Viruses that invade the brain during the first trimester cause devastating loss of neurological function. The later you go in the pregnancy, the more likely it is to cause less disease. But, even when you're infected during delivery, babies always have hearing problems, visual problems, and even cognitive problems.