Crossposted from: http://disillusion.us

I don’t know how many parts this is going to go, but here we go anyway: So I’ve been talking about adding a city government to our network for a month or so now, and we finally got them added to our network last week on Wednesday. Yay!

However, when I was over there Wednesday night moving the clients from local accounts to domain accounts I noticed something. Not a big deal, it was just a virus message, w32.downadup.b, while I understand the virus is a big deal, I figured Symantec Endpoint would catch it and eliminate it. Little did I know, I was way wrong.

I was on call this week, and I knew there was going to be problems with this starting Thursday night. I have Friday’s off and it’s been a quiet week so far, so I thought I would have a relaxing night… till I got phonecalls from Jane. Everyone was getting virus messages. It started with Dispatch; the computers were rendered almost completely useless because Symantec was working hard to prevent it. I’m not exactly sure these PCs got infected, but something is amiss. I got phone calls off and on all weekend. My boss went over there on Friday to replace one of the machines because they were unhappy with my response “Restart the computer, its the best I can tell you” I got a phone call on Saturday while playing DnD, the new PC was “infected” as well. Serves him right, that did a lot of good; half this shit started because nobody would listen to me. I fired an email off to my supervisor and boss just in case.

I got no phonecalls on Sunday except an unhappy city employee because I wouldn’t help her on my day off. She wanted my boss’s email so she could complain. Yea, I was on call, but I was told to keep overtime to a minimum. Thinking Symantec did its job, I came into believe that everything was going to be fine come Monday morning. Holy shit. I was wrong.

I came in today expecting to have an easy Monday but, I’ve never been so busy and occupied in my life. Every PC in the network got hit, including servers. I’m not sure why there was no AV on any of the servers (there was a reason, I believe) and I also thought they were patched, but that assumption was incorrect. I started out at the City Hall. I tried to google an answer to a simple question related to Outlook and I realised that I could not access any webpages that are associated with Microsoft. I tried googling things about downadup, but every article that wasn’t a press release I couldn’t access. Wat? I got into the office and through the course of time I figured out piece by piece what had happened.

We use an ISA server to filter traffic, no http traffic comes out of our network unless it passes through our ISA server, or else it gets dropped at the firewall. The ISA machine got hit, so any traffic searching for answers got dropped because downadup blocks access to certain pages based on their titles. So anything that connected to the internet at work had to connect to this server to get http. Great. Then I remembered our Ubuntu box, which bypasses the proxy so that we can apt-get easily. Sweet. I was able to do some research on the virus and removal methods, which didn’t work at all. We were at its mercy.

So our network was hit with downadup, the only way to do any research or downloading was done through a Ubuntu box. More in my next post.