Do you ever get that feeling that it is all going to pot? Or more accurately I should ask, Do you ever NOT get that feeling?!
At least it happened on a Friday, so I can fully justify my double rations of Guinness tonight. And maybe whiskey too. And I think I'll char some meat while I'm at it.
Walked into work early and bright eyed - Boss had asked me to pick him up from mechanics at dawn (?!), so I was asleep extra early last night and almost ready for what I was walking into. Tech takes one look at me and says 'Looks like email is under attack'. Should/Coulda/Woulda turned right around and used a sick day if I was a different person. Man, if only I was a different person more often!
Get to my desk, and open my 10 tabs of vendor portals - before I even get a chance to log into Barracuda, I notice heavy activity all last night on our WAN. We normally have next to zero overnight activity unless someone leaves a browser open. This is not good. Immediately thoughts go to spambot type issues. A few clicks in Fortiguard and it is clear this is from a single endpoint. I dispatch the tech to recover the computer from Engineering (why is it always the Engineers?!) and I take a deep breath.
Is this the time I fail? Is this the time I react wrong? Is this going to be a all weekend event? Do I have the skills? Can I Do This?!
Every time I've ever fought a battle, I have these thoughts. Now that I'm management I try my best to not show my team, but they are certainly still there. Confidence, Man! You Can Do This!! You've got the tools, use them.
Fortiguard confirms that removing said PC eliminated WAN traffic. Initial log analysis shows this to be Yahoo in origin... We saw some increased activity early this week from the Tech's computer to the same IP... nothing like this, but it was the same IP. Both have the latest Thunderbird - Engineer just got the update yesterday. Could it be Thunderbird?
At the same time I'm digging into email - it's off the charts - we are denying dozens of email every minute and have been since around 1am. Our mail server is configured to respond to unknown users so we are getting bounce backs from non-existent accounts that we tried to respond to. Still could this be some kind of spambot on a users system here that is relaying off yahoo maybe? Hop on to WHM (exim) and after running a few mail delivery reports it looks like we are not sending the spam - at least I don't have to deal with getting Blacklisted. All mail protocols are filtered by the firewall, so I shouldn't have to worry about an internally hosted mail server - that mail would be dropped before getting out to the net.
I decide to bring in the help of my domain host, LiquidWeb. They have some pretty spectacular (linux) support (likely windows too, but ewww!). LW Tech is brought up to date and he is off looking for signs of trouble on my server.
At this point I'm feeling a little better... It looks like I probably do have the skills to deal with this. A run a few more reports and look in a few more places, but it seems that the two events are not related. As unlikely as that is. Still no new WAN activity, so lets plug the Engineers PC back in and see what happens - we checked that their endpoint was up and running and that all of our restrictions were in place and functioning... browser history shows aol.com and yahoo.com and nothing else... another hit on Yahoo. PC goes back on network and no new activity.
LW Tech gets back to the chatbox and says we are all clean. Recommends a small change to our DKIM record, but confirms what I was seeing - we are simply responding to external traffic, we are not the originators of it. I correct the DKIM (why does Network Solutions change their interface every season?) and scratch a spambot exploit off my list.
Return Engineers PC, check Barracuda. All is well. Spam storm is dying down, now maybe one every two minutes. Normal WAN activity. I go to lunch an hour late.
Return from spirited run to Chinese To-Go to find WAN is back to off the charts. Logs show hundreds (500+) connection attempts to that same yahoo owned IP every minute. Over 18GB in denied connections in 20 minutes. This thing is pounding my logs and my ability to have situational awareness. The firewall is denying all that activity, but the sheer volume of it is stunning.
Not much left to do. I create a firewall rule to drop all Yahoo. Takes a little time to find all the domains and IP's, but once I have it defined, the traffic is no more. Fortunately we do not use Yahoo for any legitimate needs. I'll try to disable the rule next week - maybe Yahoo can get there act together by then.
This is a story from Today. Let me know if you are also having issues with Yahoo, I can't believe I'm the only one.
Keep up the fight, Brothers and Sisters! You Can Do It!