Logging Stories from the Field

jzhen's picture

Our brilliant field engineer, Dimitri McKay (his blog) brings another fun and insightful story from the field:  "I recently went on-site for a proof of concept. I’ve always loved these exercises, as it gives me a chance to help a customer see that which was invisible in the past, whether it be virus-outbreaks, users abusing bandwidth via bit-torrent and file sharing, or VOIP phones assaulting DHCP servers for IP addresses. This particular customer had an interesting configuration.  They had been sending their critical/alert and emergency firewall logs to a 3rd party security operations center. That SOC was supposed to monitor the firewall data for any risky traffic, identify any anomalies, and report the instant there’s an issue. Because of this, we wanted to forward those specific messages on to the 3rd party security operations center. We configured the LogLogic appliance to collect ALL firewall messages, and send off all Emergency, Critical and Alert messages to that 3rd party vendor.  Over the next couple hours, we turned routed stream after stream of syslog data to the LogLogic appliance. Slowly raising our MPS rates, collecting data for alerting, collecting data for reporting; overall collecting data to sift and sort through. We cranked the logging levels way up. Our goal was to "abuse" the box, and get as much out of this POC as possible. The firewalls, routers, switches, and Unix/Linux boxes were logging; we then added several dozen Windows hosts logging via Lasso. Now came the fun part. We then started drilling down into the data. I illustrated to our customer the agile reports, we ran searches across a mountain of data, and I showed the onlookers what interesting information we could mine from that mass of daily log data.  At one point, I ran an FTP report. Suddenly there were questions: "Why are there like 450 active FTP connections from Germany?" Which led to more questions: who is that? what are they attempting to do? Within several minutes, we were able to see that an FTP server had been left wide open in the DMZ with 'anonymous' logins allowed. We were also able to see the file names being uploaded/downloaded via the firewall port 21 traffic logs.  Next, we began looking at the logs from the compromised server itself.  We were able to see that the server that had been compromised had been actually compromised back in March. The logs also revealed that they attempted a dictionary attack (over nearly 3 months) hoping to get access to the box, and the fear was: if they accessed to the box, and ran something like l0ftcrack on it, the box had once been part of the domain, and so user accounts and passwords could be revealed.  The box was a virtual machine "test box" that a developer had fired up, added to the domain, didn't harden, and had then transferred it to the DMZ violating a half dozen or more security protocols.  In that same DMZ was their email system, which likely had the same login/pass combo, and if that was compromised, who knows. At the end of the day we were able to identify a compromised machine, make the security officer look like a superstar, and illustrate just how fast and agile our reports and search capabilities are. If we didn't prove that LogLogic is the clear best fit, we certainly created a perfect use case for Log Management. " Enjoy more of Dimitri's writing on his blog!