My wife is pretty cool. She's one of the new generation of self-employed eBay mom's. But even before she started an eBay biz, she was a bargain shopper. One of the best. And I don't mean, "Look what I got for 35% off at Macy's!" I mean, "Look what I found for 99% off on the clearance rack!" Seriously - if bargains are ancient treasure, I married Indiana Jones.
Anyway, somebody over at SonicWall put together a deal that would make my wife very proud. Check it out: SonicWall bought Aventail this week for $25M cash. I don't write much about the firewall / VPN market, but if you know me you know that I've been doing firewall stuff for the past decade, and I still keep an eye on it. SonicWall is really only a player in SMB/SOHO market. Their enterprise stuff isn't even on the radar where Cisco, Juniper, and Check Point are concerned. But the weird thing about Aventail is that every time I talk to a client or a colleague about SSL VPN, Aventail keeps coming up. They're a small vendor, but they have a slick product, and they do well in trade press comparos. So the fact that they were purchased comes as no surprise. And with this deal, SonicWall just bought their way back into the market for practically pennies. Nice.
Plus, it's about time we saw an acquisition that made sense. It seems like the last 9mos have been nothing but mismatched purchases.
Thursday, June 14, 2007
Wednesday, June 13, 2007
Malware Hunting for Lazy Idiots
What I want to write about right now is a couple of neat new tricks I figured out while tracking down some malware today. It's JavaScript de-obfuscation and packed executables that won't run in SysAnalyzer. Fun! Especially since this time I was successful, front-to-back. I think I may still post a detailed description. But not right now. I haven't got time. Yards don't mow themselves.
But what I did want to mention was that Niels Provos has released a new version of SpyBye. The cool part is that it built on my Mac with zero difficulty. Ironically, it builds on OpenBSD, but dumps core complaining about an unrecognized symbol from libevent. What's cool about the new SpyBye is the '-x' switch that lets you run it as an interactive proxy from your browser. This makes analysis easier to do since you can let the browser step through the scripts and iframes and all that crap that sucks up your time when trying to do it manually. What's not that cool about SpyBye are the good_patterns/bad_patterns files, and how limited and basic the content is. For example, none of the aforementioned malware or its droppers set off SpyBye. Moreover, the exploit still works on vulnerable browsers. I've been playing with adding regex patterns to the bad_patterns file to get it to detect all of the known-bad stuff I already have Snort signatures for. Once that's done maybe I'll post them up here or e-mail them to Niels or something.
The reason I love the idea of SpyBye, HoneyC, or any other honeyclient-ish tool is that they all imply an easy way to find new malware that your IDS and AV don't already stop. It sounds so much easier than rifling through proxy logs or tcpdump payloads looking for "stuff that don't look right." But I always have a little bit of buyer's (OK, compiler's?) remorse when I get something like SpyBye up and running. Sure, it works the way it was written to work, but it doesn't magically find lots of cool zero-day browser sploits in the wild. Even when I know they're there. I guess finding the "really cool" malware is still hard and requires some luck. Which sucks, because like the title says, I would much rather be a lazy idiot.
But what I did want to mention was that Niels Provos has released a new version of SpyBye. The cool part is that it built on my Mac with zero difficulty. Ironically, it builds on OpenBSD, but dumps core complaining about an unrecognized symbol from libevent. What's cool about the new SpyBye is the '-x' switch that lets you run it as an interactive proxy from your browser. This makes analysis easier to do since you can let the browser step through the scripts and iframes and all that crap that sucks up your time when trying to do it manually. What's not that cool about SpyBye are the good_patterns/bad_patterns files, and how limited and basic the content is. For example, none of the aforementioned malware or its droppers set off SpyBye. Moreover, the exploit still works on vulnerable browsers. I've been playing with adding regex patterns to the bad_patterns file to get it to detect all of the known-bad stuff I already have Snort signatures for. Once that's done maybe I'll post them up here or e-mail them to Niels or something.
The reason I love the idea of SpyBye, HoneyC, or any other honeyclient-ish tool is that they all imply an easy way to find new malware that your IDS and AV don't already stop. It sounds so much easier than rifling through proxy logs or tcpdump payloads looking for "stuff that don't look right." But I always have a little bit of buyer's (OK, compiler's?) remorse when I get something like SpyBye up and running. Sure, it works the way it was written to work, but it doesn't magically find lots of cool zero-day browser sploits in the wild. Even when I know they're there. I guess finding the "really cool" malware is still hard and requires some luck. Which sucks, because like the title says, I would much rather be a lazy idiot.
SIM Sizing - RAM
Sorry for the pregnant pause on this topic. We went camping this weekend and my brain, well, rebooted. Good for me and my blood pressure. Bad for anyone waiting for the final episode of this gripping trilogy.
Anyhoo, there are basically two places you need to plan for RAM expansion in your SIM. Again, the database. Not much of a surprise. The two drivers here are the log frequency as this will cause the database to store INSERTs into RAM while waiting for disk. This problem will be obvious as your performance will suffer noticeably. The other thing that can cause the amount of RAM your database is using to increase are standard things like the number of simultaneous users or the number of simultaneous reports that run. Also reports or table views (ArcSight calls these 'active channels') that look back through weeks or months worth of events will definitely burn RAM on the database server.
The other component that can run into RAM problems is the point on your SIM where correlation rules run. The reason correlation rules eat memory is because they are based on matching 2 or more events over a window of time. This is not unlike the state tracking table on your firewall.
Let's say you have a rule that looks for a firewall allow message and an IDS alert that have the same source IP address. Then let's say that that rule has a 2 minute window because you have time sync issues. Your SIM is going to track every firewall allow message and every IDS alert that match your filter for 2 minutes, comparing each new IDS event to each firewall event in the table as well as comparing each new firewall event to each IDS event in the table. Now imagine someone fat-fingers that rule or you have really bad time sync issues and you set that window to 20 minutes. I think you get the idea.
The solution to keeping this from spiraling out of control is simply:
1) Write correlation rule filters as simply and precisely as possible.
2) NTP!@ (If you haven't already. This is important for your SIM to function well for so many reasons.)
3) Since you have time sync figured out now, use small windows in your correlation rules.
Anyhoo, there are basically two places you need to plan for RAM expansion in your SIM. Again, the database. Not much of a surprise. The two drivers here are the log frequency as this will cause the database to store INSERTs into RAM while waiting for disk. This problem will be obvious as your performance will suffer noticeably. The other thing that can cause the amount of RAM your database is using to increase are standard things like the number of simultaneous users or the number of simultaneous reports that run. Also reports or table views (ArcSight calls these 'active channels') that look back through weeks or months worth of events will definitely burn RAM on the database server.
The other component that can run into RAM problems is the point on your SIM where correlation rules run. The reason correlation rules eat memory is because they are based on matching 2 or more events over a window of time. This is not unlike the state tracking table on your firewall.
Let's say you have a rule that looks for a firewall allow message and an IDS alert that have the same source IP address. Then let's say that that rule has a 2 minute window because you have time sync issues. Your SIM is going to track every firewall allow message and every IDS alert that match your filter for 2 minutes, comparing each new IDS event to each firewall event in the table as well as comparing each new firewall event to each IDS event in the table. Now imagine someone fat-fingers that rule or you have really bad time sync issues and you set that window to 20 minutes. I think you get the idea.
The solution to keeping this from spiraling out of control is simply:
1) Write correlation rule filters as simply and precisely as possible.
2) NTP!@ (If you haven't already. This is important for your SIM to function well for so many reasons.)
3) Since you have time sync figured out now, use small windows in your correlation rules.
Thursday, June 7, 2007
June GRSec Announced
GRSec will be getting together over the summer as promised. Note the move to Tuesday night. I've heard from several people that Wednesdays and Thursdays don't work. So we're looking forward to some new faces. Hope to see you there!
Wednesday, June 6, 2007
TJX CEO Apology
Apparently TJX CEO Carol Meyrowitz (who, it bears mentioning, was not CEO at the time that the breach occurred) apologized for The Biggest Data Breach Ever at a shareholder meeting earlier this week. Unfortunately the entirety of the meeting is not online yet, but boston.com quotes her as saying,
"But we had locks."
I'm forced to assume that this is a metaphor, not meant to be taken literally. You know, cuz shareholders are dumb and don't understand words like "authentication" or "encryption." So reading between the lines here, Meyrowitz is contradicting what a well-respected Gartner source has said about the lack of wireless security.
I hope Ms. Meyrowitz isn't offended if I don't take her word for it. Either way, it's what she apparently didn't say that bothers me. There seems to have been no talk of what steps TJX has taken or how much they've invested in improving IT security at TJX in order to reduce the risk of a second breach. I'm no stock trader, but uh...
Recommendation: Strong Sell
"But we had locks."
I'm forced to assume that this is a metaphor, not meant to be taken literally. You know, cuz shareholders are dumb and don't understand words like "authentication" or "encryption." So reading between the lines here, Meyrowitz is contradicting what a well-respected Gartner source has said about the lack of wireless security.
I hope Ms. Meyrowitz isn't offended if I don't take her word for it. Either way, it's what she apparently didn't say that bothers me. There seems to have been no talk of what steps TJX has taken or how much they've invested in improving IT security at TJX in order to reduce the risk of a second breach. I'm no stock trader, but uh...
Recommendation: Strong Sell
Tuesday, June 5, 2007
SIM Sizing - CPU (+ Performance Tuning)
There are a couple of places across a traditional SIM that are more susceptible to performance degradation than others. Here's the short list:
The Database
There basically two things that can beat up your database. The first is basically drowning it with INSERT's of new log data. While this will manifest as poor performance and high CPU utilization, the problem is most likely disk array write performance. The answer to that problem is most likely expensive. Sorry.
The second thing that can kill database performance is event searches. This can happen in reporting or table views or pattern discovery or even in charts and graphs. (It can also happen in correlation rule filters - more on that below.) Think of it this way; whatever means you are using to search events, especially historic events (double-especially if there's compression in the mix here) has to be translated in to some horrid, fugly SELECT statement, probably with multiple JOIN's. Use these in rules, graphs, or regularly scheduled reports and you can drown your database server in work to the point that the stuff you're actively doing is unusably slow. The answer here is a combination of giving lots of CPU to your database servers and writing smart search/filter statements.
Correlation Rules
Correlation rules are the heart & soul of SIM technology. For more on what they do, check out my old ISSA 'Intro to SIM' preso deck (PDF Link). There are a number of things that can screw you here, and I already mentioned the first one above. Writing filters that are too complex or simple filters that are too vague will come back to haunt you.
Like an IDS, you will need to tune the correlation rules that your SIM ships with. A lot of this will be about eliminating noise and false positives, just like IDS. But also like IDS, some of the tuning will be performance-related. In addition to the filters you write you will also want to think about things like the number of events to match on, time frame (how long to wait for event2 after event1 occurs), etc. A cool thing that ArcSight includes is a real-time graph partial rule matches. In the example below, you can see there are two rules there that need tweaking and will probably free up measurable memory and CPU cycles once I do.
One last tip on rules and performance: If your rule creates a new meta-event, make certain that the new event does not match its own correlation filter. Trust me on this one. It's worth the extra time to double-check before turning it on.
Log Agents/Parsers/Handlers
The final place where CPU load can grow quickly is your log collection points. Somewhere between the log source and the database is code that your SIM uses to convert the log from its original format to a standard format for insertion into the database. The frequency with which log entries hit this code can have an impact on performance. This is where all of the regex matching, sorting, asset category searching, severity calculation, and so on occurs. For well-defined log formats and sources (like firewalls), this tends not to be that intense a process since you have very little diversity to be handled. But for UNIX servers with a variety of services running, there is the potential for serious friction as these parsers try and figure out what the actual log source is and what the message means.
If you have something like a UNIX a server farm that generates thousands of events per second and you want to push it through your SIM, you will need to spread this load out or buy big hardware to handle it.
The Database
There basically two things that can beat up your database. The first is basically drowning it with INSERT's of new log data. While this will manifest as poor performance and high CPU utilization, the problem is most likely disk array write performance. The answer to that problem is most likely expensive. Sorry.
The second thing that can kill database performance is event searches. This can happen in reporting or table views or pattern discovery or even in charts and graphs. (It can also happen in correlation rule filters - more on that below.) Think of it this way; whatever means you are using to search events, especially historic events (double-especially if there's compression in the mix here) has to be translated in to some horrid, fugly SELECT statement, probably with multiple JOIN's. Use these in rules, graphs, or regularly scheduled reports and you can drown your database server in work to the point that the stuff you're actively doing is unusably slow. The answer here is a combination of giving lots of CPU to your database servers and writing smart search/filter statements.
Correlation Rules
Correlation rules are the heart & soul of SIM technology. For more on what they do, check out my old ISSA 'Intro to SIM' preso deck (PDF Link). There are a number of things that can screw you here, and I already mentioned the first one above. Writing filters that are too complex or simple filters that are too vague will come back to haunt you.
Like an IDS, you will need to tune the correlation rules that your SIM ships with. A lot of this will be about eliminating noise and false positives, just like IDS. But also like IDS, some of the tuning will be performance-related. In addition to the filters you write you will also want to think about things like the number of events to match on, time frame (how long to wait for event2 after event1 occurs), etc. A cool thing that ArcSight includes is a real-time graph partial rule matches. In the example below, you can see there are two rules there that need tweaking and will probably free up measurable memory and CPU cycles once I do.
One last tip on rules and performance: If your rule creates a new meta-event, make certain that the new event does not match its own correlation filter. Trust me on this one. It's worth the extra time to double-check before turning it on.
Log Agents/Parsers/Handlers
The final place where CPU load can grow quickly is your log collection points. Somewhere between the log source and the database is code that your SIM uses to convert the log from its original format to a standard format for insertion into the database. The frequency with which log entries hit this code can have an impact on performance. This is where all of the regex matching, sorting, asset category searching, severity calculation, and so on occurs. For well-defined log formats and sources (like firewalls), this tends not to be that intense a process since you have very little diversity to be handled. But for UNIX servers with a variety of services running, there is the potential for serious friction as these parsers try and figure out what the actual log source is and what the message means.
If you have something like a UNIX a server farm that generates thousands of events per second and you want to push it through your SIM, you will need to spread this load out or buy big hardware to handle it.
Reader Mail
Anonymous writes,
"Do you use their Log Management product Logger as well? If so (or if not), what do you see as the differenc [sic] between ESM and Logger?"
We're working on bringing in a demo unit of the ArcSight Logger appliance, but do not have it deployed today. I've read the cut sheets (PDF Link) and sat through the Webex for Logger, so I can sort of answer your question.
Think of ArcSight Logger like Snare plus syslog-ng plus a Google search appliance. You can shovel common event streams into it either using native means (like syslog) or using the ArcSight Connector agents. Logger can then feed events based on boolean filters (like those used in the main ESM product) "upstream" to the ESM product.
Unlike ArcSight ESM, Logger only lets you search & sort events. There's none of the visualization or correlation that ESM has. There's also none of the reporting, case management, asset data, patterns, and so on and so forth.
We're looking at Logger to possibly fulfill two roles. The first is that I'd like to reduce some of the redundancy we currently have with software agents. Having one box grab data from multiple sources and feed it to the ESM Manager server would simplify things on my end a lot. It would also eliminate one of the few remaining points where data loss could occur during downtime of any of our ESM component servers. The second is that I'd like to give our operations and engineering teams easy access to log data without having to deploy and support ESM Consoles for all of them. As I point out in my next post on SIM sizing, one of the places you are likely to encounter performance problems is the database and people searching for events. Offloading some of that through Logger will hopefully save us from having to spend more on database server hardware down the road.
I hope that answers your question.
"Do you use their Log Management product Logger as well? If so (or if not), what do you see as the differenc [sic] between ESM and Logger?"
We're working on bringing in a demo unit of the ArcSight Logger appliance, but do not have it deployed today. I've read the cut sheets (PDF Link) and sat through the Webex for Logger, so I can sort of answer your question.
Think of ArcSight Logger like Snare plus syslog-ng plus a Google search appliance. You can shovel common event streams into it either using native means (like syslog) or using the ArcSight Connector agents. Logger can then feed events based on boolean filters (like those used in the main ESM product) "upstream" to the ESM product.
Unlike ArcSight ESM, Logger only lets you search & sort events. There's none of the visualization or correlation that ESM has. There's also none of the reporting, case management, asset data, patterns, and so on and so forth.
We're looking at Logger to possibly fulfill two roles. The first is that I'd like to reduce some of the redundancy we currently have with software agents. Having one box grab data from multiple sources and feed it to the ESM Manager server would simplify things on my end a lot. It would also eliminate one of the few remaining points where data loss could occur during downtime of any of our ESM component servers. The second is that I'd like to give our operations and engineering teams easy access to log data without having to deploy and support ESM Consoles for all of them. As I point out in my next post on SIM sizing, one of the places you are likely to encounter performance problems is the database and people searching for events. Offloading some of that through Logger will hopefully save us from having to spend more on database server hardware down the road.
I hope that answers your question.
Subscribe to:
Posts (Atom)