Apparently TJX CEO Carol Meyrowitz (who, it bears mentioning, was not CEO at the time that the breach occurred) apologized for The Biggest Data Breach Ever at a shareholder meeting earlier this week. Unfortunately the entirety of the meeting is not online yet, but boston.com quotes her as saying,
"But we had locks."
I'm forced to assume that this is a metaphor, not meant to be taken literally. You know, cuz shareholders are dumb and don't understand words like "authentication" or "encryption." So reading between the lines here, Meyrowitz is contradicting what a well-respected Gartner source has said about the lack of wireless security.
I hope Ms. Meyrowitz isn't offended if I don't take her word for it. Either way, it's what she apparently didn't say that bothers me. There seems to have been no talk of what steps TJX has taken or how much they've invested in improving IT security at TJX in order to reduce the risk of a second breach. I'm no stock trader, but uh...
Recommendation: Strong Sell
Wednesday, June 6, 2007
Tuesday, June 5, 2007
SIM Sizing - CPU (+ Performance Tuning)
There are a couple of places across a traditional SIM that are more susceptible to performance degradation than others. Here's the short list:
The Database
There basically two things that can beat up your database. The first is basically drowning it with INSERT's of new log data. While this will manifest as poor performance and high CPU utilization, the problem is most likely disk array write performance. The answer to that problem is most likely expensive. Sorry.
The second thing that can kill database performance is event searches. This can happen in reporting or table views or pattern discovery or even in charts and graphs. (It can also happen in correlation rule filters - more on that below.) Think of it this way; whatever means you are using to search events, especially historic events (double-especially if there's compression in the mix here) has to be translated in to some horrid, fugly SELECT statement, probably with multiple JOIN's. Use these in rules, graphs, or regularly scheduled reports and you can drown your database server in work to the point that the stuff you're actively doing is unusably slow. The answer here is a combination of giving lots of CPU to your database servers and writing smart search/filter statements.
Correlation Rules
Correlation rules are the heart & soul of SIM technology. For more on what they do, check out my old ISSA 'Intro to SIM' preso deck (PDF Link). There are a number of things that can screw you here, and I already mentioned the first one above. Writing filters that are too complex or simple filters that are too vague will come back to haunt you.
Like an IDS, you will need to tune the correlation rules that your SIM ships with. A lot of this will be about eliminating noise and false positives, just like IDS. But also like IDS, some of the tuning will be performance-related. In addition to the filters you write you will also want to think about things like the number of events to match on, time frame (how long to wait for event2 after event1 occurs), etc. A cool thing that ArcSight includes is a real-time graph partial rule matches. In the example below, you can see there are two rules there that need tweaking and will probably free up measurable memory and CPU cycles once I do.
One last tip on rules and performance: If your rule creates a new meta-event, make certain that the new event does not match its own correlation filter. Trust me on this one. It's worth the extra time to double-check before turning it on.
Log Agents/Parsers/Handlers
The final place where CPU load can grow quickly is your log collection points. Somewhere between the log source and the database is code that your SIM uses to convert the log from its original format to a standard format for insertion into the database. The frequency with which log entries hit this code can have an impact on performance. This is where all of the regex matching, sorting, asset category searching, severity calculation, and so on occurs. For well-defined log formats and sources (like firewalls), this tends not to be that intense a process since you have very little diversity to be handled. But for UNIX servers with a variety of services running, there is the potential for serious friction as these parsers try and figure out what the actual log source is and what the message means.
If you have something like a UNIX a server farm that generates thousands of events per second and you want to push it through your SIM, you will need to spread this load out or buy big hardware to handle it.
The Database
There basically two things that can beat up your database. The first is basically drowning it with INSERT's of new log data. While this will manifest as poor performance and high CPU utilization, the problem is most likely disk array write performance. The answer to that problem is most likely expensive. Sorry.
The second thing that can kill database performance is event searches. This can happen in reporting or table views or pattern discovery or even in charts and graphs. (It can also happen in correlation rule filters - more on that below.) Think of it this way; whatever means you are using to search events, especially historic events (double-especially if there's compression in the mix here) has to be translated in to some horrid, fugly SELECT statement, probably with multiple JOIN's. Use these in rules, graphs, or regularly scheduled reports and you can drown your database server in work to the point that the stuff you're actively doing is unusably slow. The answer here is a combination of giving lots of CPU to your database servers and writing smart search/filter statements.
Correlation Rules
Correlation rules are the heart & soul of SIM technology. For more on what they do, check out my old ISSA 'Intro to SIM' preso deck (PDF Link). There are a number of things that can screw you here, and I already mentioned the first one above. Writing filters that are too complex or simple filters that are too vague will come back to haunt you.
Like an IDS, you will need to tune the correlation rules that your SIM ships with. A lot of this will be about eliminating noise and false positives, just like IDS. But also like IDS, some of the tuning will be performance-related. In addition to the filters you write you will also want to think about things like the number of events to match on, time frame (how long to wait for event2 after event1 occurs), etc. A cool thing that ArcSight includes is a real-time graph partial rule matches. In the example below, you can see there are two rules there that need tweaking and will probably free up measurable memory and CPU cycles once I do.
One last tip on rules and performance: If your rule creates a new meta-event, make certain that the new event does not match its own correlation filter. Trust me on this one. It's worth the extra time to double-check before turning it on.
Log Agents/Parsers/Handlers
The final place where CPU load can grow quickly is your log collection points. Somewhere between the log source and the database is code that your SIM uses to convert the log from its original format to a standard format for insertion into the database. The frequency with which log entries hit this code can have an impact on performance. This is where all of the regex matching, sorting, asset category searching, severity calculation, and so on occurs. For well-defined log formats and sources (like firewalls), this tends not to be that intense a process since you have very little diversity to be handled. But for UNIX servers with a variety of services running, there is the potential for serious friction as these parsers try and figure out what the actual log source is and what the message means.
If you have something like a UNIX a server farm that generates thousands of events per second and you want to push it through your SIM, you will need to spread this load out or buy big hardware to handle it.
Reader Mail
Anonymous writes,
"Do you use their Log Management product Logger as well? If so (or if not), what do you see as the differenc [sic] between ESM and Logger?"
We're working on bringing in a demo unit of the ArcSight Logger appliance, but do not have it deployed today. I've read the cut sheets (PDF Link) and sat through the Webex for Logger, so I can sort of answer your question.
Think of ArcSight Logger like Snare plus syslog-ng plus a Google search appliance. You can shovel common event streams into it either using native means (like syslog) or using the ArcSight Connector agents. Logger can then feed events based on boolean filters (like those used in the main ESM product) "upstream" to the ESM product.
Unlike ArcSight ESM, Logger only lets you search & sort events. There's none of the visualization or correlation that ESM has. There's also none of the reporting, case management, asset data, patterns, and so on and so forth.
We're looking at Logger to possibly fulfill two roles. The first is that I'd like to reduce some of the redundancy we currently have with software agents. Having one box grab data from multiple sources and feed it to the ESM Manager server would simplify things on my end a lot. It would also eliminate one of the few remaining points where data loss could occur during downtime of any of our ESM component servers. The second is that I'd like to give our operations and engineering teams easy access to log data without having to deploy and support ESM Consoles for all of them. As I point out in my next post on SIM sizing, one of the places you are likely to encounter performance problems is the database and people searching for events. Offloading some of that through Logger will hopefully save us from having to spend more on database server hardware down the road.
I hope that answers your question.
"Do you use their Log Management product Logger as well? If so (or if not), what do you see as the differenc [sic] between ESM and Logger?"
We're working on bringing in a demo unit of the ArcSight Logger appliance, but do not have it deployed today. I've read the cut sheets (PDF Link) and sat through the Webex for Logger, so I can sort of answer your question.
Think of ArcSight Logger like Snare plus syslog-ng plus a Google search appliance. You can shovel common event streams into it either using native means (like syslog) or using the ArcSight Connector agents. Logger can then feed events based on boolean filters (like those used in the main ESM product) "upstream" to the ESM product.
Unlike ArcSight ESM, Logger only lets you search & sort events. There's none of the visualization or correlation that ESM has. There's also none of the reporting, case management, asset data, patterns, and so on and so forth.
We're looking at Logger to possibly fulfill two roles. The first is that I'd like to reduce some of the redundancy we currently have with software agents. Having one box grab data from multiple sources and feed it to the ESM Manager server would simplify things on my end a lot. It would also eliminate one of the few remaining points where data loss could occur during downtime of any of our ESM component servers. The second is that I'd like to give our operations and engineering teams easy access to log data without having to deploy and support ESM Consoles for all of them. As I point out in my next post on SIM sizing, one of the places you are likely to encounter performance problems is the database and people searching for events. Offloading some of that through Logger will hopefully save us from having to spend more on database server hardware down the road.
I hope that answers your question.
Thursday, May 31, 2007
SIM Sizing
This is another topic I'm lifting from LinkedIn Answers. How do you properly size hardware for a SIM implementation and/or growth? I won't lie, I don't have the answer. I have been caught off guard by storage and performance issues in the SIM environment I work with. It's surprising how seemingly little things can have a huge impact on log volume and how that cascades into performance impact across your SIM.
When sizing any application, you are defining requirements for disk storage, disk performance, RAM, and CPU. So I will break this topic up into four posts, one dealing with each specific issue.
Disk Storage
This is probably the single hardest issue to tackle, and is most effected by aggregate event volume and will absolutely change at least once over the course of a year. This is also affected by your retention policy. You'll have to decide how long data stays in the SIM's tables before being shuffled loose this mortal coil.
If you don't work with a SIM already, you may wonder why folks that work with SIM talk about 'events' instead of 'log entries.' Aside from the vague philosophical issues around how many log entries make an event or how many events are documented in a log entry, the answer is simple - your SIM generates new events. This becomes a new sizing problem, "If I throw 1M firewall log entries and 500K EventLog entries at the SIM on a daily basis, how many new events will the SIM generate?" So now you're starting to get an idea of why SIM sizing is tricky. You can't just get by on `wc -l /var/log/messages` when trying to figure out how much storage you need.
Also, your log sources are going to change. Yes, you will probably fall in love with your SIM and want to put everything in it. But even if you don't your logs will change on you. Software updates to your servers or firmware upgrades to your embedded devices can and will change what and how data is logged. A recent real-life example is when we replaced our old content filtering solution for a new one that uses the firewall for enforcement. This didn't increase the number of firewall events we received at all. But it added all sorts of data about URLs and users to the traffic that was subject to content filtering. This doubled the byte count of about 35% of all firewall events inserted into the database. Surprise!
My advice is plan to expand. We're talking SAN-attached storage, volumes that can be resized online, and so on. Also plan to monitor table stats on a regular basis. You want to know before you run out of space that you need to expand.
When sizing any application, you are defining requirements for disk storage, disk performance, RAM, and CPU. So I will break this topic up into four posts, one dealing with each specific issue.
Disk Storage
This is probably the single hardest issue to tackle, and is most effected by aggregate event volume and will absolutely change at least once over the course of a year. This is also affected by your retention policy. You'll have to decide how long data stays in the SIM's tables before being shuffled loose this mortal coil.
If you don't work with a SIM already, you may wonder why folks that work with SIM talk about 'events' instead of 'log entries.' Aside from the vague philosophical issues around how many log entries make an event or how many events are documented in a log entry, the answer is simple - your SIM generates new events. This becomes a new sizing problem, "If I throw 1M firewall log entries and 500K EventLog entries at the SIM on a daily basis, how many new events will the SIM generate?" So now you're starting to get an idea of why SIM sizing is tricky. You can't just get by on `wc -l /var/log/messages` when trying to figure out how much storage you need.
Also, your log sources are going to change. Yes, you will probably fall in love with your SIM and want to put everything in it. But even if you don't your logs will change on you. Software updates to your servers or firmware upgrades to your embedded devices can and will change what and how data is logged. A recent real-life example is when we replaced our old content filtering solution for a new one that uses the firewall for enforcement. This didn't increase the number of firewall events we received at all. But it added all sorts of data about URLs and users to the traffic that was subject to content filtering. This doubled the byte count of about 35% of all firewall events inserted into the database. Surprise!
My advice is plan to expand. We're talking SAN-attached storage, volumes that can be resized online, and so on. Also plan to monitor table stats on a regular basis. You want to know before you run out of space that you need to expand.
Thursday, May 24, 2007
Like an Orange on a Toothpick
My ego's going to have to do some sit-ups this weekend. It's getting huge. I'm on CNN via the WOODTV WiFi Arrest story.
Supastah!
Supastah!
Un-believ-able
It's hack-ish (not meaning hacker-ish) to pick on Oracle for their "unbreakable" branding claim. But until Oracle gets to a place where they can fix buffer overflows in less than a year and XSS in less than 4 years, they really need to put a muzzle on their people when it comes to talking publicly about security.
I say this because the un-flapp-able Mary Ann Davidson gave the keynote at AusCERT 2007 and - I kid you not - compared software to US Marines. I don't think I disagree with the spirit of Mary Ann's point in her speech, but the irony of the situation is overwhelming. Seriously, either she went rogue and hoped nobody would notice, or Oracle needs new PR people. Someone should've talked her out of this.
Oracle is easily the least cooperative of the big vendors when it comes to security. Sure, Apple's been vilified recently for playing hardball with security researchers, but at least they release patches! Oracle's name is mud with researchers and bug reporters - just ask David Litchfield. (PDF link) And given their reputation, one they've spent the past decade earning, Mary Ann Davidson saying,
"Why do we need all these [security] products in the first place? Because software can't defend itself."
or,
"You are going to have to have some kind of proof that you paid attention in development - even to the level of training people and what kind of software lifecycle you have."
...is somewhere between hilarious and offensive. Before Oracle officers go around touting vendor-driven defenses, perhaps they ought to spend a little time talking about investing in software QA & bugfix processes and resources. This argument is already over. Microsoft has spent the past 5 years showing the world that you can solve security problems by throwing money at them. So Oracle, it's time to take your own medicine and step up.
In other words, it's time for Oracle to clean up their own backyard and Mary Ann Davidson needs to get the hell off my porch.
I say this because the un-flapp-able Mary Ann Davidson gave the keynote at AusCERT 2007 and - I kid you not - compared software to US Marines. I don't think I disagree with the spirit of Mary Ann's point in her speech, but the irony of the situation is overwhelming. Seriously, either she went rogue and hoped nobody would notice, or Oracle needs new PR people. Someone should've talked her out of this.
Oracle is easily the least cooperative of the big vendors when it comes to security. Sure, Apple's been vilified recently for playing hardball with security researchers, but at least they release patches! Oracle's name is mud with researchers and bug reporters - just ask David Litchfield. (PDF link) And given their reputation, one they've spent the past decade earning, Mary Ann Davidson saying,
"Why do we need all these [security] products in the first place? Because software can't defend itself."
or,
"You are going to have to have some kind of proof that you paid attention in development - even to the level of training people and what kind of software lifecycle you have."
...is somewhere between hilarious and offensive. Before Oracle officers go around touting vendor-driven defenses, perhaps they ought to spend a little time talking about investing in software QA & bugfix processes and resources. This argument is already over. Microsoft has spent the past 5 years showing the world that you can solve security problems by throwing money at them. So Oracle, it's time to take your own medicine and step up.
In other words, it's time for Oracle to clean up their own backyard and Mary Ann Davidson needs to get the hell off my porch.
Wednesday, May 23, 2007
Sparta WiFi Arrest
Now that it's on Slashdot, you've probably heard about a Sparta man that was arrested on a felony warrant for using a coffee shop's free, open wireless network without patronizing said coffee shop. The scoop on this incident started with a WOODTV story (and they reused some footage from February, so I was on TV again, which my ego liked!) To be honest, I have done this very thing many times before. My favorite bar in East Lansing has a patio and is next door to a coffee shop with free WiFi. On a beautiful summer afternoon, the choice between coffee and beer is about as easy as they come.
Anyway, this poor guy got saddled with a $400 fine and 40hrs of community service for doing the electronic equivalent of using the bathroom without buying something. According to the law, it could've been $5K and a year in jail, because the underlying crime would've been trespass. At least, that's my layman's understanding of the law. And according to the arrest warrant the judge signed, it could've been $10K and 5 years in the pokey.
I don't see a positive here. I just don't. This incident:
1. has wasted taxpayer money on a benign and trivial incident.
2. will confuse law enforcement further about where the lines around this law are drawn.
3. has victimized a guy that was too cheap to buy a cup of coffee in order to check his e-mail. Seriously, if the guy didn't want to spend $4 for a latte, where's the justice in taking $400 from him for something that neither he nor the coffee shop owner thought to be a crime?
4. has raised awareness of laptop owners that they're responsible for the wireless networks they connect to regardless of their understanding of the technology. If this has any effect, it will be a chilling one, giving people one more thing to worry about when it comes to open WiFi hotspots.
5. will perhaps even be referenced in the future to let businesses off the hook for not securing their business wireless networks.
6. illustrates the enourmous chasm separating the law and lawyers from the understanding of technology.
Anyway, this poor guy got saddled with a $400 fine and 40hrs of community service for doing the electronic equivalent of using the bathroom without buying something. According to the law, it could've been $5K and a year in jail, because the underlying crime would've been trespass. At least, that's my layman's understanding of the law. And according to the arrest warrant the judge signed, it could've been $10K and 5 years in the pokey.
I don't see a positive here. I just don't. This incident:
1. has wasted taxpayer money on a benign and trivial incident.
2. will confuse law enforcement further about where the lines around this law are drawn.
3. has victimized a guy that was too cheap to buy a cup of coffee in order to check his e-mail. Seriously, if the guy didn't want to spend $4 for a latte, where's the justice in taking $400 from him for something that neither he nor the coffee shop owner thought to be a crime?
4. has raised awareness of laptop owners that they're responsible for the wireless networks they connect to regardless of their understanding of the technology. If this has any effect, it will be a chilling one, giving people one more thing to worry about when it comes to open WiFi hotspots.
5. will perhaps even be referenced in the future to let businesses off the hook for not securing their business wireless networks.
6. illustrates the enourmous chasm separating the law and lawyers from the understanding of technology.
Subscribe to:
Posts (Atom)