This is another topic I'm lifting from LinkedIn Answers. How do you properly size hardware for a SIM implementation and/or growth? I won't lie, I don't have the answer. I have been caught off guard by storage and performance issues in the SIM environment I work with. It's surprising how seemingly little things can have a huge impact on log volume and how that cascades into performance impact across your SIM.
When sizing any application, you are defining requirements for disk storage, disk performance, RAM, and CPU. So I will break this topic up into four posts, one dealing with each specific issue.
Disk Storage
This is probably the single hardest issue to tackle, and is most effected by aggregate event volume and will absolutely change at least once over the course of a year. This is also affected by your retention policy. You'll have to decide how long data stays in the SIM's tables before being shuffled loose this mortal coil.
If you don't work with a SIM already, you may wonder why folks that work with SIM talk about 'events' instead of 'log entries.' Aside from the vague philosophical issues around how many log entries make an event or how many events are documented in a log entry, the answer is simple - your SIM generates new events. This becomes a new sizing problem, "If I throw 1M firewall log entries and 500K EventLog entries at the SIM on a daily basis, how many new events will the SIM generate?" So now you're starting to get an idea of why SIM sizing is tricky. You can't just get by on `wc -l /var/log/messages` when trying to figure out how much storage you need.
Also, your log sources are going to change. Yes, you will probably fall in love with your SIM and want to put everything in it. But even if you don't your logs will change on you. Software updates to your servers or firmware upgrades to your embedded devices can and will change what and how data is logged. A recent real-life example is when we replaced our old content filtering solution for a new one that uses the firewall for enforcement. This didn't increase the number of firewall events we received at all. But it added all sorts of data about URLs and users to the traffic that was subject to content filtering. This doubled the byte count of about 35% of all firewall events inserted into the database. Surprise!
My advice is plan to expand. We're talking SAN-attached storage, volumes that can be resized online, and so on. Also plan to monitor table stats on a regular basis. You want to know before you run out of space that you need to expand.
Thursday, May 31, 2007
Thursday, May 24, 2007
Like an Orange on a Toothpick
My ego's going to have to do some sit-ups this weekend. It's getting huge. I'm on CNN via the WOODTV WiFi Arrest story.
Supastah!
Supastah!
Un-believ-able
It's hack-ish (not meaning hacker-ish) to pick on Oracle for their "unbreakable" branding claim. But until Oracle gets to a place where they can fix buffer overflows in less than a year and XSS in less than 4 years, they really need to put a muzzle on their people when it comes to talking publicly about security.
I say this because the un-flapp-able Mary Ann Davidson gave the keynote at AusCERT 2007 and - I kid you not - compared software to US Marines. I don't think I disagree with the spirit of Mary Ann's point in her speech, but the irony of the situation is overwhelming. Seriously, either she went rogue and hoped nobody would notice, or Oracle needs new PR people. Someone should've talked her out of this.
Oracle is easily the least cooperative of the big vendors when it comes to security. Sure, Apple's been vilified recently for playing hardball with security researchers, but at least they release patches! Oracle's name is mud with researchers and bug reporters - just ask David Litchfield. (PDF link) And given their reputation, one they've spent the past decade earning, Mary Ann Davidson saying,
"Why do we need all these [security] products in the first place? Because software can't defend itself."
or,
"You are going to have to have some kind of proof that you paid attention in development - even to the level of training people and what kind of software lifecycle you have."
...is somewhere between hilarious and offensive. Before Oracle officers go around touting vendor-driven defenses, perhaps they ought to spend a little time talking about investing in software QA & bugfix processes and resources. This argument is already over. Microsoft has spent the past 5 years showing the world that you can solve security problems by throwing money at them. So Oracle, it's time to take your own medicine and step up.
In other words, it's time for Oracle to clean up their own backyard and Mary Ann Davidson needs to get the hell off my porch.
I say this because the un-flapp-able Mary Ann Davidson gave the keynote at AusCERT 2007 and - I kid you not - compared software to US Marines. I don't think I disagree with the spirit of Mary Ann's point in her speech, but the irony of the situation is overwhelming. Seriously, either she went rogue and hoped nobody would notice, or Oracle needs new PR people. Someone should've talked her out of this.
Oracle is easily the least cooperative of the big vendors when it comes to security. Sure, Apple's been vilified recently for playing hardball with security researchers, but at least they release patches! Oracle's name is mud with researchers and bug reporters - just ask David Litchfield. (PDF link) And given their reputation, one they've spent the past decade earning, Mary Ann Davidson saying,
"Why do we need all these [security] products in the first place? Because software can't defend itself."
or,
"You are going to have to have some kind of proof that you paid attention in development - even to the level of training people and what kind of software lifecycle you have."
...is somewhere between hilarious and offensive. Before Oracle officers go around touting vendor-driven defenses, perhaps they ought to spend a little time talking about investing in software QA & bugfix processes and resources. This argument is already over. Microsoft has spent the past 5 years showing the world that you can solve security problems by throwing money at them. So Oracle, it's time to take your own medicine and step up.
In other words, it's time for Oracle to clean up their own backyard and Mary Ann Davidson needs to get the hell off my porch.
Wednesday, May 23, 2007
Sparta WiFi Arrest
Now that it's on Slashdot, you've probably heard about a Sparta man that was arrested on a felony warrant for using a coffee shop's free, open wireless network without patronizing said coffee shop. The scoop on this incident started with a WOODTV story (and they reused some footage from February, so I was on TV again, which my ego liked!) To be honest, I have done this very thing many times before. My favorite bar in East Lansing has a patio and is next door to a coffee shop with free WiFi. On a beautiful summer afternoon, the choice between coffee and beer is about as easy as they come.
Anyway, this poor guy got saddled with a $400 fine and 40hrs of community service for doing the electronic equivalent of using the bathroom without buying something. According to the law, it could've been $5K and a year in jail, because the underlying crime would've been trespass. At least, that's my layman's understanding of the law. And according to the arrest warrant the judge signed, it could've been $10K and 5 years in the pokey.
I don't see a positive here. I just don't. This incident:
1. has wasted taxpayer money on a benign and trivial incident.
2. will confuse law enforcement further about where the lines around this law are drawn.
3. has victimized a guy that was too cheap to buy a cup of coffee in order to check his e-mail. Seriously, if the guy didn't want to spend $4 for a latte, where's the justice in taking $400 from him for something that neither he nor the coffee shop owner thought to be a crime?
4. has raised awareness of laptop owners that they're responsible for the wireless networks they connect to regardless of their understanding of the technology. If this has any effect, it will be a chilling one, giving people one more thing to worry about when it comes to open WiFi hotspots.
5. will perhaps even be referenced in the future to let businesses off the hook for not securing their business wireless networks.
6. illustrates the enourmous chasm separating the law and lawyers from the understanding of technology.
Anyway, this poor guy got saddled with a $400 fine and 40hrs of community service for doing the electronic equivalent of using the bathroom without buying something. According to the law, it could've been $5K and a year in jail, because the underlying crime would've been trespass. At least, that's my layman's understanding of the law. And according to the arrest warrant the judge signed, it could've been $10K and 5 years in the pokey.
I don't see a positive here. I just don't. This incident:
1. has wasted taxpayer money on a benign and trivial incident.
2. will confuse law enforcement further about where the lines around this law are drawn.
3. has victimized a guy that was too cheap to buy a cup of coffee in order to check his e-mail. Seriously, if the guy didn't want to spend $4 for a latte, where's the justice in taking $400 from him for something that neither he nor the coffee shop owner thought to be a crime?
4. has raised awareness of laptop owners that they're responsible for the wireless networks they connect to regardless of their understanding of the technology. If this has any effect, it will be a chilling one, giving people one more thing to worry about when it comes to open WiFi hotspots.
5. will perhaps even be referenced in the future to let businesses off the hook for not securing their business wireless networks.
6. illustrates the enourmous chasm separating the law and lawyers from the understanding of technology.
Thursday, May 17, 2007
May GRSec is Next Wednesday @ GRBC
Come hang out with infosectarians in West Michigan next Wednesday at GRBC. Stu Berman will be there and has assured me that he is bringing international guests. So it will definitely be a good time. See you there!
ArcSight 4.0 Released
As far back as a year ago there was working code that ArcSight was calling "four-point-oh." So there's no big news here other than the fact that this week ArcSight released installers and docs and all of that good stuff to their software site for general consumption. Add to that the fact that I can't find the announcement on their web site or in general press, and I figure that makes it blogworthy.
What's also blogworthy is the fact that if you're an existing customer and want to upgrade, you're stuck until August when ArcSight releases the upgrade-capable installers with SP1. Or, like with 3.5, you can pay their pro services team to do the upgrade for you before then. Anyway, I'm spoiling the feature list here:
What's also blogworthy is the fact that if you're an existing customer and want to upgrade, you're stuck until August when ArcSight releases the upgrade-capable installers with SP1. Or, like with 3.5, you can pay their pro services team to do the upgrade for you before then. Anyway, I'm spoiling the feature list here:
Key features of ArcSight ESM v4.0 include :
Identity Correlation ArcSight ESM v4.0 identity correlation can model the typical behavior of groups, machines, or individuals (as reflected in events) and provides a framework to access any other form of session data through mappings with dynamic variables. This information can be used or shown in rules, reports, active lists, active channels, and data monitors. Improved Asset Management & Scalability ArcSight ESM v4.0 introduces the ability to manage up to one million assets while maintaining performance, including maintaining memory usage in-line, processing, correlation, and ensuring sustained EPS (events per second). Trend Reporting & Report Generation Performance Trend Reporting enables the ready historical trending often required for regulatory compliance reporting. Trend reporting can track a trend over a specified period of time, and highlight changes in risks or threats during that period. Trend reporting improves report generation performance for regularly scheduled reports by tracking trends over a user-specified time and by keeping the data easily accessible. New Report and Template Designer ArcSight ESM v4.0 provides a new, more powerful and highly flexible reporting system. You can use this design capability to create well-defined reports for different scenarios or audiences. This feature offers options for unique queries and to define the overall look-and-feel for presenting information. These new features include the ability to report on several data queries simultaneously, using multiple charts and grids in one report. Report formats, layout, and overall look-and-feel can be customized to your needs. Historical Correlation ArcSight ESM v4.0 enhances the Verify Rules with Events capability (previously known as Replay with Rules) so you can define actions based on processing historical data through the correlation engine. ArcSight Packages ArcSight ESM v4.0 introduces a new feature called packages. A package is an ArcSight resource that acts as a portable container for group resources or content (e.g., rules, filters, data monitors, reports, etc). Resource Validation Enhancements ArcSight ESM v4.0 enhances resource validation beyond rule- and network-modeling, adding the ability to validate cross-resource dependencies automatically, and interactively, through the Console. This enables the ArcSight Manager to detect resource conflicts introduced during resource modification, creation, upgrading or importing. ArcSight ESM v4.0 64-bit The 64-bit JVM version of ArcSight ESM v4.0 will be made available as part of a controlled release. Customers who are interested in participating should contact Technical Support for additional information.
Thursday, May 10, 2007
Windows Logoff Events
Eric Fitzgerald at Microsoft posted a nice write-up on the subtleties of Windows logoff events. If you collect EventLog data in your SIM or if you have occasion to read EventLog files, I recommend reading Eric's post as well as the rest of mine.
To summarize (translate?), Eric's saying don't trust logoff events to indicate an actual logoff. It could be a timeout or a kerberos token expiring or being reclaimed by the server.
There's something of an exception to this, and you can probably find it in your EventLogs. EventID 538 - "User Logoff" - records a connection type with a decimal value. The value can be 2 or 3. Most of what you will see are type 3 connections, which can mean several different things. But type 2 logoffs indicate the end of an interactive (think RDP) session. That, again, doesn't guarantee that someone actually clicked Start -> Log Off, but it does indicate a definitive end to the session, whether it's a forced disconnect by the client or server, or a clean logoff.
The use case for logoff events is primarily forensic. "When was so-and-so using that system / at work?" "Who was logged on to server X between time A and time B?" And the fact that these events are soft and wonky is frustrating, but being aware of the squishiness of their meaning is important when using them in an investigation. This is where your SIM can really help, because sometimes the best indicator of a logoff/shutdown isn't a single event, but rather the end of activity. "I know Mr. Schmeaux stopped working at 3:30pm that day because there were no more events from his username or workstation IP address after that time."
To summarize (translate?), Eric's saying don't trust logoff events to indicate an actual logoff. It could be a timeout or a kerberos token expiring or being reclaimed by the server.
There's something of an exception to this, and you can probably find it in your EventLogs. EventID 538 - "User Logoff" - records a connection type with a decimal value. The value can be 2 or 3. Most of what you will see are type 3 connections, which can mean several different things. But type 2 logoffs indicate the end of an interactive (think RDP) session. That, again, doesn't guarantee that someone actually clicked Start -> Log Off, but it does indicate a definitive end to the session, whether it's a forced disconnect by the client or server, or a clean logoff.
The use case for logoff events is primarily forensic. "When was so-and-so using that system / at work?" "Who was logged on to server X between time A and time B?" And the fact that these events are soft and wonky is frustrating, but being aware of the squishiness of their meaning is important when using them in an investigation. This is where your SIM can really help, because sometimes the best indicator of a logoff/shutdown isn't a single event, but rather the end of activity. "I know Mr. Schmeaux stopped working at 3:30pm that day because there were no more events from his username or workstation IP address after that time."
Tuesday, May 8, 2007
One for the RSS aggregator: Chinese bot/sploit blogs
OK, so I can't read Chinese (or Japanese, or Korean, or...) characters to save my life. But in the course of my recent adventures in obfuscated JavaScript droppers, I stumbled across something interesting. I put the first piece of some obfuscated JavaScript in Google, and I got 2 hits!
I was really hoping to find a page on this particular type of encoding and where and how it's been used in the past. Instead, I found it posted to a pair of blogs in China, with no accompanying perl scripts for decoding the payload, so I can only assume the intent of the poster(s).
I was really hoping to find a page on this particular type of encoding and where and how it's been used in the past. Instead, I found it posted to a pair of blogs in China, with no accompanying perl scripts for decoding the payload, so I can only assume the intent of the poster(s).
Monday, May 7, 2007
Quick & Dirty JavaScript Sandbox
It seems like more and more browser attacks are using obfuscated JavaScript to make analysis harder. Some things are as simple as UTF encoding, others are far more inventive and confusing. Just like packed executables before, there are legit reasons for wanting to obscure JavaScript, like making it harder for people to steal your code.
So when your IPS alerts on suspicious JavaScript (which is almost never blocked in a default configuration), you can:
Today, however, I ran into a higher-than-usual volume of alerts, all of which were based on the presence of an unescape() call. In anticipation of having to do this again, and the VM being a poor solution to begin with, I built a Java sandbox, starting with a JavaScript interpreter.
Here's the recipe:
1. Cygwin (optional, but you know I love it, and it makes certain things easier)
2. Current Sun JRE for Win32
3. Rhino JavaScript engine
Create an unprivileged local user who's not even a member of 'Everybody'. You're never going to log in as this user anyway. Now unpack the JRE and Rhino to a directory where that user can view them. If you have Cygwin, build a home directory for your user, and then create a bash shell shortcut with that directory in the "Start In" line. Now use RunAs to launch your shell as the unprivileged user, and start Rhino:
Now you can dump JavaScript to the shell and watch it execute with relatively low risk of pwnage. Rhino also has a GUI debugger that's ideal for stepping through more advanced JavaScript trickery.
So when your IPS alerts on suspicious JavaScript (which is almost never blocked in a default configuration), you can:
- A) Investigate, get a sample of the offending page and potentially spend hours trying to work back through it by hand.
- B) Investigate, browse the page with your browser to see what happens, and potentially get pwned.
- C) Ignore it, and hope the local AV got it.
Today, however, I ran into a higher-than-usual volume of alerts, all of which were based on the presence of an unescape() call. In anticipation of having to do this again, and the VM being a poor solution to begin with, I built a Java sandbox, starting with a JavaScript interpreter.
Here's the recipe:
1. Cygwin (optional, but you know I love it, and it makes certain things easier)
2. Current Sun JRE for Win32
3. Rhino JavaScript engine
Create an unprivileged local user who's not even a member of 'Everybody'. You're never going to log in as this user anyway. Now unpack the JRE and Rhino to a directory where that user can view them. If you have Cygwin, build a home directory for your user, and then create a bash shell shortcut with that directory in the "Start In" line. Now use RunAs to launch your shell as the unprivileged user, and start Rhino:
Now you can dump JavaScript to the shell and watch it execute with relatively low risk of pwnage. Rhino also has a GUI debugger that's ideal for stepping through more advanced JavaScript trickery.
Thursday, May 3, 2007
Rothman Redux
Apparently nothing fires me up like Mike Rothman eulogizing SIM. Again.
"SIMs not dead, eh? - Then why is almost every SIM vendor announcing a dedicated log management appliance?"
Perhaps because Oracle or SQL tables are a lousy (and expensive) place to store your logs for years and years. Or perhaps because you don't want to shell out $10K/seat for a full featured console so your sysadmins can search your logs once a week while on a troubleshooting mission. Or, perhaps most likely of all, because infosec customers love appliances.
"How many more data points do we need about the evolving SIM space before we can finally start shoveling dirt on it?"
Let's not forget to also bury heuristic AV, behavioral IPS, deep packet inspection firewalls, and every other infosec product 'next' that has come to pass over the last decade. They all suck and nobody buys them.
Anyway, Mike's point is that since SIM vendors copy each other and are trying to sell log appliances because they discovered that agents don't scale as aggregation points, that SIM is over. Clearly.
"SIMs not dead, eh? - Then why is almost every SIM vendor announcing a dedicated log management appliance?"
Perhaps because Oracle or SQL tables are a lousy (and expensive) place to store your logs for years and years. Or perhaps because you don't want to shell out $10K/seat for a full featured console so your sysadmins can search your logs once a week while on a troubleshooting mission. Or, perhaps most likely of all, because infosec customers love appliances.
"How many more data points do we need about the evolving SIM space before we can finally start shoveling dirt on it?"
Let's not forget to also bury heuristic AV, behavioral IPS, deep packet inspection firewalls, and every other infosec product 'next' that has come to pass over the last decade. They all suck and nobody buys them.
Anyway, Mike's point is that since SIM vendors copy each other and are trying to sell log appliances because they discovered that agents don't scale as aggregation points, that SIM is over. Clearly.