Managing ArcSight ESM Tools

Last week Tim and I presented at the ArcSight user conference on using Tools in the ESM console to augment incident response and investigation. I'm hoping to have the sanitized slide deck up this week, and maybe a little bit of code to go with it.

At the end of the talk - and then a couple of times in the hallway - people asked how we manage all of these Tools. It's a great question, and the answer is, not very well. But here's the best way that I know how in ESM 4.0.

If you've ever looked at the tools editor in the ESM console, you've seen this dialog, which is pretty basic:


By default, this info is recorded in your AST file (C:\arcsight\Console\current\paul.ast), which is just a text file with a bunch of values declared. The values in the file look like this:

console.ui.tools[toolName].program=
console.ui.tools[toolName].workingdir=
console.ui.tools[toolName].iconFile=
console.ui.tools[toolName].parameters=
console.ui.tools[toolName].showInToolBar=
console.ui.tools[toolName].isExportTool=


And then there's this:

console.ui.toolsList=

Which is pretty self-explanatory. It occurs once in the AST file and is just a CSV list of the tool names, used to populate the console's Tools menu.

I wrote a Perl program that can parse an AST file and extract tool data from it for the purposes of sharing. It's designed to help scale and distribute tools across your analysts' consoles so that they don't have to manually recreate and test them. In order for it to be useful, there are some best practices. Here's what I do.

1. Cygwin (Surprise!) If you've been reading my blog for any period of time, you knew this was coming. If your analysts use Mac OS X or Linux for their ESM console platform, not to worry. They can play, too. Beyond, Perl, bash, and Python, that's kind of the point.

2. Standardize on a source directory for scripts. Put all of your scripts in the same spot. Pathing is difficult to manage by hand, so by defining a standard (I use /usr/local/bin/arcsight), you have less to do each time you distribute a new tool.

3. Use a repository like Subversion or CVS for scripts and other tool artifacts. That way, you can make a change to your tool, check it in, and the other analysts can check it out quickly and easily. No messy manual copies. Also, when you foul something up, you have revision history to go back to. That can be a life saver if you are - like me - not a developer with good testing habits.

4. Use consoleupdates.txt on the ESM manager to distribute tool configs. Here's how you do that:

Let's say my ESM user id is 'paul' and I have developed a whole bunch of tools following the first three rules above. I can use this Perl script to create an export of the tool configs for use on the server. It looks like this in Cygwin:

$ arc_tool.pl export all /cygdrive/c/arcsight/Console/current/paul.ast > ~/consoleupdates.txt
$ scp consoleupdates.txt arcsight@esmmanager:/opt/arcsight/manager/config
$ ssh arcsight@esmmanager
Password:
arcsight@esmmanager:~ $ chown arcsight.arcsight $ARCSIGHT_HOME/config/consoleupdates.txt
arcsight@esmmanager:~ $ chmod 644 $ARCSIGHT_HOME/config/consoleupdates.txt

If you want your analysts to get the updated tool list, they need to log out of the console, move their AST file somewhere safe, and log back in to the console. No restart of the manager is necessary. Now they just need to create the same script directory you did and check out your scripts from your CVS server.

The Perl script I wrote also supports listing the tool names in an AST file as well as exporting single tool configurations, so it's more than just a one-trick pony. It's got another half a trick.

My advice at this time is not to invest a ton of time in doing this unless it's a weekly headache for your security team, but it is worth doing if you've already got the moving parts in place (like a CVS server). The reason is that ArcSight has already fixed the issue of sharing tool configs in ESM 4.5. So once that's released (later this year?), some of this will be a non-issue for you. I suspect that the first two or three best practices I list above will still be valid in ESM 4.5, so it's still a valuable exercise if you have any number of custom tools already.

ArcSight User Conference 2008

I'm on the floor of the ArcSight "Protect '08" conference this morning. Tim and I gave our talk on ArcSight ESM Tools yesterday, and I will post some version of those slides and some of the code after I return from the conference.

Right now I'm listening to Hugh Njemanze give his keynote on product lines. There's a lot of interesting stuff in the release pipe; Logger 3.0, ESM 4.5, a new Connector appliance, IdentityView content for ESM, and something called "McLovin."

Anyway, here's what's been good so far:

• Customer presentations (other than mine, I mean) - I missed out last year, these are the best talks so far.
• Location - the new hotel is within walking distance of stuff (and by stuff I mean not trees and the NSA.)
• Networking - Always the best part of this conference. I love standing around with free beer, talking to other folks about what they're doing with their SIM, and sharing ideas. Looking forward to more tonight.
  

Here's what's been not-so-good:

• Wireless - the hotel wireless has been unreliable and overloaded. Frankly, I'm surprised I've been able to stay on long enough to get this post up.
• Vendor/sponsor floor - no offense to these guys, but the freebies this year are unimpressive. I've already got a pen, thanks.
• No bag - Instead of a "conference bag," everyone was issued a plastic file folio thing. Not that I needed another bag, but I can't smoosh the one foam squeezy thing I did get from a vendor booth into this blue plastic thing.

And I would be remiss if I didn't drop a product scoop or two:

• Logger 3.0 has adopted a more-ESM-like boolean filter interface. Big improvement over the chained-regex search in 2.5 and earlier.
• Demo of Logger 3.0 shows that searches of data (no details on data set) are roughly 80x faster than a similar sized search on 2.5. (The claim is 100x faster, but I counted. Still, that's a significant improvement.)
• Hugh has hinted that the slick, high-performance append-only storage stuff that Logger has is going to be integrated into ESM in some release beyond 4.5. That could mean the end of the Oracle / PartitionArchiver storage model. It won't be missed.

Visual Analysis of 'Ideas in Security'

Amrit Williams, former Gartner analyst and CTO at BigFix is one of the bloggers that I follow regularly. Amrit's a very smart guy and I respect what he has to say. He recently wrote a pair of blog posts (here and here) that compliment eachother.

Now, in the details of what he has to say, Amrit and I are in agreement. But I got to thinking about the second post and how it relates to the first post. And, well, I fired up Visio and mapped the relationships between Amrit's greatest and worst ideas lists.

If we look at the great ideas that didn't spawn or perpetuate the worst ideas, then we're not left with much. Just segmentation and theory of least privilege. If we drop out planning and segmentation because they're not actually security ideas - just good ideas that work lots of places - we're left with Theory of Least Privilege as the one great idea to come out of security. Oddly, that seems about right.

Evidence FAIL

So, first read this .

John Dozier, self-described "SuperLawyer" of the Internet, thinks you kids and your DefCon are a bunch of punks. Stay off his lawn.

Of course, I disagree. DefCon used to be a hacker conference by hackers for hackers. Now it's the BlackHat afterparty-slash-olympics. But what it isn't is a bunch of criminals. Sure, there's some mischief, and a few folks even break the rules. But everyone I know who attended DefCon this year (and that number is solidly in the double-digits), works in InfoSec, and uses what they learn at DefCon in their professional lives.

Compelling as my argument may fail to be to people like Mr. Dozier, his argument is weaker than mine. Let's dissect, shall we:

Defcon ... began August 8 and it looks like the hackers sitting in the audience and participating in the hacking competitions spent two days trying to hack into the Dozier Internet Law website using SQL Injection Attacks, Mambo Exploits, encoded cross site scripting attempts, shared ciphers overflow attempts, and the like.

The favorite and most common ISP access was from Vietnam and China, with Beijing the host and doorway of the Olympic Games as well as many, many hackers.

OK, so what we have here is a number of known, old, web attacks from China against his web server that coincide with the timing of DefCon. And aside from the timing, there's nothing to implicate anybody having anything to do with DefCon. My guess is that this wasn't even an actual human being at all, but rather an ASPROX scan that Dozier's IDS detected.

The graph above shows what these hackers do. They come to Vegas to learn how to hack into systems and create havoc.

The funny thing about this is that, with the notable exception of Dan Kaminsky's DNS attacks, there aren't IDS signatures for the research presented at DefCon. So any attacks that did come as a result of learning done at DefCon wouldn't be on that graph.

The frustrated perpetrators (they never got access) were sitting in the Riviera Hotel ballrooms, I suspect...

First, the key word there is suspect. Mr. Dozier has zero evidence that these IDS alerts had anything to do with DefCon. None. Not a shred. Second, they would've gotten in.

Going after law firm websites and administration areas that contain attorney/client protected communications and documentation, and even court ordered "sealed" files, is a direct attack on the integrity of the judicial process and the judiciary

If you have documents that are sealed by a court order stored on your company website, then you have problems. Most federal district courts won't allow you to electronically file with the court to have a document "sealed" if that document must be or otherwise is included in the filing. Those general orders aren't accidents. It's a recognition on the part of the judiciary that electronic documents are inherently less secure. But I digress.

Many attendees commit criminal acts while in attendance in organized war games.

This is simply untrue. There are organized wargames, conducted on an air-gapped network off the Internet or any other network. This is perfectly legal. The US Air Force has staffed a team in the past. By the way, congratulations to Chris Eagle and sk3wl0fr00t on their CTF win. They bested two-time champs 1@stplace, who are some of the smartest people I know, and who are all highly ethical InfoSec professionals.

Others commit criminal acts as they learn the tools of the trade in the very ballroom during speaker presentations. They hack into banks, into personal computers, into businesses, into government agencies, and steal private information, cost businesses billions of dollars annually, and ruin the financial well-being and impair the emotional stability of individuals all across our country.

This is sensational and unsubstantiated. Or as a judge would describe it, hearsay.

This is the mob of the 21st century;

No, John, this is the mob of the 21st century.

The only "security researchers" in attendance, I suspect, are the good guys.

Yes, the security researchers at DefCon are the good guys. And I promise you that the DoD and DoJ agree, as many of the speakers, attendees, volunteers, and contestants at DefCon are paid consultants to these organizations.

UPDATE: John Sawyer has an excellent write-up on this issue and on this year's DefCon (unlike John Dozier, he was actually there) on his blog, Evil Bits, over at Dark Reading. Go read.

On Blended Threats

Dave Hull over at Trusted Signal has an interesting post on his blog right now about blended threats. (Unfortunately, I can't find a permalink for it, so I don't know how long you'll be able to read it.)

If it's not still there for you to read, let me give you the gist of it. There's been some recent research into and discussion of blended threat scenarios by some very smart people.

So what is a blended threat? It's where two or more lesser-severity vulnerabilities are exploited in conjunction with each other to lead to a greater compromise. An example would be a pen-test I did some years back where we found a SQL injection vulnerability in a low-value web app with no insert/delete grant to an older, unpatched version of Oracle. Individually, you wouldn't rank either vuln especially high. You could break the web app, but there wasn't sensitive data in there, and you couldn't tamper with the data itself. The Oracle database wasn't exposed to the Internet directly. But by using SQL injection to attack Oracle, I broke out into the server OS, reverse tunneled a command shell, and had the Administrator password in very short order. Which was also the Administrator password of the other servers I could talk to.

Myself and others have been predicting the emergence of wide scale blended threat attacks since at least about 2002/2003. And so far we've been wrong, which is good. For now, blended attacks are, as Dave points out, the stuff of professional pen-testers and other intelligent intruders. But frankly, I don't know why.

The problem with blended threats is that they're harder to identify and calculate risk for. CVSS doesn't provide a way for scoring vuln A when also in the presence of vuln B. And this has lead to vendors delaying patches or downplaying the severity of vulnerabilities based on the assumption that any vulnerability the only vulnerability present.

This creates an opening in the patching cycle for malware/botnet folks to capitalize on if the right blended threat comes along. Maybe we haven't seen it becauuse, to date, these folks simply haven't needed to go there in order to be successful.

What Role Will Security Researchers Play a Decade From Now?

The whole Dan Kaminsky DNS Thing has gotten me thinking about disclosure. I intentionally haven't blogged about it because, well, the speculation around Dan's finding has turned into something of a spectacle. And you didn't need to read yet another blog post about the sky falling.

But on the eve of Black Hat, Dan's talk is less than a week away, and I can't help feeling like we've gotten no closer to understanding the issue of disclosure than we were a year ago. So, all I'm going to say about Dan's recent "situation" is that I, for one, am impressed by the level of care and coordination that went into working with vendors to get patches. This is hard. Researchers hate it because vendors can be uncooperative, incompetent, and downright vindictive. So, thank you, Dan, for spending what must have been countless hours on conference calls and e-mail getting vendors onboard.

Now that that's out of the way, let's talk about research, disclosure, and the future. Dino Dai Zovi noted in a recent blog post that the 90's were the era of full disclosure, and that that is now over. (It's an excellent post. Go read the whole thing.) And this is evident in a number of ways. For one, ZDI and other pay-per-sploit buyers. For another, in-the-wild 0days showing up for sale from malware vendors like the MPack team.

And then there's the ongoing "debate" (read: stalemate) between researchers and vendors about protocol, grace periods, and credit.

So disclosure is a mess. But I don't think it has to stay this way, at least not in the USA. Researchers who publish - as opposed to sell - have the opportunity to become consumer advocates. By cooperating with vendors in a way that still holds them accountable, researchers can demonstrate value to the consumer public. When that becomes the prevalent sentiment, then other interesting things like grants and nonprofits make it possible for researchers to earn a living without having to also do consulting or sell their exploits to a third party.

And that's the dead horse I'm beating in the disclosure race - the consumers of IT products don't have a voice in the disclosure dialogue and desperately need one. Researchers can, if they're able to forego infighting and ego theatre, be that voice.

Coffee Shop Warfare

It seems like I can't go to a coffee shop, conference center, or bar these days without some jackass on the network abusing the bandwidth. Running MMO games, BitTorrent, gnutella, or even just a large FTP/HTTP download will saturate the wireless access point, let alone the modest DSL line it's connected to, rendering it unusable for the other patrons there. This is just plain rude. And since the barrista can make a mean caramel cappucino, but doesn't have the ability to blacklist your MAC on the AP (which I realize isn't a very effective control, but hey - maybe you'd get the message then?), we're all stuck to suffer.

And I wouldn't do anything hostile on a public network. But in the name of network self-defense, there are a couple of tools you might want to take with you to the coffee shop next time.

• Wireshark - The quickest, easiest way to identify the abuser's MAC/IP is with a sniffer like Wireshark, tcpdump, or iptraf.

• Snort - Snort with flexresp2 enabled, bound to your wireless interface, and the p2p.rules set enabled and modified with "resp:reset_both,icmp_host" is an effective deterrent for people using P2P file-sharing software.

• Ettercap - More severe than Snort, you can use Ettercap to perform ARP poisoning and essentially blackhole the client(s) of your choice by MAC address. You could also use this tool to sniff unencrypted traffic between clients and the AP (and points beyond). But you wouldn't do this. It would be uncivilized, and possibly illegal.

There are lots of other wireless tools out there that have some application here, but many of them either go to far to be civil (Void11) or legal (Hotspotter), so I don't recommend them. For that matter, what I do recommend is getting your own EVDO card. Then you don't have to put up with rude WiFi users in the first place.