Wednesday, November 18, 2009

ArcSight Logger VS Splunk

You are here because you are searching for information on Splunk vs. ArcSight Logger. I actually wrote this post months before posting it, but sat on it for reasons that may become apparent as you read on.

If you want to hear me talk about my experience with Logger 4.0 through the beta process and beyond, you can check out the video case study I did for ArcSight. In short, Logger is good at what it does, and Logger 4.0 is fast. Ridiculously fast.

But that's not what I want to talk about. I want to talk about the question that's on everyone's mind: ArcSight Logger vs. Splunk?

Comparing features, there's not a strong advantage in either camp. Everybody's got built-in collection based on file and syslog. Everybody's got a web interface with pretty graphs. The main way Logger excels here is in its ability to natively front-end data aggregation for ArcSight's ESM SIEM product. But if you've already got ESM, you're going to buy Logger anyway. So that leaves price and performance as the remaining differentiators.

Splunk can compete on price, especially for more specialized use cases where Logger needs the ArcSight Connector software to pick up data (i.e. Windows EventLog via WMI, or database rows via JDBC). And if you don't care about performance, implying that your needs are modest, Splunk may be cheaper for you for even the straightforward use cases because of the different licensing model that scales downward. So for smaller businesses, Splunk scales down.

For larger businesses, Logger scales up. For example, if you need to add storage capacity to your existing Logger install, and you didn't buy the SAN-attached model, you just buy another Logger appliance. You then 'peer' the Logger appliances, split or migrate log flows, and continue to run search & reporting out of the same appliance you've been using, across all peer data stores. With Splunk? You buy and implement more hardware on your own. And pay for more licenses.

My thinking on performance? Logger 4.0 is a Splunk killer, plain and simple. To analogize using cars, Splunk is a Ford Taurus for log search. It gets you down the road, it's reliable, you can pick the entry model up cheap, and by now you know what you're getting. Logger 4.0, however, is a Zonda F with a Volvo price tag.

To bring the comparison to a fine point, I'd like to share a little story with you. It's kind of gossipy, but that makes it fun.

When ArcSight debuted Logger 4.0 and announced its GA release at their Protect conference last fall, they did a live shoot-out of a Logger 7200 running 4.0 with a vanilla install of Splunk 4 on comparable hardware and the same Linux distro (CentOS) that Logger is based on. They performed a simple keyword search in Splunk across 2 million events, which took just over 12 minutes to complete. That's not awful. But that same search against the same data set ran in about 3 seconds on Logger 4.

This would be an interesting end to an otherwise pretty boring story if it weren't for what happened next. Vendors other than ArcSight - partners, integrators, consultants, etc. - participate in their conference both as speakers and on the partner floor. One of these vendors, an integrator of both ArcSight and Splunk products, privately called ArcSight out for the demo. His theory was that a properly-tuned Splunk install would perform much better. Now, it's a little nuts (and perhaps a little more dangerous) to be an invited vendor at a conference and accuse the conference organizer of cooking a demo. But what happened next is even crazier. ArcSight wheeled the gear up to this guy's room and told him that if he could produce a better result during the conference that they would make an announcement to that effect.

Not one to shy away from a technical challenge, this 15-year infosec veteran skipped meals, free beer, presentations, more free beer, and a lot of sleep to tweak the Splunk box to get better performance out of it. That's dedication. There's no doubt in my mind that he wanted to win. Badly. I heard from him personally at the close of the conference that not only did he not make significant headway, but that all of his results were worse than the original 12 minute search time.

You weren't there, you're just reading about it on some dude's blog, so the impact isn't the same. But that was all the convincing I needed.

But if you need more convincing; we stuffed 6mos of raw syslog from various flavors of UNIX and Linux (3TB) into Logger 4 during the beta. I could keyword search the entire data set in 14 seconds. Regex searches were significantly worse. They took 32 seconds.

3 comments:

Martin said...

Great post, it's good to get some benchmarks minus the sales brochures. I use the free version of Splunk 4 quite a bit, and it is very slow compared to my custom-built log solution which implements full-text indexing. However, I'm confused as to how any technology can regex search 3 TB in 32 seconds. That implies that it is analyzing over 93 GB/sec, which is far over the theoretical threshold for any drive I've seen. I must conclude that this is hitting many striped disks. Can you say how many? The fastest consumer-grade drives are 6 GB/sec, so your hardware would need over 15 disks to get 93 GB/sec. That's just for access, which implies that ArcSight is able to regex search data at SAS line speed, which I find hard to believe. So, I guess I'm asking for clarification on the specs of the machine as well as clarification on the type of regex. Specifically, was there some method used to index the regex so that a full data scan was not required? Or are you implying that any arbitrary regex can be executed at 93 GB/sec?

PaulM said...

Hi Martin,

The unit we had for our beta was an L7200x,and you can check out the specs here:

http://www.arcsight.com/products/products-logger/

The 7200 models have 6 disks. I can't provide a technical explanation for the performance results other than to say that Logger relies on several proprietary technologies that ArcSight developed specifically for high-speed search. This includes search optimization that, for combined keyword, boolean, and regex searches, the fastest search types run first, and the slower searches run against that result set.

The other thing I'll say is that, having been an ArcSight ESM customer since before it was even called ESM, I hope that the storage and search technologies they've built for Logger are rolled forward into ESM. Even if that means having to migrate to an appliance, the performance gains over Oracle 10g will be measured in orders of magnitude.

Paul

VK said...

Martin, good question. I am not sure about the data that was sent by Paul but Logger does a great job of compressing data. The average compression rate is 10:1. So, depending on how repetitive the 3TB syslog data was, it will probably end up being 300 GB or less on Logger. Plus, Logger has full-text indexing and field based indexing which makes searches super fast. Hope this helps.