Wednesday, March 28, 2007

Pete Lindstrom Can Take My SSL Certs

In case you missed it, Pete Lindstrom, the infosec blog scene's answer to Snidely Whiplash, challenged some of the conventional wisdom around using SSL. Twice.

Pete caught some well-deserved flack for musing that SSL serves little to no purpose. But on the point that SSL on Internet web sites only really serves to encrypt data between the browser and the web server, I agree completely. Pete's assessment is that SSL basically serves to prevent sniffing. I disagree. Pete also guesses that sniffing is a rare attack against web sites. No surprise, I disagree. So Pete is at least half right.

SSL on public web sites doesn't do much to secure them. But it doesn't do much to prevent sniffing, either. Point and click tools, like the one I used in this TV news story, make it possible for public WiFi hotspots or even corporate networks to be subverted and SSL MITM sniffing to occur and for passwords or credit card numbers to be stolen as people shop online nearby. And if you think your users read those SSL warnings from IE and click 'No' when they see them, I've got a bridge in Second Life I'd like to sell you.

Tuesday, March 27, 2007

Stupid Snort Tricks

When it comes to tuning IDS and customizing content for SIM, I definitely take my own medicine, or drink my own kool-aid, as it were. Today I am working on cleaning up the Snort rules on an IDS sensor. The rules on this sensor and the others in its group are managed via Oinkmaster. The down side to this is that researching rules and writing Oinkmaster directives can be slow and tedious. But here's some ugly shell script that makes it go faster.

In this example, I have decided that all of the rules related to Microsoft bulletins from 2003 do not apply to me because of patching. This script lets me isolate sid values from rules that contain the string "bulletins/MS03-" and put them into one big comma-delimted line for pasting into my oinkmaster.conf file.

$ for b in `for a in 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30; do grep bulletin/MS03- *.rules | cut -d\; -f$a |grep sid; done | cut -d: -f2`; do echo -n $b,; done; echo

If I want to be extra anal about it and only find those sid values for rules that aren't already disabled, I can use egrep to do this:

$ for b in `for a in 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30; do egrep ^alert *.rules |grep bulletin/MS03- | cut -d\; -f$a |grep sid; done | cut -d: -f2`; do echo -n $b,; done; echo

And finally, here's some ugly shell script that will count up all of your active rules. There's a way to do this while running Snort from the command-line as well.

$ for a in `egrep ^include /etc/snort/snort.conf |cut -d/ -f2 |grep \.rules`; do egrep ^alert $a; done |wc -l

Monday, March 26, 2007

Me vs. Mike Rothman: Could I Possibly Win?

Let me start by saying that Mike Rothman is something of an infosec sage, and has the kind of resume in this field that only dozens of other people have. When he was a VP at META Group (now Gartner), I was still camping out for jsbx tickets. (OK, I was also working on this Linux HOWTO on proxy firewalls, too. But I was then and pretty much still am a nobody.) So, it is with all due respect that I am picking on an article that Mr. Rothman wrote for Also, to answer the question posed in the title, no, no I stand zero chance.

"Plagued by expensive and integration-heavy implementations, SIM products and vendors have never lived up to their promise, taking millions of venture capital with it."

Mike doesn't seem to be impressed with the current SIM offerings. And, to be fair, it's not all good news. Some SIM products are weak, some are expensive, and some are both, and maybe some are neither. What a SIM needs to work is access to your logs. That's pretty easy to deliver in most cases either via syslog, WMI, flat file access, SNMP, and so on. Not much integration.

"Just think: How great would it be to look at one screen, or one dashboard, and be able to pinpoint problems, maybe even before they occur?"

Hellz yeah! But be real. No vendor can drop that in your lap. Or, if they can, you probably also live in a 3-bedroom house with a white picket fence, 2.6 kids, and a dog, Mr. Generic. Let's not bag on the vendors because they don't deliver the moon. SIM is a tool, and you're going to learn how to use it or you're going to hate it.

"One problem is the overactive nature of SIM; its inputs, like firewalls and IPS devices, are inherently noisy. If the inputs are rife with false positives, it has historically been difficult for SIM offerings to provide actionable information without a tremendous amount of experimentation and tuning."

This is a straw man. If the original data is flawed, then the analysis of that data will yield flawed results. I don't see how that's the fault of your SIM product. But Mike does have a good point - tune your IDS already. It's 2007.

"Also, SIM products seem to address problems after it's too late; by the time information is correlated from log files, the attack has already happened. "

Another straw man from the Gartner IDS/IPS debate. Yeah, it sucks how SIM doesn't do something it wasn't designed to do. Actually, what sucks more is that vendors are taking this idea to heart and trying to make SIM act as an IPS, too.

And in today's environments, where attacks can proliferate throughout the world in a matter of minutes, playing catch-up can be crippling."

This is the sentence that inspired me to post a response to Mike's article. SIM isn't there to stop attacks from happening. It's not a defensive tool, it's an analysis tool. This makes it ideal for incident response. Something bad has happened, and if all of your logs are in one place, you have a great tool for organizing and searching for related evidence, defining the scope of the response, and monitoring post-response activity.

First, security management is increasingly being integrated with network behavior anomaly detection (NBAD), providing pseudo real-time visibility into what's happening on your network."

I agree with Mr. Rothman that analyzing netflow is a good idea. Being able to access that data from inside your SIM is an even better idea. The problem is that logging netflow with a ratio of one event (one row in your database) to each unique connection presents serious throughput and storage issues. So this is where SIM vendors should get to work integrating with FlowCollector, NetworkVantage, and other netflow collector/analyzer products. I know that some SIM vendors are going to bundle this functionality, but a SIM interface into the third-party product data is much more elegant and practical in my book than having the SIM do the netflow collection itself. I think the horse has already left the barn here, though.

I have two more random points that I want to make on the topic of SIMs and their role in security. These aren't in direct response to Mike's article, but I think they're worth bringing up here.

First, it seems that SIM products are getting pigeonholed as consoles for IDS and other network data. This is a very narrow view of what SIM can do. In fact, I would say there's more value in operating system and application logs. I know from some of the other companies I've talked to that there are SIM installs out there that do nothing but application logs. That's very cool.

Second, there's a lot of innovation in the SIM market right now, and it comes in places security folks often don't look first. Some of the coolest GUI design work that I've seen outside of windowing environments has been in the SIM arena.

On Dashboards and Pink Days

I love my wife, and one of the reasons I love her so much is because she and I can entertain ourselves in ways that must look inane and boring from a distance. Last night for example, we sat on the couch watching the late news cast of a local CBS affiliate and making fun of it the whole way. When the weather came on, she didn't let up, calling out their awful graphics, including a 7-day 'outlook' of colored bars without a legend that included several "blue" days, a "yellow" day, a "green" day, and a "pink" day. She had me laughing so hard I was in tears by the time the sports came on.

The point here is that there are lots of bad graphics out there. Bad graphics are those that don't mean anything to your audience. When you're in the audience, it's really easy to spot them. When it's your job to make them - especially if you are intimately familiar with the data behind them - it's not so easy. Good graphics stand alone. So here are a couple of tips on building dashboards:

  1. Legends and Titles. Don't just have them, have good ones. They should be clear and explain the meaning of and differences between colors, symbols, etc.
  2. Data Points. Don't try and cram too much different data into a single chart or graph. Have one purpose per graph - for example, don't display moving averages and sums in the same area. But do have a high density of data.
  3. Scale. When placing graphs next to each other, trying to make the layout symmetrical can often skew the interpretation of the graph. For instance, one bar graph with peak values in the hundreds of thousands next to another bar graph with peak values in the tens of thousands will probably confuse your audience, even if there is a y-axis legend explaining this. Similarly, if data is important, showing it to scale next to other data that occurs at different orders of magnitude can hide that data or make it seem irrelevant.
While this sounds sort of obvious, chances are whatever you're using, from gnuplot scripts to a commercial SIM package, probably sabotages you in one or more of these ways out of the box. The best way to fix any of this is to ask the people you're showing it to - in a discreet fashion, lest they pretend to know what's going on to save face - whether or not the graphs you give them are useful and summarize the data they actually care about. Otherwise, someone like my wife will tease you mercilessly. You'll have it coming.

Friday, March 23, 2007

Intro to SIM Deck

In February of 2006 I gave a preso at Grand Rapids' ISSA on SIM. This is a short deck - the first 11 slides - that has a nice overview on SIM concepts and what makes a product like ArcSight different from a product like eIQ FirewallAnalyzer (eIQ has their own SIM now).

The rest of the deck, which GR-ISSA members have access to, is lots of screen shots from ArcSight and a bit more about what we do internally with it for security ops. It's not really top secret, so if you'd like to see the whole deck, feel free to e-mail me.

More on SIM to come. Next week I call out Mike Rothman for suggesting that SIM can be "saved" by bolting crap onto it.

Wednesday, March 21, 2007

Health & Human Services Is Teething

Lots and lots of people have declared HIPAA irrelevant and ineffective because of the lack of direct federal oversight and the perception that the penalties it could potentially level at an organization were weak in comparison to things like SOX.

But, OH SNAP!! The Health and Human Services Inspector General is auditing providers. I've got in my inbox a copy of the FAX sent to _____ Hospital in Georgia about their audit. No mention of complaint or prior incidence. Just a friendly, "Hi, we're coming to audit you," letter complete with data collection document.

It has always been my stance that if what's lacking in compliance is enforcement, then it's important to comply, because enforcement is only a budget line item away. So I guess I'm saying, "I told you so!" to everyone who has greatly exaggerated the rumors of HIPAA's death.

So, uh... think warm thoughts doc, cuz that thermometer is mighty cold.

Tuesday, March 20, 2007

Review: Chuvakin's Database Log Analysis Paper

Dr. Anton Chuvakin was kind enough to share with me a copy of his 2-part paper on database log analysis, which was published in CSI Alert Newsletter, Feb 2007. I enjoyed his paper(s), but I have to acknowledge that this is a topic near and dear to my current work. In Part I of his paper, Dr. Chuvakin gets right to the heart of database log analysis: You should do database logging better than your database does by default because it's probably a compliance issue for you.

Chuvakin also explores some of the pitfalls of database logging:
  • Performance hits from turning on auditing
  • A databases stores its different log data in multiple formats (in a table, in a redo file, in a text file, etc.)
  • Finding data in database logs that is meaningful to security ops and incident response

"Introduction to Database Log Analysis, Part II" lays out some practical considerations for any organization preparing to implement database logging. It's hard to keep this topic general, since the differences in audit logging between Oracle and Microsoft SQL Server, for example, are significant enough that the strategies for tackling log gathering, storage, and analysis are very different.

As you might expect from a guy in his line of work, Chuvakin goes on to describe why you might want to use a SIM or similar tool to aggregate and read database logs. I happen to agree with him that you need something more than LogMiner if you want to use your logs for anything more than troubleshooting. But Chuvakin gives a list of example reasons for using a SIM for compliance and operations purposes around database logging, not all of which wash with me:

Change management: modern RDBMS is a complicated piece of software with plenty of configuration files and other things to change. And being aware of all the changes constitutes one of the control objectives for IT governance as well as essential for protecting the environment.

True that COBIT and ITIL both rely on change management and that it is a common compliance requirement for things like SOX and SAS. However, I don't know that a SIM is a good candidate for this. Operationally, I use our change control documentation to explain, for instance, why a bunch of files were changed or new users created. I don't use the SIM's ability to detect these events as the impetus for documenting the change - the cart's way out ahead of the horse by the time the SIM gets to it, and if you do it right, change docs always precede the change. So, at best, this is a secondary control that the SIM is providing. Not that it's a bad idea, it's just not a business driver for log analysis.

Authentication, Authorization and Access: logging access control decision such as login failures and successes as well as access to data and various database objects is of interest for auditors as well as important for security, such as insider privilege abuse. While logging all access to data is less common, it is one of the emerging trends in logging.

Good idea. Do this with your directory and server OS, too. And RADIUS. And client VPN. And prox cards. And voicemail. And... oh, you get the idea.

Threat Detection: while both unauthorized changes and access control decisions are essential for security, database logs may be used to discover and analyze direct exploitation attempts as well, at least in some cases. Even accessing the database system at an unusual time will likely be of interest to as security team.

Log analysis is probably your best option here, unless you stick an IDS sensor between your database servers and everything else on your network. Which if your database supports a web app, you should definitely do.

Performance: DBAs are tasked with monitoring database performance and logs provide one of the avenues for doing so. This is especially important to those orgs with strict service level agreements (SLA) for database performance

While a SIM may be able to create averages and other statistics in near-real time, this is not the tool for database performance monitoring. There's lots of stuff out there that does this already and much better than your SIM can.

Business Continuity: knowing of database software starts and stops is essential since business depend on databases for their revenue streams and this a downed database directly leads to losing money

Again, lots of stuff that can do this better than a SIM. Moreover, SIM's use data analysis instead of active checks. If your database shuts down, a SIM can tell you about it once it sees a log entry for it. If it hangs, then maybe it will detect a drop in activity and determine that the event per second delta is abnormal. Of course, this can occur during normal use. IMHO, SIMs suck for tracking service up/down states.

Overall, I enjoyed reading this paper. It's clear that Dr. Chuvakin has some good ideas on this topic, and I would recommend that anyone who is considering database log auditing read this paper prior to the initial design meeting. It's not a comprehensive guide, but it will get you thinking in the right direction about the issues your organization will likely face.

Monday, March 19, 2007

Insider Threats

It seems to me that over the past year or two there's been a refocus on the "insider threat" against your network within infosec trade press as well as the general infosec product space. I recall reading in the CSI/FBI Survey that in 2000 the Internet overtook internal systems as the most frequent source of attacks against networks. I can't fathom that that statistic has reversed, what with the number of SQL Slammer, SSH & VNC scans, etc. that constantly fill up my firewall logs. It begs the question, then, why refocus on insider threats? There could be a number of explanations, including the fact that everybody that is going to buy a firewall has bought one. But I think it's a mixed bag of good news and bad news.

The good news is that we've gotten better at border security. The number of remote-root exploits found drops each year. Mass-infection worms like CodeRed and Sasser are fast becoming a memory for most corporate network admins. Microsoft operating systems and most Linux distros automatically patch themselves now. But none of this addresses the problems you have by giving people passwords, so the "insider threat" still applies to you. Dang. Now you need to search everybody's e-mail and web traffic for corporate secrets being leaked. Or something. Anyway, that is the good news - we don't totally suck at security anymore.

Here's the bad news - your internal users are still not your biggest threat. It's still the Internet. It's spyware and adware and spam and bots and all of that crap that you can't block with firewall rules. Your users are still bombarded with this stuff constantly via e-mail and web sites, services that you can't turn off. And the malware authors have turned AV into an arms race - changing their software daily if need be to stay ahead of signature updates.

My point is that insider threats, while real, are a distraction. Addressing attacks from the inside of your network is now easier than addressing the new external threats effectively. And that's why people are focusing on it. It's not the biggest battle you face right now, but it is the one you have the best chance of winning.

Wednesday, March 14, 2007

On Web App Scanners

Dark Reading today covered SPI's announcement of AMP 3, their clone of AppScan Enterprise.

I haven't seen AMP, so for me to bag on it is a little unfair. But to me, the notion of automated scanning and reporting at the web-app level seems like a flawed idea. So to be clear, I'm not picking on AMP, but rather AMP and all of the other products like it.

I totally see the value of doing web-app security testing. But I worry that the click-n-fire scanning that these "enterprise" products offer is getting companies to pay more money for less depth by sacrificing quality for quantity. Regular scans are great, but it is my opinion that these products will find the low-hanging fruit and miss other vulnerabilities. Not because they can't find some of the more difficult vulns, but because the out-of-the-box scans come in two flavors; noisy, and incomplete. We already have this at the network level. Are companies ready to admit defeat at the app level already, too?

Why SIM's are cool

One of the things I've wanted to blog on - and one of the things I want this blog to sort of 'be about' - is the technical and operational sides of SIM systems. So instead of coming up with a big idea article on SIM and why it's cool, I'll give you an example from today that made my day.

At work, our security team uses a SIM to do lots of things, and one of the more important ones is incident response and investigation. Today I was watching events from McAfee ePO and stumbled across the following event:

Name: 'JavaScript security violation detected and blocked'
Destination Address: XX.XX.36.92
File Name: 'Script executed by IEXPLORE.EXE'
String1: 'JS/Exploit-BO.gen'
String2: 'trojan'

This is a detect/block message for a generic JavaScript exploit that was detected while the user was in Internet Explorer. There's no good way to tell what was actually going on here. The possibility is that there was exploit code there and that it was blocked. There is also the possibility that there was other exploit code present that was not blocked. So I searched for firewall events using this filter:

((source_address = "XX.XX.36.92") AND
((request_url EndsWith ".js") OR
(request_url EndsWith ".htm") OR
(request_url EndsWith ".html")))

(Note: I did not have to type this all out. It was a short series of mouse clicks in the SIM GUI.)

And it took no time at all to find the firewall log event I was looking for:

Name: accept
Source Address: XX.XX.36.92
Destination Address: XX.XX.121.99
Request Url: http://XX.XX.121.99:80/incs/sfhover.js

So, it took less than 2 minutes to track back to a URL and get a look at the JavaScript code. Fire up a shell and:

$ wget http://XX.XX.121.99:80/incs/sfhover.js
--13:12:40-- http://XX.XX.121.99/incs/sfhover.js
=> `sfhover.js'
Connecting to XX.XX.121.99:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 751 [application/x-javascript]

100%[====================================>] 751 --.--K/s

13:12:40 (4.75 MB/s) - `sfhover.js' saved [751/751]
$ cat sfhover.js
for (var i=0; i
elemsArray[e][i].onmouseover=function() {
this.className+=" sfhover";


It was an onmouseover call that set McAfee off, and in this case, it's benign. Two minutes to run down a possible exploit with code and get back to work. Sorry, but that's just cool.

Saturday, March 10, 2007

March GRSec - It's On Like Donkey Kong!

There definitely will be a March GRSec meetup!

Probably downtown Grand Rapids. If you have an opinion on the venue, now's the time to speak up. I'll have to make arrangements and announce our destination by Friday the 16th.

How You Know It's Spring in Michigan

Well, you sure can't tell by the weather. :-)

But you know Spring is here when the first batch of Bell's Oberon arrives in stores. Two weeks from now - Monday, March 26th - it will officially be Spring in Michigan.

Friday, March 9, 2007

Who Names These Things?

Yesterday, the SEC launched Operation Spamalot to combat pump-n-dump stock fraud that utilizes spam to artificially 'pump' some penny stock price. Aside from thinking that the name is atrocious and feeling sorry for Eric Idle, I think this is a good idea. Pump-n-dumps are an old trick, only the spam part is new.

So starting today, if your stock symbol shows up in spam, the SEC will suspend trading of your stock. They've already suspended 35 companies.

Aviram Jenik at Securiteam is concerned that this could be used to perform DoS attacks against bigger stocks. While technically possible, it's unlikely for a number of reasons. First, unlike firewalls receiving shun commands from IDS sensors, there will be people making these decisions. Second, if you look, it's pretty easy to see a pump-and-dump on paper. I collected some examples from stock spam I got this morning for you to look at. I'll bet you can see one way to differentiate them from the big boys right off the bat. The trading history tells the rest of the story.


Wednesday, March 7, 2007

February Catch Up

Here's some random stuff I've been meaning to post up here as follow-ups to posts from February. I've been pretty busy with work and am late on these. Sorry.

Python code: I've improved on my original, first Python program by adding the ability to create a whitelist file full of regular expressions. This makes it easy to isolate only those hostnames you want to find without knowing what they are. In case you're wondering, being able to do fast PTR record lookups against your DHCP ranges looking for things you don't know about is the poor man's NAC (oh, you thought EAP was the poor man's NAC?). Most Windows machines will announce their FQDN and register with Windows DHCP/DNS making them findable by doing reverse DNS lookups. Use the whitelist to exclude the stuff you know about like 'myinternaldomain.local'. Link here.

Nepenthes and ops: In this post, I mention that Tim Crothers presented on an easy way to work honeypots into network security ops. And then I totally neglected to describe how that works. Just to be clear, this is Tim's idea, not mine. I'm reposting it without permission. Hopefully he's cool with that.

Step 1. Build a VMWare image as similar to your corporate workstation image as possible. Specifically, keep it at the same patch level and run the same anti-virus or other security software with the same signatures.
Step 2. Install Linux on some old computers. Now install and configure Nepenthes on these as well.
Step 3. Deploy the Nepenthes boxes where they can collect malware: on a DSL/cable connection with no firewall, on a darknet, on a workstation network, outside the corporate firewall.
Step 4. Regularly (or automatically) review nepenthes.log and check the binaries directory for captured malware.
Step 5. Carefully transfer malware (via password-protected ZIP, for example) to the VM built in step 1.
Step 6. Disconnect the VM from the corporate network and unleash the malware. See if your AV tools detect it. Use SysAnalyzer to see what it does.
Step 7. If AV doesn't detect it, send a sample to AV vendor asking for emergency update. Deploy emergency update.

The thing I like about this is the simplicity of it. And being proactive on malware definitely won't hurt you. This stuff changes so fast it's difficult for AV vendors to keep up.

Accept Additional Risk: Cancel or Allow?

At my current place of employment, there has been a push to (and a push back against) adopt Mac OS X as a supported platform. Apparently Mac's are no longer just for indie musicians and graphic designers. In addition to graphic designers, we have programmers and UNIX admins and other random hipsters interested in toting around a new MacBook.

I recently discussed this issue in depth with my boss, the CISO. He worked with the desktop guys during the XP roll-out to reduce local admins, implement group policy security measures, turn off services, and basically harden the desktop image to where it is fairly resilient. The concerns about Mac's are that they still lack the spiffy security policy lockdown stuff that XP has and that they are now very much on the list of platforms actively being analyzed for vulnerabilities. The whole x86 CPU (and therefore x86 shellcode) thing doesn't help, either.

Once you separate this issue from the nerd jihad, Mac OS X isn't so bad. In fact, a little bit of additional work, some means of centralized software management and an AV client is enough to bring it into alignment with the XP machines and make it a reasonably low risk venture. The TCO/ROI and in-house support stuff is someone else's problem.

After stopping to think about what it takes to bring a new OS into the security fold, it's not Mac's that worry me. It's Windows Mobile on phones. Single-user OS, barely-there authentication, and when it's in a cradle and the EVDO data link is up, it's a potential back door into your network around your firewall and IDS.

Damn. Cuz I don't care what you say, the Treo 700 is way more nerdtastic than a 17" MacBook.

Tuesday, March 6, 2007

Channel Certs

Much like this blog, I'm a few weeks behind on this one, but I haven't seen it anywhere else besides Channel Insider, so I'm going to bring it up. I'm glad to see Symantec kill their VAR certification requirements, but disappointed at the same time.

In case you don't know, VAR's and other channel resellers are typically required to have some number of people complete some number of certification exams for their products in order to gain various levels of partner status. What you may also not know - but probably want to know if you are a VAR customer - is that these are not the same exams that you or your employees can train for and take. They're much easier. And they're not proctored - typically delivered via web site, it's easy to cheat or have someone else take the exam for them.

I used to work for a VAR and I acquired an even dozen of these things. Bigger vendors like Microsoft and Cisco make partners get the real sit-down certs because they can, and that's a good thing. Of course, Symantec is a BIG vendor, but outside of Win32 AV, their market share in any single infosec product space is smallish. I understand their decision to pull their channel cert program because they need all of the channel help they can get for their SGS appliances (formerly [C]Raptor) and the like. But I wish they, and other companies that have VAR channels, would hold their partners to stricter standards. It hurts everyone, but mostly VAR customers, when a vendor lauds a VAR as a 'Name Brand integration specialist' and in reality the VAR sends someone out to install your new, brightly-colored 2U appliance who has read the product PDF's and has a direct line to Tier-3 support, but has never actually installed one before.

Some customer must be that engineer's first install. How do you know it's not you?