Thursday, December 11, 2014

GrrCON 2014 Executive Summit Keynote

In October I gave the keynote entitled "Effects of Cyber Security Breaches" at a closed session of GrrCON.  The presentation was about the new relationship between cybersecurity, the board of directors, and a realistic look at the impacts of data breaches.

I've been asked for these slides by a number of folks and have just now gotten around to publishing them, including my presenter's notes.

Tuesday, August 20, 2013

BSides Detroit Presentation

In June I gave a presentation at BSides Detroit entitled, "Broke, Note Broken: An Effective Information Security Program With a $0 Budget."  Here's the video:

I have teased the BSides Detroit organizers that they ought to rename their conference to ASides Detroit because, unlike other BSides events, it doesn't coincide with another security conference, and also because it is has the best content and activities of any security conference in Detroit.  If you're in Michigan or the Great Lakes region at all, I recommend making plans to attend next year.  I'll be there.

Also, here are some other upcoming security-related events taking place in Michigan:

Saturday, October 13, 2012

GrrCON 2012 Forensics Challenge Walkthrough

This is a walk-through of the GrrCON 2012 Forensics Challenge that was designed by Jack Crook (@jackcr). Special thanks to Jack for making it so much fun and challenging!

  • You can read about the challenge here.
  • You can download the challenge files from the links here.
  • You can watch Jack's MiSec presentation on the challenge here.

1. How was the attack delivered?

Open out.pcap in Wireshark, find first TCP session, and follow the TCP stream.

Oh, look! A file that ends in *.doc.exe, that can't be good! Note the "MZ" file magic number and "This program cannot be run in DOS mode" text -- sure signs that this is a Win32 executable file.

Answer: HTTP download of

2. What time was the attack delivered?

In Wireshark, find the HTTP GET request ACK packet from the stream we just looked at. In the Frame section of the packet, locate the timestamp.

Answer: Apr 27, 2012 22:00:59

3. What was that name of the file that dropped the backdoor?

Looking at the same HTTP GET request in Wireshark, what was the name of the file from the URL?

Answer: swing-mechanics.doc.exe

4. What is the ip address of the C2 server?

In Wireshark, clear the current TCP stream filter, and browse through the packets.  As the HTTP session with the malware dropper ends, we see a new outbound connection to TCP port 443 from our victim. The destination address is the command & control (C2) server for our back door.


5. What type of backdoor is installed?

Run foremost to extract the Win32 EXE file we found in the first question: 

foremost -t exe out.pcap

Foremost will create an output directory with a subdirectory named 'exe' that should contain our backdoor. Now upload the file to VirusTotal. You should see that VirusTotal has already scanned this file. When I did it on 10/6, the last scan date was 9/28, the first day of GrrCON. :)

Answer: Poison Ivy

6. What is the mutex the backdoor is using?

This is the first answer to the challenge you have to work hard for. In order to do this the right way, you must use the memory dump to identify which process initiated the connection to the C2 server, then use its PID to find the base address and memory range, then use that to match any mutexes for that range.  (You can cheat here and Google search Poison Ivy mutexes and see if any of them are present in the mutantscan output, too, but as I said, that's cheating. :))

First, find the process that's connecting to the C2 from question #5. -f memdump.img connscan |grep

We see it's the process at PID 1096. So now we need find out what process it is and, more importantly, it's base address and memory range. -f memdump.img psscan |grep 1096

Uh-oh, that's explorer.exe, isn't it? Process injecting basterds! The base address of our pwned process is 0x0214a020. Any mutexes we find in that range are of interest to us. -f memdump.img -s mutantscan |grep 0x0214

There happens to be a funny-looking mutex close to our base address. It's not a guarantee that this is the answer we're looking for, but a quick Google search for "Poison Ivy mutex" validates the finding.

Answer: ")!VoqA.I4"

7. Where is the backdoor placed on the filesystem?

To answer this question, we'll examine the SleuthKit file (compromised.timeline) that Jack was kind enough to include in this challenge. We'll start by looking around the time that the backdoor was downloaded (4/27/12, 22:00:59) in question #2 and working forward. Also, we'll want to look for any files that are the same size as the one we extracted from the pcap file with foremost (8,192 bytes) in question #5.

(Note: If you got stuck here at GrrCON because you assumed that the filesystem time and the packet capture time were perfectly in sync, you learned the most valuable lesson there is in DFIR. There is always drift in timestamps between sources. Unfortunately, you learned it the hard way.)  

At 21:59:20 on 4/27/12, we find the prefetch temp file for the swing-mechanics.doc.exe file, and immediately after it, another file that matches the size of that file from our pcap being written to c:\windows\system32\svchosts.exe.  

(Note: There is a svchost.exe file in %systemroot%\system32 on WinXP and Win7, but there is no svchosts.exe, another clue that this is not legit.)

Answer: C:\WINDOWS\system32\svchosts.exe

8. What process name and process id is the backdoor running in?

Now, if you got this far, but cheated at question #6 instead of doing the work, you may have run the volatility pslist module, saw svchost.exe, and given the wrong answer. Oops! Cheaters never prosper. We already know that the right answer is explorer.exe and its PID is 1096.

Answer: explorer.exe 1096

9. What additional tools do you believe were placed on the machine?

Back to where we left off in question #7. Keep working forward in the SleuthKit timeline, and...

That's weird. Prefetch files indicate a program launch. Somebody ran net.exe, ipconfig.exe, and ping.exe. Likely our attacker testing network connectivity. ;-) But wait! There's more!

One thing DFIR will do to you is make you something of an expert on the names of files and folders that live within C:\WINDOWS, and, well, this doesn't look right for a lot of reasons. And when we see a file named sysmon.exe (which normally is in \WINDOWS\system32) created in this folder, and it's the same size as our binary from question #5 (8,192 bytes), we know we're looking at more bad stuff.  So in addition to a backup copy of our backdoor, there is a text file of some kind and four additional executables named f.txt, g.exe, p.exe, r.exe, and w.exe.

You can also find the handles to these files in memory with volatility as well: -f memdump.img filescan |grep -i svchosts.exe -f memdump.img filescan |grep -i systems

Answer: g.exe, p.exe, r.exe, w.exe, and sysmon.exe

10. What directory was created to place the newly dropped tools?

Answer: C:\WINDOWS\system32\systems

11. How did the attacker escalate privileges?

We can assume from the work we did in #9 and #10, that those binaries aren't copies of calc.exe, so likely one of those was used for privilege escalation. 

It would be great if we could extract them from the memory image, but since we didn't see them when we used the volatility psscan module, our chances aren't very good. Maybe we can find the command syntax that was used and get an idea of which tool was used to do what? -f memdump.img cmdscan

Well, that looks like an FTP command mixed in with Jack creating the memory dump we're analyzing, which is interesting, but not what we're looking for. Yet.

(Note: There's a good point to be made here about how by gathering the evidence, evidence was also destroyed. In the process of copying down and running mdd, Jack also overwrote most of the cmd.exe history that we are interested in for this question.)

So, no easy win to be had here. Maybe we can find what we're looking for in strings. We'll use strings and the volatility strings module to pull all of the strings >5 in length out of our memory image and see if we can find anything in there. By looking at the SleuthKit timeline file, we see that w.exe was the first of the suspicious binaries to be launched first, so we'll look for that in particular.

strings -n 5 -t o memdump.img >strings.txt -f memdump.img strings -s strings.txt >vol-strings.txt
grep w\.exe vol-strings.txt

Bingo! Those command line arguments are the username, domain name, NTLM hash, and program to run for a pass the hash attack tool of some sort. So now our attacker is running cmd.exe as Administrator. Possibly for the whole COMPANY-A domain.  

If this is your network, this is where you excuse yourself to put on clean shorts.

Answer: pass-the-hash attack

12. What level of privileges did the attacker obtain?

See above.

Answer: Administrator

13. How was lateral movement performed?

Once we understand that the attacker has become COMPANY-A\Administrator on what is probably an Active Directory domain controller, they can go wherever they want. Let's look for hostnames that aren't and see if they did anything interesting there.

grep -i \.company-a\.com vol-strings.txt

In addition to our own hostname, we see the following:

If we look back to when we ran the volatility connscan plugin for question #6, we saw a bunch of NetBIOS and LDAP connections to, which is That's pretty much a tell-tale sign that dc01 is the COMPANY-A domain controller. Which means our attacker is in fact a domain admin, and can pivot freely onto dc01 and anything else he wants. Maybe he'll do something like map a drive later. Who knows?

Answer: Credential re-use as Administrator

14. What was the first sign of lateral movement?

Now that I think of it, was there anything from in the pcap file?

ssldump -n -r out.pcap

You can do this in Wireshark, too, but one trick I wanted to show off is ssldump's ability to summarize all of the TCP sessions in a pcap file. Oh, and it looks like dc01 is also phoning home to the C2 server. Hope you packed two pair of clean shorts.

Answer: Well, I saw the C2 traffic from dc01 in the pcap file before any of the other evidence, so that's my answer.  

(Note: I think the login to the domain controller as Administrator from a workstation, which came first, should also be caught by the security ops team if they are monitoring the security EventLog on the domain controllers. (Which they should be!))

15. What documents were exfiltrated?

For this one, we had to wander around in the vol-strings.txt file we made to put the pieces together:

less vol-strings.txt

So I admit, this is a bit of a guess, but all of this looks suspicious to me. Here we have a set of files that look like the kinds of things we would want to exfiltrate. Then not that far away, we have a net use mapping a drive to dc01, then making a local directory named "1" (if you look around some more, you discover it's C:\WINDOWS\system32\systems\1). After that, the files from the shared drive are copied to that folder, and that folder is compressed in a rar file and password protected. Then an FTP command is made. That looks like data exfiltration to me.

Answer: confidential1.pdf, confidential2.pdf, confidential3.pdf

(Note: You will see later that I'm close, but managed to miss about half of the files.)

16. How and where were the documents exfiltrated?

Answer: FTP to

17. What additionl steps did the attacker take to maintain access?

Unless I missed something (which is actually quite likely), we already talked about this in question #14.

Answer: Installed Poison Ivy RAT on dc01

18. How long did the attacker have access to the network?

So, there are two possible answers Jack is looking for. There's the cynical defeatest answer, "Clearly as long as he wanted." Or there's the specific answer whereby we look at the time from the start of the first C2 connection to the end of the data exfiltration. For that, we fire up Wireshark:

The first screen is the first packet of the first C2 connection, when our attacker actually got control of the first victim system. The second screen has the filter tcp.flags.fin == 1 applied to prove a point. The last packets in the pcap file are ACKs for the C2 connections to both and (res-lab01 and dc01 respectively).  The FTP connections complete at 22:13:26, but the C2 goes on, likely past the end of the file.

Answer: 12 minutes, 21 seconds (or indefinitely)

19. What is the secret code inside the exfiltrated documents?

To get at the secret documents, we need to recreate our own copy of the encrypted rar file used for exfiltration, then decrypt it, extract the files, and view them. Fortunately, we already know everything we need to get this done.

First, extract the rar file from the pcap.

foremost -t rar out.pcap

Now, extract the files. We'll need the password, but we caught that in the volatility strings we sifted through for question #15.

unrar x -pqwerty 00002134.rar

Note the bottom of the screenshot there. Jack's got an evil sense of humor. Those aren't even PDFs! But that is in fact an OpenOffice file.

Answer: 76bca1417cb12d09e74d3bd4fe3388e9

20. What is the password for the backdoor?

Whilst reading the article we Googled for about Poison Ivy mutexes for question #6, we also learned that Poison Ivy doesn't typically use packers or cryptors, and that the C2 server and password are coded in the binary file. I wonder if they'll stick out like a sore thumb...

strings 00000002.exe

We totally recognize that IP address as the C2 server from question #4. You'll never guess Jack's favorite baseball team.

Answer: tigers

If you made it this far, thanks for reading. Hope you liked it. Oh, and your reward for reading the whole thing: In case you didn't catch it, this challenge is essentially a mirror of the attack on RSA that led to the theft of their token seed data files in 2011. When I made that realization, it gave me lulz. Jack, you're the man! :)

One more bonus! Even though it wasn't one of the challenge answers, here's the spear-phish that started the whole thing:

I have been watching you swing for the last few weeks. I believe I have come up with a major break through in your mechanics. If these adjustments are made I believe you will be back up in the bigs and batting .280 within no time. Please review the attached document before our next hitting session. Here
Lloyd McClendon

Wednesday, October 26, 2011

SIEM Market Redux

Roughly a year and a half ago, Rocky DeStefano and I had a conversation about the SIEM market in which he predicted mass acquisitions. It took longer than he originally guessed, but...

I'll throw another one out there - after more than 2 years of steady speculation of a Splunk IPO, they hired David Conte as CFO fresh from his setting up the sale of IronKey to Imation. Expect Splunk to be acquired in 2012, or at least try really hard.

Friday, May 13, 2011

GrrCON: West Michigan Security Conference

The Grand Rapids chapter of ISSA has announced a new event called GrrCON. It's a 1-day security conference that will be held in Grand Rapids, MI in September 2011. This will be one to keep an eye on over the next month or so as they get the speaker line-up solidified. I expect some cool talks and even a few surprises!

Twitter: @GrrCON
LinkedIn Events: GrrCON 2011

Saturday, October 9, 2010

Information Security for Business Majors

I recently had the pleasure of guest lecturing to a group of MBA students at Grand Valley State University on the topic of Information Security. This was a fun presentation for me to put together because it challenged me to think of how to present the business value of information security in a way that's meaningful and relate-able to a wider audience not already indoctrinated with the market and regulatory constraints in which I operate. And in this case, I think I pulled it off.

So here are the slides from that presentation, minus a few that won't translate and aren't core to the presentation itself anyway. I've also included my slide notes by title below.

1. Title

What part of the car allows it to go fast? I think it’s brakes.

How fast would you drive if your car had no brakes?

Security is like brakes - it's a set of controls, only some of which are always on, that allows your company to take bigger risks with greater confidence.

Who can define what a DoS attack is?

And can anyone describe to me how the DoS handshake works? (in the book, figure 6.6, page 269)

I am sorry to inform you that you will never need to know this. Every major vendor out there fixed this bug nearly a decade ago.

That’s the nature of security – you don’t get to stop learning or adapting, because the attackers don’t stop learning and adapting.

I hope to share with you this evening things that will take a little longer to become obsolete.

2. OK, so how bad is it really?

3. Are you scared? ...or skeptical?

Gauge your response to the previous slide.

If you were scared, consider whether or not you would panic in the face of a catastrophic security event.

If you were skeptical, consider whether or not you would take a threat serious enough to be prepared.

4. The sky is always falling!

The average time for an unpatched Windows server on the Internet to be compromised is 3-6 hours.

The overwhelming majority of data breaches are caused by human error.

If you have any one of these things, hackers can monetize them.

Computers of any kind can be rented out to send spam or launch DDoS attacks.

Personal data, referred to as “dumps” are stolen and sold by the thousand on the Internet

Money in bank accounts is transferred by EFT and then wire transfer out of the country where it is laundered.

Credit card numbers are used to purchase stolen goods which are shipped overseas.

Despite all of this, consumer-based ecommerce continues to grow 15-20% annually.

If you sell to consumers, the Internet isn’t where you want to be, it’s where you HAVE to be.

5. Information Security's Business Value

Information security can be summed up as “loss avoidance”

The value proposition is that these efforts are less expensive than the consequences of not having them.

Regulation makes some parts of security the price of admission, the rest is about striking a balance between security and flexibility.

Bruce Schneier’s book, Beyond Fear.

6. How Information Security Works

Known as the CIA Triad, these are the “ilities” that security controls impact directly.

There are other “ilities





But even at its best, security is only an enabler of these things. At either extreme, security blocks them.

7. The Goals of Security

8. Policy

9. Controls

Preventive IT controls are not infallible, and covering 100% of corner cases with your controls costs too much and hamstrings your actual business.

Auditing controls are time-consuming, and usually any damage is already done by the time an audit discovers it.

Monitoring controls are typically based on sampling, which means you might miss something. More intended as a quality or health check.

10. Tools of The Trade - Preventative

11. Tools of The Trade - Auditing

12. Tools of The Trade - Monitoring

13. Risk Management (1)

14. Risk Management (2)

15. Incident Response

I like the Richard Clarke quote from your book. “If you spend as much on information security as you spend on coffee, you will be hacked, and you’ll deserve to be hacked.”

Of course, Mr. Clarke is wrong, because having a security incident is not an issue of if, but an issue of..?

Wrong. Not “when” but “how often.”

16. (graph)

17. Awareness & Consultation

Consulting on projects or with operations teams leads to better security outcomes because security is considered earlier in the process.

Raising awareness and then inviting people to share concerns is a great way to organically scale your visibility to issues.

By being proactive and meeting colleagues where they are, you gain goodwill for your security efforts. This is a key piece of a successful security program. Strong-arm tactics are a guaranteed path to failure. Without goodwill and trust, the security practice in your company quickly becomes an obstacle for people to bypass in order to get their jobs done. This is how you lose your job.

18. How IT Security Fails

19. (image)

20. (image)

21. (image)

22. You say "potato," I say "No."

23. Communication

24. Why Buying Security Fails

Buying and integrating security technology only works some of the time, and that time is not right now.

Information security is an arms race.

Technology is both the weaponry and the battlefield.

Security is not a problem that can be solved.

Security is a practice that must be maintained with people and process.

25. Discussion

Thursday, May 20, 2010

The SIEM Market Discussion Continues

Bill Roth of LogLogic commented on my Twitter exchange with Rocky DeStefano of Visible Risk where we talked about LogLogic's announcement that they were discounting their SIEM product. I then wrote a reply, and it got a little long. So I made it a blog post instead.

Rocky, Paul:
The ClueTrain Manifesto calls markets "conversations", so here goes.....

I think you're falling into a the trap of "conventional wisdom". First off, the basic assumption that the world falls neatly into the SIEM categorization is just plain false. I stand by LogLogic's all starts with log management as the crucial piece, without that key use cases like network forensics are not even possible. Second, the notion that dropping the price is bad is just plain weird. Is LogLogic dropping the price to sell more? Sure we are. Are we dropping the price to take market share? Sure we are. Are we seeing a great response? Sure we are. Since when is saving people money a bad thing?

And we're always interested in a podcast. :)

Bill Roth, EVP

Hi Bill,

Thanks for the comment! And thanks for participating in the dialogue. I think it's awesome that LogLogic is out front and engaging on its business decisions. Very refreshing!

As to your point about log management being that crucial initial component of a SIEM implementation, I agree completely. Log management has also developed as its own market segment as well, independent of SIEM. But I don't need to tell you that. :-)

On the topic of LogLogic's decision to discount its SIEM product, I didn't mean - and I don't believe Rocky did either - that charging less for SIEM is bad, or even a bad business move.

That said, I do believe that for some significant portion of potential customers log management is a commodity technology. However, from my own experience and from everything I've seen to date, SIEM is not a commodity technology, and I'm not convinced it will be. As such, I don't see price as a strong competitive differentiator in the SIEM market.

Following the recent recession, where IT capital budgets still haven't caught up to the (hopefully sustained) economic upturn, I imagine the feedback on LogLogic's price cut has been positive, and that you'll see some SIEM sales where you wouldn't have but for the discount. But in the mid- to long-term, I have my doubts as to whether there is any meaningful gain in market share to be had for LogLogic - or any SIEM vendor for that matter - simply by competing on price with other SIEM vendors.

Let's be frank, if price were a big piece of why companies choose a particular SIEM, Cisco MARS would have the lion's share of the market and ArcSight would be folding. Instead, it's the other way around.