A Geek Raised by Wolves

ssdeep 2.8 Released

Fri, 25 May 2012 12:31:45 GMT

I have published version 2.8 of the ssdeep tool for fuzzy hashing. This is a bug-fix release, most notably quashing an issue on Win32 regarding spurious spaces in filenames. You can download the Windows binary or the *nix source code.

Links from Storytelling Talk

Fri, 25 May 2012 12:29:27 GMT

Here are all of the links I referenced in my talk on storytelling in computer forensics. The video should be posted soon. These are all great examples of storytelling!

Congressional Medal of Honor - Formal stories

"Fenimore Cooper's Literary Offenses" - Mark Twain's rules for Storytelling

Lamps as characters:
Pixar Jr.
IKEA Lamp

The 500 Mile Email - An excellent technical story

Clean All The Things

Cindy Murphy's case study, http://bit.ly/IkLZku

The Happy Secret to Better Work TED Talk

Receiving Mode-S Beacons with the Universal Software Radio Peripheral - Why screen captures are hard

http://www.storytellinginstitute.org/

DFIROnline, Bitlocker library, and SysInternals Updates

Tue, 15 May 2012 13:06:26 GMT

I will be giving a talk at 8pm EDT this Thursday on storytelling in computer forensics as part of the DFIROnline series, http://www.writeblocked.org/dfironline.html. The results of your work are useless unless you can convey them to somebody else. Along with formal reports and testimony, storytelling can be a great way to communicate. This talk will explore some of the aspects of storytelling and how they can be applied to our field.

Continuing with the links on Bitlocker Disk Encryption, there's now an open source library for handling such volumes. I haven't tried it out yet, http://code.google.com/p/libbde/

Finally, Mark Russinovich has updated some of the popular SysInternals tools, including AutoRuns, Strings, and LiveKD, http://blogs.technet.com/b/sysinternals/archive/2012/05/14/updates-autoruns-v-11-3-livekd-v-5-2-strings-v-2-5-and-trojan-horse-mark-s-sequel-to-zero-day-available-for-pre-order.aspx.

Free program for decrypting Bitlocker protected volumes

Thu, 03 May 2012 10:27:58 GMT

Romain Coltel has published a tool called Dislocker for decrypting and mounting Bitlocker protected drives on OS and Linux, http://www.hsc.fr/ressources/outils/dislocker/. The program can use FUSE to mount the drives on the system, or optionally decrypt the drive wholesale. It can use a recovery password (i.e. the string of digits), an external key file (BEK), or a clear key. I haven't had a chance to test it yet, but it's great to see a Free solution! Thanks Romain!

Decrypting Damaged BitLocker Protected Volumes

Tue, 01 May 2012 14:54:58 GMT

Recently I had the chance to examine a Windows 7 system protected by Bitlocker Drive Encryption (BDE). While I was ultimately successful in recovering the encrypted drive, the case showed me how some of my 2009 paper on BDE [1] was inaccurate or omitted pertinent information. The remainder of this post corrects and fills in the gaps of that paper and provides some details about the changes Microsoft has made to Bitlocker since it was published.

Before getting started with those details, I have to credit Nitin and Vipin Kumar for posting details and source code for reading Bitlocker protected volumes [2]. Their work was invaluable when writing the original paper, and proved so again in this case.

First, in my paper I was incorrect when I stated the size of the Full Volume Encryption Key (FVEK) was always 512 bits. As Kumar and Kumar note, the size of the key varies based on the algorithm being used. The FVEK is 512 bits if either of the Elephant diffuser modes is used. But if they are being used, the key is the same size as the encryption strength. That is, when working in AES128 mode the FVEK is 128 bits, and when in AES256 mode, the FVEK is 256 bits.

Second, when the Elephant diffuser is not in use, each sector is encrypted and decrypted using AES in CBC mode with the initialization vector set to all zeros. The sector number has no impact on the encryption process. As a side note, the practical effect of this decision is that identical sectors will appear identical in both ciphertext and plaintext. Whether or not that's a practical advantage for an attacker is debatable, but my personal recommendation is to use one of the Elephant diffuser modes.

Third, my paper did not specify how Windows would deal with a BDE protected volume if the volume header becomes damaged. My current case involved such a damaged drive and I now have an idea of how Windows handles this situation: it doesn't. Neither BDE nor the repair-bde [3] program were able to make heads or tails of the volume. I had to write a custom program, "Scarlet", which could decrypt the volume [4].

Finally, the changes in Bitlocker version two are documented in my presentation on BitLocker to go [5]. These include things like the new metadata format and passwords as volume protectors.

[1] Jesse Kornblum, Implementing BitLocker Drive Encryption for Forensic Analysis, Journal of Digital Investigation, 2009, (5)3,pp. 75-84. http://jessekornblum.com/publications/di09.html.
[2] Nitin and Vipin Kumar, Analysis of Window Vista Bitlocker Drive Encryption, http://nvlabs.in/
[3] Microsoft Corporation, How to use the BitLocker Repair Tool to help recover data from an encrypted volume in Windows Vista or in Windows Server 2008, 2010, http://support.microsoft.com/kb/928201.
[4] Why Scarlet? Because frankly, I don't give a damn how you get the keys, but you have to have the keys to decrypt the drive. Margaret Mitchell (novel) and Sidney Howard (screenplay), Gone with the Wind, Warner Brothers pictures, 1939.
[5] Jesse Kornblum, BitLocker to Go, DoD Cyber Crime Conference, 2010 http://jessekornblum.com/presentations/dodcc10-1.pdf.

Ask the Guru, Computer Forensics Storytelling

Mon, 23 Apr 2012 18:42:19 GMT

I have joined the Twitterverse! Look for me at https://twitter.com/jessekornblum and please be kind to the new guy.

My company, Kyrus, is hosting an "Ask the Guru" forum. If you have a question about anything in computer forensics or computer security, please drop us a line! We will answer your questions as they come in.

It seems my post on calling for storytelling in computer forensics has earned me a spot on DFIROnline. I'll be on the show on May 17th, talking about the different kinds of stories, how they can be told, and you can learn to tell better ones. See you then!

Windows Internals 6

Mon, 09 Apr 2012 16:15:44 GMT

According to the email, my copy of Windows Internals sixth edition has shipped!

sdhash version 1.8 released, compiling on OS X

Fri, 06 Apr 2012 11:29:29 GMT

The latest version of sdhash, Vassil Roussev's similarity program, has been released. The new code is version 1.8 and has been ported to C++. I haven't had a chance to test it out yet, but this version adds a flag to generate hashes from a specified list, adds an API, and fixes a minor bug.

The new code doesn't compile out of the box on OS X, but here's how to do it. Version 1.8 relies on the c++0x standard, which means you may need to update your C++ compiler. On my system, for example, I had to use a C++ compiler installed via MacPorts' gcc44 package. You'll also need to make a change to the Makefile. The lines

CC = g++
LD = g++

should be changed to:

CC = g++
LD = $(CC)

After that, the commands I used were:

$ sudo port install gcc44
$ wget http://roussev.net/sdhash/sdhash-1.8.zip
$ unzip sdhash-1.8.zip
$ cd sdhash-1.8
$ make CC=/opt/local/bin/g++-mp-4.4
$ ./sdhash 
sdhash-1.8 by Vassil Roussev, Candice Quates, Mar 2012
[...]

References:

sdhash Homepage, http://roussev.net/sdhash/sdhash.html
MacPorts, http://macports.org/

Tell a Story

Thu, 05 Apr 2012 11:49:59 GMT

I would like to put a slightly different spin on the recent calls for case studies. Yes, please share what you've been working on. But don't think of it as a "case study". Case studies are dry and dull. I did this, then this happened, and after examining nine thousand registry keys there was data under HKCU/Software/Microsoft/Windows/CurrentVersion/NobodyReads/ThisFar/Without/FallingAsleep.

Instead, tell us a story.

Humans are story tellers. From the beginning of our history we have told stories and eagerly listened to them. We have recognizable archetypes present in our tales: the hero, the wise old man, the trickster, and the warrior. You can see them in Odysseus. You can see them in Star Wars.

A technical description of an exciting subject, such as tracking airplanes in real time using a software radio and Google Earth [1], can be made as dry as dust.

The best stories both entertain and educate. They may be followed by an appendix of technical details, but they have a story, with characters. You are star of your investigations. What did you do? What did you see? What were you thinking and feeling? Tell us!

Do you want to change minds? Tell a tale. All of the scholarly articles in the world didn't have the same impact as The Jungle [2]. Use the power of words. Nobody objected to lean finely-textured beef. But within a month pink slime was off the market.

Do you enjoy the TED talks? They convey fantastic scientific information in story form. The science of happiness is based on statistics and studies, psychology and pharmacology. But when told as a story, it begins with a happy baby unicorn [3].

Protect your clients, hide their details. Change the names to protect the innocent. But when you do change those names, remember Dragnet. The show famous for giving you "just the facts", is, in fact, a marvelous example of stories told well.

References:

Friday Links

Fri, 30 Mar 2012 15:14:19 GMT

Three links for your Friday:

There's a great post on the Carbon Black blog about the efficacy of anti-virus products over time. They've explored how long it takes for A/V software to detect malicious software, if ever.

There have been some exciting developments regarding parsing Windows SuperFetch files. SuperFetch was introduced in Windows Vista. The system "learns" when you typically run certain programs and loads the resources necessary to run them into memory just before they are needed.

Last October ReWolf published some details about the SuperFetch files, including how they're compressed the structures they contain. We still don't know how to intprete these data for forensics, but it's a step closer to meaningful analysis!

Nick Stone has published a new tool, DeepDigest, a Qt-based GUI program based on the md5deep suite for recursive hashing. I haven't had a chance to test it out, but Nick has said it's only supported on Linux for now. (Seriously, test before you run. It could work flawlessly. It could erase your hard drive and play Bee Gees songs.) I'm really glad the code in md5deep is being used in other projects. Hooray for open source!

Doing Our Part Against Zeus

Mon, 26 Mar 2012 15:03:15 GMT

My company, Kyrus, has helped in the fight against the Zeus botnet! Read all about it:

md5deep 4.1 with Windows executable identification

Tue, 14 Feb 2012 16:23:54 GMT

This morning I have published md5deep version 4.1. There is one new feature, an expert mode which processes Windows PE (executable) files. Traditionally expert mode has been used to include or exclude symbolic links, block files, etc. But a recent feature request asked for the ability to recognize and hash PE files. Using the functionality I wrote for Miss Identify, I've added the feature to md5deep and hashdeep.

Here's an example of the new feature in action. First, we recursively hash a directory tree without any restrictions:

C:\temp>md5deep -r .
2baa55c512b251ba3ca882fcf14bde7f  C:\temp\bar\EVILEVIL.txt
3fed0738937bb96527cf6e7b17299d23  C:\temp\bar\sha1deep.exe
6dd4566eb245627b49f3abb7e4502dd6  C:\temp\bar\sometext.txt
3fed0738937bb96527cf6e7b17299d23  C:\temp\bin\hashdeep.exe
4bcd10a9e5a367e91df7dbc55f7a22f5  C:\temp\foo.txt
607e033a16006ed1e9987cfc62562f72  C:\temp\hexdump.exe

Note the two "text" files, foo.txt and EVILEVIL.txt. When we request that md5deep only hash Windows executables, we see the latter was mislabeled! The program displays a warning about this file and hashes it:

C:\temp>md5deep -r -o e .
C:\temp\bar\EVILEVIL.txt: Is Windows executable but does not have executable extension
2baa55c512b251ba3ca882fcf14bde7f  C:\temp\bar\EVILEVIL.txt
3fed0738937bb96527cf6e7b17299d23  C:\temp\bar\sha1deep.exe
3fed0738937bb96527cf6e7b17299d23  C:\temp\bin\hashdeep.exe
607e033a16006ed1e9987cfc62562f72  C:\temp\hexdump.exe

There is also one bug fix in this release, better handling of junction points on Windows. As usual you can download a Windows executable or the *nix source code.

Using the latest NSRL hashes

Thu, 09 Feb 2012 23:48:39 GMT

I've updated the Kyrus NSRL server to use the most recent hash set. We had been one set back. More details, including a rebuttal to criticism of the NSRL backed up with experimental evidence, is over at the Kyrus blog.

Updating findaes

Tue, 07 Feb 2012 00:44:37 GMT

I have updated my findaes utility. This program searches for 128, 192, and 256-bit AES keys in input files. It was intended to search memory images for the keys used by programs like BitLocker and TrueCrypt, but can be used on any kind of data. The new version is much faster! For each key schedule found, the program displays the offset and the key itself, like this:

C:\> findaes file.vmem 
Searching file.vmem
Found AES-128 key schedule at offset 0x23f20cc: 
6f 98 76 7f 65 b0 6e a6 6d 6f 65 48 60 6f ad be 
Found AES-128 key schedule at offset 0x23f2354: 
93 8e 8b d3 9b 14 d6 a3 4d 30 83 fb 11 96 74 ee 
Found AES-256 key schedule at offset 0x3fc93008: 
c9 4d a2 7b e9 a0 76 18 67 18 a3 26 e4 33 08 1c 7f ed b0 b2 9c 9f 31 5c 51 03 bb 52 b8 01 2d 4e 
Found AES-256 key schedule at offset 0x3fc944d4: 
ea ef 70 ee 22 c4 a1 3a 21 cb 5e 53 ea 2e 98 c8 a6 21 ef 9e d6 d7 92 fb f9 70 b2 cc 94 64 f7 2e

It takes some work to use these keys to decrypt a protected volume. Unlike getting the passphrase, which can be used directly with the encryption program in question. But this is what some computer forensics tools are doing behind the scenes to decrypt protected volumes.

The Windows executable and source code come in one download. The code is public domain. Enjoy!

Slides from DoD Cyber Crime Conference

Fri, 27 Jan 2012 23:55:36 GMT

As promised, I've published the slides and scripts I used during my talks at the 2012 DoD Cyber Crime conference. These are:

Audits, Triage, and the Future of Hashing - Along with the slides, there is the Python script for doing fast hash matching. That is, matching hashes using the file size as well as the hash. As described in the presentation, comparing the sizes first is much faster!

Cake and Grief Counseling will be Available: Using Artificial Intelligence for Computer Forensics Without Jeopardizing Humanity

Kyrus Beta Testing NSRLquery Server

Thu, 26 Jan 2012 18:06:04 GMT

Kyrus is beta testing a public NSRLquery server and we invite you try it out! This server allows you to submit file hashes to determine if those files are present in the National Software Reference Library (NSRL). Our server, nsrl.kyr.us, is free to use. You can submit MD5 hashes using the nsrllookup client. It's designed to use hashes such as those generated by md5deep or md5sum.

Feel free to try it out or use it in your next investigation. For example, you could be working on-site and want to consult the NSRL. Didn't bring all 1.5GB of it with you? No problem! Pipe the output of md5deep into nsrllookup, like this:

C:\> md5deep -r * | nsrllookup -s nsrl.kyr.us
305e40dee29d261d0a3dc466f2184e35  unknown.exe
607e033a16006ed1e9987cfc62562f72  EVILEVIL.exe

By default the server returns the hashes of those files which are not in the NSRL. If you instead want the hashes of the files which are in the NSRL, just add the -k flag. For example:

C:\> md5deep -r * | nsrllookup -s nsrl.kyr.us -k
e97295de2a9fde547feab4fe41df16ca  mspaint.exe
eee470f2a771fc0b543bdeef74fceca0  msiexec.exe

If you'd rather not pipe the output directly, you can use a previously saved file of hashes:

C:\> type known.txt | nsrllookup -s nsrl.kyr.us

C:\> nsrllookup -s nsrl.kyr.us < known.txt

There are a few other command line options. Use the -h flag to see them all.

If you try out the server, please let me know what you think! Post a comment below or send mail to jessek [at] kyr [dot] us.

md5deep version 4.0.1, critical bug fixes

Sun, 22 Jan 2012 17:53:18 GMT

I have published version 4.0.1 of the md5deep suite. This release fixes three critical bugs and I urge all users to upgrade immediately. You can view details of these bugs, which caused incorrect hashes of input via stdin on Windows, a hang on DFXML output generation, and a "Too many open files" error on OS X.

If you don't know what that means, don't worry about it. But upgrade anyway. [grin]

Special thanks to Dr. Simson Garfinkel who helped to track down and fix these problems.

You can download Windows binaries or the *nix source code.

The WoW Effect and the md5deep suite

Fri, 20 Jan 2012 13:43:52 GMT

Last November Christian Wojner of the Computer Emergency Response Team Austria published a whitepaper, The WOW-Effect. Here's the abstract:

The 64-bit version of Microsoft Windows includes file-system virtualization features to run 32-bit programs. File access is transparently redirected to other directories in certain cases. This feature can easily fool an analyst looking at a running system and can have a massive impact on infection-driven forensics, malware analysis and comparable investigations. In the worst case this can lead to an entirely wrong interpretation of a case/situation. While this issue is not entirely new, it is necessary to raise the IT-Security community's awareness, as some of the common tools and procedures in use need to be adapted in the presence of the files system redirector.

It's a great paper and explains the Windows on Windows issues with examples. In one of those examples Mr. Wojner uses a 32-bit version of md5sum.exe to hash C:\Windows\system32\ieapfltr.dll. The program is silently redirected by WoW and hashes C:\Windows\SysWOW64\ieapfltr.dll instead.

The md5deep suite contains 64-bit versions of each program precisely to avoid the filesystem redirection of WoW. Starting with version 3.8, released in April 2011, these 64-bit programs aren't redirected. Additionally the 32-bit versions of the programs display a warning message when executed on a 64-bit system. The 64-bit version of each program has the suffix '64'. For example, the 64-bit version of md5deep.exe is md5deep64.exe.

Repeating the example from the WoW-Effect paper, here's an attempt to hash the DLL in question using the 32-bit version of md5deep on a 64-bit Windows 7 machine:

C:\>md5deep C:\Windows\System32\ieapfltr.dll C:\Windows\SysWOW64\ieapfltr.dll
md5deep: WARNING: You are running a 32-bit program on a 64-bit system.
md5deep: You probably want to use the 64-bit version of this program.
ee9d715af1b928982f417238b9914484  C:\Windows\System32\ieapfltr.dll
ee9d715af1b928982f417238b9914484  C:\Windows\SysWOW64\ieapfltr.dll

The 32-bit version of the program produced the same (incorrect) hash as in the paper. But that warning message, displayed to standard error, should catch the eye of the user. (Well, ok, it's a correct hash, but of a different file than was specified on the command line.) Repeating the command with the 64-bit version of the program gives us the non-redirected view of the filesystem and the correct hash for ieapfltr:

C:\>md5deep64 C:\Windows\System32\ieapfltr.dll C:\Windows\SysWOW64\ieapfltr.dll
8eada158d964e3fd1999ad96c9c507ff  C:\Windows\System32\ieapfltr.dll
ee9d715af1b928982f417238b9914484  C:\Windows\SysWOW64\ieapfltr.dll

The moral of the story is to use the 64-bit versions of the md5deep suite on 64-bit systems! There are several ways for you or your scripts to determine if you're running on a 32 or 64 bit platform. For example, as noted in Microsoft's WoW64 Implementation Details, the environment variable PROCESSOR_ARCHITECTURE will be x86 on 32-bit systems, and either AMD64 or IA64 on 64-bit systems. Here's one way to pick the right program for the system's architecture using a batch script:

@if "%PROCESSOR_ARCHITECTURE%" == "x86" (set MD5DEEP=md5deep.exe) else (set MD5DEEP=md5deep64.exe)
%MD5DEEP% -re C:\Windows\System32\*

The '@' symbol in front of the if statement causes that line not to be displayed when the batch script is run. This is handy if you're saving the output of this command to be used later, such as for hash matching.

References:

The WoW-Effect, full paper
WoW64 File System Redirector. Details on which directories are redirected under WoW64.
Thanks to Dave Kleiman for posting the WoW affect paper on the win4n6 mailing list.

md5deep and hashdeep version 4.0.0

Fri, 06 Jan 2012 13:42:12 GMT

I am pleased to announce the official release of md5deep and hashdeep version 4.0.0. These programs represent a huge amount of work by me and Dr. Simson Garfinkel. We've rewritten lots of the programs in C++, added regression testing, added some new features, and in general done a much needed overhaul on the programs. After a long beta period, we feel confident about releasing these programs.

Please download the new versions and test them out. Although not much is different on the surface, there is a lot of new code under the hood. Make sure your favorite scripts and tasks still work! Try to break the new code. All bugs and nits should be reported on the SourceForge Bug Tracker, but you can also comment here.

You can grab the Windows binary or the *nix source code.

NSRL Query Tool

Tue, 03 Jan 2012 11:51:23 GMT

The National Software Reference Library, or NSRL, is a wonderful resource that is difficult to use. The set contains millions of hashes of known files, which could be used in an investigation to eliminate them from further consideration. But the sheer size of this set, at millions of hashes, makes it unwieldy at best. There are many duplicates, but even when those are eliminated, the resulting files are so big as to crash many programs. The md5deep suite, for example, will crash while attempting to load the whole thing.

But thanks to Robert Hansen of Redjack, there is now a tool for using the NSRL practically! He's written a client/server program, NSRLQuery, which takes the output of sha1deep and compare it against the NSRL. (Why SHA-1 hashes? The NSRL contains MD5, SHA-1, and CRC32 hashes. You have to pick one...) The results are written to files hits.txt and misses.txt. The former are the files from the NSRL, the latter are those which are not.

Lightning Talk Report

Wed, 14 Dec 2011 14:44:07 GMT

Last night I gave a six minute "lightning talk" at the first SANS 360 event in Washington DC. Without question preparing for this talk was far more difficult than any talk I've given in the past five years. Why? Because with only six minutes, there is no time for thinking on stage. You only have enough time to tell your story and nothing else.

The format worked well, and I hope SANS continues to use the lightning talk format. It won't work for every topic, but I think it's a great addition to the lineup.

If you missed the live talk, you can see the archived video online.

SANS 360 on December 13th

Wed, 07 Dec 2011 11:05:12 GMT

Next week I am going to be giving one of ten Lightning Talks at the SANS Cyber Defense Initiative in Washington DC. These 360 second presentations will cover a wide array of topics in the Digital Forensics and Incident Response world. The speakers also include Ovie Carroll, Harlan Carvey, Jeff Hamm, Christopher Porter, Chris Taylor, Mary Horvath, Jon Stewart, Mike Cloppert, and Jerod Alexander. It's free to attend, but you must register in advance.

If you can't make it to Washington DC you can watch the presentations live on the web!

See you there!

Second beta of md5deep version four

Wed, 30 Nov 2011 13:24:08 GMT

Based on your feedback, we have revised the new md5deep version 4.0 and published a second beta. This version should eliminate the performance issues some users were reporting and fix a few bugs. Thank you for all the great feedback. Please keep it coming!

You can download a Windows executable or the *nix source code.

First beta of md5deep version 4

Mon, 07 Nov 2011 14:02:57 GMT

I am thrilled to announce the first beta of md5deep version four. This is an almost total rewrite of the program thanks to Dr. Simson Garfinkel. We have ported the program to C++ and added some new features. The most noticable will be multiprocessor support, which should greatly speed up your work, especially when processing multiple files. The programs can now also produce output in DFXML mode.

Again, this is beta code. We are releasing it for your testing purposes only. Please do not use this code for live cases yet. Report any bugs or problems you find to the SourceForge bug tracker and we will address them in turn. If all goes well we hope to release the official md5deep version 4 on 1 Jan 2012.

You can download directly the Windows binary or *nix source code.

ssdeep version 2.7 released

Fri, 30 Sep 2011 11:46:44 GMT

I have published ssdeep version 2.7, the latest version of my fuzzy hashing program. This version adds two features. First, you can now process up to 100MB of standard input. If you need more than that, let me know and we can work something out. But the program now accepts piped input, like this:

$ some-process FILE | ssdeep
ssdeep,1.1--blocksize:hash:hash,filename
768:zRAlerwBDvABneZLBj9WXLftbmo4/Zhc+:J8BvABU9j9WXLftizh2+,"stdin"

The program now also displays an error message if it doesn't process any file large enough to produce meaningful output. This often happens when people are testing out the program and try hashing "hello world". When this happens now, emphasis added:

$ echo "hello world" > foo $ ssdeep foo ssdeep,1.1--blocksize:hash:hash,filename 3:iKFSMPv:rJPv,"/tmp/foo" ssdeep: Did not process files large enough to produce meaningful results

You can download a Windows binary or the source code.