A Geek Raised by Wolves

md5deep version 3.5

2009-12-22T12:54:24Z

This morning I've published md5deep version 3.5. This release adds a new feature, the ability to hash a set of files specified in an input file (-f mode), and fixes two bugs. You can download Windows executables or the source code or *nix, OS X, BSD, Cray, Commodore 64, etc.

Caffeinated Crash Course in Computer Forensics

2009-12-01T19:56:09Z

As part of MIT's Independent Activity Period, I'll be giving a two-hour Caffeinated Crash Course in Computer Forensics on the MIT campus on Wednesday, January 13th. MIT gives students the whole month of January to do as they please. They can take a fun course, go to charm school, or just relax. The Student Information Processing Board puts on a series of short classes all about computers, programming, and the like. I've agreed to teach a block and hope you can come check it out. The class will run from 6pm-8pm in room 66-160.

The course will be a two-hour high-speed, low drag introduction to computer forensics. You'll see what kinds of data can be recovered from various devices found in the real world and in the cloud. There's treasure everywhere! Topics covered include hidden data in office documents, filesystems, flash storage, cell phones, GPS devices (which are mostly Linux boxen, anyway), computer memory (RAM), social networking sites, the Lobby 7 Anomalous Activity Hypothesis, Jesse's Dead Body Theory of Evidence, and the story of FRED. By the end you'll understand information hasn't been destroyed unless the media has been raised to the Curie Point.

Fuzzy Hashing example

2009-11-15T14:51:26Z

Over at The Digital Standard, Chris Pogue has written an excellent example of how fuzzy hashing can be used in computer forensics. Thanks Chris, and have fun with fuzzy hashing!

When I'm Sixty Four (Bits)

2009-11-03T12:28:46Z

Sixty-Four bit computing is here! What does that mean? What will the impact be on computer forensics? I've written a tech note, When I'm Sixty Four (Bits) to explain the changes. (You could also use it as justification for buying some new hardware...) The first paragraph:

Forensic examiners are going to have to process computers running 64-bit operating systems in the near future. While this won't affect people doing document forensics, it will require significant changes for those who do code based forensics such as reverse engineering, malware analysis, etc.

BitLocker To Go, Google Earth Forensics at DoD Cyber Crime Conference

2009-09-09T14:35:00Z

I have been selected to give two presentations at 2010 DoD Cyber Crime Conference in January 2010 in St. Louis, MO. Unfortunately the St. Louis Blues will be out of town during the conference. Does anybody have some ideas on what to see and do during the off hours? The ~~Stella Artois~~ Anheuser-Busch tour? The gourmet burger bar? What else?

My first talk will be on BitLocker To Go, how Microsoft has extended BitLocker Disk Encryption to removable devices like USB sticks. You can learn how the technology works, how it uses passwords and smart cards, its applications for force protection, and how the protected data can be accessed during forensic examination. The second talk will cover Google Earth cache file forensics. You'll see what data is in the file, how it's stored, and how it can be viewed.

Speaking of BitLocker, we discussed the tool in the most recent CyberSpeak podcast, published on Monday. The show was recorded live at the SANS What Works in Computer Forensics conference a few months ago. You can listen as Ovie and Bret interview me, Harlan Carvey, Ken Bradley, and Rob Lee on a host of topics.

Hibernation File Signatures

2009-08-18T20:31:51Z

There are four legal values for the magic value at the start of the Windows hibernation file. This post details the values, hibr, (null), wake, and link, and the circumstances under which they are created. They represent, in order, valid hibernation data, successfully restored hibernation data, unsuccessfully restored hibernation data, and a link to a hibernation file elsewhere. During this work I also found three noteworthy things about the hibernation process.

The file format of the Windows hibernation file begins with a PO_MEMORY_IMAGE structure. This structure begins with a four byte magic value. Matt Suiche and others have written that this magic field can take on several values, but only two have been explained. The first value, hibr or HIBR indicates the file contains valid hibernation data. (The uppercase version is found on Windows Vista and later.) The other known value, all zeros, indicates the file has been previously successfully restored. In fact, when the file has been successfully restored, the first 0x1000 bytes of the file should all be zeros.

The other values thought to be legal in a hibernation file, wake and link, indicate either a failed restore operation or a link to another partition's hiberfil.sys file. This post explains how these methods work and provides methodologies for creating them.

These experiments were conducted using a VMware Fusion virtual machine running Windows XP Service Pack 3. The system was booted and hibernated normally. The virtual disk was mounted and the hibernation file was recorded. Here's the start of the hiberfil.sys:

0000000: 6869 6272 0000 0000 13f6 0000 a800 0000  hibr............
0000010: 5a5b 0000 0010 0000 0000 0000 0000 0000  Z[..............
0000020: 1aea 39fe 1320 ca01 d431 cc19 0000 0000  ..9.. ...1......
0000030: ff3f 03a0 0200 0000 1000 0000 0000 beff  .?..............
0000040: 00bf 0601 0000 0000 0003 0000 5045 0000  ............PE..
0000050: c6fb 0000 3d84 0000 0300 0000 5584 0000  ....=.......U...
0000060: 8145 839d 0000 0000 58f0 ef1e 0200 0000  .E......X.......
0000070: fe01 9004 0000 0000 797f 0e3f 1900 0000  ........y..?....
0000080: 3409 0000 9304 0000 2100 0000 c70f 0000  4.......!.......
0000090: 482d 0000 3d84 0000 20d4 3908 dd08 0000  H-..=... .9.....
00000a0: 0300 0000 0000 0000 0000 0000 0000 0000  ................

The virtual machine was restarted and allowed to begin restoring. When the progress bar was just shy of completion the VM was powered off. The virtual disk was mounted again and the hibernation file recorded. The second hibernation file was identical to first except for the first four bytes. Those four bytes had changed from hibr to wake:

0000000: 7761 6b65 0000 0000 13f6 0000 a800 0000  wake............
0000010: 5a5b 0000 0010 0000 0000 0000 0000 0000  Z[..............
0000020: 1aea 39fe 1320 ca01 d431 cc19 0000 0000  ..9.. ...1......
0000030: ff3f 03a0 0200 0000 1000 0000 0000 beff  .?..............
0000040: 00bf 0601 0000 0000 0003 0000 5045 0000  ............PE..
0000050: c6fb 0000 3d84 0000 0300 0000 5584 0000  ....=.......U...
0000060: 8145 839d 0000 0000 58f0 ef1e 0200 0000  .E......X.......
0000070: fe01 9004 0000 0000 797f 0e3f 1900 0000  ........y..?....
0000080: 3409 0000 9304 0000 2100 0000 c70f 0000  4.......!.......
0000090: 482d 0000 3d84 0000 20d4 3908 dd08 0000  H-..=... .9.....
00000a0: 0300 0000 0000 0000 0000 0000 0000 0000  ................

The VM was again restarted and it displayed a message, "The last attempt to restart the system from its previous location failed. Attempt to restart again?" The options given were "Delete restoration data and proceed to system boot menu" and "Continue with system restart" (Screenshot).

Researchers attempting to verify this result can also hibernate a system, manually edit the hibernation file while it's powered off, and then restart the system to achieve the same effect. Although this will allow you to see how Windows handles the file, it does not allow you to verify that Windows creates this file.

The link code allows the system to boot another hibernation file found elsewhere. According to the blog post How Windows Starts Up (Part the Second) , the link signature should be followed by an ARC path similar to those found in the boot.ini file, like this:

linkmulti(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect

Note there is no space between the link signature and the ARC path. The ARC path specifies another partition where the bootloader searches for a hiberfil.sys file to restore.

There are three other noteworthy facts I discovered during this research. First, I found some information on Hibernate Once Restart Many (HORM) mode. This is a feature of Windows XP embedded that allows the system to be booted multiple times using the same hibernation file. HORM mode depends on the Enhanced Write Filter (EWF) which is not included with other editions of Windows XP. There are methods to enable HORM on Windows XP, but they are cumbersome at best. The best indicator of HORM mode is the presence of a file resmany.dat in the root directory. See also Microsoft's description of HORM and the official documentation.

Second, I verified that a Windows system will attempt to parse or even restore the contents of hiberfil.sys, even if Hibernation mode is not enabled in the Control Panel.

Finally, hibernation is not supported on any Windows XP, Server 2003, Vista, or Server 2008 system with more than 4GB of RAM. See http://support.microsoft.com/kb/888575/ for details.

Cached TrueCrypt Passphrases

2009-08-09T20:09:55Z

Chris Neilson, a senior at the University of Denver, has discovered how TrueCrypt caches passphrases in memory when the user requests the program to do so. Please note that caching passphrases is not the default behavior, but when the user requests it, here's how they are stored in memory.

The passphrases appear on a page mostly full of zeros. There's no pattern to the offset inside of a memory page, but there are some zeros usually several dozens, a 32-bit representation of the passphrase length, the passphrase (in ASCII), and more zeros.

Although my cryptoscan plugin for the Volatility framework has proven to be ineffective against modern versions of TrueCrypt, it could easily be modified to find these cached passphrases. (Sorry I can't be more helpful and do the work myself, but this would make a great introduction to the Volatility architecture.) Enjoy!

Last Call for Volatility Bugs Before Release

2009-07-30T10:43:35Z

The Volatility Framework is looking to wrap up our month-long call for bugs before the next official release. The framework is a great way to experiment with memory forensics on Windows XP systems and I recommend it for anybody interested in this new field. You can grab the last official release, Version 1.3 Beta 1 from the web site or the latest code (patched yesterday!) directly from http://code.google.com/p/volatility/ before the final release next week.

If you find something that should be fixed, please either leave a comment here or write to the developers mailing list. We appreciate the feedback and are looking forward to big release Real Soon Now^TM.

Fuzzy Hashing version 2.2

2009-07-22T01:58:12Z

Good news everybody! I've published a new version of the ssdeep program for fuzzy hashing. The new version adds a long-requested feature: the capability to compare files of previously generated signatures. That is, let's say you compute some lists of fuzzy hashes like this:

C:\> ssdeep -r C: > list1.txt C:\> ssdeep -r D:\Malware Samples > list2.txt C:\> ssdeep -r E:\Temp\New Malware > list3.txt

You can now find any similar files in those two lists like this:

C:\> ssdeep -x list1.txt list2.txt list3.txt list1:C:\Windows\System32\ntoskrn1.exe matches list2:D:\Malware Samples\VIRUS.EXE (83)

Notice the filename of the known hashes is given in the output along with the matching filenames.

Volatility Call for Bugs

2009-07-05T16:38:05Z

We are aiming to release a new version of the Volatility Framework for memory analysis at the start of August 2009. Although no software is ever perfect, we'd like to publish the best possible code. As such, we're asking everybody to please test out the current version and tell us about any bugs they find.

The code is available on the Volatility homepage on Google Code. As noted on their site, you can download the latest version using Subversion as follows:

svn checkout http://volatility.googlecode.com/svn/trunk/ volatility-read-only

You can download some sample memory images from NIST or hogfly's memory exemplar project, but please feel free to test the framework with your own memory images. Lots of testing has been done with the above images, so the best way to find new bugs is with new memory images! Remember the framework only supports Windows XP Service Pack 2 and 3.

When you find a bug, you can post a comment here, visit the developers on IRC at #Volatility on freenode.net, or write to the developer's mailing list.

Even better than finding a bug is fixing one! By all means please take a stab at fixing whatever problems you find. The best way to submit patches is to make changes to the checked out code and generate a patch file. You can create a patch file like this:

svn diff > mypatches

Here's an example patch to let Volatility run on big-endian systems. Good hunting!

DoD Cyber Crime deadline extended

2009-06-29T21:45:15Z

It's beginning to become standard operating procedure, but the deadline for submissions for the 2010 DoD Cyber Crime Conference has been extended until 1 August.

As you can see from the 2009 conference and 2008 conference, the deadline has been extended at least once each year. While I appreciate the difficulty of vetting and scheduling so many talks, maybe they could just make the initial deadline a little later? Submitting a talk six months in advance makes it extremely difficult to present something truly cutting edge.

Windows Internals 5th Edition published

2009-06-29T13:23:01Z

Windows Internals 5th edition has been published! I first wrote about this book in January 2008, so it's only 18 months behind schedule. The book is the latest edition of the fantastic Microsoft Windows Internals books by Mark Russinovich and David Solomon. If you want to know what's going on under the hood with Windows to a ridiculous level, read this book!

Looking back on sample memory images

2009-06-12T11:09:15Z

Four years after posting sample memory images I am proud of my decision, glad to have given researchers a common data set to work with, and heartened by the community's respect for my privacy.

Back in the summer of 2005 the only publicly available memory images were the two from the DFRWS Memory Analysis Challenge. Partially inspired by the challenge, I decided to publish a set of standard memory images for researchers. The idea was to create a common data set we could all use. When somebody developed a new technique, they could explain the method and show the results of using it on the common data set. Other researchers could repeat the method on the same data and then critique the method.

I posted a set of five memory images. Three of them were from a test system and contained no real data. They were just a clean operating system install booted up. The last two of the images, however, xp-laptop-2005-06-25 and xp-laptop-2005-07-04, were memory images from a computer that had been used in the real world. The system, a laptop borrowed from a friend, was being used by S— for schoolwork.

At the time I thought about the risks of posting data from a machine S— and my friend had been using, but my conclusion was "I've rebooted the machine and only my data should be in RAM."

Oh, how wrong I was.

There has been an astounding amount of memory analysis research since that summer. We've developed methods to recover amazing things. Network packets, connection information, hidden processes, handle tables, hooked tables, entire executables, malicious code samples, and more. We should be especially thankful to moyix for his work on examining the Windows Registry in memory images. The registry contains a gold mine of forensic information such as password hashes, SSIDs from wireless networks and information about USB devices.

I'm singling out moyix for an important reason. When he first discovered the easily crackable password hashes in my memory images, he contacted me before publishing anything. He gave me a chance to protect myself. I appreciated the heads up, and while I would never ask anybody to withold publication, his pause helped keep me and S— safe.

Interesting side note about SSIDs and how forensic analysis has its limits. Cached SSIDs are stored in the registry hive labeled \WINDOWS\system32\config\software. For example, in the xp-laptop-2005-07-04 memory image it's at offset 0xe1658b60. In that hive, look at the key Microsoft\WZCSVC\Parameters\Interfaces. You'll see some entries that are GUIDs. Looking at the values under those GUIDs reveals some binary data that contains the SSIDs.

In those SSID entries you'll find an entry "Mayorga". A Bing search¹ reveals the Mayorga Coffee Factory . Although not direct evidence that I was at the Mayorga, it's definitely circumstantial. As a point of fact, however, I never went to the Mayorga coffee house before July 2005. The SSID could either be a remnant of the machine's owner or a red herring. There's nothing that prevents any access point from being named "Mayorga".

Posting my memory images unquestionably helped other researchers advance the field. The images gave them live data to experiment with and a method for others to verify their results. My heart still skips a beat every time I hear somebody has found a new kind of data in those memory images, but I'm confident the next person to make a major advance will talk to me before publishing anything.

In a similar vein, I urge others to post live data sets for research, but please be careful. Once you post something to the Internet you can never ever ever make it go away. There are probably data in my memory images we can't even conceive of yet, and analysis techniques always improve.

If you haven't already, check out the memory snapshot project (files). Hogfly is posting memory images from systems infected with live malware. While I applaud Hogfly's efforts, I don't think the systems are running with live data. That is, the malware is the only thing running on the system, making it easy to see. In an ideal world I'd like to see memory images from a system with real-world data that has malware on it. That would make for good research and training!

At this point, however, things get more complicated. Nobody should post memory images with data from people who haven't consented. But how much consent is needed? Is data posted publicly on the web free to use? My memory images contained pages from the New York Times, which is probably fine. But what about something posted to LiveJournal or Facebook? Email messages? Mailing list posts? Whatever you do, please be responsible.

1. Yes, a Bing search. A researcher always tries out new things.

md5deep version 3.4

2009-06-10T21:42:32Z

I've released version 3.4 of md5deep. This is a bug-fix release and addresses two serious problems. First, there was a memory leak while processing directories on Windows. Second, the -n mode, or Unused hashes mode has been fixed. My apologies for the errors.

Conference Updates

2009-06-03T16:58:36Z

The Call for Papers is open for the 2010 DoD Cyber Crime Conference. Don't be late; abstracts are due on July 1st!

Also, don't forget about the SANS Computer Forensics Summit in early July. Yours truly will be on the Essential Forensic Tools. See you there!

Essential Tools Panel at SANS Forensic Summit

2009-04-20T17:43:29Z

I've been selected to present as part of the Essential Tools Panel of the SANS Forensics Summit 2009. I'll be presenting on which tools I use regularly and why. With me on the panel are Troy Larson from Microsoft, Mark McKinnon, and Lance Mueller. It should be a great talk. Be there or be square!

md5deep version 3.3

2009-04-04T13:57:05Z

This morning I posted md5deep version 3.3. This is a bug-fix release intended to address two issues on Microsoft Windows. First, the program can now handle 64-bit timestamps, which previously could have caused a crashed. Second, the program now skips all reparse points (e.g. junction points, symbolic links, etc). There come up often on Windows Vista and Windows 7 and can cause a lot of extra work for the program. The resulting code is not perfect, someday the user should be able to control the recursion process, but it's better this way than before. Enjoy!

Fuzzy Hashing in FTK

2009-03-27T02:47:08Z

So apparently I've been asleep at the switch. Fuzzy hashing has been incorporated into AccessData's flagship Forensic Toolkit! Not only have they added the feature but they've also written a great paper describing fuzzy hashing and how it works in FTK.

Now I know what some of you are thinking. How did AccessData include fuzzy hashing, which is licensed under the GPL2, in a proprietary program like FTK? Well, to tell you the truth, in all this excitement I kind of lost track myself. (Wait... wrong speech.)

I think AccessData rewrote fuzzy hashing. The edit distance code, for example, has been replaced with some database calls. I don't know how they're computing the rolling and FNV hashes, but if they took the time to rewrite the edit distance code they probably rewrote the rest too. The edit distance code dates back to 1989 and was last updated in 1993. There's no sense in rewriting something that's been working for fifteen years unless you absolutely must.

Regardless, go forth and be fuzzy!

Suspicious Plugin Broken

2009-03-03T14:54:06Z

Several people have written to me asking about the Suspicious plugin I posted a while back. Unfortunately I'm not sure what's going wrong. There is some kind of conflict with the existing plugins, but I haven't been able to figure it out. Sorry!

Fixing the 'suspicious.py' Volatility module

2009-02-14T16:06:57Z

Several people have asked about the following error when using the suspicious.py plugin for Volatility:

Traceback (most recent call last):
  File "volatility", line 219, in 
    main()
  File "volatility", line 201, in main
    MemoryRegistry.Init()
  File "/Users/jessek/Volatility-1.3_Beta/forensics/registry.py", line 269, in Init
    OBJECT_CLASSES = VolatilityObjectRegistry(object2.Object)
  File "/Users/jessek/Volatility-1.3_Beta/forensics/registry.py", line 244, in __init__
    raise Exception("Object %s has already been defined by %s" % (obj,self.objects[obj]))
Exception: Object _EPROCESS has already been defined by class 'example3._eprocess'="'example3._EPROCESS'">

The error is caused by a collision of the _EPROCESS object in suspicious.py and example3.py. The easiest way to solve the problem is to remove example3.py from the memory_plugins directory.

Revised journal articles on BitLocker, Buffalo

2009-02-05T05:10:13Z

Good news everyone! Thanks to a revised legal agreement from the journal Digital Investigation I have been able to publish the edited versions of my papers Implementing BitLocker Drive Encryption for Forensic Analysis and Using Every Part of the Buffalo in Windows Memory Analysis. Although there isn't much new content in the latter, the former was almost entirely rewritten between its original submission and the present form. Enjoy!

Slides from DoD Cyber Crime 2009

2009-01-31T14:13:05Z

As promised, a version of the slides from my talk, Practical Methods for Dealing with Full Disk Encryption at the 2009 DoD Cyber Crime Conference is now online. I've redacted the law enforcement sensitive material. Enjoy!

md5deep version 3.2 with CSV mode

2009-01-25T16:15:07Z

This morning I've publishing version 3.2 of the md5deep/hashdeep suite. Although there are several important bug fixes in this release, I've also added a new feature, comma separated values mode (CSV) for md5deep. This feature has been requested every so often and I've finally added it. You can see the complete list of changes before downloading the Windows version or source code.

List of Volatility plugins

2009-01-15T23:24:08Z

I'm creating a list of the plugins for the Volatility memory analysis framework over on the Forensics Wiki. If you've written a plugin, please add it to the list!

Using Fuzzy Hashing from C#

2009-01-14T17:33:52Z

Recently ssdeep user Jose Cintron wanted to use fuzzy hashing in a C# program. After some significant head scratching he came up with the following code necessary to use at least part of the DLL from his program:

using System.Runtime.InteropServices;	// Needed for DllImport

public class ssdeepWrapper
{
     // fuzzy.dll should be somewhere in the path or specify exactly
     // where to find it.
     [DllImport("fuzzy.dll")]
         public static extern int fuzzy_hash_filename(string fname, 
StringBuilder result);
     [DllImport("fuzzy.dll")]
         public static extern int fuzzy_compare(string sig1, string sig2);
}

Any thoughts? Has anybody else tried using fuzzy hashing from a different language?