Tuesday, 19 May 2015 00:00

At home among strangers: recovering data from FAT, NTFS, UFS without quitting Linux

Rate this item
(0 votes)

Every other man and his dog wrote about recovering data from a file system. There is a wide variety of means to perform this task, including debugs utility, which can easily retrieve any erased files from ext2. But what about other file systems? How can we recover a lost file from a flash drive or adjacent NTFS-partition? Even the most hard-working bloggers are silent about it. And yet, it's very simple and prosy.

It is not always convenient to reboot the operating system to another to perform checking file systems, file recovery, partition resizing and other actions on data. Imagine that two operating systems have been installed on your computer for several years: Windows and Linux. You boot very rarely the first one and only in cases of emergency; you use the second one daily and have been thinking about going completely over to Linux and uninstalling Windows, the only thing is that you can't convert NTFS-partition, which holds years of accumulated data, to ext3 by any tool. You have to keep the two OSs, because even though NTFS-section is accessible from Linux (using ntfs-3g), you still have to reboot to Windows to solve file system problems.

And what if the FAT file system has fried on your Flash-drive? Rebooting to Windows again? Or you accidentally deleted a file in the UFS file system pertinent to installed adjacently FreeBSD? Maybe you're a system administrator and don't have around a Windows recovery disk at the most seasonable time? I will give answers to all these questions at once: you can perform almost all actions on recovering FAT, NTFS, UFS file systems, restoring files stored in them, diagnostics, and lots more without quitting Linux. You'll learn how to do it from this article.

Tool kit

Before I directly describe the processes of recovery, diagnostics and reanimating dead files, it is my duty to acquaint you with a list of tools used. First, we need tools to work with file systems (creating, testing, obtaining information). All of them are presented in three packages:

  1. dosfstools - utilities for working with file systems such as FAT. The package contains only two utilities: mkfs.vfat (mkfs.dos) for creating a file system and fsck.vfat (fsck.dos) for performing a file system check.
  2. ufsutils - a set of tools to work with UFS and its derivatives (e.g, FFS, used in FreeBSD). Contains eight tools including mkfs.ufs, fsck.ufs, tunefs.ufs (FS setting), growfs.ufs (resizing), and others.
  3. ntfsprogs - various tools for working with NTFS. Does not contain a tool for creating or complete checking (basic check is available) a file system, but includes a lot of useful tools, such as ntfscp for copying files without setting the partition, ntfsundelete "file reanimating" tool, ntfsresize resizing utility, ntfsclone for partition cloning, and others.

Also, we may need tools for working with hard disk partitions. There are three of the most advanced utilities of this type: parted, developed for creating partitions, resizing, moving, creating and checking file systems; gpart - utility-restorer of zapped partition table and TestDisk - analog of gpart with pseudo-graphical interface and some useful features.

It should be noted that parted is just a nice wrapper over the described tools for working with file systems, but they too handle almost everything that parted does. And over the very parted there is another wrapper, called gparted. It only creates a user-friendly graphical GTK interface styled after Partition Magic.

In the TestDisk package you will find a utility PhotoRec, designed to recover different file types on a partition regardless of the used file system. It's basic idea is to find and recover files based on their metadata without analyzing the structure of the file system. PhotoRec can recover images (bmp, jpg, png, tiff, raf, raw, rdc, x3f, crw, ctg, orf, mrw), audio files (wav, au, mp3, wma), video files (avi, mov , mpg), archives (bz2, tar, zip), documents (doc, pdf, html, rtf), files with utility sources (c, pl, sh). A number of utilities of this type can be found in the Sleuth Kit package, for which there is autopsy web-interface.

Use cases

In the following sections, we will consider several common use cases of the described utilities. Firstly, it is a detailed description of< file recovery using three different approaches; secondly, repairing file system after crash; thirdly, partition cloning for several machines, and fourthly, the description of the data migration to a smaller partition.

Casting ressurection

The above-mentioned utility ntfsundelete from the ntfsprogs package is designed to reanimate dead files on NTFS. It is very easy to use and extremely flavorful. If you accidentally zapped a file and immediately set the partition, be sure - ntfsundelete will be able to restore it in one piece.

First we need to look the list all deleted files through:

# ntfsundelete /dev/sda1

The third column will display the percentage of the file integrity. If it equals to 100% - everything is OK, the file can be reanimated in one piece. A smaller value indicates that some portions of the file have already been zapped by new data, so after recover the file is, as they say, dead. In some cases, the possibility of recovering even a half-dead file can make difference, but for now we will focus on fully intact instances. To do this, execute the following command:

# ntfsundelete -p 100 /dev/sda1

Wow, there are a lot of them indeed! Let's make the utility display only the files deleted in the past 2 days:

# ntfsundelete /dev/sda1 -p 100 -t 2d

This is more like it. Let's recover the file, whose inode number (the first column) is 11,172, to the directory /undeleted:

# ntfsundelete /dev/sda1 -u -i 11172 -d /undeleted

Files can be recovered based on a mask:

# ntfsundelete /dev/sda1 -u -m "*.doc"

They can be filtered by length:

# ntfsundelete /dev/hda1 -S 5k-6m

Or, we can recover all deleted files, and then find our bearings.

# ntfsundelete /dev/sda1 -u -m "*" -d /undeleted

The utility recovers files with all the attributes, including the name and creation time. This software is a joy to use.

To recover data from all other file systems, including FAT, UFS, EXT3 and any other indeed, it's best to use PhotoRec. Run the program:

# photorec

Select a test device in the main menu (e.g, /dev/sda). Click <Enter> and select the partition table type (for PCs it is Intel). Next, select the partition, and then on the next screen - the type of the file system (ext2/ext3 or other). Specify the directory where you want to allocate the recovered files, and press "Y". The directory must be on a different partition/disk, otherwise you risk twisting the knife by zapping deleted files with new data.

That's all, the recovery process started. It may take from 10 minutes to several hours, depending on "oldness" of the file system and the number of deleted files. You can stop the process at any time by pressing <Ctrl-C>, and resume it from the point of interruption, re-launching PhotoRec.

In the directory you selected, you will find a lot of subdirectories named like recup_dir.1, recup_dir.2, each of which contains a large number of files of different types. PhotoRec does not restore names, so you will have spend certain time to sort out a mess.

PhotoRec has other drawbacks, too:

  1. It glitches quite often, and files can be corrupted, so they should necessarily be checked for "not being dead".
  2. The program looks for files by patterns. If you deleted a file whose format is not supported by PhotoRec - the game is over.

Therefore, you should have other tools of analysis and lost data recover along with photorec. The best utility set in this field is Sleuth Kit, containing a wide variety of tools which various hacking investigation authorities and advanced system administrators like to employ in their work. We are miles away from that, and we are interested in only two of the entire set of utilities: fls and icat, designed to search and retrieve files (both existing and deleted).

Let's see a list of deleted files using a utility fls:

# fls -rd /dev/sdb1
r/r * 117: dsc0005.jpg
r/r * 119: dsc0006.jpg
r/r * 122: dsc0007.jpg
r/r * 125: dsc0008.jpg
r/r * 128: dsc0009.jpg

The '-r' indicator makes the tool recursively go over all directories, and '-d' - display only deleted files.

Most likely, the list will be very long, and also it will include the inode list, which has already been given to other files (realloc line in the third column), so we will filter it out and direct to less:

# fls -rd /dev/sda1 | grep -v
'(realloc)' | less

In the third column you will see the inode-file numbers, and in the fourth - their names. To retrieve the file from the file system, use the icat command ('-r' indicator is used to recover a deleted file):

# icat -r /dev/sda1 1023 >
/home/vasya/tmp/my_file

To recover all the files, you can use the following command:

# for i in `fls -rd /dev/sda1 |
grep -v '(realloc)' |\
awk {'print $3'}|tr -d [:]`; do icat -r -f fat /dev/sdb1 $i >\
/home/vasya/tmp/inode-$i ;done

If you want to find a concrete file, you can
simple grep fls:

# fls -rd /dev/sda1 | grep -v
'(realloc)' | grep my_file.jpg

A remarkable feature of Sleuth Kit tools is that they use lots of different techniques for searching deleted files and their parts. They include analysis of the file system control structures, various heuristic approaches and pattern matching. In fact, using Sleuth Kit you can reanimate even files zapped in ext3 (though developers of ext3 claim that it is impossible).

File system repair

It is very simple to repair a broken file system. It is enough to use the standard tools of fsck.vfat (for FAT12, FAT16 and FAT32), fsck.ufs (for UFS, UFS2, FFS) and ntfsfix (for NTFS).

Unfortunately, ntfsfix can not fix NTFS up completely. It only fixes some of its problems and sets an indicator of the forced file system check, so that the next Windows reboot will entail running chkdsk for a full FS test.

Using a virtual machine, we can avoid the need to reboot to Windows. To do this:

  1. Launch the virtual machine and install Windows to the virtual hard disk.
  2. Set the partition containing NTFS file system.
  3. Start the virtual machine, specify the virtual disk with Windows as primary drive, and our real hard drive as secondary.
  4. Using the Windows standard features, run the check of NTFS-partition.

Partitions copying

Let's say you bought a new hard drive and now you want to move a few partitions from the old drive to the new one. If you start to do this by standard methods, by creating a new partition and copying files manually, you will risk having a lot of problems associated with the encoding of file names, special files, protected files, and also you will lose a lot of time. It is better to use the partition cloning method.

UNIX-users clone partitions using standard dd utility, which can be used in conjunction with any file system. To do this, create a partition on a new disk that is equal-sized to the source, and execute the "dd if=partition1 of=partition2 bs=1m" command. Similarly, you can copy NTFS-partition too, but there is a more suitable tool in the ntfsprogs package for this purpose .

The ntfsclone utility is identical to dd command except for two peculiarities. Firstly, it does not copy unoccupied ares of a file system, that is why the moving is faster, and the partition image (if you are creating an image) requires less space. Secondly, ntfsclone is able to store the image in a special compressed file, which is convenient to transfer to other machines.

To clone a partition it is enough to execute the following command:

# ntfsclone --overwrite /dev/hda1 /dev/hdb1

And to create the image:

# ntfsclone --save-image --output backup.img /dev/hda1

The ntfsclone utility is especially useful if you decide to copy the installed Windows on a fleet of other machines (in a classroom or office). It's enough to install Windows on a single machine and create the image that can then be put in the file-sharing and uploaded to other machines using Linux LiveCD. You will have to copy the MBR-record of the disk so that they can be booted:

# sfdisk -d /dev/sda > /share/sda-sfdisk.dump

# dd if=/dev/sda bs=512 count=1 of=/share/sda-mbr.dump

And then write it to the disc of each machine:

# sfdisk /dev/sda < /share/sda-sfdisk.dump

# dd if=/share/sda-mbr.dump of=/dev/sda

Transferring Data

What if you decided to go completely over to Linux, but do not want to use a variety of tricks and ntfs-3g to get access to your old data located on NTFS-partition? After all, this section can occupy a large part of the disk, and there is no way to simply copy its contents to the new partition, formatted to ext3/ext4. In this case, you will get help from utilities from the ntfsprogs package, or rather from one of them - ntfsresize, which will allow copying the data in small chunks to the new file system with subsequent reduction in size of NTFS-partition and increase of ext3/ext4-partition. For this you will need any LiveCD with ntfsprogs and e2fsprogs 1.41 or later version (for supporting ext4, that is, if you are going to transfer data on it). It is also most welcome that LiveCD has the latest gparted, because it is difficult and dangerous to resize manually. Besides resizing the file system itself, you will have to resize a partition using fdisk; if you make a single mistake, you will have to try everything back.

So, let's boot LiveCD and set hard disk partitions. Suppose its size is 120 GB. Of these 120 GB, 80 GB are packed to capacity NTFS-partition, and the remaining 30 GB (yes, just 30; because after converting marketing gigabytes into the actual ones, the disk space equals to about 111 GB) - the partition with installed Linux, which occupies 5 GB. Hence, our free space is about 25 GB. Let us move files from NTFS-partition to ext3/ext4-partition until their total size becomes equal to the size of free space. As a result, the free space becomes completely used, and NTFS "reduces" by 25 GB. Set both partitions and execute gparted. Select the NTFS-partition, click the second mouse button, select Resize/Move and decrease the partition size by the free space size, select ext3/ext4-partition and increase it by the same free space size (you will have to move the partition to the beginning of the disk, and then enlarge). In this way we get extra 25 GB of free space which allows us to copy some files, and then again fulfill resizing. Having done four of such passes, we completely remove the NTFS-partition and extend ext3/ext4-partition to the entire disk.

Conclusions

As you could see, Linux is not just able to manipulate many third-party file systems, but also has a variety of tools to modify them, conduct diagnostics and perform other actions. You'll never achieve stalemate, having a Linux-based LiveCD based, which just is the Holy Grail of any system administrator and user.

WWW

foremost.sourceforge.net - Foremost, another popular tool for recovering files from patterns.

www.sysresccd.org - System Rescue CD contains all the utilities mentioned in the article.

If you need to recover your data, contact one of Data Retrieval offices in Texas: Houston, Austin, Dallas or San Antonio

Last modified on Tuesday, 19 May 2015 21:48
Data Recovery Expert

Viktor S., Ph.D. (Electrical/Computer Engineering), was hired by DataRecoup, the international data recovery corporation, in 2012. Promoted to Engineering Senior Manager in 2010 and then to his current position, as C.I.O. of DataRecoup, in 2014. Responsible for the management of critical, high-priority RAID data recovery cases and the application of his expert, comprehensive knowledge in database data retrieval. He is also responsible for planning and implementing SEO/SEM and other internet-based marketing strategies. Currently, Viktor S., Ph.D., is focusing on the further development and expansion of DataRecoup’s major internet marketing campaign for their already successful proprietary software application “Data Recovery for Windows” (an application which he developed).

Leave a comment

Make sure you enter the (*) required information where indicated. HTML code is not allowed.