>> ext3grep

In English

Recover deleted files from an ext3 partition

How I deleted part of my thesis work:

  $ rm -rf ps3/

and how ext3grep saved my bum :)

Background:

I had just adjusted the backup scripts, so I thought: "OK, I am safe, the script replicates the data on another machine and also copy a tarball on the university machine; I can _clean_up_ this old location..."

...wonder what?

Stupid me, I forgot to RUN the backup script!
And the last backup was two months old.

Time of the disaster:
After June 25 2008 19:00
Before June 25 2008 20:30

I promptly umounted the partition after a couple of errors, because I was messing with "mount --bind" on the precious data directory: I also had the data exported on a nfs share and I wanted to show an "alias" to the clients.
(Yes, I was playing fool with precious data...)

Let's see what I can recover.

  $ export IMAGE=/dev/sda7
  $ ./ext3grep $IMAGE --dump-names

It shows files which I did NOT delete, they were accessible before I umounted the partition.
Hmm, the files I am searching for do not appear at all, not even as DELETED, argh!
And even if the ps3/ dir is there, it is empty when I "--ls --inode" it.
Was it because of the "mount --bind" mess?

Let's search for a directory name which we know was in ps3/

  $ ./ext3grep $IMAGE --search 00_hello_spe
  Running ext3grep version 0.7.0
  Number of groups: 820
  Minimum / maximum journal block: 1545 / 35886
  Loading journal descriptors... sorting... done
  Journal transaction 408734 wraps around, some data blocks might have been lost of this transaction.
  Number of descriptors in journal: 25996; min / max sequence numbers: 404956 / 450124
  Blocks containing "00_hello_spe": 11708416

Good, there's a block of HOPE :)

Show the content of the block:

  $ ./ext3grep $IMAGE --ls --block 11708416
  Running ext3grep version 0.7.0
  Number of groups: 820
  Minimum / maximum journal block: 1545 / 35886
  Loading journal descriptors... sorting... done
  Journal transaction 408734 wraps around, some data blocks might have been lost of this transaction.
  Number of descriptors in journal: 25996; min / max sequence numbers: 404956 / 450124
  Group: 357

  Block 11708416 is a directory. The block is Unallocated

            .-- File type in dir_entry (r=regular file, d=directory, l=symlink)
            |          .-- D: Deleted ; R: Reallocated
  Indx Next |  Inode   | Deletion time                        Mode        File name
  ==========+==========+----------------data-from-inode------+-----------+=========
     0    1 d 5849089  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  .
     1    2 d       2                                         drwxr-xr-x  ..
     2    3 d 5849091  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  01_hello_spe_argp
     3    4 d 5849090  D 1214417007 Wed Jun 25 20:03:27 2008  drwxr-xr-x  00_hello_spe
     4    5 d 5849100  D 1214417007 Wed Jun 25 20:03:27 2008  drwxr-xr-x  02_vsum1_spe
     5    6 d 5849206  D 1214417008 Wed Jun 25 20:03:28 2008  drwxr-xr-x  quad
     6    7 d 5849101  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  02_vsum1_ppe
     7    9 d 5849119  D 1214417008 Wed Jun 25 20:03:28 2008  drwxr-xr-x  03_mat_mat_spu
     8    9 d 5849175  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  single_spe_simd_dev_template
     9  end d 5849210  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  mat_mat_simd
    10  end d 5849184  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  single_spe_simd_matmat64
    11  end d 5849159  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  single_spe_simd_matmat_4x4
    12  end r 5849200  D 1214417009 Wed Jun 25 20:03:29 2008  rrw-r--r--  .single_spe_simd_matmat_4x4.swp
    13  end r 5849204  D 1214417008 Wed Jun 25 20:03:28 2008  rrw-r--r--  .single_spe_simd_matmat_4x4.swpx

Ok, this is all the stuff I have to recover.

Let's start form a simple one, important stuff is deeper in the dir tree, so a first level try will show if I can recover stuff at all:

     8    9 d 5849175  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  single_spe_simd_dev_template
  $ ./ext3grep $IMAGE --ls --inode 5849175
  Running ext3grep version 0.7.0
  Number of groups: 820
  Minimum / maximum journal block: 1545 / 35886
  Loading journal descriptors... sorting... done
  Journal transaction 408734 wraps around, some data blocks might have been lost of this transaction.
  Number of descriptors in journal: 25996; min / max sequence numbers: 404956 / 450124
  Inode is Unallocated
  Loading sda7.ext3grep.stage2............ done
  There is no directory block associated with inode 5849175.

Argh, no directory block associated with inode 5849175.
Let's see what the journal has to say:

  $ ./ext3grep $IMAGE --show-journal-inode 5849175
  Running ext3grep version 0.7.0
  Number of groups: 820
  Minimum / maximum journal block: 1545 / 35886
  Loading journal descriptors... sorting... done
  Journal transaction 408734 wraps around, some data blocks might have been lost of this transaction.
  Number of descriptors in journal: 25996; min / max sequence numbers: 404956 / 450124
  Copies of inode 5849175 found in the journal:

  --------------Inode 5849175-----------------------
  Generation Id: 2552992623
  uid / gid: 1000 / 1000
  mode: drwxr-xr-x
  size: 0
  num of links: 0
  sectors: 0 (--> 0 indirect blocks).

  Inode Times:
  Accessed:       1214417009 = Wed Jun 25 20:03:29 2008
  File Modified:  1214417009 = Wed Jun 25 20:03:29 2008
  Inode Modified: 1214417009 = Wed Jun 25 20:03:29 2008
  Deletion time:  1214417009 = Wed Jun 25 20:03:29 2008

  Direct Blocks:

  --------------Inode 5849175-----------------------
  Generation Id: 2552992623
  uid / gid: 1000 / 1000
  mode: drwxr-xr-x
  size: 4096
  num of links: 3
  sectors: 8 (--> 0 indirect blocks).

  Inode Times:
  Accessed:       1214415156 = Wed Jun 25 19:32:36 2008
  File Modified:  1210872172 = Thu May 15 19:22:52 2008
  Inode Modified: 1208277183 = Tue Apr 15 18:33:03 2008
  Deletion time:  0

  Direct Blocks: 11758520

Ok, so there is still a version of the inode with mode: d with a Direct Block defined, let's see if we can get anything out of it:

  $ ./ext3grep $IMAGE --ls --block 11758520
  Running ext3grep version 0.7.0
  Number of groups: 820
  Minimum / maximum journal block: 1545 / 35886
  Loading journal descriptors... sorting... done
  Journal transaction 408734 wraps around, some data blocks might have been lost of this transaction.
  Number of descriptors in journal: 25996; min / max sequence numbers: 404956 / 450124
  Group: 358

  Block 11758520 is a directory. The block is Unallocated

            .-- File type in dir_entry (r=regular file, d=directory, l=symlink)
            |          .-- D: Deleted ; R: Reallocated
  Indx Next |  Inode   | Deletion time                        Mode        File name
  ==========+==========+----------------data-from-inode------+-----------+=========
     0    1 d 5849175  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  .
     1    2 d 5849210  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  ..
     2    3 r 7749646  D 1214417009 Wed Jun 25 20:03:29 2008  rrw-r--r--  Makefile
     3    4 d 5849176  D 1214417009 Wed Jun 25 20:03:29 2008  drwxr-xr-x  spu
     4    5 r 5849182  D 1214417009 Wed Jun 25 20:03:29 2008  rrw-r--r--  template.c
     5    6 r 5849177  D 1214417009 Wed Jun 25 20:03:29 2008  rrw-r--r--  template.d
     6    8 r 5849183  D 1214417009 Wed Jun 25 20:03:29 2008  rrwxr-xr-x  template
     7    8 r 5849186  D 1214417009 Wed Jun 25 20:03:29 2008  rrw-r--r--  4913
     8   10 r 5849181  D 1214417009 Wed Jun 25 20:03:29 2008  rrw-r--r--  template.o
    10  end r 5849186  D 1214417009 Wed Jun 25 20:03:29 2008  rrw-r--r--  problem_data.h
    11  end r 5849187  D 1214417009 Wed Jun 25 20:03:29 2008  rrw-r--r--  problem_data.h~

Yay, we walked down the directory. Can we restore a file?

  $ ./ext3grep $IMAGE --restore-inode 7749646
  $  mv RESTORED_FILES/inode.7749646 Makefile 

Yatta! File restored!

And I repeated that manually for all the files I had to recover.
Well, I know that I could have automated the process with a wrapper script, but the important thing is that the files are now back, and replicated on multiple hosts as originally planned :)

Many many thanks to Carlo Wood and ext3grep: