Recovering lost files

Using Snapshots

All of the file servers in Nikola take snapshots of important filesystems every day. These snapshots are a exact image of the filesystem in question at a specific time. We have two forms of these snapshot systems: zfs and wafl. The melodi and mtml labs only support zfs snapshots, while tial supports wafl and zfs.

In order to recover lost or damaged files, you must simply cd into the snapshot directory of the filesystem, and then into the directory that will most likely have your file before the damage occured. Then simply copy whichever files you need out of that directory and you're done.

ZFS Snapshots

If you are in the labs melodi or mtml, the snapshot system you will be using is ZFS. In order to access the snapshot directory you will need to cd into .zfs/snapshot in the filesystem where your files are located. NOTE: the directory .zfs is a truely hidden directory, so it won't show up even if you ls -a, so the only way to check for its existence is to cd into it. For example, if you have a file in your home directory you must recover, you would execute the command:

cd ~/.zfs/snapshot

Once you are in your snapshot directory, executing ls will present you with a number of directories in the form xdaysago with x being an integer from 0-6. Using the example from above, if you cd into the directory 3daysago, you will find an exact replica of your home directory as it was exactly 3 days ago. You may then copy whatever files you need out of the snapshot directory and you have successfully recovered your files. ZFS does not support hourly snapshots.

WAFL Snapshots

If you are in the lab tial, the snapshot system you will likely be using is WAFL, though you may come across ZFS so don't hesitate to check for both of them. For WAFL, the snapshot directory you must cd into is simply .snapshot. You can check for the existence of .snapshot by executing ls -a, for it is not truely hidden like ZFS. For example, if you need to recover a file from your home directory, you would execute the command:

cd ~/.snapshot

Once you are in your snapshot directory, executing ls will present you with a number of directories in the form hourly.x, nightly.x, and weekly.x, with x being an integer. There are 6 hourly snapshots kept, 7 nightly snapshots for each day of the week, and 2 weekly snapshots. Using the example from above, if you cd into the hourly.3 directory, you will find a exact replica of your home directory from 3 hours ago, or if you cd into the nightly.4 directory, you will find an exact replica of your home directory from 4 days ago. If the integer is '0', it represents this hour, today, or this week. Once you have found the files you need, you simply need to copy them out of the snapshot directory and you have successfully recovered them.

Request a restore

It is important to note the backup schedule for your lab. If you delete a file the same day you created it we won't have had a chance to back it up yet.

Send your restore request with the following information to your lab's help queue, likely melodi-bugs@nikola-rt.ee.washington.edu or tial-bugs@nikola-rt.ee.washington.edu

1. The name of the file or directory to be restored.
2. The full path to the file or directory. ex. /g/ssli/projects/madeup/fred/ginger
3. The date from which the file can be recovered. ex. "It was good last Friday the 23rd of June."
4. The oldest possible date the file existed (if known).
5. Where you want us to put the restored file/directory. e.g. "restore it to the place where it used to be" or "restore it to my transitory directory" or "restore it to my home directory."

Allow 2-3 working days for restores. Small restores are usually handled quickly (unless the tape drives are busy) but large restores can take multiple days - especially if we have to pull them from an offsite backup.