How to recover archived files from Glacier

Parice Brandies -

So you enabled archiving on your Object Storage bucket in RONIN for secure and affordable long-term storage of your files, but now you need one (or all) of those files back again.

Well, never fear, the process to retrieve these files is pretty quick and simple once you know how, and this blog post will teach you just that!

How do I restore an archived file from Glacier?

If you only want to restore a few small files, you can easily restore files from Glacier using Cyberduck. If you want to learn how to connect to your object storage bucket in Cyberduck, check out this tutorial.

Otherwise, if you need to restore a large amount of files, want to customise the restore options (such as number of days you want the file restored for), or just prefer using the command-line, below we will show you all the important AWS CLI commands you need to know.

Firstly, you will want to make sure the AWS CLI tools are installed and then configure the AWS CLI with the key to the bucket where your archived files are sitting - see this blog post for a tutorial.

Once the AWS CLI has been installed and configured with your bucket key, you can list all of the files you have in Glacier Flexible Retrieval (formerly just Glacier) by running the following command:

 aws s3api list-objects --bucket mybucket.store.ronin.cloud --query 'Contents[?StorageClass==`GLACIER`][Key]' --output text
Note: For the purposes of this tutorial, we will assume your bucket is called mybucket.store.ronin.cloud - make sure you replace this with your specified bucket name instead.

If you selected the Glacier Instant Retrieval, or Glacier Deep Archive options when enabling archiving on your object storage bucket in RONIN, replace GLACIER in the command above with GLACIER_IR or DEEP_ARCHIVE respectively. You can also list non-archived files by replacing GLACIER with STANDARD.

To restore a particular file from your bucket run the following command:

aws s3api restore-object --bucket mybucket.store.ronin.cloud --key dir1/example.txt --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Bulk"}}'

Now, let's explain this command:

  • aws s3api restore-object - the AWS CLI command for restoring an object from Glacier
  • --bucket mybucket.store.ronin.cloud - the bucket you are trying to restore a file from
  • --key dir1/example.txt - the path to the file you wish to restore (in this example it is a file called example.txt that is within a folder in the bucket called dir1)
  • --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Bulk"}}' - the restore request details where Days is how many days the object should be restored back to standard storage for (in this case 7 days) and Tier is the retrieval method. In this example we have used the Bulk retrieval method, as it is free.

There are a number of retrieval methods available depending on what Glacier storage class your file is stored in:

  • Expedited - 1-5 minutes, ~$0.03 per GB from Glacier Flexible Retrieval only (costs are region dependent)
  • Standard - 3-5 hours, ~$0.01 per GB from Glacier Flexible Retrieval and ~$0.02 from Glacier Deep Archive (costs are region dependent)
  • Bulk - 5-12 hours, FREE from Glacier Flexible Retrieval and ~$0.0025 from Glacier Deep Archive (costs are region dependent)
Note: Glacier Instant Retrieval files will always be restored in milliseconds, so you can omit the "GlacierJobParameters":{"Tier":"Bulk"} section from your command. The charge for restoring objects from Glacier Instant Retrieval is $0.03 per GB.

Once you have initiated a restore request for a file, you can check its progress by running the following command:

aws s3api head-object --bucket mybucket.store.ronin.cloud --key dir1/example.txt

The output of this command will tell you whether the restore request is still being processed, and once it is processed it will tell you the “expiry date” when the object will be archived back to Glacier again.

What if I want to restore a file to standard storage permanently?

To restore a file back to standard storage permanently, once the file has been restored from Glacier using the command above, you need to run an additional command which essentially uses a copy command to overwrite the file back to standard storage indefinitely:

aws s3 cp s3://mybucket.store.ronin.cloud/dir1/example.txt s3://mybucket.store.ronin.cloud/dir1/example.txt --storage-class STANDARD
Note: Make sure archiving is also turned off on your bucket in RONIN if you don't want these files to be archived again.

What if I want to restore all my files back to standard storage?

Unfortunately there is no one single command that will restore all of your files in an Object Storage bucket from Glacier back to standard storage. However, you can combine the first two commands in this blog post to automate the process for you:

aws s3api list-objects --bucket mybucket.store.ronin.cloud --query "Contents[?StorageClass=='GLACIER'][Key]" --output text | xargs -I {} sh -c "aws s3api restore-object --bucket mybackup.store.ronin.cloud --key \"{}\" --restore-request Days=7,GlacierJobParameters={Tier=Bulk} || true"

This command will first list all of the files you have in the GLACIER storage class in your bucket and then pass these files one by one to the aws s3api restore-object command via xargs.

If you have file names that need to be escaped (such as single quotes), use sed to clean them up as they are parsed in:

aws s3api list-objects --bucket mybucket.store.ronin.cloud --query "Contents[?StorageClass=='GLACIER'][Key]" --output text | sed -e 's/"/"\\""/g' -e 's/.*/"&"/' | xargs -I {} sh -c "aws s3api restore-object --bucket mybucket.store.ronin.cloud --key \"{}\" --restore-request Days=7,GlacierJobParameters={Tier=Bulk} || true"

If you still get a "Broken Pipe" run xargs with the -p option to go through line by line and find the file that isn't formatting correctly :)

Alternatively, you could provide xargs with your own list of files.

Tip: If you are running the command on a machine with multiple CPUs, you can run xargs with the -P option to run the command in multi-threaded mode and speed things up a bit! For example, if you have 8 CPUs, you can restore 8 files simultaneously by adding -P8 to your xargs command.

Now go and treat yourself to a coffee while you wait for those archived files to defrost!