Accessing your RONIN object storage with Rclone

Rclone is a command-line program that allows you to copy, sync, and move files between your local system to various cloud storage services, one of those being AWS object storage (S3). Although there are a number of other tools that can assist with the most basic use case of transferring files between a local machine and an AWS object storage bucket (like the AWS CLI), Rclone has the advantage of being able to be configured to connect to multiple cloud services. For example, when users also have files in Google Drive, Dropbox, One Drive, on-prem machines etc, and want to be able access and move files with a single application. This blog post will teach you how to install and configure Rclone to manage files in your AWS object storage bucket.

To learn how to configure Rclone to connect to other services, please refer to the documentation here.

Step 1. Installation

Install Rclone based on your current operating system following the Rclone installation documentation. Here we are following the steps for Linux OS.

sudo apt install unzip
curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64
sudo cp rclone /usr/bin/
sudo chown root:root /usr/bin/rclone
sudo chmod 755 /usr/bin/rclone
sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb

Test Rclone is installed successfully - you should see the manual information:

rclone	

Note: Do not run the rclone config command, we will be setting up the config manually in the next step to make things a bit quicker.

Step 2. Configuration

Open/create the Rclone config file:

nano /home/ubuntu/.config/rclone/rclone.conf

Add the following information to the config file that is required to be able to connect to your object storage bucket in RONIN, including:

• Type of storage = s3 (AWS term for object storage)
• Provider = AWS
• Access key ID = found in your read-write or read-only permission csv file
• Secret access key = found in your read-write or read-only permission csv file
• Region = region of your RONIN project - found on your project dashboard

[myobjectstore]
type = s3
provider = AWS
access_key_id = AKIAIOSFODNN7EXAMPLE
secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
region = ap-southeast-2

Note: You can replace "myobjectstore" with the name of your object storage bucket in RONIN so that it is easier to recognise.

Make sure you save the file before exiting with Ctrl+X, then Y

Step 3. Testing

Copy the path to your object storage bucket - this can be found by clicking the 3 dots in the top right hand corner of your object storage card in RONIN and then clicking "Connect Info":

Test that you can connect to your object storage bucket by running the ls command to list all files in your object store replacing myobjectstore with the name you inserted into the Rclone config file, and myobjectstorepath with your object store path copied from above:

rclone ls myobjectstore:myobjectstorepath

## e.g. rclone ls myobjectstore:myobjectstore.store.ronin.cloud

If you have any files in your object storage bucket, these should be listed in the output. Otherwise, if your object store is empty, as long as you don't see any errors, you should be ready to start using Rclone.

Rclone Command Examples

There are a number of ways you can transfer and manage files in your object storage bucket with Rclone. We have included some common examples below showing how to move files between your object storage bucket and a local filesystem, but please refer to the documentation for a more extensive list and more information about all the additional flags and options for each command.

• Create some test files and a test folder with a file in it to try out some of the commands below:

truncate -s 5K move.txt
truncate -s 5K copy.txt
mkdir folder
truncate -s 5K folder/file.txt

• Move a file called move.txt from your current folder to your object store:

rclone move move.txt myobjectstore:myobjectstore.store.ronin.cloud/move.txt --s3-no-check-bucket

The move.txt file should now exist in your bucket, but no longer on your machine.

Note: The --s3-no-check-bucket flag is required whenever you are uploading data to the object store. The --no-traverse flag may be useful in some situations too - more info here.

• Move a file called move.txt from your object store back to the current folder:

 rclone move myobjectstore:myobjectstore.store.ronin.cloud/move.txt .

The move.txt file should now exist on your machine, but no longer in your bucket.

• Copy a file called copy.txt from your current folder to your object store:

rclone copy copy.txt myobjectstore:myobjectstore.store.ronin.cloud/copy.txt --s3-no-check-bucket

The copy file should now exist on both your machine AND in your bucket.

• Copy a file called copy.txt from your object store to the current folder:

 rclone copy myobjectstore:myobjectstore.store.ronin.cloud/copy.txt .

This will just replace the existing "copy.txt" file on your machine with itself since no changes have been made to it. If changes had been made to the version in the bucket, these changes would be applied to the file on your local machine.  

• Sync a local folder called "folder" to your object store (i.e. make the folders identical, modifying the object store folder only):

rclone sync folder/ myobjectstore:myobjectstore.store.ronin.cloud/folder

This will upload the folder to your object storage bucket along with any files in it.

• Sync a folder in your object store to a local folder (i.e. make a local folder identical to a folder in your object store, modifying your current folder only):

rclone sync myobjectstore:myobjectstore.store.ronin.cloud/folder folder

Note: Syncing makes the source and destination folders IDENTICAL, so if there are any additional files in the destination folder that are not present in the source folder, they will be DELETED.

• Check if your local folder is identical to the folder in your object store:

rclone check folder myobjectstore:myobjectstore.store.ronin.cloud/folder

This is helpful to either check that your sync command worked, OR to run prior to your sync command to see what might change.

• Delete a file called copy.txt from your object store:

rclone delete myobjectstore:myobjectstore.store.ronin.cloud/copy.txt

• Delete a whole folder called "folder" from your object store:

rclone delete myobjectstore:myobjectstore.store.ronin.cloud/folder 

Note: This will delete any files in the folder too.

• Mount your object storage bucket to a new empty folder:

mkdir /mnt/myobjectstore/
rclone mount myobjectstore:myobjectstore.store.ronin.cloud /mnt/myobjectstore/ --daemon

This allows you to view your object storage bucket like a local filesystem folder. Any files that are added to this folder will automatically be synced to the object storage bucket. Any files that are deleted from this folder will automatically be deleted from the object storage bucket. To learn more about mounting S3 buckets see: https://blog.ronin.cloud/mountpoint/

Remember, while this blog article focuses on AWS object storage, Rclone can be used to transfer files to and from many different cloud storage providers making it easier for you to migrate and manage files across different platforms - see here for a list of all of the supported providers.