rclone

Rclone is a powerful copy/sync tool for cloud (at least 20 providers supported), as well as sftp backends. It also supports some special backends – most notably the crypt backend, allowing you to transparently and easily save data to an untrusted cloud. (The cryptography used is modern and quite trustworthy).

caveats first!

Let’s get the “caution” out of the way: one thing to watch out for is when file permissions are critically important – it does not preserve permissions for files unless you specify -M (or --metadata), and for directories even that flag is not honored. In general, rclone is NOT a suitable replacement for cp -a or rsync -a if mode bits are important. (There is a simple workaround for cases where you really need to use rclone; see this article).

A lesser point is that many people seem to think rclone is rsync for the cloud. This is not true, at least as far as rsync’s best feature, delta transfer, is concerned. If a 1 GB file changes a few bytes, rclone will resend the entire file on the next sync.

multi-threading

Despite the point above, about the lack of “delta transfer”, I often prefer rclone – it is simply much faster because it is multi-threaded. You’ll find this difference show up especially when copying directories containing lots of small files.

instant gratification

Anyone who’s ever looked at rsync’s output knows that it is either too detailed, flowing off the screen as each file is copied, or it is lacking a progress indicator. Of course it has a powerful set of options to control that, but they are pretty arcane. (Even for me!)

Rclone shines in this; just using -P (--progress) it displays sufficient detail on overall progress and files currently in transit, overwriting this information in-place on the screen rather than continuously scrolling off the screen. Try it… just create a temporary directory (which you can delete later), and run rclone -P copy ~/.cache my-new-temp-dir and watch the display.

`--track-renames`

Rclone has a --track-renames option that can be even better than rsync’s delta transfer in certain situations. As the name implies, if you renamed a bunch of files on the source side, neither rclone (by default), nor rsync, would realise that – they’d delete the file with the old name on the destination, and copy the new file. The --track-renames option, fixes this elegently. Of course, it requires that the source and destination backends have at least one hashing algorithm in common, as well as that the destination supports server-side renames of files, but most backends should be fine.

(I was going to write about how it does this here, but the official documentation of this feature, at https://rclone.org/docs/#track-renames, is much better than I could ever do, naturally. So go read that! However, note that this feature does not work with encrypted remotes, making it mostly useless for untrusted/cloud storage).