shauryadocs: fix broken links and wrong s3 directory bucket flag name9dbfd9d2 months agoHistory
3588 lines · 134.3 KB · MarkdownRawDownload

title: "Documentation" description: "Rclone Usage"

Usage

Rclone is a command line program to manage files on cloud storage. After download and install, continue here to learn how to use it: Initial configuration, what the basic syntax looks like, describes the various subcommands, the various options, and more.

Configure

First, you'll need to configure rclone. As the object storage systems have quite complicated authentication these are kept in a config file. (See the --config entry for how to find the config file and choose its location.)

The easiest way to make the config is to run rclone with the config option:

rclone config

See the following for detailed instructions for

Basic syntax

Rclone syncs a directory tree from one storage system to another.

Its syntax is like this

rclone subcommand [options] <parameters> <parameters...>

A subcommand is an rclone operation required (e.g. sync, copy, ls).

An option is a single letter flag (e.g. -v) or a group of single letter flags (e.g. -Pv) or a long flag (e.g. --progress). No options are required. Options can come after the subcommand or in between parameters too or on the end, but only global options can be used before the subcommand. Anything after a -- option will not be interpreted as an option so if you need to add a parameter which starts with a - then put a -- on its own first, eg

rclone lsf -- -directory-starting-with-dash

A parameter is usually a file path or rclone remote, eg /path/to/file or remote:path/to/file but it can be other things - the subcommand help will tell you what.

Source and destination paths are specified by the name you gave the storage system in the config file then the sub path, e.g. "drive:myfolder" to look at "myfolder" in Google drive.

You can define as many storage paths as you like in the config file.

Please use the --interactive/-i flag while learning rclone to avoid accidental data loss.

Subcommands

rclone uses a system of subcommands. For example

rclone ls remote:path # lists a remote
rclone copy /local/path remote:path # copies /local/path to the remote
rclone sync --interactive /local/path remote:path # syncs /local/path to the remote

The main rclone commands with most used first

See the commands index for the full list.

Copying single files

rclone normally syncs or copies directories. However, if the source remote points to a file, rclone will just copy that file. The destination remote must point to a directory - rclone will give the error Failed to create file system for "remote:file": is a file not a directory if it isn't.

For example, suppose you have a remote with a file in called test.jpg, then you could copy just that file like this

rclone copy remote:test.jpg /tmp/download

The file test.jpg will be placed inside /tmp/download.

This is equivalent to specifying

rclone copy --files-from /tmp/files remote: /tmp/download

Where /tmp/files contains the single line

test.jpg

It is recommended to use copy when copying individual files, not sync. They have pretty much the same effect but copy will use a lot less memory.

Syntax of remote paths

The syntax of the paths passed to the rclone command are as follows.

/path/to/dir

This refers to the local file system.

On Windows \ may be used instead of / in local paths only, non local paths must use /. See local filesystem documentation for more about Windows-specific paths.

These paths needn't start with a leading / - if they don't then they will be relative to the current directory.

remote:path/to/dir

This refers to a directory path/to/dir on remote: as defined in the config file (configured with rclone config).

remote:/path/to/dir

On most backends this is refers to the same directory as remote:path/to/dir and that format should be preferred. On a very small number of remotes (FTP, SFTP, Dropbox for business) this will refer to a different directory. On these, paths without a leading / will refer to your "home" directory and paths with a leading / will refer to the root.

:backend:path/to/dir

This is an advanced form for creating remotes on the fly. backend should be the name or prefix of a backend (the type in the config file) and all the configuration for the backend should be provided on the command line (or in environment variables).

Here are some examples:

rclone lsd --http-url https://pub.rclone.org :http:

To list all the directories in the root of https://pub.rclone.org/.

rclone lsf --http-url https://example.com :http:path/to/dir

To list files and directories in https://example.com/path/to/dir/

rclone copy --http-url https://example.com :http:path/to/dir /tmp/dir

To copy files and directories in https://example.com/path/to/dir to /tmp/dir.

rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir

To copy files and directories from example.com in the relative directory path/to/dir to /tmp/dir using sftp.

Connection strings {#connection-strings}

The above examples can also be written using a connection string syntax, so instead of providing the arguments as command line parameters --http-url https://pub.rclone.org they are provided as part of the remote specification as a kind of connection string.

rclone lsd ":http,url='https://pub.rclone.org':"
rclone lsf ":http,url='https://example.com':path/to/dir"
rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
rclone copy :sftp,host=example.com:path/to/dir /tmp/dir

These can apply to modify existing remotes as well as create new remotes with the on the fly syntax. This example is equivalent to adding the --drive-shared-with-me parameter to the remote gdrive:.

rclone lsf "gdrive,shared_with_me:path/to/dir"

The major advantage to using the connection string style syntax is that it only applies to the remote, not to all the remotes of that type of the command line. A common confusion is this attempt to copy a file shared on google drive to the normal drive which does not work because the --drive-shared-with-me flag applies to both the source and the destination.

rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:

However using the connection string syntax, this does work.

rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive:

Note that the connection string only affects the options of the immediate backend. If for example gdriveCrypt is a crypt based on gdrive, then the following command will not work as intended, because shared_with_me is ignored by the crypt backend:

rclone copy "gdriveCrypt,shared_with_me:shared-file.txt" gdriveCrypt:

The connection strings have the following syntax

remote,parameter=value,parameter2=value2:path/to/dir
:backend,parameter=value,parameter2=value2:path/to/dir

If the parameter has a : or , then it must be placed in quotes " or ', so

remote,parameter="colon:value",parameter2="comma,value":path/to/dir
:backend,parameter='colon:value',parameter2='comma,value':path/to/dir

If a quoted value needs to include that quote, then it should be doubled, so

remote,parameter="with""quote",parameter2='with''quote':path/to/dir

This will make parameter be with"quote and parameter2 be with'quote.

If you leave off the =parameter then rclone will substitute =true which works very well with flags. For example, to use s3 configured in the environment you could use:

rclone lsd :s3,env_auth:

Which is equivalent to

rclone lsd :s3,env_auth=true:

Note that on the command line you might need to surround these connection strings with " or ' to stop the shell interpreting any special characters within them.

If you are a shell master then you'll know which strings are OK and which aren't, but if you aren't sure then enclose them in " and use ' as the inside quote. This syntax works on all OSes.

rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir

On Linux/macOS some characters are still interpreted inside " strings in the shell (notably \ and $ and ") so if your strings contain those you can swap the roles of " and ' thus. (This syntax does not work on Windows.)

rclone copy ':http,url="https://example.com":path/to/dir' /tmp/dir

You can use rclone config string to convert a remote into a connection string.

Connection strings, config and logging

If you supply extra configuration to a backend by command line flag, environment variable or connection string then rclone will add a suffix based on the hash of the config to the name of the remote, eg

rclone -vv lsf --s3-chunk-size 20M s3:

Has the log message

DEBUG : s3: detected overridden config - adding "{Srj1p}" suffix to name

This is so rclone can tell the modified remote apart from the unmodified remote when caching the backends.

This should only be noticeable in the logs.

This means that on the fly backends such as

rclone -vv lsf :s3,env_auth:

Will get their own names

DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name

Valid remote names

Remote names are case sensitive, and must adhere to the following rules:

  • May contain number, letter, _, -, ., +, @ and space.
  • May not start with - or space.
  • May not end with space.

Starting with rclone version 1.61, any Unicode numbers and letters are allowed, while in older versions it was limited to plain ASCII (0-9, A-Z, a-z). If you use the same rclone configuration from different shells, which may be configured with different character encoding, you must be cautious to use characters that are possible to write in all of them. This is mostly a problem on Windows, where the console traditionally uses a non-Unicode character set - defined by the so-called "code page".

Do not use single character names on Windows as it creates ambiguity with Windows drives' names, e.g.: remote called C is indistinguishable from C drive. Rclone will always assume that single letter name refers to a drive.

Adding global configuration to a remote {#globalconfig}

It is possible to add global configuration to the remote configuration which will be applied just before the remote is created.

This can be done in two ways. The first is to use override.var = value in the config file or the connection string for a temporary change, and the second is to use global.var = value in the config file or connection string for a permanent change.

This is explained fully below.

override.var

This is used to override a global variable just for the duration of the remote creation. It won't affect other remotes even if they are created at the same time.

This is very useful for overriding networking config needed for just for that remote. For example, say you have a remote which needs --no-check-certificate as it is running on test infrastructure without a proper certificate. You could supply the --no-check-certificate flag to rclone, but this will affect all the remotes. To make it just affect this remote you use an override. You could put this in the config file:

[remote]
type = XXX
...
override.no_check_certificate = true

or use it in the connection string remote,override.no_check_certificate=true: (or just remote,override.no_check_certificate:).

Note how the global flag name loses its initial -- and gets - replaced with _ and gets an override. prefix.

Not all global variables make sense to be overridden like this as the config is only applied during the remote creation. Here is a non exhaustive list of ones which might be useful:

  • bind_addr
  • ca_cert
  • client_cert
  • client_key
  • connect_timeout
  • disable_http2
  • disable_http_keep_alives
  • dump
  • expect_continue_timeout
  • headers
  • http_proxy

Lines 1–500 of 3588.