<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"><HTML><HEAD><TITLE>rsync FAQ</TITLE></HEAD><!--#include virtual="header.html" --> <H2 align="center">Frequently Asked Questions</H2> <table><tr valign=top><td><ul><li><a href="#1">the transfer fails to finish</a><br><li><a href="#2">rsync recopies the same files</a><br><li><a href="#3">is your shell clean</a><br><li><a href="#4">memory usage</a><br><li><a href="#5">out of memory</a><br><li><a href="#6">rsync through a firewall</a><br><li><a href="#7">rsync and cron</a><br></ul></td><td>&nbsp;&nbsp;&nbsp;&nbsp;</td><td><ul><li><a href="#8">rsync: Command not found</a><br><li><a href="#9">spaces in filenames</a><br><li><a href="#10">ignore "vanished files" warning</a><br><li><a href="#11">read-only file system</a><br><li><a href="#12">multiplexing overflow 101:7104843</a><br><li><a href="#13">inflate (token) returned -5</a><br></ul></td></tr></table> <hr><h3><a name=1>the transfer fails to finish</a></h3> <p>If you get an error like one of these: <pre>rsync: error writing 4 unbuffered bytes - exiting: Broken pipersync error: error in rsync protocol data stream (code 12) at io.c(463)</pre> <p>or <pre>rsync: connection unexpectedly closed (24 bytes read so far)rsync error: error in rsync protocol data stream (code 12) at io.c(342)</pre> <p>please read the <a href="issues.html">issues and debugging page</a>for details on how you can try to figure out what is going wrong. <hr><h3><a name=2>rsync recopies the same files</a></h3> <p>Some people occasionally report that rsync copies too many files whenthey expect it to copy only a few. In most cases the explanation isthat you forgot to include the --times (-t) option in the original copy,so rsync is forced to (efficiently) transfer every file that differs inits modified time to discover what data (if any) has changed. <p>Another common cause involves sending files to an Microsoft filesystem:if the file's modified time is an odd value but the receiving filesystemcan only store even values, then rsync will re-transfer too many files.You can avoid this by specifying the --modify-window=1 option. <p>Yet another periodic case can happen when daylight-savings timechanges if your OS+filesystem saves file times in local time instead ofUTC.  For a full explanation of this and some suggestions on how toavoid them problem, see <a href="daylight-savings.html">this document</a>. <p>Something else that can trip up rsync is a filesystem changeing thefilename behind the scenes.  This can happen when a filesystem changesan all-uppercase name into lowercase, or when it decomposes UTF-8 behindyour back. <blockquote> <p>An example of the latter can occur with HFS+ on Mac OS X:  if youcopy a directory with a file that has a UTF-8 character sequence in it,say a 2-byte umlaut-u (\0303\0274), the file will get that characterstored by the filesystem using 3 bytes (\0165\0314\0210), and rsync willnot know that these differing filenames are the same file (it will, infact, remove a prior copy of the file if --delete is enabled, and thenrecreate it). <p>You can avoid a charset problem by passing an appropriate --iconvoption to rsync that tells it what character-set the source files are,and what character-set the destination files get stored in.  Forinstance, the above Mac OS X problem would be dealt with by using--iconv=UTF-8,UTF8-MAC (UTF8-MAC is a pseudo-charset recognized by MacOS X iconv in which all characters are decomposed). </blockquote> <p>If you think that rsync is copying too many files, look at theitemized output (-i) to see why rsync is doing the update (e.g. the 't'flag indicates that the time differs, or all pluses indicates that rsyncthinks the file doesn't exist).  You can also look at the stats producedwith -v and see if rsync is really sending all the data.  See also the--checksum (-c) option for one way to avoid the extra copying of filesthat don't have synchronized modified times (but keep in mind that the-c option eats lots of disk I/O, and can be rather slow). <hr><h3><a name=3>is your shell clean</a></h3> <p>The "is your shell clean" message and the "protocol mismatch" messageare usually caused by having some sort of program in your .cshrc, .profile,.bashrc or equivalent file that writes a message every time you connectusing a remote-shell program (such as ssh or rsh).  Data written in thisway corrupts the rsync data stream. rsync detects this at startup andproduces those error messages.  However, if you are using rsync-daemonsyntax (host::path or rsync://) without using a remote-shell program (no--rsh or -e option), there is not remote-shell program involved, and theproblem is probably caused by an error on the daemon side (so check thedaemon logs). <p>A good way to test if your remote-shell connection is clean is to trysomething like this (use ssh or rsh, as appropriate): <blockquote><pre>ssh remotesystem /bin/true &gt; test.dat</pre></blockquote> <p>That should create a file called test.dat with nothing in it. Iftest.dat is not of zero length then your shell is not clean.  Look at thecontents of test.dat to see what was sent. Look at all the startup files onremotesystem to try and find the problem. <hr><h3><a name=4>memory usage</a></h3> <p>Rsync versions before 3.0.0 always build the entire list of files to betransferred at the beginning and hold it in memory for the entire run.  Rsyncneeds about 100 bytes to store all the relevant information for one file,so (for example) a run with 800,000 files would consume about 80M ofmemory.  -H and --delete increase the memory usage further. <p>Version 3.0.0 slightly reduced the memory used per file by not storing fieldsnot needed for a particular file.  It also introduced an incremental recursionmode that builds the file list in chunks and holds each chunk in memory only aslong as it is needed.  This mode dramatically reduces memory usage, but itonly works provided that both sides are 3.0.0 or newer and certain options thatrsync currently can't handle in this mode are not being used. <hr><h3><a name=5>out of memory</a></h3> <p>The usual reason for "out of memory" when running rsync is that you aretransferring a _very_ large number of files.  The size of the files doesn'tmatter, only the total number of files.  If memory is a problem, first try touse the incremental recursion mode: upgrade both sides to rsync 3.0.0 ornewer and avoid options that disable incremental recursion (e.g., use<tt>--delete-delay</tt> instead of <tt>--delete-after</tt>).  If this is notpossible, you can break the rsync run into smaller chunks operating onindividual subdirectories using <tt>--relative</tt> and/or exclude rules. <hr><h3><a name=6>rsync through a firewall</a></h3> <p>If you have a setup where there is no way to directly connect twosystems for an rsync transfer, there are several ways to get a firewallsystem to act as an intermediary in the transfer.  You'll find full detailson the <a href="firewall.html">firewall methods</a> page. <hr><h3><a name=7>rsync and cron</a></h3> <p>On some systems (notably SunOS4) cron supplies what looks like a socketto rsync, so rsync thinks that stdin is a socket. This means that if youstart rsync with the --daemon switch from a cron job you end up rsyncthinking it has been started from inetd. The fix is simple&mdash;justredirect stdin from /dev/null in your cron job. <hr><h3><a name=8>rsync: Command not found</a></h3> <p>This error is produced when the remote shell is unable to locate the rsyncbinary in your path. There are 3 possible solutions: <ol> <li>install rsync in a "standard" location that is in your remote path.  <li>modify your .cshrc, .bashrc etc on the remote system to include the paththat rsync is in <li>use the --rsync-path option to explicitly specify the path on theremote system where rsync is installed </ol> <p>You may echo find the command: <blockquote><pre>ssh host 'echo $PATH'</pre></blockquote> <p>for determining what your remote path is. <hr><h3><a name=9>spaces in filenames</a></h3> <p>Can rsync copy files with spaces in them? <p>Short answer: Yes, rsync can handle filenames with spaces. <p>Long answer:  <p>Rsync handles spaces just like any other unix command line application.Within the code spaces are treated just like any other character so afilename with a space is no different from a filename with any othercharacter in it. <p>The problem of spaces is in the argv processing done to interpret thecommand line.  As with any other unix application you have to escape spacesin some way on the command line or they will be used to separate arguments.  <p>It is slightly trickier in rsync (and other remote-copy programs likescp) because rsync sends a command line to the remote system to launch thepeer copy of rsync (this assumes that we're not talking about daemon mode,which is not affected by this problem because no remote shell is involvedin the reception of the filenames).  The command line is interpreted by theremote shell and thus the spaces need to arrive on the remote systemescaped so that the shell doesn't split such filenames into multiplearguments. <p>For example: <blockquote><pre>rsync -av host:'a long filename' /tmp/</pre></blockquote> <p>This is usually a request for rsync to copy 3 files from the remotesystem, "a", "long", and "filename" (the only exception to this is for asystem running a shell that does not word-split arguments in its commands,and that is exceedingly rare).  If you wanted to request a single file withspaces, you need to get some kind of space-quoting characters to the remoteshell that is running the remote rsync command.  The following commandsshould all work: <blockquote><pre>rsync -av host:'"a long filename"' /tmp/rsync -av host:'a\ long\ filename' /tmp/rsync -av host:a\\\ long\\\ filename /tmp/</pre></blockquote> <p>You might also like to use a '?' in place of a space as long as thereare no other matching filenames than the one with spaces (since '?' matchesany character): <blockquote><pre>rsync -av host:a?long?filename /tmp/</pre></blockquote> <p>As long as you know that the remote filenames on the command lineare interpreted by the remote shell then it all works fine.  <hr><h3><a name=10>ignore "vanished files" warning</a></h3> <p>Some folks would like to ignore the "vanished files" warning, whichmanifests as an exit-code 24.  The easiest way to do this is to createa shell script wrapper.  For instance, name this something like"rsync-no24": <blockquote><pre>#!/bin/shrsync "$@"e=$?if test $e = 24; then    exit 0fiexit $e</pre></blockquote> <hr><h3><a name=11>read-only file system</a></h3> <p>If you get "Read-only file system" as an error when sending to a rsyncdaemon then you probably forgot to set "read only = no" for that module. <hr><h3><a name=12>multiplexing overflow 101:7104843</a></h3> <p>This mysterious error, or the similar "invalid message 101:7104843", canhappen if one of the rsync processes is killed for some reason and a messagebeginning with the four characters "Kill" gets inserted into the protocolstream as a result.  To solve the problem, you'll need to figure out why rsyncis being killed. <hr><h3><a name=13>inflate (token) returned -5</a></h3> This error means that rsync failed to handle an expected error from thecompression code for a file that happened to be transferred with a block sizeof 32816 bytes.  You can avoid this issue for the affected file by transferringit with a manually-set block size (e.g. --block-size=33000), or by upgradingthe receiving side to rsync 3.0.7. <hr> <!--#include virtual="footer.html" -->