rsync-sidekick

Why?

rsync is a fantastic tool. Yet, by itself, it's a pain to use for repeated backing up of media files (videos, music, photos, etc.) that are reorganized frequently.

rsync-sidekick is a safe and simple tool that is designed to run before rsync is run.

What?

rsync-sidekick propagates following changes (or any combination) from source directory to destination directory:

1. Change in file modification timestamp
2. Rename of file/directory
3. Moving a file from one directory to another

Additionally, it does the following things:

1. Directory timestamp synchronization (with -d flag)
2. Local copying of duplicate-content files at the destination (with -c flag)
3. Copying from archive/backup directories on the destination side (with -a flag)

It works with local directories, remote hosts via SSH (using a remote agent or SFTP fallback) and even inside Docker containers! 🙂

Note:

• This tool does not delete any files or folders (under any circumstances) — that's why it's safe to use 😌
  • Your files are just moved around
  • Now, if you're uncomfortable with this tool even moving your files around, consider using the --dry-run option
• This tool does not actually transfer files — that's for rsync to do 🙂
• Since you'd run rsync after this tool is run, any changes that this tool couldn't propagate would just be propagated by rsync
  • So the most that you might lose is some time with rsync doing more work than it could have — Which is likely still much less than not using this tool at all 😄

How to install?

1. Install Go version at least 1.25
   • On Ubuntu: snap install go
   • On Mac: brew install go
   • For anything else: Go downloads page
2. Run command:

   go install github.com/m-manu/rsync-sidekick/v2@latest

3. Add following line in your .bashrc/.zshrc file:

   export PATH="$PATH:$HOME/go/bin"

How to use?

Step 1: Run this tool

# Local to local:
rsync-sidekick /Users/manu/Photos/ /Volumes/Portable/Photos/

# Local to remote (faster if rsync-sidekick is also installed on remote host):
rsync-sidekick /Users/manu/Photos/ user@server:/backup/Photos/

# Remote to local:
rsync-sidekick user@server:/data/Photos/ /Users/manu/Photos/

Step 2: Run rsync as you would normally do

# Note the trailing slashes below. Without them, rsync's behavior is different!
rsync -av /Users/manu/Photos/ /Volumes/Portable/Photos/

Command line options

Running rsync-sidekick --help displays following information:

rsync-sidekick is a tool to propagate file renames, movements and timestamp changes from a source directory to a destination directory.

Usage:
	 rsync-sidekick <flags> [source] [destination]

where,
	[source]        Source directory (local path or user@host:/path)
	[destination]   Destination directory (local path or user@host:/path)

flags: (all optional)
      --archive-one-file-system       don't cross filesystem boundaries when scanning archive paths
                                      (by default archives DO cross boundaries, e.g. into btrfs snapshot subvols)
                                      (works locally and with remote-exec, not with SFTP)
  -a, --archive-path stringArray      additional directory on the destination side to scan for copy sources
                                      (can be specified multiple times; files are copied from archive, never moved;
                                      implies --copy-duplicates)
  -c, --copy-duplicates               copy files locally at destination when content already exists there
                                      (avoids re-transfer of duplicate-content files via rsync)
  -n, --dry-run                       show what would be done, but don't actually perform any actions
  -x, --exclusions string             path to file containing newline separated list of file/directory names to be excluded
                                      (even if this is not set, files/directories such these will still be ignored: $RECYCLE.BIN, desktop.ini, Thumbs.db etc.)
  -h, --help                          display help
      --list                          list files along their metadata for given directory
      --one-file-system               don't cross filesystem boundaries when scanning source and destination (like rsync -x)
                                      (works locally and with remote-exec, not with SFTP)
  -f, --progress-frequency duration   frequency of progress reporting e.g. '5s', '1m' (default 2s)
      --reflink                       use cp --reflink=auto for copy actions (instant on CoW filesystems like btrfs/XFS)
                                      (only effective when copies are performed via --copy-duplicates or --archive-path)
      --sftp                          force SFTP mode (don't try remote-execution)
  -s, --shellscript                   instead of applying changes directly, generate a shell script
                                      (this flag is useful if you want to run the shell script as a different user)
  -p, --shellscript-at-path string    similar to --shellscript option but you can specify output script path
                                      (this flag cannot be specified if --shellscript option is specified)
      --sidekick-path string          remote rsync-sidekick command (e.g. "sudo rsync-sidekick") (default "rsync-sidekick")
  -i, --ssh-key string                path to SSH private key for remote connections
  -d, --sync-dir-timestamps           also propagate directory timestamps from source to destination
  -v, --verbose                       generates extra information, even a file dump (caution: makes it slow!)
      --version                       show application version (v2.0.1) and exit

More details here: https://github.com/m-manu/rsync-sidekick

Advanced options

Syncing remote paths (via SSH)

rsync-sidekick supports syncing to/from remote hosts via SSH. Either the source or the destination (not both!) can be a remote path in the form user@host:/path.

Remote-execution mode (default): If rsync-sidekick is installed on the remote host, it will be used as an agent for fast remote scanning and action execution. This is the recommended setup.

SFTP fallback: If rsync-sidekick is not available on the remote host, it falls back to SFTP mode automatically. You can also force SFTP mode with --sftp.

# Use a specific SSH key:
rsync-sidekick -i ~/.ssh/my_key /Users/manu/Photos/ user@server:/backup/Photos/

# Specify the remote rsync-sidekick path:
rsync-sidekick --sidekick-path /usr/local/bin/rsync-sidekick /local/path/ user@server:/remote/path/

# Run the remote agent with sudo (useful when syncing to root-owned directories):
rsync-sidekick --sidekick-path "sudo rsync-sidekick" /local/path/ user@server:/remote/path/

Copying duplicate-content files (--copy-duplicates)

By default, rsync-sidekick only moves files at the destination. If the same content exists at multiple paths at source but only one of those paths exists at the destination, the extra copies are left for rsync to transfer.

With --copy-duplicates (or -c), rsync-sidekick will copy the file locally at the destination instead, saving network transfer time:

# Source has photo.jpg at paths A, B, C (same content). Destination only has it at A.
# Without -c: B and C must be transferred by rsync.
# With -c: rsync-sidekick copies A→B and A→C locally at the destination.
rsync-sidekick -c /Users/manu/Photos/ /Volumes/Portable/Photos/

Using archive directories (--archive-path)

The --archive-path (or -a) flag lets you specify additional directories on the destination side that are scanned for content matches. Files are only copied from archives, never moved. This is useful when you have an old backup or archive that might contain files matching orphans at the source.

Archive paths imply --copy-duplicates behavior automatically.

# Scan an old backup directory for matching content:
rsync-sidekick -a /Volumes/OldBackup/Photos/ /Users/manu/Photos/ /Volumes/Portable/Photos/

# Multiple archive paths:
rsync-sidekick -a /archive1/ -a /archive2/ /source/ /destination/

# Works with remote destinations too (archives must be on the remote host):
rsync-sidekick -a /remote/archive/ /local/source/ user@server:/remote/dest/

Reflink copies (--reflink)

On CoW (copy-on-write) filesystems like btrfs or XFS, the --reflink flag makes copy actions use cp --reflink=auto, which is instant and uses no additional disk space. On filesystems that don't support reflinks, it falls back to a regular copy automatically. This flag only has an effect when copies are being performed (via --copy-duplicates or --archive-path).

# Instant zero-cost copies on btrfs:
rsync-sidekick -c --reflink /Users/manu/Photos/ /mnt/btrfs-backup/Photos/

Staying on one filesystem (--one-file-system)

On btrfs or other setups with nested mount points / subvolumes, --one-file-system prevents rsync-sidekick from crossing filesystem boundaries when scanning source and destination directories (similar to rsync -x). Each subvolume on btrfs has a different device ID, so this effectively keeps the scan within a single subvolume.

A separate --archive-one-file-system flag controls the same behavior for archive paths. By default, archives do cross filesystem boundaries, since you typically point -a at a parent directory containing multiple btrfs snapshot subvolumes.

Both flags work locally and with remote-exec mode. They have no effect in SFTP mode, since SFTP does not expose filesystem/device boundaries.

# Scan only within the @ subvolume, don't descend into nested subvols:
rsync-sidekick --one-file-system -c --reflink /mnt/data/@ /mnt/backup/@

# Archives cross into snapshots by default (no extra flag needed):
rsync-sidekick --one-file-system -c --reflink -a /mnt/backup/.snapshots/@/ /mnt/data/@ /mnt/backup/@

Running this from a Docker container

Not everyone needs this. But if you do, below is a simple example:

# Run rsync-sidekick:
docker run --rm -v /Users/manu:/mnt/homedir manumk/rsync-sidekick rsync-sidekick /mnt/homedir/Photos/ /mnt/homedir/Photos_backup/

# Then run rsync: (note the trailing slashes -- without them, rsync's behavior is different)
docker run --rm -v /Users/manu:/mnt/homedir manumk/rsync-sidekick rsync /mnt/homedir/Photos/ /mnt/homedir/Photos_backup/

FAQs

Why was this tool created? Doesn't rsync provide flags for doing what this tool does? 🤔

rsync options such as --detect-renamed, --detect-renamed-lax, --detect-moved and --fuzzy don't work reliably and sometimes are dangerous! rsync-sidekick is reliable alternative to all these options and much more!

How will I benefit from using this tool?

Using rsync-sidekick before rsrync makes your backup process significantly faster than using only rsync. Sometimes this performance benefit can even be 100x😲, if the only changes at your source directory are the 3 types mentioned earlier in this article.