Automated File Synchronization with Rsync, Expect, and Cron

- Easy Backup / Synchronize / Share Files Between Multiple PCs

If you're like me, you probably have two or more Linux system that you frequently switch between throughout the day. It's common to have multiple desk/lab systems at work, while at home manage a server, laptop, and media center.

In a multi-machine, multi-network environment, each system gets customized with unique settings, scripts, documents, and code that need to be manually duplicated across every system. With no easy way to do this, most people resort to manually copying files or making gradual ad-hoc changes on every system, and eventually making typos that linger undetected.

If this sounds familiar to you, then this post is the solution.

After going through this tedium for a few weeks, I implemented a simple way to easily keep multiple Linux machines in sync, and it can also be used to quickly bootstrap a new machine with all of your customized settings. An added benefit is that no 3rd-party software or background process are required, such as the case with DropBox (however if you feel DropBox will work in your situation, it might be the best option).

Overview

The automated file synchronization works as follows:

1. All files to sync are moved into a new directory (e.g. ~/rsync/) under the home path.
2. Original file locations are replaced with symbolic links to the new location.
3. A script synchronizes files in the sync sub-directory to an online server using rsync and SSH.
4. Any changes to the local files, are eventually uploaded to the server. Also any newer files on the server will replace the out-of-date files on the local systems.
5. The rsync command does not handle merging multiple changes, so some care may be needed to make sure any changes are made to freshly updated files. This can be accomplished by manually running the sync script.
6. Lastly, to simplify "syncing" a fresh machine, a setup script can be used to quickly create all custom symlinks.

Online Server Storage

A file server is used as the repository to store and retrieve all data. Ideally the server must be accessible over the Internet and available 24/7. One could designate a local server, if every machine resides on the same network. The only requirements are that the machine allows for SSH logins and has enough storage space to meet your needs.

In my case, I use an online web-hosting company called NearlyFreeSpeech.net. NFS is very unique in that there are no sign-up fees, and you are only billed for the amount of services you use, such as storage space and transfer amounts. For my setup, the cost is somewhere around $0.02 to $0.05 per month. Assuming you have only a few megabytes of data you wish to synchronize, their prices are extremely low.

Synchronization Script

Now for the interesting part. The synchronization script requires rsync and Expect to login to the remote server and perform any file updates. Expect automates the SSH login process so it works without user interaction. Another method to automate SSH logins is to use shared keys, but some web-hosts do not enable this feature. Since this method is very common, I will not cover it here. If you are using NFS hosting, then you should stick with the default method.

Ensure you have the required dependencies for the script.

sudo apt-get install rsync expect

Save the script to the ~/bin or /usr/local/bin directory.

File: sync.sh

#!/bin/bash ROOT="/home/private/" SITE="ssh.phx.nearlyfreespeech.net" USER="my_user_name" PASSWORD="my_password" RSYNC_OPTS="-e \\\"ssh -o StrictHostKeyChecking=no\\\" -azuv" auto_rsync() { expect -c "eval spawn -noecho rsync --exclude .*.swp $RSYNC_OPTS $1 $2       match_max 10000k       expect \"*?assword:*\"       send -- \"$PASSWORD\r\"       expect eof" } sync() { FILE=$(basename $1) DEST=$(dirname $1) # download remote site file to current location auto_rsync $USER@$SITE:$ROOT$FILE $DEST # update remote site file if newer than backup auto_rsync $1 $USER@$SITE:$ROOT } # backup specific files sync "$HOME/rsync"

Note: Modify the following variables in the script before using:

ROOT	remote directory location to store all files
SITE	remote SSH server hostname
USER	remote SSH user name
PASSWORD	remote SSH password

How To Use the Sync.sh Script

The last line of the script defines the local directory/file to backup to the remote SSH site. The command sync "$HOME/rsync" tells the script that all files in the ~/rsync/ directory should be copied to the remote site. If a newer version exists on the remote site it will not be updated, but instead copied to the local machine.

I find that it is easiest to move all synchronized files under the ~/rsync/ directory so that the script will automatically manage them. Another method would be to add additional directives to specific individual file names. Some users might find this method to be better suited for their needs. Keep in mind that you can not have duplicate filenames on the server in the ROOT directory, and that each directive line initiates a new SSH login (i.e. connection setup overhead).

Symlink Generator Script

If you wish to synchronize common environment files, then a symbolic link must be used in the path to link back to a file synchronized in ~/rsync/. To simplify the management of system files stored in the ~/rsync/ directory, there is a second script to auto-generate symbolic links. First the script:

File: ~/rsync/install.sh
#!/bin/bash RSYNC="$HOME/rsync" ln -sf $RSYNC/.bashrc ~/ ln -sf $RSYNC/.bash_aliases ~/ ln -sf $RSYNC/.vim ~/ ln -sf $RSYNC/.gvimrc ~/ ln -sf $RSYNC/.vimrc ~/ ln -sf $RSYNC/.forward ~/ ln -sf $RSYNC/passwordmaker.rdf ~/ mkdir -p ~/bin ln -sf $RSYNC/bin/sync.sh ~/bin/

Warning: This script will replace any destination files that already exist. Be sure that you have saved copies of any files you do not wish to erase.

This script lists many of the files in my home directory that I synchronize across machines. I store the script in the ~/rsync/ directory, so that it is synchronized across each machine for easy execution. Before running the script for the first time on a new machine, make sure you have a backup of any files with localized changes you want to keep.

Notice that .vim is actually a directory. All the files and sub-directories stored in .vim will work correctly with this single symlink.

Another interesting feature is that the sync.sh script is actually stored in $RSYNC/bin/sync.sh. By doing this, the script can be quickly updated across all of the systems.

Cron Setup

To automate the synchronization process, a cron command can be added to each machine. The example defines a bi-hourly update of your files. You can increase or decrease the frequency as needed. Edit your crontab file with the command below:

crontab -e

Then add the following line to the end of the file:

36 */2 * * *     ~/bin/sync.sh > /dev/null

The first number should be a random value between 0 and 59. This defines the minute of each even hour to connect to the remote SSH server. Using a random value prevents multiple systems from trying to update the SSH server at the same time.

.... Read more!