Edit [2021-02-17]: we have reversed the changes back to the original, as it seems to work better as it was. The suggested change was from using -auvhPW instead of -rutvahz (see below shell script commands). Also, we highly recommend the AOMEI for full system backups and hard drive cloning. We now use a combination of rsync, Windows File History and AOMEI.
Edit [2020-08-14]: we have made some changes to this guide based on user feedback.
Over our years providing software development services, we have used many different file backup techniques, including: manually copying files, running Windows backups, 3rd-party software solutions, Rich/Robo Copy, shell scripts (which we’ll talk about), and we even tried one of those external hard-drive software bundles, like WD Passport.
We reluctantly installed Windows 10 on our main computers, figuring that Windows 10 has now had ample time to get most of the kinks worked out, and to our surprise “Backup and Restore” from Windows 7 is being deprecated, and the “Backup and Restore (Windows 7)” in Windows 10 seems to fail no matter what was tried. The only option is to use their new backup option, File History, which we use to an extent.
However, we ran into issues with File History when trying to select specific folders under the AppData directory. It didn’t seem to target them properly and the only way to add them was to add the entire AppData directory, instead of specific directories from within.
So now we create our own backup scripts using a file-transfer command-line tool called rsync.
rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License and is currently being maintained by Wayne Davison.
It’s a bit of a process to get this setup on Windows, but it’s worth it. If you have a linux system, then you already have rsync installed by default.
- Download and install Notepad++; used to instill proper line-endings for the shell script we will create.
- Download, install, and configure Cygwin; this program simulates a shell in Windows, and also provides rsync as an option during installation.
- Go through some rsync command-line examples.
Notepad++ is a great tool for quick notes and file editing/previewing. It allows for multiple documents to be open at once, and it remembers unsaved documents for you. You can download Notepad++ from here: https://notepad-plus-plus.org/
We will use this program to create a shell file (.sh extension) that we can run from Cygwin. Notepad++ helps us solve an issue where regular text files created in Windows have different line ending characters (\r\n) than those that are used in a Unix/Linux environment (\n).
We’ll use Notepad++ to set the line endings of our shell script to be that of Unix decent. Open Notepad++ and create a new file, then click on Edit > EOL Conversion > Unix (LF), then save your file with a .sh extension (eg. mybackup.sh).
In the very first line of that file, write the following:
All remaining commands will go below this line.
2. Download, install and configure Cygwin
Cygwin provides a local shell (command-line) environment with configuration for many of the same capabilities as you’d have on a Linux OS or Mac machine, and this program will allow us to run the rsync command.
- Go to https://www.cygwin.com/
- Download your supported executable (.exe), 64 or 32-bit, file from the links found under the section Installing Cygwin.
- Open the file to begin the installation process…
- During installation, you will be asked which modules to install; you can use the search box at the top to help locate the modules we need. Find and select both ssh and rsync.
- Once installed, open the Cygwin terminal.
- If you make a mistake or need to install additional modules, you may run the Cygwin installer again to select additional modules.
3. Rsync commands
To view all accessible drives from Cygwin, type the following in your Cygwin terminal:
From here, we can access our files on our main drives, and plan our backup strategy. Let’s take a look at a rsync command that copies everything from C-drive to D-drive:
rsync -rutvahz --stats /cygdrive/c/ /cygdrive/d/
The above command uses some very specific options/flags, which are explained below, starting with -auvhPW:
- a = archive, includes -r (recursive, into directories) and -t (times, preserve)
- u = update, skips files if they are newer
- v = verbose, to improve quality of transfer
- h = human-readable output
- z = compress during transfer (removed)
- P = equivalent to –partial –progress (providing file transfer progress %)
- W = copy files whole (equivalent to –whole-file)
- stats = provides statistics about the transfer upon completion
- Other flags can be found, and are described, on the rsync website.
This first example will copy all files from your user (MyUserName) /Documents/ folder on your C-drive, to a folder called /Documents/ on your D-drive, which can be an external hard drive, USB stick, etc.:
rsync -rutvahz --stats /cygdrive/c/Users/MyUserName/Documents/ /cygdrive/d/Documents/
- exclude ‘folder/file’ = allows you to dictate which files or folders to skip during the sync. You may chain –exclude flags to ignore multiple directories:
rsync -rutvahz --stats --exclude 'Private' --exclude 'SemiPrivate' /cygdrive/c/Users/MyUserName/Documents/ /cygdrive/d/Documents/
You may run the above commands right from the Cygwin terminal, or – especially if you have multiple rsync commands – you can save those commands to mybackup.sh and then run the following command: