I came across a real life-saver today…
All of my backup scripts are based on using RSYNC. This is because I am used to working between linux systems on which I always have root/sudo access and also have SSH. However I was recently tasked with a “fun” project that involved me needing to script the syncing of some directories between a linux server and a windows server running third-party SFTP software… that doesn’t allow SSH, SCP, etc.
Now, IMHO, the built-in SFTP program for Ubuntu has a syntax that is a royal PIA for any kind of complex or sophisticated file copy operations. Therefore, getting RSYNC like “mirroring” capability using plain SFTP was going to be a scripting nightmare (perhaps not, maybe I didn’t dig deep enough but that is certainly what it looked like to me…).
After much searching of the Google (thankfully Donald Trump hasn’t turned the internet off yet…) I came across a small program called LFTP referenced on several sites linked below in my “references” section.
First I installed LFTP which is in the ubuntu repositories (love Ubuntu…)
Once installed, here is the scriptable syntax:
The above tells LFTP to execute two commands separated by a semi-colon. The first opens up a connection to the SFTP server using the username “jdoe” and password “passw0rd”. Notice you need to specify the protocol by pre-appending “sftp://” to the url. Also I am using a non default port with the -p flag of “9922”. If you are using key authentication just put some random text where the password should be.
The next command after the semi-colon says I want to “mirror” the two directories and the flags do the following:
-c, –continue – continue a mirror job if possible
-e, –delete delete files not present at remote site
-R, –reverse reverse mirror (put files, vs. a normal mirror used for pulling files I guess??)
-L, –dereference download symbolic links as files
One final note, I think RSYNC is still preferable as I believe (not sure but I think this is correct) it can transfer byte-level changes. So if you have large files and something small changes it will only transfer the changes and not the entire file again. I also don’t know how “intelligent” LFTP is… i.e. does it just check for file names or does it do more in-depth checks like making sure file-size matches or even MD5 sums? I don’t know. So I will continue to prefer RSYNC where I can use it but I will keep LFTP handy going forward when RSYNC isn’t an option.
References:
http://allanmcrae.com/2011/07/syncing-files-across-sftp-with-lftp/
https://www.linux.com/community/blogs/133-general-linux/10459
http://matthewbotos.com/blog/2007/08/04/ssh-keys-and-lftp
http://linux.about.com/od/commands/l/blcmdl1_lftp.htm
Thank you this continues to be helpful
I like rsync and was a regular user of it. But alternatives of it on windows are very less and if they are there they dont seem to be useful like this software. However, after searching a lot I found GS Richcopy 360 works really well and provides all the options that rsync provided plus its multi threaded file transfer saves a lot of time while transferring file. I love it, try it hope it helps you too!
nice article
Yo do this task I personally prefer using gs richcopy 360, works really well. Does the task as expected with much hassle and errors. I am a happy customer of it.
simple and well written and 100% covers the crux of this.
The MAJOR advantage of rsync its wonderful ability to transfer only parts of files, as you say, This can make HUGE difference when some of the files transfered are log files.
Thanks JP!
I love RSYNC when it is an option! in my experience it is phenomenally smart/efficient and it just works. Thanks for the feedback!