I sometimes have to backup a lot of data from users that move out of the university that I work at. Rsync tends to be very slow and serial at transferring data. I user globus-url-copy from the [Globus Toolkit] to transfer data. I do work on High Performance Linux clusters with high speed networking with decently fast storage, but nevertheless I use Globus-FTP at home to backup my backups on my Macs and Windows boxes.
The Globus Toolkit is meant for research grids, but works wonders for local transfers. It basically allows you to transfer data by breaking up the data into smaller chunks just like Torrents except it is file by file. Even lots of small files tend to be fast. You can pick your block size and sync directories with various levels of checksums. There is so much to play around with it, but the guide in the link above provides some really good values to try. You can even use multiple interfaces to copy the same files. For remote copies there is a server to set up but you can just use ftp or local files.
Anybody else have any tips?
[link] [2 comments]