Rsync, SCP Network Transfer Speed

Probably something with negligible overhead like nc. A quick test showed it 20% faster than scp for me on a 1.4GB file.

A nearby group at work that must replicate large disk images across geos regulary, is quite fond of https://tsunami-udp.sourceforge.net/ , but I am not sure it applies to your use case; but it might be interesting to research UDP-based protocols.

(Otherwise, I've used nc on occasion too, but make sure the inconvenience is worthwhile. Usage of replicated file systems, deltas, etc. are other stuff to research.)

rsync will do block level transfers in some cases, which will be faster, and if only deltas need to be transmitted you will see even more efficiency. scp works at the file level exclusively, so the overhead of the filesystem factors into the equation. any udp based protocol will be fast to transfer, but the overhead of the file system does still come into play

in general:

rsync block level transfers - most efficient
udp based file transfers - fastest transfer
tcp based file transfers - most resilient

i think the drbd project is used to replicate data between hosts that you are running NAS or SAN services on, and that is near real-time replication with similar functionality of rsync. you have 10Gb links between hosts. are you able to use Infiniband, or the open source alternative iSER? this usually has a requirement of Mellanox or Marvell cards because there are drivers that allow the CPU to be bypassed when shuttling bits around between memory and the network stack.

when thinking about how to go about this, think of the subsystems involved. how many, how much overhead each adds, what the slowest links are and what protocols are used. you can also look at tweaking kernel settings via sysctl to improve networking and afford better performance. most kernels are shipped with tuning configs that offend the fewest number of customers. tweaks i have on my nas allocate more memory to the networking stack, and set larger buffers. with the bonded 4 port Gb NICs, i also increase ring buffers, which helps.

aside from using the appropriate transfer mechanism, tuning and tweaking things may help improve the overall performance, as well.

ZFS can be an excellent solution in this space; it tracks block-level changes, and can send precomputed deltas. It doesn't have the overhead rsync does of comparing two files byte-for-byte, it already knows what changed since the last sync. It can be very efficient.

However, it's a "study filesystem". You will need to put in quite substantial effort learning it, and you'll probably have to build your own sync solutions. @Jim Salter wrote a utility called syncoid, but it's not good with recursively snapshotting and sending filesystems, so I had to write my own scripts. Your needs may, ahem, sync better with syncoid.

Rsync is very fast, to speed it up disable encryption and potentially (you'll need to test this) enable compression. You can also look to the network and enable jumbo frames.

Use 'rsync' because it has the '-z' option to compress the file.
If you read the man page, then under SETUP you will find that "For remote transfers modern rsync uses ssh", so you'll also transfer it encrypted.