scvalex.net

9. On Transfer Speeds

Let’s say we’re trying to copy a \(1\) GiB file from one machine to another over a gigabit ethernet LAN. How long will it take? (Spoiler: it’s quite a bit more than the simple calculation says.)

Our setup contains two machines connected via gigabit ethernet to a router. We’re going to transfer \(1\) GiB (\(2^30\) bytes) between them. A quick calculation tells us that this should take \(8.6\)s, but does it?

We have a few ways of transferring the file; let’s start with the simplest: netcat.

# Start `netcat` on one of the machines
edge13 % nc -l 12345 > /dev/null

# Create a 1 GiB file on the other machine
edge12 % dd bs=1024 count=1048576 \
              if=/dev/zero of=file.bin
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied,
12.6983 s, 84.6 MB/s

# Transfer the file over via `netcat`
edge12 % time nc edge13 12345 < file.bin

I see it takes \(13.27\)s to transfer \(1\) GiB between the two machines with netcat (average over \(5\) runs). That comes up to \(647.31\) mbps. This is pretty bad, considering the network’s theoretical speed of \(1000\) mbps.

Now, let’s try scp:

# Transfer the file over via `scp`
edge12 % time scp file.bin edge13:/dev/null

I see it takes \(15.30\)s to transfer \(1\) GiB between the two machines with scp (average over \(5\) runs). That comes up to \(561.43\) mbps, which is worse than netcat. That’s not unexpected, though, considering that scp runs over ssh, which must add quite a bit of overhead.

Interestingly enough, netcat doesn’t seem to be as fast as it can be. Running a pair of very simple programs, I see that \(1\) GiB of data could be transferred in \(11.95\)s. That comes up to \(718.82\) mbps. The size of the receiver’s buffer seems to play a big role in determining the speed of the transfer. Anything smaller than \(1\) KiB, or larger than \(64\) KiB increases the transfer time by full seconds.

The moral of this story is this: when estimating how long a particular file transfer will take, you should guess that it will be at least twice as slow as the network speed would allow.