On a rather hefty rsync job between two local ext4 filesystems, writing on the target (a mechanical hard drive which should be able to sustain 100 mb/s) would fluctuate strongly (between 0 mb/s and 160 mb/s) and effectively slow down to a few mb/s on average after a few minutes.
While I didn’t find the root cause, two setting (in combination) seem to fix the issue.
Filesystem pre-allocation
This one is easy: rsync should be called with the –preallocate parameter which reduces file fragmentation and reduces hard drive seeks.
Ext4 commit interval
This one requires mounting the write target ext4 filesystem with the commit option [1]. I started getting good results with 60 seconds – values below that would lead to a measurable reduction in throughput.
References
[1] Ext4 documentation
https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
George's Techblog 