Slow rsync on ext4 filesystem

On a rather hefty rsync job between two local ext4 filesystems, writing on the target (a mechanical hard drive which should be able to sustain 100 mb/s) would fluctuate strongly (between 0 mb/s and 160 mb/s) and effectively slow down to a few mb/s on average after a few minutes.

While I didn’t find the root cause, two setting (in combination) seem to fix the issue.

Filesystem pre-allocation

This one is easy: rsync should be called with the –preallocate parameter which reduces file fragmentation and reduces hard drive seeks.

Ext4 commit interval

This one requires mounting the write target ext4 filesystem with the commit option [1]. I started getting good results with 60 seconds – values below that would lead to a measurable reduction in throughput.

References

[1] Ext4 documentation
https://www.kernel.org/doc/Documentation/filesystems/ext4.txt

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.