I don’t know how I never noticed this, but the mechanical hard disks in the perfect NAS emit a clicking noise at regular intervals that never stops. The NAS runs Ubuntu 18.04 server, an md software RAID 6 with dmcrypt on top, formatted with ext4.
Disk I/O is way too low to show anything useful on iotop, but btrace was more helpful:
# btrace /dev/md127
9,127 1 0 0.000000000 0 m N md bitmap_daemon_work
9,127 1 0 5.119993735 0 m N md bitmap_daemon_work
9,127 2 0 10.239998722 0 m N md bitmap_daemon_work
This keeps repeating every 5 seconds or so. Good, so I’m not crazy, some process indeed keeps updating the RAID.
Since the array is hidden behind dmcrypt, let’s move a layer up and monitor the dmcrypt device:
# btrace /dev/mapper/data
253,0 0 1 0.000000000 3282 Q WSM 49464 + 8 [kmmpd-dm-0]
253,0 0 2 0.318423986 426 C WSM 49464 + 8 [0]
There isn’t awfully lot written about kmmpd, it seems to have something to do with ext4’s multiple mount protection. The kmmpd daemon writes a heartbeat at regular intervals to the filesystem which causes the update cascade. The good news is that this seems to be a configurable flag that can be toggled with tune2fs. I chickened out of disabling it completely, but maxing out the update interval seems to work nicely (you may want to replace /dev/mapper/data with the device hosting your ext4 filesystem):
tune2fs -E mmp_update_interval=299 /dev/mapper/data
Disabling mmp entirely:
tune2fs -O ^mmp /dev/mapper/data
IMPORTANT: a rather severe side effect is that fsck will wait till the current interval expires, which worst case means half an hour delay before fsck starts working.