Constant disk access with ext4 caused by kmmp

I don’t know how I never noticed this, but the mechanical hard disks in the perfect NAS emit a clicking noise at regular intervals that never stops. The NAS runs Ubuntu 18.04 server, an md software RAID 6 with dmcrypt on top, formatted with ext4.

Disk I/O is way too low to show anything useful on iotop, but btrace was more helpful:

# btrace /dev/md127
  9,127  1        0     0.000000000     0  m   N md bitmap_daemon_work
  9,127  1        0     5.119993735     0  m   N md bitmap_daemon_work
  9,127  2        0    10.239998722     0  m   N md bitmap_daemon_work

This keeps repeating every 5 seconds or so. Good, so I’m not crazy, some process indeed keeps updating the RAID.

Since the array is hidden behind dmcrypt, let’s move a layer up and monitor the dmcrypt device:

# btrace /dev/mapper/data
253,0    0        1     0.000000000  3282  Q WSM 49464 + 8 [kmmpd-dm-0]
253,0    0        2     0.318423986   426  C WSM 49464 + 8 [0]

There isn’t awfully lot written about kmmpd, it seems to have something to do with ext4’s multiple mount protection. The kmmpd daemon writes a heartbeat at regular intervals to the filesystem which causes the update cascade. The good news is that this seems to be a configurable flag that can be toggled with tune2fs. I chickened out of disabling it completely, but maxing out the update interval seems to work nicely (you may want to replace /dev/mapper/data with the device hosting your ext4 filesystem):

tune2fs -E mmp_update_interval=299 /dev/mapper/data

Disabling mmp entirely:

tune2fs -O ^mmp /dev/mapper/data

IMPORTANT: a rather severe side effect is that fsck will wait till the current interval expires, which worst case means half an hour delay before fsck starts working.