Today one of the Nutanix Customer called me saying their Nutanix Cluster was down. Upon checking noticed all cluster services were down. When tried to start the cluster Genesis logs pointed to insufficient disk space. On further inspection Linux File system on all CVM was at 100% usage for inodes.

nutanix@NTNX-XXXXXXX-XXXXX-B-CVM:192.168.X.XXX:~$ allssh df -i

================== 192.168.X.XXX =================

Filesystem Inodes IUsed IFree IUse% Mounted on

devtmpfs 4114406 434 4113972 1% /dev

tmpfs 4118255 1 4118254 1% /dev/shm

tmpfs 4118255 4036 4114219 1% /run

tmpfs 4118255 16 4118239 1% /sys/fs/cgroup

/dev/sda2 655360 655360 0 100% /

/dev/loop0 65536 230 65306 1% /tmp

/dev/sda3 2621440 99593 2521847 4% /home

tmpfs 4118255 1 4118254 1% /run/user/0

tmpfs 4118255 1 4118254 1% /run/user/1000

/dev/sdc1 244191232 811486 243379746 1% /home/nutanix/data/stargate-storage/disks/VGJ3VHSG

/dev/sdb1 244191232 813612 243377620 1% /home/nutanix/data/stargate-storage/disks/VGJ3XMZG

/dev/sda4 113278976 267907 113011069 1% /home/nutanix/data/stargate-storage/disks/S455NC0N900742

================== 192.168.X.XXX =================

Filesystem Inodes IUsed IFree IUse% Mounted on

devtmpfs 4114406 434 4113972 1% /dev

tmpfs 4118255 1 4118254 1% /dev/shm

tmpfs 4118255 4036 4114219 1% /run

tmpfs 4118255 16 4118239 1% /sys/fs/cgroup

/dev/sda2 655360 655360 0 100% /

/dev/loop0 65536 256 65280 1% /tmp

/dev/sda3 2621440 199946 2421494 8% /home

tmpfs 4118255 1 4118254 1% /run/user/0

tmpfs 4118255 1 4118254 1% /run/user/1000

/dev/sdc1 244191232 763959 243427273 1% /home/nutanix/data/stargate-storage/disks/VGJ6TR5G

/dev/sda4 113278976 301854 112977122 1% /home/nutanix/data/stargate-storage/disks/S455NC0N900734

/dev/sdb1 244191232 763815 243427417 1% /home/nutanix/data/stargate-storage/disks/VGJ6XASG

================== 192.168.X.XXX =================

Filesystem Inodes IUsed IFree IUse% Mounted on

devtmpfs 4114406 434 4113972 1% /dev

tmpfs 4118255 1 4118254 1% /dev/shm

tmpfs 4118255 4036 4114219 1% /run

tmpfs 4118255 16 4118239 1% /sys/fs/cgroup

/dev/sda2 655360 655360 0 100% /

/dev/loop0 65536 110 65426 1% /tmp

/dev/sda3 2621440 100851 2520589 4% /home

tmpfs 4118255 1 4118254 1% /run/user/0

tmpfs 4118255 1 4118254 1% /run/user/1000

/dev/sda4 113278976 261394 113017582 1% /home/nutanix/data/stargate-storage/disks/S455NA0N906186

/dev/sdc1 244191232 754575 243436657 1% /home/nutanix/data/stargate-storage/disks/VGJ6Y34G

/dev/sdb1 244191232 756298 243434934 1% /home/nutanix/data/stargate-storage/disks/VGJ3BGWG

This issue is already identified by known scenarios on version AOS 6.5.3. where the /var/spool/postfix/maildrop directory fills up the inodes.

Nutanix Tech note : https://portal.nutanix.com/kb/6082 can be referred for the same.

Basically, You have to run the below command will clear-up the files.

nutanix@NTNX-XXXXXXX-XXXXX-B-CVM:192.168.X.XXX:~$ allssh ‘sudo du –inode /var/spool/postfix/maildrop’

Confirm inode usage is Normal

nutanix@NTNX-XXXXXXX-XXXXX-B-CVM:192.168.X.XXX:~$ allssh df -i

================== 192.168.X.XXX =================

Filesystem Inodes IUsed IFree IUse% Mounted on

devtmpfs 4114406 434 4113972 1% /dev

tmpfs 4118255 1 4118254 1% /dev/shm

tmpfs 4118255 596 4117659 1% /run

tmpfs 4118255 16 4118239 1% /sys/fs/cgroup

/dev/sda2 655360 57441 597919 9% /

/dev/loop0 65536 237 65299 1% /tmp

/dev/sda3 2621440 99715 2521725 4% /home

tmpfs 4118255 1 4118254 1% /run/user/0

tmpfs 4118255 1 4118254 1% /run/user/1000

/dev/sdc1 244191232 811483 243379749 1% /home/nutanix/data/stargate-storage/disks/VGJ3VHSG

/dev/sdb1 244191232 813592 243377640 1% /home/nutanix/data/stargate-storage/disks/VGJ3XMZG

/dev/sda4 113278976 267565 113011411 1% /home/nutanix/data/stargate-storage/disks/S455NC0N900742

================== 192.168.X.XXX =================

Filesystem Inodes IUsed IFree IUse% Mounted on

devtmpfs 4114406 434 4113972 1% /dev

tmpfs 4118255 1 4118254 1% /dev/shm

tmpfs 4118255 596 4117659 1% /run

tmpfs 4118255 16 4118239 1% /sys/fs/cgroup

/dev/sda2 655360 57443 597917 9% /

/dev/loop0 65536 267 65269 1% /tmp

/dev/sda3 2621440 200108 2421332 8% /home

tmpfs 4118255 1 4118254 1% /run/user/0

tmpfs 4118255 1 4118254 1% /run/user/1000

/dev/sdc1 244191232 763957 243427275 1% /home/nutanix/data/stargate-storage/disks/VGJ6TR5G

/dev/sda4 113278976 301781 112977195 1% /home/nutanix/data/stargate-storage/disks/S455NC0N900734

/dev/sdb1 244191232 763813 243427419 1% /home/nutanix/data/stargate-storage/disks/VGJ6XASG

================== 192.168.X.XXX =================

Filesystem Inodes IUsed IFree IUse% Mounted on

devtmpfs 4114406 434 4113972 1% /dev

tmpfs 4118255 1 4118254 1% /dev/shm

tmpfs 4118255 596 4117659 1% /run

tmpfs 4118255 16 4118239 1% /sys/fs/cgroup

/dev/sda2 655360 57430 597930 9% /

/dev/loop0 65536 118 65418 1% /tmp

/dev/sda3 2621440 100257 2521183 4% /home

tmpfs 4118255 1 4118254 1% /run/user/0

tmpfs 4118255 1 4118254 1% /run/user/1000

/dev/sda4 113278976 261402 113017574 1% /home/nutanix/data/stargate-storage/disks/S455NA0N906186

/dev/sdc1 244191232 754575 243436657 1% /home/nutanix/data/stargate-storage/disks/VGJ6Y34G

/dev/sdb1 244191232 756298 243434934 1% /home/nutanix/data/stargate-storage/disks/VGJ3BGWG

Restart genesis service on all cluster:

nutanix@NTNX-XXXXXXX-XXXXX-B-CVM:192.168.X.XXX:~$ cluster restart_genesis

Start Cluster:

nutanix@NTNX-XXXXXXX-XXXXX-B-CVM:192.168.X.XXX:~$ cluster start

Once Cluster/inode usage is normal, upgrade the AOS latest release to avoid any future issues.

Hope this helps 😊