OpenVZ disk space issues

I started to receive an error in one of my Proxmox OpenVZ containers which indicated that the quota had been exceeded for disk space. Immediately I logged into the container and performed a df -h to view the disk usage. To my surprise only 50% of the disk space had been used.

Error in the logs:

kernel: VZ QUOTA: file softlimit exceeded for id=103

OpenVZ containers have two storage related limits;

• Disk space – this is the amount of space which can be used by the container, usually measured in gigabytes.
• inodes – this loosely translates to the number of files which can be stored on the filesystem.

Below is an example of the disk based limits configuration.

DISKSPACE="10485760:11534336"
DISKINODES="2000000:2200000"

The problem with my container was that although I wasn’t using all the allocated disk space, I was using all of the allocated inodes as I had a java application running which created 1000s of new small files every hour. CCTV recording software such as Zoneminder would likely have the same problem as it creates many small image files when recoding movement from a camera.

You can test how many inodes are in use by running df -i on the container.  An example below of a very low inode usage:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 10000000 116024 9883976 2% /

Increase disk space and inode quota

Proxmox does not allow you to change the inodes limits directly from the GUI. You will need to edit the configuration file for the container directly on the OS or use the vz command. The config files are stored in /etc/vz/conf by default and named VMID.conf. Edit the required config file for your container and increase the DISKSPACE and DISKINODES values as required.

vi /etc/vz/conf/103.conf

DISKSPACE="10485760:11534336"
DISKINODES="4000000:4400000"

OpenVZ has the concept of HARD and SOFT limits. A soft limit will trigger a warning but allow the applications to continue to use disk space/ inodes however when the hard limit is hit new requests for storage will be denied. A hard limit is like a physical disk being full.

DISKSPACE="SOFT:HARD"
DISKINODES="SOFT:HARD"

You can also use the word UNLIMITED to remove any virtual limits on storage however you will still be constrained by the underlying physical storage limits.

You will need to restart your container for the settings to take effect.

To avoid having to restart your container, you can use the vzctl command to apply the new limits immediately, even to a running container.

vzctl set 103 --diskspace 45G:49G --save

vzctl set 103 --diskinodes 4000000:4400000 --save