OpenVZ disk space issues

OpenVZ disk space issues

Get Social!

proxmox logo gradI started to receive an error in one of my Proxmox OpenVZ containers which indicated that the quota had been exceeded for disk space. Immediately I logged into the container and performed a df -h to view the disk usage. To my surprise only 50% of the disk space had been used.

Error in the logs:

kernel: VZ QUOTA: file softlimit exceeded for id=103

OpenVZ containers have two storage related limits;

  • Disk space – this is the amount of space which can be used by the container, usually measured in gigabytes.
  • inodes – this loosely translates to the number of files which can be stored on the filesystem.

Below is an example of the disk based limits configuration.

DISKSPACE="10485760:11534336"
DISKINODES="2000000:2200000"

The problem with my container was that although I wasn’t using all the allocated disk space, I was using all of the allocated inodes as I had a java application running which created 1000s of new small files every hour. CCTV recording software such as Zoneminder would likely have the same problem as it creates many small image files when recoding movement from a camera.

You can test how many inodes are in use by running df -i on the container.  An example below of a very low inode usage:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 10000000 116024 9883976 2% /

Increase disk space and inode quota

Proxmox does not allow you to change the inodes limits directly from the GUI. You will need to edit the configuration file for the container directly on the OS or use the vz command. The config files are stored in /etc/vz/conf by default and named VMID.conf. Edit the required config file for your container and increase the DISKSPACE and DISKINODES values as required.

vi /etc/vz/conf/103.conf
DISKSPACE="10485760:11534336"
DISKINODES="4000000:4400000"

OpenVZ has the concept of HARD and SOFT limits. A soft limit will trigger a warning but allow the applications to continue to use disk space/ inodes however when the hard limit is hit new requests for storage will be denied. A hard limit is like a physical disk being full.

DISKSPACE="SOFT:HARD"
DISKINODES="SOFT:HARD"

You can also use the word UNLIMITED to remove any virtual limits on storage however you will still be constrained by the underlying physical storage limits.

You will need to restart your container for the settings to take effect.

To avoid having to restart your container, you can use the vzctl command to apply the new limits immediately, even to a running container.

vzctl set 103 --diskspace 45G:49G --save
vzctl set 103 --diskinodes 4000000:4400000 --save

Backup all Proxmox OpenVZ containers in one go

Category : How-to

Get Social!

proxmox logo gradThe below script is a bash script which works with Proxmox and the OpenVZ commands to backup all known containers to a specified folder.

You will need to set BACKUP_PATH to be the folder where you would like the backups to be stored and COMPRESS to a value which specifies the compression used. COMPRESS values can be:

  • 0 – no compression.
  • 1 – default compression (usually lzo).
  • gzip – gzip compression.
  • lzo – lzo compression.

Paste the below file into /bin/backup_all and make sure it’s executable.

vi /bin/backup-all
#!/bin/bash
#
# Filename : backup_all
# Description : Backup all OpenVZ containers in Proxmox
# Author : James Coyle
#
# Version:
# -Date      -Author     -Description
# 20-11-2013 James Coyle Initial
#
#

BACKUP_PATH=/var/lib/vz/dump
COMPRESS="lzo"

# Check dir exists
if [ ! -d $BACKUP_PATH ]; then
  echo "The directory $BACKUP_PATH does not exist."
  exit 99
fi

IFS=$'\n'
VMARRAY=($(vzlist -a -H))
VMIDS=""

for V in ${VMARRAY[@]}
do
  VMIDS="$VMIDS ${V:7:3}"
done

if [ -n $VMIDS ]; then
  vzdump $VMIDS --dumpdir $BACKUP_PATH --mode snapshot --compress $COMPRESS --remove 0
fi

echo "Backup of VMID(s) $VMIDS complete."

Updates to Proxmox 3.1 including signed console applet

Category : Tech News

Get Social!

proxmox logo grad

New updates to Proxmox 3.1 have been released which can be applied using apt-get.

The main feature in this suite of updates is that the Java VNC console is now signed which should result in far fewer annoying messages when starting the console. There are a few other updates, including bug fixes and support for iSCSI targets (istgt) on ZFS.

Run the update and upgrade command to apply the recent updates. There is no need to shut down any running guests or restart your host with this update.

Run apt-get update to download the information on the latest available updates.

apt-get update

Run apt-get upgrade to download and install the latest update packages. Press y when prompted to download and install the displayed packages.

apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
 libpve-access-control libpve-common-perl libpve-storage-perl pve-manager vncterm
5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,634 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
Do you want to continue [Y/n]? y

Restore a single Proxmox OpenVZ Container From The command Line

Get Social!

proxmox logo gradI mostly use Proxmox from the command line, or terminal, and I have created a few scripts to perform common and repetitive tasks.

The below script will restore a single OpenVZ container to the latest backup file available in the dump directory. The scripts takes a parameter for the container VMID to restore from backup. If the container exists, it will be stopped and removed before restoring the latest backup file available in the backup directory.

The script iterates through all of your backup files and only restores the latest based on the date in the file name, and not the date of the file creation or modified.

You will need to set the BACKUP_PATH variable to the location of your backup folder with no trailing slash, and BACKUP_EXT with the extension used for your chosen backup format.

If you save this script in the /bin then you can call the script from the terminal without having to move to the scripts directory. Create the file and paste the below script into it.

vi /bin/restore_one
#!/bin/bash
#
# Filename : restore_one
# Description : Restores a single OpenVZ Proxmox container to the latest backup file
#               available in the dump folder.
# Author : James Coyle
#
# Version:
# -Date -Author -Description
# 01-11-2013 James Coyle Initial
#
#

BACKUP_PATH=/var/lib/vz/dump
BACKUP_EXT=tar.lzo

# Do not change
SEARCH_PATH=$BACKUP_PATH/vzdump-openvz-$1-*.$BACKUP_EXT

function display-useage
{
  echo "Useage $0 [vmid to restore]"
  echo "Example: $0 999"

}

# Check dir exists
if [ ! -d $BACKUP_PATH ]; then
  echo "The directory $BACKUP_PATH does not exist"
  exit 99
fi

# Check if argument is present
if [ -z "$1" ]
then
  echo "Argument not present."
  display-useage
  exit 99
fi

# Check if vmid is available, on or off
VMON=$(vzlist | grep -P "[ ]+$1[ ]+")
VMOFF=$(vzlist --stopped | grep -P "[ ]+$1[ ]+")

if [ -n "$VMON" ]; then
  echo "Requesting stop of container."
  vzctl stop $1
  echo "Requesting deletion of container."
  vzctl delete $1
elif [ -n "$VMOFF" ]; then
  echo "Container is stopped."
  echo "Requesting deletion of container."
  vzctl delete $1
else
  echo "Container is not live."
fi

# Get unique VMIDs
for F in $SEARCH_PATH
do
  FILENAME=${F##*/}
  FILE_DATE=${FILENAME:18:19}
  FILE_DATE=${FILE_DATE//[_\-]/}

  if [ -z "$BACKUP_FILE" ]; then
    BACKUP_FILE=$F
  fi

  TEST_FILENAME=${BACKUP_FILE##*/}
  TEST_FILE_DATE=${TEST_FILENAME:18:19}
  TEST_FILE_DATE=${TEST_FILE_DATE//[_\-]/}
  if [ "$FILE_DATE" -gt "$TEST_FILE_DATE" ]; then
    BACKUP_FILE=$F
  fi
done

if [ -n $BACKUP_FILE ]; then
  # Restore VM
  echo "Restoring $1 with $BACKUP_FILE..."
  vzrestore $BACKUP_FILE $1
else
  echo "No backup file for VMID $1 exists."
fi

Make the script executable using chmod.

chmod +x /bin/restore_one

Use the below command, and substitute [VMID] with the container VMID to restore, to run the script.

restore_one [VMID]

See my other script on restoring multiple OpenVZ containers in Proxmox.


Restore all Proxmox OpenVZ Containers From The command Line

Get Social!

proxmox logo gradI use Proxmox to host a development environment using OpenVZ containers. I take frequent backups of all OpenVZ containers in the event I need to roll back any development work.

The below script restores all OpenVZ containers which are available in the backup folder, but not available in the Proxmox GUI. Using this script, you can remove the containers in Proxmox which you would like to restore and run the script to restore the latest backup.

The script iterates through all of your backup files and only restores the latest based on the date in the file name.

You will need to set the BACKUP_PATH variable to the location of your backup folder with no trailing slash, and BACKUP_EXT with the extension used for your chosen backup format.

If you save this script in the /bin then you can call the script from the terminal without having to move to the scripts directory. Create the file and paste the below script into it.

vi /bin/restore_all
#!/bin/bash
#
# Filename : restore_all
# Description : Restores all missing OpenVZ containers in Proxmox to the latest version available in the dump folder.
# Author : James Coyle
#
# Version:
# -Date       -Author      -Description
# 01-11-2013  James Coyle  Initial
#
#

BACKUP_PATH=/var/lib/vz/dump
BACKUP_EXT=tar.lzo

# Do not change
SEARCH_PATH=$BACKUP_PATH/vzdump-openvz-*.$BACKUP_EXT

# Check dir exists
if [ ! -d $BACKUP_PATH ]; then
  echo "The directory $BACKUP_PATH does not exist"
  exit 99
fi

# Get unique VMIDs
for F in $SEARCH_PATH
do
  FILENAME=${F##*/}
  FILE_DATE=${FILENAME:18:19}
  FILE_DATE=${FILE_DATE//[_\-]/}
  VMID=${FILENAME:14:3}
  if [ -n $FILE_VIMS[$VMID] ]; then
    FILE_VIMS[$VMID]=$F
  fi

  TEST_FILENAME=${FILE_VIMS[$VMID]##*/}
  TEST_FILE_DATE=${TEST_FILENAME:18:19}
  TEST_FILE_DATE=${TEST_FILE_DATE//[_\-]/}
  if [ "$FILE_DATE" -gt "$TEST_FILE_DATE" ]; then
    FILE_VIMS[$VMID]=F
  fi
done

# Restore VM
for I in ${!FILE_VIMS[*]}
do
  echo "Restoring $I with ${FILE_VIMS[$I]}..."
  vzrestore ${FILE_VIMS[$I]} $I
done

Make the script executable using chmod.

chmod +x /bin/restore_all

Use the below command to run the script and restore all containers which are missing from backup.

restore_all

Mount NFS volume in a Proxmox OpenVZ container

Category : How-to

Get Social!

openvz-logo-150px_new_3There are various options for adding additional storage to an OpenVZ container. You can add additional storage to the containers root volume to simply increase the overall storage available to the container. For external storage, or storage on another disk to the root partition of the container there are bind mounts.

With some light work you can also use NFS mounts inside an OpenVZ container. Before NFS will work in a container a command needs to be ran on the host to enable NFS features in the container.

If you do not enable NFS on the container, you will get the following error:

mount: unknown filesystem type 'nfs'

Open a Terminal on the host machine and run the below command to check that the modules are loaded in the kernel:

modprobe nfs

Then run the below command to enable NFS on the container. Make sure container is turned off or restart the container after issuing the command.

vzctl set 998 --features "nfs:on" --save

This writes a change to the containers config file. To avoid using the command, you could simply edit the config file directly and add the below text to the bottom of the file:

FEATURES="nfs:on"

Start the container and make sure that the required packages are installed.

apt-get install nfs-common

If you do not have the required packages installed you may receive the following error

mount: wrong fs type, bad option, bad superblock on 192.168.50.252:/dspool/compressed,
 missing codepage or helper program, or other error
 (for several filesystems (e.g. nfs, cifs) you might
 need a /sbin/mount.<type> helper program)
 In some cases useful info is found in syslog - try
 dmesg | tail or so

Finally, run the mount command to mount your NFS directory.

mount -t nfs 10.10.10.5:/storage/compressed /mnt/testmount

Visit our advertisers

Quick Poll

What type of VPN protocol do you use?

Visit our advertisers