dd Cheat Sheet

dd Cheat Sheet

Tags :

Category : Cheat Sheets

Get Social!

dd is one of the most versatile IO tools available for Linux. It’s used in a variety of ways ranging from Disk Benchmarking through to creating SWAP files and copying downloaded disk images to physical disks.

dd takes the following common switches:

  • if is the input file name and location.
  • of is the name and location of the output file.
  • bs is the block size that will be used to read and/ or write the file. Increasing this can help with performance  or dictate how much data will be read or written.
  • count is the number of blocks that will be used.
  • seek is the number of blocks on the output file that will be skipped before writing any data.
  • skip is the number of blocks that will be skipped on the input file before starting to read data.
  • conv is a comma separated list of additional parameters that can be used. See the man dd for more information.

The below headings will list a few example uses of dd in a typical Linux environment.

Backup disk partition with dd

You can use dd to copy an entire disk partition to a virtual disk file. This can be useful for creating a backup or to clone the disk to another machine.

dd if=/dev/sda1 of=~/localdisk_sda1.img

You can use this method to read a CD-ROM, USB drive or Flash disk to a file in the same way – just make sure the device is inserted and point the if= part of the dd command to the relevant /dev/ device.

You could also compress the image as part of the process with gzip.

dd if=/dev/sda1 | gzip -c > ~/localdisk_sda1.img.gz

Restore disk partition with dd

Similar to the above command, you can use dd to replace a disk’s partition with a virtual disk file.

dd if=~/localdisk_sda1.img of=/dev/sda1

If you compressed the image then you can decompress it first all in one go:

gunzip -c ~/localdisk_sda1.img.gz | dd of=/dev/sda1

Create a fixed size file with dd

You can create a fixed size file with DD that will be created in the location you specify.

dd if=/dev/zero of=/root/test bs=1024 count=1

This will create a file in /root/test of 1024 bytes in size. Increase either bs or count to change the size of the file. The resulting size will be bs * count. You can also use shorhand sizes such as K, M and G with bs, for example bs=1G,

dd if=/dev/zero of=upload_test bs=file_size count=1

Create a SWAP file with dd

dd can be used to create a SWAP file that can be used as a SWAP device by your computer. This is often needed with smaller instances on Cloud providers such as AWS.

The starting point is the same as the above command to create a file with the size that you’d like to use for swap. See my other blog post for more info.

Split a file with dd

dd can be used to read just part of a file, given offset and length coordinates. The below example will skip the first 100 bytes of the file and output the proceeding 10 bytes (byte 101 – 111).

dd if=filetosplit of=partfile bs=1 count=10 skip=100

You could repeat this process to split a large file into multiple smaller files, to be able to email it for example.

dd if=filetosplit of=partfile1 bs=1 count=100
dd if=filetosplit of=partfile2 bs=1 count=100 skip=100
dd if=filetosplit of=partfile3 bs=1 count=100 skip=200

Merge multiple files with dd

You can merge multiple files into a single file with dd. Following on from the above split example, the below will rejoin the 3 file parts into a single file.

dd if=partfile1 of=joinedfile bs=1 count=100
dd if=partfile2 of=joinedfile bs=1 count=100 seek=100
dd if=partfile3 of=joinedfile bs=1 count=100 seek=200

Convert text to lower case with dd

You can use the conv switch with dd to transform ascii text from upper case to lower case and visa-versa. Using lcase and ucase in the conv switch will instruct dd to convert the text as it’s written.

The below example will convert all characters in the filetoconvert.txt. file to lower case.

dd if=filetoconvert.txt of=convertedfile.txt conv=lcase

 

Restore a single Proxmox OpenVZ Container From The command Line

Tags :

Category : How-to

Get Social!

proxmox logo gradI mostly use Proxmox from the command line, or terminal, and I have created a few scripts to perform common and repetitive tasks.

The below script will restore a single OpenVZ container to the latest backup file available in the dump directory. The scripts takes a parameter for the container VMID to restore from backup. If the container exists, it will be stopped and removed before restoring the latest backup file available in the backup directory.

The script iterates through all of your backup files and only restores the latest based on the date in the file name, and not the date of the file creation or modified.

You will need to set the BACKUP_PATH variable to the location of your backup folder with no trailing slash, and BACKUP_EXT with the extension used for your chosen backup format.

If you save this script in the /bin then you can call the script from the terminal without having to move to the scripts directory. Create the file and paste the below script into it.

vi /bin/restore_one
#!/bin/bash
#
# Filename : restore_one
# Description : Restores a single OpenVZ Proxmox container to the latest backup file
#               available in the dump folder.
# Author : James Coyle
#
# Version:
# -Date -Author -Description
# 01-11-2013 James Coyle Initial
#
#

BACKUP_PATH=/var/lib/vz/dump
BACKUP_EXT=tar.lzo

# Do not change
SEARCH_PATH=$BACKUP_PATH/vzdump-openvz-$1-*.$BACKUP_EXT

function display-useage
{
  echo "Useage $0 [vmid to restore]"
  echo "Example: $0 999"

}

# Check dir exists
if [ ! -d $BACKUP_PATH ]; then
  echo "The directory $BACKUP_PATH does not exist"
  exit 99
fi

# Check if argument is present
if [ -z "$1" ]
then
  echo "Argument not present."
  display-useage
  exit 99
fi

# Check if vmid is available, on or off
VMON=$(vzlist | grep -P "[ ]+$1[ ]+")
VMOFF=$(vzlist --stopped | grep -P "[ ]+$1[ ]+")

if [ -n "$VMON" ]; then
  echo "Requesting stop of container."
  vzctl stop $1
  echo "Requesting deletion of container."
  vzctl delete $1
elif [ -n "$VMOFF" ]; then
  echo "Container is stopped."
  echo "Requesting deletion of container."
  vzctl delete $1
else
  echo "Container is not live."
fi

# Get unique VMIDs
for F in $SEARCH_PATH
do
  FILENAME=${F##*/}
  FILE_DATE=${FILENAME:18:19}
  FILE_DATE=${FILE_DATE//[_\-]/}

  if [ -z "$BACKUP_FILE" ]; then
    BACKUP_FILE=$F
  fi

  TEST_FILENAME=${BACKUP_FILE##*/}
  TEST_FILE_DATE=${TEST_FILENAME:18:19}
  TEST_FILE_DATE=${TEST_FILE_DATE//[_\-]/}
  if [ "$FILE_DATE" -gt "$TEST_FILE_DATE" ]; then
    BACKUP_FILE=$F
  fi
done

if [ -n $BACKUP_FILE ]; then
  # Restore VM
  echo "Restoring $1 with $BACKUP_FILE..."
  vzrestore $BACKUP_FILE $1
else
  echo "No backup file for VMID $1 exists."
fi

Make the script executable using chmod.

chmod +x /bin/restore_one

Use the below command, and substitute [VMID] with the container VMID to restore, to run the script.

restore_one [VMID]

See my other script on restoring multiple OpenVZ containers in Proxmox.

Restore all Proxmox OpenVZ Containers From The command Line

Tags :

Category : How-to

Get Social!

proxmox logo gradI use Proxmox to host a development environment using OpenVZ containers. I take frequent backups of all OpenVZ containers in the event I need to roll back any development work.

The below script restores all OpenVZ containers which are available in the backup folder, but not available in the Proxmox GUI. Using this script, you can remove the containers in Proxmox which you would like to restore and run the script to restore the latest backup.

The script iterates through all of your backup files and only restores the latest based on the date in the file name.

You will need to set the BACKUP_PATH variable to the location of your backup folder with no trailing slash, and BACKUP_EXT with the extension used for your chosen backup format.

If you save this script in the /bin then you can call the script from the terminal without having to move to the scripts directory. Create the file and paste the below script into it.

vi /bin/restore_all
#!/bin/bash
#
# Filename : restore_all
# Description : Restores all missing OpenVZ containers in Proxmox to the latest version available in the dump folder.
# Author : James Coyle
#
# Version:
# -Date       -Author      -Description
# 01-11-2013  James Coyle  Initial
#
#

BACKUP_PATH=/var/lib/vz/dump
BACKUP_EXT=tar.lzo

# Do not change
SEARCH_PATH=$BACKUP_PATH/vzdump-openvz-*.$BACKUP_EXT

# Check dir exists
if [ ! -d $BACKUP_PATH ]; then
  echo "The directory $BACKUP_PATH does not exist"
  exit 99
fi

# Get unique VMIDs
for F in $SEARCH_PATH
do
  FILENAME=${F##*/}
  FILE_DATE=${FILENAME:18:19}
  FILE_DATE=${FILE_DATE//[_\-]/}
  VMID=${FILENAME:14:3}
  if [ -n $FILE_VIMS[$VMID] ]; then
    FILE_VIMS[$VMID]=$F
  fi

  TEST_FILENAME=${FILE_VIMS[$VMID]##*/}
  TEST_FILE_DATE=${TEST_FILENAME:18:19}
  TEST_FILE_DATE=${TEST_FILE_DATE//[_\-]/}
  if [ "$FILE_DATE" -gt "$TEST_FILE_DATE" ]; then
    FILE_VIMS[$VMID]=F
  fi
done

# Restore VM
for I in ${!FILE_VIMS[*]}
do
  echo "Restoring $I with ${FILE_VIMS[$I]}..."
  vzrestore ${FILE_VIMS[$I]} $I
done

Make the script executable using chmod.

chmod +x /bin/restore_all

Use the below command to run the script and restore all containers which are missing from backup.

restore_all

Visit our advertisers

Search

Quick Poll

Which type of virtualisation do you use?
  • Add your answer
Results

Recent Posts

Visit our advertisers

Site Links
Blog
Rated Sites

JamesCoyle.net Limited