dd Cheat Sheet

dd Cheat Sheet

Tags :

Category : Cheat Sheets

Get Social!

dd is one of the most versatile IO tools available for Linux. It’s used in a variety of ways ranging from Disk Benchmarking through to creating SWAP files and copying downloaded disk images to physical disks.

dd takes the following common switches:

  • if is the input file name and location.
  • of is the name and location of the output file.
  • bs is the block size that will be used to read and/ or write the file. Increasing this can help with performance  or dictate how much data will be read or written.
  • count is the number of blocks that will be used.
  • seek is the number of blocks on the output file that will be skipped before writing any data.
  • skip is the number of blocks that will be skipped on the input file before starting to read data.
  • conv is a comma separated list of additional parameters that can be used. See the man dd for more information.

The below headings will list a few example uses of dd in a typical Linux environment.

Backup disk partition with dd

You can use dd to copy an entire disk partition to a virtual disk file. This can be useful for creating a backup or to clone the disk to another machine.

dd if=/dev/sda1 of=~/localdisk_sda1.img

You can use this method to read a CD-ROM, USB drive or Flash disk to a file in the same way – just make sure the device is inserted and point the if= part of the dd command to the relevant /dev/ device.

You could also compress the image as part of the process with gzip.

dd if=/dev/sda1 | gzip -c > ~/localdisk_sda1.img.gz

Restore disk partition with dd

Similar to the above command, you can use dd to replace a disk’s partition with a virtual disk file.

dd if=~/localdisk_sda1.img of=/dev/sda1

If you compressed the image then you can decompress it first all in one go:

gunzip -c ~/localdisk_sda1.img.gz | dd of=/dev/sda1

Create a fixed size file with dd

You can create a fixed size file with DD that will be created in the location you specify.

dd if=/dev/zero of=/root/test bs=1024 count=1

This will create a file in /root/test of 1024 bytes in size. Increase either bs or count to change the size of the file. The resulting size will be bs * count. You can also use shorhand sizes such as K, M and G with bs, for example bs=1G,

dd if=/dev/zero of=upload_test bs=file_size count=1

Create a SWAP file with dd

dd can be used to create a SWAP file that can be used as a SWAP device by your computer. This is often needed with smaller instances on Cloud providers such as AWS.

The starting point is the same as the above command to create a file with the size that you’d like to use for swap. See my other blog post for more info.

Split a file with dd

dd can be used to read just part of a file, given offset and length coordinates. The below example will skip the first 100 bytes of the file and output the proceeding 10 bytes (byte 101 – 111).

dd if=filetosplit of=partfile bs=1 count=10 skip=100

You could repeat this process to split a large file into multiple smaller files, to be able to email it for example.

dd if=filetosplit of=partfile1 bs=1 count=100
dd if=filetosplit of=partfile2 bs=1 count=100 skip=100
dd if=filetosplit of=partfile3 bs=1 count=100 skip=200

Merge multiple files with dd

You can merge multiple files into a single file with dd. Following on from the above split example, the below will rejoin the 3 file parts into a single file.

dd if=partfile1 of=joinedfile bs=1 count=100
dd if=partfile2 of=joinedfile bs=1 count=100 seek=100
dd if=partfile3 of=joinedfile bs=1 count=100 seek=200

Convert text to lower case with dd

You can use the conv switch with dd to transform ascii text from upper case to lower case and visa-versa. Using lcase and ucase in the conv switch will instruct dd to convert the text as it’s written.

The below example will convert all characters in the filetoconvert.txt. file to lower case.

dd if=filetoconvert.txt of=convertedfile.txt conv=lcase

 

Linux User Management Cheat Sheet

Tags :

Category : Cheat Sheets

Get Social!

This is my cheat sheet on Linux user administration covering functions such as adding, and removing users and assigning them to groups.

Add a new Linux user

Use the useradd command to add a new user.

useradd [USERNAME]

Change a users password

The user account is locked until you set a password with the passwd command.

passwd [USERNAME]

Add user to group – new user

If you are adding a new user, you can add it to a group in the same command. This command will create a user of [USERNAME] in group [GROUPNAME].

useradd -G [GROUPNAME] [USERNAME]

Add user to group – existing user

If the user already exists, you can add it to an existing group with the usermod command.

usermod -a -G [GROUPNAME] [USERNAME]

Delete a user

Run the userdel command to remove an existing Linux user.

userdel [USERNAME]

View existing users and groups

Run the below cat command to view existing Linux users. You sill see the user names and user IDs of all users on your server.

cat /etc/passwd | sort

Use this cat command to view the existing Linux groups.

cat /etc/group | sort

Change a users home directory location

You can change the users home directory with the usermod command.

usermod -d -m [NEW_DIRECTORY] [USERNAME]

Change a users UID

Use the usermod -u command to change the user ID of a user.

usermod -u [UID] [USER_NAME]

 

OpenSSL Certificate Cheat Sheet

Tags :

Category : Cheat Sheets

Get Social!

openssl-logoThese commands cover the basics of OpenSSL and are valid for either Windows or Linux with the exception that paths may need to be corrected for the respective platform.

Install OpenSSL

For windows http://www.openssl.org/related/binaries.html

For Ubuntu

sudo apt-get install openssl

Create Private Key

The last argument in the below line is the key length. This can be changed to 2048 or 4096 if required for better encryption.

openssl genrsa -des3 -out server.key 1024

Generate a CSR (Certificate Signing Request)

You will be asked for the details of the certificate such as domain name and address when running this command.

openssl req -new -key server.key -out server.csr

Remove Passphrase from Key

Some applications do not allow for the private key to have a passphrase. The below commands will remove the passphrase – be careful as it will mean the key is no longer protected and can be viewed by anyone with read access to the file.

openssl rsa -in server-with-passphrase.key -out server.key

Generating a Self-Signed Certificate

Once you have generated a key and CSR you will need to sign the request and generate the public certificate. If you do not have a certificate authority you can sign the certificate yourself. The below will generate a certificate which is valid for one year.

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Convert x509 to pem

openssl x509 -inform der -in server.crt -out server.pem

pkcs12 to pem – key only

Use the below command to extract only the key from a pkcs12 certificate.

openssl pkcs12 -nocerts -in c:\server.pfx -out c:\server-key.key

pkcs12 to pem – certificate only

Use the below command to extract only the public certificate from a pkcs12 certificate.

openssl pkcs12 -nokeys -in server.pfx -out server-cert.cer

Check a private key

You can check a private key with the below command.

openssl rsa -in privateKey.key -check

Check a certificate

Use the below command to check a certificate.

openssl x509 -in certificate.crt -text -noout

 

update-rc.d Cheat Sheet

Tags :

Category : Cheat Sheets

Get Social!

Linux penguinDebian and Ubuntu use the service command to control services and update-rc.d for adding and removing services from start up. Using the service command we can start, stop, restart and display all available services. With update-rc.d we can add and remove services and add them to the Ubuntu/ Debian start up scripts. As Linux operating systems have multiple states, or runlevels, you need to make sure you add any new services to the correct runlevels. For example, you would not want to start a web service application before starting networking.

See my post on runlevels for more information about runlevels in Linux..

What services are available for startup?

Use the status-all switch to list all services which are registered with the OS and issues them a status command. You will then get one of the following displayed next to each service:

  • [ + ] – Services with this sign are currently running.
  • [ – ] – Services with this sign are not currently running..
  • [ ? ] – Services that do not have a status switch.
service --status-all

Sample output:

service --status-all
 [ ? ] acpid
 [ - ] apparmor
 [ ? ] apport
 [ ? ] atd
 [ - ] bootlogd
 [ ? ] console-setup
 [ ? ] cron
 [ ? ] dbus
 [ ? ] dmesg
 [ ? ] dns-clean
 [ ? ] friendly-recovery
 [ ? ] glusterfs-server
 [ - ] grub-common
 [ ? ] gssd
 [ ? ] hostname
 [ ? ] hwclock
 [ ? ] hwclock-save
 [ ? ] idmapd
 [ ? ] irqbalance
 [ ? ] killprocs
 [ ? ] module-init-tools
 [ ? ] network-interface
 [ ? ] network-interface-container
 [ ? ] network-interface-security
 [ ? ] networking
 [ ? ] ondemand
 [ ? ] passwd
 [ ? ] plymouth
 [ ? ] plymouth-log
 [ ? ] plymouth-ready
 [ ? ] plymouth-splash
 [ ? ] plymouth-stop
 [ ? ] plymouth-upstart-bridge
 [ ? ] portmap
 [ ? ] portmap-wait
 [ ? ] pppd-dns
 [ ? ] procps
 [ ? ] rc.local
 [ ? ] resolvconf
 [ ? ] rpcbind-boot
 [ - ] rsync
 [ ? ] rsyslog
 [ ? ] screen-cleanup
 [ ? ] sendsigs
 [ ? ] setvtrgb
 [ + ] ssh
 [ ? ] statd
 [ ? ] statd-mounting
 [ - ] stop-bootlogd
 [ - ] stop-bootlogd-single
 [ ? ] sudo
 [ ? ] udev
 [ ? ] udev-fallback-graphics
 [ ? ] udev-finish
 [ ? ] udevmonitor
 [ ? ] udevtrigger
 [ ? ] ufw
 [ ? ] umountfs
 [ ? ] umountnfs.sh
 [ ? ] umountroot
 [ - ] unattended-upgrades
 [ - ] urandom
 [ ? ] whoopsie

Start a service

Starting a service is done using the command service followed by the service name and the command to start the service.

service apache2 start

Stop a service

Use the stop keyword with service to stop a service.

service apache2 stop

Check the status of a service

Each service has a status, usually running or not running. Some services, such as network, may have a different output and output more information on the service.

service apache2 status

Remove a service

Use the remove keyword with update-rc.d to remove the service start up command for an application. You will need to use the -f switch if the applications /etc/init.d start up file exists.

update-rc.d -f apache2 remove

 Add a service

Adding a service to Ubuntu or Debian is done with the update-rc.d command. You can specify which runlevels to start and stop the new service or accept the defaults. The init.d file will be added to the relevent rc.d startup folders.

 update-rc.d apache2 defaults

Setting Start and Kill priority

If you need more control over when your service is asked to start and stop, you may need to set the start and kill (S and K) values.

For a given runlevel, you may have several services starting. For example, you may have apache2 and mysql both starting on runlevel 3. Ideally, you’d want mysql to start before apache2 and shutdown after apache2. In this case we need to give mysql the priority in startup, but apache2 the priority in shutdown.

When starting, the lower the number, the earlier it will start. A start priority of 10 will start before a priority of 20. When killing, it’s the opposite. A higher number will be killed before a lower number.

To set the start and kill priority we simply add them to the above update-rc.d command with the start priority first, followed by the kill priority.

update-rc.d apache2 defaults [START] [KILL]

The below command will start mysql first, then apache2. On shutdown, the kill will be the reverse of the start with apache2 being killed first and mysql second.

update-rc.d apache2 defaults 90 90
update-rc.d mysql defaults 10 10

Because, in our example, both start and kill priorities are the same we can shorted the command to just

update-rc.d apache2 defaults 90
update-rc.d mysql defaults 10

Manually set the RunLevel to Start and Kill a service

You can manually specify the Linux RunLevel that the system must be in to Start and Kill your service. See my other blog post for more information on RunLevels.

To extend the above example, we can specify exactly which RunLevel apache2 will be started and stopped.

update-rc.d apache2 start 10 2 3 4 5 . stop 90 0 1 6 .

apache2 will be started (as long as it isn’t already) when the system enters RunLevel 2, 3, 4 or 5 with a priority of 10. It will then be asked to stop when the system enters RunLevel 0, 1 or 6 with a priority of 90.

Let me know in the comments if you think anything is missing.

logrotate Cheat Sheet

Tags :

Category : Cheat Sheets

Get Social!

logrotate Most Linux applications create log files that need to be managed by either archiving or deleting old log files. This process is called log file rotation. The most common log rotation utility for Linux is conveniently called logrotate. logrotate is configured using the main configuration file, or the logrotate configuration folder. The configuration file usually holds the global configuration and the pointer to the configuration folder. In common Linux distributions such as Ubuntu and Debian, the logrotate configuration file can be found:

/etc/logrotate.conf

And the configuration folder can be found:

/etc/logrotate.d/

The configuration folder is usually where you would add new log file configurations which are unique for a particular log file, or set of log files. For example, if you were to add a new log rotation action for the Ubuntu package manager apt, you may use something like below:

vi /etc/logrotate.d/apt
/var/log/apt/term.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}
/var/log/apt/history.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

logrotate runs each day by default, and is invoked using the daily cron job. The below subject detail the common components of creating a logrotate configuration file.

Create a new empty template

To create a new logrotate configuration file, you need to create a new file in /etc/logrotate.d/. You will then need to add a reference to the log files you wish to rotate. This can be directly to a single file or use pattern matching to match a group of log files. The below example matches all log files in /var/log/myapp/ which have a .log extension.

/var/log/myapp/*.log {

}

You will need to add further commands to this template before it becomes useful. Further commands will be added inside the { and } tags.

Rotate based on log file size

Use the size keyword to rotate the log file when it exceeded a given file size. The below example rotates a file when it reaches 10 KB. 

/var/log/myapp/*.log {
  size 10k
}

Rotate based on time (Month, Week or Day)

You can rotate logs using the monthly, weekly or daily keyword to create a new log based on duration. The keywords explain them selves, and they can be used in conjunction with the size keyword to rotate on which ever criteria is met first.

/var/log/myapp/*.log {
  size 10k
  weekly
}

Limit how many log files are kept after rotation by number

The rotate keyword allows you to specify how many old, rotated, log files are kept before logrotate deletes them. The rotate keyword requires an integer to specify how many old log files are kept.

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
}

The above example will keep 8 old log files.

Limit how many files are kept after rotation by date

You can specify how long to keep rotated files using the maxage keyword. Any rotated log files which are older than maxage will be deleted. The below example will keep rotated log files for 56 days.

/var/log/myapp/*.log {
  size 10k
  weekly
  maxage 56
}

Compress rotated log files

Log files which have been rotated can be compressed to save disk space. Gzip is used by default.

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
  compress
}

You can change the default gzip compression to another format by specifying the compresscmd command and a different executable to use. The below example changes the compression format to bzip2 for better file compression.

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
  compress
  compresscmd /bin/bzip2
}

Ignore missing log files

If a log file does not exist when logrotate is running then an error will be thrown. You can use the keyword missingok to avoid this scenario and instruct logrotate to ignore the log file if it does not exist.

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
  missingok
}

Continue writing to the same log file after rotation

Usually when a log file is rotated the log file is moved to a new location. Some applications may throw an error, and others may continue to write to the relocated file. The copytruncate keyword copies all the log in the file to a new file and then truncates the original file. This keeps the original log file in place and also allows rotation to continue.

/var/log/myapp/*.log {
  size 10k
  weekly
  rotate 8
  copytruncate
}

 

Let me know in the comments if you think anything is missing.

chkconfig Cheat Sheet

Tags :

Category : Cheat Sheets

Get Social!

Linux penguinControlling startup services in Linux distributions such as Red Hat Enterprise Linux (RHEL), CentOS and Oracle Enterprise Linux (OEL) can be done using a Gnome GUI or a command line utility. The command line utility is called chkconfig and can list existing, add new or remove services from the operating systems startup list.

As Linux operating systems have multiple states, or runlevels, you need to make sure you add any new services to the correct runlevels. For example, you would not want to start a web service application before starting networking. See my post on runlevels.

What services are available for startup?

Use the –list switch to see your existing services and when they should be running.

chkconfig --list

An example output is below. This shows all of the machine runlevels and what the state of the service will be.

chkconfig --list

Note: This output shows SysV services only and does not include native systemd services. SysV configuration data might be overridden by native systemd configuration.

modules_dep 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netconsole 0:off 1:off 2:off 3:off 4:off 5:off 6:off
network 0:off 1:off 2:off 3:on 4:off 5:off 6:off

In this example, there are three services available. netconsole is not available at any runlevels and is therefore never started automatically, network is available only in runlevel 3 and modules_dep is available in runlevel 2 and runlevel 3.

You could also use the example below to detail the runlevels of a single service only.

chkconfig network --list

Add a new service with chkconfig

Adding a new service is  easily done with the below command. The below example shows the service network being enabled to start at the next machine boot.

chkconfig network on

Use the –level switch to enable the service at specific runlevels. Use the below example to enable the service at runlevel 3.

chkconfig network on --level 3

Remove a service with chkconfig

Removing an existing service is done with the below command. The below example shows the service network being disabled from automatic start.

chkconfig network off

Use the –level switch to remove the service from specific runlevels. Use the below example to disable the service at runlevel 3.

chkconfig network off --level 3

Start a service

Starting a service is done using the command service followed by the service name and the command to start the service.

service network start

Stop a service

Use the stop keyword with service to stop a service.

service network stop

Check the status of a service

Each service has a status, usually running or not running. Some services, such as network, may have a different output and output more information on the service.

service network status

 

Let me know in the comments if you think anything is missing.

Next Page »

Visit our advertisers

Search

Quick Poll

Which type of virtualisation do you use?
  • Add your answer
Results

Recent Posts

Visit our advertisers

Site Links
Blog
Rated Sites

JamesCoyle.net Limited