Set up Fail2ban for Proxmox Web GUI

Set up Fail2ban for Proxmox Web GUI

Get Social!

fail2ban_logoFail2ban is an application that scans log files in real time and bans malicious IP addresses based on a set of rules and filters you can set.

For this blog post, we’re going to look at capturing invalid login attempts to the Proxmox Web GUI and ban any IP addresses from accessing the Web GUI if they fail to authenticate 3 times from the same IP address.

Fail2ban is made up of three main component parts:

  • Filter – a Filter is a pattern or regular expression that we wish to search for in the log files. In our case, we want to search for the words ‘authentication failure’ in the log because that’s what the pvedaemon writes when a failed login attempt occurs.
  • Action – an Action is what we’ll do if the filter is found. What we need to do is ban any IP address where the filter is triggered 3 times.
  • Jail – a Jail in Fail2ban is the glue that holds it all together – this ties a Filter, together with an Action and the relevant log file.

Install Fail2ban

Installing Fail2ban on Debian/ Proxmox is as easy as it gets – just use the apt package manager.

apt-get install fail2ban

Fail2ban is mostly Python, so it’ll need to be installed on the system or apt-get  will install it as a dependency.

Note: by default Fail2ban will enable itself on SSH connections, blocking invalid IPs after 6 invalid attempts. 

Configure Fail2ban for the Proxmox Web GUI

There are several steps to setting up Fail2ban. As mentioned earlier in the post, we want to ban any users IP address from accessing the Proxmox Web GUI if they have failed to authenticate 3 times. We shouldn’t block them indefinitely because it may be a simple password issue that they can resolve with the account administrator. We’ll configure Fail2ban to ban failed attempts for an hour.

Because banning a user after 3 invalid attempts is a fairly basic thing in the world of Fail2ban, we won’t need to create an Action as listed above. We’ll need to create a Jail and a Filter.

The Jail

A Jail in Fail2ban is the core configuration that  combines a Filter, an Action (although this may be default Fail2ban behaviour) and a log file.

The default configuration for Fail2ban is found in /etc/fail2ban/jail.conf and contains many predefined entries for common processes such as FTP and Apache. We shouldn’t edit this file directly when adding new entries, instead, we should create the below file which will be used to override the default jail.conf.

vi /etc/fail2ban/jail.local

Add the following (this file may not already exist):

[proxmox-web-gui]
enabled  = true
port     = http,https,8006
filter   = proxmox-web-gui
logpath  = /var/log/daemon.log
maxretry = 3
bantime = 3600

The above entry has set a ruleset name of proxmox-web-gui, and the following:

  • enabled – this simply states that this ruleset is active.
  • port – set sthe port that any bans should act on
  • filter – this sets the file name of the filter that we’ll use to detect any login failures. More about this in the next section.
  • logpath – the name or pattern (for example /var/log/apache/*.log) of the log to monitor for the failed logins. This is the file that the above filter will work on.
  • maxretry – this is how many times should the filter detect a problem before starting the ban.
  • bantime – this is how long, in minutes, that the ban be in effect for.

The Filter

Now that we have specified the log file to look in we need to specify how to find the event we need to look for. For our example, Proxmox writes a specific string each time a failed login occurs which looks like the belew:

authentication failure; rhost=10.10.10.10 [email protected] msg=no such user ('[email protected]')

Our Filter, therefore, needs to look for this text and pull out the IP address.

Create a Filter file called proxmox-web-gui.conf in /etc/fail2ban/filter.d/.

vi /etc/fail2ban/filter.d/proxmox-web-gui.conf

Add the following:

[Definition]
failregex = pvedaemon\[[0-9]+\]: authentication failure; rhost=<HOST> user=.* msg=.*

This will match the text that Proxmox writes to the daemon.log file when a failed login is detected. It’s got a Fail2ban specific keyword <HOST> which is what’s used to indicate to Fail2ban where the offending IP address is in the log entry. Fail2ban can then block this IP address as indicated in our Jail file.

Testing Fail2ban Filters

Fail2ban provides a nice little utility to test your Filter definitions to make sure they are working as you intend. First things first – we need an entry in our log file for an invalid login attempt. Go to your Proxmox Web GUI and enter some invalid login credentials.

The command to use is fail2ban-regex which has two parameters; the log file location and the Filter location.

fail2ban-regex /var/log/daemon.log /etc/fail2ban/filter.d/proxmox-web-gui.conf

An example of the output is below. The text Success, the total number of match is 1 states that there is one match in the log for our pattern in the proxmox-web-gui.conf file.

fail2ban-regex /var/log/daemon.log /etc/fail2ban/filter.d/proxmox-web-gui.conf

Running tests
=============

Use regex file : /etc/fail2ban/filter.d/proxmox-web-gui.conf
Use log file   : /var/log/daemon.log


Results
=======

Failregex
|- Regular expressions:
|  [1] pvedaemon\[[0-9]+\]: authentication failure; rhost=<HOST> user=.* msg=.*
|
`- Number of matches:
   [1] 1 match(es)

Ignoreregex
|- Regular expressions:
|
`- Number of matches:

Summary
=======

Addresses found:
[1]
    10.27.4.98 (Fri May 29 12:31:14 2015)

Date template hits:
770 hit(s): MONTH Day Hour:Minute:Second
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second Year
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second
0 hit(s): Year/Month/Day Hour:Minute:Second
0 hit(s): Day/Month/Year Hour:Minute:Second
0 hit(s): Day/Month/Year Hour:Minute:Second
0 hit(s): Day/MONTH/Year:Hour:Minute:Second
0 hit(s): Month/Day/Year:Hour:Minute:Second
0 hit(s): Year-Month-Day Hour:Minute:Second
0 hit(s): Year.Month.Day Hour:Minute:Second
0 hit(s): Day-MONTH-Year Hour:Minute:Second[.Millisecond]
0 hit(s): Day-Month-Year Hour:Minute:Second
0 hit(s): TAI64N
0 hit(s): Epoch
0 hit(s): ISO 8601
0 hit(s): Hour:Minute:Second
0 hit(s): <Month/Day/Year@Hour:Minute:Second>

Success, the total number of match is 1

However, look at the above section 'Running tests' which could contain important
information.

Restart fail2ban for the new Jail to be loaded.

service fail2ban restart

To check your new Jail has been loaded, run the following command and look for the proxmox-web-gui Jail name next to Jail List.

fail2ban-client -v status
INFO   Using socket file /var/run/fail2ban/fail2ban.sock
Status
|- Number of jail:      2
`- Jail list:           ssh, proxmox-web-gui

Try to log into the Proxmox Web GUI with an incorrect user 3 and see your IP address appear in the Currently banned section.

fail2ban-client -v status proxmox-web-gui
INFO   Using socket file /var/run/fail2ban/fail2ban.sock
Status for the jail: proxmox-web-gui
|- filter
|  |- File list:        /var/log/daemon.log
|  |- Currently failed: 0
|  `- Total failed:     3
`- action
   |- Currently banned: 1
   |  `- IP list:       10.10.10.10
   `- Total banned:     1

 


Create SSH Key Authentication Between Nodes

Category : How-to

Get Social!

Secure key authentication is one of the more secure ways to grant users access to a Linux server. The standard password authentication which is usually used to login to a server is replaced with an SSH key which is presented when authenticating. This increases security as passwords can eventually be cracked using brute force or even guessed in some circumstances. SSH keys are impossible to guess and almost impossible to to hack using brute force due to their length and complexity.

A SSH key is actually two strings of characters – one which is private and is used to connect to the server and another which is public which sits on the server itself.

Run the below command to create the key pair on the client machine.

ssh-keygen -t rsa

Accept the default location to save the key which will be inside the current users home directory:

Enter file in which to save the key (/home/james/.ssh/id_rsa):

For additional security, you can add a passhrase to the private key. This means the key cannot be used without the passphrase which increases the security of the key itself. Simply press return if you do not wish to use a passphrase.

Note: if you are using the key for applications to gain access to other servers, it’s unlikely that a passphrase will be supported.  

Enter passphrase (empty for no passphrase):
Your identification has been saved in /home/james/.ssh/id_rsa
Your public key has been saved in /home/james/.ssh/id_rsa.pub. 
The key fingerprint is:
46:ba:02:fd:2f:9c:b9:39:ec:6c:90:50:d8:ec:7b:00 james@testpc
The key's randomart image is:
+--[ RSA 2048]----+
|   +             |
|  E +            |
|   +    .        |
|  ..o  o         |
|  ...+. S        |
|   .+..o         |
|    .=oo         |
|     oOo         |
|     o=+.        |
+-----------------+

The two keys have been created;

  • Private: /home/james/.ssh/id_rsa
  • Public: /home/james/.ssh/id_rsa.pub

The final step is to copy the public key to the machine which you are going to connect to. In Debian or Ubuntu you can use the ssh-copy-id – you will need to change [USER] for the user who you will connect to the remote machine as and [SERVER] to the hostname or IP address of the remote server you will connect to.

ssh-copy-id [USER]@[SERVER]

Not all Linux distributions will contain the required ssh-copy-id utility, many CentOS/ Red Hat distributions do not for example, so you will need to use the manual method. Again, you will need to substitute the [USER] and [SERVER] attributes to the details of your remote machine.

cat /home/james/.ssh/id_rsa.pub | ssh [USER]@[SERVER] "cat >> ~/.ssh/authorized_keys"

It is not always recommended for security reasons but you can copy this public key to multiple machines so that you can use the same private key to connect to multiple remote machines.


Speed up Multiple apt-get install Requests by Caching the Repository

Get Social!

Linux penguinapt-get is the tool used in Debian and Ubuntu to manage packages installed on the system. Each time an update is available, or you install a new package the files will be downloaded from one of the central repository servers out on the internet and installed on your system.

There are two main problems with this:

  • Your servers may not be on able to access the internet directly for security reasons
  • Installing the same package on multiple servers will result in downloading the package the same amount of times. This could be slow or expensive in terms of bandwidth.

To solve the problem you can mirror the source repository on your own local server and add that as a source for your servers to update from. The main issue with this is that each distribution has a huge catalog of package which would take up vast amounts of space. Multiply this by the different releases of operating system in your environment and you could be talking terabytes of space.

Various utilities have been created to work round this problem such as apt-proxy, apt-cacher and debproxy. These utilities work by only caching some of the most used packages and fetching the rest from the source.

The below example will use apt-cacher-ng which is a middle man who sits in between the server being updated and the source repository out on the internet. It chooses to cache some regularly or recently used files locally and will recall them when they are requested which greatly speeds up the process for the requesting machine. The cache is frequently cleared to make sure that disk space is only being used for the most necessary packages. This drastically reduces resource required to run the service, whilst speeding up package downloading and guaranteeing that all packages are available.

Setting up apt-cacher-ng server

The apt-cacher-ng utility sits on a server which must be able to access both the public network and any internal network which your other servers may sit on.

Run apt-get install to install the proxy utility.

apt-get install apt-cacher-ng

The default installation of apt-cacher-ng holds details of both Ubuntu and Debian source repositories and is ready to use.

If you need to change the settings of the application such as the port it listens on, edit the below file:

vi /etc/apt-cacher-ng/acng.conf

You can now access the web interface using using the local machine’s IP or host name and the port. The default port is 3142.

apt-cache-nr-homescreen

This page shows that apt-cacher-ng is working correctly and is ready to cache the first source requests.

The next step is to add the server location to your clients. Create the below file and add details of your caching server.

/etc/apt/apt.conf.d/02proxy

Add the below line and edit [SERVER_IP] and [SERVER_PORT] to match your apt-cacher-ng configuration.

Acquire::http { proxy "http://[SERVER_IP]:[SERVER_PORT]"; };

Eg.

Acquire::http { proxy "http://10.10.10.1:3142"; };

Finally, run the update command on your clients to cause the proxy to cache the package lists. Packages will also be cached soon as you start to install or updates packages on your client.

To make sure that apt-cacher-ng is doing it’s job, tail the log to make sure entries are appearing.

tail -f /var/log/apt-cacher-ng/apt-cacher.log

In addition, you can also view the webpage for statistics on cache hits and misses:

http://[SERVER_IP]:[SERVER_PORT]/acng-report.html?doCount=Count+Data#top


update-rc.d Cheat Sheet

Get Social!

Linux penguinDebian and Ubuntu use the service command to control services and update-rc.d for adding and removing services from start up. Using the service command we can start, stop, restart and display all available services. With update-rc.d we can add and remove services and add them to the Ubuntu/ Debian start up scripts. As Linux operating systems have multiple states, or runlevels, you need to make sure you add any new services to the correct runlevels. For example, you would not want to start a web service application before starting networking.

See my post on runlevels for more information about runlevels in Linux..

What services are available for startup?

Use the status-all switch to list all services which are registered with the OS and issues them a status command. You will then get one of the following displayed next to each service:

  • [ + ] – Services with this sign are currently running.
  • [ – ] – Services with this sign are not currently running..
  • [ ? ] – Services that do not have a status switch.
service --status-all

Sample output:

service --status-all
 [ ? ] acpid
 [ - ] apparmor
 [ ? ] apport
 [ ? ] atd
 [ - ] bootlogd
 [ ? ] console-setup
 [ ? ] cron
 [ ? ] dbus
 [ ? ] dmesg
 [ ? ] dns-clean
 [ ? ] friendly-recovery
 [ ? ] glusterfs-server
 [ - ] grub-common
 [ ? ] gssd
 [ ? ] hostname
 [ ? ] hwclock
 [ ? ] hwclock-save
 [ ? ] idmapd
 [ ? ] irqbalance
 [ ? ] killprocs
 [ ? ] module-init-tools
 [ ? ] network-interface
 [ ? ] network-interface-container
 [ ? ] network-interface-security
 [ ? ] networking
 [ ? ] ondemand
 [ ? ] passwd
 [ ? ] plymouth
 [ ? ] plymouth-log
 [ ? ] plymouth-ready
 [ ? ] plymouth-splash
 [ ? ] plymouth-stop
 [ ? ] plymouth-upstart-bridge
 [ ? ] portmap
 [ ? ] portmap-wait
 [ ? ] pppd-dns
 [ ? ] procps
 [ ? ] rc.local
 [ ? ] resolvconf
 [ ? ] rpcbind-boot
 [ - ] rsync
 [ ? ] rsyslog
 [ ? ] screen-cleanup
 [ ? ] sendsigs
 [ ? ] setvtrgb
 [ + ] ssh
 [ ? ] statd
 [ ? ] statd-mounting
 [ - ] stop-bootlogd
 [ - ] stop-bootlogd-single
 [ ? ] sudo
 [ ? ] udev
 [ ? ] udev-fallback-graphics
 [ ? ] udev-finish
 [ ? ] udevmonitor
 [ ? ] udevtrigger
 [ ? ] ufw
 [ ? ] umountfs
 [ ? ] umountnfs.sh
 [ ? ] umountroot
 [ - ] unattended-upgrades
 [ - ] urandom
 [ ? ] whoopsie

Start a service

Starting a service is done using the command service followed by the service name and the command to start the service.

service apache2 start

Stop a service

Use the stop keyword with service to stop a service.

service apache2 stop

Check the status of a service

Each service has a status, usually running or not running. Some services, such as network, may have a different output and output more information on the service.

service apache2 status

Remove a service

Use the remove keyword with update-rc.d to remove the service start up command for an application. You will need to use the -f switch if the applications /etc/init.d start up file exists.

update-rc.d -f apache2 remove

 Add a service

Adding a service to Ubuntu or Debian is done with the update-rc.d command. You can specify which runlevels to start and stop the new service or accept the defaults. The init.d file will be added to the relevent rc.d startup folders.

 update-rc.d apache2 defaults

Setting Start and Kill priority

If you need more control over when your service is asked to start and stop, you may need to set the start and kill (S and K) values.

For a given runlevel, you may have several services starting. For example, you may have apache2 and mysql both starting on runlevel 3. Ideally, you’d want mysql to start before apache2 and shutdown after apache2. In this case we need to give mysql the priority in startup, but apache2 the priority in shutdown.

When starting, the lower the number, the earlier it will start. A start priority of 10 will start before a priority of 20. When killing, it’s the opposite. A higher number will be killed before a lower number.

To set the start and kill priority we simply add them to the above update-rc.d command with the start priority first, followed by the kill priority.

update-rc.d apache2 defaults [START] [KILL]

The below command will start mysql first, then apache2. On shutdown, the kill will be the reverse of the start with apache2 being killed first and mysql second.

update-rc.d apache2 defaults 90 90
update-rc.d mysql defaults 10 10

Because, in our example, both start and kill priorities are the same we can shorted the command to just

update-rc.d apache2 defaults 90
update-rc.d mysql defaults 10

Manually set the RunLevel to Start and Kill a service

You can manually specify the Linux RunLevel that the system must be in to Start and Kill your service. See my other blog post for more information on RunLevels.

To extend the above example, we can specify exactly which RunLevel apache2 will be started and stopped.

update-rc.d apache2 start 10 2 3 4 5 . stop 90 0 1 6 .

apache2 will be started (as long as it isn’t already) when the system enters RunLevel 234 or 5 with a priority of 10. It will then be asked to stop when the system enters RunLevel 01 or 6 with a priority of 90.

Let me know in the comments if you think anything is missing.


Latest git on Ubuntu

Category : How-to

Get Social!

octocat-githubI have recently been playing around with Git on Ubuntu and noticed that the version in the standard repository is a few versions behind the latest release.

Using the PPA repositories, you can install the latest version of the Git client software without having to build it from scratch.

Make sure you have the python-software-properties package installed.

apt-get install python-software-properties

Then add the repository and install git.

add-apt-repository ppa:git-core/ppa
apt-get update
apt-get install git

If you already have git installed, you can simply update to the latest version.

apt-get upgrade

Add launchpad PPA repository to Ubuntu

Get Social!

launchpad.net logoMost Linux based systems use a software repository which is either local (a CD-ROM) or remote (a web address) to install new software and manage software updates to already installed software. For Ubuntu/ Debian based Linux distributions apt-get is used to interact with these software repositories, for Red Hat/ CentOS its yum.

With a default Linux installation a suite of repositories are installed to manage the core operating system and install additional applications. As these repositories are critical to the Linux distribution it is difficult for software developers to get their software included in them because they have to be verified for stability and security. This means that the repositories are often behind the official release schedule of 3rd party software or don’t include the software at all. In older versions of Linux support may have been dropped altogether in favour of maintaining the newer versions of the distribution.

It’s here where Launchpad comes in. Developers can add their software to the Launchpad or PPA repository which can be added to a Linux distribution to enable installation of additional software which is not available in the core repositories.

GlusterFS, for example, is at version 3.2.5 in the core Ubuntu 12.04 distribution however the official release of GlusterFS is 3.4. You could build the GlusterFS binaries directly from source and I’ll cover that in a future blog post, but we are not going to do that here.

Kindly, semiosis has created a GlusterFS repository on Launchpad which we can add to our Ubuntu installation to deliver the latest (or thereabouts) version of the software.

Although this example details the Ubuntu GlusterFS 3.4 specifically, any Launchpad repository is added in the same way. For other software visit https://launchpad.net/ubuntu/+ppas and use the search function.

Make sure you have the following python utility installed which is used to add the repository to your Ubuntu sources list:

apt-get install python-software-properties

Use the command add-apt-repository with the username who created the repository on Launchpad and the repository name. You can find the user and repository that you require by searching on https://launchpad.net/

add-apt-repository ppa:[USERNAME]/[REPOSITORY NAME]

Glusterfs example:

add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.4

 


Visit our advertisers

Quick Poll

Do you use GlusterFS in your workplace?

Visit our advertisers