I wrote a new script today to keep me up-to-date on how full the boot partition is on my Ubuntu servers. I actually administer quite a few of them and it can become a real issue if the boot drive hits 100% full, which it commonly does. The reason for this is that the boot partition by default is quite small (usually under 200 MB) and will fill up, often over the course of only a few months, with kernel files.
I have seen some servers carry on just fine when this occurs, I have also seen other servers exhibit some really odd behavior. Either way, it is best avoided.
The problem (and the blessing) is that Ubuntu Server is Linux… which means it requires very little administrative intervention month to month because (unlike another well-known and much used server platform) Linux tends to just work and work and work and work.
Very early on I went in for a job interview in IT. At the time I was a freshly minted MCITP, and knew very little about anything outside of Microsoft. The gentleman interviewing me asked me what my experience with Linux was, to which I replied “very little.” He then thought for a moment and said, “Well that really doesn’t matter, we have several Linux servers and the primary issue we run into is that we forget about them for 3 – 5 years until a power supply or spinning disk dies.”
That sums up most Linux setups in a nutshell. All that to say, it legitimately might be several months in-between administrative server logins. A fact which most admins are quite thrilled about.
When I started realizing that Ubuntu Server was going to require regular intervention, albeit fairly lightweight stuff, I was a bit bummed. It probably isn’t a horrible thing, rock solid stability aside it is still a good idea to login now and again to keep your software packages up to date. However that tendency to “set it and forget it” remains.
Long story short, I needed an automated alert system.
Solution, a shell script running as a cron job that will shoot me an email.
If you are following along, we are now going to dive in to the practical instruction bit. Before you get started there is one main pre-requisite for an email-sending script… your server needs to be able to send emails. Let’s start there.
There are two (or perhaps many more but I am going to oversimplify) categories of email sending systems for linux. Notice the emphasis on SENDING. I am not worried about setting up mailboxes and receiving email, just sending it out. So two categories for sending… those programs that rely on and forward messages to an external SMTP server, and those that can send out directly. NOTE: I am no email guru, and this description might be grossly inaccurate but it is the best I can come up with based on my experience. My preference is to send mail directly from my box as I want as few external dependencies as possible. The easiest and most lightweight program I have come across to do just that is Exim.
Rather than reinvent the wheel by writing a guide on setting up Exim, I will just point you to this one right here: https://www.digitalocean.com/community/tutorials/how-to-install-the-send-only-mail-server-exim-on-ubuntu-12-04
UPDATE 02.04.2016 – I decided to reinvent the wheel after getting tired of constantly having to manually configure Exim… If you want a quick one-liner to install AND configure Exim4 for you on Ubuntu, please take a look at this link: Install Exim4 MTA with One Command
One other note, if you are setting this up, like I am, to just allow the server to send out periodic alerts, it is important to only allow mail to Exim from 127.0.0.1. Furthermore, if you have a firewall the server only needs port 25 open outbound and you don’t need anything open inbound. This decreases the external attack surface to almost nothing and keeps things secure.
Once you are all set, here is your script:
use="$(df -h | grep boot | awk '{print $5+0}')"
server="$(hostname)"
ip="$(ifconfig eth0 | grep "inet addr" | awk '{print $2}')"
contents="$(ls -lha /boot | grep generic | awk '{print $9,$5}')"
kernel="$(uname -a | awk '{print $3}')"
email="[email protected]"
if [ "$use" -ge "65" ];
then
echo "Boot partition usage is "$use"% this is not okay"
echo -e "server: $server\n\nIP $ip\n\nThe boot partition is "$use"% full!\n\nCurrent Kernel: $kernel\n\nBoot Partition Contents:\n$contents" | mail -s "$server, boot "$use"% full!" $email
else
echo "Boot partition usage is "$use"% this is okay"
fi
This script does one of two things based on how full the boot partition is. If the partition is 65% or more full, it outputs a message to the command line (really that is just for testing) and it sends an email alert message with some particular information. If it is less than 65% full, it just outputs a message to the command line. The “else” statement could actually be taken out entirely for production usage.
The main magic happens with all of the variables at the top. I will walk through those now:
$use – this variables gets the output from the “df” command. By default df lists all of your partitions and how full they are. We use grep and awk to strip everything else and return only a number representing how full our boot partition is.
$server – will contain the hostname of the server this script is running on.
$ip – will contain one of the local IP addresses on the nic eth0.
$contents – will contain a neatly formatted list of all of the files and their sizes in the boot directory.
$email – here is where you enter your email address that you want to receive the alerts.
All of that is thrown into an if-then loop. If $use is greater than or equal to 65 (the threshold we set for receiving an alert), then…
As far as the “then” bit goes I am just going to break down the email command.
else
The first part of the command is constructing the body of our email. Using the “-e” flag will let us use “\n” to signal a new line in our email body. From there we start putting in variables to copy all of our information into the body of our email. The final output is then piped “|” into the mail command which calls on Exim to send the email. The “-s” flag is for our subject, which follows in double qoutes and then finally our $email variable contain our email address that Exim will send the message to.
The final email that will arrive in your inbox should look something like this:
IP addr:192.168.25.25
The boot partition is 72% full!
Current Kernel: 3.13.0-71-generic
Boot Partition Contents:
abi-3.13.0-71-generic 1.2M
abi-3.13.0-73-generic 1.2M
config-3.13.0-71-generic 162K
config-3.13.0-73-generic 162K
initrd.img-3.13.0-71-generic 20M
initrd.img-3.13.0-73-generic 20M
System.map-3.13.0-71-generic 3.3M
System.map-3.13.0-73-generic 3.3M
vmlinuz-3.13.0-71-generic 5.6M
vmlinuz-3.13.0-73-generic 5.6M
Once all of this is put together and tested, you can chmod +x the saved script file and drop it into the daily cron folder, which on Ubuntu should be at “\etc\cron.daily. If you want to run it on a more specific schedule you can instead setup an entry in crontab. The only other thing I will note is to make sure you script filename doesn’t have any kind of extension, otherwise it will be ignored.
Hope this helps other Ubuntu Server admins stay on top of things! Happy Friday!
Thanks, very helpful.