Media Temple does some stupid interesting things with its linux installs. Particularly it seems like they don’t like clients upgrading their own boxes… So they don’t include some core packages that are needed to run the automated upgrade on Ubuntu. Now, I got on the support line and the front line cannon fodder (aka Tier 1 support) gave me the following entertaining fiction to avoid helping:
(more…)
The Heartbleed bug is what I would professionally classify as seriously scary stuff. Basically there is some kind of heartbeat functionality built into OpenSSL. Often, in tech talk, this kind of thing is used for remote service monitoring (i.e. if I have a pulse my service is at least up). In this case, I am not sure the specific application. What was discovered was that this feature could be exploited by a remote hacker to gain access to information stored in the servers memory in 64k chunks at a time… and it was undetectable. Pretty much anything could have been leaked, including the SSL private keys themselves. This means a person could then decrypt in-transit information between the server and client and have access to all kinds of goodies like usernames and passwords. (you can read more here)
This is a veritable nightmare and affects pretty much all of us running websites on linux hosts that make use of SSL. So, I highly recommend you dive in quickly and fix this issue, fast. The first step…
Test your site to see if is vulnerable to the heartbleed bug. You can check it using the tool here:
Okay… don’t flip out if you are vulnerable. Pretty much all of my SSL sites were as they use OpenSSL like the other 2/3rds of all internet sites. I have since patched them all.
I host hosted on pretty much all Ubuntu 13.04 servers running Apache 2. My instructions for fixing the bug are geared towards Ubuntu.
Second step… back your SSL keys up!
We are going to be completely wiping your current SSL install OUT on your web server. This means you need to backup your cert files. Here is how I did mine on one particular server. (later on it is a very good idea to regenerate your server’s private key and replace all of your SSL keys anyhow, but it isn’t a bad idea to keep what you have around…)
First, make a directory tree where you can copy all of your certs to.
sudo -s
mkdir /sslbackup
mkdir /sslbackup/private
mkdir /sslbackup/certs
mkdir /sslbackup
mkdir /sslbackup/private
mkdir /sslbackup/certs
Next, if you are running Apache, which I am, you can determine where the SSL cert files are that you need to backup by doing the following (this is for apache2, running on Ubuntu 13.1, your site config files may be somewhere else):
cd /etc/apache2/sites-enabled
On Ubuntu and Debian this folder is usually used to house the “running config” files for each of your websites. Any of your sites that use ssl (i.e. the url is httpS://whatever.com) need to be checked. In my example lets say I have a site called contoso.com and it has a config file named contoso.com-config… Run the following command against that file:
egrep -wi --color 'SSLEngine|SSLCertificateFile|SSLCertificateKeyFile|SSLCertificateChainFile' contoso.com-config
Run that command for each of your site config files and note the output. You should get something like this:
SSLEngine on
# SSLCertificateFile directive is needed.
SSLCertificateFile /etc/ssl/certs/contoso.crt
SSLCertificateKeyFile /etc/ssl/private/contoso.key
# Point SSLCertificateChainFile at a file containing the
# the referenced file can be the same as SSLCertificateFile
SSLCertificateChainFile /etc/ssl/private/contosoint.crt
# SSLCertificateFile directive is needed.
SSLCertificateFile /etc/ssl/certs/contoso.crt
SSLCertificateKeyFile /etc/ssl/private/contoso.key
# Point SSLCertificateChainFile at a file containing the
# the referenced file can be the same as SSLCertificateFile
SSLCertificateChainFile /etc/ssl/private/contosoint.crt
There are three files in the above case. Public and private keys and the chain file. You can now back those files up to your backup directories. Example:
cp /etc/ssl/certs/contoso.crt /sslbackup/certs/contoso.crt
cp /etc/ssl/private/contoso.key /sslbackup/private/contoso.key
cp /etc/ssl/private/contosoint.crt /sslbackup/private/contosoint.crt
cp /etc/ssl/private/contoso.key /sslbackup/private/contoso.key
cp /etc/ssl/private/contosoint.crt /sslbackup/private/contosoint.crt
That should backup all of your cert files safely so that when we totally wipe-out OpenSSL we don’t have to worry about losing our cert files.
Next, it is time to remove OpenSSL. In my case I am on Ubuntu so I use the ‘apt’ package manager. Your distro may vary.
apt-get update
apt-get purge openssl #This is where your certs may get wiped if you didn't back them up above
apt-get purge libssl-dev
apt-get autoremove && apt-get autoclean
apt-get purge openssl #This is where your certs may get wiped if you didn't back them up above
apt-get purge libssl-dev
apt-get autoremove && apt-get autoclean
Next we need to update to the latest version of openSSL. I was running Ubuntu 13.04 on my server though and this was an issue because it is no longer support. To fix this problem, I upgraded to 13.10 by doing the following:
do-release-upgrade
If that works for you, GREAT… I, however, am hosting on Media Temple and had no such luck. To work around I had to do this:
apt-get install update-manager-core python-apt
cd ~
mkdir tmp && mkdir vartmp && mount --bind ~/tmp /tmp && mount --bind ~/vartmp /var/tmp
do-release-upgrade
cd ~
mkdir tmp && mkdir vartmp && mount --bind ~/tmp /tmp && mount --bind ~/vartmp /var/tmp
do-release-upgrade
The upgrade then ran…
Afterwords I did the following
apt-get update
apt-get install openssl libssl-dev
service apache2 restart
apt-get install openssl libssl-dev
service apache2 restart
Now… after this was done… I had all kinds of additional personal hell. This came particularly from the upgrade of Ubuntu to 13.04 to 13.10. I did this on two separate servers and it broke all kinds of things. More on that in my next post!
Assuming you didn’t have to upgrade Ubuntu then you can go back to the heartbleed vulnerability checking site:
And test your site again. Once I resolved my other issues and had my sites back up again, this showed the vulnerability to be remediated!
Now… the HOLE has been closed. Which is great. However the particularly scary thing about this portal of doom is that there was no way to detect if anyone had ever exploited it. Yep… people could have stolen stuff and there is no way to know. They could have stolen your SSL private keys… hence… the recommendation now that we have slammed the door shut is to:
-> Regenerate new private keys.
-> Submit new CSR to your CA.
-> Obtain and install new signed certificate.
-> Revoke old certificates.
(information taken from discussion here)
I will not be going into the details of how to do all of that in this post.
References:
http://heartbleed.com/
http://www.mysqlperformanceblog.com/2014/04/08/openssl-heartbleed-cve-2014-0160/
http://docs.openvpn.net/heartbleed-vulnerability-on-access-server-1-8-1-2-0-5/
http://serverfault.com/questions/587329/heartbleed-what-is-it-and-what-are-options-to-mitigate-it
http://askubuntu.com/questions/429385/upgrade-openssl-on-ubuntu-12-04
https://www.openssl.org/source/
http://www.cyberciti.biz/faq/searching-multiple-words-string-using-grep/
http://askubuntu.com/questions/444702/how-to-patch-cve-2014-0160-in-openssl
I run google adsense on several of my drupal sites. That is a whole other discussion for a whole other time. Suffice to say, adsense now gives your site a health ranking or score for ad display.
Crawler Errors are one particular problem that can hurt your adsense score.
Drupal, by default, has a pretty complete “robots.txt” file right out of the box. However I have found that it needed a few lines added to it.
(more…)
Today I am working on setting up a BackupPC server to take remote internal centralized backups of some of our other servers on the cheap.
I already had BackupPC installed and the basics configured but I needed to add a new drive to the system (for additional backup data storage) and I also needed to setup a new NIC connection. My Ubuntu Server is running on Microsoft Hyper-V 3.0 on a Server 2012 host machine so adding all the new hardware was as simple as a few clicks.
Normally I am a command-line guy but this server is going to be managed ongoing by folks who are less Linux savvy so I wanted to install some additional software that would make their life easier. To that end, I am using Webmin.
During the course of adding additional storage to my VM I ran into some headaches related to Hyper-V and Linux storage formatting of GPT disks larger than 2 TB.
Sounds like a very specific use case? I think it is quicky becoming more common as A.) Storage gets cheaper and therefore larger and B.) Microsoft Hyper-V sees more adoption as it is now decently featured and has attractive pricing for people with existing Windows infrastructure. Hopefully this article will help you avoid the trouble I ran into when setting up a new large disk on an Ubuntu Hyper-V VM…
(more…)
Most UTM (unified threat management) Firewall devices worth their price tag include a VPN server as part of the mix. In my experience, a UTM is an excellent choice for a small office and/or most smaller enterprises as several of the higher-end devices scale quite far. For a larger, corporate network though, while a UTM (or two or three) might be part of the security mix, larger dedicated components often make more sense.
That being said, if you have a UTM, and it includes a VPN solution, you may be considering taking advantage of this for remote network access. While I wouldn’t necessarily advise against doing this, before going to far down that road I would tell you to look into deploying OpenVPN Access Server instead with Google Authenticator. Here is why…
(more…)