The Heartbleed bug is what I would professionally classify as seriously scary stuff. Basically there is some kind of heartbeat functionality built into OpenSSL. Often, in tech talk, this kind of thing is used for remote service monitoring (i.e. if I have a pulse my service is at least up). In this case, I am not sure the specific application. What was discovered was that this feature could be exploited by a remote hacker to gain access to information stored in the servers memory in 64k chunks at a time… and it was undetectable. Pretty much anything could have been leaked, including the SSL private keys themselves. This means a person could then decrypt in-transit information between the server and client and have access to all kinds of goodies like usernames and passwords. (you can read more here)
This is a veritable nightmare and affects pretty much all of us running websites on linux hosts that make use of SSL. So, I highly recommend you dive in quickly and fix this issue, fast. The first step…
Test your site to see if is vulnerable to the heartbleed bug. You can check it using the tool here:
Okay… don’t flip out if you are vulnerable. Pretty much all of my SSL sites were as they use OpenSSL like the other 2/3rds of all internet sites. I have since patched them all.
I host hosted on pretty much all Ubuntu 13.04 servers running Apache 2. My instructions for fixing the bug are geared towards Ubuntu.
Second step… back your SSL keys up!
We are going to be completely wiping your current SSL install OUT on your web server. This means you need to backup your cert files. Here is how I did mine on one particular server. (later on it is a very good idea to regenerate your server’s private key and replace all of your SSL keys anyhow, but it isn’t a bad idea to keep what you have around…)
First, make a directory tree where you can copy all of your certs to.
mkdir /sslbackup
mkdir /sslbackup/private
mkdir /sslbackup/certs
Next, if you are running Apache, which I am, you can determine where the SSL cert files are that you need to backup by doing the following (this is for apache2, running on Ubuntu 13.1, your site config files may be somewhere else):
On Ubuntu and Debian this folder is usually used to house the “running config” files for each of your websites. Any of your sites that use ssl (i.e. the url is httpS://whatever.com) need to be checked. In my example lets say I have a site called contoso.com and it has a config file named contoso.com-config… Run the following command against that file:
Run that command for each of your site config files and note the output. You should get something like this:
# SSLCertificateFile directive is needed.
SSLCertificateFile /etc/ssl/certs/contoso.crt
SSLCertificateKeyFile /etc/ssl/private/contoso.key
# Point SSLCertificateChainFile at a file containing the
# the referenced file can be the same as SSLCertificateFile
SSLCertificateChainFile /etc/ssl/private/contosoint.crt
There are three files in the above case. Public and private keys and the chain file. You can now back those files up to your backup directories. Example:
cp /etc/ssl/private/contoso.key /sslbackup/private/contoso.key
cp /etc/ssl/private/contosoint.crt /sslbackup/private/contosoint.crt
That should backup all of your cert files safely so that when we totally wipe-out OpenSSL we don’t have to worry about losing our cert files.
Next, it is time to remove OpenSSL. In my case I am on Ubuntu so I use the ‘apt’ package manager. Your distro may vary.
apt-get purge openssl #This is where your certs may get wiped if you didn't back them up above
apt-get purge libssl-dev
apt-get autoremove && apt-get autoclean
Next we need to update to the latest version of openSSL. I was running Ubuntu 13.04 on my server though and this was an issue because it is no longer support. To fix this problem, I upgraded to 13.10 by doing the following:
If that works for you, GREAT… I, however, am hosting on Media Temple and had no such luck. To work around I had to do this:
cd ~
mkdir tmp && mkdir vartmp && mount --bind ~/tmp /tmp && mount --bind ~/vartmp /var/tmp
do-release-upgrade
The upgrade then ran…
Afterwords I did the following
apt-get install openssl libssl-dev
service apache2 restart
Now… after this was done… I had all kinds of additional personal hell. This came particularly from the upgrade of Ubuntu to 13.04 to 13.10. I did this on two separate servers and it broke all kinds of things. More on that in my next post!
Assuming you didn’t have to upgrade Ubuntu then you can go back to the heartbleed vulnerability checking site:
And test your site again. Once I resolved my other issues and had my sites back up again, this showed the vulnerability to be remediated!
Now… the HOLE has been closed. Which is great. However the particularly scary thing about this portal of doom is that there was no way to detect if anyone had ever exploited it. Yep… people could have stolen stuff and there is no way to know. They could have stolen your SSL private keys… hence… the recommendation now that we have slammed the door shut is to:
-> Regenerate new private keys.
-> Submit new CSR to your CA.
-> Obtain and install new signed certificate.
-> Revoke old certificates.
(information taken from discussion here)
I will not be going into the details of how to do all of that in this post.
References:
http://heartbleed.com/
http://www.mysqlperformanceblog.com/2014/04/08/openssl-heartbleed-cve-2014-0160/
http://docs.openvpn.net/heartbleed-vulnerability-on-access-server-1-8-1-2-0-5/
http://serverfault.com/questions/587329/heartbleed-what-is-it-and-what-are-options-to-mitigate-it
http://askubuntu.com/questions/429385/upgrade-openssl-on-ubuntu-12-04
https://www.openssl.org/source/
http://www.cyberciti.biz/faq/searching-multiple-words-string-using-grep/
http://askubuntu.com/questions/444702/how-to-patch-cve-2014-0160-in-openssl
Pingback: Heartbleed – One Final Step – Change out your Server Private Keys and Re-key Your SSL Certs | KiloRoot.com