“Enterprise Log Search and Archive” (AKA ELSA) is an open-source project started by an individual who needed the ability to not just collect logs from a bunch of devices but to also quickly search and parse them. And by a “bunch” I mean on the scale of millions and millions of logs. A lot of traditional open-source systems use REGEX to parse logs, which is fine on a small scale but quickly falls apart under volume. ELSA takes a different approach and in the words of it’s creator allows “Google-fast searching on a massively large set of logs” and it does so by using a combination of MySQL and:
“Syslog-NG version 3, which includes support for the pattern-db parser. It allows Syslog-NG to be given an XML file specifying log patterns to normalize into fields which can be inserted into a database. It does this with a high-speed pattern matching algorithm (Aho-Corasick) instead of a traditional regular expression. This allows it to parse logs at over 100k logs/second on commodity hardware. Combined with MySQL’s ability to bulk load data at very high rates (over 100k rows/second)”
Great… so what I would have imagined was an impossible task has been solved, but what about actually digging through your parsed logs?
” I found Sphinxsearch.com, which specializes in open-source, full-text search for MySQL. Sphinx was able to index log tables at rates of 50k logs/second, and it provided a huge added feature: distributed group-by functionality. So, armed with Syslog-NG, MySQL, and Sphinx, I was able to put together a formal Perl framework to manage bulk loading log files written by Syslog-NG into MySQL and indexing the new rows.”
Honestly, this gent sounds like a bit of a genius, especially if it really works on commodity hardware.
It sounds too good to be true honestly however there is a growing community of people that say it does exactly what it says.
Now… here is my actual use-case…
A year or so ago I tried setting up OSSEC into one of the offices where I work and deploying OSSEC agents on each end-point workstation. This has proven to be more of a mess than anything else. It works fine until it doesn’t. OSSEC insists on encrypting all of your data streams which requires setting up each invdividual end-point which is a bit cumbersome. Automating the process isn’t a simple affair either. The office environment it was installed into shifts and changes constantly (systems get rebuilt with some frequency) which has led to a system that is only half-implemented.
So a few weeks ago I started looking into the easiest way to put in a basic syslog server with basic search functionality and the easiest to install-and-setup event log to syslog converter I could find. That is how I came across ELSA and another project called “Eventlog-to-Syslog”.
Right now I am in the process of setting up a sandbox in my office with several servers and endpoints to see how easy deployment is. I will tell you how it goes in the coming days/weeks! In the meantime, if you want to give all of this a spin with me, check out the links in the references section below!
UPDATE 03/02/2015: So a couple of things… I got this installed in a sandbox environment and it all went relatively smoothly except for one small “gotcha”… The install is actually a shell script and the whole system relies on a apache. Apache changed how configuration files were handled when they went from 2.2 –> 2.4. Ubuntu 12.04 LTS uses Apache 2.2 and the newer versions of Ubuntu use 2.4… The setup scripts were made for Apache 2.2. So if you deploy ELSA as it currently is on a newer Ubuntu base system, it won’t work. Rather than figure out how to fix it manually I just opted to install ubuntu 12.04 LTS. Hopefully this will save someone else a reformat.
Secondly… I am not going to pursue using ELSA at this time. Rather, I am going to give GrayLog a try as the web interface looks a bit more polished and I didn’t find ELSA’s log searching capabilities particularly intuitive. More on that venture soon!