3WoO: You got your OSSEC in my Logstash

There are a number of popular topics on the OSSEC mailing list, OSSEC-wui being the one I dread the most. OSSEC-wui is an old project that is currently unmaintained. It's a "web user interface" providing a view of the logs, agent status, and syscheck information.

The wui parsed OSSEC's plain text log files, requiring the web server to have access. Because of this the web server needed to be a member of the ossec group, and had to be able to access the OSSEC directories. That means no chrooted httpd unless you install OSSEC inside of the chroot. I didn't like this, and didn't use the wui.

Another problem with the wui was the log viewing interface. Working with OSSEC's rules and decoders has shown me how crazy the log space is. Everyone needs to create a new format, or thinks certain bits are not important/very important. It's crazy! There's also the possibility of attack through log viewers. XSS via a log webpage? It's definitely possible, and I've heard people talking about that possibility. Now, how much do I want to trust an unmaintained php application with access to potentially malicious log messages?

Fortunately there are alternatives: Graylog2, Splunk, logstash, Loggly, ArcSight, OSSIM, Octopussy, and probably dozens of others. Some are commercial and some are open source. Some have more features, and some are a bit more spartan. Paul Southerington even created a Splunk app for OSSEC called Splunk for OSSEC. It integrates Splunk and OSSEC quite nicely.

Two of the downsides to Splunk in my opinion are that it is not open source, and there is a 500 megabyte limit per day. I'm guessing that I shouldn't have ever hit the 500MB limit since I'm running this setup at home, but I did. Nine times in one month. Since this is a home use setup a commercial license for Splunk just wasn't worth it, so I migrated to logstash.

Here's what logstash has to say about itself:

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.

logstash has a simple interface

Logstash is written in ruby, utilizing some of the benefits of jruby. If you're using a version of logstash before 1.1.0 you may want to install libgrok as well. I believe it's being integrated into 1.1.0, but my information may be a bit off. I'll be using libgrok later in this post. That's the extent of the requirements.

A quick note about grok: It requires a relatively recent version of libpcre. CentOS (and probably RHEL) use a relatively ancient version of libpcre. While there are probably ways to get it working, I didn't want anything to do with that jazz, so I moved to Debian. I think it says something when Debian has a more up-to-date package than your former distro of choice.

I'm using logstash as an endpoint for logs, but you can also use it to push logs somewhere else. You could install a minimal logstash instance on one system, and have it provide that system's logs to another logstash instance somewhere else.

Looking at the documentation you'll see that logstash puts all configurations into one of three categories: input, filter, or output. An input is a source of log events. A filter can modify, exclude, or add information to an event. And an output is a way to push the logs into something else.

One of the most important outputs logstash offers is elasticsearch. If you're going to use logstash's web interface, you'll need this output. elasticsearch provides search capabilities to logstash. logstash and especially elasticsearch will use all of the RAM you give them. Luckily elasticsearch is clusterable, so adding more resources can be as simple as configuring a new machine. Sizing your elasticsearch installation is important, and knowing your events per second may help.

The logstash monolithic jar file contains the logstash agent, web interface, and elasticsearch. It allows you to run all 3 pieces of logstash with 1 command and to even use the embedded elasticsearch. This is much more limited than using a separate elasticsearch installation, but may be adequate for small or test installations.

In the following screen shot the first process is an elasticsearch node. The other 2 java processes are logstash's agent and web processes. I have 1 more elasticsearch node using slightly more memory than this one as well. Setting it up this way gives me a chance to play with it a bit.

logstash processes

I run the agent with the following command:
java -jar ./logstash-1.0.16-monolithic.jar agent -f ./mylogstash.conf

And the web process:
java -jar ./logstash-1.0.17-monolithic.jar web

This is getting a bit long, so I'll be detailing mylogstash.conf in my next post.