Querying Splunk with Clojure

Introduction

Two months ago I was complaining that I didn’t have a job, and then suddenly I had one. I work for Splunk now, an enterprise-level machine data analysis company. Their core product is pretty magical: you can shove just about any logfile, real-time stream, script-generated datasource, or anything else at it, and then interrogate the data to understand and monitor your networks, server farms, client connections, whatever. It does an amazing job of correlating data through little more than keyword observation. There’s even a free version, which, while limited to a half-gig of data, is a good way to start off taking apart your weblog files.

They have me niftying up the third-party web framework, the thing large companies use to integrate the data we collect with their own network intelligence dashboards, visualization systems, whatever. It’s all my kind of thing: Python back-end, Javascript/Backbone/jQuery front-end stuff, lots of clever closures and event handling.

But I decided, for entertainment purposes only, to learn more about a part of the system I know almost nothing about: the Java SDK. And it wasn’t enough to work in a language I don’t know (since I don’t know Java) with SDK’s I’ve never seen before. I had to go make it work with a language that, to the best of my knowledge, no one has ever demonstrated compatibility with Splunk before. I had to make it work in Clojure.

Yes, I’m a Hipster Hacker. I mean, come on, if you’re not making it difficult for yourself, what is education for, anyway?

The complete source code for this project is available at Github.

Disclaimer

I am a newbie to all of this. Clojure, Splunk, even Java. This is entirely raw and beginner level stuff. I did this for my own edification. This code in no way represents the state of the art at Splunk. It’s definitely not warranted in any way by me or anyone else, and it’s licensed under the Apache Public License, V2. Use at your own risk. Don’t make me go ALL CAPS on you.

( It's big, let's put it someplace kind. )

Now you know how to talk to the basic Splunk service using Clojure. I’m sure if you’re comfortable with Clojure none of the Java access weirdness surprised you at all, but for me this was a pretty good exercise. At times I felt like I was writing in a “fantasy LISP”, a Lisp that actually, you know, had real world applicability, and that should have done the things I would have expected of a LISP. That fantasy LISP was pretty close to the real deal; it only took me a few hours of lein run sessions to knock out all the bugs.