Splunk - the swiss army knife of big data

techweekeurope.co.uk

What is it?

Splunk Inc. is a vendor of a solution to monitor, search, analyze, visualize and act on massive streams of real-time and historical machine data.

The solution is widely in use by more than 4,800 enterprises in over 85 countries. The added value of the usage Splunk software is to gain operational intelligence that informs business and customer understanding, improves service and up-time, reduces cost and mitigates cyber-security risk.

Enterprises’ IT systems and infrastructure contain websites, applications, servers, networks, sensors, mobile devices and similar - All of them generate massive amounts of machine data.

The machine data can be turned into valuable insights with help of Splunk.



On the basis of these analyses it is possible to determine problems and investigate security incidents in shorter time than hours or days. End-to-end monitoring of the infrastructure increase the quality of the service and avoids service degradation or outages. Gain real-time visibility into customer experience, transactions and behavior.

Splunk bridges the gap between simple log management and security information and event management (SIEM) products from other vendors.

What separates out Splunk from the world of Syslog servers and SIEM tools is Splunk Apps, a library of add-ons and plugins that make Splunk smarter about particular types of log information, change its look-and-feel, or add new types of domain specific analysis.

If you want to make Splunk work, you've got to be ready to abandon the slick GUI and dive deep into difficult technical configuration, editing configuration files, writing regular expressions, and taking the time to understand where your data are coming from and how Splunk will see them.

How does it work?

Splug goes through of log / data files, parsing and indexing them. After this different searches can be run via the Splunk interface. Splunk supports technologies like Big-data based on Hadoop. With the help of Map Reduce it identifies the events from the log records, and counts the number of occurrence.

de.splunk.com

There are different possible graphical dashboards, that could be built. The result data can be exported via the API to excel for additional regressions and analysis.

Horizontal scaling is fully supported and is the way to go for data processing with Splunk. The bottleneck of the data processing is usually CPU and challenging parameter is RAM. Splunk Enterprise has built in ability to handle a clustered topology.

How much does it cost?

The software licensing model works like this. The trial version included full functionality and can be used for 60 days. Only Splunk Enterprise works as a distributed system.

A Splunk Enterprise License for up to 500mb of indexed data costs $500/mo, and scales up with the organization’s indexing needs. The license is only based on the indexed data, not on the number of queries into the data.

Nice thing is that it does not matter how many instances (machines) are in a cluster the price of the license stays the same!