On-boarding data into Splunk is a fundamental skill and a commonplace task for Splunk Admins, and for your correspondent as a Splunk PS Consultant. The technical how-to’s of data onboarding apply here, but what I want to prepare you for is situations where getting the data into Splunk presents a political or technical tarpit you might not expect.
If you are a Splunk Admin doing the on-boarding, this can help you manage expectations, ask the right questions ahead of time, and line up the right people to help. If you’re planning on having a Splunk PS Consultant work with you to onboard data, you will reduce the risk around that engagement.
New Data Sources
When going after a new data source for Splunk, think about what you or your internal customers intend to do with it and then ask the subject matter experts how you might get that. With in-house application logs, this may turn out to be a journey, where you get the existing logs into Splunk and find that they do not tell you enough; you might have to work with the application developers to change the logging format into something more Splunk-friendly. Fortunately, Splunk lends itself to iteration, so you can start with what you have and work up to great. Even with merely-okay logs, you can probably build a simple dashboard or teach the developers a 15-minute Splunk search lesson and win some political capital for the next development cycle.
Reporting Systems
When onboarding data from a reporting system, make sure it will meet your requirements on the Splunk side. Reporting systems frequently only store aggregate data at a daily or weekly level, which is not much help if your Splunk requirements are for troubleshooting, alerting, or correlating events with other data sources – it’s the right data points at the wrong granularity. Talk to the reporting team and find out if there are different levels of granularity available to you, or if they can easily develop something that fits your requirements better.
Enterprise Applications or Networking Devices
When asked to get data from an enterprise application or networking device into Splunk, check first to see if there is a Splunk Technology Add-On (“TA”) for it. That can save you a lot of time accessing the data and working out field extractions, and the documentation will usually help in understanding the technical configuration steps. Check carefully to be sure the TA is compatible with both your version of Splunk and with your version of the application or device. Also, be sure to check if the TA is “Splunk Built”, “Splunk App Inspect Passed”, or “Splunk Supported”. Many Apps and add-ons are developed by the community and can vary in overall quality. Typically apps that are “Splunk Built”, or “Splunk App Inspect Passed” will be more thoroughly vetted, and “Splunk Supported” apps are officially covered by your support contract!
No TA?
If there is not a TA, do you have access to a relational database for the application? Does the application have a REST API, and if so, is there API reference documentation available? These are common avenues for getting the data you need; if you can get the help of SMEs, particularly a knowledgeable DBA, developer, or application admin, it will be a huge help. In my experience, you will typically need these SMEs to be available for an hour or two to help with configuration over the course of a few days. You may also need to get the help of the vendor (a resource I highly recommend if you are stalled in finding or accessing the data), your application admins probably know whom to ask for on the vendor side. Some vendors charge extra for access to this sort of data, so be prepared for that possibility – even if you have the right project sponsorship and available funding, the approval cycle may take weeks or even months.
Data Sensitivity
Finally, always bear in mind the possible sensitivity of the data, and if it seems possible that you will be on-boarding anything sensitive, consider how it needs to be approached. You might need to mask the data at rest in Splunk, or you might place it in a restricted index accessible only to users in certain roles.
About SP6
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.
