As ITSI 4.0 was announced at Splunk .conf18, a new set of features and enhancements were introduced to the applause of the crowd, but as an experienced implementer of Splunk ITSI (IT Service Intelligence), I feel very few companies start with their best foot forward. Let me tell you why and what you can do to follow in the footsteps of those that are.
For Starters, What is Splunk ITSI?
I find a lot of people will handicap their Splunk ITSI implementation by pigeonholing Splunk ITSI into infrastructure monitoring. Now, I can’t say it’s a surprising opinion as Splunk ITSI seems to fill all of the usual checkboxes that would make you think it’s an enhanced monitoring tool, but trust me when I say Splunk ITSI is more valuable as a Service Insight and Event Analytics tool.
A service monitoring platform is a tool that will allow you to monitor the top-level services that IT delivers to the business for its day-to-day operations. Think Business Process Monitoring. A regular monitoring tool, on the other hand, will allow you to monitor groups of devices and applications. Do you see what I’m getting at? The business is not interested in what infrastructure components are down. They’re hungry to know what services are in danger of interrupting the business.
How Successful ITSI Implementations Start
It’s important to align your strategy to meet the goals of the business. Let me drive the next point with an analogy:
Imagine the company buys you a beautiful new and expensive sports car and you’re expected to break it in via a round trip that will take you through two different kinds of roads. The first road is the autobahn where you’ll be taking the car to its top speed. The second is a backroad where you’ll be forced to go slow. Now picture your audience eying your progress. What do you think they want to see first?
Time runs out on ITSI implementations that took the slow road because they never got their strategy right. The stakeholders will wonder what is holding value up as you’re pelted with requests from the technical, business, and every other type of stakeholder in between. Why? Because you took the slow road first. Keep this in mind:
“Patience is like air in a sealed room.
No matter how much you start with, eventually, you will run out.”
– Siberia (movie)
Start with this exercise: Discover what are the pain points, why you were brought in, and by who, identify how much time you have, who the audience is, and who’s first in line to get results. Patience will not be your friend; you must decide how to deliver the most value in the least amount of time. More often than not the outcome will equate to building out top-level business KPI’s and glass tables in the first iteration. Hold off on the deep dives and technical KPIs.
So, What is a Service?
A service – at least in the sense that we care about – is a capability in which the business is critically dependent. It will vary from industry to industry but some examples might be:
- The online sales process supported by an e-commerce platform.
- Start of day order shipments supported by a nightly batch process.
- The ability to create and deliver quotes to customers enabled by a risk assessment tool.
All of these are top-level business services – not infrastructure components. I know this is a big departure in the way most of us think of monitoring in Splunk. Partly because we are used to implementing Splunk for IT Infrastructure and partly due to there being very few examples of Splunk ITSI reaching this zenith of service monitoring I speak of. Think of it as mixing business process monitoring with end-user experience motoring. I hope I’m not losing you as next come the actions to take.
How to Approach a Splunk ITSI Implementation
Start from the top
The crucial shift you’ll need to accomplish is to approach a new project looking from the top down. Identify what the business value of this project is and forget for a moment your years of infrastructure monitoring experience – where you learned to start from the bottom up, looking at individual components.
Find services worth monitoring
Substitute infrastructure questions for questions that will help you discover services that are critical capabilities, at risk of failure, and costly to restore for the business. Whip up a small risk assessment for these services on a spreadsheet as these are the services that IT and the business will be more interested in. The risk assessment will help a champion find you and you will absolutely need a champion.
Identify the main components or service dependencies
Once you have some services worth monitoring, follow up with the service owners to pin down what makes delivering those business services such a challenge. Map the systems that make these services possible as you will need to monitor them later on via KPIs
Define the application dependencies
Now you enter more familiar territory as you meet with the SMEs or IT leads to correlate specific infrastructure components (servers, routers, storage devices, etc.) with the applications they support.
Hunt down the data sources
Find the data you will need to onboard to support the KPIs for the applications and their dependencies. Here we come to the aspect we’re most with comfortable as Splunkers – onboarding, parsing, and extracting fields from data.
Don’t get stuck in the weeds!
Well, at least try not to. This will tend to happen automatically as you unconsciously pay more attention to the technical aspects of the engagement and not enough to where the value is actually delivered. It’s where Splunkers are usually more comfortable but fight it.
Make sure that you build and design for purpose and your sights always return to the goal. Ask yourself, “how is this adding value?” If you’re having trouble explaining it then take a second look at the stated business objectives and realign.
To sum things up: The more you make ITSI about Business Process Monitoring and Service Monitoring the easier it is to justify ITSI.
It’s difficult to add great value to traditional infrastructure monitoring as that is a mature field with many offerings. The higher up you climb towards monitoring business processes the easier it is to innovate using Splunk ITSI.
Keep These Tips in Mind
- Be light on your feet: Don’t let the lack of data slow you down. If you can’t get real-time data, get static data. If you can’t get real data then simulate it.
- Demo often: Build a glass table mock-up early. Show the goods as soon as possible even in the early stages when most of it is smoke and mirrors. Your champion will use this continuous firepower to get detractors onboard and keep superiors excited.
- Find a champion: I can’t stress this enough. You will need a visionary that gets it. He’s for changing the status quo, not afraid of bringing up less than ideal situations to light, and will guide the project through the data procurement process.
- Don’t lose alignment and keep in mind that everything you develop must be ultimately aligned with an issue the business is trying to solve. If Splunk ITSI is not being used to make an existing business process obsolete then it must enhance it by making it faster, more reliable, or efficient.
- Get plugged into the problem management cycle. Find out about new use cases or risks to your Service Insight and Event Analytics platform.
- Shift your monitoring strategy. A successful Splunk ITSI implementation will require a mind shift in the way you view monitoring. IT infrastructure monitoring is straightforward: You ask what are the KPI’s, what are the thresholds, the list of devices, and move on. But if you start down that path with Splunk ITSI you’re setting up the platform to be compared against a myriad of other monitoring tools – it’s a battle it was not designed to win. Instead of volume go for the quality. Do a Cost/Benefit analysis for each KPI that comes your way and focus on the ones that deliver the most value for the effort. After the low-hanging fruit is gone then you can think of increasing volume.
For a step-by-step approach take a look at this .conf2016 break out by Martin Wiser and Bill Babilon from Splunk: Anatomy of a Successful IT Service Intelligence Deployment
Finally, I’ll leave you for a Splunk ITSI implementation I was heavily involved in. Hopefully, this will give you a grounded idea of what Splunk ITSI looks like when it’s applied to business needs: Domino’s Delivery of a Faster Response Was No Standard Order
Now go for the gold and leave boring bronze tiered monitoring behind!
About SP6
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.
