Recently we hosted a webinar titled, “12 Steps to Splunk Success,” presented by SP6’s Managing Partner, Jim Barge. During the webinar, Jim reviewed common hurdles our Professional Services Engineers recognize as common across many Splunk customers, along with solutions to these challenges.
As the title of the webinar suggests, there are a number of hurdles that your organization may face during an initial Splunk deployment or just in the general upkeep of your environment. Challenges may vary from one organization to another, but one challenge commonly encountered by our team is the lack of Splunk training by both administrators and end-users.
(Woefully) Inadequate Splunk Training for Users and Administrators
Jim surveyed SP6’s 13 Splunk-certified Professional Services Consultants, who have collectively delivered more than 200 Splunk engagements, and asked the question, “What gets in the way of customer success?” The one point that almost every one of our consultants recognized as an impediment to success is insufficient knowledge and training around Splunk for both administrators and users, whether that’s the front end of Splunk (with SPL) or the back-end architecture and ongoing administration. Organizations simply don’t put enough emphasis on enabling Splunk users and administrators, to ensure the optimized value of the platform.
Collectively, our team has been involved in over 200 Splunk engagements. The answers below reflect what our team typically encounters.
Question: On a scale of 1 to 10 with 10 being the highest, please assess the overall Splunk capabilities of the typical Splunk USER that you encounter at customers.
Question: On a scale of 1 to 10 with 10 being the highest, please assess the overall Splunk capabilities of the typical Splunk ADMIN that you encounter at customers.
Our team rated the average Splunk user at just over a 3 out of 10 in terms of Splunk knowledge. Admins fared slightly better with an average score of 4.25 out of 10. It’s clear from their collective experience that there’s room for improvement in Splunk training at most organizations.
“If you think education is expensive, try ignorance.”– Derek Bok
Your organization isn’t alone. The first step in being more successful with Splunk is understanding that lack of Splunk training is a challenge for most organizations and then making a commitment to training your Splunk team. This can be difficult in a typical IT setting. You likely have other projects to work on, fires to fight, and other systems to maintain. You and your team will need to balance fighting fires with making strategic advances in your Splunk knowledge and practices, but it’s critical to make the commitment to improving the internal knowledge of Splunk. For Splunk users, in particular, there is a learning curve attached to Splunk’s SPL query language that can be cumbersome.
Solutions to Improve Splunk Training and Knowledge
Develop a Training Plan:
Manage training like any project plan that you might undertake. First, who will be using the Splunk platform and needs to be trained? Start out by listing out all of your Splunk Users and Admins and then document a formal training plan to execute upon, including timelines when specific training will be attained. Run it and prioritize it like any other project plan that’s important to your business. Track training progress to ensure that it actually gets completed. Encourage certifications and track those as well. Don’t let training, which is absolutely critical to enablement, adoption, and value realization, occur (or not occur) in a haphazard manner.
Leverage Free Splunk Training:
Splunk offers (2) formal EDU (education) classes that can be found via the Company’s website, which are foundational, building-block virtual courses to get people more familiar with Splunk. Splunk Fundamentals 1 (for Splunk users) and a Splunk Infrastructure Overview (for Splunk Admins), are “must-haves” for any of an organization’s employees who are new to Splunk. Our Professional Services team actually prefers that users take these courses before they begin an engagement so that users will understand the basics of Splunk right from the start.
Leverage Your Power User(s):
During this time, it’s also important to identify a power user or user(s) within your organization who has taken the lead with Splunk, dug into its capabilities, or otherwise embraced the platform. This type of person can be pivotal to the ongoing success of your training program, and with knowledge transfer to other Splunk users. This user can be leveraged to provide weekly lunch-and-learns or other internal training sessions.
Utilize Additional Available Resources for Splunk Training and Education:
There are additional community-based training resources that will significantly advance the knowledge of Splunk users and admins, and should absolutely be taken advantage of. Beyond better-known resources like Splunk answers, wikis, and user groups, there are a number of resources available to your team, many of which are completely free:
- Exploring Splunk Search Processing Language (SPL) Primer and Cookbook – Many people aren’t aware of this free and very helpful resource. This downloadable e-book was written by the third employee who was employed by Splunk. You can access it on Splunk’s website here.
- Splunk docs – This is a great resource for learning about specific Splunk features or capabilities. Some good examples for newer users include their SPL Cheat Sheet and About the Search Tutorial.
- Video – If you prefer videos to written content, Splunk Education also has a YouTube channel where you can learn about anything from basic searching to creating alerts and more. There are also both basic and more advanced e-learning sessions available from Splunk Education.
- The SP6 Blog – Time and time again, we meet Splunk users and admins who use our blog resources to improve their Splunk environments and learn how to take the next step with Splunk. If you’re haven’t done so already, visit our blog and subscribe to receive weekly updates with content created by our Splunk Professional Services consultants. Our stated objective of SP6 content is that the content is not simply marketing “fluff, but rather incredibly rich content that can help advance the knowledge of Splunk users, administrators, and architects.
5. Run In-house Workshops:
All-day, in-house user workshops can be accomplished in two ways. First, you can leverage your Splunk power user to share his or her experiences. You can also leverage partners or Splunk themselves. Splunk Solutions Architects (essentially, technical engineers attached to their Sales Engineering team) will come work with your team and provide onsite workshops for your team.
6. Start Building Training and Knowledge Transfer into Professional Services Engagements:
On most Professional Service engagements, organizations are concerned with activities that are certainly critical to Splunk; for instance, adding new data sources. In some cases, a customer may ask us to get as many data sources into Splunk in a 40-hour block of time as possible. Those same customers may not ask for any front-end analytics (reports, dashboards, queries, or alerts) attached to those data sources. That is generally a mistake, as the front-end analytics are most valuable with Splunk.
More to the point, once data sources are ingested, we suggest a block of 4 hours be built into a Services engagement with Splunk users or admins, whereby formal knowledge transfer takes place between our Splunk SME and those people – with their own data, in their own environment. Splunk users and admins need to be made aware of what to do with Splunk when PS departs, and blocking 4 hours of Professional Services engagements for group training and knowledge transfer is an extremely valuable and effective use of time.
Splunk Training is an Ongoing Process
Remember, Splunk training is not an event, it’s an ongoing process. Even if you have accomplished initial Splunk training, your team should continue to carry out weekly lunch and learns or other training sessions where they can explore questions that are relevant to them and also get answers on some of the specific challenges that they’re encountering within your organization’s environment.
You may still include general Splunk training as a part of your ongoing training sessions, but you may not rely solely on it. This is where your power user, who is most familiar with Splunk and SPL can take the lead in sharing his or her Splunk experience and knowledge to help your team make the most of its environment. Finally, if you simply don’t have enough internal expertise to keep training moving forward or to manage other aspects of your Splunk environment it may be helpful to engage a Managed Splunk Services provider who can assist with the ongoing “care and feeding” of your Splunk environment.
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.