Case Study
SP6 Success program for Splunk
Case Study
Ivy League University
Summary
A large university system with over 35,000 faculty and students had a large (3+ terabyte of daily log data) and complex Splunk deployment being used for a wide variety of purposes: IT infrastructure monitoring, Information Security and DevOps.
This organization had significant issues supporting Splunk which prevented achieving optimal value realization.
SP6 was able to significantly:
- Improve required Compliance tied to data retention.
- Drive down the cost of Splunk licensing at renewal.
- Reduce organizational risk tied to the organization always being one resignation away from lack of Splunk management (and the downstream effects on users in security, IT operations and DevOps).
- Reduce the cost (via administrative time) and complexity of managing Splunk.
The Problem
Like many organizations, the sole employee managing Splunk was highly technical and proficient in certain areas (in this person’s case, Linux administration and network engineering). However, Splunk was not this person’s “bread and butter”. This individual, despite being responsible for Splunk, didn’t know the solution very well.
This part-time Splunk Administrator had inherited the system from a prior Admin who happened to be a developer that took on Splunk responsibilities. In doing so, this prior Admin overengineered Splunk. There were many unnecessarily complex, custom-built components with little documentation. That prior Admin moved within the University to another role.
Because Splunk was not within this employee’s core capabilities, and because they were managing an environment that was developed outside of best practices and extremely (and unnecessarily) difficult to maintain, the Splunk Admin was challenged with effectively managing the Splunk platform. Key challenges included:
- Data integrations – which are central and critical to log aggregation, analytics and SIEM solutions – would break, and there was no understanding of why (or how to remediate).
- Dropped data integrations pose significant risks to any organization, as the data being collected:
– Is required, in many instances, for Compliance purposes.
– Is no longer available for security detections or systems troubleshooting.
– At some point, this individual responsible for managing Splunk left the University. - There was an absence of in-house expertise to (a) simply Splunk’s engineering and (b) make it work efficiently.
These Problems Are Not Unique
The (3) biggest issues with this organization are quite common:
- Mission critical system which is single threaded through one individual.
- Splunk is not this individual’s core competency (lack of expertise, yielding poor engineering and use).
- The employee has other responsibilities in the organization; managing other systems and tools, leaving inadequate time to support Splunk.
The Solution
This University made the decision to partially outsource Splunk administration to SP6, through SP6’s Success Program for Splunk. They did this in conjunction with hiring another employee to own the system internally. The intent was to:
- Have one of the University’s most mission-critical technology solutions managed by Splunk experts.
- No longer be single threaded and vulnerable to staff changes.
- As a result, get better use from Splunk through better engineering
and use.
SP6 deployed their Objectives Based Methodology. This approach starts with the identification and prioritization of needs that would deliver the highest organizational impact. It flows to execution of those Splunk-related priorities.
SP6’s approach is also based upon an assigned Splunk SME that is dedicated (part-time) to the customer. This assured that the University benefitted from not only expertise, but consistent, intimate knowledge of the University’s systems.
The Results
Through the expertise of SP6 Splunk experts who were assigned to deliver part-time, monthly services on behalf of the University, SP6 was able to:
- Ensure Data Compliance.
Integrations are now stable and report legally required data to the University’s data store. - Significantly Reduce Future License Costs by up to 40%. The University was leveraging Splunk’s workload licensing model (SVCs). Prior to SP6’s involvement, poor engineering and misconfigurations caused searches to run improperly. Unnecessary searches were hammering server utilization, necessitating Splunk Cloud compute that – after SP6 re-engineering – was reduced by 40%.
- De-risk the University’s Compliance, Security, IT Ops and DevOps operations. The Splunk platform is no longer single threaded through one individual. Regardless of staff turnover at the University, SP6 ensures that there is consistent data collection and analytics that are the foundation of security detections and accelerated IT and application troubleshooting, providing better systems availability.
- Knowledge Transfer. The University is on their third in-house Splunk Admin in as many years. SP6 is helping up-skill that staff member through knowledge transfer. SP6’s intent is not for the University to be reliant upon our services; but rather, enable customers. In a large and complex Splunk environment, SP6 ensures that both University personnel and assigned SP6 team members together to drive value through this mission critical tool.
