Professional Services (PS) time delivered by external Splunk consulting teams is valuable, so it’s important for organizations to maximize the amount of work-time delivered by service providers. Here are some tips to ensure that you get the most out of your Splunk professional services engagements. These tips include:
- Have a well-defined and communicated State of Work / Work Order (SOW)
- Have IT access and environment ready
- Trust but verify
- Build a development (dev) environment
- Designate administrators who have the ability to make changes
- Set a cadence for status reporting
- Set expectations for communication
- Use PS as a means to knowledge transfer to Full-time staff
- Use technology for sharing and archiving data
- Require project documentation
Have a well-defined and communicated SOW – and one that is reviewed by the PS consultant that will actually be performing the work
This goes without saying, right?
Despite the simplicity of this best practice, problems can arise in the following scenarios:
- The SOW is poorly written (not enough detail)
- The SOW was written by someone other than the person who will be performing the services work
The second scenario above is very common. Many times, the SOW may have been written by a technical sales engineer and then handed off to a post-sales services team. Our professional services consultants have kick-off calls with every client the week prior to starting the project, so they can determine exactly what you hoping to accomplish through the project. At times, there can be differences between the SOW (or the SOW was poorly written) and what is elicited in these kick-off calls.
A well-written and agreed upon SOW will also form the basis for determining project success, as well as minimizing scope creep. It’s important to stay on point and not let too many “oh by the ways” enter the project, at the risk of derailing the original work that was expected.
At the end of the project, there needs to be a starting point to tie success back to. A well-defined SOW with strong communication from both sides is key for success during the engagement.
Have the IT Environment Ready
Make sure the Splunk consultant can hit the ground running is to have your environment ready. This includes:
- Setting the consultant a valid Active Directory account
- Allowing the consultant access to the file system
- Provisioning the hosts (hardware and software that Splunk will reside on)
- Opening the firewall ports
- Secure a development environment
For Splunk Enterprise Security (ES) engagements:
- Provide a list of IT assets and identities
Trust But Verify
It’s common for a systems administrator to ask their networking peers to open up a firewall rule and assume that task has been completed. The administrator should always check that this was actually completed before the professional services team arrives. This can be done by installing telnet on the source host and checking if the socket is open on the remote host. Not checking this beforehand can result in days lost due to change management.
Installing telnet
Successfully connected on that IP & Port
Blocked on that IP & Port
Build a Development (Dev) Environment
As the old saying goes, you should never test in production. Your dev environment can be at a much smaller scale than your production environment and even run on virtual machines. You should always test your changes in this environment before applying to production. Make sure you have a dedicated development environment ready before professional services arrive. It’s also good to know that you could also include this in the SOW if you need one built.
Designate Administrators Who Have the Ability to Make Changes
If you choose to have a Splunk professional services provider such as SP6 deliver work, there should be a dedicated systems administrator available who can make changes to accomplish objectives in the SOW. In small to medium-sized organizations, this designated technical point-of-contact should be able to make these changes. In larger organizations where a formal Change Management (CM) process exists, this person will help run point on making requests and navigating the CM process.
Set a Cadence for Status Reporting
Splunk professional services projects are typically what we would classify as short to medium-term in nature. Unlike, by contrast, ERP or traditional data warehouse or BI projects that can last quarters and years, Splunk projects tend to be measured in weeks or months. Some projects are as short-term as one to two weeks. The cadence for status reporting reflects these project durations. Our best practices:
- A brief, daily meeting between PS and the client every morning.
- On short projects such as a week or two, daily status reports should be emailed to the client point-of-contact; there is just too much that has to happen in too short a period of time to delay status reporting on these short assignments.
- On longer-term/larger team projects, there will typically be more integration with the client’s PMO or another type of group that has its own standards for status reporting. The integration with the PMO is usually because these larger/longer projects have more budget which requires more Project Management and oversight. Even in these environments, SP6’s PS staff documents a weekly status report.
- If the professional services firm is responsible for Project Management, then there are tasks for activities including Project Plans, Budget Burn Down reporting, Tasks/Activities, Risks and Risk Management, etc. The professional services firm needs to be responsible for reporting these items to a customer, beyond communicating status.
Set Expectations for Communication
There will always be contingencies that have downstream effects within projects; for instance, a professional services consultant may need access to a particular system and not be getting it, or the server team may be responsible for standing up an environment and that environment hasn’t been delivered.
A seasoned, mature services consultant knows to stay ahead of issues, specifically, communication and contingencies. Less hardened consultants may not have the consulting chops on knowing how to deliver bad news, wait until much later in the project, and then say “I didn’t get this done because __(fill in the blank of the dependency that failed)__”. How can you avoid these potentially adverse issues?
- Designate a point-of-contact within the organization to which the Splunk PS consultant should escalate these issues and dependencies
- Make clear to any consultants that you expect these dependencies to be communicated when they are first encountered – and not used as an excuse later in the project
On a shorter, for instance, two-week Splunk engagements, if a professional services consultant was not good about raising flags about roadblocks, a week could go by until they are raised. Roadblocks need to be immediately communicating and removed.
Use Professional Services as a Means to Knowledge Transfer to Full-Time Staff
A professional services partner is brought on-site to complete certain technical deliverables, right?
Well, mostly. In addition, there is a big area of opportunity to not only leverage PS expertise to perform certain product and technology-related tasks that they bear expertise in, but to knowledge transfer to customer’s full-time (FTE) staff.
- How is this best accomplished? In many cases, having the customer’s own employees taking the ‘steering wheel’ (keyboard) and being guided by the services consultant.
- It’s most beneficial for the administrator to do the Splunk configurations with the guidance of the Splunk PS consultant so they can learn by doing. When PS and your organization’s FTE are not physically in the same location, WebEx (or similar technology) can be used to allow the administrator to share their screen and gives the ability to view the screens of everyone else in the meeting.
While this may seem counterintuitive and could dent the productivity of an organization’s staff during this mentoring time, having the customer’s staff doing the work, under the guidance and stewardship of the professional services expert, really facilitates knowledge transfer. SP6 has had engagements where the client staff “drives” 100% of the time, typing what our consultant instructs them to type. In doing so, the leave-behind value is the customer having FTE staff that gained valuable instruction on, in this case, Splunk, and are capable of performing Splunk administration on their own.
This instruction also goes a long way toward a customer realizing the full benefits of the investment they are making in the product itself. In those instances when the work is performed by the organization’s staff under the direction and instruction of the services consultant, clients typically see greater use and adoption of, and value extracted from, their investment in Splunk. When this hands-on work and instruction do not occur, we’ve witnessed adoption being very slow.
Use Technology for Sharing and Archiving Data
Having access to collaboration tools while on an engagement is beneficial for both parties because it allows the professional services consultant and administrator to share files, Splunk searches, code samples, and historical information in real-time without having to worry about being constrained by things such as file access, file size limitations across email systems and other constraints. Splunk Professional Services tool of choice is HipChat. HipChat archives information about the previous days which could be easily forgotten and also makes it easy to export the entire conversation and send it to the client after the engagement is over.
Require Project Documentation
Most technology practitioners aren’t fond of documentation. Your best consultants, however, realize that proper documentation is critical, for common-sense reasons. Your own FTE staff will be responsible for the administration of these systems long after professional services work is completed. Ensure that the work that PS completes is properly documented, including technology diagrams.
About SP6
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.
