File system permissions in Linux, the suggested operating system for Splunk, can be worrisome for Splunk administrators. Oftentimes, a Splunk Administrator needs correct permissions to read system logs; however, by default in Linux, these permissions are not available, or only available to the one Linux superuser account.
To work around this, a common configuration in the field for Linux-based systems is where the root is configured as the user to run the Splunk system. This configuration alleviates the complaints that the Splunk non-root user does not have the proper permissions to access and view the data. Naturally, running the program as the root user will allow access to the files that are being read.
The drawback in doing this is that running Splunk as the root user creates a security vulnerability for these systems. Due to this, best practices from Splunk Professional Services suggest not running Splunk as the root user. Surprisingly, a large number of Splunk consulting engagements that I have been on have ignored this security risk. This article suggests ways to reduce the Splunk security risks associated with root access, stay in bounds on Splunk best practices, while simultaneously providing Splunk Administrators the access to the data they need access to.
The traditional fix that does not involve running the Splunk software as root is to chmod the files or directories with permissions that can be set for the owner, group, and everyone. It is not uncommon to manipulate the group ownership of a file so that the non-root Splunk user can view and subsequently index its contents. While that solution works, there is a large risk of causing other issues on the system.
What next?
Thankfully, the getfacl and setfacl commands are available and allow directory/file access control for file access. The command runs as root or as an authorized pseudo-user. Individual users and/or groups can be given read, write and/or execute permissions.
setfacl -m u:splunk:rx <full path and filename>File permissions fix
The setfacl command however only solves the issue if you are aware of which files needed to have their permissions modified before deploying your configuration. To fix the file access permissions after the fact can be done with a search of the splunk.log using Splunk for the host in question or a script can be run like the one below.
The script below is a great example that will help to alleviate the permissions concerns. The script works by reading the splunkd.log and picking out the files that have been reported with insufficient permissions. The script will then process the command above with the spunk user as the intended recipient of the new file permissions. Bundled into the script are a couple of options that allow setting:
- Splunk location (Enterprise or Universal Forwarder)
- SUF=”/opt/splunkforwarder/var/log/splunk”
- #
- SDEF=”/opt/splunk/var/log/splunk”
- Debug (logged output)
- DEBUG=1
- Test Run (additional verification step if needed)
- TEST=1
The ideal location to run the script is from the /tmp directory and as the root user account. Logic has been added into the script where each file is tested for existing permissions. If the permissions already exist a message is logged with that information.
The Script:
If you’d like to copy and use this script, download the text version here.
#! /bin/bash
#
# This script will modify the acl rights of files retrieved from
# the /opt/splunk/var/log/splunk/splunkd.log
# that need to allow the [splunk] user to have read and execute access.
#
#
#
####################### script initialization ##
#
# To run on splunkforwarder (UF) or a custom path
# set set value to the [splunkd.log] path for this installation.
# If path exist then log will be read from that location.
#################################################
#
SUF=”/opt/splunkforwarder/var/log/splunk”
#
SDEF=”/opt/splunk/var/log/splunk”
if [ -d $SUF ]
then
grep permission $SUF/splunkd.log | awk -F “‘” ‘{print $2}’ > /tmp/splperm
else
grep permission $SDEF/splunkd.log | awk -F “‘” ‘{print $2}’ > /tmp/splperm
fi
#
touch /tmp/aclresults
touch /tmp/acltest
FILES=”/tmp/splperm”
rFILES=”/tmp/aclresults”
aclEXIST=”/tmp/acltest”
INPUT=`cat $FILES`
###############################################
######################## script options #######
#
# To see full before and after results per file set value to 1
DEBUG=1
# To initialize a test run and verify correct files set value to 1
TEST=1
###############################################
######################## main script ##########
echo `date` > $rFILES
for f in $INPUT
do
getfacl $f | grep splunk > $aclEXIST
if [ ! -s $aclEXIST ]
then
echo `date` ” Processing splunk user rights for $f” >> $rFILES
if [ $DEBUG -eq 1 ]
then
echo “Before:” >> $rFILES
getfacl $f >> $rFILES
fi
if [ $TEST -eq 0 ]
then
echo “This is not a test run for $f” >> $rFILES
setfacl -m u:splunk:rx $f
else
echo “This is a test run for $f” >> $rFILES
fi
if [ $DEBUG -eq 1 ]
then
echo “After:” >> $rFILES
getfacl $f >> $rFILES
fi
else
echo `date` “Splunk user rights for $f not needed” >> $rFILES
fi
done
################### main script end ########Review
File permissions can be a big problem when trying to index data into Splunk. The steps to mediate the issues can range from simple to difficult. Using chmod may open other issues with the system and its overall security. Using the getfacl and setfacl commands expand on the chmod command in Linux allowing per-user access, while also allowing greater control over file permissions and access. These commands make them the preferred method of balancing Splunk Administration and system controls.
Using the script above automates the repair process that will allow files with permissions errors to be indexed into the Splunk environment. The script goes a little further by providing output logs that can be audited if the script is to run on a schedule or on-demand.
As always, Splunk Answers can the go-to source for help and further information on this topic can be found on Splunk Answers (https://answers.splunk.com/index.html). Splunk professional services firms such as SP6 are also always here to help Splunk customers with these and many other issues, to make sure that your Splunk instance is optimized and driving significant business value for your organization.
This article was authored by Cedric Milan, a senior Splunk Professional Services consultant with SP6. Cedric is a Splunk Certified Consultant II, the highest level of certification in the Splunk domain. SP6 is a leading Splunk business partner, assisting customers with software acquisition and Splunk consulting services.
About SP6
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.
