Things to Ponder: Splunk conf files -> Lessons learned in school and why they should apply to programming
In a Splunk environment, Splunk operations are how a specific Splunk instance performs its assigned functions. Splunk operations are controlled by configuration files (a.k.a. conf files).
Typically, these files are located in one of several file/folder locations in any Splunk instantiation (Splunk Enterprise or Splunk Universal Forwarder). For example, in a Splunk Enterprise installation on a Linux OS compute engine, some typical folder locations for conf files are:
/opt/splunk/etc/system/default
/opt/splunk/etc/system/local
/opt/splunk/etc/apps/<app name>/default
/opt/splunk/etc/apps/<app name>/local
The general folder structure is similar for a Windows installation:
C:\Program Files\Splunk\etc\system\default
C:\Program Files\Splunk\etc\system\
C:\Program Files\Splunk\etc\apps\<app name>\default
C:\Program Files\Splunk\etc\apps\<app name>\local
The folder location for Splunk Universal Forwarder locations is the same, except replacing the word “splunk” with “splunkforwarder”.
These conf files turn on or off certain functions, based on what they control. For example, conf files determine whether the web user interface (UI) is visible, whether an input config is turned on to accept data and whether a setting of using encryption is in force.
These config file settings are essential to the proper running of the Splunk instance. Incorrectly configuring a setting can have wide-ranging effects, including the Splunk instance not performing a task in the expected manner of the instance not even being able to run at all. In the worst-case scenario, data may be modified, deleted, or lost. Proper configuration settings in conf files are of utmost importance, to say the least!
Things to Ponder: Splunk conf files -> Lessons learned in school and why they should apply to programming
conf files: Foundation
Journey back in time for a moment, to sometime in the elementary school days of your youth. Somewhere, in an English language class we were all required to take, we were informed of a few universal language truths. One such truth was the strict observation that when writing directions, instructions, or general guidance, one should refrain from the use of double-negatives. “In standard English, each subject-predicate construction should only have one negative form.” (1) What that means is that a negative form is created by adding a negation to the verb.
Some correct English examples:
The affirmative assertion: I will allow all data to flow. → simple statement to DO something
The correct negative assertion: I will not allow all data to flow. → simple statement to NOT do something
According to Grammerly.com, “A double negative is a non-standard sentence construction that uses two negative forms.” (2) Double negatives are created by adding a negation to the verb and to the modifier of the noun (adjectives, adverbs, etc.) or to the object of the verb; it’s like saying “no” twice!
An example of a double negative:
A double negative assertion: I will not allow no data to flow. – a confusing way to DO something (I’m not going to not do it)
So why is this relevant to Splunk one might wonder? It is because, within a majority of Splunk configuration files, there can be stanzas (bounded by “[“ and “]” brackets), which have directional settings applied to them using positive or negative assertions. Stanzas can also be thought of as “classes” from a programming frame of reference. One such setting often seen under various stanzas is the setting of “disabled”
conf files: Exposition
Splunk conf files use choices in settings to control pieces of Splunk instance operations. The purpose of a choice or selection in a Splunk conf file is to allow or disallow or select a specific condition or operation to occur. The choice of words for that selection is varied and can be the genesis for the casual reader to select an operation not consistent with their desired outcome.
According to techterms.com, “Boolean, or Boolean logic, is a subset of algebra used for creating true/false statements…Since computers operate in binary (using only zeros and ones), computer logic can often be expressed in Boolean terms. For example, a true statement returns a value of 1, while a false statement returns a value of 0.”(3)
One such control is whether a particular setting is enabled or disabled. In some conf files, Splunk uses simple English language textual values such as a “true” or “false” choice to indicate if a setting is allowed to be enabled or disabled. In other Splunk conf files, it uses the machine or computer Boolean coding of a zero “0” or a one “1” to indicate the choice. There are even a few Splunk conf file operations that use “yes” or “no” choices to turn some operation or function on or turn it off.
Exploration
A typical example of a Splunk conf setting stanza (could also be called a “class”) is found in the transforms.conf conf file:
transforms.conf
[default]
CAN_OPTIMIZE = [true|false]
* Optional. Controls whether Splunk software can optimize this extraction out
(another way of saying the extraction is disabled).
* NOTE: This option should be rarely set to false.
* Default: true
In the above sample, line #4 of the setting under the [default] stanza CAN_OPTIMIZE illustrates the options available to use with it; “true” or “false”. Line #8 indicates what the “default” selection is. This seems to present a simple, unambiguous selection with absolute clarity of purpose: CAN_OPTIMIZE is true (or enabled, if you will).
During my undergraduate college years, in my academic computer programming courses, I was exposed to the concepts of Boolean logic. When applying that knowledge to Splunk conf file settings, I remind myself what the settings of a “1” or a “0” translate to in English verbiage. A “1” is the affirmative (the equivalent of a “yes” or an “enabled”), while a “0” represents the opposite or negative affirmation.
In this next example, the choices are presented in Boolean form.
inputs.conf
[http]
disabled = [0|1]
* Whether or not the event collector input is active.
* Set this setting to "1" to disable the input, and "0" to enable it.
* Default: 1 (disabled).
In this example, line #4 of the setting under the [http] stanza illustrates the “disabled” options available to use with it; in this case “0” or “1”.
Line #7 indicates the “default” selection. Does the “1” indicate that this setting is indeed turned on, or is it disabled? This selection can provide less clarity, depending on the reading. If the selection is “0”, meaning the negative assertion, one could read it as “This setting of ‘disabled’ is NOT turned on, therefore, it is enabled” or “This assertion is a negative, therefore I am not enabling the stanza”; both of which are correct, but confusing!
In yet another example (though infrequently seen in later versions of Splunk), the terms “yes” or “no” can be used to present the selection choice. A sample of such a setting is indicated below.
inputs.conf
[tcp://<remote server>:<port>]
listenOnIPv6 = [no|yes|only]
* Whether or not the input listens on IPv4, IPv6, or both
* Set to 'yes' to listen on both IPv4 and IPv6 protocols.
* Set to 'only' to listen on only the IPv6 protocol.
* Default: The setting in the [general] stanza of the server.conf file
With this example, the “yes” or “no” selections add to the vagueness of the final choice when combined with the presumptive positive selection of the IPv4 choice. The descriptive text does not even address what the result would be for the choice of “no”!
Having three different potential ways to indicate an affirmative or negative setting assertion adds to the potential for misconfiguration in a conf file. Too often, the choices can be applied in a “double negative” fashion; that is, indicating a “disabled” or negative condition with a negative assertion such as “false”. Thus “disabled = false” really translates to “that setting is not being disabled; it is enabled !” This is the exact “double-negative” that our English teachers tried to impress upon us as being unworthy of proper English prose and voice. It indicates neither clarity nor precision in its presentation.
conf files: Conclusion
If I were able to go back in time, I would bring my 4th grade English teacher to the code room where Splunk was being developed. I would ensure that all coders, for every section and sub-section of the entire code-base listened to, heard, and understood the concept of double-negatives. They would hopefully be enlightened as to how much joy (sarcasm) it presents as it amplifies the ambiguity of Splunk configuration settings when using double negative assertions. Please coders and developers, use “enabled = true” or “enabled = false” !!!! Make it clear for everyone !!!
Today, even a thorough reading of each conf file’s spec file sometimes fails to provide a succinct and clear understanding of the purpose and result for a choice in some settings. I am not an English major, but some concepts were planted deeply in my brain. Simplicity and clarity of language enhance a coder’s ability to generate useful, meaningful, decisive code, as well as help those of us tasked with understanding and using it to better perform our tasks.
About SP6
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.
(1),(2) – https://www.grammarly.com/blog/3-things-you-must-know-about-double-negatives/
(3) – https://techterms.com/definition/boolean
