A Beginner’s Guide to Splunk App/User Context Configuration Files

By Eric Levy, Splunk Consultant

This article will give Splunk admins a beginner’s guide to Splunk App/User Context Configuration Files. In a previous post, I went over several Splunk configuration files that one would find in a global context – independent of any apps or users. As a quick refresher, configuration files in Splunk (or “conf files”) dictate almost all settings in a Splunk environment. Splunk can have several of the same conf files in several directories, and “merge” via precedence rules. These rules differ depending on the context. Here, we will discuss Splunk conf files in the app/user context.

When is the App/User Context Applicable?

This context applies when editing conf files that pertain to a certain app or dealing with knowledge objects/actions owned by certain users. Search related activities is most common to this context, and most conf files discussed will have direct effects on search related processes.

App/User Context Conf File Precedence

Conf files can exist in default or local directories within apps or the overall system. Generally, local settings take precedence over default settings, and it is recommended users make changes in local directories. Further ordering depends on global or app/user context. For the app/user context, the baseline precedence order is as follows (taken from Splunk documentation):

  1. User directories for current user (/opt/splunk/etc/users/) – local, then default
  2. App directories for current running app (/opt/splunk/etc/apps/running_app_name)– local, then default
  3. App directories for other apps (/opt/splunk/etc/apps/other_app_name/) – local, then default
  4. System directories (/opt/splunk/etc/system/) – local, then default

For example, if I had savedsearches.conf files stored within:

  1. /opt/splunk/etc/users/admin/search/local/savedsearches.conf
  2. /opt/splunk/etc/apps/search/local/savedsearches.conf
  3. /opt/splunk/etc/apps/search/default/savedsearches.conf
  4. /opt/spunk/etc/apps/TA-splunksearches/local/savedsearches.conf
  5. /opt/splunk/etc/system/local/savedsearches.conf

The first settings to take precedence would be the ones set in the admin user directory, as they would have any options set per user. Then, any settings set locally within the running app (in this case, search) would take precedence, followed by the app’s default settings. That is followed by settings in any other app contexts, and finally settings in the system directories that would apply if not overridden by a higher precedence file.

App directory names can determine additional precedence. For app/user context, Splunk uses reverse-lexicographical order for priority. For example, a file that starts with the letter “Z” has a higher precedence than a file with the letter “A”.

Here are some common conf files found in the app/user context in Splunk:

ALERT_ACTIONS.CONF

alert_actions.conf provides settings for saved search actions. Different options exist for configuring alerts to send emails, RSS notifications, scripts, scheduled searches, and custom alert actions.

The [default] stanza dictates settings for all alert_actions within the conf file but can be overridden by individual alert_action stanzas.

Here are some settings commonly configured in [default]:

maxresults = <integer> – the max number of search results sent through alerts

hostname = [protocol]<host>[:<port>] – the hostname in the URL sent through alerts

maxtime = <integer>[m|s|h|d] – the allowed amount of time for an alert to execute before it’s aborted

command = <string> – the search command that executes the action

label = <string> – sets the label in the UI for custom alert actions

description = <string> – sets the description in the UI for custom alert actions

FIELDS.CONF

fields.conf provides settings for Splunk to handle indexed and extracted fields, multivalue fields, and how search time handles field values.

Again, the [default] stanza dictates settings for all fields that can be overwritten by individual field stanzas.

For local settings, the stanza name may be the field name itself, or a wildcard expression for a certain sourcetype formatted like this: 

sourcetype::<sourcetype>::<wildcard>

Additional settings:

TOKENIZER = <regex> – indicates a field value is part of a larger token (pertinent to multivalue fields)

INDEXED = <Boolean> – specifies if the field is created at index or search time

INDEXED_VALUE = several options:

  • Set to “true” if the value is in the raw event text
  • Set to “false” if not
  • Can also set to a sed-cmd string to substitute a field value
  • Can also set to a simple substitution string
MACROS.CONF

macros.conf provides settings to configure search macros, or reusable pieces of SPL referred to by a name between two backticks. This makes it very easy to use the same search strings in several different searches. Arguments can be passed into macros for more flexibility, and macros can contain eval statements.

Each stanza in macros.conf is specific to each search macro, and its naming convention is the name of the macro unless specified otherwise.

Available settings:

Available settings:

args = <string>, <string>, … – any arguments passed into the macro

definition – the search terms referred by the macro. Any arguments must be separated by dollar signs.

validation = <string> – verifies that the macro has appropriate arguments. Should return NULL if valid, or a Boolean/string if false.

errormsg  = <string> – the error message returned by the macro if passed bad arguments and the validation setting is a Boolean statement.

Iseval = true|false – specifies if the definition is an eval expression that returns a string of the expanded macro

description = <string> – optional setting to describe what the macro does

PROPS.CONF

I went over props.conf in the global context in my last post, but this file can exist in both the global and the app/user context, depending on what settings are being used.

Use the EXTRACT or REPORT settings in props.conf to create search time field extractions. Typically, EXTRACT is the setting to use, but REPORT can be used across multiple sources/sourcetypes/hosts, to apply multiple extractions, to set up delimiter extractions, multivalue fields, to extract fields that start with numbers or underscores, and to manage formatting.

Example formatting: EXTRACT-hostname = \d{4}\sHostName=(?<HostName>\w{5})\s

Use the rename = <string> setting to rename a sourcetype at search time.

SAVEDSEARCHES.CONF

savedsearches.conf provides possible setting/value pairs for saved searches. The standard settings, under each stanza representing a saved search, are:

disabled = <Boolean> – is the search enabled or not

search = <string> – the SPL for the saved search

dispatchAS = [user|owner] – who does the search run as?

There are settings for additional configurations of saved searches including scheduling, notifications, workload management, and event actions.

TRANSFORMS.CONF

This file is used in tandem with props.conf to enact more complex regex transformations at both index time (global context) and search time (app/user context).

In a global context, transforms.conf uses regex to override host and sourcetypes values, mask sensitive data, create index time field extractions, and set up fields from lookup tables.

The app/user context uses regex or delimiters to help create search time field extractions. When the extraction occurs depends on which Splunk instance includes the transforms.conf file – for the app/user context, put transforms.conf on a search head. Settings for search time extractions include:

REGEX = <regex> – the regular expression used

FORMAT = <string> – required for index time extractions, but optional for search time extractions. Specifies formatting, particularly using $<number> to specify the capture group to output the regex.

SOURCE_KEY = <string> – defines the field(s) where Splunk applies the regex – the default is _raw

DELIMS = <quoted string list> – only applicable for search time extractions, to extract fields based on a delimiter separating field values

FIELDS = <quoted string list> – gives the option to provide field names to extracted fields

For further reading, here is a full list of Splunk conf files, and once again, here is a list to my original blog post about global conf files. Happy Splunking!