A Beginner’s Guide to Splunk Global Configuration Files
By: Eric Levy | Splunk Consultant
Splunk is an invaluable tool for data analysis and provides flexible options to configure an environment tailored to the business using it. But beyond text boxes, drop-down menus, and radio buttons in the Web environment, or executing commands on the command line – how does Splunk log and make changes? Enter configuration files.
Configuration files (or “conf files”) – using the .conf file extension – are a series of files that dictate almost all settings in a Splunk environment. This includes data inputs, outputs, data modification, indexes, clustering, performance tweaks, and much more. Splunk deployments can have several conf files of the same name in various directories, and “merge” via precedence rules.
Different conf files exist in a global context and an app/user context, the latter of which typically are used for search related activities. Here we will talk about conf files used in the global context, although usage at search time will be touched upon where applicable.
CONF FILE PRECEDENCE
Conf files can exist in default or local directories within apps or the overall system. Generally, local settings take precedence over default settings, and it is recommended users make changes in local directories. Further ordering depends on global or app/user context. For the global context, the baseline precedence order is as follows:
- System local directory
- App local directory
- App default directory
- System default directory
EDITING CONF FILES
Settings within a conf file follow a header, attribute structure in separate stanzas. The header is indicated in [brackets], with attributes below them. Attributes are case sensitive, and settings may not apply if typed incorrectly. Values for attributes are not case sensitive.
To edit them, one must have file system access. Using vi or nano via the command line or a text editor is most common. To apply new changes in Splunk after editing a conf file, save it and then restart Splunk.
Here is some more information on some of the most common global conf files:
INPUTS.CONF
inputs.conf is used for specifying input data sources.
On a forwarder or single server, inputs.conf specifies data sources used in the deployment. On an indexer, inputs.conf specifies the port that a forwarder sends data (set in outputs.conf on the forwarder). The port used to send data from one Splunk instance to the next is typically 9997. On a search head, inputs.conf ingests internal logs by default.
The available settings in inputs.conf depend on the data source being imported. The header is always formatted as [data_type://<path>], where the data_type could be monitor input, tcp/udp (network input), scripted input, etc. The <path> is the file path, but this setting may also vary depending on the data_type.
Attributes include setting the data source’s host, index, source, and sourcetype. Specifying the sourcetype is important for processing in props.conf. Additional settings for specifying the host are available as well. Whitelist and blacklist options allow you to filter files to import within a path. Beyond these, each data_type has its own unique settings.
PROPS.CONF
props.conf provides several processing settings to modify data coming into Splunk. This file can apply either at index time (global context) or at search time (app/user context).
Sourcetypes are separated by stanza with their own list of attributes. On a forwarder, parsing is limited, but indexers and search heads can provide index-time and search-time parsing.
The “Big 8” settings, as recommended by Splunk Education, are as follows:
- LINE_BREAKER = (regex): Determines the start of a new line.
- SHOULD_LINEMERGE = (TRUE/FALSE): Typically set to false when used with single lined events, and not very efficient, but tries to clean up event boundaries.
- TRUNCATE = (integer): Specifies the maximum line length in bytes (set to 0, lines will not be truncated).
- MAX_TIMESTAMP_LOOKAHEAD = (integer): The maximum number of characters for a timestamp within an event.
- TIME_PREFIX = (regex): A prefix that Splunk looks for up until the timestamp begins.
- TIME_FORMAT = (format): Tells Splunk the timestamp’s format.
- EVENT_BREAKER_ENABLE = (TRUE/FALSE): Used in load balancing data to indexers on a universal forwarder.
- EVENT_BREAKER = (regex capture group): Specifies the event boundaries for multi-line events on a Universal Forwarder, if EVENT_BREAKER_ENABLE = true.
OUTPUTS.CONF
outputs.conf tells the Splunk instance where to forward data.
This file is required on a forwarder. Indexers don’t typically need one, but search heads can have it to forward logs back to the indexers. This is especially important in clustered environments.
The stanza header lists the type of output (tcpout, syslog, etc.), and then the target instance, separated by a colon. For example, tcp output going to indexer 1 would follow the format [tcpout:indexer1]. Overall output settings don’t include the target (e.g., [tcpout]).
Attributes for an overall tcpout stanza include:
- defaultGroup = (comma separated list) – Lists target group names.
- indexAndForward = (Boolean) – When set to true, automatically indexes data before forwarding it (only available on heavy forwarders).
Attributes for a target group tcpout stanza include:
- server = (comma separated list) – One or more IPs to send data over.
- indexerDiscovery = (string) – Set it to the IP of the manager node to enable indexer discovery in a clustered environment.
outputs.conf settings are not limited to tcpout – different types of outputs have their own unique sets of settings.
SERVER.CONF
server.conf configures a wide variety of server options in Splunk.
It includes a [general] stanza where one can configure the serverName, hostName, sessionTimeout, site (in a multisite cluster), parallelIngestionPipelines (for increased processing), and more.
Another important stanza in server.conf is [clustering], which determines an instance’s role in an indexer cluster. Settings include:
- mode = (string) – Sets the mode for this instance (manager, peer, searchhead, disabled).
- manager_uri = (uri) – The URI of the cluster manager.
- pass4SymmKey = (string) – A password set between cluster instances to successfully send data.
- rolling_restart = (string) – On the manager, determines how indexers restart during a rolling restart.
- multisite = (Boolean) – On the manager, determines if the cluster is in multisite mode.
- replication_factor = (integer) – On the manager, determines the number of copies of rawdata in a cluster.
- search_factor = (integer) – On the manager, determines the number of copies of searchable data in a cluster.
- available_sites = (list) – Lists the sites in a cluster (required for a multisite cluster).
Also included are options to configure SSL, search head clustering, KV Store configuration, and much more.
INDEXES.CONF
indexes.conf sets properties for Splunk indexes.
Each stanza represents a given index with attributes, but default settings can go in the [default] stanza.
There are several general settings, but here are some attributes that can be set on a per-index basis:
- disabled = (Boolean) – Turns an index on or off.
- deleted = true – If present, the index is set for deletion.
- homePath = (string) – Sets the path of the hot and warm buckets for this index.
- coldPath = (string) – Sets the path of the cold bucket for this index.
- thawedPath = (string) – If a frozen bucket is thawed, this specifies the path for that bucket.
- maxGlobalDataSizeMB = (integer) – The maximum size of a warm bucket, in MB.
- maxDataSize = (integer) – The maximum size of a hot bucket, in MB, before it rolls over to a warm bucket.
OTHER USEFUL GLOBAL CONF FILES
There are 56 unique configuration files in Splunk and nearly 30 of them are global – too many to discuss here! They are all important in their own ways, but here are a few others to take note of:
- authorize.conf.- Handles user roles and their permissions.
- collections.conf – Handles KV Store settings for apps.
- deploymentclient.conf – Handles settings for clients of a deployment server.
- limits.conf – Handles limits for search commands.
- serverclass.conf – Handles creating different groups of deployment clients on a deployment server.
- transforms.conf – Handles data transformations (typically with regex), in tandem with props.conf. Can exist in both the global and app/user contexts.
For further reading, this is a full list of Splunk conf files, and here is more on how to edit a conf file.
Happy Splunking!