How to Dynamically Extract and Set Field Names in Splunk
By Brent Mckinney, Splunk Team Lead
This blog demonstrates how to dynamically set field names based on values in your Splunk events, and set them to corresponding field values. This requires a unique configuration, as opposed to doing traditional field extractions with static field names.
Let’s Get Started
For this example, let’s say we have the following events with the sourcetype test_sourcetype:
11/30/2023 01:34:30 field_name=”response_code” field_value=”200″
11/30/2023 02:30:39 field_name=”response_code” field_value=”200″
11/30/2023 03:36:47 field_name=”response_code” field_value=”404″
11/30/2023 04:39:31 field_name=”response_code” field_value=”500″
11/30/2023 05:34:55 field_name=”response_code” field_value=”200″
11/30/2023 06:32:21 field_name=”response_code” field_value=”200″
11/30/2023 07:39:54 field_name=”response_code” field_value=”200″
11/30/2023 08:38:29 field_name=”response_code” field_value=”200″
11/30/2023 09:36:34 field_name=”response_code” field_value=”404″
11/30/2023 10:31:01 field_name=”response_code” field_value=”500″
By default, Splunk will extract these fields as follows:
field_name = response_code
field_value = 200
But what if we want the fields extracted like this?
response_code = 200
Rather than creating a custom, static field name, and extracting a corresponding value, Splunk provides the ability to extract both the field name AND value, and apply them accordingly. This is done using field transformations applied to our sourcetype.
Field Transformations
The first thing we need to do is create a regular expression to capture the fields of interest. Our initial regex looks like this:
field_name=\”(?<field_name>\w*)\”\sfield_value=\”(?<field_value>\w*)\””
This regex extracts the desired fields, but by itself, sets them in 2 fields, field_name and field_value respectively. In order to have the value of field_name be our field name, and the value of field_value be the corresponding field value, we need adjust our regex to use specific syntax that Splunk expects for the transformation. This involves using KEY and VAL to extract and correlate the field/value pair.
Our regex now looks like this:
field_name=\”(?<_KEY_1>\w*)\\sfield_value=\”(?<_VAL_1>\w*)\””
The value of <string> specified in _KEY_<string> and _VAL_<string> will ensure the field name extracted is set to the field value extracted. In order to implement we can add our transformation:
Navigate to Settings > Fields > Field transformations and provide the following then Save.
Name: testing_extraction
Type: regex-based
Format: leave this blank
Regular expression: field_name=\”(?<_KEY_1>\w*)\”\sfield_value=\”(?<_VAL_1>\w*)\”
Source Key: _raw
Next we set our field extraction to the appropriate sourcetype, to use the above transformation. To start, navigate to Fields > Field extractions:
Name: test_extraction
Apply to sourcetype named test_sourcetype
Type: Uses transform
Extraction/Transform: testing_extraction
The above values should match the sourcetype of the data you are extracting these fields from. The value of Extraction/Transform MUST match the name of the Field Transformation. Once saved, running a search on the sourcetype will show the field names and values extracted and available to filter on.
Note, this can NOT be done directly in SPL at search time. It must be done via field transformation. You may also need to check permissions on both the field transformation AND field extraction so that users can use the new fields.
You should now be able to dynamically set field names based on values in your Splunk events, and set them to corresponding field values. As always, we are available if you have any questions – just fill out the form below: