Extracting Nested JSON Fields in Splunk
By Aaron Dobrzeniecki, Senior Splunk Consultant
Splunk, as we all know, is an analytical tool used to collect and evaluate data from different types of sources, helping organizations monitor their systems in real-time. One very common data format used is JSON. The data is structured in such a way that allows Splunk to extract the fields natively, granted you are working with perfect JSON. Nested JSON fields are a little trickier. To work with nested JSON fields in Splunk, you can easily extract them using props.conf and transforms.conf. This is a solution oriented blog post!
Please see an example event I will be using for this blog.
2024-12-10 17:57:16.998 INFO C2P { “id”: “1001”, “type”: “Regular” }, { “id”: “1002”, “type”: “Chocolate” }, { “id”: “1003”, “type”: “Blueberry” }, { “id”: “1004”, “type”: “Devil’s Food” } { “id”: “5001”, “type”: “None” }, { “id”: “5002”, “type”: “Glazed” }, { “id”: “5005”, “type”: “Sugar” }, { “id”: “5007”, “type”: “Powdered Sugar” }, { “id”: “5006”, “type”: “Chocolate with Sprinkles” }, { “id”: “5003”, “type”: “Chocolate” }, { “id”: “5004”, “type”: “Maple” }
When ingesting this data into Splunk JUST by using the Great 8 props.conf settings as well as KV_MODE=JSON, I do not get my nested JSON fields extracted. Please see my testing below:
[test]
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
TRUNCATE = 9999
SHOULD_LINEMERGE = FALSE
EVENT_BREAKER_ENABLE = TRUE
EVENT_BREAKER = ([\r\n]+)[\d\-]+
KV_MODE = JSON
After adding in a REPORT to my props.conf and adding a transforms.conf with this setting, my fields extracted accordingly. Please see my configuration below:
props.conf
[aaron_test]
LINE_BREAKER = ([\r\n]+)[\d\-]+
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3Q
MAX_TIMESTAMP_LOOKAHEAD = 23
TRUNCATE = 9999
SHOULD_LINEMERGE = FALSE
EVENT_BREAKER_ENABLE = TRUE
EVENT_BREAKER = ([\r\n]+)[\d\-]+
REPORT-nested = aaron_nested
transforms.conf
[aaron_nested]
REGEX = \”(\w+)\”:[\n\r\s]*\”*([^\”\,]+)
FORMAT = $1::$2
MV_ADD = true
The MV_ADD in the transforms.conf extracts the fields as multivalue fields so we can capture all values! NOTE! MV_ADD is only valid for search-time field extractions! The FORMAT setting specifies the format of the event, including any field names or values you want to add. If the REGEX setting does not have any capture groups, the matching fails.
As you can see, both the id and type fields get pulled out when I run my search. Instead of having to create field extractions for each and every value, you can do the field=value formula using props.conf and transforms.conf. Please see the regex used in the screenshot below:
In conclusion, extracting nested JSON fields in Splunk is essential for extracting valuable insights form complex data structures. By using props.conf and transforms.conf you can easily and efficiently extract nested JSON fields in your complex data. This capability allows for greater flexibility and precision when working with data. By using the REPORT in props.conf associated with a transforms.conf, we are able to modify our extractions WITHOUT modifying the indexed data.
Learn more here!
About the Author
Aaron Dobrzeniecki has just over 7.5 years of experience with Information Technology. He first worked for an MSP broadening his skills with Tier 1 and Tier 2 Help Desk support for just over 40 small companies. Aaron has been working with Splunk for 7 years starting on the apps and add-ons team troubleshooting issues with Splunk apps and add-ons. He then worked on the Admin on Demand team creating content, assisting with Splunk core issues or tasks. As well as providing Splunk Best Practices. He has provided excellent service to the Splunk customers he has worked with and continues to get praise from his customers. Aaron was also a Customer Success Manager for 14 companies working with their Splunk teams to better their environments. As a Certified Splunk Consultant, Aaron continues to increase his knowledge and follow the Splunk Best Practices while maintaining an excellence in communication. Aaron graduated with his Bachelors’ majoring in Security Informatics from Indiana University in 2015.