Who Updated My Search?
By Kyle Moreau, Security Engineer Consultant II
As a Splunk administrator, it is important to be able to audit changes performed by your users. Sometimes tracking down these changes is not very straight forward. Splunk’s solution for viewing changes made to configurations comes in the form of the _configtracker index. This index logs configuration changes that were made in the UI. It includes important information like the stanza, file name and path, and the date/time the change occurred. Some limitations of this index are the user who made the change isn’t present and the changes are sometimes split into multiple logs (depending on where the change were made).
Below is an example showing a simple change to a saved search. These two logs show the before and after fields for a single stanza:
Old Values:
New Values:
Were you able to spot the change? It might not be so easy find a configuration change if you are looking at a stanza for something like a correlation search in Splunk Enterprise Security, which contains many additional properties. To fix the issue of multiple logs for a single configuration change, use the search below (please note this, and all following searches, specify the “savedsearches” conf file):
| index=_configtracker sourcetype=”splunk_configuration_change” data.path=”*savedsearches.conf” |
| | spath output=modtime data.modtime |
| | spath output=path data.path |
| | spath output=stanza data.changes{}.stanza |
| | spath output=name data.changes{}.properties{}.name |
| | spath output=new_value data.changes{}.properties{}.new_value |
| | spath output=old_value data.changes{}.properties{}.old_value |
| | stats values(host) as host, values(stanza) as stanza, values(name) as name, values(old_value) as old_value, values(new_value) as new_value by modtime, path |
The above search pulls the fields together and combines the logs using the “modtime” field. This will display a single stanza configuration change with all the stanza information the log contains. To view just the change, combine the name and old/new_value fields, compare each line of the stanza, and output the differences:
| index=_configtracker sourcetype=”splunk_configuration_change” data.path=”*savedsearches.conf” |
| | spath output=modtime data.modtime |
| | spath output=path data.path |
| | spath output=stanza data.changes{}.stanza |
| | spath output=name data.changes{}.properties{}.name |
| | spath output=new_value data.changes{}.properties{}.new_value |
| | spath output=old_value data.changes{}.properties{}.old_value |
| | eval old = mvzip(name, old_value, “: “) |
| | eval new = mvzip(name, new_value, “: “) |
| | stats values(host) as host, values(stanza) as stanza, values(old) as old, values(new) as new by modtime, path |
| | eval max_len = if(mvcount(old) > mvcount(new), mvcount(old), mvcount(new)) |
| | eval indices = mvrange(0, max_len) |
| | mvexpand indices |
| | eval left = mvindex(old, indices) |
| | eval right = mvindex(new, indices) |
| | where left != right OR isnull(left) != isnull(right) |
| | stats values(host) as host, values(stanza) as stanza, values(left) as old_value, values(right) as new_value by modtime, path |
| | sort – modtime |
Now we have a good idea of what configurations were changed and where. This final step will pull in the user that made the change, to make a truly powerful search. The logs containing user changes exist in the index _audit. In this example, a join is used to pull the user information out of _audit index, using the modified time as the common field. The _configtracker modtime and the audit log timestamp fields should both match up to the nearest second.
| index=_configtracker sourcetype=”splunk_configuration_change” data.path=”*savedsearches.conf” |
| | spath output=modtime data.modtime |
| | spath output=path data.path |
| | spath output=stanza data.changes{}.stanza |
| | spath output=name data.changes{}.properties{}.name |
| | spath output=new_value data.changes{}.properties{}.new_value |
| | spath output=old_value data.changes{}.properties{}.old_value |
| | eval old = mvzip(name, old_value, “: “) |
| | eval new = mvzip(name, new_value, “: “) |
| | eval modtime = strptime(modtime, “%a %b %d %H:%M:%S %Y”) |
| | join modtime type=left |
| [ search index=_audit sourcetype=audittrail |
| | where match(action, “(?i)edit|update|add|create|modified”) |
| | eval modtime = strptime(timestamp, “%m-%d-%Y %H:%M:%S”) |
| | where isnotnull(user) |
| | stats count values(user) as user by modtime |
| | fields – count] |
| | stats values(host) as host, values(stanza) as stanza, values(old) as old, values(new) as new, values(user) as user by modtime, path |
| | eval max_len = if(mvcount(old) > mvcount(new), mvcount(old), mvcount(new)) |
| | eval indices = mvrange(0, max_len) |
| | mvexpand indices |
| | eval left = mvindex(old, indices) |
| | eval right = mvindex(new, indices) |
| | where left != right OR isnull(left) != isnull(right) |
| | convert ctime(modtime) |
| | stats values(host) as host, values(stanza) as stanza, values(user) as user, values(left) as old_value, values(right) as new_value by modtime, path |
| | sort – modtime |
You can now see the full picture and track down configuration changes on the fly. If you have any questions about diving deeper into auditing your environment, reach out to us.
About the Author
Kyle Moreau has over 8 years of information technology experience, 6 of which are within information security. As a security engineer, he leads projects for various security tools including SIEM and Endpoint. Maintained and developed solutions to improve the security posture of a fortune 500 organization. Kyle has successfully completed full ES implementations for multiple organizations. Previous experience as a security analyst, focused on incident response and investigation as well as keeping current in the latest attack methods.