Use Case Testing to Validate Your Notables
By Kyle Moreau, Cybersecurity Engineer II
You are in charge of maintaining notables in Splunk Enterprise Security and are tasked with validating all of the notables. This task will test the accuracy of the notable and how efficient the search is. When searching back through your correlation searches, you notice a few of them have never triggered a notable. You need to determine if this is because the activity is rare and hasn’t occurred yet (and hopefully never) or if there is something wrong with the search. You’ll need to find a way to test these notables and validate they will work if/when the activity occurs.
The first and most important step is to start by reviewing the search logic.
1. Determine if the index, source type, and fields exist in your logs.
2. Make modifications to match the search criteria with the logs in your environment.
3. Go into ES and review the correlation search settings.
Validate the search schedule:
Make sure it is set up to trigger a notable when the criteria is met:
Once you have determined the correlation search is set up properly and main components of the SPL are accurate, you can start testing the logic:
Review Historical data
Search further back than you normally would to trigger the notable. The simplest way to validate the SPL is by matching logs that are already in Splunk. If the search is broken up into bucks or defines earliest/latest, comment these out and run your search back.
For example, you find logs that match your search criteria from 30 days ago, you can modify your correlation search to look back to that day:
Modify the search
The historical search couldn’t exactly match your SPL; you might have success by making small changes to the search. Modifying thresholds, changing field values, adding to lookup tables are all methods that can help generate a match without completely losing the search integrity.
In this example we are looking for a count of over 10 IP Addresses user per user:
Within this time range and threshold, we get 0 results. For testing we can expand the time range, or change the threshold to get results:
Perform live testing
Generating activity from the source of your logs might not always be realistic, but it is likely the most accurate way to test a notable search. One of the best examples of this is a red team engagement with an internal team or 3rd party vendor hired by your organization.
In some cases, you can test generating logs necessary to trigger a notable by performing activity with the help of a product admin or network engineer. For example, you want to test a notable that fires when a user is added to a specific group in active directory. With appropriate permission/authorization, an admin can perform this action with a test user to generate the traffic.
Effective testing of notable searches not only enhances your Splunk environment’s reliability but also minimizes the risk of missing critical alerts. Whether you use historical data, make temporary changes to your search, or simulate events, each approach plays a valid role in building robust and actionable searches.
Want to optimize your Splunk environment further? Reach out here to learn about our extensive Splunk services and how we can support your goals.
About the Author
Kyle Moreau has over 8 years of information technology experience, 6 of which are within information security. As a security engineer, he leads projects for various security tools including SIEM and Endpoint. Maintained and developed solutions to improve the security posture of a fortune 500 organization. Kyle has successfully completed full ES implementations for multiple organizations. Previous experience as a security analyst, focused on incident response and investigation as well as keeping current in the latest attack methods.