Taking Action with Workload Management in Splunk Cloud
By Aaron Dobrzeniecki, Team Lead, Support and Kamal Dorairaj, Senior Splunk Consultant
If you’ve ever used Splunk, you already know one important fact: it’s awesome! However, during peak load times—when everyone is running searches and ingesting logs into their indexes—things can slow to a crawl. This becomes frustrating for both users and admins: searches get queued, and even simple queries take far longer than they should. If only there were a way to limit resource usage for user searches and abort inefficient searches quickly. Good news—there is!
Workload Management (WLM) is Splunk’s powerful tool that lets you control resource allocation. In this post, we’ll discuss two Admission Rules and one Workload Rule.
Admission Rules
Admission rules automatically filter out searches before they start, based on user-defined conditions.
No ALLTIME Searches
This rule prevents users from running all-time searches, which can be costly in large environments with sizable indexes.
Predicate:
search_time_range=alltime AND role!=sc_admin
This ensures that only the sc_admin role can run all-time searches. Other users will see a custom message, such as: ‘Please narrow your search time range.’ Implementing this rule saves Splunk resources and reduces search delays.
No Index Wildcards
New users often run searches like index=*, which is inefficient. Searches should include fields like index, sourcetype, host, and source to narrow results.
Predicate:
index=* AND role!=sc_admin
Users (except sc_admin) will receive a message like: ‘Please specify an index.’ In environments with 100+ indexes and large data volumes, this rule significantly conserves resources.
Workload Rule
Workload rules contain user-defined conditions based on a set of predicates, when a search meets the condition, it is triggered with a specific action taken, for example, abort the search.
No_Realtime Searches
Real-time searches are discouraged because they lock one CPU core on the Search Head and one on every indexer for the duration of the search.
Predicate:
search_mode=realtime AND runtime>1s AND role!=sc_admin
Users (except sc_admin) will see: ‘Real-time searches are not allowed.’ Instead, use scheduled reports or alerts that run every 5–15 minutes to achieve similar results.
Rule Ordering
Rule order matters. For example:
- Rule 1: If runtime > 5 mins, display a message.
- Rule 2: If runtime > 10 mins, abort the search.
The second rule will never execute because once a rule applies to a job, Splunk does not re-check that job. To prioritize, set the most critical rule to order 1.
Workload Pools
Moving a search to a limited_perf pool does not immediately degrade performance. Restrictions apply only when the Search Head is under contention. Pools can borrow resources dynamically, and restrictions harden only during contention.
On-Premise Considerations
- Workload Management works only on Linux, not Windows.
- Splunk must run under systemd.
- Configure cgroups on all servers for resource allocation.
- If polkit rules are missing, install them using the enable boot-start command to allow non-root users to manage Splunk services.
Run preflight checks via CLI:
./splunk check workload-config
Fix any reported issues (e.g., missing properties in Splunkd.service).
Troubleshooting
Use this SPL to check search pool CPU allocation:
index=_introspection host=<indexer_name> component=PerProcess | timechart span=10s sum(data.normalized_pct_cpu) by data.process_type
Example Rules
Workload Rules:
WildcardSearch_OR_Alltime:
NOT (role=admin OR role=sc_admin) AND index=* AND runtime>1m
Action: Move to limited_perf pool.
Searches_LongerThan30min:
NOT (role=splunk_secops_admin OR role=splunk_secops_user OR role=admin OR role=sc_admin) AND runtime>30m
Action: Move to limited_perf pool.
Admission Rules:
Index Wildcard Searches:
NOT (role=splunk_secops_admin OR role=splunk_secops_user OR role=admin OR role=sc_admin) AND index=*
Action: Filter search.
Message: ‘Please do not use wildcard searches on indexes. Explicitly define the index.’
Splunk’s Workload Management lets you filter inefficient searches and manage resource usage with admission and workload rules to help reduce delays and keep searches running efficiently. Learn more here!
About the Authors
Aaron Dobrzeniecki has just over 9 years of experience with Information Technology. He first worked for an MSP broadening his skills with Tier 1 and Tier 2 Help Desk support for just over 40 small companies. Aaron has been working with Splunk for over 8 years starting on the apps and add-ons team troubleshooting issues with Splunk apps and add-ons. He then worked on the Admin on Demand team creating content, assisting with Splunk core issues or tasks. As well as providing Splunk Best Practices. He has provided excellent service to the Splunk customers he has worked with and continues to get praise from his customers. Aaron was also a Customer Success Manager for 14 companies working with their Splunk teams to better their environments. As a Certified Splunk Consultant, Aaron continues to increase his knowledge and follow the Splunk Best Practices while maintaining an excellence in communication. Aaron graduated with his Bachelors’ majoring in Security Informatics from Indiana University in 2015.
Kamal Dorairaj has over 22 years of diverse IT experience with full cycle development of various Applications and Systems. In his first 10 years; he worked as a CRM Consultant at the client locations includes Middle East, India, US; in various domains includes Retail, Telecom, eCommerce. Later he worked as a Lead SRE at Stubhub (was eBay company) for 7 years as a SME in Splunk, GCP. He is a PMP certified, ITIL certified, CompTia Security+ certified and Splunk Architect, Splunk ES Admin, Splunk Developer certified. He also completed Cybersecurity for Managers, Business Analytics from MIT Sloan and MBA in Technology Management. His core interest area has been, ‘Get more business value out of Big data’.