Creating Custom Splunk Commands to Change Cron Schedules

By: Antonio Patino Noguez | Splunk Consultant

In a customer environment, we noticed search latency at the top of every hour. After troubleshooting, I noticed that scheduled searches were set to run at the same time. Although it is simple to change the cron schedule for these searches through Splunk web, there were way too many searches, making the process painful. Using some filters and a REST call in a search, I could grab different batches of scheduled searches – but then I wondered, how can I change the cron schedules of these batches via search? 

Splunk gives us the ability to create custom search commands. To create a custom command, you need to define the command in a commands.conf file. After the line of SPL is determined to be a custom command, Splunk checks the commands.conf and runs the corresponding Python script. Splunk passes each pipe of SPL through the script through STDIN and writes them out through STDOUT. Splunk then returns to the main search and completes the search.

So, first things first, I needed to create both a commands.conf file and a Python script. By opening the text editor of my choice, I simply defined my command in a stanza and filled out the necessary attributes. 

Here is an example of that:

           [custom_command]

           filename=cron_python.py

           chunked=true

           python.version=python3

Next, I need to create my Python script. The goal here was to collect all the scheduled searches that my search returned and iterate through them while changing their scheduled time to run. Leveraging the Splunk SDK, I have the following loop in my script:

       for splrecord in records:

           title = splrecord[‘title’]

           cron_schedule = splrecord[‘cron_schedule’]

           newtracker_update = service.saved_searches[str(title)]

           #kwargs = {“cron_schedule”: self.cron_schedule} 

           kwargs = {

               “cron_schedule”: self.cron_schedule,

           }

Now it’s time to bundle these files in a custom app. One important thing to remember to add to the app is the splunklib library. This is so the Python script can leverage the Splunk SDK.

Since I gathered the schedule searches in “batches” via SPL, I could leverage my custom command to change the cron schedules just by running my search. It works like magic!

In conclusion, creating custom commands can really help tackle specific tasks that may not be easily handled with out-of-the-box Splunk enterprise. This allows for flexibility to handle virtually all issues that may arise in your Splunk environment. For further reading, here are The 3 Most Common Splunk Issues and How to Solve Them.

Got a unique Splunk issue? TekStream can help you solve it. Contact us today.