Troubleshooting Splunk Forwarders Tutorial: Send Commands to Multiple Splunk Forwarders (Linux)
By Alex Trejo, Splunk Consultant
When working in a large Splunk environment it is common to troubleshoot multiple Splunk forwarders at one time. One approach to managing Splunk Forwarders is connecting directly to the hosts to troubleshoot. Connecting to a host, running commands, repeating on another host. This can be extremely time consuming when managing a large Splunk environment with hundreds of Splunk Forwarders. A solution to this time-consuming problem is creating a script which sends commands to multiple Splunk Forwarders.
In this example, we will create a shell script which will be used to search for text in multiple Splunk forwarders. To begin, we will create a text file which will contain the list of the Splunk forwarders we are attempting to connect to.
Here you would replace the IPs 10.0.0.X with the IPs that match your hosts.
Next, we will decide which commands we want to send to the hosts. In our case, we want to search for text that resides in a directory on the hosts. Thus, we want to run the command:
After you have decided which commands you want to run, begin creating the shell script that will be used to contact the hosts. Create a HOST_FILE variable that will read the path to the HostsList.txt file we created earlier.
You will want to replace <path to host file> with the path to your host file.
Next, create a USERNAME variable that will read the username we will be using to connect to the hosts.
Make sure to replace with your username.
For the variables being used in this example, /path/to/directory and “text_to_find” we will be reading the values as inputs and assigning them to the variables PATH_TO_TEXT and TEXT_TO_FIND.
Next, we will create the remote script which will be the commands you want to send to the forwarders. In our case, the text searching commands.
Optional: Before running the commands to the forwarders you can initiate output that will display that we are attempting to connect to the hosts.
Now create a “for loop” which will loop through each host listed in our HOSTS_FILE.
The variable DST is each host and will be referenced again when we are referencing a host and not the HOST_FILE.
Next, you will run an SSH command which will run to connect to each DST.
Finally, finish by closing the shell script.
Notice, that you will still need to input a password every time the SSH connection is made with a forwarder. If you want to avoid this you will have to download SSHPASS. This will allow you to include the password in the SSH command bypassing the manual password input every time a connection is made. Once you download SSHPASS you can replace the SSH command with:
Replacing <password> with your password (Not in quotation marks).
This is what the final shell script for running commands on multiple Splunk forwarders will look like.
That will do it! You should find that your forwarders have each been reviewed for trouble, and should have saved quite a bit of time with this efficient method of sending multiple commands. Happy Splunking everyone!