Version Source Control for your Splunk Environment
By: Zubair Rauf | Splunk Consultant
When Splunk environments grow in organizations, the need for source control also grows with it. It is good practice to use the widely available source control tools that are available for enterprise level source control.
There are many Version Source Control (VCS) software available online, but the one most widely used is the open source, Git, which has proven to be a very powerful tool for distributed source control. Using Git, multiple Splunk Admins can work with their local repositories and the changes shared separately.
To take the conversation further, I would separate the need for version control in two major segments
- User Applications
- Administrative Applications
I have broken down the applications into two segments to enable ease of management for Splunk Admins. The User Applications should consist of the search artifacts that are built and developed as use cases evolve and change often, whereas the Administrative applications I would classify as those which are mostly used to deploy setup Splunk like TAs and other deployment apps. These applications rarely change once set up unless new data sources are on-boarded, there are significant changes to the architecture, etc.
In the context of this blog post, we will focus on the administrative applications. These apps are the backbone on your Splunk deployment and should be cautiously changed to make sure there is no downtime in the environment. Changing these files could cause irreparable damage to the way data is indexed to Splunk, causing loss to indexed events, especially when changing sourcetypes, etc.
As I already mentioned, there are numerous flavors of source control and depending on your taste, you can use either. If you’re starting off fresh with source control, Git is easy to set-up and you can use it with Github or Atlassian Bitbucket. Both these tools can help you get started in a matter of minutes, where you can create repositories and setup source control for your distributed Splunk environment.
The Git server will host all the master repos in the Splunk Environment for all the administrative apps. The admins who need make edits do it in the following two ways;
- Edit the master directly.
- Create local clones of the master, make the required edits, commit them to the local branch and then push it out to the remote repo.
Ideally, no one should edit the master branch directly to reduce the risk of unwanted changes to the master files. All admins should edit in local branches, and then once the edits are approved, they should be merged to the master.
There should be three Master repos with their respective apps and TAs in those repos. These repos should correspond to the following servers;
- Cluster Master for Indexers
- Deployer for Search Head Cluster
- Deployment Server for Forwarders
To deploy the repos to the servers, you can use git hooks or tie your git deployment back into your puppet or chef environment. This is based on your discretion and how you are comfortable with distributed deployment in your organization. The repos should be deployed to the following directories
- Cluster Master to $SPLUNK_HOME/etc/master-apps/
- Deployer to $SPLUNK_HOME/etc/shcluster/apps
- Deployment Server to $SPLUNK_HOME/etc/deployment-apps
After the updated repos are deployed to the respective directories, you can push them out to the client nodes using Splunk commands.
If you are interested in more information, please reach out to us and someone will get in touch with you and discuss options on how TekStream can help you manage your Splunk environment.