Migrating a Splunk Deployment Server to New Hardware
By Kyle Moreau, Cybersecurity Engineer II
The first step in migrating your deployment server is to review your environment. Take note of the version of Splunk Enterprise that is running. Review local apps, deployment apps, server classes, and any custom configurations. Create a backup of your configurations.
The next step is to install and configure Splunk on the new host. Take into consideration the version of Splunk that is currently being used. If you are not using the most up to date version, review the official Splunk documentation for any major changes. After you install and configure Splunk, validate the web UI and ensure there are no errors that need to be addressed before configuring it as a deployment server. At this step, you will also want to add any custom apps or configurations that are not deployment server related (such as configuring ldap authentication).
After Splunk is configured and running on the new server, you can configure it as a deployment server. Copy over the deployment of apps and the server classes configuration. Modify the deployment client app configuration file to point to the new deployment server.
If a deployment client app does not already exist in your environment, it is highly recommended you modify your configurations before proceeding with this process. A common issue is that the deployment client configuration is created on the universal forwarder under $SPLUNK_HOME/etc/system/local. This is an issue because the deployment server cannot make changes to configurations outside of $SPLUNK_HOME/etc/apps. If you proceed without fixing this issue, you will have to manage your forwarders locally one by one.
You may need to restart Splunk to be able to view the configurations after they are copied over. Compare the configurations between the two servers to verify that they match.
Once the configurations are complete, all that is left is to update the forwarders to communicate with the new host. If your environment is following Splunk best practices, this is a simple task.
Update the deployment client app on the original deployment server to point to the new server. Make sure that the app is set to reload the Splunk service. Reload the deployment server and wait about 15 minutes; you should see the forwarders check in to the new deployment server and stop checking into the old deployment server.
If the change is not successful, there are likely configurations on the forwarder that conflict with the app or networking issues with the new server. If you are running into issues or want to learn more about migrating Splunk servers, please reach out to us using the form below.
Need expert help with your Splunk migration? Contact TekStream today →
About the Author
Kyle Moreau has over eight years of information technology experience, six of which are within information security. As a security engineer, he leads projects for various security tools including SIEM and Endpoint. Maintained and developed solutions to improve the security posture of a fortune 500 organization. Kyle has successfully completed full ES implementations for multiple organizations. Previous experience as a security analyst, focused on incident response and investigation as well as keeping current in the latest attack methods.