pass4SymmKey does not hash when pushed from Cluster Master

by Aaron Dobrzeniecki | Senior Splunk Consultant

When working with a pass4SymmKey, the goal is to get the password to hash when Splunk restarts on the servers you have placed them on. I have noticed while working with many customers that pushing a pass4SymmKey from the Cluster Manager WILL NOT hash the password on the indexers. The screenshots below are using Splunk 8.2.5 and I am pushing a pass4SymKey to the indexers.

NOTE, pushing passwords from a CM in Splunk version 9.0+ will hash the pass4SymmKey on the indexer AND cluster manager side.

On the Cluster Manager:

Terminal window showing the contents of the server.conf file on a Splunk Cluster Master, including a plain-text pass4SymmKey used for cascading replication

In newer versions of Splunk they have included a directory called manager-apps. As you can see above, I am using master-apps which you can still use in 8.2.5. In order to migrate over from using master-apps vs manager-apps, you need to move ALL of the apps in master-apps to manager-apps and apply the cluster bundle. NOTE! You cannot use master-apps and manager-apps at the same time.

I have applied the bundle to the indexers and this image shows the checksums matching!

Splunk terminal output showing cluster-bundle-status across multiple indexers, indicating active and validated bundles with replication status marked as "success".

Splunk Indexer 1:

Splunk configuration file on indexer 1 showing an unencrypted pass4SymmKey in the cascading_replication stanza.

Splunk Indexer 2:

Terminal output from indexer 2 displaying the same plaintext pass4SymmKey as indexer 1 in the server.conf file.

Splunk Indexer 3:

Splunk configuration file from indexer 3 showing an unencrypted cascading_replication pass4SymmKey similar to other indexers.

As you can see from the multiple screenshots above, the pass4SymmKey did not hash on the indexers when pushed from the Cluster Manager. Also, in the first screenshot you can see that the password is also in plain text on the Cluster Manager. This imposes a security risk as well on the Cluster Manager. If an unauthorized user was able to gain access to the Cluster Manager, they could easily find this pass4SymmKey.

Now, the stanza I have used in my example is for cascading bundle replication from the Search Heads. Having a similar pass4SymmKey under that stanza on all indexers, allows the indexers to send each other knowledge object bundles originating from the Search Head. Pushing ANY pass4SymmKey in ANY stanza from the Cluster Manager will not hash the value. Click here for more information on cascading bundles on Splunk.

Solution/Resolution
An absolute way I have found to make sure the pass4SymmKey hashes is to place the stanza and pass4SymmKey in $SPLUNK_HOME/etc/system/local/server.conf and restart Splunk. I know what you must be thinking, “We have a large number of indexers that we would need to manually touch in order to place the pass4SymmKey in $SPLUNK_HOME/etc/system/local/server.conf”. IF YOU DO NOT have any sort of automation tool (puppet, ansible, bitbucket CICD, or even just a script that will reach out and touch every indexer), you will have to manually touch each box. If you do not have any automation tools but you have a person who can write a script to touch each box, try that before the manual process.

Manually Setting pass4SymmKey in Splunk

-Navigate to the CLI or backend of the indexers
-cd $SPLUNK_HOME/etc/system/local (Make sure you are the user that is running Splunk)
-If there is no current server.conf, you can create one
-vi server.conf
-Place the specific stanza and pass4SymmKey setting you will be using as below:

Magnified section of a configuration file showing the same plaintext pass4SymmKey used in multiple nodes.

Note: **You will need to restart the indexers for the hashing to occur. In an indexer cluster you will want to perform a rolling restart instead of restarting each indexer manually. **

-Navigate to the GUI or the backend of the Cluster Manager

–GUI: Settings -> Indexer Clustering -> Edit (Top Right) -> Rolling Restart -> Begin Rolling Restart (check the widgets as necessary)

–Back End/CLI: Navigate to $SPLUNK_HOME/bin and run the following command
./splunk rolling-restart cluster-peers

The screenshots below will show you, that after manually placing my pass4SymmKey in $SPLUNK_HOME/etc/system/local/server.conf and performing a rolling restart on the cluster; the pass4SymmKey hashes!

Encrypted pass4SymmKey displayed in server.conf from indexer 1’s system local directory, differing from the plaintext versions.
Encrypted pass4SymmKey shown in the system local directory of indexer 2, indicating local hashing applied.
System local configuration on indexer 3 showing a hashed version of the pass4SymmKey used for cascading replication.

In Conclusion

Splunk versions pre 9.0 WILL NOT hash the pass4SymmKey when pushed from the Cluster Manager. If you would like to apply a pass4SymmKey on versions before 9.0, you will either need to use an automation tool to reach every indexer, or touch each server manually as explained above. Please see all server.conf files from each version of Splunk after the push from the Cluster Master/Manager.

Splunk 8.2.5 Indexer
Terminal output showing a plain text pass4SymmKey value set to aarontest123 within the server.conf file on Splunk instance idx1, located in /opt/splunk/etc/slave-apps/_cluster/local/.
Splunk 8.2.5 Cluster Master
Terminal output showing a plain text pass4SymmKey value aarontest123 within the server.conf file on the Splunk Cluster Master (master1) in the /opt/splunk/etc/master-apps/_cluster/local/ directory.
Splunk 9.0 Indexer
Terminal output from idx1 showing a hashed pass4SymmKey stored in server.conf under /opt/splunk/etc/peer-apps/_cluster/local/, indicating hashing occurs when key is set locally rather than pushed from Cluster Master.
Splunk 9.0 Cluster “Manager”
Terminal output on master1 showing a hashed pass4SymmKey value in the server.conf file located in /opt/splunk/etc/manager-apps/_cluster/local/, suggesting a manual edit on the Cluster Manager with proper hashing behavior.
Splunk 9.0.1 Indexer
Terminal output from idx1 again displaying a hashed pass4SymmKey in the server.conf file located in /opt/splunk/etc/peer-apps/_cluster/local/, reinforcing that peer-nodes apply hashing correctly when configured independently.
Splunk 9.0.1 Cluster “Manager”
Terminal view of another hashed pass4SymmKey entry in server.conf under /opt/splunk/etc/manager-apps/_cluster/local/ on the master node, used to demonstrate inconsistent behavior when keys are pushed versus manually configured.

There you have it! If you still have questions about this, or any other Splunk item that you’d like help with, contact us! Submit your comments below and we will be in touch right away.