Lesser-known Splunk Enterprise features Part 1: Debug/Refresh

By Nate Hufnagel, Splunk Consultant II, Cybersecurity Engineering

Intro

The Splunk Enterprise platform supports several styling & behavior customizations that customers can take full advantage of. Some of these features are more obvious as your Splunk admin becomes more familiar with the UI, while others are not as obvious, even hidden. This two part series will discuss two of these hidden but powerful features of Splunk: debug/refresh and _bump.

First up: debug/refresh. This feature can be used to reload all splunkd registered EAI (Extensible Administration Interface) handlers via REST call. Put more plainly – most of Splunk’s configs and static assets can be reloaded or refreshed without having to restart the splunkd service. This includes props, transforms, inputs, dashboards, and many others. The full list of endpoints can be found in the Splunk docs, or by executing debug/refresh. Let’s take a look at a few examples.

In this first example, we’ll add a new search-time field extraction to props.conf and update our sourcetype without restarting Splunk. This is our list of fields before making any changes:

A screenshot of a Splunk interface showing a pop-up window detailing the distribution of "ErrorMessage" field values, with a list of selected and interesting fields visible on the left pane.

We’ll add another field called “ErrorMessage_2” to the sourcetype in props.conf. This can also be done through Splunk Web (Settings>Add Data)
[<sourectype>]
…
EXTRACT-ErrorMessage_2 = ErrorMessage\=(?<ErrorMessage_2>[^\]]+)
…

Next, we’ll edit our Splunk Web URL to the following:
http(s)://<host:mport>/debug/refresh
…where “host” is the hostname and “mport” is the web port number (8000 in this case).
Click “Refresh”, then it takes about a minute while all the endpoints reload.
Once the reload is complete, we’ll see the “Entity refresh control page” in our browser, as well as a list of all endpoints that we’re successfully reloaded.

A screenshot of a Splunk terminal showing the entity refresh control page, displaying the successful refreshing of Splunk configurations and EAI handlers.

Now, we’ll navigate back to Splunk Web and run the same search as before: http(s)://<host>:<mport> (should still be logged in).

A screenshot of the Splunk interface showing a detailed analysis pop-up window for the "ErrorMessage_2" field, displaying the top 10 values and their distribution.

After running the same search, we see our new field!

We can also reload more complex changes, like a custom app. For example, an app with a custom navigation bar and embedded HTML links can be installed and used without having to restart Splunkd!

In this next example, our custom app is named “html-link”.  It contains on custom navigation tab called “Hyperlink” and an html link that can take us to an external site.  We’ll follow the same steps as above to achieve this without restarting Splunkd:

              Install the app

              Perform the debug/refresh

              Navigate back to Splunk Web home

              Launch app

A screenshot of the Splunk Enterprise user interface showing the "Apps" dropdown menu open, listing various installed applications and options like "Search & Reporting" and "Manage Apps."

Before installing the app

A screenshot of a command line interface showing a tar command used to extract a file (html-blink_current.tgz) into the Splunk apps directory (/opt/splunk/etc/apps).

Installing app

A screenshot of the Splunk debug Entity refresh control page within a terminal, detailing the process for refreshing splunkd resources and EAI handlers.

Performing debug/refresh

A screenshot of the Splunk Enterprise interface displaying the Apps dropdown menu, which lists various applications, including a newly added "html-link" app, along with options to "Manage Apps" and "Search & Reporting".
A screenshot of the Splunk Enterprise interface showing the "Hyperlink" tab active, with text instructing the user to click a link labeled "Debug-refresh-test-2" for learning more about regex.

Navigating to/launching to app

To learn more about what Splunk resources can be reloaded with debug/refresh, check out the link to the Splunk docs below. Be on the lookout for Part 2, where we’ll dive into the “_bump” feature.

Link to Docs

https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/CustomizationOptions

https://dev.splunk.com/enterprise/docs/devtools/customrestendpoints/customresteai

Let us know what you think! And for more information on TekStream offerings, visit our Splunk Services page.