Auditing Apps for Splunk 8.0
By: Eric Howell | Splunk Consultant
The release of Splunk 8.0 marked a pivotal change in the functional workings of Splunk; the tool transitioned from leveraging Python 2 to Python 3. This shift is due to the fact that support was dropped for Python 2 by the governing vendor on January 1st, 2020. Due to this change, administrators working to maintain a supported, healthy environment will be required to perform a comprehensive review of app-Splunk version compatibility and upgrades in addition to usual upgrade procedures.
Compile List of Installed Apps and TAs
In each environment, make an inventory of the deployed Splunk architecture components:
- Search Heads – taking into account any stand-alone instances that might not be in the primary cluster (example: Enterprise Security)
- Monitoring Console
- Deployment Server
- License Master
- Cluster Master
- Heavy Forwarders – if applicable
For each of these components, compile a list of the installed and active applications (apps) and Technology Add-Ons (TAs). This can be done by running the following SPL query in the Search UI:
| rest /services/apps/local
| stats count(title) by splunk_server label version eai:acl.app author
| rename label AS App, version AS Version, eai:acl.app AS Base, author AS Author
| table splunk_server, App, Base, Author, Version
Review Installed Apps on SplunkBase
The above search will provide you with the necessary information to compare the installed version of an App against that found within Splunkbase, including the App Name, Author, Version, and installation status. This information will need to be compared against what is found by locating the app within Splunkbase at https://Splunkbase.Splunk.com .
As an example, using the Splunk Add-on for Microsoft Windows:
|Splunk Add-on for Microsoft Windows||Splunk||4.8.2||Yes|
Searching this add-on in SplunkBase leads us to the following link: https://splunkbase.splunk.com/app/742/
The SplunkBase page for each app contains information regarding app version (adjustable via the dropdown indicated in the next image) and compatible versions of Splunk Enterprise. The details tab often contains app-specific information (frequently linking to Splunk supported documentation) and can provide insight into the appropriate upgrade path for the app. These upgrade paths are critical to follow due to major adjustments often made in newer iterations of any app. These iterative changes can cause negative impact if not accounted for: loss of data, non-functional commands, and new formats for dashboards.
If your environment includes apps that are not found in SplunkBase (very likely due to custom app creation), use your best judgment. The Upgrade Readiness app, which is discussed later in this document, will provide further insight into likely xml or python related complications found in any apps that are scanned. It is advised further in this document to create a Dev environment to test these upgrades prior to releasing in Prod, and these apps are perfect candidates for additional testing outside of Prod.
Fig 1. SplunkBase page breakdown
Compile the list of installed apps and TAs in the environment and cross-reference them with SplunkBase to provide insight:
- Does the current version of the App support the version of Splunk you are upgrading to?
- What is the upgrade path for the App?
- Does the app still benefit from ongoing development?
- What Apps can be removed from the environment or will cause conflict once the upgrade has been performed?
SplunkBase link: https://SplunkBase.Splunk.com/app/4698/
Running the Upgrade Readiness App provides will provide further insight into whether your apps are ready for the upgrade to Splunk 8. As Python 2 is no longer vendor-supported, continued use of apps reliant on Python 2 can leave your environment vulnerable to intrusion or bad actors. The Upgrade Readiness App will advise which of your apps contain python files that are not dual-compatible or strictly compatible with Python 3, and it will also indicate xml files that support Splunk’s Advanced XML which has been sunset and replaced by SimpleXML. Additional details can be found here:
After running the Readiness App each of the installed apps on the Splunk instance should return a value result, such as, Passed, Warning, Skipped, etc.
Please note that this app will need to be installed on each instance of Splunk for comprehensive review.
From the above steps, the path forward should emerge after documenting the findings.
- Identify which apps/TAs can be and require upgrade
- Identify Apps that are no longer supported
- Remove and/or disable Apps that are no longer relevant in the environment or will cause issues post-upgrade.
- Identify Apps that have not been thoroughly documented and will require additional testing (ideally in a Dev environment).
Once the plan has been developed with the steps above, separate the apps by the appropriate configuration management tool/server (Cluster Master, Deployer, etc)
The major contributing factor to lost functionality when upgrading to Splunk 8.0+ is found in apps that leverage a great deal of Python files that are not dual-compatible with Python 2 and Python 3. This is discussed in greater detail in the links below:
To maintain a functional, supported version of the Enterprise Security app throughout the upgrade process, it will likely be necessary to upgrade Apps as you upgrade Splunk. Several apps are heavily Python-dependent in their operation and will feature a Python-version change between app versions.
These Python version-specific apps, if they are being leveraged in your environment, should be upgraded during the same scheduled change window as the Splunk Enterprise upgrade to 8.0+. Otherwise, they will cease to function correctly (or at all) due to their reliance on Python 3. The apps that require this specific process are listed here:
- Splunk Enterprise Security App ver 6.1
- Splunk Machine Learning Toolkit ver 5.0
- Deep Learning Toolkit ver 3.0
- Python for Scientific Computer ver 2.0
Want to learn more about auditing apps in Splunk 8.0? Contact us today!