Useful Online Resources for the Aspiring Splunk Admin
By Eric Levy, Splunk Consultant
No matter how much training you go through and how many certifications you get, you can truly never know everything about Splunk. Some of it isn’t even provided in training itself – to properly use Splunk, you’ll need experience around the command line, data sources, and syslog. Even beyond that, you’ll never truly memorize every possible setting in props.conf or how to form every possible regular expression.
Here is a list of resources that my fellow TekStream colleagues and I compiled that we use on a near daily basis to assist in our Splunk work.
OFFICIAL SPLUNK RESOURCES
Splunk Docs
You will never go a day using Splunk without referring to the official documentation, colloquially known as the “Docs”. It has pages on every product offered by Splunk, including premium apps and select supported apps in Splunkbase, and provides detailed installation steps, configuration steps, upgrade steps, and best practices. Nearly every .conf file and SPL command/function has its own page for reference, on top of formatting standards for time values and dashboards (both Classic and Studio). The added benefit is that Splunk keeps previous versions of the Docs available if you aren’t running on the latest version. While the Docs are an essential resource, be wary of vague wording on some pages.
Splunk Community Posts
If you didn’t find it in the Docs, somebody else didn’t either. The Splunk community has thousands of Q&A threads for tricky searches, error messages, unconventional log sources, and other specific use cases. Oftentimes community posts and the docs go hand in hand as several users link back to the docs for more information on an answer. It’s sites like here that show how limitless the sky can be with Splunk.
Splunkbase
A repository for apps to install in Splunk – some supported, others not. Includes technical add-ons (TAs) for data sources, front-facing apps with useful dashboards, alerts, and searches, and even custom visualizations (like a calendar, for example). While you’ll always interact with Splunkbase in some way or another (as Splunk communicates with it directly to install apps), the site itself is useful for documentation and important updates provided by the developers. Don’t see an app for your purposes? Develop one and submit it for others to use!
ENTERPRISE SECURITY
Splunk ES Detections
This constantly updated list provided by Splunk has several threat detections to use in Enterprise Security. Each detection has steps for implementation, security context, and known false positives. Keep in mind that some are marked “experimental” and may not be confirmed as effective.
Risk Based Alerting Guide
Risk Based Alerting is a powerful and unique feature in Enterprise Security for prioritizing threat sources, and this free PDF is a guide for how to use it and get the most value out of your security notables.
UNOFFICIAL SPLUNK RESOURCES
Splunk Subreddit
Unaffiliated with Splunk, but the subreddit is a sufficient alternative to the Splunk Community posts when searching for an unconventional use case.
Stack Overflow
Similar to the Splunk Community posts and the subreddit, here is another great resource to find unconventional use cases.
GoSplunk
An open repository for SPL queries and dashboards, essential for anyone getting started with SPL and/or looking for inspiration for their own projects. They also have a blog with some helpful Splunk information, although they haven’t been updated in over 5 years.
TekStream Splunk Blog Posts
Our esteemed Splunk Consultants from all over the country always post helpful information, tips, and resources to this very blog. Be sure to follow us on LinkedIn for updates and new blogs as they come in!
NON-SPLUNK ESSENTIALS
RegexOne
A perfect starting ground for learning regular expressions.
Regex101
Once you’ve mastered RegexOne, Regex101 becomes an invaluable tool for figuring out regular expressions for field extractions and line breaking. Don’t be surprised if you use this website every week, if not every day.
JSON Formatter
Helpful for developing apps and deciphering complicated log files (although Splunk is generally quite good displaying these events via INDEXED_EXTRACTIONS or KV_MODE).
Crontab Guru
Some reports and alerts may need to be run on an unconventional interval for their own reasons or for spreading out search jobs. Splunk uses cron jobs to set up such alerting and this site makes formatting them easy. Just copy and paste directly into Splunk when finished.
Character Count Tool
MAX_TIMESTAMP_LOOKAHEAD and TRUNCATE are essential props.conf settings to help Splunk parse time and guide each sourcetype on event length. Instead of manually counting characters, paste an event or a timestamp here to get a value immediately.
Epoch Time Converter
I’ve developed use cases for my own customers to filter dashboard panels based on time intervals, which involve weaving around values set by the time picker with directly comparing epoch values. Other times, I may want to verify that the timestamps coming in are being read correctly by Splunk into the right time zone offset. This converter is perfect for translating dates to and from epoch time for these use cases and beyond.
This list is in no way comprehensive – there are infinite use cases for Splunk that may require other tools. These are just the ones that me and my fellow consultants at TekStream use regularly. Happy Splunking!