Why Splunk on AWS?

AWS is the world’s most comprehensive and widely-adopted cloud platform, offering over 165 services from data centers all over the globe. AWS allows you to build sophisticated applications with increased flexibility, scalability and reliability. The platform serves businesses from everyone from government agencies and Fortune 1,000 companies to small businesses and entrepreneurial startups.

Should your business consider using AWS? Changing databases to AWS is easy with the AWS database migration service, or by using AWS managed services or an AWS consulting agency.  Here’s why AWS is amongst the leading cloud computing services.

Reason #1: It’s Flexible 

Anyone can sign up for AWS and use the services without advanced programming language skills. AWS prioritizes consumer-centered design thinking, allowing users to select their preferred operating system, programming language, database, and other vital preferences. They also provide comprehensive developer resources and informative tools available to help maintain AWS’s ease of use and keep it up-to-date. 

Whether your team has the time to learn AWS or has access to AWS consulting, training users is simple. AWS offers their services with a no-commitment approach. Many software solutions use this as a way to market monthly subscriptions, but AWS services are charged on an hourly basis. As soon as you terminate a server, billing won’t include the next hour.   

With AWS, you can spin-up a new server within a matter of minutes compared to the hours or days it takes to procure a new traditional server. Plus, there’s no need to buy separate licenses for the new server. 

Reason #2: It’s Cost-Effective 

With AWS, you pay based on your actual usage. AWS’s cloud solution makes paying for what you use the standard for database storage, content delivery, compute power, and other services. No more fixed server cost or on-prem monitoring fees. Your cost structure scales as your business scales, providing your company with an affordable option that correlates with its current needs. This results in lower capital expenditure and faster time to value without sacrificing application performance or user experience. Amazon also has a strong focus on reducing infrastructure cost for buyers. 

Reason #3: It’s Secure 

Cloud security is the highest priority at AWS. Global banks, military, and other highly-sensitive organizations rely on AWS, which is backed by a deep set of cloud security tools. Maintaining high standards of security without managing your own facility allows companies to focus on scaling their business, with the help of:  

  • AWS multiple availability zones (which consist of one or more data centers around the globe with redundant power, networking, and connectivity) that aid your business in remaining resilient to most failure modes, like natural disasters or system failures. 
  • Configured, built-in firewall rules that allow you to transition from completely public to private or somewhere in between to control access according to circumstance.

Multiple Migration Options

Depending on your unique business and tech stack needs, AWS offers companies multiple options for realizing its host of benefits. For Splunk, those options include:

  1. Migrate your Splunk On-Prem directly to AWS
  2. Migrate your Splunk On-Prem to Splunk Cloud (which sits on AWS)

Migrating to the cloud can be a business challenge, but AWS makes it simpler. While on the journey towards stronger digital security and efficiency, AWS can save time and resources. With its flexibility, cost-effectiveness, and security, you can easily deploy a number of software-based processes to an inclusive cloud-based solution.

Automated Patching with Ansible and the TekStream MSP

By: Karla Broadrick | Technical Architect & Team Lead


TekStream is excited to roll out the use of Ansible for automated patching of customer environments managed under the Managed Services Program (MSP).

What is Ansible?

Ansible is an IT automation engine.  It can be used to automate any number of repetitive system administration tasks such as infrastructure provisioning, configuration management, task automation, and application deployment.  It can be utilized in both On-Prem and cloud environments.  Instructions are organized into roles and playbooks, allowing flexibility for tasks to be performed based on a set of input parameters.  The setup for Ansible is simple and doesn’t rely on an agent on remote hosts.  All of this leads to a great degree of flexibility in how and where Ansible is used to automate every day IT tasks.

How is TekStream utilizing Ansible?

TekStream has created an Ansible playbook that can be used to automate OS and application patching in customer’s environments.  The playbook automatically creates backups, downloads designated patches, shuts down the application, applies both Windows and Linux OS patches, applies WebLogic and other Oracle patches and reboots the host as needed.

The TekStream Ansible solution for patching is targeted at OS patching and Oracle WebCenter and WebLogic application patching, but has the flexibility to be extended into other areas.  TekStream uses this solution in our MSP customer environments.


The benefits to TekStream MSP customers of using Ansible for patching are many including

  • Consistency: Patches are applied the same every time in every environment. This means less mistakes are made and ensures uniformity and quality that our MSP customers can rely on.
  • Time savings: Automation saves time. No longer is there a need to manually log into each server, download and transfer patches, apply each one, and wait in between each step.  With the TekStream Ansible solution for patching, a single command kicks off the entire process.  Our MSP customers can rest assured that their downtime is minimized.
  • Scalability: This automated solution lends itself to be scaled quite easily.  Whether you have a single server environment or a large cluster, the Ansible playbook can be utilized.  The servers can be patched in parallel, without the bottleneck of required manual intervention.  This means that we are able to patch a large clustered environment with a single period of downtime.
  • Easy knowledge transfer/training: Because Ansible takes care of 99% of the work, there is very little knowledge required to do patching. Once configured for a customer environment, the solution is able to run on its own.  This makes knowledge transfer and training incredibly simple.  It also eliminates the fear of lost knowledge during personnel changes.
  • Reliability: All of the benefits listed above equates to reliability. Your organization can rest assured that patching will be done correctly, every time, and in an acceptable timeframe.
  • Cost Savings: Ansible allows us to do more in less time.  For TekStream MSP customers, this translates to cost savings and allows us to stretch your dollars further.

Interested in streamlining the patching and maintenance of your system?  Contact TekStream to learn about how we can help with Ansible and our MSP program.

Implementing Right-Click Integrations in Splunk

By: Eric Howell | Splunk Consultant


Splunk provides Admins the opportunity to build a huge variety of customizations, visualizations, apps, and a near-infinitude of different options to finely tune your Splunk experience and have it best suit your needs. There is a use case and a solution for almost any task you can throw at Splunk. A use case that has been brought up frequently is to implement a “Right-Click Integration” to allow information from Splunk to be passed to another tool. Originally, this seemed a daunting task that could require the implementation of custom java script or a unique python script to solve for. Ultimately, the solution was much simpler and the logic is already found in Splunk – Workflow Actions!

Workflow actions allow for Splunk to perform a variety of tasks against available web resources for information found in ingested field/value pairs. They can be very easy to set up either through the UI or by deploying a workflow_actions.conf, as best depends on the size of your Splunk environment, and they offer quite a bit of additional functionality. Utilizing this functionality, your “Right-Click” integration may require an extra click, but it will be fully functional and should be fully serviceable.

Leveraging Workflow Actions to Pass Values and Fields

As discussed above, to set up a workflow action, an admin can leverage the WebUI on one of the Search Heads in an environment or deploy the workflow_actions.conf file as appropriate. This will allow a user to expand an event and pass a value from a field to an external source. In the examples below, we will use VirusTotal as an example.

WebUI Setup

For this example, we will be posting the value of a field to VirusTotal, allowing us to search their IOC database and verify if the value we are passing is a known, bad entity.

  1. Navigate to the Settings Dropdown in the Splunk navigator (In the top right by default) and select Fields under Knowledge

2. Next, you will select “Add new” in the “Workflow actions” category

3. This next step encompasses what this workflow action will do. We begin with creating a Label for the action, which will be what users see when they move to access it. Note, you can leverage             tokens in this descriptor to input values dynamically, based on what is clicked. In the case here, it will show the specific value that is being presented to VirusTotal.

4. Then you will indicate, in a comma-separated list, the fields that this workflow action can apply to. Leaving this blank will appear in all field menus.


5. If needed, the next step will be to select which eventtypes this workflow_action is applicable for. For additional reading on eventtypes: Follow this Link

a. Eventtypes are categorizations of data input by the user or an admin. An example would be when a specific IP address is found in your web data (sourcetype=access_combined src_ip=, and you want to apply specific categorization to data that meets that criteria so that it appears with a new field/value pair (e.g. eventtype=local_access)

6. Next you will choose if the action is available in the Event Menu, Fields Menu, or both. This selection is a dropdown. Additional reading available here: Follow this Link

a. Event Menu – Will allow for you to apply this workflow action on the whole event. When an event returned by your search is expanded, the “Event Actions” dropdown will hold any workflow action with “Event Menu” selected here.

b. Fields Menus – Similar to the above, on an expanded event, the workflow action can be accessed by clicking the dropdown under “Action” in the list of included fields to allow one to apply the workflow action directly to the value of a specific field.

7. Next is to choose the Action type – meaning whether the workflow action acts as a “link” to another web location or whether it will run another “search”

8. The next step is configuring the properties based on which of the above Action Types was chosen

a. Link Configuration – Allows you to specify a URI (also accepts tokens), decide whether to open the link in a new window, select whether you intend to utilize a POST or GET method, and which arguments to pass to the web site.

b. Search Configuration – Allows you to specify a search string (token-friendly), dictate which app to run the search in, determine a name of a view (page) for the search to open in, whether to run the search in the same window, and input a timeframe.

9. Next click Save


Your workflow action is ready to test. Here is an example of what you can see when looking in an event view with an expanded event.

Deploying a workflow_actions.conf

Like most other configuration files, workflow_actions.conf is easily deployed via the method of your choosing (manually, scripted, or through the use of the Deployer). Below is what will be generated by Splunk in $SPLUNK_HOME/etc/system/local when created through the web UI:



display_location = both

label = Search VirusTotal for $@field_value$

link.method = post

link.uri = https://www.virustotal.com/en/search/

link.target = blank

link.postargs.1.key = query

link.postargs.1.value = $@field_value$

type = link


This same information is deployable in an app of your choosing to $SPLUNK_HOME/etc/apps/ as appropriate for environments with clustered Search Heads or through other means of configuration management.

There are a wide variety of possible options with the stanza, and I recommend reviewing Splunk’s documentation regarding the specific .conf file here:


Leveraging your workflow_actions with older versions of Splunk Enterprise Security

If you are utilizing a version of the Splunk Enterprise Security app prior to 5.3, even if your workflow actions have global permissions, the app will not import configurations from others unless they follow a strict naming convention. If you want your workflow action to be accessible from the Incident Review dashboard, for example, you will need to make sure your app follows the one of these naming conventions:

  • DA-ESS-*
  • SA-*
  • TA-*
  • Splunk_SA_*
  • Splunk_TA_*
  • Splunk_DA-ESS_*


Want to learn more about right-click integrations in Splunk? Contact us today!

Deep Freeze Your Splunk Data in AWS

By: Zubair Rauf | Splunk Consultant


In today’s day and age, storage has become a commodity, but, even now, reliable high-speed storage comes at a substantial cost. For on-premise Splunk deployments, Splunk recommends RAID 0 or 1+0 disks capable of at least 1200 IOPS and this increases in high-volume environments. Similarly, in bring-your-own-license cloud deployments, customers prefer to use SSD storage with at least 1200 IOPS or more.

Procuring these disks and maintaining them can carry a hefty recurring price tag. Aged data, that no longer needs to be accessed on a daily basis but has to be stored because of corporate governance policies or regulatory requirements, can effectively increase the storage cost for companies if done on these high-performance disks.

This data can securely be moved to Amazon Web Services (AWS) S3, S3 Glacier, or other inexpensive storage options of the Admin’s choosing.

In this blog post, we will dive into a script that we have developed at TekStream which can move buckets from Indexer Clusters to AWS S3 seamlessly, without duplication. It will only move one good copy of the bucket and ignore any duplicates (replicated buckets).

During the process of setting up indexes, Splunk Admins can decide and set data retention on a per-index basis by setting the ‘frozenTimePeriodInSecs’ setting in indexes.conf. This allows the admins to be flexible on their retention levels based on the type of data. Once the data becomes of age, Splunk deletes it or moves it to frozen storage.

Splunk achieves this by referring to the coldToFrozenScript setting in indexes.conf. If a coldToFrozenScript is defined, Splunk will run that script; once it successfully executes without problems, Splunk will go ahead and delete the aged bucket from the indexer.

The dependencies for this script include the following;

–   Python 2.7 – Installed with Splunk

–   AWS CLI tools – with credentials already working.

–   AWS Account, Access Key and Secret Key

–   AWS S3 Bucket

Testing AWS Connectivity

After you have installed AWS CLI and set it up with the Secret Key and Access Key for your account, test connectivity to S3 by using the following command:

Note: Please ensure that the AWS CLI commands are installed under /usr/bin/aws and the AWS account you are using should have read and write access to S3 artifacts.

If AWS CLI commands are set-up correctly, this should return a list of all the S3 buckets in your account.

I have created a bucket titled “splunk-to-s3-frozen-demo”.

Populate the Script with Bucket Name

Once the S3 bucket is ready, you can copy the script to your $SPLUNK_HOME/bin folder. After copying the script, edit it and change the name of your S3 Bucket where you wish to freeze your buckets.

Splunk Index Settings

After you have made the necessary edits to the script, it is time to update the settings on your index in indexes.conf.

Depending on where your index is defined, we need to set the indexes.conf accordingly. On my demo instance, the index is defined in the following location:

In the indexes.conf, my index settings are defined as follows;


Note: These settings are only for a test index, that will roll any data off to frozen (or delete if a coldToFrozenScript is not present) after 600 seconds.

Once you have your settings complete in indexes.conf, please restart your Splunk instance. Splunk will read the new settings at restart.

After the restart, I can see my index on the Settings > Indexes page.

Once the index is set up, I use the Add Data Wizard to add some sample data to my index. Ideally, this data should roll over to warm, and the script should be moved to my AWS S3 bucket after 10 minutes.

The remote path on S3 will be set up in the following order:

If you are running this on an indexer cluster, the script will not copy duplicate buckets. It will only copy the first copy of a bucket and ignore the rest. This helps manage storage costs and does not keep multiple copies of the same buckets in S3.

Finally, once the script runs successfully, I can see my frozen Splunk bucket in AWS S3. If you are running this on an indexer cluster, the script will not copy duplicate buckets. It will only copy the first copy of a bucket and ignore the rest. This helps manage storage costs and does not keep multiple copies of the same buckets in S3.

Note: This demo test was done on Splunk Enterprise 8.0 using native Python 2.7.1 that ships with Splunk Enterprise. If you wish to use any other distribution on Python, you will have to modify the script to be compatible.

If there is an error and the bucket does not transfer to S3, or it is not deleted from the source folder, then you can troubleshoot it with the following search:

This search will show you the stdout error that is thrown when the script runs into an error.

To wrap it up, I would highly recommend that you do implement this in a dev/sandbox environment before rolling it out into production. Doing so will ensure that it is robust for your environment and make you comfortable with the set-up.

To learn more about how to set-up AWS CLI Tools for your environment, please refer to the following link; https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

If you have any questions or are interested in getting the script, contact us today!

Sneak Peek Into Our Approach to Migrating Splunk On-Prem to AWS How to Migrate Splunk On-Prem to AWS

Splunk on AWS offers a special kind of magic. Splunk makes it simple to collect, analyze and act on data of all kinds. AWS applications for Splunk allow users to ingest that data quickly, with visualization tools, dashboards, and alerts. Together, they help organizations see through the noise. When a notable event (such as a potential breach) occurs, you can find and act on it quickly, making the combo a powerful tool for risk management. 

Running Splunk on the cloud gives organization resiliency, with all the advantages of scalability, flexibility, cost-optimization, and security.  Yet migrating to the cloud poses many challenges and implementing the new system alone can be intimidating, costly, and time-consuming if not done correctly. AWS database migration services are available to mitigate the impact of this necessary shift for your business. 

Finding an experienced and expert AWS and combined Splunk managed services partner to help you navigate can ease the process. Here’s a quick look into how to handle the change from Splunk On-Prem to AWS.

How Splunk Licensing Works

Each of your Splunk Enterprise instances require a license, which specifies how many gigabytes per day a given Splunk Enterprise instance can index and which features you have access to. Multiple types of licenses are available, and distributed deployments, consisting of multiple instances, require a few extra steps. 

Choosing the correct Splunk licensing option can be confusing. It requires outlining the types of business problems you wish to solve with Splunk, then estimating how much data usage you will need to perform this work over time.

Finding a Partner with Licensing Expertise

Non-compliance with licensing can lead to overages and penalties. As your advisory partner, TekStream can work with you to ensure that your Splunk and AWS licenses are in order.

TekStream has extensive experience navigating the specifics of complex license structures and contracts. Our Splunk Enterprise consultants will leverage their years of experience to help you assess your needs, accurately estimate your data usage, and determine the optimal license types and quantities for your unique needs.

In addition to Splunk licensing for new implementations, TekStream will also help your organization save money on licensing renewals. We will examine your Splunk usage to date, pinpoint areas where you may be overpaying, and provide you with viable alternatives to reduce your costs without sacrificing efficiency.

The 4 Most Common Licensing Structures

When selecting a licensing structure for Splunk on AWS, there are 4 main options. The best option will vary depending on the organization. Through careful analysis of your current licensing structures and your desired future state, we will work with you to determine the optimal licensing structure.  

Option 1: Migrate Your Existing Perpetual or Term License to AWS

Option 2: Convert Your Current License to Splunk Cloud (which would run on AWS)

Option 3: Convert to a Term or Infrastructure License (if on a Perpetual License)

Option 4: Pay-As-You-Go as part of a 3rd-Party Hosted MSP Solution

Each option has its pros and cons depending on an organization’s goals and usage. A partner can help you select the best option. TekStream’s deep experience overseeing complex data migrations empowers us to act as true consultative partners. We have the experience needed to quickly scope challenges and present solutions for your unique situation.

Take the risk out of your Splunk migration to AWS. We are so confident in our battle-tested strategy and proven database migration process that we guarantee that your database migration will be completed on-time and on-budget (when using TekStream’s Proven Process). We also guarantee optimal and cost-effective license and cloud subscriptions. 


Download the
Ultimate Guide

To find out more specifics about our proven process and get an in-depth look into our services, read The Ultimate Guide to Migrating Your Splunk On-Prem to Amazon Web Services. 

Optimizing Splunk Searches

By: Yetunde Awojoodu| Splunk Consultant


It may be interesting to learn that not all searches need to be optimized. Your search should be optimized if it will be administered often or queries large amounts of data. With Splunk Processing Language (SPL), you can obtain the same set of results using a different array of commands, however, when constructing your searches, it is important to consider the impact to memory and network resources in your environment.

How to Determine Whether Your Search is Unoptimized

  • It runs for a long period of time
  • It retrieves larger amounts of data than needed from the indexes

(View the amount of data retrieved from the indexer in the Job Inspector)

  • You tend to hit the disk search quota while running the search
  • It results in a slow or sluggish system

Creating Optimized Searches

To optimize the speed at which your search runs, it is important that you minimize the amount of processing time required by each component of the search. Your search can be slow because of the complexity of your query to retrieve events from the index. Below are some useful guidelines:

  1. Choose an Appropriate Time Frame

Set the time picker to search within the exact time window your results should be found. This will limit the number of buckets that will need to be searched in the specified index. For instance, if you specify a search for the past 24 hours, only buckets in the specified index with data for the last 24 hours will be searched. “All time” searches are discouraged and “Real-time” searches should almost never be used due to resource consumption.

  1. Use an Efficient Search Mode

Splunk has three search modes which are Fast, Smart and Verbose. Change your search mode depending on what you need to see. Select verbose mode sparingly, using only when needed. Since it returns all of the fields and event data it possibly can, it takes the longest time to run. See more on Splunk search modes here.

  1. Retrieve only what is needed

How you construct your search has a significant impact on the number of events retrieved from disk. Be restrictive and specific when retrieving events from the index. If you need only a portion of the whole data, limit your search early to extract the portion you need before any data manipulation or calculations. You can specify an index, source type, host, source, specific word or phrases in the events. Include as many search terms as possible in your base search. Also, use the head command to limit events retrieved when you need just a subset of the data and remove unnecessary fields from the search results by using commands such as fields and where.


index=audit sourcetype=access_combined host=admin (action=failed OR action=cancelled) | stats count by user

  1. Use Efficient SPL Commands

a) Choice of Commands

As mentioned above, you can arrive at the same results using a different combination of commands but your choice will determine the efficiency of your search. Below are a few tips:

      • Joins and Lookups – Avoid using multiple joins and lookups in your search. They are very resource-intensive. Perform joins and lookups only on the required data and consider using append over join.
      • Eval – Perform evaluations on the minimum number of events possible
      • Stats – Where possible, use stats command over the table command
      • Table – Use table command only at the end of your search since it cannot run until all results are returned to the search head.
      • Dedup – The stats command is more efficient than dedup. Consider listing the fields you want to dedup in the “by” clause of the stats command. For instance,

…| stats count by user action source dest


…| stats latest by user action source dest

Rather than

…| dedup user action source dest

b) Order of Commands

The order in which commands are specified in a search is extremely important since this determines where the commands are executed which could be at the search head or on the indexers. When part or all of a search is run on the indexers, the search processes in parallel and search performance is much faster. It is good to parallelize as much work as possible. The aim is to not overburden the search head by making the indexer(s) do some of the work.

If your commands can be arranged so that they execute on the indexer, the overall search will execute quicker. Move commands that bring data to the search head as late as possible in your search criteria

Streaming and Non-Streaming Commands

Understanding streaming and non-streaming commands is important in discussing the order of commands in a search. Non-streaming commands include transforming commands such as stats, timechart, top, rare, dedup, sort and append which operate on the entire result set of event data and are always executed on the search head regardless of order. To optimize your searches, place non-streaming commands as late as possible in your search string.

There are two types of streaming commands – Distributable and Centralized Streaming Commands. Distributable Streaming Commands such as eval, fields, rename, replace and regex operate on each event returned by a search regardless of the event order. They can be executed on the indexer. However, if any of the preceding search commands is executed on the search head, the distributable command will be executed on the search head as well. When possible, allow distributable streaming commands to precede non-streaming commands.

Similar to the distributable streaming commands, centralized streaming commands operate on each event returned by a search but event order is important and commands execute only on the search head.

To inspect your search, take a look at the Job Inspector and Search Job Properties. There are two search pieces: remoteSearch and reportSearch that show where parts of your search string are executed. RemoteSearch is the part of the search string executed on the remote nodes (indexers) while the reportSearch is the part executed on the search head.

Let’s look at a simple example to illustrate this:

index=security user=admin failed

| timechart count span=1h

| stats avg(count) as average

In this example, the base search retrieves events from the index and the search head executes the timechart and stats commands which are both transforming commands and outputs the results.

A second example:

index=network sourcetype=cisco_wsa_squid usage=” violation”

| stats count AS connections by username usage

| rename username as violator

| search connections >=10

In the above example, stats command is a transforming command so it will be executed on the search head. “Rename” is a distributable streaming command which could execute on the indexer but because it occurs after a transforming command, it will also execute on the search head.

For better performance, reorder the commands as follows so that “rename” precedes “stats” and is therefore executed on the indexer.

index=network sourcetype=cisco_wsa_squid usage=violation

| rename username as violator

| stats count AS connections by username usage

| search connections >=10

  1. Check the Job Inspector

Finally, check the job inspector tool to examine the overall stats of your search including where Splunk spent its time. Use the tool to troubleshoot search performance and understand the impact of knowledge objects (lookups, tags) on processing.

Reference Links






Want to learn more about optimizing your Splunk searches? Contact us today!

Splunk Alerts: Using Tokens to Prioritize Email Notifications

By: Brandon Mesa | Splunk Consultant



Splunk Enterprise enables alerting through a variety of native and external alert actions. You can enhance your Splunk environment by creating alerts that add custom functionalities. Common alert actions include logging events, outputting results to a lookup, running a script, telemetry endpoints, sending emails, and many more. Additional Alert actions can be found at www.splunkbase.com.

Use case

A client wants to monitor their Splunk environment and be notified when their servers are not performing as expected. The client wants to monitor various key performance indicators, including, CPU, Disk, and Memory. While the client wants to monitor the performance of all their infrastructure, the main priority is ensuring production servers have as little downtime as possible. For this reason, the client would like to be alerted through email notifications when their servers are not performing as expected. Splunk search results that return a production server not performing up to expectations should send an email notification with the “Priority” set to the “highest” setting, which will deliver an email to recipients with the high importance icon. This will enable end-users to easily identify email notifications that should be prioritized.

To monitor server health, a Splunk alert should be created and configured. Returned results that meet the alert’s defined threshold should send email notifications to target recipients. The email alert should dynamically set the email priority based on the returned search results.

For example, if 3 servers are returned when the scheduled search is run, then the server environment should be dynamically evaluated by Splunk. If a server from the production environment is in the list returned, then Splunk will send the email notification marked as “important”, if not, then the email notification will be sent without any “high priority” settings configured.


The “Send email” alert action, an out-of-box functionality that comes with Splunk Enterprise, partially meets the use case needs. Using the “Send email” alert action enables dynamic email notifications through the use of tokens. When an alert runs and search results are returned, the “Send email” action, evaluates the returned results to define the values for the tokenized fields set at configuration time.

Another way to configure email notifications is by using the SPL command “sendemail”. Various command arguments can be used with the “sendemail” command to configure email notifications. Furthermore, like the “Send email” alert action, the “sendemail” SPL command, enables users to dynamically define various email configuration settings with the use of tokens.


By default, Splunk enables the use of tokens for select configurations for email notifications. Various configuration fields are enclosed within the “$” symbol. By enclosing a field name with the “$” symbol ($<token>$), field values can be dynamically defined based on various criteria, including returned results. Tokens can be set through the alert action GUI, or via SPL, by using the “sendemail” command. Splunk allows the following fields to be tokenized for email notifications:

  • To
  • Cc
  • Bcc
  • Subject
  • Message
  • Footer

More information on tokens, email notifications, and setting up alert actions can be found below:





The goal is to notify users when servers meet or exceed a defined threshold. By using the “Send email” alert action, or the “sendemail” SPL command, we can alert specific recipients when servers are not performing up to expectations. We’ll also integrate the use of tokens to configure our email notification and define various email settings on the fly. The goal is to dynamically define our target recipients, as well as the email subject, message, and priority based on the returned results. To define these field values based on returned results we can use the tokens below for our configuration settings:

  • $result.To$
  • $result.Cc$
  • $result.Bcc$
  • $result.Subject$
  • $result.Message$

Using the token values above to configure our email alert will dynamically define the value for the target recipients as well as email subject and message. However, if we take a closer look at the configuration screen below, we notice that the “Priority” field is a drop-down that requires the selection of a static setting for the overall configuration of the alert being created:




By default, the “Send email” alert action requires the email priority to be set by selecting an option from the drop-down menu. Passing tokens to the “Priority” field is not an out-of-box feature with Splunk, whether the email notification is configured through SPL or via the GUI as an alert action. The goal of this blog is to walk you through how to optimize your Splunk environment to enable dynamic email “Priority” configuration with the use of tokens.


Let’s consider the out-of-box functionalities in a Splunk environment:

  • “Send email” Alert action which enables triggered alerts to send email notifications
  • SPL command (“sendemail”) that can send email notifications based on returned SPL results
  • Tokens can be applied to various fields including To, Cc, Bcc, Subject, Message, and Footer to generate dynamic field values based on results returned.

We know the “Priority” field can’t be tokenized and therefore cannot be dynamically defined. This limits our email notifications to have a static email priority configuration setting. If the email priority setting is set to normal, and a result of high importance is returned, we’re at a loss. Email notifications will not be configured to send as important in this specific case. Now, let’s consider the functionalities needed to meet the defined requirements:

  • Enable dynamic email prioritization based on our search results
  • Pass tokens to our “Priority” field, as shown below:


We can enhance our Splunk environment to include this functionality by creating a new alert action and SPL command that mimics the functionality of the native “Send email” alert action and “sendemail” SPL command. Configurations for alert actions can be found in the alert_actions.conf file, while search command configurations are located on the command.conf file.

Let’s take a look at our Splunk environment, perhaps we can identify files which are related to our current “Send email” alert action or “sendemail” command. Follow the steps below in your Splunk Enterprise instance, via the CLI:

  1. $ cd $SPLUNK_HOME/etc/apps/search/default
  2. $ cat commands.conf – you should see a “sendemail” stanza

Looking at our “sendemail” stanza in the default commands.conf, we see that “sendemail.py” is executed to enable the “sendemail” command functionality. The goal is to clone, analyze, and modify the new custom sendemail.py script to enable the tokenization of the priority field.

To mimic the native “sendemail” command and “Send email” alert action, complete the following steps:

  1. Create a custom app with the necessary permission settings. More information on creating a custom app:


  1. Create a new commands.conf and alert-actions.conf file that replicates the configurations settings from the native “Send email” alert action and “sendemail” command functionalities.

You can do this by copying the stanza settings from the default .conf files and adjusting the settings as needed. Remember, our goal is to create a new SPL command (“sendemail”) and alert action (“Send email”) which mimics the native Splunk functionalities, yet enables the tokenization for the “Priority” field.

Modify the configuration settings to create a new command and alert action as needed. For example, you can name your new command “custom_sendemail” to differentiate it from the native Splunk SPL “sendemail” command.

  1. Place your commands.conf and alert-actions.conf in the default directory of your custom app.
  2. Place your modified copy of the “sendemail.py” script into the bin directory of your custom application. Remember, this script should reflect source code changes that enable the tokenization of the priority field.
  3. In addition to the commands.conf, alert-actions.conf, and sendemail.py files ensure your custom app has the appropriate *.html files to ensure your custom alert action is available in the user interface. You can use the “Developer Tools” feature within the Google Chrome browser to copy and modify the HTML source code for the “Send email” alert action.

See below:

At this point, your new custom app should include the following directories along with the following files:

  • custom_app
    • appserver
      • static
        • png
      • bin
        • py
      • default
        • conf
        • conf
        • conf
        • data
          • ui
            • alerts
          • html
        • metadata
          • meta


The custom commands.conf and sendemail2¬.py files will create a new SPL command that enables the tokenization of the “Priority” field. The alert_actions.conf and email_priority.html files will create a new alert action in your Splunk environment which enables you to pass tokens to the “Priority” field. For example, rather than having a drop-down menu for the email “Priority” selection, this alert action will enable you to pass the $result.Priority$ token and further enable dynamic email notifications.


Your custom app should be deployed to your Splunk search heads and placed in the $SPLUNK_HOME/etc/apps/ path. If you are running a search head cluster, the custom app should be placed on the deployer’s, $SPLUNK_HOME/etc/shcluster, and pushed down to the cluster members.

Once your custom app is deployed to your Splunk environment, you can configure dynamic email notifications one of two ways:

  • By creating an alert and selecting your new alert action. Pass the $result.FieldName$ token to the “Priority” field to configure email priority based on returned search results.
  • By creating and saving a SPL search that uses the new “sendemail2” command. Verify that you have included the “Priority” command argument and pass the token within the search result as follows: Priority=$result.FieldName$. This will ensure your email notification priority is dynamically configured based on returned search result values.

Want to learn more about using tokens to prioritize email notifications?  Contact us today!

Using Federal RMF and Splunk for IT Systems That Do Not Require Regulatory Compliance

By: Don Arnold | Splunk Consultant


Regulatory compliance for IT systems is an area that can be mysterious and not well understood by many IT managers or security professionals.  Some industries are mandated to obtain regulatory compliance certifications for their IT systems, ie HIPAA for healthcare, PCI DSS for credit card transactions, Sarbanes Oxley for accounting, ISO for manufacturing, and FISMA for Federal systems.  However, compliance only applies to a small portion of IT systems and the rest are left to implement a mix of devices and techniques to secure their systems.  However, without a definitive roadmap to follow many IT systems can be left with holes in their security programs, which can increase the risk of attack and make their systems vulnerable to compromise.

Since the majority of IT systems are not required to implement compliance standards, it doesn’t mean they can’t use these standards as a roadmap to better secure their systems.  Many IT security professionals try to build a security practice within their network by using technical solutions.  Though this is good, following a security framework helps to ensure that all areas of security are covered.

One of the best and most complete security frameworks that is publicly available and comes at no cost to use is the Risk Management Framework from the U.S. Government, better known as the Federal RMF.  The Federal RMF uses a number of special publications from the National Institute of Standards and Technology (NIST) to help organize a company’s cybersecurity implementation into a life cycle framework.  The steps in the life cycle include preparation, categorization of the system, selection of applicable security controls, implementation, assessment, authorization, and monitoring.  Each step specifically identifies what happens and uses NIST documents to outline how to follow each one.

The last step in the RMF life cycle is monitoring and the NIST SP 800-53 specifically identifies an entire family of controls for Continuous Monitoring.  Splunk, an industry-leading SIEM, can be implemented to meet the Continuous Monitoring needs of Account Management, Privileged Access, Login Access and other items.  In addition, Splunk has several add-on applications, such as Infosec and Enterprise Security, that can assist with out-of-the-box data searches and dashboards to ensure your cybersecurity Continuous Monitoring needs are being met.

If your IT system is subject to regulatory compliance from HIPAA, PCI, SOX, etc., then the framework has a detailed roadmap for you to follow.  However, if your system does not require compliance, but you’re looking for a framework as a guideline to follow to ensure you’re covering your cybersecurity needs, then the Federal RMF and Splunk are some of the best tools on the market today.  TekStream Solutions has cybersecurity engineers on staff with years of experience implementing cybersecurity solutions and can assist you with building a strong cybersecurity program using the Federal RMF.

Want to learn more about Splunk and cybersecurity? Contact us today!







Machine Learning with Splunk: Testing Logistic Regression vs Support Vector Machines (SVM) using the ML Toolkit

By: Brent McKinney | Splunk Consultant


If you’ve ever spent time working with Machine Learning, it’s highly likely you’ve come across Logistic Regression and Support Vector Machines (SVMs). These 2 algorithms are amongst the most popular models for binary classification. They both share the same basic goal: Given a sample x, how can we predict a variable of interest y?

For example, let’s say we have a dataset with samples (x) containing the following,

and we want to determine a single variable y: Is this person diabetic or not?

Logistic Regression and SVMs are perfect candidates for this!

The problem now lies in finding the means to test this on a sizeable dataset, where we have hundreds or thousands of samples. Coding machine learning algorithms can become quite a task, even for experienced developers. This is where Splunk comes in!

Splunk’s Machine Learning Toolkit makes testing ML algorithms a breeze. The Machine Learning Toolkit is an app, completely free to download on Splunkbase and allows users to visualize and compare results from ML algorithms quickly, without having to code them.

To stay consistent with our previous example, I will be demonstrating this with a public dataset from Kaggle.com, in the form of a CSV file. This dataset includes real data, that is already labeled and clean. Since our data is properly labeled, this can be will serve as a supervised learning problem. This simply means that our task will learn a function that maps an input (our x features) to an output (our y value – diabetic or not) based on the labeled items. I’ve posted the link to the MLTK app, as well as the dataset used in this example, as sources at the bottom of this page.

To install the MLTK app: Once you’ve downloaded the Machine Learning Toolkit from Splunkbase, log into your Splunk instance, and click the Apps dropdown at the top of the screen. Select “Manage Apps” and then click the button “Install app from file”. From here select Choose File and select the MLTK app folder (no need to untar the file, Splunk will unpack the folder on the server!). Click Upload.

To upload a csv file: You can upload the csv file by clicking Settings>Lookups>Lookup table files>New Lookup Table File. Select MLTK for the app, our csv as the upload file, and give a name with the .csv extension (diabetes.csv). Then go to Settings>Lookups>Lookup definition>New Lookup Definition to define the lookup. We’ll select the MLTK app, “diabetes” for the name, “File-based” for the type, and the csv file for the Lookup file.

Once the Machine Learning Toolkit has been installed, and the dataset file has been uploaded to Splunk, we can get to work.

From your Splunk instance, navigate to the Machine Learning Toolkit app by selecting it from the “App” dropdown menu at the top of the screen.

From here we can see there are several options available, each depending on the problem you are trying to solve. We want to categorize our data into 2 categories: diabetic and not diabetic. So for our example, we will use “Predict Categorical Fields”.

To start, select “Experiments” from the navigation bar, and then “Create New Experiment”. Select the appropriate experiment type, then add a title and description.


Once all looks good, select Create.

Now we are brought to the experiment screen. To use our diabetes dataset, we will need to use the SPL inputlookup command in the search bar. Note the search must begin with a | as this is a generating command.

This will return the data we uploaded from the CSV file.

As we can see, there are a few parameters that need to be set. The first being the algorithm we want to use. We will be testing Logistic Regression and SVM. The default is Logistic Regression so we can leave it as-is for now.

The next parameter is “Field to Predict”. This represents the variable we want to discover, y. This list is populated with fields found in our csv file. In our example, our  y variable is “Outcome”, which gives a value of 1 for samples that are diabetic, and a value of 0 for samples that are NOT diabetic.



The next parameter is “Fields to use for predicting”. As the name implies, these are the variables that make up our feature sample x. The algorithms will use these features to determine our Outcome variable. The more relevant fields we select here, the more accurate our algorithms will be when calculating a result, so in this case we will select all of them.

Once these parameters have been set, all we need to do is decide how we want to split the data into training and testing.

Machine Learning algorithms use the training data to determine a function that most accurately produces the desired output. So to achieve the best accuracy, we want to use a majority of the data for training. Once the algorithm is trained on the dataset, it runs this function on the test data and gives an output based on the samples it saw during training. For this example, I will use 80% for training, and 20% for testing.

(Note, while we want to use as much training data as possible, we must have some test data. If we use 100% of the data for training, then any test data will have already been seen by the algorithm, and therefore not give us any insightful results.)

Now that all of our parameters are set, we are ready to see results!

Select Fit Model to run Logisitic Regression on our data.

Once the algorithm is finished, we are given 3 panels.

The first returns a results table containing our test data. The columns on the right of the bold line show our original x features for each sample. The columns on the left of the bold line show the output that the algorithm predicted for each sample, compared to the actual output in the dataset, for each sample, highlighting the ones it got wrong.

The panel on the bottom left shows the degree of accuracy of the algorithm for our given dataset. From this we can conclude that if we were to give this model a new sample, it would determine whether or not the sample is diabetic or not with a 77% degree of accuracy.

The bottom right panel gives a little more detail, showing how well the algorithm did at predicting each outcome. We can see that for our particular example, it did slightly better at determining samples that were not diabetic, as opposed to samples that were.

Now let us compare this to a SVM model. Considering that we want to use the same dataset and parameters, all we need to do is change the algorithm.

Once that is set, we can select Fit Model to run SVM on our data.

Right away we can see that using Support Vector Machines gives us substantially better results than Logisitic regression. Both algorithms give the same details format, but we can see that using SVM resulted in a 97% accuracy when predicting on our test data, in comparison to LR resulting in 77%.


To conclude, Splunk’s Machine Learning Toolkit provides an easy-to-use environment for testing and comparing Machine Learning algorithms. In this demonstration, we used Splunk and Machine Learning to create models to predict whether a given sample is diabetic or not. While this demonstration focused on SVMs and Logistic Regression, there are many more algorithms available in Splunk’s Machine Learning Toolkit to play around with, including Linear Regression, Random Forests, K-means Clustering, and more!

Link to download Machine Learning Toolkit app:


Link to download dataset used in this example:



Want to learn more about Machine Learning with Splunk? Contact us today!