TekStream Recognized in 2021 Splunk Global and Regional Partner Awards

TekStream Named 2021 Global Services Partner of the Year and AMER Professional Services Partner of the Year for Outstanding Performance


TekStream today announced it has received the 2021 Global Services Partner of the Year and 2021 AMER Professional Services Partner of the Year awards for exceptional performance and commitment to Splunk’s Partner+ Program. The 2021 Global Services Partner of the Year Award recognizes a partner with excellence in post-sale and professional services implementations. This partner demonstrates a strong commitment to technical excellence, certifications, and customer satisfaction. The 2021 AMER Professional Services Partner of the Year Award recognizes an AMER Splunk partner that is actively engaged in services implementations, in addition to having a strong commitment to training and certification of their organization. For more information on Splunk’s Partner+ Program, visit the Splunk website.

“We are delighted to have won the 2021 Global Services Partner of the Year and 2021 AMER Professional Services Partner of the Year awards. It is a fantastic achievement to be awarded and even more satisfying to contribute to the success of Splunk and our customers. Our team is very excited to be recognized for its efforts and expertise and will wear this prized recognition proudly,” said Matthew Clemmons, Managing Director at TekStream.

“Congratulations to TekStream for being named the 2021 Splunk Global Services Partner of the Year and 2021 AMER Professional Services Partner of the Year,” said Bill Hustad, VP, Global GTM Partners, Splunk. “The 2021 Splunk Global Partner Awards highlight partners like TekStream that deliver successful business outcomes, as well as help our joint customers leverage Splunk’s Data-to-Everything Platform to drive value and unlock insights. Additionally, TekStream shares our commitment of prioritizing customer success.”

The Splunk Partner Awards recognize partners of the Splunk ecosystem for industry-leading business practices and dedication to constant collaboration. All award recipients were selected by a group of Splunk executives, theater leaders, and the global partner organization.

“We are very honored to have been selected by Splunk for not just one, but two Partner of the Year awards. TekStream prides itself on doing what is right for the customer above all else, and our commitment to that mantra drives everything that we do. We value our partnership and look forward to helping Splunk grow the ecosystem on its way to $5B,” said Karl Cepull, Senior Director, Operational Intelligence at TekStream.

About TekStream

TekStream accelerates clients’ digital transformation by navigating complex technology environments with a combination of technical expertise and staffing solutions. We guide clients’ decisions, quickly implement the right technologies with the right people, and keep them running for sustainable growth. Our battle-tested processes and methodology help companies with legacy systems get to the cloud faster, so they can be agile, reduce costs, and improve operational efficiencies. And with 100s of deployments under our belt, we can guarantee on-time and on-budget project delivery. That’s why 97% of clients are repeat customers. For more information visit https://www.tekstream.com/.

JSON Structured Data & the SEDCMD in Splunk

By: Khristian Pena | Splunk Consultant



Have you worked with structured data that is not following its structure? Maybe your JSON data has a syslog header. Maybe your field values have an extra quote, colon, or semicolon and your application team cannot remediate the issue. Today, we’re going to discuss a powerful tool for reformatting your data so automatic key-value fields are extracted at search-time. These field extractions utilize KV_MODE in props.conf to automatically extract fields for structured data formats like JSON, CSV, and from table-formatted events.



KV_MODE = [none|auto|auto_escaped|multi|json|xml]

This article will focus on the JSON structure and walk through some ways to validate, remediate and ingest this data using the SEDCMD.  You may have used the SEDCMD to anonymize, or mask sensitive data (PHI,PCI, etc) but today we will use it to replace and append to existing strings.


JSON Structure

JSON supports two widely used (amongst programming languages) data structures.

  • A collection of name/value pairs. Different programming languages support this data structure in different names. Like object, record, struct, dictionary, hash table, keyed list, or associative array.
  • An ordered list of values. In various programming languages, it is called as array, vector, list, or sequence.


An object starts with an open curly bracket { and ends with a closed curly bracket } Between them, a number of key value pairs can reside. The key and value are separated by a colon : and if there are more than one KV pair, they are separated by a comma ,


  “Students“: [

                                      { “Name“:”Amit Goenka” ,

  “Major“:”Physics” },

                                      { “Name“:”Smita Pallod” ,

  “Major“:”Chemistry” },

                                      { “Name“:”Rajeev Sen” ,

  “Major“:”Mathematics” }




An Array starts with an open bracket [ and ends with a closed bracket ]. Between them, a number of values can reside. If more than one values reside, they are separated by a comma , .



  “name“: “Bidhan Chatterjee”,

  “email“: “bidhan@example.com”



  “name“: “Rameshwar Ghosh”,

  “email“: “datasoftonline@example.com”




JSON Format Validation:
Now that we’re a bit more familiar with the structure Splunk expects to extract from, let’s work with a sample. The sample data is JSON wrapped in a syslog header. While this data can be ingested as is, you will have to manually extract each field if you choose to not reformat it. You can validate the structure by copying this event to https://jsonformatter.curiousconcept.com/ .

Sample Data:

May 14 13:28:51 <redacted_hostname> github_audit[22200]: { “above_lock_quota”:false, “above_warn_quota”:false, “babeld”:”eebf1bc7″, “babeld_proto”:”http”, “cloning”:false, “cmdline”:”/usr/bin/git upload-pack –strict –timeout=0 –stateless-rpc .”, “committer_date”:”1589477330 -0400″, “features”:” multi_ack_detailed no-done side-band-64k thin-pack include-tag ofs-delta agent=git/″, “frontend”:”<redacted>”, “frontend_pid”:17688, “frontend_ppid”:6744, “git_dir”:”/data/user/repositories/7/nw/75/42/9d/4564/6435.git”, “gitauth_version”:”dcddc67b”, “hostname”:”<redacted>”, “pgroup”:”22182″, “pid”:22182, “ppid”:22181, “program”:”upload-pack”, “quotas_enabled”:false, “real_ip”:”″, “remote_addr”:”″, “remote_port”:”15820″, “repo_config”:”{\”ssh_enabled\”:\”true\”,\”ldap.debug_logging_enabled\”:\”true\”,\”auth.reactivate-suspended\”:\”true\”,\”default_repository_permission\”:\”write\”,\”allow_private_repository_forking\”:\”true\”}”, “repo_id”:6435, “repo_name”:”<redacted>”, “repo_public”:true, “request_id”:”43358116096ea9d54f31596345a0fc38″, “shallow”:false, “status”:”create_pack_file”, “uploaded_bytes”:968 }


The errors are noted and highlighted below:

As we can see, the timestamp, hostname and thread field are outside of the JSON object.


Replace strings in events with SEDCMD

You can use the SEDCMD method to replace strings or substitute characters. This must be placed on the parsing queue prior to index time. The syntax for a sed replace is:

SEDCMD-<class> = s/<regex>/\1<replacement>/flags

  • <class> is the unique stanza name. This is important because these are applied in ABC order
  • regexis a Perl language regular expression
  • replacementis a string to replace the regular expression match.
  • flagscan be either the letter g to replace all matches or a number to replace a specified match.
  • \1 – use this flag to insert the string back into the replacement


How To Test in Splunk:

Copy the data sample text into a notepad file and upload using Splunk’s built in Add Data feature under Settings to test. Try out each SEDCMD and note the difference in the data structure for each attribute.



Props.conf – Search Time Field Extraction


KV_MODE = json


Want to learn more about JSON structured data & the SEDCMD in Splunk? Contact us today!

Custom Document Security in Oracle WebCenter Portal 12c

By: Greg Becker | Splunk Consultant, Team Lead


During a recent project, a customer had business units that had a requirement to provide more granular security in the Documents hierarchy than the out-of-the-box WebCenter Portal 12c product can accommodate. For this project, we used the security features that are available within WebCenter Content using document accounts and security groups. With this approach, all that we had to do was configure a unique folder entry point from Portal into Framework Folders within Content and the remainder was simply configurations within WCC. The remainder of this document outlines that process.

This approach requires configurations in various locations:

  1. Group membership in the Identity Management store
  2. Security Accounts and Security Groups in WebCenter Content
  3. Credential map settings in WebCenter Content
  4. Folder security settings in WebCenter Content


Group Membership:

From an Administrative perspective (AD/LDAP)

  • Create the new AD/LDAP group naming, come up with a consistent naming convention and use any existing suggestions from the AD/LDAP team
  • Get the groups created and assign users to the groups.
  1. Login to the appropriate Identity Management tool
  2. Create a new group and assign users to that group
    1. Name the new group
    2. Assign users to the new group
    3. Click “Create” to create the new group with selected users
    4. Repeat for any additional groups that are required


Create the new Security Accounts in WebCenter Content:

  1. Login to WebCenter Content as an administrator
  2. Navigate to Administration -> Admin Applets
  3. Start the User Admin applet
  4. Choose Security -> Predefined Accounts

5. Add the new accounts to accommodate the defined folder structure. Below is an example structure:


Update the Credential Map in WebCenter Content:

  1. Login to WebCenter Content as an administrator
  2. Navigate to Administration -> Credential Maps
  3. Choose the appropriate credential map to modify
  4. In this example we will copy the Allowancing section and make the appropriate edits (this is where the IdM group maps to UCM accounts and security groups)

5. Add the new entries for the new folder

6. Click Update to save


Add the new folder with security attributes in Content UI for WebCenter Content:

  1. Login to WebCenter Content UI as an administrator (https://<WebCenter Portal>/wcc/faces/wccmain)
  2. Navigate to the appropriate top-level folder already assigned to the portal that you are working with:

3. Click the icon in the upper right to create a new folder

4. Name the folder HAZMAT and click Save

5. In the folder tree right-click on the new folder and choose Properties to modify the security for the folder

6. In the Folder Properties dialog box switch to the Security tab and ensure that the new account is selected and then click Save



At this point, you should be able to login with the users that you assigned to the new group in IdM and verify that they see the appropriate new folder. Logging in with the new user shows the following folder result:


Specify the starting folder within WebCenter Content:

When you have created your folders and set permissions within WebCenter Content you can then specify the starting folder within your Portal – this is the key to taking advantage of the custom security. The following image shows where the configuration happens for the Content Manager.



After you have configured this solution in a test environment you can use the following guidelines to migrate from one environment to another.

  • Migrate AD/LDAP groups and user configs as needed
  • Use CMU bundles inside Content to migrate security accounts (or re-create manually)
  • Copy over Credential Map
  • Use Archiver to move the folder structure to PROD
  • Use Archiver to move the entire batch of content if it was moved into the TEST environment
    • Alternatively, only do some test content and only contribute ‘real’ content when PROD is ready
  • Validate


Want to learn more about custom document security in Oracle WebCenter Portal 12c? Contact us today!

Splunk KvStore Migration

By: Christopher Winarski | Splunk Consultant and

Bruce Johnson | Director, Enterprise Security


Migrating your Splunk environment can be a daunting task to some. With the worry of missing valuable data. Did my users’ settings migrate properly? Did all my applications migrate properly? Did all my lookup tables survive the migration? If you find yourself performing a Splunk migration you may be asking yourself some of these questions. Well, today I try to take one of those worries off your chest by walking you through a Splunk KvStore Migration, more specifically migrating the Splunk KvStore from a Search Head Cluster to a new Search Head Cluster. E.g On-prem Shcluster to AWS Shcluster

KvStore stores data in key-value pairs known as Collections. These tables of data are located in your collections.conf files. Records contain each entry of your data, similar to a row in a database table. Using KvStore as opposed to csv files you can define the storage definition schema for your data, perform create-read-update-delete operations on individual records using Splunk REST API and lookups using the Splunk search language. KvStore excels in performance when you start getting large lookups with many data points which is especially prevalent within Enterprise Security, one of Splunk’s Premium Apps.

The normal export/migration is to use csv export which is not really practical for large KvStores due to the limitations to file sizes on most operating system’s is what drive was used for mongodb in the first place. Gemini KvStore Tools helps to circumvent the normal semi-workable, tedious migration process.

Gemini KV Store Tools comes with some custom commands built for the Splunk search bar that makes our life/migration less complicated. The commands we are interested in for this migration are:

  • | Kvstorebackup
  • | Kvstorerestore

Requirements for this process:

  • You must already be utilizing Splunk’s KvStore for your lookups.
  • Downloaded and installed “Gemini KV Store Tools” application in both the originating environment Search Head Cluster and the new environment Search Head Cluster you are migrating too. https://splunkbase.splunk.com/app/3536/
  • You must have already migrated/copied the applications from the old Search Head Cluster. We are interested in the collections.conf within these applications.
    • tar -zcf apps.tgz /opt/splunk/etc/shcluster/apps
  • The collections.conf files must be present on the new environment before proceeding


Step 1: Of the original Search Head Cluster, Identify the kvstore captain, and log into the GUI environment, then open the search app. The KvStore captain is the instance in the search head cluster that receives the write operations regarding the KvStore collections where the Search head captain is the instance in the search head cluster that schedules jobs, pushes knowledge bundles to search peers, and replicates any runtime changes to knowledge objects throughout the search head cluster.**note** This may be different than the Search Head captain


Step 2: On this instance, also log into the backend and create a directory under the /tmp directory named “kvstore_backup”. Ensure Splunk has read/write permissions to this folder.

cd /tmp

mkdir kvstore_backup

sudo chown -R splunk:splunk /tmp/kvstore_backup


Step 3: Creates a json file per each collection to the destination path in the kvstore_backup folder, as well as should see “Success” per each collection zipped within the original environment. In the search bar on original environment KvStore captain, run:

| kvstorebackup path=”/tmp/kvstore_backup” global_scope=”true” compression=”true”


Step 4: Check KvStore monitoring console to verify if collection counts are listed and save the page to refer to the results/counts to verify later. (old environment)

Monitoring Console > Search > KvStore:Instance


Step 5: Now that you have created your collection backups and have verified that the number of records per is correct. Go on each new search head cluster member (CLI) and edit server.conf to have:


                                                                        oplogSize = 10000

Also, on each instance, you have to edit/change the search head replication factor to 1 in the new environment on each search head cluster member. (server.conf)


replication_factor = 1

Once both are set, restart the instance. Do this for every search head cluster member in the new environment.


Step 6: Identify and get Search Head captain to be the same instance as the kvstore captain.

Ensure the kvstore captain = Search Head Cluster captain.

Useful commands:

./splunk show shcluster-status

./splunk show kvstore-status

Transfer captaincy to one node by bootstrapping the kvstore captain as the search head captain.

On the KvStore captain, we want to make it also the search head captain, run this command (CLI):

./splunk edit shcluster-config -mode captain -captain_uri <URI>:<management_port> -election false

On each other non-captain instance, run this command (CLI):

./splunk edit shcluster-config -mode member -captain_uri <URI>:<management_port> -election false

This will allow you to specify the captain as the kvstore captain and get rid of dynamic captaincy for this purpose. At the end we will want to revert our search head cluster back to a dynamic captaincy.


Step 7: Once you have the kvstore captain = search head captain, Log into the CLI of the other search head nodes (every search head cluster member that is not the captain/kvstore captain). Starting the instance after cleaning the local kvstore will initialize a kvstore synchronization upon startup with kvstore captain.

SHUTDOWN Splunk: ./splunk stop

run: ./splunk clean kvstore –local

Then start splunk: ./splunk start


Step 8: SCP kvstore_backup from Step:2 to new environment search head captain/kvstore captain. Make sure that splunk has permissions to access the file. Follow these steps for guidance.

Old instance where the backup was created from:

scp -r kvstore_backup ec2-user@IPADDRESS:/tmp

Move file to /opt folder on kvstore/search head captain:

mv kvstore_backup /opt/kvstore_backup

Change ownership of the file and internal files to splunk for permissions

sudo chown -R splunk:splunk /opt/kvstore_backup


Step 9: Kvstore Gemini Tools is to be installed on the new search head cluster prior to running this step, if you have not done so please insure it is installed within the new search head cluster. Once the kvstore_backup has the permissions and is in place on the backend of the kvstore captain/search head captain. Now log on to the GUI of that splunk instance, open search and run:

| kvstorerestore filename=”/opt/kvstore_backup/*.json.gz”

On big restores, can take many minutes for the restore to complete, be patient and let the search run.


Step 10: Verify lookups return the same results in the new environment as back in the old environment with the saved page(step 4) , run:

| inputlookup <Lookup definition>


Step 11: We want to revert the search head cluster back to a dynamic captaincy now (the static captaincy bootstrapping was just used for the migration) and also change our replication factor back to the original setting in the environment.

You can do this by logging on to each instance CLI, stopping splunk then on the search head cluster captain, run:

./splunk edit shcluster-config -mode captain -captain_uri <URI>:<management_port> -election true

On the other non captain search head cluster members run:

./splunk edit shcluster-config -mode member -captain_uri <URI>:<management_port> -election true

Then we want to edit the config file again to revert replication factor back to the original number that was set before the migration. (server.conf)


replication_factor = 2

**The “2” is arbitrary here, as this should be set to the number that was present prior to the migration**

That’s it! Migrations can be a scary endeavor and if not prepared, one can easily lose data. If you seek further assistance don’t hesitate to reach out to us here at TekStream Solutions. We would be happy to help! No Splunk project is too small or too big.

How to Set Up Splunk DB Connect to Connect to Multiple MSSQL Databases and Some Tips & Tricks

By: Jon Walthour |Team Lead, Senior Splunk Consultant


Over the years, I have found one tried and true method for getting Splunk connected to multiple Microsoft SQL Server instances spread across a corporate network—connect to Windows from Windows. That is to say, run the DB Connect application from Splunk on a Splunk Enterprise Heavy Forwarder, installed on a Windows environment. Why must Splunk be running Windows? It certainly doesn’t if you’re going to authenticate to the MSSQL instances with local database accounts. That authentication process can be handled by the database driver. However, when multiple connections to multiple MSSQL instances are required, as is often the case, a bunch of local account usernames and passwords can be a nightmare to manage for everyone involved. So, Windows AD authentication is preferred. When that becomes a requirement, you need a Windows server running Splunk. I tried getting Splunk running on Linux to connect to SQL Server using AD authentication via Kerberos for a month and never got it to work. Using a Windows server is so much simpler.

To accomplish this, the first thing you need to do is request two things from your Infrastructure teams—a service account for Splunk to use to connect to all the SQL Server instances and a server running Microsoft Windows. The service account must have “logon as a service” rights and the Windows server must meet the requirements for Splunk reference hardware with regards to CPUs, memory and storage. The best practice for Splunk generally speaking is to use General Policy Objects (GPOs) to define permissions so that they are consistent across a Windows environment. Relying on local Admin accounts can result in challenges, particularly across some of the “back-end” Splunk instances such as Splunk Search Head to Indexer permissions.

Once the server and service account have been provisioned, install Splunk Enterprise and Splunk DB Connect (from Splunkbase) on the it. Here’s the first trick: go into Settings > Control Panel > Services and configure the splunkd service to run under the service account. This is crucial. You want not just the database connections to be made using the service account, but the Splunk executables to be running under that account. This way, all of Splunk is authenticated to Active Directory and there are no odd authentication issues.

After you have Splunk running under the MSSQL service account with DB Connect installed as an app in the Splunk instance, you’ll want to install the Java Runtime Environment (JRE) software, either version 8 (https://www.oracle.com/java/technologies/javase-jre8-downloads.html) or version 11 (https://www.oracle.com/java/technologies/javase-jdk11-downloads.html), and download the appropriate MSSQL driver based on Splunk’s documentation (https://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Installdatabasedrivers), which either the Microsoft drivers for the open source jTDS drivers. Personally, I’ve had better outcomes with the Microsoft drivers in this scenario.

Once you’ve downloaded the SQL database driver archive, unzip it. In the installation media, find the library “mssql-jdbc_auth-<version>.<arch>.dll” appropriate to the version and architecture you downloaded and copy it to the C:\Windows\System32 directory. Then, find the file jar “mssql-jdbc-<version>.<jre version>.jar” appropriate to your JRE version and copy it to $SPLUNK_HOME\etc\apps\splunk_app_db_connect\drivers.

Now, log into Splunk and go the Splunk DB Connect app. It will walk you through the configuration of DB Connect. In the “General” section, fill in the path to where you installed the JRE (JAVA_HOME). This is usually something like “C:\Program Files\Java\jre<version>”. The remaining settings you can leave blank. Just click “Save”. This will restart the task server, which is the java-based processing engine of DB Connect that runs all the database interactions.

In the “Drivers” section, if the MS SQL drivers are not listed with green checkmarks under the “Installed” column, click the “Reload” button to have the task server rescan the drivers folder for driver files. If they still do not have green checkmarks, ensure the right driver files are properly placed in $SPLUNK_HOME/etc/apps/splunk_app_db_connect/drivers.

Next, navigate to Configuration > Databases > Identities and click “New Identity”. Enter the username and password of the service account you’re using for the MSSQL connections and give it an appropriate name. Check “Use Windows Authentication Domain” and enter the appropriate value for your Active Directory domain. Save the identity.

Navigate to Configuration > Databases > Connections and click “New Connection”. Pick the identity you just created and use the “MS-SQL Server using MS Generic Driver With Windows Authentication” connection type. Select the appropriate timezone the database you’re connecting to is in. This is especially important so that Splunk knows how to interpret the timestamps it will ingest in the data. For the “host” field, enter the hostname or IP address of the MSSQL server. Usually the default port of 1433 doesn’t need to be changed nor the default database of “master”. Enable SSL if you’re connection is to be encrypted and I always select “Read Only” when creating a database input to make sure there is no way to input can change any data in the connected database.

Finally, a few miscellaneous tips for you.

For the “Connection Name” of database connections, I always name them after their hostname and port from the JDBC URL Settings. This is because in a complex DB Connect environment, you can have many inputs coming from many different databases. A hostname/port number combination, however, is unique. So, naming them with a pattern of “hostname-port#” (e.g., “sql01.mycompany.com-1433”) will prevent you from establishing duplicate connections to the same MSSQL installation.

Another tip is that you can edit the connection settings for your JDBC driver directly in the configuration. This is typically only useful when your development team has come up with specific, non-standard configurations they use for JDBC drivers.

Sometimes complex database queries that call stored procedures or use complex T-SQL constructions can be more than the JDBC driver and Task Server can handle. In that case, I ask the MSSQL DBAs if they will create a view for me constructed of the contents of the query and provide me select rights on the view. That leaves all the complex query language processing with SQL server rather than taxing the driver and DB Connect.

When dealing with ingesting data from a SQL server cluster, the usual construction of the JDBC connection string created by DB Connect won’t do. With a clustered environment, you also need to specify the instance name in addition to the hostname and port of the SQL Server listener. So, after setting up the connection information where the host is the listener and the port is the listener port, click the “Edit JDBC URL” checkbox and add “;instance=<database instance name>” to the end of the JDBC URL to ensure you connect to the proper database instance in the cluster. For example, the get to the “testdb” instance in the “sql01” cluster, you’d have a JDBC URL like: “jdbc:sqlserver://sql01.mycompany.com:1433;databaseName=master;selectMethod=cursor;integratedSecurity=true;instance=testdb”

I hope these directions and tips have been helpful in making your journey into Splunk DB Connect simpler and straightforward.

Happy Splunking!

Want to learn more about setting up Splunk DB Connect to connect to multiple MSSQL databases? Contact us today!

Creating Splunk Alerts (and Setting Permissions!) Through REST API

By: Marvin Martinez | Senior Developer


Creating Alerts via the Spunk REST API is fairly straightforward once you know exactly what parameters to use to ensure that Splunk recognizes the Saved Search as an Alert.  The same applies for ACL permissions on these alerts and other Splunk Knowledge Objects.

First things first, let’s create a scheduled search via REST using the “/services/saved/searches” endpoint.  The curl code below creates a simple search to pull some data from the _internal index for the last 10 minutes.

Note that, to create the saved search, all that was needed was authorization (a token in this case) and a couple of parameters in the call: (1) a name for the search and (2) the search itself.

This will create a search in the Searches, Reports and Alerts screen in Splunk Web.

As you can see, the search has been created and shows up as a Report.  But what if you need this to be an alert?! Even more importantly, what if you want to set this up with specific permissions?  Well, luckily, like essentially everything else in Splunk, this can also be done via the REST API.

To create a new search as an alert, you’ll need to call the same endpoint as shown above with the parameters mentioned below. Otherwise, call the “/services/saved/searches/{name}” endpoint if you’re modifying a search that’s already created.  For the purposes of this write-up, I will call the endpoint to manage an already created search (“/services/saved/searches/{name}”).

In order for Splunk to recognize the search as an alert, and not a Report, the following parameters have to be set correctly and passed along in your POST REST call.  The table below outlines the parameter name and a brief description of what they mean.

Parameter Description
alert_type ‘number of events’ (if this is set to ‘always’, which is the default, Splunk thinks it’s just a report)
is_scheduled true (this is a Boolean setting that Splunk checks to make sure there’s a set schedule for the report, which is required for alerts)
cron_schedule */10 * * * * (a cron schedule that represents the schedule which the alert will run on)
alert_comparator ‘greater than’ (this is the operator used in the alert settings to determine when to send the alert – associated with the alert_threshold below)
alert_threshold 0 (this is the number to compare with the operator above. i.e. only alert when results > 0)


The curl command for the REST call is shown below.  Note the aforementioned parameters that are now being included.

But how does this search now look in the Searches screen?  As can be seen in the image below, once the REST command has been executed successfully, your Alert should now be reflected appropriately as an “Alert”.

For further confirmation of these settings, click the Edit link under Actions, and click Advanced Edit from the drop-down menu.  This will bring up a lengthy listing of all the settings for this search.  If using the REST API is not your style, this is where you can alternately set these settings from Splunk Web.

The listing looks something like this:

All that’s left now is to set your permissions as desired.  To do this, you’ll need to call a new endpoint.  You’ll use the previous endpoint you used to manage a specific saved search, but you’ll add a new section at the end for “acl” (i.e. ‘https://localhost:8089/services/saved/searches/ATestRESTSearch/acl’).  This acl extension/option is available for any endpoint but, in this use case, we’ll use it to manage the permissions for the alert we created above.

In the case of a saved search, you’ll need to include the following parameters in your REST call:

Parameter Description
sharing ‘app’ – this can also be ‘global’ or ‘user’, depending on what the scope of the access you want this search to have (This is required when updating the ACL properties of any object)
app ‘search’ – this is the name of the app that this search belongs to.  (For saved searches, this is required when updating ACL properties of these objects)
perms.read A comma-delimited list indicating what roles to assign read permissions to
perms.write A comma-delimited list indicating what roles to assign write permissions to


A curl command that was used in this case is shown below.  In this example, the alert is being updated to give read permissions to admin and user-mmtestuser1.  Additionally, it is being updated to give write permissions to admin and power roles.

As an added bonus, here is an example of how Postman was leveraged to make this final call, in case that’s your REST API-calling tool of your choice.  The Authorization tab, in this example, was set to Basic Auth type with admin credentials.  In the Body tab, you’ll set your parameters to the REST call as “x-www-form-urlencoded” values.  Note the 4 parameters mentioned above shown included in the call below.

Once the REST call is made, navigate to your “Searches, Reports, and Alerts” screen in Splunk Web, and click to Edit Permissions of your alert.  You’ll notice that your permissions are now reflected just the way you designated them in your REST call.

The Splunk REST API is a great alternative, and a necessity for many, to using Splunk Web to create and manage knowledge objects.  Anything that can be done in Splunk Web can be done via the REST API, though it sometimes can be a bit hard to easily understand the process for how to achieve some of these desired actions.  Now, you can easily create alerts and set the permissions just the way you want…and all through REST!

Want to learn more about creating alerts via the Spunk REST API? Contact us today!


Using an External Application to Pull Splunk Search Results

By: Aaron Dobrzeniecki | Splunk Consultant


Have you ever wanted to pull logs from Splunk without actually being physically signed into the Splunk Search Head? With an external application, such as Postman, you can query the Splunk REST API endpoint to actually provide you with results from a search being run.

When Splunk runs a search, it creates a search ID which we can use to grab the results from the REST endpoint. We will be testing out two ways to get the results of a search. The first way is to grab the name of the Splunk search and query it against the /services/saved/searches/{search_name}/dispatch endpoint, which will provide us with the sid. We then use the sid to grab the results of the search, which will fire off the search and will poll for results as they come in. The second way to get the search results is by doing an export on the search name which will run the search and get the results without polling.

First things first, you need to make sure that the user you are authenticating to Splunk with has the “Search” capability, as well as access to search the necessary indexes. It’s that simple! If you are setting up a user for a particular person make sure they only have access to what they need. Giving further access is not necessary and can cause security issues.
In this example we are using the Postman application to query the Splunk REST API to grab search results from a couple of different reports/saved searches. Things we are going to need include:

  • Splunk user account with the Search capability. We need that user to be able to search the index we are going to be grabbing our data from.
  • We also need to know the Splunk URL we are going to be pulling from. In this case, I am using my localhost as an example. We will also be querying the Splunk management port of 8089 to get our results set.

The image above shows the type of request I am doing (POST), the REST API being used to query my search ID (/services/saved/searches/{name_of_search}/dispatch), and the authentication type of username and password. What the URL above is doing is it is reaching out to Splunk and grabbing the SID (search id) of the search named Index Retention Getting Close. With this search id we will be able to run a GET on the Splunk REST API and grab the results of the search.

Below I will be showing you two Splunk REST API endpoints that you can query (using POST) to get the Search ID for a specified search. The first endpoint is for searches that do not have Global permissions. As long as the user you are authenticating with has a role that has access to read the search, you can query the endpoint of /servicesNS/nobody/{app}/saved/searches/{name}/dispatch to retrive the Search ID. The second endpoint you can query if the search has Global permissions and you have read access is simply /services/saved/searches/{name}/dispatch to retrieve the Search ID. The two scenarios are below.

The image above shows the rest endpoint that can be used to grab a specific search ID that is in an app and has specific permissions. As long as my account has access to the app and search inside the app, I will be able to query it. For this example, we have changed the permissions of the search to be App only.

The image above is the results of the search in json, using the search ID we queried from the REST API.

The image above gives us the same results except they are in xml format.

The image above shows the search ID of the search with REST API I am querying. Since that search now has Global permissions, we do not need to use the ServicesNS endpoint. When you do a POST with a dispatch on the name of a search/report you will get the Search ID. As you can see the search ID is circled. We will be using this search ID to query the results of the search and show the actual search results in the Postman application. The Splunk REST API you will want to query next is the /services/search/jobs/{sid}/results?output_mode= (atom | csv | json | json_cols | json_rows | raw | xml). Any of those values will get you the results of the search in the format selected. In this example, I will be showing you json and xml.

As you can see above, the data results are shown in xml format for the search we were wanting to get results from.

This image shows the same results but in json format. With the options above for data output, you can query the Splunk REST API to get the search results and have them show in your preferred format.

Way 2: Query the REST API to show the results by using an export on the search name which will run the search and get the results without polling. Take a look at the screenshot below which queries the /services/search/jobs endpoint to stream in the results of the search as they come in.

Remember, you need to have the Search capability in Splunk, as well as you have to be able to read the results of the search. Whether that is setting Global permissions or having a role that has read access to the app and search. Below are some links referencing the Splunk REST API. If you have any questions at all regarding querying the Splunk REST API from an external application, please let me know!




Want to learn more about using an external application to pull Splunk search results? Contact us today!

Driving Growth by Leveraging AWS and Document Understanding

Your company is sitting on a potential gold mine of stored data. Tucked away on servers and cloud-based drives are the answers and insights you need to take your business to that next level of growth. Advancements in machine learning and artificial intelligence have made it easier (and less expensive) to analyze this data through Document Understanding. Companies that leverage the Amazon Web Services (AWS) platform to support their needs, tying in a Document Understanding initiative can have a fundamental impact on driving growth and securing a more profitable bottom-line.

What is Document Understanding?

Historically, the chief hurdle to analyzing this data is that much of this data is unstructured – composed of text-based files, reports, survey results, social media posts, notes, and random PDFs. Sifting through this quagmire was expensive and inefficient as it had to be done by hand.

That was the old way.

Fueled by natural language processing (NLP) and machine learning (ML), these systems analyze text-based documentation (PDFs, notes, reports) to uncover insights. The machine-learning capabilities allow you to “teach” the AI how to read your specific documentation and guide its insight discovery.

How Document Understanding Can Benefit Your Enterprise Corporation

Enterprise companies are already tapping the power of AWS’s Document Understanding solution to garner essential insights into critical business functions.  Regardless of industry vertical, businesses are using Document Understanding to:

  • – Instantly search for information across multiple scanned documents, PDFs, images, reports, and stored text files.
  • – Redact critical information from documents and identify compliance threats in real-time.
  • – Digitize, store, and analyze customer feedback and request forms.
  • – Identify overarching communication trends and isolate specific messaging that can be used to improve the customer experience or marketing campaign.

And this is just the proverbial tip of the benefits iceberg. Through the machine learning aspect of Document Understanding, you can tailor your use of this technology to identify and analyze the data sets that have the most impact on your business and bottom line.

Driving Document Understanding through Intelligence with AWS Content Process Automation

As a certified AWS Advanced Consulting Partner, we are excited to announce the launch of our new AWS Content Process Automation (CPA) offering. Our new CPA tool integrates with the AWS platform to provide a structured process and streamlined toolset for implementing and managing an ongoing Document Understanding initiative.

Through our new AWS CPA offering, brands can:

  • – Make previously inaccessible data actionable at scale.
  • – Automate tendencies but necessary business processes.
  • – Improve compliance and risk management.
  • – Identify opportunities to increase operational efficiency and reduced costs.

How TekStream CPA Works

Historically, analyzing sizeable unstructured data sets for actionable information has been a time-consuming and costly initiative. Most of the work had to be done manually – which can be both costly and inefficient.

Our new CPA offering leverages artificial intelligence and machine learning, along with defined scope and direction, to increase the speed and accuracy for data discovery while eliminating much of the manual aspect of data mining.

Using machine-learning services like Amazon Textract and Amazon Rekognition, TekStream CPA inspects documents, images, and video (collectively called “files”), gathers key information and insights, and automatically stores these files logically to ensure easier access to critical information. Amazon Augmented AI (Amazon A2I) routes files requiring further review to content specialists and information managers to edit associated information, take corrective actions, and approve files for storage.

TekStream CPA relentlessly and automatically investigates content to find key insights and associations that might not be easily discovered by the naked eye. Users and administrators establish business rules defining what information is important, how it will be managed, and the storage rules for documents, images, forms, video files, and unstructured data. This ensures critical business facts and figures are available for business operations.

Built for Growth

Use these systems to gain a deeper understanding of internal and consumer audience sentiment around your brand or a specific product.

Analyzing your unstructured data sets is only part of the business growth equation. To achieve a true return on your investment and drive a noticeable impact on your bottom-line, you also need to transform your insights into actions. By leveraging serverless technologies like Amazon Lambda through our Content Process Automation tool, administrators can create functions to call their own services for file conversions, reformatting, and many more to meet specific business criteria.

Start driving business growth today. TekStream has deep experience helping clients across multiple industries accelerate their digital transformation and begin leveraging the power of Document Understanding to push their business forward. Reach out to us today to learn more about what CPA and Document Understanding can do for your business.

Want to learn more about unlocking value from your unstructured data? Download our latest eBook, “9 Steps to Unlocking Value from Your Unstructured Data and Content.”

8 Benefits to Using Document Understanding to Mine Unstructured Data

What if we told you that your business was sitting on a mountain of untapped business intelligence or that hidden away in archived emails, documents, and customer survey results are the very insights you need to drive growth and improve your bottom line? These types of text-based documents are a form of “unstructured data” and (alongside image libraries, data streams, and similar data deposits) account for nearly 80% of all the data that an enterprise company generates and stores.

How do you analyze all of this data to identify the specific insights that can drive change and improve performance in your organization? Through Document Understanding.

Understanding Document Understanding

Document Understanding is one of the three core AI capabilities fueling the unstructured data analysis industry (the other two being Computer Vision and IoT analysis). This system leverages the power of natural language processing and machine learning to analyze text-based documents (PDFS, notes, reports) to uncover actionable business insights.

The machine-learning capabilities of these systems allow your organization to “teach” the AI to read your specific documentation and discover insights that specific to your brand and audience.

8 Benefits of Analyzing Your Company’s Unstructured Data

The fact that the market size for natural language processing is estimated to reach over $16B by 2021 proves that organizations large and small are investing in tools and systems that analyze their unstructured data. This means that these companies are confident that the benefit of this work will outweigh the costs of these new systems.

While these benefits differ between industries, some of the key benefits to mining unstructured data includes:

1. Finding Opportunities to Improve Your Customer Experience

Retain more customers (and win over new fans) by using Document Understanding to analyze customer surveys and reviews to identify where your company can provide better customer service.

2. Discover New Opportunities in The Market

What is the “next big thing” in your industry? How will you ensure your company will stay relevant to consumers over the next 20 years? Turn your data lake into a blue ocean by mining your unstructured data for relevant insights and consumer trends.

3.  Know Your Audience Better With Sentiment Analysis

Use these systems to gain a deeper understanding of internal and consumer audience sentiment around your brand or a specific product.

4. Make Key Decisions Faster and More Accurately

Quit getting bogged down with analysis paralysis. Get the data you need to identify and take action on the “right” decision when it counts most.

5. Improve Team Productivity and Reduce/Remove Outdated Data Processing Techniques

Through automation, you can eliminate data processing bottlenecks and instead focus your employees on more high-value tasks.

6. Identify and Eliminate Unnecessary Cost Centers

Get a handle on your waste by understanding what areas of your business are costing you money (without providing a correlating ROI).

7.  Gain a Better Understanding of Your Customer Behavior and Buying Triggers

Improve the performance of your marketing campaigns and customer retention efforts by gaining more in-depth insight into what makes your customers your customers in the first place.

8.  Avoid Costly Regulatory or Compliance Issues

Uncover regulatory or compliance issues before they negatively impact your company.

Start with The End in Mind

Ready to get started analyzing your unstructured data, but not sure where to begin? We recommend starting with the end goal in mind. What is your highest unstructured data analysis priority? Are you sitting on a mountain of customer surveys? Are you curious about where your hidden costs centers are?

Understand which aspects of your unstructured data analysis will have an immediate impact on your business’s bottom line. Then work backward to develop the tools and systems you need to discover this intelligence.

If you are not sure where to begin, we can help. We’ve helped companies across a myriad of industries turn their unstructured data into business growth rocket fuel. Contact us today to learn how we can do the same for you.

If you’d like to learn more about how to unlock value from your unstructured data? Download our free eBook, “9 Steps to Unlocking Value from Your Unstructured Data and Content.”