Splunk KvStore Migration

/0 Comments/in , /by

By: Christopher Winarski | Splunk Consultant and

Bruce Johnson | Director, Enterprise Security

 

Migrating your Splunk environment can be a daunting task to some. With the worry of missing valuable data. Did my users’ settings migrate properly? Did all my applications migrate properly? Did all my lookup tables survive the migration? If you find yourself performing a Splunk migration you may be asking yourself some of these questions. Well, today I try to take one of those worries off your chest by walking you through a Splunk KvStore Migration, more specifically migrating the Splunk KvStore from a Search Head Cluster to a new Search Head Cluster. E.g On-prem Shcluster to AWS Shcluster

KvStore stores data in key-value pairs known as Collections. These tables of data are located in your collections.conf files. Records contain each entry of your data, similar to a row in a database table. Using KvStore as opposed to csv files you can define the storage definition schema for your data, perform create-read-update-delete operations on individual records using Splunk REST API and lookups using the Splunk search language. KvStore excels in performance when you start getting large lookups with many data points which is especially prevalent within Enterprise Security, one of Splunk’s Premium Apps.

The normal export/migration is to use csv export which is not really practical for large KvStores due to the limitations to file sizes on most operating system’s is what drive was used for mongodb in the first place. Gemini KvStore Tools helps to circumvent the normal semi-workable, tedious migration process.

Gemini KV Store Tools comes with some custom commands built for the Splunk search bar that makes our life/migration less complicated. The commands we are interested in for this migration are:

  • | Kvstorebackup
  • | Kvstorerestore

Requirements for this process:

  • You must already be utilizing Splunk’s KvStore for your lookups.
  • Downloaded and installed “Gemini KV Store Tools” application in both the originating environment Search Head Cluster and the new environment Search Head Cluster you are migrating too. https://splunkbase.splunk.com/app/3536/
  • You must have already migrated/copied the applications from the old Search Head Cluster. We are interested in the collections.conf within these applications.
    • tar -zcf apps.tgz /opt/splunk/etc/shcluster/apps
  • The collections.conf files must be present on the new environment before proceeding

 

Step 1: Of the original Search Head Cluster, Identify the kvstore captain, and log into the GUI environment, then open the search app. The KvStore captain is the instance in the search head cluster that receives the write operations regarding the KvStore collections where the Search head captain is the instance in the search head cluster that schedules jobs, pushes knowledge bundles to search peers, and replicates any runtime changes to knowledge objects throughout the search head cluster.**note** This may be different than the Search Head captain

 

Step 2: On this instance, also log into the backend and create a directory under the /tmp directory named “kvstore_backup”. Ensure Splunk has read/write permissions to this folder.

cd /tmp

mkdir kvstore_backup

sudo chown -R splunk:splunk /tmp/kvstore_backup

 

Step 3: Creates a json file per each collection to the destination path in the kvstore_backup folder, as well as should see “Success” per each collection zipped within the original environment. In the search bar on original environment KvStore captain, run:

| kvstorebackup path=”/tmp/kvstore_backup” global_scope=”true” compression=”true”

 

Step 4: Check KvStore monitoring console to verify if collection counts are listed and save the page to refer to the results/counts to verify later. (old environment)

Monitoring Console > Search > KvStore:Instance

 

Step 5: Now that you have created your collection backups and have verified that the number of records per is correct. Go on each new search head cluster member (CLI) and edit server.conf to have:

[kvstore]

                                                                        oplogSize = 10000

Also, on each instance, you have to edit/change the search head replication factor to 1 in the new environment on each search head cluster member. (server.conf)

[shclustering]

replication_factor = 1

Once both are set, restart the instance. Do this for every search head cluster member in the new environment.

 

Step 6: Identify and get Search Head captain to be the same instance as the kvstore captain.

Ensure the kvstore captain = Search Head Cluster captain.

Useful commands:

./splunk show shcluster-status

./splunk show kvstore-status

Transfer captaincy to one node by bootstrapping the kvstore captain as the search head captain.

On the KvStore captain, we want to make it also the search head captain, run this command (CLI):

./splunk edit shcluster-config -mode captain -captain_uri <URI>:<management_port> -election false

On each other non-captain instance, run this command (CLI):

./splunk edit shcluster-config -mode member -captain_uri <URI>:<management_port> -election false

This will allow you to specify the captain as the kvstore captain and get rid of dynamic captaincy for this purpose. At the end we will want to revert our search head cluster back to a dynamic captaincy.

 

Step 7: Once you have the kvstore captain = search head captain, Log into the CLI of the other search head nodes (every search head cluster member that is not the captain/kvstore captain). Starting the instance after cleaning the local kvstore will initialize a kvstore synchronization upon startup with kvstore captain.

SHUTDOWN Splunk: ./splunk stop

run: ./splunk clean kvstore –local

Then start splunk: ./splunk start

 

Step 8: SCP kvstore_backup from Step:2 to new environment search head captain/kvstore captain. Make sure that splunk has permissions to access the file. Follow these steps for guidance.

Old instance where the backup was created from:

scp -r kvstore_backup ec2-user@IPADDRESS:/tmp

Move file to /opt folder on kvstore/search head captain:

mv kvstore_backup /opt/kvstore_backup

Change ownership of the file and internal files to splunk for permissions

sudo chown -R splunk:splunk /opt/kvstore_backup

 

Step 9: Kvstore Gemini Tools is to be installed on the new search head cluster prior to running this step, if you have not done so please insure it is installed within the new search head cluster. Once the kvstore_backup has the permissions and is in place on the backend of the kvstore captain/search head captain. Now log on to the GUI of that splunk instance, open search and run:

| kvstorerestore filename=”/opt/kvstore_backup/*.json.gz”

On big restores, can take many minutes for the restore to complete, be patient and let the search run.

 

Step 10: Verify lookups return the same results in the new environment as back in the old environment with the saved page(step 4) , run:

| inputlookup <Lookup definition>

 

Step 11: We want to revert the search head cluster back to a dynamic captaincy now (the static captaincy bootstrapping was just used for the migration) and also change our replication factor back to the original setting in the environment.

You can do this by logging on to each instance CLI, stopping splunk then on the search head cluster captain, run:

./splunk edit shcluster-config -mode captain -captain_uri <URI>:<management_port> -election true

On the other non captain search head cluster members run:

./splunk edit shcluster-config -mode member -captain_uri <URI>:<management_port> -election true

Then we want to edit the config file again to revert replication factor back to the original number that was set before the migration. (server.conf)

[shclustering]

replication_factor = 2

**The “2” is arbitrary here, as this should be set to the number that was present prior to the migration**

That’s it! Migrations can be a scary endeavor and if not prepared, one can easily lose data. If you seek further assistance don’t hesitate to reach out to us here at TekStream Solutions. We would be happy to help! No Splunk project is too small or too big.

How to Set Up Splunk DB Connect to Connect to Multiple MSSQL Databases and Some Tips & Tricks

/0 Comments/in , /by

By: Jon Walthour |Team Lead, Senior Splunk Consultant

 

Over the years, I have found one tried and true method for getting Splunk connected to multiple Microsoft SQL Server instances spread across a corporate network—connect to Windows from Windows. That is to say, run the DB Connect application from Splunk on a Splunk Enterprise Heavy Forwarder, installed on a Windows environment. Why must Splunk be running Windows? It certainly doesn’t if you’re going to authenticate to the MSSQL instances with local database accounts. That authentication process can be handled by the database driver. However, when multiple connections to multiple MSSQL instances are required, as is often the case, a bunch of local account usernames and passwords can be a nightmare to manage for everyone involved. So, Windows AD authentication is preferred. When that becomes a requirement, you need a Windows server running Splunk. I tried getting Splunk running on Linux to connect to SQL Server using AD authentication via Kerberos for a month and never got it to work. Using a Windows server is so much simpler.

To accomplish this, the first thing you need to do is request two things from your Infrastructure teams—a service account for Splunk to use to connect to all the SQL Server instances and a server running Microsoft Windows. The service account must have “logon as a service” rights and the Windows server must meet the requirements for Splunk reference hardware with regards to CPUs, memory and storage. The best practice for Splunk generally speaking is to use General Policy Objects (GPOs) to define permissions so that they are consistent across a Windows environment. Relying on local Admin accounts can result in challenges, particularly across some of the “back-end” Splunk instances such as Splunk Search Head to Indexer permissions.

Once the server and service account have been provisioned, install Splunk Enterprise and Splunk DB Connect (from Splunkbase) on the it. Here’s the first trick: go into Settings > Control Panel > Services and configure the splunkd service to run under the service account. This is crucial. You want not just the database connections to be made using the service account, but the Splunk executables to be running under that account. This way, all of Splunk is authenticated to Active Directory and there are no odd authentication issues.

After you have Splunk running under the MSSQL service account with DB Connect installed as an app in the Splunk instance, you’ll want to install the Java Runtime Environment (JRE) software, either version 8 (https://www.oracle.com/java/technologies/javase-jre8-downloads.html) or version 11 (https://www.oracle.com/java/technologies/javase-jdk11-downloads.html), and download the appropriate MSSQL driver based on Splunk’s documentation (https://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Installdatabasedrivers), which either the Microsoft drivers for the open source jTDS drivers. Personally, I’ve had better outcomes with the Microsoft drivers in this scenario.

Once you’ve downloaded the SQL database driver archive, unzip it. In the installation media, find the library “mssql-jdbc_auth-<version>.<arch>.dll” appropriate to the version and architecture you downloaded and copy it to the C:\Windows\System32 directory. Then, find the file jar “mssql-jdbc-<version>.<jre version>.jar” appropriate to your JRE version and copy it to $SPLUNK_HOME\etc\apps\splunk_app_db_connect\drivers.

Now, log into Splunk and go the Splunk DB Connect app. It will walk you through the configuration of DB Connect. In the “General” section, fill in the path to where you installed the JRE (JAVA_HOME). This is usually something like “C:\Program Files\Java\jre<version>”. The remaining settings you can leave blank. Just click “Save”. This will restart the task server, which is the java-based processing engine of DB Connect that runs all the database interactions.

In the “Drivers” section, if the MS SQL drivers are not listed with green checkmarks under the “Installed” column, click the “Reload” button to have the task server rescan the drivers folder for driver files. If they still do not have green checkmarks, ensure the right driver files are properly placed in $SPLUNK_HOME/etc/apps/splunk_app_db_connect/drivers.

Next, navigate to Configuration > Databases > Identities and click “New Identity”. Enter the username and password of the service account you’re using for the MSSQL connections and give it an appropriate name. Check “Use Windows Authentication Domain” and enter the appropriate value for your Active Directory domain. Save the identity.

Navigate to Configuration > Databases > Connections and click “New Connection”. Pick the identity you just created and use the “MS-SQL Server using MS Generic Driver With Windows Authentication” connection type. Select the appropriate timezone the database you’re connecting to is in. This is especially important so that Splunk knows how to interpret the timestamps it will ingest in the data. For the “host” field, enter the hostname or IP address of the MSSQL server. Usually the default port of 1433 doesn’t need to be changed nor the default database of “master”. Enable SSL if you’re connection is to be encrypted and I always select “Read Only” when creating a database input to make sure there is no way to input can change any data in the connected database.

Finally, a few miscellaneous tips for you.

For the “Connection Name” of database connections, I always name them after their hostname and port from the JDBC URL Settings. This is because in a complex DB Connect environment, you can have many inputs coming from many different databases. A hostname/port number combination, however, is unique. So, naming them with a pattern of “hostname-port#” (e.g., “sql01.mycompany.com-1433”) will prevent you from establishing duplicate connections to the same MSSQL installation.

Another tip is that you can edit the connection settings for your JDBC driver directly in the configuration. This is typically only useful when your development team has come up with specific, non-standard configurations they use for JDBC drivers.

Sometimes complex database queries that call stored procedures or use complex T-SQL constructions can be more than the JDBC driver and Task Server can handle. In that case, I ask the MSSQL DBAs if they will create a view for me constructed of the contents of the query and provide me select rights on the view. That leaves all the complex query language processing with SQL server rather than taxing the driver and DB Connect.

When dealing with ingesting data from a SQL server cluster, the usual construction of the JDBC connection string created by DB Connect won’t do. With a clustered environment, you also need to specify the instance name in addition to the hostname and port of the SQL Server listener. So, after setting up the connection information where the host is the listener and the port is the listener port, click the “Edit JDBC URL” checkbox and add “;instance=<database instance name>” to the end of the JDBC URL to ensure you connect to the proper database instance in the cluster. For example, the get to the “testdb” instance in the “sql01” cluster, you’d have a JDBC URL like: “jdbc:sqlserver://sql01.mycompany.com:1433;databaseName=master;selectMethod=cursor;integratedSecurity=true;instance=testdb”

I hope these directions and tips have been helpful in making your journey into Splunk DB Connect simpler and straightforward.

Happy Splunking!

Want to learn more about setting up Splunk DB Connect to connect to multiple MSSQL databases? Contact us today!

Creating Splunk Alerts (and Setting Permissions!) Through REST API

/0 Comments/in , /by

By: Marvin Martinez | Senior Developer

 

Creating Alerts via the Spunk REST API is fairly straightforward once you know exactly what parameters to use to ensure that Splunk recognizes the Saved Search as an Alert.  The same applies for ACL permissions on these alerts and other Splunk Knowledge Objects.

First things first, let’s create a scheduled search via REST using the “/services/saved/searches” endpoint.  The curl code below creates a simple search to pull some data from the _internal index for the last 10 minutes.

Note that, to create the saved search, all that was needed was authorization (a token in this case) and a couple of parameters in the call: (1) a name for the search and (2) the search itself.

This will create a search in the Searches, Reports and Alerts screen in Splunk Web.

As you can see, the search has been created and shows up as a Report.  But what if you need this to be an alert?! Even more importantly, what if you want to set this up with specific permissions?  Well, luckily, like essentially everything else in Splunk, this can also be done via the REST API.

To create a new search as an alert, you’ll need to call the same endpoint as shown above with the parameters mentioned below. Otherwise, call the “/services/saved/searches/{name}” endpoint if you’re modifying a search that’s already created.  For the purposes of this write-up, I will call the endpoint to manage an already created search (“/services/saved/searches/{name}”).

In order for Splunk to recognize the search as an alert, and not a Report, the following parameters have to be set correctly and passed along in your POST REST call.  The table below outlines the parameter name and a brief description of what they mean.

Parameter Description
alert_type ‘number of events’ (if this is set to ‘always’, which is the default, Splunk thinks it’s just a report)
is_scheduled true (this is a Boolean setting that Splunk checks to make sure there’s a set schedule for the report, which is required for alerts)
cron_schedule */10 * * * * (a cron schedule that represents the schedule which the alert will run on)
alert_comparator ‘greater than’ (this is the operator used in the alert settings to determine when to send the alert – associated with the alert_threshold below)
alert_threshold 0 (this is the number to compare with the operator above. i.e. only alert when results > 0)

 

The curl command for the REST call is shown below.  Note the aforementioned parameters that are now being included.

But how does this search now look in the Searches screen?  As can be seen in the image below, once the REST command has been executed successfully, your Alert should now be reflected appropriately as an “Alert”.

For further confirmation of these settings, click the Edit link under Actions, and click Advanced Edit from the drop-down menu.  This will bring up a lengthy listing of all the settings for this search.  If using the REST API is not your style, this is where you can alternately set these settings from Splunk Web.

The listing looks something like this:

All that’s left now is to set your permissions as desired.  To do this, you’ll need to call a new endpoint.  You’ll use the previous endpoint you used to manage a specific saved search, but you’ll add a new section at the end for “acl” (i.e. ‘https://localhost:8089/services/saved/searches/ATestRESTSearch/acl’).  This acl extension/option is available for any endpoint but, in this use case, we’ll use it to manage the permissions for the alert we created above.

In the case of a saved search, you’ll need to include the following parameters in your REST call:

Parameter Description
sharing ‘app’ – this can also be ‘global’ or ‘user’, depending on what the scope of the access you want this search to have (This is required when updating the ACL properties of any object)
app ‘search’ – this is the name of the app that this search belongs to.  (For saved searches, this is required when updating ACL properties of these objects)
perms.read A comma-delimited list indicating what roles to assign read permissions to
perms.write A comma-delimited list indicating what roles to assign write permissions to

 

A curl command that was used in this case is shown below.  In this example, the alert is being updated to give read permissions to admin and user-mmtestuser1.  Additionally, it is being updated to give write permissions to admin and power roles.

As an added bonus, here is an example of how Postman was leveraged to make this final call, in case that’s your REST API-calling tool of your choice.  The Authorization tab, in this example, was set to Basic Auth type with admin credentials.  In the Body tab, you’ll set your parameters to the REST call as “x-www-form-urlencoded” values.  Note the 4 parameters mentioned above shown included in the call below.

Once the REST call is made, navigate to your “Searches, Reports, and Alerts” screen in Splunk Web, and click to Edit Permissions of your alert.  You’ll notice that your permissions are now reflected just the way you designated them in your REST call.

The Splunk REST API is a great alternative, and a necessity for many, to using Splunk Web to create and manage knowledge objects.  Anything that can be done in Splunk Web can be done via the REST API, though it sometimes can be a bit hard to easily understand the process for how to achieve some of these desired actions.  Now, you can easily create alerts and set the permissions just the way you want…and all through REST!

Want to learn more about creating alerts via the Spunk REST API? Contact us today!

 

Using an External Application to Pull Splunk Search Results

/0 Comments/in , /by

By: Aaron Dobrzeniecki | Splunk Consultant

 

Have you ever wanted to pull logs from Splunk without actually being physically signed into the Splunk Search Head? With an external application, such as Postman, you can query the Splunk REST API endpoint to actually provide you with results from a search being run.

When Splunk runs a search, it creates a search ID which we can use to grab the results from the REST endpoint. We will be testing out two ways to get the results of a search. The first way is to grab the name of the Splunk search and query it against the /services/saved/searches/{search_name}/dispatch endpoint, which will provide us with the sid. We then use the sid to grab the results of the search, which will fire off the search and will poll for results as they come in. The second way to get the search results is by doing an export on the search name which will run the search and get the results without polling.

First things first, you need to make sure that the user you are authenticating to Splunk with has the “Search” capability, as well as access to search the necessary indexes. It’s that simple! If you are setting up a user for a particular person make sure they only have access to what they need. Giving further access is not necessary and can cause security issues.
In this example we are using the Postman application to query the Splunk REST API to grab search results from a couple of different reports/saved searches. Things we are going to need include:

  • Splunk user account with the Search capability. We need that user to be able to search the index we are going to be grabbing our data from.
  • We also need to know the Splunk URL we are going to be pulling from. In this case, I am using my localhost as an example. We will also be querying the Splunk management port of 8089 to get our results set.

The image above shows the type of request I am doing (POST), the REST API being used to query my search ID (/services/saved/searches/{name_of_search}/dispatch), and the authentication type of username and password. What the URL above is doing is it is reaching out to Splunk and grabbing the SID (search id) of the search named Index Retention Getting Close. With this search id we will be able to run a GET on the Splunk REST API and grab the results of the search.

Below I will be showing you two Splunk REST API endpoints that you can query (using POST) to get the Search ID for a specified search. The first endpoint is for searches that do not have Global permissions. As long as the user you are authenticating with has a role that has access to read the search, you can query the endpoint of /servicesNS/nobody/{app}/saved/searches/{name}/dispatch to retrive the Search ID. The second endpoint you can query if the search has Global permissions and you have read access is simply /services/saved/searches/{name}/dispatch to retrieve the Search ID. The two scenarios are below.

The image above shows the rest endpoint that can be used to grab a specific search ID that is in an app and has specific permissions. As long as my account has access to the app and search inside the app, I will be able to query it. For this example, we have changed the permissions of the search to be App only.

The image above is the results of the search in json, using the search ID we queried from the REST API.

The image above gives us the same results except they are in xml format.

The image above shows the search ID of the search with REST API I am querying. Since that search now has Global permissions, we do not need to use the ServicesNS endpoint. When you do a POST with a dispatch on the name of a search/report you will get the Search ID. As you can see the search ID is circled. We will be using this search ID to query the results of the search and show the actual search results in the Postman application. The Splunk REST API you will want to query next is the /services/search/jobs/{sid}/results?output_mode= (atom | csv | json | json_cols | json_rows | raw | xml). Any of those values will get you the results of the search in the format selected. In this example, I will be showing you json and xml.

As you can see above, the data results are shown in xml format for the search we were wanting to get results from.

This image shows the same results but in json format. With the options above for data output, you can query the Splunk REST API to get the search results and have them show in your preferred format.

Way 2: Query the REST API to show the results by using an export on the search name which will run the search and get the results without polling. Take a look at the screenshot below which queries the /services/search/jobs endpoint to stream in the results of the search as they come in.

Remember, you need to have the Search capability in Splunk, as well as you have to be able to read the results of the search. Whether that is setting Global permissions or having a role that has read access to the app and search. Below are some links referencing the Splunk REST API. If you have any questions at all regarding querying the Splunk REST API from an external application, please let me know!

https://docs.splunk.com/Documentation/Splunk/8.0.6/RESTTUT/RESTsearches

https://docs.splunk.com/Documentation/Splunk/8.0.6/RESTREF/RESTsearch#search.2Fjobs.2Fexport)

 

Want to learn more about using an external application to pull Splunk search results? Contact us today!

Forwarder 6.x Compatibility with Splunk 8.0

/0 Comments/in , /by

By: Forrest Lybarger | Splunk Consultant

 

If you are looking into upgrading Splunk to 8.0, you have probably come across the compatibility matrix for forwarders:

Source: https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwardersandindexers

 

This table means that Splunk does not support, nor has it tested, the use of 6.x forwarders with 8.0 indexers. It doesn’t mean that it is impossible for them to work together. In other words, you can use 6.x forwarders at your own risk. Any problems you have with these forwarders, however, will almost always be caused by the version difference and most likely fixed by upgrading.

With all the caveats out of the way, how do you get this working? Well, it depends on what exact version your forwarders have. Here are the affected versions:

  • 6.0.0 to 6.0.6
  • 6.1.0 to 6.1.4
  • 6.2.0 to 6.2.6
  • 6.3.0 to 6.3.1
  • 6.3.1511.1

The issue is that some older 6.x versions of Splunk use a different SSL protocol from 6.6.x and later versions, which makes them unable to connect via the management port (usually port 8089) and unable to communicate with the deployment server. To correct this, you need to force the newer Splunk components to use an SSL version that the older components can understand. In this case, your forwarders are the only components not upgrading to 8.0, so you only need to fix the deployment server. To avoid issues with these forwarder versions add an app with a server.conf containing this stanza to your deployment server:

[sslConfig]

sslVersions = *,-ssl2

sslVersionsForClient = *,-ssl2

cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

Allow any sslConfigs apps your environment already has to override this app by giving it a lower priority name or just add the lines from the stanza that aren’t present in your current app. You can delete this new ssl config after your forwarders are upgraded.

This fix should only be used if you must upgrade to 8.0 and can’t wait for your forwarders to upgrade. Keep in mind that this is not Splunk supported, so for now it could work (latest version as of writing this is 8.0.6), but in the future, Splunk could break this workaround. When you do implement this fix, make sure to prioritize upgrading your forwarders and understand that any problems involving data ingestion or forwarding are most likely caused by not upgrading your forwarders to at least 7.0 (latest version possible is recommended).

Want to learn more about forwarder 6.x compatibility with Splunk 8.0? Contact us today!

 

TekStream Promoted to Premier Tier in Splunk Partner+ Program

/0 Comments/in , , , /by

TekStream, an Atlanta-based digital transformation technology firm, announced it has once again achieved Premier Partner status in the Splunk Partner+ Program.

In order to achieve Premier Partner status, partners must achieve $2 million in sales over the past 12 months and staff accreditations commensurate with the tier. With its Premier status, TekStream’s Splunk customers benefit from an enhanced level of engagement, commitment, and support.

By including TekStream in its Premier Partner Tier, Splunk has recognized TekStream for its outstanding achievement and commitment to Splunk market development, strategic prioritization, and customer success.

“I’m proud of our team and the hard work they’ve put in to achieve this accomplishment.  It’s especially impressive considering the circumstances we’ve all endured this year.  I’m excited about the momentum this creates heading into 2021 for our team, our customers, and Splunk” said Matthew Clemmons, Managing Director of the Splunk practice at TekStream.

About TekStream
TekStream accelerates clients’ digital transformation by navigating complex technology environments with a combination of technical expertise and staffing solutions. We guide clients’ decisions, quickly implement the right technologies with the right people, and keep them running for sustainable growth. Our battle-tested processes and methodology help companies with legacy systems get to the cloud faster, so they can be agile, reduce costs, and improve operational efficiencies. And with 100s of deployments under our belt, we can guarantee on-time and on-budget project delivery. That’s why 97% of clients are repeat customers. For more information visit https://www.tekstream.com/

Working with Multivalue Fields in Splunk

/0 Comments/in , /by

By: Yetunde Awojoodu | Splunk Consultant

 

Have you ever come across fields with multiple values in your event data in Splunk and wondered how to modify them to get the results you need? Each field in an event typically has a single value, but for events such as email logs you can often find multiple values in the “To” and “Cc” fields. Multivalue fields can also result from data augmentation using lookups. To properly evaluate and modify multivalue fields, Splunk has some multivalue search commands and functions. If you ignore multivalue fields in your data, you may end up with missing and inaccurate data, sometimes reporting only the first value of the multivalue field(s) in your results.

In this article, I have applied a simple scenario to illustrate how different multivalue commands and functions can be used individually or combined to meet different use cases. I will cover some common search commands and functions that work with multivalue fields. Note that multivalue functions can be used with eval, where or fieldformat search commands. In my illustrations, I employed the “makeresults” command to generate hypothetical data for my searches so that anyone can recreate them without the need to onboard data. Read more on the makeresults command.

 

Scenario

Within one purchase transaction, Mary bought eggs, milk and bread. She paid for the eggs with cash and covered the remaining items using her credit card. We can assume that this purchase transaction is equivalent to a log event. The values for each multivalue field are separated by the comma delimiter.

Example 1:

Please note that in all the results, I have deliberately excluded the default field, “_time” which is a default field generated when the makeresults command is used.

 

Makemv (Command)

This command is used to split the values of a field that appear like a single value into multiple values within an event based on the delimiter. A delimiter specifies the boundary between characters.

Example 2:

The values in the “groceries” field have been split within the same event based on the comma delimiter. The values in the “payment” field remain the same. The report shows the method of payment for all three grocery items but it does not specify the actual payment method used for each item. To expand the event into three separate events, one for each item and show the exact payment for each grocery item, we will need a combination of commands and functions.

 

Mvzip (Function)

The mvzip function is used to tie corresponding values in the different fields of an event together. This helps to keep the association among the field values. This function takes two multivalue fields, X and Y, and combines them by stitching together the first value of X with the first value of field Y, then the second with the second, and so on.

Example 3:

The new field, “zipped” is the result of the mvzip function. The values of the groceries and payment fields are properly zipped together before expanding into separate events. Note that at this point, the results are still within one event.

 

Mvexpand (Command)

This command expands the values of a multivalue field into separate events, one event for each value in the multivalue field. All other single field values and unexpanded multivalue field values will remain the same in each new event.

Example 4:

Mvexpand works great at splitting the values of a multivalue field into multiple events while keeping other field values in the event as is but it only works on one multivalue field at a time. For instance, in the above example, mvexpand cannot be used to split both “zipped” and “payment” fields at the same time. The next function will come in handy to accomplish this.

 

Mvindex (Function)

Having zipped the values and created one field, “zipped”, you can now expand the “zipped” field into multiple events. The mvindex function is a little more intricate. To further tie field values together so that accurate associations are made in the process of expanding the values into separate events, mvindex will separate the existing multivalued field into two chosen fields using index values. Indexes can start at zero if labeling from the first value. For example, if values= a,e,i,o,u; a=0 e=1 i=2 o=3 u=4. You could also label from the last character with -1; a=-5 e=-4 i=-3 o=-2 u=-1 or you could choose to have a combination of both index patterns; a=0 e=1 i=2 o=-2 u=-1.

Example 5:

Mvindex was used to assign index 0 to the first value in the group which represents groceries and index 1 to the second value representing payment method so that when the fields are split, the values will not get mixed up. The “split” command was used to separate the values on the comma delimiter. Using mvindex and split functions, the values are now separated into one value per event and the values correspond correctly.

Tip – The stats command can also be used in place of mvexpand to split the fields into separate events as shown below:

Example 6:

 

Mvcount (Function)

This function can be used to quickly determine the number of values in a multivalue field using the delimiter. If the field contains a single value, the function returns 1 and if the field has no values, the function returns NULL.

Example 7:

As with single value fields, keep in mind that you may need a combination of multivalue commands/functions to get your report in the required format that will meet your specific use case.

Note: If there are situations in your data where a field is sometimes multivalue and other times null, refer here

 

Want to learn more about working with multivalue fields in Splunk? Contact us today!

 

A Use Case for Ingest Time Eval

/0 Comments/in , /by

By: Zubair Rauf | Senior Splunk Consultant

 

A few days ago, I came across an interesting challenge that a customer put in front of me. They had been facing this for some time now. The customer works with an app that logs all of its events 7 hours ahead of Eastern time, irrespective of daylight savings time. The server clock reset to midnight when Eastern time was 5:00 PM all year round. To work around this problem and make sure the events were always synced with the correct time zone, they adjusted the sourcetype for those logs every time daylight savings time started or ended.

When presented with this problem, I spent a good amount of time to find a time zone that would change with eastern time when daylight savings time changed and have the same time offset as those logs. Not having any success on that front, I started looking at alternatives to help my customer overcome their issue and I came across this powerful way to solve the problem with a one-time fix with the sourcetype.

Splunk introduced Ingest time evals with Splunk Enterprise 7.2. Ingest time evals are similar to search time evals that have helped Splunk be the powerful tool that it always has been. Ingest time evals allow you to write an EVAL formula that is executed at ingestion time to create a new indexed field or to update a field’s value. They give you more control over Splunk index time fields as well. In my particular case, having control over and being able to manipulate index time fields helped me just do the trick for my customer.

For starters, _time is an index time field that is parsed from the raw log event. If the event does not have a time, the indexer will assign it with a current time when the event is ingested. In my particular challenge, the _time field needed a fixed offset by five hours as it was five hours ahead of eastern time.

To setup ingest time evals, we have to work with transforms.conf, props.conf, and fields.conf (only if creating new fields at ingest time). To further elaborate on the process of setting up ingest time evals to create new index time fields or manipulate existing fields at index time, we have used a sample log from a Cisco device.

To do a comparison, I ingested the log file with a custom sourcetype I created to parse the events.

With the above sourcetype, the following events were ingested.

If you look closely, the date/time was parsed exactly as it appears in the raw log event. Now if the raw event had a timestamp that needed to be offset, we could change the _time field at ingest time using ingest time eval.

To make my required changes, I will have to add an INGEST_EVAL expression in a transforms stanza in transforms.conf to update the _time field at ingest time after it has been parsed out from the actual event.

In the above example, I have used INGEST_EVAL to update my _time field to add 7200 seconds to it. This translates into 2 hours. I have also used the “:=” instead of “=” so that Splunk updates the _time field and not create another _time value resulting in a multivalued _time field in the final event. In this case, “:=” will overwrite the existing value in the field.

The above screenshot shows the updated _time field after the same log file has been ingested with the updated props and transforms. If you closely look at the Time column in the above screenshot in the first event it shows the timestamp being parsed as 01/16/20 1:43:43 PM but the timestamp in the event is 01/16/2020 11:43:31 AM. This tells us that the INGEST_EVAL expression in our transforms.conf successfully worked.

At this point, I would caution you to thoroughly test your INGEST_EVAL on a dev Splunk server so that you are sure that your eval works.

Ingest time eval can also be used to create new index time fields. While updating the _time field to offset the time difference, I thought about creating some custom index fields for demonstration purposes. This would further demonstrate how powerful ingest time evals are and how they can be useful.

Considering I was updating the _time field with my new timestamp, I figured it would be good to have a field that still parses and stores the original time. I named that field orig_time. This field is basically derived from the original _time field that was parsed before it was changed into the new timestamp.

I also thought it would be good to calculate the raw length of the event at ingest time, as that would create a field for me to calculate the size of the ingested data later. I particularly leaned towards demonstrating this, because not too long ago, I was also faced with the challenge to report host-level licensing information for every index. This helps Splunk users in an organization understand how much data their hosts are sending to Splunk.

Now, this is an easy fix if your environment is small. In that case, you can use the license_usage.log file available in the _internal index to calculate your license usage by index, sourcetype, source, or host. It definitely does become a problem when your environment grows too large. When the unique tuples cross 2000 by default, the license manager starts squashing source/host values and only index, sourcetype values remain in license_usage.log.

To work with this issue, I set up a daily license usage search which calculates the length of _raw for the past day for all the indexes and stores it in a summary index. This search runs at off-peak hours when the system is not being used by other users. That helps me populate my dashboards on demand for the users who want to see this data the next day.

Having raw event size calculated for every event at index size will definitely help me rid myself of those expensive searches that need to be run every night, these searches can be less reliable in case the search head that runs the summary generating search crashes. At index time I create a new field “event_size” using INGEST_EVAL in transforms.conf. The settings used to do this are as below;

If you look closely at the settings;

Transforms.conf

I have added two new stanzas to the transforms.conf to create the evals for the new fields, orig_time, and event_size.

Fields.conf

As we are creating two new fields at ingest time, we add their names as stanzas in fields.conf and make sure these fields are indexed by adding the parameter “INDEXED = true”.

Props.conf

I have updated the TRANSFORMS parameter in the relevant sourcetype. If you notice, the order of the TRANSFORMS stanza actually dictates which transform will be applied first to the data being parsed. In this particular case, the stanza is:

TRANSFORMS = orig-time,time-offset,event-size

In this specific transform the order will be as follows:

  • orig-time will preserve the original parsed time into the orig_time
  • time-offset will update the existing _time field to be offset by 02 hours.
  • event-size will calculate the total length of the event and create a new event_size field.

If you look at the final screenshot (above) closely on the left under “Interesting Fields” you will see that there are two new fields that you can see. These include orig_time and event_size

Now to calculate the total license usage by any measure, you can use your event_size with a | tstats search which will be many folds faster than a regular search.

There can be many other uses for Ingest Time Evals, one of which is listed on the documents page. To find out more, please visit Splunk documentation at https://docs.splunk.com/Documentation/Splunk/8.0.2/Data/IngestEval#Why_use_ingest-time_eval.3F

 

If you want to learn more or have TekStream help with implementing some Splunk use cases, contact us today!

Splunk Phantom Workbooks

/0 Comments/in , /by

By: Joe Wohar | Senior Splunk Consultant

 

Splunk Phantom is an amazing software used to automate cybersecurity processes, however, many companies do not know that they could also be using Phantom for case management. Arguably the most powerful, yet unknown to many, case management feature of Phantom is the ability to create and use workbooks.

If you’re familiar with Phantom, then you know that Phantom playbooks are repeatable processes that Phantom runs through against events. Phantom workbooks are repeatable, defined processes that analysts run through against events. However, they’re typically only used when an analyst needs to get involved. When an event is determined to be a threat that needs to be investigated by an analyst, the event should be promoted to a case. This can be done with a manual change done on the event (by clicking the toolbox icon button) or by having the conditions specified in a playbook that can then turn the event into a case.

Image 1: A workbook must be selected when converting an event to a case.

 

One of the biggest advantages of workbooks is that it’s a great way of ensuring that your analysts (new or old) are following the same set of steps when working cases. SOPs help define processes for your analysts to follow, but workbooks put those processes right into the case and make the work easily trackable. Workbooks are made up of 2 trackable components: phases and tasks.

 

Phases

Phases split the investigation into different sections, such as identification, acquisition, analysis, and reporting. Individual SLAs can be set for each phase of a workbook. When SLAs are missed/breached, there is a panel on the Phantom home page for tracking that:

Image 2: Home page SLA breach tracker.

 

Phases are made up of tasks, which are where the specific steps for investigations are listed.

Image 3: Adding new phases/tasks to a workbook.

 

Tasks

Tasks are very customizable, so they can be pretty general with few trackable requirements or be very specific with many tracked steps. First, tasks can have a default owner assigned to them, which could be useful if you want to have a “review” task so that a more experienced analyst can review a newer analyst’s work, however, I think most often you’d want to leave that blank so that tasks can be assigned to the analyst working the case. The description section of the task is where you can describe the specific things that should be done in the task. If you don’t want to track specific steps, you can simply use this section to create a list of steps for analysts to follow. However, if you have very specific steps involved in a task, you may want to use the description just for describing the process and then have the steps listed as actions or playbooks. This brings us to the next part of tasks, adding actions and playbooks.

Actions and playbooks are Phantom automation being added to the human process. The actions and playbooks added to a task are limited to the actions available in your configured apps and the playbooks that you have available in your Phantom instance. Then, when an analyst goes to run the action from the investigation screen, the action is already pulled up and they just need to enter the details.

Image 4: Workbook opened in a case with 2 tasks.

 

Image 5: Pop-up window from clicking the “run query” action in the workbook.

 

Running a playbook from a workbook is even simpler. Just click the playbook and click the “Run Playbook” button.

Image 6: Pop-up window from clicking the “Disabled User” playbook in the workbook.

 

As analysts move through the tasks and complete them, the phase’s tracker will be updated to show completion and whether or not tasks were completed on time and if the phase was completed on time.

Images 7 & 8: First task complete and then both tasks completed.

 

If you’re not using Phantom for case management, then you’re likely using Phantom to create tickets and add details to them in another software, which is costing you more in hardware and licensing. By using Phantom for case management, you’ll save the cost of another software and its hardware while using software you’ve already bought at no additional cost.

Not sure how to get started with workbooks? Try taking one of your best defined SOPs and make a workbook for it. If you’re not currently a Phantom customer and would like to try it out, you can download the OVA by registering here: https://my.phantom.us/

 

Want to learn more about Phantom workbooks? Contact us today!