Archive for category: Blog

By: Khristian Pena | Splunk Consultant

 

 

Have you worked with structured data that is not following its structure? Maybe your JSON data has a syslog header. Maybe your field values have an extra quote, colon, or semicolon and your application team cannot remediate the issue. Today, we’re going to discuss a powerful tool for reformatting your data so automatic key-value fields are extracted at search-time. These field extractions utilize KV_MODE in props.conf to automatically extract fields for structured data formats like JSON, CSV, and from table-formatted events.

Props.conf

[<spec>]

KV_MODE = [none|auto|auto_escaped|multi|json|xml]

This article will focus on the JSON structure and walk through some ways to validate, remediate and ingest this data using the SEDCMD.  You may have used the SEDCMD to anonymize, or mask sensitive data (PHI,PCI, etc) but today we will use it to replace and append to existing strings.

 

JSON Structure

JSON supports two widely used (amongst programming languages) data structures.

• A collection of name/value pairs. Different programming languages support this data structure in different names. Like object, record, struct, dictionary, hash table, keyed list, or associative array.

• An ordered list of values. In various programming languages, it is called as array, vector, list, or sequence.

Syntax:

An object starts with an open curly bracket { and ends with a closed curly bracket } Between them, a number of key value pairs can reside. The key and value are separated by a colon : and if there are more than one KV pair, they are separated by a comma ,

{

  “Students“: [

                                      { “Name“:”Amit Goenka” ,

  “Major“:”Physics” },

                                      { “Name“:”Smita Pallod” ,

  “Major“:”Chemistry” },

                                      { “Name“:”Rajeev Sen” ,

  “Major“:”Mathematics” }

                                      ]

                                      }

 

An Array starts with an open bracket [ and ends with a closed bracket ]. Between them, a number of values can reside. If more than one values reside, they are separated by a comma , .

[

                                      {

  “name“: “Bidhan Chatterjee”,

  “email“: “bidhan@example.com”

                                      },

                                      {

  “name“: “Rameshwar Ghosh”,

  “email“: “datasoftonline@example.com”

                                      }

                                      ]

 

JSON Format Validation:
Now that we’re a bit more familiar with the structure Splunk expects to extract from, let’s work with a sample. The sample data is JSON wrapped in a syslog header. While this data can be ingested as is, you will have to manually extract each field if you choose to not reformat it. You can validate the structure by copying this event to https://jsonformatter.curiousconcept.com/ .

Sample Data:

May 14 13:28:51 <redacted_hostname> github_audit[22200]: { “above_lock_quota”:false, “above_warn_quota”:false, “babeld”:”eebf1bc7″, “babeld_proto”:”http”, “cloning”:false, “cmdline”:”/usr/bin/git upload-pack –strict –timeout=0 –stateless-rpc .”, “committer_date”:”1589477330 -0400″, “features”:” multi_ack_detailed no-done side-band-64k thin-pack include-tag ofs-delta agent=git/1.8.3.1″, “frontend”:”<redacted>”, “frontend_pid”:17688, “frontend_ppid”:6744, “git_dir”:”/data/user/repositories/7/nw/75/42/9d/4564/6435.git”, “gitauth_version”:”dcddc67b”, “hostname”:”<redacted>”, “pgroup”:”22182″, “pid”:22182, “ppid”:22181, “program”:”upload-pack”, “quotas_enabled”:false, “real_ip”:”10.160.194.177″, “remote_addr”:”127.0.0.1″, “remote_port”:”15820″, “repo_config”:”{\”ssh_enabled\”:\”true\”,\”ldap.debug_logging_enabled\”:\”true\”,\”auth.reactivate-suspended\”:\”true\”,\”default_repository_permission\”:\”write\”,\”allow_private_repository_forking\”:\”true\”}”, “repo_id”:6435, “repo_name”:”<redacted>”, “repo_public”:true, “request_id”:”43358116096ea9d54f31596345a0fc38″, “shallow”:false, “status”:”create_pack_file”, “uploaded_bytes”:968 }

 

The errors are noted and highlighted below:

As we can see, the timestamp, hostname and thread field are outside of the JSON object.

 

Replace strings in events with SEDCMD

You can use the SEDCMD method to replace strings or substitute characters. This must be placed on the parsing queue prior to index time. The syntax for a sed replace is:

SEDCMD-<class> = s/<regex>/\1<replacement>/flags

• <class> is the unique stanza name. This is important because these are applied in ABC order
• regexis a Perl language regular expression
• replacementis a string to replace the regular expression match.
• flagscan be either the letter g to replace all matches or a number to replace a specified match.
• \1 – use this flag to insert the string back into the replacement

 

How To Test in Splunk:

Copy the data sample text into a notepad file and upload using Splunk’s built in Add Data feature under Settings to test. Try out each SEDCMD and note the difference in the data structure for each attribute.

BEFORE:

AFTER:

Props.conf – Search Time Field Extraction

[<spec>]

KV_MODE = json

 

Want to learn more about JSON structured data & the SEDCMD in Splunk? Contact us today!

By: Christopher Winarski | Splunk Consultant and

Bruce Johnson | Director, Enterprise Security

 

Migrating your Splunk environment can be a daunting task to some. With the worry of missing valuable data. Did my users’ settings migrate properly? Did all my applications migrate properly? Did all my lookup tables survive the migration? If you find yourself performing a Splunk migration you may be asking yourself some of these questions. Well, today I try to take one of those worries off your chest by walking you through a Splunk KvStore Migration, more specifically migrating the Splunk KvStore from a Search Head Cluster to a new Search Head Cluster. E.g On-prem Shcluster to AWS Shcluster

KvStore stores data in key-value pairs known as Collections. These tables of data are located in your collections.conf files. Records contain each entry of your data, similar to a row in a database table. Using KvStore as opposed to csv files you can define the storage definition schema for your data, perform create-read-update-delete operations on individual records using Splunk REST API and lookups using the Splunk search language. KvStore excels in performance when you start getting large lookups with many data points which is especially prevalent within Enterprise Security, one of Splunk’s Premium Apps.

The normal export/migration is to use csv export which is not really practical for large KvStores due to the limitations to file sizes on most operating system’s is what drive was used for mongodb in the first place. Gemini KvStore Tools helps to circumvent the normal semi-workable, tedious migration process.

Gemini KV Store Tools comes with some custom commands built for the Splunk search bar that makes our life/migration less complicated. The commands we are interested in for this migration are:

• | Kvstorebackup
• | Kvstorerestore

Requirements for this process:

• You must already be utilizing Splunk’s KvStore for your lookups.
• Downloaded and installed “Gemini KV Store Tools” application in both the originating environment Search Head Cluster and the new environment Search Head Cluster you are migrating too. https://splunkbase.splunk.com/app/3536/
• You must have already migrated/copied the applications from the old Search Head Cluster. We are interested in the collections.conf within these applications.
  • tar -zcf apps.tgz /opt/splunk/etc/shcluster/apps
• The collections.conf files must be present on the new environment before proceeding

 

Step 1: Of the original Search Head Cluster, Identify the kvstore captain, and log into the GUI environment, then open the search app. The KvStore captain is the instance in the search head cluster that receives the write operations regarding the KvStore collections where the Search head captain is the instance in the search head cluster that schedules jobs, pushes knowledge bundles to search peers, and replicates any runtime changes to knowledge objects throughout the search head cluster.**note** This may be different than the Search Head captain

 

Step 2: On this instance, also log into the backend and create a directory under the /tmp directory named “kvstore_backup”. Ensure Splunk has read/write permissions to this folder.

cd /tmp

mkdir kvstore_backup

sudo chown -R splunk:splunk /tmp/kvstore_backup

 

Step 3: Creates a json file per each collection to the destination path in the kvstore_backup folder, as well as should see “Success” per each collection zipped within the original environment. In the search bar on original environment KvStore captain, run:

| kvstorebackup path=”/tmp/kvstore_backup” global_scope=”true” compression=”true”

 

Step 4: Check KvStore monitoring console to verify if collection counts are listed and save the page to refer to the results/counts to verify later. (old environment)

Monitoring Console > Search > KvStore:Instance

 

Step 5: Now that you have created your collection backups and have verified that the number of records per is correct. Go on each new search head cluster member (CLI) and edit server.conf to have:

[kvstore]

                                                                        oplogSize = 10000

Also, on each instance, you have to edit/change the search head replication factor to 1 in the new environment on each search head cluster member. (server.conf)

[shclustering]

replication_factor = 1

Once both are set, restart the instance. Do this for every search head cluster member in the new environment.

 

Step 6: Identify and get Search Head captain to be the same instance as the kvstore captain.

Ensure the kvstore captain = Search Head Cluster captain.

Useful commands:

./splunk show shcluster-status

./splunk show kvstore-status

Transfer captaincy to one node by bootstrapping the kvstore captain as the search head captain.

On the KvStore captain, we want to make it also the search head captain, run this command (CLI):

./splunk edit shcluster-config -mode captain -captain_uri <URI>:<management_port> -election false

On each other non-captain instance, run this command (CLI):

./splunk edit shcluster-config -mode member -captain_uri <URI>:<management_port> -election false

This will allow you to specify the captain as the kvstore captain and get rid of dynamic captaincy for this purpose. At the end we will want to revert our search head cluster back to a dynamic captaincy.

 

Step 7: Once you have the kvstore captain = search head captain, Log into the CLI of the other search head nodes (every search head cluster member that is not the captain/kvstore captain). Starting the instance after cleaning the local kvstore will initialize a kvstore synchronization upon startup with kvstore captain.

SHUTDOWN Splunk: ./splunk stop

run: ./splunk clean kvstore –local

Then start splunk: ./splunk start

 

Step 8: SCP kvstore_backup from Step:2 to new environment search head captain/kvstore captain. Make sure that splunk has permissions to access the file. Follow these steps for guidance.

Old instance where the backup was created from:

scp -r kvstore_backup ec2-user@IPADDRESS:/tmp

Move file to /opt folder on kvstore/search head captain:

mv kvstore_backup /opt/kvstore_backup

Change ownership of the file and internal files to splunk for permissions

sudo chown -R splunk:splunk /opt/kvstore_backup

 

Step 9: Kvstore Gemini Tools is to be installed on the new search head cluster prior to running this step, if you have not done so please insure it is installed within the new search head cluster. Once the kvstore_backup has the permissions and is in place on the backend of the kvstore captain/search head captain. Now log on to the GUI of that splunk instance, open search and run:

| kvstorerestore filename=”/opt/kvstore_backup/*.json.gz”

On big restores, can take many minutes for the restore to complete, be patient and let the search run.

 

Step 10: Verify lookups return the same results in the new environment as back in the old environment with the saved page(step 4) , run:

| inputlookup <Lookup definition>

 

Step 11: We want to revert the search head cluster back to a dynamic captaincy now (the static captaincy bootstrapping was just used for the migration) and also change our replication factor back to the original setting in the environment.

You can do this by logging on to each instance CLI, stopping splunk then on the search head cluster captain, run:

./splunk edit shcluster-config -mode captain -captain_uri <URI>:<management_port> -election true

On the other non captain search head cluster members run:

./splunk edit shcluster-config -mode member -captain_uri <URI>:<management_port> -election true

Then we want to edit the config file again to revert replication factor back to the original number that was set before the migration. (server.conf)

[shclustering]

replication_factor = 2

**The “2” is arbitrary here, as this should be set to the number that was present prior to the migration**

That’s it! Migrations can be a scary endeavor and if not prepared, one can easily lose data. If you seek further assistance don’t hesitate to reach out to us here at TekStream Solutions. We would be happy to help! No Splunk project is too small or too big.

By: Marvin Martinez | Senior Developer

 

Creating Alerts via the Spunk REST API is fairly straightforward once you know exactly what parameters to use to ensure that Splunk recognizes the Saved Search as an Alert.  The same applies for ACL permissions on these alerts and other Splunk Knowledge Objects.

First things first, let’s create a scheduled search via REST using the “/services/saved/searches” endpoint.  The curl code below creates a simple search to pull some data from the _internal index for the last 10 minutes.

Note that, to create the saved search, all that was needed was authorization (a token in this case) and a couple of parameters in the call: (1) a name for the search and (2) the search itself.

This will create a search in the Searches, Reports and Alerts screen in Splunk Web.

As you can see, the search has been created and shows up as a Report.  But what if you need this to be an alert?! Even more importantly, what if you want to set this up with specific permissions?  Well, luckily, like essentially everything else in Splunk, this can also be done via the REST API.

To create a new search as an alert, you’ll need to call the same endpoint as shown above with the parameters mentioned below. Otherwise, call the “/services/saved/searches/{name}” endpoint if you’re modifying a search that’s already created.  For the purposes of this write-up, I will call the endpoint to manage an already created search (“/services/saved/searches/{name}”).

In order for Splunk to recognize the search as an alert, and not a Report, the following parameters have to be set correctly and passed along in your POST REST call.  The table below outlines the parameter name and a brief description of what they mean.

Parameter	Description
alert_type	‘number of events’ (if this is set to ‘always’, which is the default, Splunk thinks it’s just a report)
is_scheduled	true (this is a Boolean setting that Splunk checks to make sure there’s a set schedule for the report, which is required for alerts)
cron_schedule	*/10 * * * * (a cron schedule that represents the schedule which the alert will run on)
alert_comparator	‘greater than’ (this is the operator used in the alert settings to determine when to send the alert – associated with the alert_threshold below)
alert_threshold	0 (this is the number to compare with the operator above. i.e. only alert when results > 0)

 

The curl command for the REST call is shown below.  Note the aforementioned parameters that are now being included.

But how does this search now look in the Searches screen?  As can be seen in the image below, once the REST command has been executed successfully, your Alert should now be reflected appropriately as an “Alert”.

For further confirmation of these settings, click the Edit link under Actions, and click Advanced Edit from the drop-down menu.  This will bring up a lengthy listing of all the settings for this search.  If using the REST API is not your style, this is where you can alternately set these settings from Splunk Web.

The listing looks something like this:

All that’s left now is to set your permissions as desired.  To do this, you’ll need to call a new endpoint.  You’ll use the previous endpoint you used to manage a specific saved search, but you’ll add a new section at the end for “acl” (i.e. ‘https://localhost:8089/services/saved/searches/ATestRESTSearch/acl’).  This acl extension/option is available for any endpoint but, in this use case, we’ll use it to manage the permissions for the alert we created above.

In the case of a saved search, you’ll need to include the following parameters in your REST call:

Parameter	Description
sharing	‘app’ – this can also be ‘global’ or ‘user’, depending on what the scope of the access you want this search to have (This is required when updating the ACL properties of any object)
app	‘search’ – this is the name of the app that this search belongs to.  (For saved searches, this is required when updating ACL properties of these objects)
perms.read	A comma-delimited list indicating what roles to assign read permissions to
perms.write	A comma-delimited list indicating what roles to assign write permissions to

 

A curl command that was used in this case is shown below.  In this example, the alert is being updated to give read permissions to admin and user-mmtestuser1.  Additionally, it is being updated to give write permissions to admin and power roles.

As an added bonus, here is an example of how Postman was leveraged to make this final call, in case that’s your REST API-calling tool of your choice.  The Authorization tab, in this example, was set to Basic Auth type with admin credentials.  In the Body tab, you’ll set your parameters to the REST call as “x-www-form-urlencoded” values.  Note the 4 parameters mentioned above shown included in the call below.

Once the REST call is made, navigate to your “Searches, Reports, and Alerts” screen in Splunk Web, and click to Edit Permissions of your alert.  You’ll notice that your permissions are now reflected just the way you designated them in your REST call.

The Splunk REST API is a great alternative, and a necessity for many, to using Splunk Web to create and manage knowledge objects.  Anything that can be done in Splunk Web can be done via the REST API, though it sometimes can be a bit hard to easily understand the process for how to achieve some of these desired actions.  Now, you can easily create alerts and set the permissions just the way you want…and all through REST!

Want to learn more about creating alerts via the Spunk REST API? Contact us today!