Archive for category: Blog

By: Yetunde Awojoodu | Splunk Consultant

 

Have you ever come across fields with multiple values in your event data in Splunk and wondered how to modify them to get the results you need? Each field in an event typically has a single value, but for events such as email logs you can often find multiple values in the “To” and “Cc” fields. Multivalue fields can also result from data augmentation using lookups. To properly evaluate and modify multivalue fields, Splunk has some multivalue search commands and functions. If you ignore multivalue fields in your data, you may end up with missing and inaccurate data, sometimes reporting only the first value of the multivalue field(s) in your results.

In this article, I have applied a simple scenario to illustrate how different multivalue commands and functions can be used individually or combined to meet different use cases. I will cover some common search commands and functions that work with multivalue fields. Note that multivalue functions can be used with eval, where or fieldformat search commands. In my illustrations, I employed the “makeresults” command to generate hypothetical data for my searches so that anyone can recreate them without the need to onboard data. Read more on the makeresults command.

 

Scenario

Within one purchase transaction, Mary bought eggs, milk and bread. She paid for the eggs with cash and covered the remaining items using her credit card. We can assume that this purchase transaction is equivalent to a log event. The values for each multivalue field are separated by the comma delimiter.

Example 1:

Please note that in all the results, I have deliberately excluded the default field, “_time” which is a default field generated when the makeresults command is used.

 

Makemv (Command)

This command is used to split the values of a field that appear like a single value into multiple values within an event based on the delimiter. A delimiter specifies the boundary between characters.

Example 2:

The values in the “groceries” field have been split within the same event based on the comma delimiter. The values in the “payment” field remain the same. The report shows the method of payment for all three grocery items but it does not specify the actual payment method used for each item. To expand the event into three separate events, one for each item and show the exact payment for each grocery item, we will need a combination of commands and functions.

 

Mvzip (Function)

The mvzip function is used to tie corresponding values in the different fields of an event together. This helps to keep the association among the field values. This function takes two multivalue fields, X and Y, and combines them by stitching together the first value of X with the first value of field Y, then the second with the second, and so on.

Example 3:

The new field, “zipped” is the result of the mvzip function. The values of the groceries and payment fields are properly zipped together before expanding into separate events. Note that at this point, the results are still within one event.

 

Mvexpand (Command)

This command expands the values of a multivalue field into separate events, one event for each value in the multivalue field. All other single field values and unexpanded multivalue field values will remain the same in each new event.

Example 4:

Mvexpand works great at splitting the values of a multivalue field into multiple events while keeping other field values in the event as is but it only works on one multivalue field at a time. For instance, in the above example, mvexpand cannot be used to split both “zipped” and “payment” fields at the same time. The next function will come in handy to accomplish this.

 

Mvindex (Function)

Having zipped the values and created one field, “zipped”, you can now expand the “zipped” field into multiple events. The mvindex function is a little more intricate. To further tie field values together so that accurate associations are made in the process of expanding the values into separate events, mvindex will separate the existing multivalued field into two chosen fields using index values. Indexes can start at zero if labeling from the first value. For example, if values= a,e,i,o,u; a=0 e=1 i=2 o=3 u=4. You could also label from the last character with -1; a=-5 e=-4 i=-3 o=-2 u=-1 or you could choose to have a combination of both index patterns; a=0 e=1 i=2 o=-2 u=-1.

Example 5:

Mvindex was used to assign index 0 to the first value in the group which represents groceries and index 1 to the second value representing payment method so that when the fields are split, the values will not get mixed up. The “split” command was used to separate the values on the comma delimiter. Using mvindex and split functions, the values are now separated into one value per event and the values correspond correctly.

Tip – The stats command can also be used in place of mvexpand to split the fields into separate events as shown below:

Example 6:

 

Mvcount (Function)

This function can be used to quickly determine the number of values in a multivalue field using the delimiter. If the field contains a single value, the function returns 1 and if the field has no values, the function returns NULL.

Example 7:

As with single value fields, keep in mind that you may need a combination of multivalue commands/functions to get your report in the required format that will meet your specific use case.

Note: If there are situations in your data where a field is sometimes multivalue and other times null, refer here

 

Want to learn more about working with multivalue fields in Splunk? Contact us today!

 

By: Joe Wohar | Senior Splunk Consultant

 

Splunk Phantom is an amazing software used to automate cybersecurity processes, however, many companies do not know that they could also be using Phantom for case management. Arguably the most powerful, yet unknown to many, case management feature of Phantom is the ability to create and use workbooks.

If you’re familiar with Phantom, then you know that Phantom playbooks are repeatable processes that Phantom runs through against events. Phantom workbooks are repeatable, defined processes that analysts run through against events. However, they’re typically only used when an analyst needs to get involved. When an event is determined to be a threat that needs to be investigated by an analyst, the event should be promoted to a case. This can be done with a manual change done on the event (by clicking the toolbox icon button) or by having the conditions specified in a playbook that can then turn the event into a case.

Image 1: A workbook must be selected when converting an event to a case.

 

One of the biggest advantages of workbooks is that it’s a great way of ensuring that your analysts (new or old) are following the same set of steps when working cases. SOPs help define processes for your analysts to follow, but workbooks put those processes right into the case and make the work easily trackable. Workbooks are made up of 2 trackable components: phases and tasks.

 

Phases

Phases split the investigation into different sections, such as identification, acquisition, analysis, and reporting. Individual SLAs can be set for each phase of a workbook. When SLAs are missed/breached, there is a panel on the Phantom home page for tracking that:

Image 2: Home page SLA breach tracker.

 

Phases are made up of tasks, which are where the specific steps for investigations are listed.

Image 3: Adding new phases/tasks to a workbook.

 

Tasks

Tasks are very customizable, so they can be pretty general with few trackable requirements or be very specific with many tracked steps. First, tasks can have a default owner assigned to them, which could be useful if you want to have a “review” task so that a more experienced analyst can review a newer analyst’s work, however, I think most often you’d want to leave that blank so that tasks can be assigned to the analyst working the case. The description section of the task is where you can describe the specific things that should be done in the task. If you don’t want to track specific steps, you can simply use this section to create a list of steps for analysts to follow. However, if you have very specific steps involved in a task, you may want to use the description just for describing the process and then have the steps listed as actions or playbooks. This brings us to the next part of tasks, adding actions and playbooks.

Actions and playbooks are Phantom automation being added to the human process. The actions and playbooks added to a task are limited to the actions available in your configured apps and the playbooks that you have available in your Phantom instance. Then, when an analyst goes to run the action from the investigation screen, the action is already pulled up and they just need to enter the details.

Image 4: Workbook opened in a case with 2 tasks.

 

Image 5: Pop-up window from clicking the “run query” action in the workbook.

 

Running a playbook from a workbook is even simpler. Just click the playbook and click the “Run Playbook” button.

Image 6: Pop-up window from clicking the “Disabled User” playbook in the workbook.

 

As analysts move through the tasks and complete them, the phase’s tracker will be updated to show completion and whether or not tasks were completed on time and if the phase was completed on time.

Images 7 & 8: First task complete and then both tasks completed.

 

If you’re not using Phantom for case management, then you’re likely using Phantom to create tickets and add details to them in another software, which is costing you more in hardware and licensing. By using Phantom for case management, you’ll save the cost of another software and its hardware while using software you’ve already bought at no additional cost.

Not sure how to get started with workbooks? Try taking one of your best defined SOPs and make a workbook for it. If you’re not currently a Phantom customer and would like to try it out, you can download the OVA by registering here: https://my.phantom.us/

 

Want to learn more about Phantom workbooks? Contact us today!

 

By: Aaron Dobrzeniecki | Splunk Consultant

 

If you have problems or questions regarding masking important data when it gets ingested into Splunk, this is the blog for you. Common use cases include masking credit card numbers, SSN, passwords, account IDs, or anything that should not be visible to the public. When masking data before it gets indexed into Splunk, you want to make sure you (if applicable) test it in a dev environment. A great website to use is www.regex101.com.

The overall methodology of how the two approaches work specifically relies on the correctness of your regular expression. Splunk will look for strings that match the defined regex pattern. You can then tell Splunk to strip out, replace the matching string, or replace part of the string. Both of the methods below do the same exact thing – match a regex and replace the values – but both methods do it in a slightly different manner.

In the example data below, I will be masking the account IDs to only show the last four digits of the account ID. There are two ways you can mask data before it gets ingested into Splunk.

Method 1:

Using props.conf and transforms.conf to modify the data so that the first 12 characters of the account ID turn into “x”‘s.

One sample event:

[02/Nov/2019:16:05:20] VendorID=9999 Code=D AcctID=9999999999999999

When ingested into Splunk using the below props.conf and transforms.conf the event will be indexed as so:

[02/Nov/2019:16:05:20] VendorID=9999 Code=D AcctID=xxxxxxxxxxxx9999

props.conf

[mysourcetype]

TRANSFORMS-data_mask=data_masking

 

transforms.conf

[data_masking]

SOURCE_KEY=_raw

REGEX=(^.*)(\sAcctID=)\d{12}(\d*)

FORMAT=$1$2xxxxxxxxxxxx$3

DEST_KEY=_raw

Specify the field you want Splunk to search for the matching data in using the SOURCE_KEY parameter. Splunk will attempt to match the regex specified in the REGEX setting. If it matches, Splunk will replace the matching portion with the value from FORMAT and then write the transformed value to the field specified in DEST_KEY (which is the same in this example). The values for FORMAT are as followed. The dollar sign digit relates to the capture groups. In the example above you can see that there are 3 total capture groups: (^.*) is the first capture group; (\sAcctID=) is the second capture group; and finally (\d*) is the third capture group (I included a third capture group to specify extra digits, if they exist in the event or not). See how we did not include the \d{12}? This is because THAT regex string is what we want to mask.

The basis behind masking your important data is to make sure that you have created the correct regex. In the example above I created the entire regex string that encompasses an entire event. In doing so, we are able to bring back the entire event using the capture groups and ridding the event of the data to be masked.

Another way to mask important data from being ingested into Splunk is to use the SEDCMD to replace the desired texts with X’s or whatever you want to show that the data has been masked. Using the same sample event above we will get the same results as above, but using a different method.

Method 2:

props.conf

[mysourcetype]

SEDCMD-replace=s/AcctID\=\d{12}/AcctID=xxxxxxxxxxxx/g

The above props.conf will mask the data as desired. The key here is to make sure that your regex string (the one that is replacing the original regex string) includes the part that you want to keep and does not include the string that you want to get rid of. With SEDCMD, Splunk replaces the current regex with the regex you specify in the third segment of the SEDCMD.

In conclusion, there are two ways to anonymize data with Splunk Enterprise:

Use the SEDCMD like a sed script to do replacements and substitutions. The sed script method is easier to do, takes less time to configure, and is slightly faster than a transform. But there are limits to how many times you can invoke SEDCMD and what it can do.

Use a regular expression transform (method 1). This method takes longer to configure, but is easier to modify after the initial configuration and can be assigned to multiple data inputs more easily.

Want to learn more about masking important data in your Splunk environment? Contact us today!