By: Joe Wohar | Splunk Consultant

 

There are multiple apps that can be used to integrate Phantom with Splunk, each exists for a different reason. Some of the functionality overlaps. The intent of this post is to provide a guide to knowing which one to leverage based upon what environment you are working in and what use cases are driving your requirement.

 

Application	Install Target	Usage
Splunk App for Phantom	Phantom	Pull event data from Splunk, push event data to Splunk, add Splunk actions to Phantom playbooks.
Phantom App for Splunk	Splunk	Push event data to Phantom
Phantom Remote Search	Splunk	Push Phantom data to Splunk
Splunk App for Phantom Reporting	Splunk	Report on Phantom data
Splunk Add-on for Phantom	Splunk	Used for monitoring Phantom as a service in Splunk ITSI

 

Splunk App for Phantom

The Splunk App for Phantom is a Phantom app used to connect Phantom to Splunk. Phantom apps that are built by Splunk are installed in Phantom by default, so no installation is required, however, you’ll need to configure an asset for it. In the asset settings, you’ll need the IP/hostname of your Splunk instance as well as a Splunk user with sufficient access to the data you wish to search. The Splunk App for Phantom can do the following: post data to Splunk as events, update notable events, run SPL queries, and pull events from Splunk to Phantom. 

• To pull events from Splunk to Phantom, you’ll need to configure the asset settings and ingest settings in your configured asset. It is recommended that you create a new label in Phantom for the events you pull in from Splunk, which will make it easier to find the events in the Analyst Queue in Phantom.
• There are four included actions which can be used in playbooks:
  • get host events – retrieves events about a specific host from Splunk
  • post data – creates an event in your Splunk instance
  • run query – runs an SPL query in Splunk and returns the results of the search to Phantom
  • update event – updates specified notable events within your Splunk Enterprise Security instance

For specific details on using these actions, search for “splunk” on the Apps page in Phantom and click the Documentation link:

 

Phantom App for Splunk https://splunkbase.splunk.com/app/3411/

The Phantom App for Splunk is a Splunkbase app that is installed in Splunk and connects Splunk to Phantom. The main function of this app is to send data from Splunk to Phantom. First, you’ll need to go through the Phantom Server Configuration page to connect Splunk to Phantom, which will require an automation user in Phantom. Then, to send events to Phantom, you’ll need to create a saved search in Splunk where the results of the search are the events you want ingested into Phantom. Open the Phantom App for Splunk and create a New Saved Search Export to start sending events over. There is also an option to create a Data Model Export, which follows the same set of steps used for exporting saved search results to Phantom:

This app also contains alert actions that can be used in Splunk Enterprise Security:

• Send to Phantom – sends the event(s) that triggered the alert to Phantom
• Run Playbook in Phantom – sends the event(s) that triggered the alert to Phantom and runs the specified playbook on them

For more information about the Phantom App for Splunk, review the following documents:

https://docs.splunk.com/Documentation/PhantomApp

https://my.phantom.us/4.6/docs/admin/splunk

 

Phantom Remote Search https://splunkbase.splunk.com/app/4153/

The Phantom Remote Search app is used for multiple reasons. Phantom has an embedded Splunk Enterprise instance built into it, however, you can configure Phantom to use an external Splunk Enterprise instance instead via this app. To do this, you’ll need to install the Phantom Remote Search app onto your Splunk instance, which contains Splunk roles needed for creating two Splunk users required by Phantom. You’ll also need to setup an HTTP Event Collector (HEC) input for receiving Phantom data. After installing the app, creating the necessary users, and creating the HEC input, you can go over to Phantom and change the “Search Settings” in the “Administration Settings”:

Click the following link for a more detailed list of instructions:

https://my.phantom.us/4.6/docs/admin/administration#SearchSettings

This app is also very useful because once you have completed the setup, Phantom will start sending data about itself over to Splunk. This allows you to shift your Phantom reporting out of Phantom and into Splunk. If your Phantom instance is brand new with no events and no active playbooks, configure an asset or create a playbook to test whether or not Phantom data is being sent to Splunk.

 

Splunk App for Phantom Reporting https://splunkbase.splunk.com/app/4399/

If you have already installed the Phantom Remote Search app onto your Splunk instance and configured your Search Settings in Phantom to use an external Splunk instance, you can install the Splunk App for Phantom Reporting onto your Splunk instance to gain insights into Phantom automation and containers:

Splunk Add-on for Phantom https://splunkbase.splunk.com/app/4726/

The Splunk Add-on for Phantom is a Splunk add-on is designed for use with Splunk ITSI to monitor your Phantom instance, although ITSI is not a pre-requisite, it can also be used with Splunk Enterprise but it publishes metrics in a manner that is consistent with ITSI health metrics. It also expects installation of the Phantom Remote Search add-on. The Phantom Remote Search add-on defines indexes and roles used by Phantom when Phantom is configured to use an external Splunk instance for search data. The Phantom Remote Search add-on is required in order to use the Content Pack for Monitoring Phantom as a Service. If you do want to use Splunk ITSI to monitor Phantom, you can follow the documentation for that here:

https://docs.splunk.com/Documentation/ITSICP/current/Config/AboutPhantom

For more information about Phantom, register at https://my.phantom.us/ which will give you access to knowledge articles, documentation, playbooks, and the OVA for Phantom so you can try it out yourself!

Need more help? Contact us today!

By: Eric Howell | Splunk Consultant

 

Splunk provides Admins the opportunity to build a huge variety of customizations, visualizations, apps, and a near-infinitude of different options to finely tune your Splunk experience and have it best suit your needs. There is a use case and a solution for almost any task you can throw at Splunk. A use case that has been brought up frequently is to implement a “Right-Click Integration” to allow information from Splunk to be passed to another tool. Originally, this seemed a daunting task that could require the implementation of custom java script or a unique python script to solve for. Ultimately, the solution was much simpler and the logic is already found in Splunk – Workflow Actions!

Workflow actions allow for Splunk to perform a variety of tasks against available web resources for information found in ingested field/value pairs. They can be very easy to set up either through the UI or by deploying a workflow_actions.conf, as best depends on the size of your Splunk environment, and they offer quite a bit of additional functionality. Utilizing this functionality, your “Right-Click” integration may require an extra click, but it will be fully functional and should be fully serviceable.

Leveraging Workflow Actions to Pass Values and Fields

As discussed above, to set up a workflow action, an admin can leverage the WebUI on one of the Search Heads in an environment or deploy the workflow_actions.conf file as appropriate. This will allow a user to expand an event and pass a value from a field to an external source. In the examples below, we will use VirusTotal as an example.

WebUI Setup

For this example, we will be posting the value of a field to VirusTotal, allowing us to search their IOC database and verify if the value we are passing is a known, bad entity.

1. Navigate to the Settings Dropdown in the Splunk navigator (In the top right by default) and select Fields under Knowledge

2. Next, you will select “Add new” in the “Workflow actions” category

3. This next step encompasses what this workflow action will do. We begin with creating a Label for the action, which will be what users see when they move to access it. Note, you can leverage             tokens in this descriptor to input values dynamically, based on what is clicked. In the case here, it will show the specific value that is being presented to VirusTotal.

4. Then you will indicate, in a comma-separated list, the fields that this workflow action can apply to. Leaving this blank will appear in all field menus.

 

5. If needed, the next step will be to select which eventtypes this workflow_action is applicable for. For additional reading on eventtypes: Follow this Link

a. Eventtypes are categorizations of data input by the user or an admin. An example would be when a specific IP address is found in your web data (sourcetype=access_combined src_ip=127.0.0.1), and you want to apply specific categorization to data that meets that criteria so that it appears with a new field/value pair (e.g. eventtype=local_access)

6. Next you will choose if the action is available in the Event Menu, Fields Menu, or both. This selection is a dropdown. Additional reading available here: Follow this Link

a. Event Menu – Will allow for you to apply this workflow action on the whole event. When an event returned by your search is expanded, the “Event Actions” dropdown will hold any workflow action with “Event Menu” selected here.

b. Fields Menus – Similar to the above, on an expanded event, the workflow action can be accessed by clicking the dropdown under “Action” in the list of included fields to allow one to apply the workflow action directly to the value of a specific field.

7. Next is to choose the Action type – meaning whether the workflow action acts as a “link” to another web location or whether it will run another “search”

8. The next step is configuring the properties based on which of the above Action Types was chosen

a. Link Configuration – Allows you to specify a URI (also accepts tokens), decide whether to open the link in a new window, select whether you intend to utilize a POST or GET method, and which arguments to pass to the web site.

b. Search Configuration – Allows you to specify a search string (token-friendly), dictate which app to run the search in, determine a name of a view (page) for the search to open in, whether to run the search in the same window, and input a timeframe.

9. Next click Save

 

Your workflow action is ready to test. Here is an example of what you can see when looking in an event view with an expanded event.

Deploying a workflow_actions.conf

Like most other configuration files, workflow_actions.conf is easily deployed via the method of your choosing (manually, scripted, or through the use of the Deployer). Below is what will be generated by Splunk in $SPLUNK_HOME/etc/system/local when created through the web UI:

 

[VirusTotalPost]

display_location = both

label = Search VirusTotal for $@field_value$

link.method = post

link.uri = https://www.virustotal.com/en/search/

link.target = blank

link.postargs.1.key = query

link.postargs.1.value = $@field_value$

type = link

 

This same information is deployable in an app of your choosing to $SPLUNK_HOME/etc/apps/ as appropriate for environments with clustered Search Heads or through other means of configuration management.

There are a wide variety of possible options with the stanza, and I recommend reviewing Splunk’s documentation regarding the specific .conf file here:

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Workflow_actionsconf

Leveraging your workflow_actions with older versions of Splunk Enterprise Security

If you are utilizing a version of the Splunk Enterprise Security app prior to 5.3, even if your workflow actions have global permissions, the app will not import configurations from others unless they follow a strict naming convention. If you want your workflow action to be accessible from the Incident Review dashboard, for example, you will need to make sure your app follows the one of these naming conventions:

• DA-ESS-*
• SA-*
• TA-*
• Splunk_SA_*
• Splunk_TA_*
• Splunk_DA-ESS_*

 

Want to learn more about right-click integrations in Splunk? Contact us today!

 

By: Yetunde Awojoodu| Splunk Consultant

 

It may be interesting to learn that not all searches need to be optimized. Your search should be optimized if it will be administered often or queries large amounts of data. With Splunk Processing Language (SPL), you can obtain the same set of results using a different array of commands, however, when constructing your searches, it is important to consider the impact to memory and network resources in your environment.

How to Determine Whether Your Search is Unoptimized

• It runs for a long period of time
• It retrieves larger amounts of data than needed from the indexes

(View the amount of data retrieved from the indexer in the Job Inspector)

• You tend to hit the disk search quota while running the search
• It results in a slow or sluggish system

Creating Optimized Searches

To optimize the speed at which your search runs, it is important that you minimize the amount of processing time required by each component of the search. Your search can be slow because of the complexity of your query to retrieve events from the index. Below are some useful guidelines:

1. Choose an Appropriate Time Frame

Set the time picker to search within the exact time window your results should be found. This will limit the number of buckets that will need to be searched in the specified index. For instance, if you specify a search for the past 24 hours, only buckets in the specified index with data for the last 24 hours will be searched. “All time” searches are discouraged and “Real-time” searches should almost never be used due to resource consumption.

2. Use an Efficient Search Mode

Splunk has three search modes which are Fast, Smart and Verbose. Change your search mode depending on what you need to see. Select verbose mode sparingly, using only when needed. Since it returns all of the fields and event data it possibly can, it takes the longest time to run. See more on Splunk search modes here.

3. Retrieve only what is needed

How you construct your search has a significant impact on the number of events retrieved from disk. Be restrictive and specific when retrieving events from the index. If you need only a portion of the whole data, limit your search early to extract the portion you need before any data manipulation or calculations. You can specify an index, source type, host, source, specific word or phrases in the events. Include as many search terms as possible in your base search. Also, use the head command to limit events retrieved when you need just a subset of the data and remove unnecessary fields from the search results by using commands such as fields and where.

Example:

index=audit sourcetype=access_combined host=admin (action=failed OR action=cancelled) | stats count by user

4. Use Efficient SPL Commands

a) Choice of Commands

As mentioned above, you can arrive at the same results using a different combination of commands but your choice will determine the efficiency of your search. Below are a few tips:

• • • Joins and Lookups – Avoid using multiple joins and lookups in your search. They are very resource-intensive. Perform joins and lookups only on the required data and consider using append over join.
    • Eval – Perform evaluations on the minimum number of events possible
    • Stats – Where possible, use stats command over the table command
    • Table – Use table command only at the end of your search since it cannot run until all results are returned to the search head.
    • Dedup – The stats command is more efficient than dedup. Consider listing the fields you want to dedup in the “by” clause of the stats command. For instance,

…| stats count by user action source dest

or

…| stats latest by user action source dest

Rather than

…| dedup user action source dest

b) Order of Commands

The order in which commands are specified in a search is extremely important since this determines where the commands are executed which could be at the search head or on the indexers. When part or all of a search is run on the indexers, the search processes in parallel and search performance is much faster. It is good to parallelize as much work as possible. The aim is to not overburden the search head by making the indexer(s) do some of the work.

If your commands can be arranged so that they execute on the indexer, the overall search will execute quicker. Move commands that bring data to the search head as late as possible in your search criteria

Streaming and Non-Streaming Commands

Understanding streaming and non-streaming commands is important in discussing the order of commands in a search. Non-streaming commands include transforming commands such as stats, timechart, top, rare, dedup, sort and append which operate on the entire result set of event data and are always executed on the search head regardless of order. To optimize your searches, place non-streaming commands as late as possible in your search string.

There are two types of streaming commands – Distributable and Centralized Streaming Commands. Distributable Streaming Commands such as eval, fields, rename, replace and regex operate on each event returned by a search regardless of the event order. They can be executed on the indexer. However, if any of the preceding search commands is executed on the search head, the distributable command will be executed on the search head as well. When possible, allow distributable streaming commands to precede non-streaming commands.

Similar to the distributable streaming commands, centralized streaming commands operate on each event returned by a search but event order is important and commands execute only on the search head.

To inspect your search, take a look at the Job Inspector and Search Job Properties. There are two search pieces: remoteSearch and reportSearch that show where parts of your search string are executed. RemoteSearch is the part of the search string executed on the remote nodes (indexers) while the reportSearch is the part executed on the search head.

Let’s look at a simple example to illustrate this:

index=security user=admin failed

| timechart count span=1h

| stats avg(count) as average

In this example, the base search retrieves events from the index and the search head executes the timechart and stats commands which are both transforming commands and outputs the results.

A second example:

index=network sourcetype=cisco_wsa_squid usage=” violation”

| stats count AS connections by username usage

| rename username as violator

| search connections >=10

In the above example, stats command is a transforming command so it will be executed on the search head. “Rename” is a distributable streaming command which could execute on the indexer but because it occurs after a transforming command, it will also execute on the search head.

For better performance, reorder the commands as follows so that “rename” precedes “stats” and is therefore executed on the indexer.

index=network sourcetype=cisco_wsa_squid usage=violation

| rename username as violator

| stats count AS connections by username usage

| search connections >=10

5. Check the Job Inspector

Finally, check the job inspector tool to examine the overall stats of your search including where Splunk spent its time. Use the tool to troubleshoot search performance and understand the impact of knowledge objects (lookups, tags) on processing.

Reference Links

https://docs.splunk.com/Documentation/Splunk/latest/Search/Writebettersearches

https://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutoptimization

https://docs.splunk.com/Documentation/Splunk/latest/Search/Quicktipsforoptimization

https://docs.splunk.com/Documentation/Splunk/latest/Search/Changethesearchmode

 

Want to learn more about optimizing your Splunk searches? Contact us today!