Archive for category: Splunk

  By: Bruce Johnson  | Director, Enterprise Security

 

Many organizations use Splunk today. Of those adopters, most have a distributed Splunk environment. Often, organizations have sensitive data traversing their network, which makes its way into Splunk. More now than ever, security is at the forefront of everyone’s mind, and securing your Splunk environment is no exception. How to properly secure a distributed Splunk environment is not a new concept, but it is still frequently underutilized or improperly implemented. With that being said, an overview of how to implement SSL between your Splunk Deployment Server and Splunk Web instances will be discussed in detail. Manually configuring the Search Head and Search Peers will be overviewed.

As Splunk ships with OpenSSL, this method will be discussed in examples. Ensure you are using the proper version of OpenSSL on each Splunk instance. The steps provided assume you are configuring on a Linux-based host, and Splunk is installed in the /opt/splunk OR /opt/splunkforwarder directory, and using Splunk default ports.

Deployment Server and Splunk Web

You will want to secure your traffic from your web browser to your Deployment Server, as non-ssl traffic transfers raw data. Cleartext makes it easy for those that know how to intercept traffic to read your data. Use SSL certificates to help secure your data by turning your cleartext into ciphertext, especially when you need to access instances outside of your network. There are, of course, default certificates that ship with Splunk. It is a best practice to go with either a self-signed or a purchased CA-signed certificate instead. How to utilize self-signed certificates will be discussed.

First, make a directory for your self-signed certificates, to ensure you don’t interfere with the default Splunk certificates. Then, traverse to that folder. While in that folder, create your private key (PK) that you will utilize to sign your certificates. Ensure access to this folder is limited to personnel that needs access, as private keys should never be shared. Encrypted data can be decrypted by anyone that has the private key.

Next, you will need to generate your custom Certificate Signing Request (CSR). Use your CSR to create your public certificate (.pem), which is what you will distribute to your various Splunk instances. With the root certificate created to act as a CA, you will then utilize the CSR, CA certificate, and private key to generate and sign a server certificate that is valid for three years. Use the server certificate by distributing it to your indexers, forwarders, and other Splunk instances, which communicate over management port 8089. We will only discuss, however, implementing it on your Deployment Server.

• 1. mkdir /opt/splunk/etc/auth/custcerts
• 2. cd /opt/splunk/etc/auth/custcerts
• 3. /opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out mattCAPK.key 2048
  • a. Enter a secure password, then again to confirm.
• 4. /opt/splunk/bin/splunk cmd openssl rsa -in mattCAPK.key -out mattCAPKNoPW.key
  • a. Removing the password makes it easier for testing.
  • b. You’ll need to enter the secure password you created in step 3 above.
• 5. /opt/splunk/bin/splunk cmd openssl req -new -key mattCAPKNoPW.key -out mattCACert.csr
  • a. Enter details to questions asked.
• 6. /opt/splunk/bin/splunk cmd openssl x509 -req -in mattCACert.csr -sha512 -signkey mattCAPKNoPW.key -CAcreateserial -out mattCACert.pem -days 1095

Now you will need to generate your server certificate:

• 7. /opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out mattServerPK.key 2048
• 8. /opt/splunk/bin/splunk cmd openssl rsa -in mattServerPK.key -out mattServerNoPW
  • a. Again, removing the password makes testing easier.
• 9. Use your new server private key mattServerPK.key to generate a CSR for your server certificate. (use sdsp01.nielsen.com for the common name in the CSR).

Similar to steps 1-6, you will use the private key to create the CSR, then both to create the server certificate.

• 10. /opt/splunk/bin/splunk cmd openssl req -new -key mattServerNoPW.key -out mattServerCert.csr
• 11. /opt/splunk/bin/splunk cmd openssl x509 -req -in mattServerCert.csr -SHA256 -CA mattCACert.pem -CAkey mattCAPKNoPW.key -CAcreateserial -out mattServerCert.pem -days 1095

You’ll now want to concatenate them all together (you will do this two different times in these steps). The format and reasoning are explained here:

• 12. cat mattServerCert.pem mattServerNoPW.key mattCACert.pem > mattNewServerCert.pem

At this point, you will need to update the server.conf file on your Deployment Server. This file is located in the /opt/splunk/etc/system/local/ directory. You can get more granular in the stanzas if you prefer, and the options are listed in Splunk docs.

• 13. Find the [sslConfig] stanza.
• 14. [sslConfig]
• 15. enableSplunkdSSL = true
• 16. serverCert = /opt/splunk/etc/auth/custcerts/mattNewServerCert.pem
• 17. caCertFile = /opt/splunk/etc/auth/custcerts/mattCACert

Here you will need to restart Splunk on your Deployment Server instance.

• 18. /opt/splunk/bin/splunk restart\

You will need to generate a key specifically for the web UI for the Deployment Server. Please note that you must remove the password for the Splunk Web portion, as it’s not compatible with a password.

• 19. /opt/splunk/bin/splunk cmd openssl genrsa -des3 -out mattWebPK.key 2048
• 20. /opt/splunk/bin/splunk cmd openssl rsa -in mattWebPK.key -out mattWebPKNoPW.key
• 21. /opt/splunk/bin/splunk cmd openssl req -new -key mattWebPKNoPW.key -out mattWebCert.csr
• 22. /opt/splunk/bin/splunk cmd openssl x509 -req -in mattWebCert.csr -SHA256 -CA mattCACert.pem -CAkey mattCAPKNoPW.key -CAcreateserial -out mattWebCert.pem-days 1095
• 23. cat mattWebCert.pemmattCACert.pem > mattWebCertificate.pe

(You should be noticing a trend by now!)

You will now need to update the web.conf [settings] stanza, which is located in the /opt/splunk/etc/system/local/ directory path.

• 24. [settings]
• 25. enableSplunkWebSSL = true
• 26. privKeyPath = /opt/splunk/etc/auth/custcerts/mattWebPKNoPW.key
• 27. serverCert = /opt/splunk/etc/auth/custcerts/mattWebCertificate.pem

** For reasons of Splunk magic, the Deployment Server has issues pushing certs to Deployment Peers, so configure them individually/manually. An app would be much simpler method, though.

Once implemented, test within your browser. I have had issues with Google Chrome (see Image 1 below), but Firefox allows the page to be load as desired (see Image 2 below for reference). You will find that https only works at this point, and http no longer will.

SHs & Search Peers/Indexers

When adding search peers (indexers) to a search head, many admins will simply use the Splunk user interface (UI), or the command-line interface (CLI). In many situations, these are efficient and complete methods. There are, however, use cases that require adding search peers via editing distsearch.conf directly. This manner provides more granular and advanced features to be implemented. When editing distsearch.conf directly, key files need to be distributed manually to each search peer. This is in contrast to the two other methods, which implement authentication automatically.

Upon adding your search peers to your search head(s) via editing distsearch.conf, the key files need to be copied to the proper path. On your search head(s), copy the public key file, and place it in your search peer(s)’ file structure (file location(s) examples follow this paragraph). If adding search peer(s) to multiple search heads, then each search head’s public key file needs to be in its own folder named after the search head (utilize the actual serverName that is listed in server.conf for the folder name). Once the files have been properly copied over, simply restart Splunk on each Splunk search peer instance. The file location examples are as follows:

On your search head:

• $SPLUNK_HOME/etc/auth/distServerKeys/sh1PublicKey.pem

On your search peers/indexers:

• $SPLUNK_HOME/etc/auth/distServerKeys/searchHead1Name/sh1PublicKey.pem
• $SPLUNK_HOME/etc/auth/distServerKeys/searchHead2Name/sh2PublicKey.pem

Each instance of your Splunk deployment can, and should, be configured to use SSL. Each instance has its own caveats and nuances that need special attention to detail to configure properly. You should also look into securing your traffic between your forwarders and indexers.

Contact us for more help on configuring SSL for your distributed Splunk environment!

  By: William Phelps  |  Senior Technical Architect

 

This blog covers the basic steps for configuring the SignalFx agent and configuring a Python application running in Gunicorn to send trace data to SignalFx via the agent if Gunicorn is being executed within a Docker container.

Let’s start with a high-level overview of the technologies involved in the solution:

• – The SignalFx Tracing Library for Python automatically instruments Python 2.7 or 3.4+ applications to capture and report distributed traces to SignalFx within a single function.
• – The library accomplishes this by configuring an OpenTracing-compatible tracer to capture and export trace spans.
• – The tracer can also be used to embed additional custom instrumentation into the automatically generated traces. This blog will concentrate solely on the auto-instrumentation approach.
• – The SignalFx-Tracing Library for Python then works by detecting libraries and frameworks referenced by the application and then configuring available instrumentors for distributed tracing via the Python OpenTracing API 2.0. By default, the tracer footprint is small and doesn’t declare any instrumentors as dependencies.
• – Gunicorn, or ‘Green Unicorn,’ is a Python web server gateway interface (WSGI) HTTP Server for UNIX/Linux. The Gunicorn server is a pre-fork worker model and is broadly compatible with various web frameworks, simply implemented, light on server resources, and is fairly speedy.

The following notes are provided as general prerequisites or assumptions:

• – The SignalFx agent is already installed and initially configured on the Docker host. (The Otel collector is not yet compatible with this configuration.)
• – Alternately, the SignalFx agent can be deployed as its own Docker container, but this article assumes a local installation on the Docker host.
• – The SignalFx agent is already sending traces to the proper SignalFx realm and account.
• – The Docker host is assumed to be Linux. (RHEL/Centos/Oracle/AWS). The steps would be similar for Ubuntu/Debian, but the commands shown will be RHEL-centric.
• – Python 3 is installed on the host.
• – Ensure that proper access to Terminal or a similar command-line interface application is available.
• – Ensure that the installing Linux username has permission to run “curl” and “sudo.”

Process Steps

The general overall flow for this process is relatively short:

1. Configure the SignalFX agent to monitor Docker containers.
2. Create a Gunicorn configuration to support the tracing wrapper.
3. Create a Dockerfile to deploy the application.

Docker Monitoring Configuration

If the SignalFx agent was installed previously, navigate to the folder where the agent.yaml resides.

Edit the agent.yaml file to enable Docker monitoring. Under the “Observers” section, add a type of “Docker” as shown, and under “Monitors,” add a type of “docker-container-stats.”

Also, under the “Monitors” section, ensure that for the type “signalfx-forwarder,” the attribute “listenAddress” is set to 0.0.0.0:9080, and not “localhost” or “127.0.0.1”.

Additionally, under the type “signalfx-forwarder,” uncomment the attribute “defaultSpanTags.”

Uncomment and set a meaningful value for “environment” as shown. This value will be sent with every trace from this host/SignalFX agent invocation and is used as a filtering device. “Environment” is a child attribute of “defaultSpanTags.” Be aware of the appropriate indentation, as YAML is very strict.

Save the file. At this point, SignalFx will be able to send Docker metrics data, but it likely will not be sending anything. A quick look at the logs via journalctl:

…will probably show a permissions issue with reading the Docker socket. Add the user “signalfx-agent” to the “Docker” group and restart the SignalFx service to address this issue.

 

Gunicorn Configuration

Use the following steps to configure Gunicorn for auto-instrumentation of Python for SignalFX. Again, the assumption for this blog is that Gunicorn is being deployed to a Docker container.

1. In Python’s application root directory, create a file called “gunicorn.config.py”.
2. The contents of this file should appear as follows (or be modified to include the following):
   

Local Docker Configuration

At a high level, the Dockerfile simply consists of the directives to create the necessary environment to add Gunicorn and the Python application to the container. These directives include:

• – Creating the expected directory structure for the application.
• – Creating a non-root user to execute Gunicorn and the application inside the container. (Running as root is not recommended and will likely cause the container build to fail.)
• – Setting environment variables to pass to the container context.
• – Loading the Splunk OpenTelemetry dependencies.
• – Launching Gunicorn.

Please note the “ENV” directive for “SPLUNK_SERVICE_NAME” in step 7. The service name is the only configuration option that typically needs to be specified. Adjust this value to indicate the Python service being traced.

Other options can be found in the GitHub documentation under “All configuration options.”

This Dockerfile is using the Python3.8 parent image as the FROM target. Accordingly, the “pip” instruction in step 8 may need to be altered based on the parent image. The requirements file argument, however, should still be valid.

The requirements file appears as follows. This file lists out Gunicorn and specific Paste libraries, along with basic setup items and a version of Flask. The actual requirements for a project may vary. “Splunk-opentelemetry” in turn will load packages it requires. As such, this requirements file is not to be considered the complete library source.

Setting the PATH variable typically is NOT needed as shown in step 10. However, this ensures that the correct environment is present prior to running the bootstrap. The PATH must include the user’s “.local/bin” and “.local” folders from the home directory.

Finally, in step 12, note the use of both “- -paste” and “-c”. “ – -paste” of an .ini file allows additional configuration to be added to the build. “-c” is required to get the SignalFx configuration loaded that was defined in “gunicorn.config.py” earlier. This initialization line is shown to illustrate that both parameters can be used simultaneously. “-c” should follow “paste” if both are used.

Running the Dockerfile will generate a lot of output, but the final lines should look something like this:

Checking in the SignalFx UI should show the new service defined in step 7 of the Dockerfile.

Contact us for more help on configuring SignalFx for Docker & Gunicorn!

      By: Zubair Rauf  |  Splunk Consultant, Team Lead

 

A few days ago, I came across a very rare use case in which a user had to reindex a specific subset of raw Splunk events into another index in their data. This was historical data and could not be easily routed to a new index at index-time.

After much deliberation on how to move this data over, we settled on the summary index method, using the collect command. This would enable us to search for the specific event we want and reindex them in a separate index.

When re-indexing raw events using the “collect” command, Splunk automatically assigns the source as search_id (for ad-hoc search) and saved search name (for scheduled searches), sourcetype as stash, and host as the Splunk server that runs the search. To change these, you can specify these values as parameters in the collect command. The method we were going to follow was simple – build the search to return the events we cared about, use the collect command with the “sourcetype,” “source,” and “host” values to get the original values of source, sourcetype, and host to show up in the new event.

To our dismay, this was not what happened, as anything added to those fields is treated as a literal string and doesn’t take the dynamic values of the field being referenced. For example, host = host would literally change the host value to “host” instead of the host field in the original event. We also discovered that when summarizing raw events, Splunk will not add orig_source, orig_sourcetype, and orig_index to the summarized/re-indexed events.

To solve this problem, I had to get creative, as there was no easy and direct way to do that. I chose props and transforms to solve my problem. This method is by no means perfect and only works with two types of events:

• – Single-line plaintext events
• – Single-line JSON events

Note: This method is custom and only works if all steps are followed properly. The transforms extract field values based on regex, therefore everything has to be set up with care to make sure the regex works as designed. This test was carried out on a single-instance Splunk server, but if you are doing it in a distributed environment, I will list the steps below on where to install props and transforms.

The method we employed was simple:

1. Make sure the target index is created on the indexers.
2. Create three new fields in search for source, sourcetype, and host.
3. Append the new fields to the end of the raw event (for JSON, we had to make sure they were part of the JSON blob).
4. Create Transforms to extract the values of those fields.
5. Use props to apply those transforms on the source.

I will outline the process I used for both types of events using the _internal and _introspection index in which you can find both single-line plain text events and single-line JSON events.

Adding the Props and Transforms

The best way to add the props and transforms is to package them up in a separate app and push them out to the following servers:

1. Search Head running the search to collect/re-index the data in a new index (If using a cluster, use the deployer to push out the configurations).
2. Indexers that will be ingesting the new data (If using a clustered environment, use the Indexer cluster Manager Node, previously Master Server/Cluster Master to push out the configuration).

Splunk TA

I created the following app and pushed it out to my Splunk environment:

TA-collect-raw-events/
└── local
├── props.conf
└── transforms.conf


The files included the following settings:

Transforms.conf

[setDynamicSource]
FORMAT = source::$1
REGEX = ^.*myDynamicSource=\"([^\"]+)\"
DEST_KEY= MetaData:Source

[setDynamicSourcetype]
FORMAT = sourcetype::$1
REGEX = ^.*myDynamicSourcetype=\"([^\"]+)\"
DEST_KEY= MetaData:Sourcetype

[setDynamicHost]
FORMAT = host::$1
REGEX = ^.*myDynamicHost=\"([^\"]+)\"
DEST_KEY= MetaData:Host

[setDynamicSourceJSON]
FORMAT = source::$1
REGEX = ^.*\"myDynamicSourceJSON\":\"([^\"]+)\"
DEST_KEY= MetaData:Source

[setDynamicSourcetypeJSON]
FORMAT = sourcetype::$1
REGEX = ^.*\"myDynamicSourcetypeJSON\":\"([^\"]+)\"
DEST_KEY= MetaData:Sourcetype

[setDynamicHostJSON]
FORMAT = host::$1
REGEX = ^.*\"myDynamicHostJSON\":\"([^\"]+)\"
DEST_KEY= MetaData:Host

Props.conf

[source::myDynamicSource]
TRANSFORMS-set_source = setDynamicSource, setDynamicSourcetype, setDynamicHost

[source::myDynamicSourceJSON]
TRANSFORMS-set_source = setDynamicSourceJSON, setDynamicSourcetypeJSON, setDynamicHostJSON

Once the TA is successfully deployed on the Indexers and Search Heads, you can use the following searches to test this solution.

Single-line Plaintext Events

The easiest way to test this is by using the _internal, splunkd.log data as it is always generating when your Splunk instance is running. I used the following search to take ten sample events and re-index them using the metadata of the original event.

index=_internal earliest=-10m@m latest=now
| head 10
| eval myDynamicSource= source
| eval myDynamicSourcetype= sourcetype
| eval myDynamicHost= host
| eval _raw = _raw." myDynamicSource=\"".myDynamicSource."\" myDynamicSourcetype=\"".myDynamicSourcetype."\" myDynamicHost=\"".myDynamicHost."\""
| collect testmode=true index=collect_test source="myDynamicSource" sourcetype="myDynamicSourcetype" host="myDynamicHost"

Note: Change testmode=false when you want to index the new data as testmode=true is to test your search to see if it works.

The search does append some metadata fields (created with eval) in the original search to the newly indexed event. This method will use license to index this data again as the sourcetype is not stash.

Single-line JSON Events

To test JSON events, I am using the Splunk introspection logs from the _introspection index. This search also extracts ten desired events and re-indexes them in the new index. This search inserts metadata fields into the JSON event:

index=_introspection sourcetype=splunk_disk_objects earliest=-10m@m latest=now
| eval myDynamicSourceJSON=source
| eval myDynamicSourcetypeJSON=sourcetype
| eval myDynamicHostJSON=host
| rex mode=sed s/.$//g
| eval _raw = _raw.",\"myDynamicSourceJSON\":\"".myDynamicSourceJSON."\",\"myDynamicSourcetypeJSON\":\"
".myDynamicSourcetypeJSON."\":\",\"myDynamicHostJSON\":\"".myDynamicHostJSON."\
"}"
| collect testmode=true index=collect_test source="myDynamicSourceJSON" sourcetype="myDynamicSourcetypeJSON" Host="myDynamicHostJSON"

The data in the original events does not pretty-print as JSON, but all fields are extracted in the search as they are in the raw Splunk event.

The events are re-indexed into the new index, and with the | spath command, all the fields from the JSON are extracted as well as visible under Interesting Fields.

One thing to note here is that this is not a mass re-indexing solution. This is good for a very specific use case where there are not a lot of variables involved.

To learn more about this or if you need help with implementing a custom solution like this, please feel free to reach out to us.

      By: Yetunde Awojoodu  |  Splunk Consultant

 

Depending on your use case or what you are looking to achieve with your Splunk Processing Language (SPL), you may need to query multiple data sources and merge the results. The most intuitive command to use when these situations arise is the “join” command, but it tends to consume a lot of resources – especially when joining large datasets. I will be describing a few other commands or functions that can be applied when combining data from multiple sources in Splunk, including their benefits and limitations.

“OR” Boolean Operator

The most common use of the “OR” operator is to find multiple values in event data, e.g. “foo OR bar.” This tells the program to find any event that contains either word. However, the “OR” operator is also commonly used to combine data from separate sources, e.g. (sourcetype=foo OR sourcetype=bar OR sourcetype=xyz). Additional filtering can also be added to each data source, e.g., (index=ABC loc=Ohio) OR (index=XYZ loc=California). When used in this manner, Splunk runs a single search, looking for any events that match any of the specified criteria in the searches. The required events are identified earlier in the search before calculations and manipulations are applied.

Syntax for “OR” Operator:

(<search1>) OR (<search2>) OR (<search3>)

Pros:

• – Merges fields and event data from multiple data sources
• – Saves time since it does only a single search for events that match specified criteria and returns only the applicable events before any other manipulations

Cons:

• – Only used with base searches. Does not allow calculations or manipulations per source, so any further calculations or manipulations will need to be performed on all returned events

Example: In the example below, the OR operator is used to combine fields from two different indexes and grouped by the customer_id, which is common to both data sources.

Append Command

Append is a streaming command used to add the results of a secondary search to the results of the primary search. The results from the append command are usually appended to the bottom of the results from the primary search. After the append, you can use the table command to display the results as needed. Note that the secondary search must begin with a generating command. It is important to also note that append searches are not processed like subsearches where the subsearch is processed first. They are run at the point they are encountered in the SPL.

Syntax for Append:

<primary search> ... | append [<secondary search>]

Pros:

• – Displays fields from multiple data sources

Cons:

• – Subject to a maximum result rows limit of 50,000 by default
• – The secondary search must begin with a generating command
• – It can only run over historical data, not real-time data

Example: In the example below, the count of web activities on the Splunk User Interface is displayed from _internal index along with count per response from the _audit index.

The last four rows are the results of the appended search. Both result sets share the count field. You can see that the append command just tacks on the results of the subsearch to the end of the previous search, even though the results share the same field values.

Multisearch Command

Multisearch is a generating command that runs multiple streaming searches at the same time. It requires at least two searches and should only contain purely streaming operations such as eval, fields, or rex within each search. One major benefit of the multisearch command is that it runs multiple searches simultaneously rather than sequentially as with the append command. This could save you some runtime especially when running more complex searches that include multiple calculations and/or inline extractions per data source. Results from the multisearch command are interleaved, not added to the end of the results as with the append command.

Syntax for the Multisearch Command:

| multisearch [<search1>] [<search2>] [<search3>] ...

Since multisearch is a generating command, it must be the first command in your SPL. It is important to note that the searches specified in square brackets above are not actual subsearches. They are full searches that produce separate sets of data that will be merged to get the expected results. A subsearch is a search within a primary or outer search. When a search contains a subsearch, Splunk processes the subsearch first as a distinct search job and then runs the primary search.

Pros:

• – Merges data from multiple data sources
• – Multisearch runs searches simultaneously, thereby saving runtime with complex searches
• – There is no limit to the number of result rows it can produce
• – Results from the multisearch command are interleaved allowing for a more organized view

Cons:

• – Requires that the searches are entirely distributable or streamable
• – Can be resource-intensive due to multiple searches running concurrently. This needs to be taken into consideration since it can cause search heads to crash.

Example: In the example shown below, the multisearch command is used to combine the action field from the web_logs index and queue field from the tutorial_games index using eval to view the sequence of events and identify any roadblocks in customer purchases. The results are interleaved using the _time field.

Union Command

Union is a generating command that is used to combine results from two or more datasets into one large dataset. The behavior of the union command depends on whether the dataset is a streaming or non-streaming dataset. Centralized streaming or non-streaming datasets are processed the same as append command while distributable streaming datasets are processed the same as multisearch command.

Syntax for Union Command:

| union [<search2>] [<search2>] … OR … | union [<search>]

However, with streaming datasets, instead of this syntax:
<streaming_dataset1> | union <streaming_dataset2>

Your search is more efficient with this syntax:
... | union <streaming_dataset1>, <streaming_dataset2>

Pros:

• – Merges data from multiple data sources
• – Can process both streaming and non-streaming commands, though behavior will depend on the command type
• – As an added benefit of the max out argument, which specifies the maximum number of results to return from the subsearch. The default is 50,000 results. This value is the maxresultrows setting in the [searchresults] stanza in the limits.conf file.

Example: The example below is similar to the multisearch example provided above and the results are the same. Both searches are distributable streaming, so they are “unioned” by using the same processing as the multisearch command.

In the second example below, because the head command is a centralized streaming command rather than a distributable streaming command, any subsearches that follow the head command are processed using the append command. In other words, when a command forces the processing to the search head, all subsequent commands must also be processed on the search head.

Comparing the Four Options

The table below shows a comparison:

#	OR	Append	Multisearch	Union
1.	Boolean Operator	Streaming command	Generating command	Generating Command
2.	Used in between searches	Used in between searches	Must be the first command in your SPL	Can be either the first command or used in between searches. Choose the most efficient method based on the command types needed
3.	Results are interleaved	Results are added to the bottom of the table	Results are interleaved	Results are interleaved based on the time field
4.	No limit to the number of rows that can be produced	Subject to a maximum of 50,000 result rows by default	No limit to the number of rows that can be produced	Default of 50,000 result rows with non-streaming searches. Can be changed using maxout argument.
5.	Requires at least two base searches	Requires a primary search and a secondary one	Requires at least two searches	Requires at least two searches that will be “unioned”
6	Does not allow use of operators within the base searches	Allows both streaming and non-streaming operators	Allows only streaming operators	Allows both streaming and non-streaming operators
7.	Does only a single search for events that match specified criteria	Appends results of the “subsearch” to the results of the primary search	Runs searches simultaneously	Behaves like multisearch with streaming searches and like append with non-streaming
8.	Transforming commands such as chart, timechart or stats cannot be used within the searches but can be specified after		Transforming commands such as chart, timechart or stats cannot be used within the streaming searches	Behaves like multisearch with streaming searches and like append with non-streaming

I hope you now have a better understanding of the different multisearch command options presented and will make the most optimized choice for your use case.

Want to learn more about combining data sources in Splunk? Contact us today!