Archive for category: Blog

  By: Bruce Johnson  | Director, Enterprise Security

 

Many organizations use Splunk today. Of those adopters, most have a distributed Splunk environment. Often, organizations have sensitive data traversing their network, which makes its way into Splunk. More now than ever, security is at the forefront of everyone’s mind, and securing your Splunk environment is no exception. How to properly secure a distributed Splunk environment is not a new concept, but it is still frequently underutilized or improperly implemented. With that being said, an overview of how to implement SSL between your Splunk Deployment Server and Splunk Web instances will be discussed in detail. Manually configuring the Search Head and Search Peers will be overviewed.

As Splunk ships with OpenSSL, this method will be discussed in examples. Ensure you are using the proper version of OpenSSL on each Splunk instance. The steps provided assume you are configuring on a Linux-based host, and Splunk is installed in the /opt/splunk OR /opt/splunkforwarder directory, and using Splunk default ports.

Deployment Server and Splunk Web

You will want to secure your traffic from your web browser to your Deployment Server, as non-ssl traffic transfers raw data. Cleartext makes it easy for those that know how to intercept traffic to read your data. Use SSL certificates to help secure your data by turning your cleartext into ciphertext, especially when you need to access instances outside of your network. There are, of course, default certificates that ship with Splunk. It is a best practice to go with either a self-signed or a purchased CA-signed certificate instead. How to utilize self-signed certificates will be discussed.

First, make a directory for your self-signed certificates, to ensure you don’t interfere with the default Splunk certificates. Then, traverse to that folder. While in that folder, create your private key (PK) that you will utilize to sign your certificates. Ensure access to this folder is limited to personnel that needs access, as private keys should never be shared. Encrypted data can be decrypted by anyone that has the private key.

Next, you will need to generate your custom Certificate Signing Request (CSR). Use your CSR to create your public certificate (.pem), which is what you will distribute to your various Splunk instances. With the root certificate created to act as a CA, you will then utilize the CSR, CA certificate, and private key to generate and sign a server certificate that is valid for three years. Use the server certificate by distributing it to your indexers, forwarders, and other Splunk instances, which communicate over management port 8089. We will only discuss, however, implementing it on your Deployment Server.

• 1. mkdir /opt/splunk/etc/auth/custcerts
• 2. cd /opt/splunk/etc/auth/custcerts
• 3. /opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out mattCAPK.key 2048
  • a. Enter a secure password, then again to confirm.
• 4. /opt/splunk/bin/splunk cmd openssl rsa -in mattCAPK.key -out mattCAPKNoPW.key
  • a. Removing the password makes it easier for testing.
  • b. You’ll need to enter the secure password you created in step 3 above.
• 5. /opt/splunk/bin/splunk cmd openssl req -new -key mattCAPKNoPW.key -out mattCACert.csr
  • a. Enter details to questions asked.
• 6. /opt/splunk/bin/splunk cmd openssl x509 -req -in mattCACert.csr -sha512 -signkey mattCAPKNoPW.key -CAcreateserial -out mattCACert.pem -days 1095

Now you will need to generate your server certificate:

• 7. /opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out mattServerPK.key 2048
• 8. /opt/splunk/bin/splunk cmd openssl rsa -in mattServerPK.key -out mattServerNoPW
  • a. Again, removing the password makes testing easier.
• 9. Use your new server private key mattServerPK.key to generate a CSR for your server certificate. (use sdsp01.nielsen.com for the common name in the CSR).

Similar to steps 1-6, you will use the private key to create the CSR, then both to create the server certificate.

• 10. /opt/splunk/bin/splunk cmd openssl req -new -key mattServerNoPW.key -out mattServerCert.csr
• 11. /opt/splunk/bin/splunk cmd openssl x509 -req -in mattServerCert.csr -SHA256 -CA mattCACert.pem -CAkey mattCAPKNoPW.key -CAcreateserial -out mattServerCert.pem -days 1095

You’ll now want to concatenate them all together (you will do this two different times in these steps). The format and reasoning are explained here:

• 12. cat mattServerCert.pem mattServerNoPW.key mattCACert.pem > mattNewServerCert.pem

At this point, you will need to update the server.conf file on your Deployment Server. This file is located in the /opt/splunk/etc/system/local/ directory. You can get more granular in the stanzas if you prefer, and the options are listed in Splunk docs.

• 13. Find the [sslConfig] stanza.
• 14. [sslConfig]
• 15. enableSplunkdSSL = true
• 16. serverCert = /opt/splunk/etc/auth/custcerts/mattNewServerCert.pem
• 17. caCertFile = /opt/splunk/etc/auth/custcerts/mattCACert

Here you will need to restart Splunk on your Deployment Server instance.

• 18. /opt/splunk/bin/splunk restart\

You will need to generate a key specifically for the web UI for the Deployment Server. Please note that you must remove the password for the Splunk Web portion, as it’s not compatible with a password.

• 19. /opt/splunk/bin/splunk cmd openssl genrsa -des3 -out mattWebPK.key 2048
• 20. /opt/splunk/bin/splunk cmd openssl rsa -in mattWebPK.key -out mattWebPKNoPW.key
• 21. /opt/splunk/bin/splunk cmd openssl req -new -key mattWebPKNoPW.key -out mattWebCert.csr
• 22. /opt/splunk/bin/splunk cmd openssl x509 -req -in mattWebCert.csr -SHA256 -CA mattCACert.pem -CAkey mattCAPKNoPW.key -CAcreateserial -out mattWebCert.pem-days 1095
• 23. cat mattWebCert.pemmattCACert.pem > mattWebCertificate.pe

(You should be noticing a trend by now!)

You will now need to update the web.conf [settings] stanza, which is located in the /opt/splunk/etc/system/local/ directory path.

• 24. [settings]
• 25. enableSplunkWebSSL = true
• 26. privKeyPath = /opt/splunk/etc/auth/custcerts/mattWebPKNoPW.key
• 27. serverCert = /opt/splunk/etc/auth/custcerts/mattWebCertificate.pem

** For reasons of Splunk magic, the Deployment Server has issues pushing certs to Deployment Peers, so configure them individually/manually. An app would be much simpler method, though.

Once implemented, test within your browser. I have had issues with Google Chrome (see Image 1 below), but Firefox allows the page to be load as desired (see Image 2 below for reference). You will find that https only works at this point, and http no longer will.

SHs & Search Peers/Indexers

When adding search peers (indexers) to a search head, many admins will simply use the Splunk user interface (UI), or the command-line interface (CLI). In many situations, these are efficient and complete methods. There are, however, use cases that require adding search peers via editing distsearch.conf directly. This manner provides more granular and advanced features to be implemented. When editing distsearch.conf directly, key files need to be distributed manually to each search peer. This is in contrast to the two other methods, which implement authentication automatically.

Upon adding your search peers to your search head(s) via editing distsearch.conf, the key files need to be copied to the proper path. On your search head(s), copy the public key file, and place it in your search peer(s)’ file structure (file location(s) examples follow this paragraph). If adding search peer(s) to multiple search heads, then each search head’s public key file needs to be in its own folder named after the search head (utilize the actual serverName that is listed in server.conf for the folder name). Once the files have been properly copied over, simply restart Splunk on each Splunk search peer instance. The file location examples are as follows:

On your search head:

• $SPLUNK_HOME/etc/auth/distServerKeys/sh1PublicKey.pem

On your search peers/indexers:

• $SPLUNK_HOME/etc/auth/distServerKeys/searchHead1Name/sh1PublicKey.pem
• $SPLUNK_HOME/etc/auth/distServerKeys/searchHead2Name/sh2PublicKey.pem

Each instance of your Splunk deployment can, and should, be configured to use SSL. Each instance has its own caveats and nuances that need special attention to detail to configure properly. You should also look into securing your traffic between your forwarders and indexers.

Contact us for more help on configuring SSL for your distributed Splunk environment!

  By: William Phelps  |  Senior Technical Architect

 

This blog covers the basic steps for configuring the SignalFx agent and configuring a Python application running in Gunicorn to send trace data to SignalFx via the agent if Gunicorn is being executed within a Docker container.

Let’s start with a high-level overview of the technologies involved in the solution:

• – The SignalFx Tracing Library for Python automatically instruments Python 2.7 or 3.4+ applications to capture and report distributed traces to SignalFx within a single function.
• – The library accomplishes this by configuring an OpenTracing-compatible tracer to capture and export trace spans.
• – The tracer can also be used to embed additional custom instrumentation into the automatically generated traces. This blog will concentrate solely on the auto-instrumentation approach.
• – The SignalFx-Tracing Library for Python then works by detecting libraries and frameworks referenced by the application and then configuring available instrumentors for distributed tracing via the Python OpenTracing API 2.0. By default, the tracer footprint is small and doesn’t declare any instrumentors as dependencies.
• – Gunicorn, or ‘Green Unicorn,’ is a Python web server gateway interface (WSGI) HTTP Server for UNIX/Linux. The Gunicorn server is a pre-fork worker model and is broadly compatible with various web frameworks, simply implemented, light on server resources, and is fairly speedy.

The following notes are provided as general prerequisites or assumptions:

• – The SignalFx agent is already installed and initially configured on the Docker host. (The Otel collector is not yet compatible with this configuration.)
• – Alternately, the SignalFx agent can be deployed as its own Docker container, but this article assumes a local installation on the Docker host.
• – The SignalFx agent is already sending traces to the proper SignalFx realm and account.
• – The Docker host is assumed to be Linux. (RHEL/Centos/Oracle/AWS). The steps would be similar for Ubuntu/Debian, but the commands shown will be RHEL-centric.
• – Python 3 is installed on the host.
• – Ensure that proper access to Terminal or a similar command-line interface application is available.
• – Ensure that the installing Linux username has permission to run “curl” and “sudo.”

Process Steps

The general overall flow for this process is relatively short:

1. Configure the SignalFX agent to monitor Docker containers.
2. Create a Gunicorn configuration to support the tracing wrapper.
3. Create a Dockerfile to deploy the application.

Docker Monitoring Configuration

If the SignalFx agent was installed previously, navigate to the folder where the agent.yaml resides.

Edit the agent.yaml file to enable Docker monitoring. Under the “Observers” section, add a type of “Docker” as shown, and under “Monitors,” add a type of “docker-container-stats.”

Also, under the “Monitors” section, ensure that for the type “signalfx-forwarder,” the attribute “listenAddress” is set to 0.0.0.0:9080, and not “localhost” or “127.0.0.1”.

Additionally, under the type “signalfx-forwarder,” uncomment the attribute “defaultSpanTags.”

Uncomment and set a meaningful value for “environment” as shown. This value will be sent with every trace from this host/SignalFX agent invocation and is used as a filtering device. “Environment” is a child attribute of “defaultSpanTags.” Be aware of the appropriate indentation, as YAML is very strict.

Save the file. At this point, SignalFx will be able to send Docker metrics data, but it likely will not be sending anything. A quick look at the logs via journalctl:

…will probably show a permissions issue with reading the Docker socket. Add the user “signalfx-agent” to the “Docker” group and restart the SignalFx service to address this issue.

 

Gunicorn Configuration

Use the following steps to configure Gunicorn for auto-instrumentation of Python for SignalFX. Again, the assumption for this blog is that Gunicorn is being deployed to a Docker container.

1. In Python’s application root directory, create a file called “gunicorn.config.py”.
2. The contents of this file should appear as follows (or be modified to include the following):
   

Local Docker Configuration

At a high level, the Dockerfile simply consists of the directives to create the necessary environment to add Gunicorn and the Python application to the container. These directives include:

• – Creating the expected directory structure for the application.
• – Creating a non-root user to execute Gunicorn and the application inside the container. (Running as root is not recommended and will likely cause the container build to fail.)
• – Setting environment variables to pass to the container context.
• – Loading the Splunk OpenTelemetry dependencies.
• – Launching Gunicorn.

Please note the “ENV” directive for “SPLUNK_SERVICE_NAME” in step 7. The service name is the only configuration option that typically needs to be specified. Adjust this value to indicate the Python service being traced.

Other options can be found in the GitHub documentation under “All configuration options.”

This Dockerfile is using the Python3.8 parent image as the FROM target. Accordingly, the “pip” instruction in step 8 may need to be altered based on the parent image. The requirements file argument, however, should still be valid.

The requirements file appears as follows. This file lists out Gunicorn and specific Paste libraries, along with basic setup items and a version of Flask. The actual requirements for a project may vary. “Splunk-opentelemetry” in turn will load packages it requires. As such, this requirements file is not to be considered the complete library source.

Setting the PATH variable typically is NOT needed as shown in step 10. However, this ensures that the correct environment is present prior to running the bootstrap. The PATH must include the user’s “.local/bin” and “.local” folders from the home directory.

Finally, in step 12, note the use of both “- -paste” and “-c”. “ – -paste” of an .ini file allows additional configuration to be added to the build. “-c” is required to get the SignalFx configuration loaded that was defined in “gunicorn.config.py” earlier. This initialization line is shown to illustrate that both parameters can be used simultaneously. “-c” should follow “paste” if both are used.

Running the Dockerfile will generate a lot of output, but the final lines should look something like this:

Checking in the SignalFx UI should show the new service defined in step 7 of the Dockerfile.

Contact us for more help on configuring SignalFx for Docker & Gunicorn!

  By: Jon Walthour  |  Team Lead, Senior Splunk Consultant

 

Problem: Take two multi-site indexer clusters and meld them into one with all the buckets from cluster A residing in and being managed by cluster B. Then, with all the buckets transferred to cluster B, cluster A indexer hardware can be decommissioned.

TL;DR: It is not possible to do this, because the buckets from cluster A will never be moved outside of cluster A’s storage by cluster B’s cluster manager.

Step 1 is to make the clusters identical in configuration. You’d have to:

1. Ensure both clusters are running on the same version of Splunk.
2. Ensure the indexes.conf files on both clusters were identical—they both contain all the indexes, that each index stanza is named the same in both (e.g. one indexer cluster can’t put its Windows Event Logs in an index named “windows” and the other put them in one named “wineventlog”). Configurations would need to be changed and directories renamed to make both clusters the same in terms of index names and locations of hot/warm and cold buckets. Bottom line: You need to be able to use the same indexes.conf in either cluster since, when they were merged, they would be.
3. The contents of the controller apps and agent apps directories would also need to be identical in terms of contents and versions across both clusters. Again, they are eventually going to share the controller app’s contents. So, in preparation, they must be made identical.

Step 2: Turn cluster A from a multi-site indexer cluster into a single-site indexer cluster. The replicated buckets on site2 in cluster A are redundant and will get in the way. To do this, site search and replication factors in cluster A will need to be set to “total” values of “2” on each. Then, wait for all the fixup tasks to complete to get the cluster to meet site_replication_factor and site_search_factor. Finally, you would follow these steps to convert cluster A to a single-site indexer cluster.

Step 3: Now add single-site cluster A to multi-site cluster B as a new site. For instance, if multi-site cluster B has three sites, site1, site2, and site3, cluster A now becomes site4 of cluster B. You do this by:

1. Edit server.conf on the cluster manager, adding “site4” to the “available_sites” attribute under the “clustering” stanza and restart Splunk.
2. Add peers to the new site by running:
   splunk edit cluster-config -mode peer -site site4 -master_uri https://:8089 -repiication_port 8080 -secret
splunk restart
3. Add search heads by running:
   splunk edit cluster-config -mode searchhead -site site4 -master_uri https://:8089 -replication_port 8080 -secret
splunk restart
4. Point forwarders at the new cluster by adding the following to their server.conf:
   [general]
site=site4

Step 4: In theory, then, you would next decommission site4, causing all its buckets to be dumped into the other three sites. Here’s the rub, though, and it’s a big one: the buckets created when site4 was cluster A won’t migrate off the site. Any buckets created when site4 was cluster A will continue to only exist on site4 and the cluster manager will never migrate them elsewhere into other sites.

This is clearly stated in the documentation for decommissioning a site of a multi-site indexer cluster:

If a peer on the decommissioned site contains any buckets that were created before the peer was part of a cluster, such buckets exist only on that peer and therefore will be lost when the site is decommissioned…Similarly, if the decommissioned site started out as a single-site cluster before it became part of a multisite cluster, any buckets that were created when it was a single-site cluster exist only on that site and will be lost when the site is decommissioned.

It is immaterial here that cluster A was once a multi-site indexer cluster. The condition is the same. They are buckets from a foreign cluster and the cluster manager will leave them segregated as long as they exist. So, the only way your end goal could be accomplished is if you could hold on to the hardware for one of the sites in cluster A until such time as all its buckets aged out to frozen and were, presumably, deleted. Then, you would be at a state where all the buckets in site4 were created after it became site4 and no buckets remained from when it was cluster A. Only at that time could you decommission site4 and reclaim the hardware without suffering any data loss.

Contact us for more help with merging multi-site indexer clusters:

      By: Zubair Rauf  |  Splunk Consultant, Team Lead

 

A few days ago, I came across a very rare use case in which a user had to reindex a specific subset of raw Splunk events into another index in their data. This was historical data and could not be easily routed to a new index at index-time.

After much deliberation on how to move this data over, we settled on the summary index method, using the collect command. This would enable us to search for the specific event we want and reindex them in a separate index.

When re-indexing raw events using the “collect” command, Splunk automatically assigns the source as search_id (for ad-hoc search) and saved search name (for scheduled searches), sourcetype as stash, and host as the Splunk server that runs the search. To change these, you can specify these values as parameters in the collect command. The method we were going to follow was simple – build the search to return the events we cared about, use the collect command with the “sourcetype,” “source,” and “host” values to get the original values of source, sourcetype, and host to show up in the new event.

To our dismay, this was not what happened, as anything added to those fields is treated as a literal string and doesn’t take the dynamic values of the field being referenced. For example, host = host would literally change the host value to “host” instead of the host field in the original event. We also discovered that when summarizing raw events, Splunk will not add orig_source, orig_sourcetype, and orig_index to the summarized/re-indexed events.

To solve this problem, I had to get creative, as there was no easy and direct way to do that. I chose props and transforms to solve my problem. This method is by no means perfect and only works with two types of events:

• – Single-line plaintext events
• – Single-line JSON events

Note: This method is custom and only works if all steps are followed properly. The transforms extract field values based on regex, therefore everything has to be set up with care to make sure the regex works as designed. This test was carried out on a single-instance Splunk server, but if you are doing it in a distributed environment, I will list the steps below on where to install props and transforms.

The method we employed was simple:

1. Make sure the target index is created on the indexers.
2. Create three new fields in search for source, sourcetype, and host.
3. Append the new fields to the end of the raw event (for JSON, we had to make sure they were part of the JSON blob).
4. Create Transforms to extract the values of those fields.
5. Use props to apply those transforms on the source.

I will outline the process I used for both types of events using the _internal and _introspection index in which you can find both single-line plain text events and single-line JSON events.

Adding the Props and Transforms

The best way to add the props and transforms is to package them up in a separate app and push them out to the following servers:

1. Search Head running the search to collect/re-index the data in a new index (If using a cluster, use the deployer to push out the configurations).
2. Indexers that will be ingesting the new data (If using a clustered environment, use the Indexer cluster Manager Node, previously Master Server/Cluster Master to push out the configuration).

Splunk TA

I created the following app and pushed it out to my Splunk environment:

TA-collect-raw-events/
└── local
├── props.conf
└── transforms.conf


The files included the following settings:

Transforms.conf

[setDynamicSource]
FORMAT = source::$1
REGEX = ^.*myDynamicSource=\"([^\"]+)\"
DEST_KEY= MetaData:Source

[setDynamicSourcetype]
FORMAT = sourcetype::$1
REGEX = ^.*myDynamicSourcetype=\"([^\"]+)\"
DEST_KEY= MetaData:Sourcetype

[setDynamicHost]
FORMAT = host::$1
REGEX = ^.*myDynamicHost=\"([^\"]+)\"
DEST_KEY= MetaData:Host

[setDynamicSourceJSON]
FORMAT = source::$1
REGEX = ^.*\"myDynamicSourceJSON\":\"([^\"]+)\"
DEST_KEY= MetaData:Source

[setDynamicSourcetypeJSON]
FORMAT = sourcetype::$1
REGEX = ^.*\"myDynamicSourcetypeJSON\":\"([^\"]+)\"
DEST_KEY= MetaData:Sourcetype

[setDynamicHostJSON]
FORMAT = host::$1
REGEX = ^.*\"myDynamicHostJSON\":\"([^\"]+)\"
DEST_KEY= MetaData:Host

Props.conf

[source::myDynamicSource]
TRANSFORMS-set_source = setDynamicSource, setDynamicSourcetype, setDynamicHost

[source::myDynamicSourceJSON]
TRANSFORMS-set_source = setDynamicSourceJSON, setDynamicSourcetypeJSON, setDynamicHostJSON

Once the TA is successfully deployed on the Indexers and Search Heads, you can use the following searches to test this solution.

Single-line Plaintext Events

The easiest way to test this is by using the _internal, splunkd.log data as it is always generating when your Splunk instance is running. I used the following search to take ten sample events and re-index them using the metadata of the original event.

index=_internal earliest=-10m@m latest=now
| head 10
| eval myDynamicSource= source
| eval myDynamicSourcetype= sourcetype
| eval myDynamicHost= host
| eval _raw = _raw." myDynamicSource=\"".myDynamicSource."\" myDynamicSourcetype=\"".myDynamicSourcetype."\" myDynamicHost=\"".myDynamicHost."\""
| collect testmode=true index=collect_test source="myDynamicSource" sourcetype="myDynamicSourcetype" host="myDynamicHost"

Note: Change testmode=false when you want to index the new data as testmode=true is to test your search to see if it works.

The search does append some metadata fields (created with eval) in the original search to the newly indexed event. This method will use license to index this data again as the sourcetype is not stash.

Single-line JSON Events

To test JSON events, I am using the Splunk introspection logs from the _introspection index. This search also extracts ten desired events and re-indexes them in the new index. This search inserts metadata fields into the JSON event:

index=_introspection sourcetype=splunk_disk_objects earliest=-10m@m latest=now
| eval myDynamicSourceJSON=source
| eval myDynamicSourcetypeJSON=sourcetype
| eval myDynamicHostJSON=host
| rex mode=sed s/.$//g
| eval _raw = _raw.",\"myDynamicSourceJSON\":\"".myDynamicSourceJSON."\",\"myDynamicSourcetypeJSON\":\"
".myDynamicSourcetypeJSON."\":\",\"myDynamicHostJSON\":\"".myDynamicHostJSON."\
"}"
| collect testmode=true index=collect_test source="myDynamicSourceJSON" sourcetype="myDynamicSourcetypeJSON" Host="myDynamicHostJSON"

The data in the original events does not pretty-print as JSON, but all fields are extracted in the search as they are in the raw Splunk event.

The events are re-indexed into the new index, and with the | spath command, all the fields from the JSON are extracted as well as visible under Interesting Fields.

One thing to note here is that this is not a mass re-indexing solution. This is good for a very specific use case where there are not a lot of variables involved.

To learn more about this or if you need help with implementing a custom solution like this, please feel free to reach out to us.