Using Federal RMF and Splunk for IT Systems That Do Not Require Regulatory Compliance

By: Don Arnold | Splunk Consultant


Regulatory compliance for IT systems is an area that can be mysterious and not well understood by many IT managers or security professionals.  Some industries are mandated to obtain regulatory compliance certifications for their IT systems, ie HIPAA for healthcare, PCI DSS for credit card transactions, Sarbanes Oxley for accounting, ISO for manufacturing, and FISMA for Federal systems.  However, compliance only applies to a small portion of IT systems and the rest are left to implement a mix of devices and techniques to secure their systems.  However, without a definitive roadmap to follow many IT systems can be left with holes in their security programs, which can increase the risk of attack and make their systems vulnerable to compromise.

Since the majority of IT systems are not required to implement compliance standards, it doesn’t mean they can’t use these standards as a roadmap to better secure their systems.  Many IT security professionals try to build a security practice within their network by using technical solutions.  Though this is good, following a security framework helps to ensure that all areas of security are covered.

One of the best and most complete security frameworks that is publicly available and comes at no cost to use is the Risk Management Framework from the U.S. Government, better known as the Federal RMF.  The Federal RMF uses a number of special publications from the National Institute of Standards and Technology (NIST) to help organize a company’s cybersecurity implementation into a life cycle framework.  The steps in the life cycle include preparation, categorization of the system, selection of applicable security controls, implementation, assessment, authorization, and monitoring.  Each step specifically identifies what happens and uses NIST documents to outline how to follow each one.

The last step in the RMF life cycle is monitoring and the NIST SP 800-53 specifically identifies an entire family of controls for Continuous Monitoring.  Splunk, an industry-leading SIEM, can be implemented to meet the Continuous Monitoring needs of Account Management, Privileged Access, Login Access and other items.  In addition, Splunk has several add-on applications, such as Infosec and Enterprise Security, that can assist with out-of-the-box data searches and dashboards to ensure your cybersecurity Continuous Monitoring needs are being met.

If your IT system is subject to regulatory compliance from HIPAA, PCI, SOX, etc., then the framework has a detailed roadmap for you to follow.  However, if your system does not require compliance, but you’re looking for a framework as a guideline to follow to ensure you’re covering your cybersecurity needs, then the Federal RMF and Splunk are some of the best tools on the market today.  TekStream Solutions has cybersecurity engineers on staff with years of experience implementing cybersecurity solutions and can assist you with building a strong cybersecurity program using the Federal RMF.

Want to learn more about Splunk and cybersecurity? Contact us today!




Machine Learning with Splunk: Testing Logistic Regression vs Support Vector Machines (SVM) using the ML Toolkit

By: Brent McKinney | Splunk Consultant


If you’ve ever spent time working with Machine Learning, it’s highly likely you’ve come across Logistic Regression and Support Vector Machines (SVMs). These 2 algorithms are amongst the most popular models for binary classification. They both share the same basic goal: Given a sample x, how can we predict a variable of interest y?

For example, let’s say we have a dataset with samples (x) containing the following,

and we want to determine a single variable y: Is this person diabetic or not?

Logistic Regression and SVMs are perfect candidates for this!

The problem now lies in finding the means to test this on a sizeable dataset, where we have hundreds or thousands of samples. Coding machine learning algorithms can become quite a task, even for experienced developers. This is where Splunk comes in!

Splunk’s Machine Learning Toolkit makes testing ML algorithms a breeze. The Machine Learning Toolkit is an app, completely free to download on Splunkbase and allows users to visualize and compare results from ML algorithms quickly, without having to code them.

To stay consistent with our previous example, I will be demonstrating this with a public dataset from, in the form of a CSV file. This dataset includes real data, that is already labeled and clean. Since our data is properly labeled, this can be will serve as a supervised learning problem. This simply means that our task will learn a function that maps an input (our x features) to an output (our y value – diabetic or not) based on the labeled items. I’ve posted the link to the MLTK app, as well as the dataset used in this example, as sources at the bottom of this page.

To install the MLTK app: Once you’ve downloaded the Machine Learning Toolkit from Splunkbase, log into your Splunk instance, and click the Apps dropdown at the top of the screen. Select “Manage Apps” and then click the button “Install app from file”. From here select Choose File and select the MLTK app folder (no need to untar the file, Splunk will unpack the folder on the server!). Click Upload.

To upload a csv file: You can upload the csv file by clicking Settings>Lookups>Lookup table files>New Lookup Table File. Select MLTK for the app, our csv as the upload file, and give a name with the .csv extension (diabetes.csv). Then go to Settings>Lookups>Lookup definition>New Lookup Definition to define the lookup. We’ll select the MLTK app, “diabetes” for the name, “File-based” for the type, and the csv file for the Lookup file.

Once the Machine Learning Toolkit has been installed, and the dataset file has been uploaded to Splunk, we can get to work.

From your Splunk instance, navigate to the Machine Learning Toolkit app by selecting it from the “App” dropdown menu at the top of the screen.

From here we can see there are several options available, each depending on the problem you are trying to solve. We want to categorize our data into 2 categories: diabetic and not diabetic. So for our example, we will use “Predict Categorical Fields”.

To start, select “Experiments” from the navigation bar, and then “Create New Experiment”. Select the appropriate experiment type, then add a title and description.


Once all looks good, select Create.

Now we are brought to the experiment screen. To use our diabetes dataset, we will need to use the SPL inputlookup command in the search bar. Note the search must begin with a | as this is a generating command.

This will return the data we uploaded from the CSV file.

As we can see, there are a few parameters that need to be set. The first being the algorithm we want to use. We will be testing Logistic Regression and SVM. The default is Logistic Regression so we can leave it as-is for now.

The next parameter is “Field to Predict”. This represents the variable we want to discover, y. This list is populated with fields found in our csv file. In our example, our  y variable is “Outcome”, which gives a value of 1 for samples that are diabetic, and a value of 0 for samples that are NOT diabetic.



The next parameter is “Fields to use for predicting”. As the name implies, these are the variables that make up our feature sample x. The algorithms will use these features to determine our Outcome variable. The more relevant fields we select here, the more accurate our algorithms will be when calculating a result, so in this case we will select all of them.

Once these parameters have been set, all we need to do is decide how we want to split the data into training and testing.

Machine Learning algorithms use the training data to determine a function that most accurately produces the desired output. So to achieve the best accuracy, we want to use a majority of the data for training. Once the algorithm is trained on the dataset, it runs this function on the test data and gives an output based on the samples it saw during training. For this example, I will use 80% for training, and 20% for testing.

(Note, while we want to use as much training data as possible, we must have some test data. If we use 100% of the data for training, then any test data will have already been seen by the algorithm, and therefore not give us any insightful results.)

Now that all of our parameters are set, we are ready to see results!

Select Fit Model to run Logisitic Regression on our data.

Once the algorithm is finished, we are given 3 panels.

The first returns a results table containing our test data. The columns on the right of the bold line show our original x features for each sample. The columns on the left of the bold line show the output that the algorithm predicted for each sample, compared to the actual output in the dataset, for each sample, highlighting the ones it got wrong.

The panel on the bottom left shows the degree of accuracy of the algorithm for our given dataset. From this we can conclude that if we were to give this model a new sample, it would determine whether or not the sample is diabetic or not with a 77% degree of accuracy.

The bottom right panel gives a little more detail, showing how well the algorithm did at predicting each outcome. We can see that for our particular example, it did slightly better at determining samples that were not diabetic, as opposed to samples that were.

Now let us compare this to a SVM model. Considering that we want to use the same dataset and parameters, all we need to do is change the algorithm.

Once that is set, we can select Fit Model to run SVM on our data.

Right away we can see that using Support Vector Machines gives us substantially better results than Logisitic regression. Both algorithms give the same details format, but we can see that using SVM resulted in a 97% accuracy when predicting on our test data, in comparison to LR resulting in 77%.


To conclude, Splunk’s Machine Learning Toolkit provides an easy-to-use environment for testing and comparing Machine Learning algorithms. In this demonstration, we used Splunk and Machine Learning to create models to predict whether a given sample is diabetic or not. While this demonstration focused on SVMs and Logistic Regression, there are many more algorithms available in Splunk’s Machine Learning Toolkit to play around with, including Linear Regression, Random Forests, K-means Clustering, and more!

Link to download Machine Learning Toolkit app:

Link to download dataset used in this example:


Want to learn more about Machine Learning with Splunk? Contact us today!

Options to Consider for Your Oracle 12c WebCenter Upgrade

By: Brandon Prasnicki | Technical Architect


If you search the Oracle knowledgebase on how to upgrade your existing Oracle WebCenter Content (WCC), Imaging, or Portal instance from 11g to 12c, your options are to do an in-place upgrade or to migrate the entire repository using Oracle WebCenter Content supported tools.  However, if an upgrade consists of new hardware (on-premise), new cloud Infrastructure (Oracle Cloud Infrastructure (OCI), Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform etc), upgraded operating systems (Microsoft Windows or Linux) along with database upgrade (Oracle Database 12c), the only supported method is to use these supported migration tools.  To move the content from one machine to the next, this process consists of the following:

  1. Install 12c on the new environment
  2. Create the 12c schemas with RCU
  3. Create and configure the 12c domain
  4. Migrate the WCC configurations with CMU and Archiver
  5. Migrate the WCC content with Archiver

While this is a straightforward approach, the question becomes:  Is this feasible?

The answer to that question is:  It depends.

With any upgrade project, TekStream Solutions evaluates the scope of the upgrade and migration and makes recommendations on the appropriate approach.  Here is a high-level outline of starting points considered during the TekStream QuickStream process:

  1. Is the repository small? This supported methodology is a good approach and alternative for instances that do not hold a lot of content.  We have seen situations for implementations that leverage WCC as a backend for implementations like Portal where the content repository isn’t very large.  For this, the supported methodology is a decent alternative.
  2. Are there opportunities to decommission old Enterprise Content Management Systems? Sometimes there is an opportunity to also mix in and decommission old content repositories. Examples include old shared filesystems not currently managed by any enterprise content management systems (CMS), or even little-used old CMS systems where, depending on the customer license structure, the ROI of rolling into an Oracle WebCenter Content (WCC) instance makes sense during the time of upgrade.  Examples of this include but are not limited to Adobe and Documentum etc.  For this, TekStream utilizes a proprietary utility called “Content Loader” to handle WCC delta migrations, and merge deprecated CMS application content.
  3. Is the repository large? For very large repositories, Tekstream uses a cost-effective approach called the “out of place” in-place upgrade which eliminates the need to migrate the content.  The ‘supported’ Oracle approach simply is not feasible, as repositories with millions of content items would take months and maybe even years to migrate.  Examples of implementations that include large repositories include Digital Asset Management (DAM), Records Management (RM) and even some regular Content Management repositories.   When Oracle states this “out of place” in-place upgrade is not a supported approach, they are strictly referring to all the ‘gothchas’ that can occur.  The support team members at Oracle are not the consultants to handle such an approach.  That is where TekStream solutions come in to guide and implement the upgrade to a successful outcome.
    1. Have we seen ‘gotchas’ in this approach? Certainly.  Every version and situation has its nuances.  TekStream’s QuickStream process digs deeper into identifying unique customer situations to account for during a migration.  Tekstream has proven to handle these challenges and deliver successful implementations.  Our background expertise performing these upgrades before has proven vital to customer success.
    2. Could a customer do this approach by themselves? Honestly, probably not.  We are here to guide you through this approach and avoid the pitfalls that can occur.  We have been through this before, and are here to guide and deliver a successful upgrade.

TekStream Solutions makes sure that the system is migrated, upgraded and in a clean, working, and supported state at the completion of the project.   This approach has proven to save customers a lot of time and money.  TekStream also offers extended support and is an Oracle Managed Services provider to give customers great peace of mind and frees up internal resources for more demanding in-house projects.

Want to learn more about Oracle 12c WebCenter upgrades? Contact us today!

Press Release: TekStream Makes 2019 INC. 5000 List for Fifth Consecutive Year

For the 5th Time, Atlanta-based Technology Company Named One of the Fastest-growing Private Companies in America with Three-Year Sales Growth of 166%

ATLANTA, GA, August 14, 2019– Atlanta-based technology company, TekStream Solutions, is excited to announce that for the fifth time in a row, it has made the Inc. 5000 list of the fastest-growing private companies in America. This prestigious recognition comes again just eight years after Rob Jansen, Judd Robins, and Mark Gannon left major firms and pursued a dream of creating a strategic offering to provide enterprise technology software, services, solutions, and sourcing. Now, they’re a part of an elite group that, over the years, has included companies such as Chobani, Intuit, Microsoft, Oracle, Timberland, Vizio, and

“Being included in the Inc. 5000 for the fifth straight year is something we are truly proud of as very few organizations in the history of the Inc. 5000 list since 2007 can sustain the consistent and profitable growth year over year needed to be included in this prestigious group of companies,” said Chief Executive Officer, Rob Jansen. “The accelerated growth we are seeing to help clients leverage Cloud-based technologies and Big Data solutions to solve complex business problems has been truly exciting. We are helping our clients take advantage of today’s most advanced recruiting and technology solutions to digitally transform their businesses and address the ever-changing market.”

This year’s Inc. 5000 nomination comes after TekStream has seen a three-year growth of over 166%, and 2019 is already on pace to continue this exceptional growth rate. In addition, the company has added 30% more jobs over the last 12 months.

“Customers continue to invest in ‘Cloud First’ strategies to move their on-premises environments to the cloud, but often struggle with how to get started.  There is a vast market for specialized experts familiar with both legacy systems and newer cloud technology platforms.  Bridging those two worlds to address rapid line of business changes and reducing technology costs are focal points of those strategies. TekStream is well-positioned to continue that thought leadership position over the next several years.” stated Judd Robins, Executive Vice President of Sales.

To qualify for the award, companies had to be privately owned, established in the first quarter of 2015 or earlier, experienced a two-year growth in sales of more than 50 percent, and garnered revenue between $2 million and $300 million in 2018.

“The continued recognition is evidence of our team’s response to client’s recruiting needs across multiple industries and sectors. The growth in hiring demands commercially and federally along with the need to deliver on changing candidate demands have fueled the work we have put into having both outsourced and immediate response contingent recruiting solutions,” stated Mark Gannon, Executive Vice President of Recruitment.

We are “The Experts of Business & Digital Transformation”, but more importantly, we understand the challenges facing businesses and the myriad of technology choices and skillsets required in today’s “always on” companies and markets. We help you navigate the mix of transformative enterprise platforms, talent and processes to create future-proof solutions in preparing for tomorrows opportunities…so you don’t have to. TekStream’s IT consulting solutions combined with its specialized IT recruiting expertise helps businesses increase efficiencies, streamline costs and remain competitive in an extremely fast-changing market. For more information about TekStream Solutions, visit or email

Integrating Oracle Human Capital Management (HCM) and Content and Experience Cloud (CEC)

By: Greg Becker | Technical Architect


During the first phase of a recent project we built an employee file repository for a Healthcare client in the Oracle Cloud Infrastructure – Classic (OCI-C) space. A number of services were used including Oracle Content and Experience Cloud (repository), Oracle Process Cloud Service (for filing the documents in a logic structure), Oracle WebCenter Enterprise Capture (for scanning) and Oracle Database Cloud Service (for custom application tables).

During the second phase of the project our clients had a requirement to automatically update metadata values on their content items stored in the CEC repository. They wanted to trigger a change based on events or updates that occur for an employee record that is stored in Oracle Human Capital Management, for example when an Employee Status changes from Active to Inactive.

Our solution was to use an Oracle Process Cloud Service process to perform the metadata updates when certain values were passed into the process. The reason for updating the metadata is so that accurate searches can be performed by end users. The tricky part of the implementation is how to call the PCS process based on the change. To accomplish this Informatica is used to determine a ‘change’ based on data from the tables within the HCM structure and then pass that change record to a DB table used by the client solution. At that point a database function was developed to action the PCS REST Web Service. The final step of the process was to build a database trigger that called the function.

First you need to do some initial setup to be able to use the APEX libraries as well as create the network ACL to connect to the PCS domain you’re using. You can find this information in various places online. You can either use SOAP or REST web services and we chose REST. If you want to call the web service using SSL (which we did) you’ll have to also create an Oracle wallet.


Function Definition:

SOAP Envelope:

Call the Function from a Trigger:


There are more than one ways to fulfil this customer requirement but these are the pieces that worked well in this case. If you have any additional integration needs between Oracle Human Capital Management and Oracle Content and Experience Cloud please contact TekStream and we’d be happy to assist you.

Iplocation: Simple Explanation for Iplocation Search Command

By: Charles Dills | Splunk Consultant

Iplocation can be used to find some very important information. It is a very simple yet powerful search command that can help with identifying where traffic from a specific IP is coming from.

To start iplocation on its own won’t display any visualizations. What it will do is add a number of additional fields that can be used in your searches that can be added to dashboards, panels, and tables. Below we will use a simple base search using Splunk example data:

From here we will add iplocation to our search, sorting by clientip. As you can see in the below screenshot, this added a few fields that we can use circled in red:

From here we can alter our search with a table to display the information we need. For example, for a company who is based and fully operates out of the US could consider and traffic going outside the us to a foreign country as unauthorized or malicious. Using the iplocation in combination with values, we are able to list out each IP address that is not located inside the US and display each by which country It is located:

The last thing we will do is clean up our table using rename and this can provide a simple way to distinguish where traffic from a specific IP address is coming from:

Want to learn more about iplocation? Contact us today!

Take your Traditional OCR up a notch

By: Greg Moler | Director of Imaging Solutions

While the baseline OCR landscape has not changed much, AWS aims to correct that. Traditional OCR engines are quite limited in what details they can provide. Being able to detect the characters is only half the battle, the ability to get meaningful data out of them becomes the challenge. Traditional OCR follows the ‘what you see is what you get’ mantra, meaning once you run your document through, the blob of seemingly unnavigable text is all you are left with. What if we could enhance this output with other meaningful data elements useful in extraction confidence? What if we could improve the navigation of the traditional OCR block of text?

Enter Textract from AWS. A public web service aimed to improve your traditional OCR experience in an easily scalable, integrable, and low cost package. Textract is built upon an OCR extraction engine that is optimized by AWS’ advanced machine learning. It has been taught how to extract thousands of different types of forms so you don’t have to worry about it. The ‘template’ days are over. It also provides a number of useful advanced features that other engines simply do not offer: confidence ratings, word block identification, word and line object identification, table extraction, and key-value output. Let’s take a quick look at each of these:

  • Confidence Ratings: Ability to intelligently make choices to accept results, or require human intervention based on your own thresholds. Building this into your work flow or product can greatly improve data accuracy
  • Word Blocks: Textract will identify word blocks allowing you to navigate through them to help identify things like address blocks or known blocks of text in your documents. The ability to identify grouped wording rather than sifting through a massive blob of OCR output can help you find the information you are looking for faster
  • Word and Line Objects: Rather than getting a block of text from a traditional OCR engine, having code-navigable objects to parse your documents will greatly improve your efficiency and accuracy. Paired with location data, you can use the returned coordinates to pinpoint where it was extracted from. This becomes useful when you know your data is found in specific areas or ranges of a given document to further improve accuracy and filter out false positives
  • Table Extraction: Using AWS AI-backed extraction technology, Table extraction will intelligently identify and extract tabular data to pipe into whatever your use case may need, allowing you to quickly calculate and navigate these table data elements.
  • Key-value Output: AWS, again using AI-backed extraction technology, will intelligently identify key-value pairs found on the document without having to write custom engines to parse the data programmatically. Optionally, send these key-value pairs to your favorite key-value engine like Splunk or Elasticsearch (Elastic Stack) for easily searchable, trigger-able, and analytical actions for your document’s data.

Contact us today to find out how Textract from AWS can help streamline your OCR based solutions to improve your data’s accuracy!

Tsidx Reduction for Storage Savings

By: Yetunde Awojoodu | Splunk Consultant


Tsidx Reduction was introduced in Splunk Enterprise v6.4 to provide users with the option of reducing the size of index files (tsidx files) primarily to save on storage space. The tsidx reduction process transforms full size index files into minified versions which will contain only essential metadata. A few scenarios to consider tsidx reduction include:

  • Consistently running out of disk space or nearing storage limits but not ready to incur additional storage costs
  • Have older data that are not searched regularly
  • Can afford a tradeoff between storage costs and search performance

How it works

Each bucket contains a tsidx file (time series index data) and a journal.gz file (raw data). A tsidx file associates each unique keyword in your data with location references to events, which are stored in the associated rawdata file. This allows for fast full text searches. By default, an indexer retains tsidx files for all its indexed data for as long as it retains the data itself.

When buckets are tsidx reduced, they still contain a smaller version of the tsidx files. The reduction applies mainly to the lexicon of the bucket which is used to find events matching any keywords in the search. The bloom filters, tsidx headers, and metadata files are still left in place. This means that for reduced buckets, search terms will not be checked against the lexicon to see where they occur in the raw data. 

Once a bucket is identified as potentially containing a search term, the entire raw data of the bucket that matches the time range of the search will need to be scanned to find the search term rather than first scanning the lexicon to find a pointer to the term in the raw data. This is where the tradeoff with search performance occurs. If a search hits a reduced bucket, the resulting effect will be slower searches. By reducing tsidx files for older data, you incur little performance hit for most searches while gaining large savings in disk usage.

The process can decrease bucket size by one-third to two-thirds depending on the type of data. For example, a 1GB bucket would decrease in size between 350MB – 700MB. The exact amount depends on the type of data. Data with many unique terms require larger tsidx files. To make a rough estimate of a bucket’s reduction potential, look at the size of its merged_lexicon.lex file. The merged_lexicon.lex file is an indicator of the number of unique terms in a bucket’s data. Buckets with larger lexicon files have tsidx files that reduce to a greater degree.

When a search hits the reduced buckets, a message appears in Splunk Web to warn users of a potential delay in search completion: “Search on most recent data has completed. Expect slower search speeds as we search the minified buckets.” Once you enable tsidx reduction, the indexer begins to look for buckets to reduce. Each indexer reduces one bucket at a time, so performance impact should be minimal.


  • Savings in disk usage due to reduced tsidx files
  • Extension of data lifespan by permitting data to be kept longer (and searchable) in Splunk
  • Longer term storage without the need for extra architectural steps like adding S3 archival or rolling to Hadoop.


The configuration is pretty straight forward and you can perform a trial by starting with one index and observing the results before taking further action on any other indexes. You will need to specify a reduction age on a per-index basis:

1. On Splunk UI:

  • Go to Settings > Indexes > Select an Index
    Set tsidx reduction policy.

2. Splunk Configuration File:

  • indexes.conf
    enableTsidxReduction = true
    timePeriodInSecBeforeTsidxReduction = <NumberOfSeconds>

The attribute “timePeriodInSecBeforeTsidxReduction” is the amount of time, in seconds, that a bucket can age before it becomes eligible for tsidx reduction. When this time difference is exceeded, a bucket becomes eligible for tsidx reduction. Default Is 604800

To check whether a bucket is reduced, run the dbinspect search command:

| dbinspect index=_internal
The tsidxState field in the results specifies “full” or “mini” for each bucket.

To restore reduced buckets to their original state, refer toSplunk Docs

A few notes

  • Tsidx reduction should be used on old data and not on frequently searched data. You can continue to search across the aged data, if necessary, but such searches will exhibit significantly worse performance. Rare term searches, in particular, will run slowly.
  • A few search commands do not work with reduced buckets. These include ‘tstats’ and ‘typeahead’. Warnings will be included in search.log

Reference Links

Want to learn more about Tsidx Reduction for Storage Savings? Contact us today!

Operating a Splunk Environment with Multiple Deployment Servers

Operating a Splunk Environment with Multiple Deployment Servers

By: Eric Howell | Splunk Consultant

Splunk Environments come in all shapes and sizes, from the small single-server installation managing all of your Splunk needs in one easily-managed box, to the multi-site, extra complex environments scaled out for huge amounts of data and all the bells and whistles to get in-depth visibility and reporting into a wide variety of circumstances as suits functionally any use case you can throw at Splunk. And, of course, everything in between.

For those multi-site, or multi-homed environments, that many data centers require for any range of needs, managing your configurations begins to get complicated between the additional firewall rules, data management stipulations, and any other broad range of issues that might crop up.

Thankfully, Splunk Enterprise allows for your administrative team, or Splunk professional services, to set up a Deployment Server to manage the configurations (bundled into apps) for all of the universal forwarders, so long as they’ve been set up as deployment clients. In a complicated environment, you may find that you need two deployment servers to manage the workload, for any number of reasons. Perhaps you are trying to keep uniform configuration management systems in multiple environments, or perhaps you are aiming to spread the communication load across multiple servers for these deployments. Whatever the use case, setting up two (or more) deployment servers is not the heartache you may be worried about, and the guide below should be ample to get you on the right track.

Multiple Deployment Servers – Appropriate Setup

To set up multiple deployment servers in an environment, you will need to designate one of the Deployment Servers as the “Master” or “Parent” server (DS1). This is likely to be the original deployment server that houses all of the necessary apps, and is likely already serving as deployment server to your environment.

The use case below will allow you to service a multi-site environment where each environment requires the same pool of apps, but is small enough to be serviced by a single deployment server.

  1. Stand up a new box (or repurpose a decommissioned server, as is your prerogative)! Install Splunk on this new server. This will act as your second deployment server (DS2).
  2. The key difference between these servers is that DS2 will actually be a client of the DS1.
  3. Initial set up is minimal, but make sure that this server has any standard configurations the rest of your environment holds, such as an outputs.conf to send its internal logs to the indexer layer, if you are leveraging that functionality.
  • You will create a deployment client app on DS2. You could use a copy of a similar app that resides on one of your heavy forwarders that poll DS1 for configuration management, but you will need to make two key adjustments in deploymentclient.conf:

  • Once this change has been made, the apps that will be pulled down from DS1 will reside in the appropriate location on DS2 to be deployed out to any servers that poll it.
  • Restart Splunk on DS2
  • Next, you will need to navigate to the ForwarderManagement UI on DS1 and create a Server Class for your Slave or ChildDeployment Servers (DS2 in this case)
  • Add all apps to this new server class
    • Allowing Splunk to restart with these apps isfine, as changes made to the originating Deployment Server (DS1) will allow DS2
      to recognize that the apps that it holds have been updated and are ready for
  • Add DS2 to this Server Class
  • Depending on the settings you have configured indeploymentclient.conf on DS2 for its polling period (phoneHomeIntervalInSecs
    attribute), and how many apps there are for it to pull down from DS1, wait an appropriate amount of time (longer than your polling period, and more) and
    verify if the apps have all been deployed.
  • After this, updates made to the apps on DS1 will propagate down to DS2.

Alternative Use Case

If you are planning to leverage multiple deployment servers to service the same group of servers/forwarders, you will want to also copy over the serverclass.conf from DS1. If all server classes have been created through the web ui, the file should be available here:


If this is your intended use case, you will also want to work with your Network Team to place the Deployment Servers behind a loadbalancer. If you do so, you’ll need to modify the following attribute in deploymentclient.conf in your deployment client app that resides on your forwarders to indicate the VLAN:

You will also need to make sure both Deployment Servers generate the same “checksums” so that servers polling in and reaching different DS servers do not redownload the full list of apps with each check-in.

To do so, you will need to modify serverclass.conf on both Deployment Servers to include the following attribute:

This attribute may not be listed by default, so you may need to include it manually. This can be included with the other attributes in your [global] stanza.

Want to learn more about operating a Splunk environment with multiple deployment servers? Contact us today!