Automate Data Gathering and Evaluation with AWS

Jay Almers | Senior Cloud Solutions Architect


Whether you are creating usage and billing reports or taking inventory of your AWS account(s) in preparation for an application or service migration, data gathering and evaluation are imperative to making actionable, informed decisions. AWS provides some very powerful tools in their management console to assist their users in capturing the information necessary to make these kinds of decisions; however, searching for and obtaining data from multiple tools and services and combining them into an aggregate report can often be a tedious task.

To make things more difficult, let’s complicate that process by capturing this same information from multiple accounts. Depending on the volume of data, the number of sources, and the complexity of manipulation you need to perform, the amount of effort needed for data gathering and evaluation of this information could be quite significant. To make things even more difficult, what happens to the data (and the resulting reports) if a number was off by one decimal place unknowingly introduced due to human error?


Anything you have to do more than twice has to be automated.”

-Adam Stone, CEO of D-Tools


Automation is used in countless industries, from automotive manufacturing to zoological research, to perform repetitive tasks and decrease the possibility of error due to human intervention. The world of Information Technology is no different. Using the data inaccuracy scenario described above, it is clear to see that the process of collecting large amounts of data from multiple sources in a single account would benefit from automation, if for no other reason than to decrease the amount of time required to generate reports. In a multi-account situation, the time savings are even greater.

Another benefit of automation is its inherent ability to decrease the chances of introducing errors due to human intervention. That’s all well and good, but you may be wondering how we can automate the process of collecting this data from AWS without utilizing the management console and ultimately making your life easier. Enter the AWS Command Line Interface (CLI), an amazing tool that can be used for everything from service provisioning to data collection. With a little bit of programming and knowledge of how the AWS CLI works, we can automate the process of collecting information from multiple sources and data points in an efficient and repeatable manner without the need for tedious and error-prone human intervention.

To provide a real-world illustration of how this type of automation can be used, I’ll provide some details from a recent engagement TekStream was a part of. Our client, a large organization with many different internal departments, wanted to migrate a large number of non-enterprise AWS accounts into their controlled Enterprise Master Account. In order to migrate these accounts into the correct organizational structure with the necessary services, controls, and policies applied, they needed to determine the service and application requirements as well as gain an understanding of other operational considerations such as dependencies, availability, security, and fault tolerance.

Capturing this amount of information would have taken a tremendous amount of time if done manually; time that could be better spent on other areas of business. We developed a series of scripts and utilities which utilized the AWS CLI and a few supporting libraries for data parsing and manipulation. The scripts collected this invaluable information and created aggregate reports that simplified the process of comparing accounts and assisted the client in making actionable, data-driven decisions in preparation for migration.

Two of the main benefits of programmatic data collection, aggregation, and parsing is using native iteration in the form of loops and conditional logic such as if/then/else statements.  For example, we needed to capture information about all the configured Virtual Private Clouds (VPC) in all available AWS regions. Manually, we would have needed to log into the console, change into each region, navigate to the VPC service, then copy and paste each VPC ID into the report. Then, for each VPC ID, we’d need to navigate to the Gateways, Subnets, Route Tables, Network ACL, DHCP Options, Security Groups, and VPC Peering sub-sections to collect all the required information needed to complete the report.

Instead, once the script was developed, we could simply supply a list of account numbers and let the scripts iterate through all regions and service endpoints, building the reports for us – trimming the required effort from multiple hours of manual data gathering and evaluation to approximately 5 minutes.

This is just one example of a countless number of use cases for employing automation to simplify your current business processes.

If you would like more information on automation and how you may be able to leverage it to make your life a little easier,
contact TekStream today!

Setting up MFA with SNS and Java

Rimpi Mathur | Technical Architect


To provide better security and avoid unauthorized access to your applications, you should enable multi-factor authentication (MFA). This blog explains how to set up MFA using AWS APIs and services.

What is MFA?

With identity theft and cybercrime on the rise, multi-factor authentication (MFA) is a good way to improve your account security. MFA adds an additional layer of identity validation along with the basic login credentials. It is a unique number that is valid for a short duration of time and is sent to a device of your choice. This number is typically sent to an email address or cell phone number, but it can also be accessed using a specialty app.


We will be using the following technologies to achieve our goal.

  • • Java 1.8 or higher
  • • Eclipse or any other IDE
  • • Maven for code build


First, let’s look at sending MFA using the AWS admin console.

Sending text message to cell phone using Amazon Simple Notification Service

  • • AWS Console Settings

  1.  Under “Simple Notification Service,” click on Test Messaging (SMS).
  2.  Make sure your account is not in the SMS sandbox; in sandbox mode, you can send a maximum of 10 text messages. Submit a request for Service Limit Increase. This normally takes a day to be approved by AWS.
  3.  From the console, purchase an origination number. This is a 10-digit toll-free number that will be used for sending out messages. This number is assigned by AWS.
  4.  To send a message from the console click on “Publish Text Message.”
  5. Enter all the information in each box.

Publish SMS message, DetailsDestination phone numberCountry-specific attributes, Entity ID, Template IDNow that we know how to send an MFA code from the console, let’s integrate this into our code.

We will see how we can send the MFA code post-login from our codebase.

Generate MFA code using Java

There are many ways of creating a  6-digit number. One of the simplest ways is to use the Random() method in Java.

  • • Code for sending messages using SNS.


    1. We will import six packages that are required for defining the settings in AWS.


    1. Sample code for creating a text message.

There are three AWS attributes that need to be set: region, maximum cost, and SMS type.

public void sendSMS(String message, String phoneNumber) {


// Initializes the message, phone number, aws attributes, and sends the code.


try {

PublishResult result = initSnsClient()
     .publish(new PublishRequest()

} catch (Exception e)


// The AWS region you will use for sending text messages; this region should be the same region that you moved to production mode (from sandbox) in the initial setting from AWS console.


private AmazonSNSAsync initSnsClient() {

AmazonSNSAsync snsClient =  AmazonSNSAsyncClientBuilder.standard()


return snsClient;


// Sets the max price for sending each text message.


private Map<String, MessageAttributeValue> initSmsAttributes() {

if(smsAttributes == null) {

smsAttributes = new HashMap<String, MessageAttributeValue>();

smsAttributes.put(“AWS.SNS.SMS.MaxPrice”, new MessageAttributeValue()




// Sets the type to Promotional (non-critical) or Transactional (message sent with high priority).


smsAttributes.put(“AWS.SNS.SMS.SMSType”, new MessageAttributeValue()

.withStringValue(“Transactional “))



return smsAttributes;


      1. This code can easily be called from any other class for sending the text message.
      • • Sending MFA code using E-mail

An e-mail can be sent using the helper class JmsTemplate


private void addMessageToQueue(final Email email) {

try {

    jmsTemplate.send(new MessageCreator() {


public Message createMessage(Session session) throws JMSException {

return session.createObjectMessage(email);




catch (Exception e) {

throw e;




Contact us for more help setting up MFA with SNS and Java!

How VMware vCenter UUID Duplicate ID Misconfiguration Can Affect the AWS CloudEndure and Elastic Disaster Recovery Solution

By: Brandon Prasnicki, Technical Architect and Gabriel Zabal, Cloud & Cybersecurity Consultant

What is CloudEndure and Elastic Disaster Recovery (EDR)?

AWS CloudEndure from Amazon Web Services and AWS Elastic Disaster Recovery are both Disaster Recovery solutions that continuously replicate your machines (including operating system, system state configuration, databases, applications, and files) into a low-cost staging area in your target AWS account and preferred Region. While the AWS Elastic Disaster Recovery is the preferred solution moving forward, only CloudEndure was available in the AWS region in the configured Customer AWS region at the time of this particular project and writing of this article.

In the case of a disaster, you can instruct CloudEndure Disaster Recovery to automatically launch thousands of your machines in their fully provisioned state in minutes.

By replicating your machines into a low-cost staging area while still being able to launch fully provisioned machines within minutes, CloudEndure Disaster Recovery can significantly reduce the cost of your disaster recovery infrastructure.

You can use CloudEndure Disaster Recovery and AWS Elastic Disaster Recovery Solution to protect your most critical databases, including Oracle, MySQL, and SQL Server, as well as enterprise applications such as SAP.

The Issue
Recently during a CloudEndure implementation, an odd behavior was observed in the CloudEndure console. The customer source environment was VMware vCenter and when machines would begin replicating and were showing replication status in a list, the servers would suddenly disappear from the list. Then in a short period of time, they would suddenly reappear. While this project was using CloudEndure, it is very likely this issue would also be seen in the new AWS Elastic Disaster Recovery solution as the agent and replication design is very similar.

The CloudEndure console provides Event Logging, and in this case, the logging was very helpful. Below you can see a message stating that the hardware on the machine changed. No hardware changes were anticipated, and certainly not this frequently.

After some internet research, it seemed likely that this behavior was due to duplicate UUIDs in the VMware vCenter environment. The CloudEndure Agent generates an ID upon installation based on the Hardware ID, and in the case of the VMware virtual machines, this ID is based on the UUID. This leads to having different Agents with the same ID.

This issue is likely due to cloning configuration and not instructing VMware cloud director to create new UUIDs when using a vApp Template. This VMware KB mentions that when you use VMware Cloud Director, all virtual machines that are created from a vApp Template receive the same VM UUID.

To confirm, these commands were run on the various hosts:

c:wmic bios get name,serialnumber,version

c:wmic csproduct get name,identifyingnumber,uuid

Once the problem machines were identified, the fix to the CloudEndure agents was as follows:

1.     Uninstall The CloudEndure Agent:

         Windows 64-bit
  • Copy the following folder to a new location:
  • C:Program Files (x86)CloudEnduredist
  • From the new location, run in CMD as an administrator:
  • install_agent_windows.exe –remove
         Windows 32-bit
  • Copy the following folder to a new location:
  • C:Program FilesCloudEnduredist
  • From the new location, run in CMD as an administrator.
  • install_agent_windows.exe –remove
         Uninstalling an Agent from a Linux Target machine:
  • Run as root or with sudo the following commands:
  • /var/lib/cloudendure/
  • /var/lib/cloudendure/install_agent –remove

2.     Re-install the CloudEndure Agent using the steps from the Cloud Endure documentation found here:


After re-installation, verify the ID is a unique value per machine:

For references see:

Query AWS Resources with a Custom Search Command in Splunk

By: Bruce Johnson  | Director, Enterprise Security


Ever wondered what to make of all those resources living in your Cloud environment? Perhaps you’ve asked yourself, “What if I could leverage data from my cloud environment to enhance day-to-day Splunk operations?” The purpose of this blog is to show you how to quickly create a Splunk custom search command that will allow you to query AWS resources on the fly. New capabilities, eh? Well, maybe not so new. Custom search commands have been supported by Splunk for many years now; we’ll simply be shedding some light on how you can easily enhance the functionality of your Splunk environment to meet your business needs.

Splunk custom search commands are “commands that extend SPL to serve your specific needs.” Custom search commands can be Generating, Streaming, Transforming, and Dataset processing, each type serving a different purpose and functionality. For example, Generating search commands fetch data from a given location without performing any additional data processing or transformation. With a Splunk Generating search command, data can be natively fetched from a Splunk index or imported from any external source such as an API. In this blog, we’ll install a custom Splunk Generating Command that interacts with the AWS API to retrieve information about real-time compute and network resources deployed in a cloud environment.

Other types of Splunk search commands:

  • Streaming: process search results one-by-one, applying one transformation to each event that a search returns. A commonly used Splunk streaming command is the eval command.
  • Dataset processing: require the entire dataset in place before the command can run. For example, the sort command is a Dataset processing command that sorts the returned results based on the defined search fields.
  • Transforming: order search results into a data table. For example, chart, top, stats, time chart.

This blog walks you through the steps of installing a custom Splunk Generating command that allow you to query real-time resource information from your AWS cloud environments such as EC2 Instances, EBS Volumes, Security Groups, Network Interfaces, Subnets, and VPCs. This custom Splunk Generating command uses the Splunk SDK to interact with the AWS API and imports data into Splunk for further event processing and correlation. This custom Splunk Generating command can be used to:

  • – Fetch information about existing AWS resources
  • – Create inventory lookups for AWS resources
  • – Correlate external data with Splunk indexed data


  • – AWS account
  • – Splunk Search Head hosted on an EC2 Instance
  • – AWS IAM Role
  • – AWS IAM Policy
  • – Custom Search Command TA

To keep the blog simple, we’ll assume that our readers have a Splunk installation already launched in AWS. This blog will walk through the process of creating the necessary AWS role and policy to ensure the EC2 instance has the required permissions to query AWS resources. Once that’s taken care of, we’ll install the custom search command TA onto the Splunk Search Head.

Create AWS IAM Policy and Role:

  1. Log in to your AWS account via the AWS Management Console.
  2. Go to the IAM service.
  3. Create an IAM Policy with the following AWS permissions.
  4. Create an IAM Role that inherits the IAM policy you created in step three.
  5. Attach the IAM role created to your Splunk Search Head.

Install Custom Search Command on Splunk Search Head

  1. Initialize an SSH session to the Splunk Search Head.
  2. Clone the Custom Search Command repository from GitHub to $SPLUNK_HOME/etc/apps: git clone
  3. Restart Splunk: $SPLUNK_HOME/bin/splunk restart
  4. Search away!

EC2 Instances:



Security Groups:

EBS Volumes:

Network Interfaces:

TA Components:

  • – Defines the AWS Client and required functions to pull AWS resources
  • – Defines the Splunk Generating Command used to pull AWS Resources
  • – splunklib: Splunk SDK Python modules
  • – commands.conf: Defines the “awssearch” SPL command to be used
  • – searchbng.conf: Defines search-assistant configurations for the “awssearch” SPL command

Learn more about Splunk custom search commands. You can find the source code for the Splunk AWS Inventory TA on GitHub.

Contact us for more help on using custom search commands in Splunk!

Maximize WorkDocs Using AWS Lambda Functions

How AWS Lambda Functions Can Be Used to Automate and Advance WorkDocs Document Repository

  By: Courtney Dooley | Technical Architect


Amazon Web Services WorkDocs is a simple document repository that has a user-friendly interface and search capabilities. A key element of WorkDocs is its API library and the ability to extend and automate functionality from most third-party process workflows. Below are three ways Lambda functions can use the WorkDocs API library to automate functionality for Folder Automation, Permissions Management, and Comments & Labels.

Folder Automation

Custom REST APIs can be created using Lambda functions triggered by APIs created in AWS API Gateway. These functions can complete folder and file modifications within WorkDocs. The API calls can create a folder in a specific existing folder or user root folder based on data provided by the API request. Even if data isn’t entirely known, such as the parent folder id, providing an owner-user and folder name can get the parent folder id using the describeUsers and describeFolderContents APIs.

If a folder or file needs to be renamed, it can be updated using the WorkDocs updateDocument and updateFolder APIs. The same functions can be used to move a file or folder from one location to another by specifying an alternative “ParentFolderId” parameter.

Finally, when content is ready for cleanup, a file or folder can be deleted using the deleteFolder, deleteDocument, and deleteFolderContents APIs. These functions move the content to the owner’s recycle bin where it can be restored or permanently deleted by the owner. The retention time limit for all recycle bins can be set via the administration settings within WorkDocs.

Permissions Management

Using the describeUsers WorkDocs API, a specific user can be identified by specifying the user’s login as the “Query” parameter. If the user has not been activated for WorkDocs, that activation can be done using the activateUser WorkDocs API. Once active, that user’s id can be added as an owner, co-owner, contributor, or viewer of any resource by using the addResourcePermissions API. All calls can be completed in the same Lambda function and requested specifying all details at once.

Specific users can be removed from a file or folder by using the removeResourcePermissions, or all permissions can be removed using the removeAllResourcePermissions API. The remove all can be used to start over and add new permissions to the resource as part of a custom “Replace All” functionality.

Users can also be notified of these changes by setting the “Notifications” JSON parameter in the addResourcePermissions. By setting the “EmailMessage” value and “SetEmail” to true within the “Notifications,” an email alert will be sent to the user informing them of the permissions change. Setting “SetEmail” value to false will keep the notification from being sent and the user will not know of the change other than seeing the resource in the “Shared with me” view.

Users can also be deactivated, deleted, created, and modified using the WorkDocs API library to allow for a full range of provisioning via Lambda functions.

Comments & Labels

Although WorkDocs does not have metadata in the traditional sense (at least not at the time of this blog post), labels and comments can be searched, which gives a unique ability to values set using those features.

Comments are available as feedback when viewing documents and are not available on folders, however, labels are not exposed to the user interface at this time and can be set on folders and documents independently.

Lambda functions can be used to get, set, and delete comments or labels on a resource using the below set of API functions:

  • – createComment
  • – deleteComment
  • – describeComments
  • – createLabels
  • – deleteLabels

describeFolder and describeDocument include the labels assigned to that resource. deleteLabels allows for a parameter to “DeleteAll” rather than just a specific label.

All APIs mentioned in this blog are available out of the box with WorkDocs and can be combined to create a custom solution that meets the demands of custom web services, process workflows, and other integrated applications.

AWS offers a robust and expansive set of services to implement custom solutions quickly and easily.

Contact us for more tips and tricks on developing AWS Cloud Solutions!

Don’t Be a Karen: Rebuilding the Terraform State File and Best Practices for Backend State File Storage

  By: Brandon Prasnicki  |  Technical Architect


It happened. It finally happened. After talking to the manager, Contractor Karen quit. She was solely responsible for managing the project’s cloud architecture with Terraform. Now that Karen left, a new resource needs to take her place and continue managing and building the cloud infrastructure. Luckily, the terraform code was in a git repository (excluding the .terraform dir), but no one is sure if it is up to date, and the state file was local to Karen’s machine and not recoverable. What to do now?

  1. Don’t be a Karen. Make it a company policy to configure the backend. A Terraform backend is the configuration on how (and where) to store your Terraform state in a centralized, remote location.
    • – A shared resource account or a production account is a good place to store terraform states.
    • – Having a remote backend is also a must for shared development environments.
  2. Use a versioned bucket. State files can get corrupt, and you may need to revert to an old version of the state file.
  3. Configure the backend. For each unique terraform state, make sure to update the key path to be reflective of the workload architecture the state file is associated with:

If it’s already too late, and you have been victimized by a Karen, then it’s time to rebuild the state file.

  1. Depending on the size of your workload, this will be a time-consuming process.
  2. For each resource, you will need to identify the key needed to import into the state. For this key, reference the terraform documentation. For example:
    • a. For a VPC you would reference this page and see that to import a VPC you would need the VPC ID:
      terraform import aws_vpc.test_vpc vpc-a01106c2
    • b. For an EC2 instance you would reference this page and see that to import an EC2 instance you would need the EC2 instance ID:
      terraform import aws_instance.web i-12345678
  3. After each import, you should run a plan and make sure the plan does not expect any changes you are not anticipating and correct them in the code if applicable. This process will take time.

Contact us for more help on rebuilding Terraform State Files!

Accessing or Restricting Website Content by Geolocation

      By: Stuart Arnett  |  Principal Architect

A public facing website in general is accessible to the entire world. However, there might be times when you want to allow or restrict content based on the geolocation of the user accessing the site. There are several reasons for wanting to do this, but the following are some examples:

  • – Security – restrict access to the website based on country (ex., only United States users can access the website or all countries except China can access the website)
  • – Translations – automatically display the website using the primary language for the country the user is located in
  • – Targeted Content – display different content to the end user, (ex., redirect the user to country specific domains of the website or provide access to different pages)

To perform any of the above scenarios, the web server will need to know the geolocation of the user. This is accomplished by using the user’s IP address. This poses a problem because the location for an IP addresses can change when IP addresses are re-assigned. Luckily for us, MaxMind, Inc. ( provides GeoIP databases and services. The GeoIP databases / services are available in both free and paid for versions. The free versions are not as accurate as the paid for versions and only provide country, city, and ASN databases. We will be focusing on the free version of the GeoIP databases.

MaxMind also provides tools to keep the databases updated and a module for Apache HTTP server to query the databases to determine the user’s country, city, and/or ASN data. We will be setting up both the GeoIP database update tool and configuring Apache HTTP server module in the demo.

Demo Environment

We will be using the following for setting up the demo:

  • Operating System: Amazon Linux 2
  • Web Server: Apache HTTP Server 2.4
  • Geolocation: MaxMind GeoLite2 Databases, MaxMind GeoIP Update, and MaxMind DB Apache Module

MaxMind Account Setup

MaxMind requires you to setup an account and create a license key to access the MaxMind GeoLite2 databases.

  1. Navigate to and setup a new account.
  2. Once your account is set up, navigate to to create a new license key for GeoIP Update versions older than 3.1.1.

Note: Make sure to copy the license key, because it is not accessible after you leave the creation page.

Install and Configure MaxMind

We will now install MaxMind databases, update application, and the Apache HTTP module on Apache Linux 2.

  1. Log into your Apache Linux 2 instance. You will need an account with root access.
  2. Enable the EPEL repository on Amazon Linux 2 to access the MaxMind packages. sudo yum install -y
  3. Install MaxMind databases and development libraries using the following command:
    sudo yum install -y libmaxminddb-devel
  4. Amazon Linux 2 already includes an older version of the MaxMind GeoIP Update package (GeoIP) which will work but does not meet MaxMind’s latest security requirements. A request has been added to the AWS update backlog to update the package. The installed version will be used for this demo, but the configuration below will change when Amazon upgrades to the newer version.
  5. Update the MaxMind GeoIP Update configuration in /etc/GeoIP.conf
    sudo vi /etc/GeoIP.conf
    Update the UserId and LicenseKey information with your MaxMind Account ID and License Key created during your MaxMind account setup above. Update the ProductIds with the values specified below (GeoLite2-ASN GeoLite2-City GeoLite2-Country). The Product IDs are used to determine which databases will be updated. # Enter your license key here
    # customers should insert their license key and user_id
    # free GeoLite users should use 000000000000 as license key
    LicenseKey 000000000000
    # Enter your User ID here ( GeoLite only users should use 999999 as user_id )
    UserId 999999# Enter the Product ID(s) of the database(s) you would like to update
    # By default 106 (MaxMind GeoIP Country) is listed below
    ProductIds GeoLite2-ASN GeoLite2-City GeoLite2-Country
  6. Update the MaxMind GeoIP Databases.
    /usr/bin/geoipupdate -v The -v command line option is used for verbose mode to provide some output when running the update. For future reference, the database files are stored in the “/usr/share/GeoIP/” folder.
  7. MaxMind updates the GeoIP databases on Tuesdays US Eastern time. Configure a crontab job to execute the geoipupdate program on Wednesdays to keep the databases updated with the latest information. crontab -e
    Add the following line to run the job on Wednesdays at 4am UTC.

    Adjust the time accordingly based on the time zone for your server.

    00 04 * * 3 sudo /usr/bin/geoipupdate
  8. Install the MaxMind Apache HTTP Module using the following commands: sudo yum install -y gcc
    sudo yum install -y httpd-devel
    mkdir ~/download/
    cd ~/download
    tar xvzf mod_maxminddb-1.2.0.tar.gz

    cd mod_maxminddb-1.2.0
    sudo make install

    The above commands install the gcc and httpd-devel packages which are required to build the mod_maxminddb Apache HTTP module. The module tarball is downloaded, built, and installed which adds the library to the Apache HTTP modules folder and updates the httpd.conf including the new library.

Install and Configure Apache HTTP Server

We now need to setup and configure Apache HTTP Server.

  1. Log into your Apache Linux 2 instance. You will need an account with root access.
  2. Install Apache HTTP Server using the following command:
    sudo yum install -y httpd
  3. Start the Apache HTTP Server and verify it is working by navigating to your server with a web browser. Make sure the Apache Test page is displayed.
    sudo systemctl start httpd
  4. Create a new httpd configuration file with the MaxMind configuration:
    sudo vi /etc/httpd/conf.d/geolocation.confAdd the following to the file:

    <VirtualHost *:80>
    <IfModule mod_maxminddb.c>

    MaxMindDBEnable On
    MaxMindDBFile ASN_DB /usr/share/GeoIP/GeoLite2-ASN.mmdb
    MaxMindDBFile CITY_DB /usr/share/GeoIP/GeoLite2-City.mmdb
    MaxMindDBFile COUNTRY_DB /usr/share/GeoIP/GeoLite2-Country.mmdb
    MaxMindDBEnv GEOIP_ASN ASN_DB/autonomous_system_number
    MaxMindDBEnv GEOIP_ASORG ASN_DB/autonomous_system_organization# MaxMindDBEnv GEOIP_CONTINENT_CODE CITY_DB/continent/code
    # MaxMindDBEnv GEOIP_CONTINENT_NAME CITY_DB/continent/names/en
    # MaxMindDBEnv GEOIP_COUNTRY_CODE CITY_DB/country/iso_code
    # MaxMindDBEnv GEOIP_COUNTRY_NAME CITY_DB/country/names/en
    MaxMindDBEnv GEOIP_CITY_NAME CITY_DB/city/names/en
    MaxMindDBEnv GEOIP_LONGITUDE CITY_DB/location/longitude
    MaxMindDBEnv GEOIP_LATITUDE CITY_DB/location/latitude
    MaxMindDBEnv GEOIP_CONTINENT_NAME COUNTRY_DB/continent/names/en
    MaxMindDBEnv GEOIP_COUNTRY_CODE COUNTRY_DB/country/iso_code
    MaxMindDBEnv GEOIP_COUNTRY_NAME COUNTRY_DB/country/names/en
    RewriteEngine OnRewriteCond %{REQUEST_URI} !^/geoip.html$
    RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^US$
    RewriteRule .* /geoip.html?continentCode=%{ENV:GEOIP_CONTINENT_CODE}&continent=%

    • – MaxMindDBEnable is used to enable or disable the MaxMind DB Apache Module.

      MaxMindDBEnable On|Off
    • – MaxMindDBFile is used to associate a name to a MaxMind database file on disk.

      MaxMindDBFile <Name> <Filesystem Path to Database File>
    • – MaxMindDBEnv is used to assign a database value to an environment variable. It will look up the database value based on the IP address making the current request.

      MaxMindDBEnv <ENV Variable Name> <Database Name><Path to Data>
    • – Environment variable values are accessed using the following syntax:

      %{ENV:<ENV Variable Name>}
      An Apache rewrite condition (RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^US$) is used in the above configuration to redirect all United States users accessing the website to the geoip.html page. The query parameters appended to the URL are used to display the additional data that is available within the MaxMind GeoLite2 database files and is for reference only.Note: If you are located outside of the United States, please update the country code accordingly to be able to verify the solution is working when you test.
  5. Create HTML file geoip.html

    sudo vi /var/www/html/geoip.html
    Add the following to the file:<h1>Geolocation Test Page</h1>
  6. Restart the Apache HTTP Server to load the new configuration.

    sudo systemctl restart httpd
  7. Navigate to the root page of your website. Assuming your IP is within the United States, you will be redirected to the geoip.html page. Review the query parameters added to the URL to view the additional data that is available within the MaxMind GeoLite2 database files based on your IP address.

You should now have a working website which allows geolocation-based rules to be applied to direct traffic. You can use this as a base and modify the configuration adding any additional rules you require for the geolocation business requirements in your website.

Want to learn more about geolocation rules and exclusions? Contact us today!

Share and Share Alike: Using a Data Model Job Server for Shared Data Model Accelerations

      By: Jon Walthour  |  Senior Splunk Consultant, Team Lead


  • – Common Information Model (CIM)
  • – DBX Health Dashboards
  • – Palo Alto app
  • – Splunk Global Monitoring Console
  • – Infosec
  • – CIM Validator
  • – CIM Usage Dashboards
  • – ArcSight CEF data models add-on
  • – SA-Investigator
  • – Threat hunting

All these Splunk apps and add-ons and many others use data models to power their searches. In order for a data model-powered search to function at peak performance, they are often accelerated. This means that at regular, frequent intervals, the searches that define these data models are run by Splunk and the results are summarized and stored on the indexers. And, because of the design of data models and data model accelerations, this summarized data stored on the indexers is tied to the search head or search head cluster that created it.

So, imagine it: You’re employing many different apps and add-ons in your Splunk deployment that all require these data models. Many times you need the same data models accelerated on several different search heads for different purposes. All these data models on all these search heads running search jobs to maintain and keep their summarized data current. All this summarized data is stored again and again on the indexers, each copy of a bucket’s summary data identical, but tied to a different search head.

In a large distributed deployment with separate search heads or search head clusters for Enterprise Security, IT Service Intelligence, adhoc searching, etc., you end up accelerating these data models everywhere you want to use them—on each search head or search head cluster, on your Monitoring Console instance, on one or more of your heavy forwarders running DB Connect, and more. That’s a lot of duplicate searches consuming CPU and memory on both your search heads and your indexers and duplicate accelerated data-consuming storage on those indexers.

There is a better way, though. Beginning with version 8.0, you can now share data models across instances—run once, use everywhere in your deployment that uses the same indexers. You accelerate the data models as usual on Search Head 1. Then, on Search Head 2, you direct Splunk to use the accelerated data created by the searches run on Search Head 1. You do this in datamodel.conf on Search Head 2 under the stanzas for each of the data models you want to share by adding the setting “acceleration.source_guid” like this:

[<data model name.]
acceleration.source_guid = guid of Search Head 1

You get the GUID from one of two places. If a standalone search head created the accelerated data, the GUID is in $SPLUNK_HOME/etc/instance.cfg. If the accelerated data was created by data model searches run on a search head cluster, you will find the GUID for the cluster in server.conf on any cluster member in the [shclustering] stanza.

That’s it, but there are a few “gotchas” to keep in mind.

First, keep in mind that everything in Splunk exists in the context of an app, also known as a namespace. So, the data models you’re accelerating are defined in the context of an app. Thus, the datamodel.conf you’re going to have on the other search heads with the “acceleration.source_guid” setting must be defined in the same namespace (the same app) as the one in which the data model accelerations are generated on the originating search head.

Second, once you set up this sharing, you cannot edit the data models on the search heads sharing the accelerated data (Search Head 2, in our example above) via Splunk web. You have to set up this sharing via the command line, and you can only edit it via the command line. You will also not be able to rebuild the accelerated data on the sharing search heads for obvious reasons, as they did not build the accelerated data in the first place.

Third, as with all other things in multisite indexer clusters, sharing data model accelerations in multisite indexer clusters gets more complicated. Basically, since the summary data hitches a ride with the primary buckets in a multisite deployment, which end up being spread across the sites, while search heads get “assigned” to particular sites, you want to set “summary_replication” to “true” in the [clustering] stanza in server.conf. This ensures that every searchable copy of a bucket, not just the primary bucket, has a copy of the accelerated data and that searches of summary data are complete. There are other ways to deal with this issue, but I’ve found simply replicating the accelerated data to all searchable copies ensures no missing data and no duplicate data the best.

Finally, when you’re running a tstats search against a shared data model, always use summariesonly=true. Again, this ensures a consistent view of the data as unsummarized data could introduce differing sources and thus incorrect results. One way to address this is to ensure the definition of the indexes that comprise the sources for the data models in the CIM (Common Information Model) add-on are consistent across all the search heads and search head clusters.

And this leads us to the pièce de résistance, the way to take this feature to a whole new level: Install a separate data model acceleration search head built entirely for the purpose of running the data model accelerations. It does nothing else, as in a large deployment, accelerating all the data models will keep it quite busy. Now, this means this search head will need plenty of memory and plenty of CPU cores to ensure the acceleration search jobs run smoothly, quickly, and do not queue up waiting for CPU resources or, worse yet, get skipped altogether. The data models for the entire deployment are managed on this job server. They are all accelerated by this instance and every other search head and search head cluster has a datamodel.conf where all data model stanzas have an “acceleration.source_guid” setting pointing to this data model job search head.

This gives you two big advantages. First, all the other search heads and clusters are freed up to use the accelerated data models without having to expend the resources to maintain them. It separates the maintenance of the data model accelerations from the use of them. Even in an environment where only one search head or search head cluster is utilizing these accelerated data models, this advantage alone can be significant.

So often in busy Enterprise Security implementations, you can encounter significant skipped search ratios because regularly run correlation searches collide with regularly run acceleration jobs and there just aren’t enough host resources to go around. By offloading the acceleration jobs to a separate search head, this risk of data model content loss because of skipped accelerations or missed notable events because of skipped correlation searches is greatly diminished.

Second, since only one instance creates all the data models, there is only one copy of the summary data on the indexers, not multiple duplicate copies for various search heads, saving potentially gigabytes of disk space. And, since the accelerations are only run once on those indexers, indexer resources are freed up to handle more search load.

In the world of medium and large distributed Splunk deployments, Splunk instances get specialized—indexers do indexing, search heads do searching. We also often have specialized instances for the Monitoring Console, the Cluster Manager, the Search Head Cluster Deployer, and complex modular inputs like DBConnect, Splunk Connect for Syslog, and the AWS add-ons. The introduction of Splunk Cloud has brought us the “Inputs Data Manager,” or IDM, instance for these modular inputs. I offer to you that we should add another instance type to this repertoire—the DMA instance to handle all the data model accelerations. No decently-sized Splunk deployment should be without one.

Want to learn more about data model accelerations? Contact us today!

Using Child Playbooks in Splunk Phantom

      By: Joe Wohar  |  Senior Splunk Consultant


Splunk Phantom is an amazing SOAR platform that can really help your SOC automate your incident response processes. It allows you to build playbooks, which are Python scripts under the covers, that will act on security events that have been ingested into the platform. If you have a well-defined process for handling your security events/incidents, you can build a Splunk Phantom playbook to run through that entire process, therefore saving your security analysts time and allow them to work on more serious incidents.

A common occurrence with Splunk Phantom users is that they create a playbook that they want to use in conjunction with other playbooks. For example, a security analyst created three playbooks: a phishing playbook, an unauthorized admin access playbook, and a retrieve user information playbook. In both a phishing event and an unauthorized admin access event, they’d like to retrieve user information. Therefore, the analyst decides to have each of those playbooks call the “retrieve user information playbook” as a child playbook. However, when calling another playbook as a child playbook, there a few gotchas that you need to consider.

Calling Playbooks Synchronously vs. Asynchronously

When adding a playbook block to a playbook, there are only two parameters: Playbook and Synchronous. The Playbook parameter is simple: choose the playbook you’d like to run as a child playbook. The Synchronous option allows you to choose whether or not you’d like to run the child playbook synchronously.

A screenshot from Splunk Phantom showing options for retrieving user information from a playbook.

Choosing “OFF” will cause the child playbook to run asynchronously. This means that the child playbook is called to run on the event the parent playbook is running against, and then the parent playbook continues down the path. If you’ve called the child playbook at the end of the parent playbook, then the parent playbook will finish running and the child playbook will continue running separately.

Choosing “ON” means that the parent playbook will call the child playbook and wait for it to finish running before moving on to the next block. So when a child playbook is called, you have two playbooks running at the same time on the event. This means that every synchronous child playbook is a performance hit to your Splunk Phantom instance. It is best to avoid running child playbooks synchronously unless absolutely necessary due to the performance impact.

Since there are cases where you might need the child playbook to be synchronous, there are a few tips to avoid causing too much of a performance impact.

  1. Keep your child playbooks short and simple. You want your child playbook to finish running quickly so that the parent playbook can resume.
  2. Avoid adding prompts into child playbooks. Prompts wait for a user to take an action. If you put a prompt into a child playbook, the parent playbook has to wait for the child playbook to finish running and the child playbook has to wait for user input.
  3. Avoid using “no op” action blocks from the Phantom app. The “no op” action causes the playbook to wait for a specified number of seconds before moving on to the next block in the path. The “no op” block causes the child playbook to take longer to run, which you usually want to avoid, but there are instances where you may need to run a “no op” action in a child playbook (covered later).
  4. When using multiple synchronous child playbooks, run them in series, not parallel. Running synchronous child playbooks in series ensures that at any given time during the parent playbook’s run, only two playbooks are running at the same time: the parent playbook and one child playbook.

Sending Data Between Parent and Child Playbooks in Splunk Phantom

When calling a child playbook, the only thing that is carried over to the child playbook is the event id number. None of the block outputs from the parent playbook are carried into the child playbook. This creates the problem of how to get data from a parent playbook into a child playbook. There are two main ways of doing this: add the data to a custom list or add data to an artifact in the container. The first option, adding data to a custom list, is a very inconvenient option due to how difficult it is to get data out of a custom list. Also, custom lists are really designed to be a list for checking values against, not storing data to be pulled later.

Adding data to an artifact in the container can be done in two different ways: update an artifact or create a new artifact. Adding data to an artifact is also much easier than adding and updating data in a custom list because there are already actions created to do both tasks in the Phantom app for Phantom: update artifact and add artifact. “Update artifact” will require you to have an artifact id as a reference so it knows which artifact in the container to update. Adding an artifact is simpler because you can always add an artifact, but you can only update an artifact if one exists.

When adding an artifact, there can be a slight delay between the time the action runs and when the artifact is actually added to the container. My advice here is when you add an artifact to a container that you want to pull data from in the child or parent playbook, add a short wait action (you only need it to wait 2 seconds) immediately after the “add artifact” action. You can have the playbook wait by adding a “no op” action block from the Phantom app for Phantom (which you should already have installed if you’re using the add artifact and update artifact actions).

Documentation Tips for Parent and Child Playbooks in Splunk Phantom

When creating a child playbook that you plan to use in multiple parent playbooks, documentation will really help you manage your playbooks in the long run. Here are a couple of quick tips for making your life easier.

  1. Use a naming convention for the child playbooks at least. I’d definitely recommend using a naming convention for all of your playbooks, but if you don’t want to use a naming convention for parent playbooks, at the very least use one for the child playbooks. Adding something like “ – [Child]” will really make it easier to find child playbooks and manage them.
  2. Put the required fields for the child playbook into the playbook’s description. Calling a child playbook is very easy, but if your parent playbook isn’t using the same CEF fields as the child playbook, you’re going to have a problem. Adding this list to the description will help let you know if you need to update your container artifact to add those needed fields or not.

Follow these tips and tricks and you’ll be setting yourself up for a performant and easy-to-manage Splunk Phantom instance for the long term.

Want to learn more about using playbooks in Splunk Phantom? Contact us today!