Forwarder 6.x Compatibility with Splunk 8.0

By: Forrest Lybarger | Splunk Consultant


If you are looking into upgrading Splunk to 8.0, you have probably come across the compatibility matrix for forwarders:



This table means that Splunk does not support, nor has it tested, the use of 6.x forwarders with 8.0 indexers. It doesn’t mean that it is impossible for them to work together. In other words, you can use 6.x forwarders at your own risk. Any problems you have with these forwarders, however, will almost always be caused by the version difference and most likely fixed by upgrading.

With all the caveats out of the way, how do you get this working? Well, it depends on what exact version your forwarders have. Here are the affected versions:

  • 6.0.0 to 6.0.6
  • 6.1.0 to 6.1.4
  • 6.2.0 to 6.2.6
  • 6.3.0 to 6.3.1
  • 6.3.1511.1

The issue is that some older 6.x versions of Splunk use a different SSL protocol from 6.6.x and later versions, which makes them unable to connect via the management port (usually port 8089) and unable to communicate with the deployment server. To correct this, you need to force the newer Splunk components to use an SSL version that the older components can understand. In this case, your forwarders are the only components not upgrading to 8.0, so you only need to fix the deployment server. To avoid issues with these forwarder versions add an app with a server.conf containing this stanza to your deployment server:


sslVersions = *,-ssl2

sslVersionsForClient = *,-ssl2

cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

Allow any sslConfigs apps your environment already has to override this app by giving it a lower priority name or just add the lines from the stanza that aren’t present in your current app. You can delete this new ssl config after your forwarders are upgraded.

This fix should only be used if you must upgrade to 8.0 and can’t wait for your forwarders to upgrade. Keep in mind that this is not Splunk supported, so for now it could work (latest version as of writing this is 8.0.6), but in the future, Splunk could break this workaround. When you do implement this fix, make sure to prioritize upgrading your forwarders and understand that any problems involving data ingestion or forwarding are most likely caused by not upgrading your forwarders to at least 7.0 (latest version possible is recommended).

Want to learn more about forwarder 6.x compatibility with Splunk 8.0? Contact us today!


How Robotic Process Automation Fuels Intelligent Document Processing

By: Troy Allen | VP Cloud Services

RPA, or Robotic Process Automation, is becoming more prevalent in the world of business process automation. In its simplest form, RPA is the practice of utilizing bots or Artificial Intelligence (AI) to analyze information and make recommendations for actions based on that analysis. RPA is not a static set of rules that a process follows. It is an evolving model that learns and adapts by examining recommendations and the actual actions taken and adjusts future actions based on what it has “learned.” In short, RPA is business process automation driven by artificial intelligence that learns as it goes to provide increasingly accurate actions and results.

As with business process automation, RPA can be utilized for many common tasks in an organization. Insurance companies leverage RPA to automate the onboarding of new clients and validate insurance claims to reduce the amount of human-based processing while increasing accuracy. Manufacturing companies utilize RPA to process Bill of Material documents and Purchase Order management. Both examples focus on automatically processing documents and associated information that normally requires a high level of human interaction and decision making.

As an essential part of RPA, Intelligent Document Processing* provides critical insights into the information being processed, helping to make the automation more accurate.

Intelligent Document Processing (IDP), a subset and critical function of RPA, interprets unstructured documents into a recognizable set of information that can be analyzed and acted upon with high levels of confidence.  Using specialized Artificial Intelligence tools, IDP allows for the recognition of critical details and associations that would normally require human review to identify. For example, with forms processing and extracting key information from tables, IDP enables RPA processes to perform highly accurate analysis and actions based on that information.

RPA and Intelligent Document Processing in Action

Onboarding new employees requires a large amount of information to be collected and processed. Typical employers accept and track candidate applications, compensation details, candidate/employee profiles, onboarding documentation, performance management documentation, and various state and federal employee documents. This can result in 15 to 20 documents being processed for each employee being hired.

Imagine an organization that hires thousands of employees for seasonal work; this can result in 2,000 or more documents that have to be reviewed, processed, and acted upon in a very quick timeframe. Considering that a single document may take a human resource worker 2 minutes to review and make critical decisions about, and up to 2 hours per document to complete its process, this can result in over 4,000 hours of processing. Assuming the average resource cost for the participants involved in the processing of new employees is $25 an hour, the company could be looking at over $100,000 just to onboard these new employees! This cost is most likely higher considering that not every candidate is a fit for the role or company and more candidates must be screened, interviewed, and processed to meet hiring goals.

With RPA and IDP, automated processes could be deployed to help minimize the amount of time each processor has to interact with the various documents and the actual process. In many cases, documents can be automatically reviewed, analyzed, categorized, and routed for action based on a well-defined business process. As an example, this can reduce the overall processing of those 2,000 employees from 4,000 hours to 2,000 hours, resulting in a $50,000 reduction in onboarding and hiring costs for seasonal employees.

What are the savings with RPA and Intelligent Document Processing?

As with any process, it takes time to establish, configure, test, and update to make the process as efficient and accurate as possible. This is true with Robotic Process Automation and IDP. Many RPA tools provide a baseline of processes and intelligence based on business processes, but no two organizations operate the same way.

These baseline processes need to be modified to meet specific organizational operations. In many cases, RPA and IDP platforms provide a solid foundation to build upon, which can save significant operational costs right out of the box. RPA and IDP are designed to learn as more information is processed which means that speed and efficiency grow exponentially.

Over time, organizations that utilize RPAs can see upwards of a 40- 50% increase in efficiency and a reduction of 50% or more in processing costs. The following chart outlines the potential return on investment (ROI) of an RPA solution with IDP with 2 automated processes that traditionally take 4 full-time employees 35% of their time to perform with a salary of $55,000 annually per employee:

How to Learn More

Contact us to learn more about our RPA and IDP solution – Intelligent Document Processing (IDP) by TekStream. Through our experience and hundreds of implementations, we help companies streamline business processes and improve decision-making with a structured approach to unstructured content and data.

Using TekStream IDP, organizations can enable their users to quickly process and manage critical business documents, images, forms, video files, and unstructured data from a wide variety of sources. Fill out the form below to see how we can help you understand how Robotic Process Automation and IDP can be leveraged within your organization. You’ll improve your processing efficiency, reduce overhead, and see a return on your RPA investment quickly so you can focus on driving your business to even greater heights of success.

*Intelligent Document Processing may also be referred to as Document Understanding.

TekStream Promoted to Premier Tier in Splunk Partner+ Program

TekStream, an Atlanta-based digital transformation technology firm, announced it has once again achieved Premier Partner status in the Splunk Partner+ Program.

In order to achieve Premier Partner status, partners must achieve $2 million in sales over the past 12 months and staff accreditations commensurate with the tier. With its Premier status, TekStream’s Splunk customers benefit from an enhanced level of engagement, commitment, and support.

By including TekStream in its Premier Partner Tier, Splunk has recognized TekStream for its outstanding achievement and commitment to Splunk market development, strategic prioritization, and customer success.

“I’m proud of our team and the hard work they’ve put in to achieve this accomplishment.  It’s especially impressive considering the circumstances we’ve all endured this year.  I’m excited about the momentum this creates heading into 2021 for our team, our customers, and Splunk” said Matthew Clemmons, Managing Director of the Splunk practice at TekStream.

About TekStream
TekStream accelerates clients’ digital transformation by navigating complex technology environments with a combination of technical expertise and staffing solutions. We guide clients’ decisions, quickly implement the right technologies with the right people, and keep them running for sustainable growth. Our battle-tested processes and methodology help companies with legacy systems get to the cloud faster, so they can be agile, reduce costs, and improve operational efficiencies. And with 100s of deployments under our belt, we can guarantee on-time and on-budget project delivery. That’s why 97% of clients are repeat customers. For more information visit

OCI DR in the Cloud

Business Continuance via Disaster Recovery is an essential element of IT and takes on many forms. The high end consists of high availability solutions that provide real-time replication of systems. While these systems provide seamless continuity during outages they are large, complex, and expensive, justifiable to support only the most critical business applications. At the other end of the continuum, however, Disaster Recovery is little more than tape backup or backup to NAS which have complicated and lengthy restore procedures which take hours or days.
A major improvement can be made in disaster recovery with a solution that provides business continuity in a model that simply extends the existing IT architecture into the Cloud.

Rackware RMM Migration/DR platform is a non-intrusive Agentless Technology with pre- and post- Migration Configuration Capabilities that is easy to set up and configure for complicated enterprise environments/applications. Rackware RMM supports both Linux and Windows-based workloads for migration to the Oracle Cloud Infrastructure.

RackWare RMM platform provides a flexible and all-encompassing solution for Migration and disaster recovery. RackWare helps Enterprises and large Organizations take advantage of the agility promised by Oracle Cloud Infrastructure. Rackware’s platform eliminates the complexity of protecting, moving, and managing large-scale applications, including critical business applications and their workloads into the Oracle Cloud. It is now possible for enterprise customers to forgo the upfront purchase of duplicate recovery hardware, the cost of set up, configuring, and maintaining that hardware by leveraging Oracle cloud infrastructure.

Rackware RMM provides the following value proposition for enterprises in the Oracle Cloud:

  • Non-disruptive / Live Captures -No agents installed, safe and secure replication of your production environments
  • Network and Application Discovery – Automatically discover network configurations and applications allowing you to reconfigure them in the OCI environment during migration
  • Universal DR Protection – RackWare support spans all physical and virtual confluences, even for complex environments with Large SQL Clusters, and Network Attached Storage
  • Seamless Failback –  To physical and virtual environments, for simple disaster recovery drills
  • Cost Reduction – Orchestration engine for multiple polices of RPOs and RTOs based on tolerance to reduce costs with less expensive compute, network, and storage utilization.

Storage Methods

There are 2 storage methods available for Disaster Recovery.

Store and Forward

Store and Forward will create an image of your source workload in storage on the RMM’s database. When using this method, the RMM will need a datastore capable of containing the amount of used data from each source hosts minus typical compression savings.

Store and Forward is required if using the auto-provision feature whereby the RMM will only provision the compute resources during a DR event or test/drill event or to offer the multi-stage protection of having data protected by a stored image and then synced from stored image to target compute resources.


RMM does not store a copy of the used data from source hosts. The RMM acts as a passthrough proxy to sync the source workload data through itself and onto the target DR instances.

How it works

RMM provides a DR solution that builds on its image mobility and elasticity features to bring economic DR to enterprises. The building blocks of RackWare’s DR solution include onboarding, cloud bursting and the policy framework to automate necessary functions. Captured images from production (origin) instances are cloned and pushed out to a local or remote DR site. Changes in production images are periodically synchronized with the remote images, keeping the original host Image and the DR image in sync. In the event of an outage at the origin site, the up to date image at the DR site can assume operations through RackWare’s fail-over mechanism.

After the production instance is repaired and operational, it’s easy to restore the origin site to any up any changes made to the CloudImage in the cloud. When the origin site is restored to its operational state, the administrator can utilize the capture from cloud feature to refresh the original Image and synchronize any changes that occurred during the outage.

Overhead on the origin Host is extremely small involving only resources to take a delta snapshot. Thus the data overhead of the WAN link incurs only the delta of information, keeping bandwidth needs and sync time to a minimum. It’s important that Image updates include user data, Operating System updates, and application installations and configuration changes so that the recovery image behaves exactly like the production image should a failover occur. The cloud DR feature supports all of these. While OS updates are more infrequent it is still important to ensure that kernel patches are kept in sync with the DR Image. When updating the OS, an image refresh operation is done from the RMM first before the sync to the CloudImage. Should the production system be compromised or inoperable, the CloudImage is automatically launched and is running with the latest synchronized changes.

Oracle & Rackware partnership provides a seamless experience to Migrate to the Oracle Cloud Infrastructure and secure customer workloads with dynamic provisioning and disaster recovery.

About TekStream
TekStream accelerates clients’ digital transformation by navigating complex technology environments with a combination of technical expertise and staffing solutions. We guide clients’ decisions, quickly implement the right technologies with the right people, and keep them running for sustainable growth. Our battle-tested processes and methodology help companies with legacy systems get to the cloud faster, so they can be agile, reduce costs, and improve operational efficiencies. And with 100s of deployments under our belt, we can guarantee on-time and on-budget project delivery. That’s why 97% of clients are repeat customers. For more information visit

TekStream Helps to Support the Launch of Professional Services in AWS Marketplace

TekStream, a digital transformation company and Amazon Web Services (AWS) Advanced Consulting Partner, announced today that it is participating in the launch of Professional Services in AWS Marketplace. AWS customers can now find and purchase professional services from TekStream in AWS Marketplace, a curated digital catalog of software, data, and services that makes it easy to find, test, buy, and deploy software and data products that run on AWS. As a participant in the launch, TekStream is one of the first AWS Consulting Partners to quote and contract services in AWS Marketplace to help customers implement, support, and manage their software on AWS. Click here for more information.

As organizations migrate to the cloud, they want to use their preferred software solutions on AWS. AWS customers often rely on professional services from TekStream to implement, migrate, support, and manage their software in the cloud. Until now, AWS customers had to find and contract professional services outside of AWS Marketplace and could not identify software and associated services in a single procurement experience. With professional services from TekStream available in AWS Marketplace, customers have a simplified way to purchase and be billed for both software and related services in a centralized place. Customers can further streamline their purchase of software with standard contract terms to simplify and accelerate procurement cycles.

“TekStream views AWS Marketplace as a strategic channel for our services to be discovered and procured,” said Judd Robins, Executive Vice President. “Complete solutions generally have a technology and a human component to make them work successfully. AWS Marketplace has always been a great catalog of technical solutions. With the addition of Professional Services in AWS Marketplace, customers now have a broader range of options to get those solutions launched and managed.”

• Database Migration QuickStart – Jumpstart your Database migration to AWS with a 1-week process to analyze and assess Oracle, Microsoft, and open-source database migrations to AWS purpose-built database solutions.
• Splunk Cloud QuickStart – Get your Top 3 IT Operations and/or Security use cases implemented leveraging Splunk with 2 weeks of services, training, and 3 months of go-live support provided by TekStream.
• Splunk CMMC QuickStart – a practical, proven, and effective solution to get you compliant in under 30 days.
• Oracle License Optimization Plan – 1 week to analyze and assess your Oracle licenses and contracts to reduce costs – paving your way to Database Freedom on AWS.
• CloudEndure Cloud Migration QuickStart – 1 week to Migrate Development, QA, or Testing On-Premise Workload to AWS
• CloudEndure Cloud Disaster Recover QuickStart – 1 week to implement and test disaster recovery for up to 3 on-premise workloads to AWS

TekStream accelerates clients’ digital transformation by navigating complex technology environments with a combination of technical expertise and staffing solutions. We guide clients’ decisions, quickly implement the right technologies with the right people, and keep them running for sustainable growth. Our battle-tested processes and methodology help companies with legacy systems get to the cloud faster, so they can be agile, reduce costs, and improve operational efficiencies. And with 100s of deployments under our belt, we can guarantee on-time and on-budget project delivery. That’s why 97% of clients are repeat customers.

Integrating Oracle Cloud ERP (Cloud Fusion) With External Applications Leveraging Oracle BI Reports

By: Greg Moler | Director of Imaging Solutions


Does your organization utilize Oracle Fusion Applications such as Oracle Cloud ERP, Cloud Human Capital Management, and Project Portfolio Management cloud?  Are you looking for ways to extend the functionality of these applications?  These platforms store a lot of critical business data, much of which has the potential to be integrated with other third-party applications.  In this article, we explore use cases and considerations for leveraging Oracle BI reports to integrate Oracle Fusion data with external applications.


What are Oracle BI reports?

Oracle’s Fusion Applications including Cloud ERP, Cloud Human Capital Management, and Project Portfolio Management cloud as well as many others, use the Oracle Business Intelligence (BI) platform to provide reports and analytics.  BI reports provide a way to query and report against the underlying data in these platforms.  In the cloud platform, these take the place of traditional database queries.  They can be used in a variety of different functions to extract data for use with external applications.


Use Cases

When it comes to potential use cases for extending your Oracle Fusion application’s data, the possibilities are endless.  If you can think of a need, we can design a way to build it.  In this section, we take a glimpse at some specific use cases for integrating data from Oracle Cloud ERP.

  • Automating GL/Project Account Coding: Manually coding invoices can be time-consuming for AP personnel. This process can be streamlined by automating business logic to programmatically apply GL and Project codes to invoices.  For example, often one attribute on the invoices drives the rest of the coding.  Another common scenario is vendor distribution sets, where each vendor has a pre-defined set of charge account strings and percentages that get applied to each invoice.  BI reports can be structured to take input from your 3rd party application such as vendor id and return coding data.
  • Workflow routing and Approval hierarchy: If your application does workflow routing or approval hierarchy, it will need to know which user’s documents should be assigned to. Often, this is based on data on the document and can be retrieved from Cloud ERP.  BI reports can be designed to return the appropriate workflow assignee based on the pertinent ERP data.  If your organization also utilizes Oracle Human Capital Management (HCM), you can extend functionality even further by directly accessing employee data in the approval hierarchy.
  • Vendor data: Vendor data stored in Oracle Cloud ERP can be incredibly useful in a variety of different scenarios. For example, vendor identification on invoices, purchase orders, and other documents.  Another common use case is the use of vendor-specific attributes or descriptive flex fields (dffs) defined in Cloud ERP.  These attributes can be easily maintained by the business in the Fusion interface and leveraged in an external application to drive automation such as account coding or workflow routing as described previously.
  • Validation data: More than likely, your application will need to be able to validate that the data on the document or object is correct before allowing it to proceed. The information needed to perform this validation is typically stored in Cloud ERP.  For example, validating whether a purchase order had enough open balance available for a line item.  BI reports provide a way to perform these validations, simply by taking input parameters from the external application and returning corresponding data from cloud Fusion.
  • Reporting data: Reporting options within Fusion Applications and Oracle Business Intelligence platform can be limited. These reporting capabilities can be supplemented by an external reporting tool.  BI reports provide a way to extract data out of Oracle for analysis and reporting in an outside application.


 Considerations when creating Oracle BI Reports:

When designing and creating Oracle BI reports for integration with an external application, there are many considerations including:

  • Data Structure: Design a reusable format for your BI reports so that data is returned in a consistent structure. This will simplify integration points with the reports.
  • Data return type: What return data type will your application require? XML? CSV?
  • Catalog folder structure: Consider how to organize your catalog folder structure. Which reports function together and should be grouped together?  Which reports should be promoted together between environments?  The easiest way to migrate reports is to use the Archive/Unarchive function on an entire folder.  It is important to consider which reports should be archived together.
  • Testing options: The most basic way to test the BI reports is through the Fusion Catalog Manger. This provides a good baseline test.  More than likely, the reports will be called via web service, so the next layer of testing should be done using a tool such as SOAP UI that will allow you to test calling web services.  This will allow you to call the web service and view responses as they will be received by the external application.  Keep in mind that all responses must be decoded before they can be consumed.


Interested in getting more out of your Cloud ERP data?  Contact TekStream to learn about how we can assist with your Cloud Fusion integration needs!

Working with Multivalue Fields in Splunk

By: Yetunde Awojoodu | Splunk Consultant


Have you ever come across fields with multiple values in your event data in Splunk and wondered how to modify them to get the results you need? Each field in an event typically has a single value, but for events such as email logs you can often find multiple values in the “To” and “Cc” fields. Multivalue fields can also result from data augmentation using lookups. To properly evaluate and modify multivalue fields, Splunk has some multivalue search commands and functions. If you ignore multivalue fields in your data, you may end up with missing and inaccurate data, sometimes reporting only the first value of the multivalue field(s) in your results.

In this article, I have applied a simple scenario to illustrate how different multivalue commands and functions can be used individually or combined to meet different use cases. I will cover some common search commands and functions that work with multivalue fields. Note that multivalue functions can be used with eval, where or fieldformat search commands. In my illustrations, I employed the “makeresults” command to generate hypothetical data for my searches so that anyone can recreate them without the need to onboard data. Read more on the makeresults command.



Within one purchase transaction, Mary bought eggs, milk and bread. She paid for the eggs with cash and covered the remaining items using her credit card. We can assume that this purchase transaction is equivalent to a log event. The values for each multivalue field are separated by the comma delimiter.

Example 1:

Please note that in all the results, I have deliberately excluded the default field, “_time” which is a default field generated when the makeresults command is used.


Makemv (Command)

This command is used to split the values of a field that appear like a single value into multiple values within an event based on the delimiter. A delimiter specifies the boundary between characters.

Example 2:

The values in the “groceries” field have been split within the same event based on the comma delimiter. The values in the “payment” field remain the same. The report shows the method of payment for all three grocery items but it does not specify the actual payment method used for each item. To expand the event into three separate events, one for each item and show the exact payment for each grocery item, we will need a combination of commands and functions.


Mvzip (Function)

The mvzip function is used to tie corresponding values in the different fields of an event together. This helps to keep the association among the field values. This function takes two multivalue fields, X and Y, and combines them by stitching together the first value of X with the first value of field Y, then the second with the second, and so on.

Example 3:

The new field, “zipped” is the result of the mvzip function. The values of the groceries and payment fields are properly zipped together before expanding into separate events. Note that at this point, the results are still within one event.


Mvexpand (Command)

This command expands the values of a multivalue field into separate events, one event for each value in the multivalue field. All other single field values and unexpanded multivalue field values will remain the same in each new event.

Example 4:

Mvexpand works great at splitting the values of a multivalue field into multiple events while keeping other field values in the event as is but it only works on one multivalue field at a time. For instance, in the above example, mvexpand cannot be used to split both “zipped” and “payment” fields at the same time. The next function will come in handy to accomplish this.


Mvindex (Function)

Having zipped the values and created one field, “zipped”, you can now expand the “zipped” field into multiple events. The mvindex function is a little more intricate. To further tie field values together so that accurate associations are made in the process of expanding the values into separate events, mvindex will separate the existing multivalued field into two chosen fields using index values. Indexes can start at zero if labeling from the first value. For example, if values= a,e,i,o,u; a=0 e=1 i=2 o=3 u=4. You could also label from the last character with -1; a=-5 e=-4 i=-3 o=-2 u=-1 or you could choose to have a combination of both index patterns; a=0 e=1 i=2 o=-2 u=-1.

Example 5:

Mvindex was used to assign index 0 to the first value in the group which represents groceries and index 1 to the second value representing payment method so that when the fields are split, the values will not get mixed up. The “split” command was used to separate the values on the comma delimiter. Using mvindex and split functions, the values are now separated into one value per event and the values correspond correctly.

Tip – The stats command can also be used in place of mvexpand to split the fields into separate events as shown below:

Example 6:


Mvcount (Function)

This function can be used to quickly determine the number of values in a multivalue field using the delimiter. If the field contains a single value, the function returns 1 and if the field has no values, the function returns NULL.

Example 7:

As with single value fields, keep in mind that you may need a combination of multivalue commands/functions to get your report in the required format that will meet your specific use case.

Note: If there are situations in your data where a field is sometimes multivalue and other times null, refer here


Want to learn more about working with multivalue fields in Splunk? Contact us today!


A Use Case for Ingest Time Eval

By: Zubair Rauf | Senior Splunk Consultant


A few days ago, I came across an interesting challenge that a customer put in front of me. They had been facing this for some time now. The customer works with an app that logs all of its events 7 hours ahead of Eastern time, irrespective of daylight savings time. The server clock reset to midnight when Eastern time was 5:00 PM all year round. To work around this problem and make sure the events were always synced with the correct time zone, they adjusted the sourcetype for those logs every time daylight savings time started or ended.

When presented with this problem, I spent a good amount of time to find a time zone that would change with eastern time when daylight savings time changed and have the same time offset as those logs. Not having any success on that front, I started looking at alternatives to help my customer overcome their issue and I came across this powerful way to solve the problem with a one-time fix with the sourcetype.

Splunk introduced Ingest time evals with Splunk Enterprise 7.2. Ingest time evals are similar to search time evals that have helped Splunk be the powerful tool that it always has been. Ingest time evals allow you to write an EVAL formula that is executed at ingestion time to create a new indexed field or to update a field’s value. They give you more control over Splunk index time fields as well. In my particular case, having control over and being able to manipulate index time fields helped me just do the trick for my customer.

For starters, _time is an index time field that is parsed from the raw log event. If the event does not have a time, the indexer will assign it with a current time when the event is ingested. In my particular challenge, the _time field needed a fixed offset by five hours as it was five hours ahead of eastern time.

To setup ingest time evals, we have to work with transforms.conf, props.conf, and fields.conf (only if creating new fields at ingest time). To further elaborate on the process of setting up ingest time evals to create new index time fields or manipulate existing fields at index time, we have used a sample log from a Cisco device.

To do a comparison, I ingested the log file with a custom sourcetype I created to parse the events.

With the above sourcetype, the following events were ingested.

If you look closely, the date/time was parsed exactly as it appears in the raw log event. Now if the raw event had a timestamp that needed to be offset, we could change the _time field at ingest time using ingest time eval.

To make my required changes, I will have to add an INGEST_EVAL expression in a transforms stanza in transforms.conf to update the _time field at ingest time after it has been parsed out from the actual event.

In the above example, I have used INGEST_EVAL to update my _time field to add 7200 seconds to it. This translates into 2 hours. I have also used the “:=” instead of “=” so that Splunk updates the _time field and not create another _time value resulting in a multivalued _time field in the final event. In this case, “:=” will overwrite the existing value in the field.

The above screenshot shows the updated _time field after the same log file has been ingested with the updated props and transforms. If you closely look at the Time column in the above screenshot in the first event it shows the timestamp being parsed as 01/16/20 1:43:43 PM but the timestamp in the event is 01/16/2020 11:43:31 AM. This tells us that the INGEST_EVAL expression in our transforms.conf successfully worked.

At this point, I would caution you to thoroughly test your INGEST_EVAL on a dev Splunk server so that you are sure that your eval works.

Ingest time eval can also be used to create new index time fields. While updating the _time field to offset the time difference, I thought about creating some custom index fields for demonstration purposes. This would further demonstrate how powerful ingest time evals are and how they can be useful.

Considering I was updating the _time field with my new timestamp, I figured it would be good to have a field that still parses and stores the original time. I named that field orig_time. This field is basically derived from the original _time field that was parsed before it was changed into the new timestamp.

I also thought it would be good to calculate the raw length of the event at ingest time, as that would create a field for me to calculate the size of the ingested data later. I particularly leaned towards demonstrating this, because not too long ago, I was also faced with the challenge to report host-level licensing information for every index. This helps Splunk users in an organization understand how much data their hosts are sending to Splunk.

Now, this is an easy fix if your environment is small. In that case, you can use the license_usage.log file available in the _internal index to calculate your license usage by index, sourcetype, source, or host. It definitely does become a problem when your environment grows too large. When the unique tuples cross 2000 by default, the license manager starts squashing source/host values and only index, sourcetype values remain in license_usage.log.

To work with this issue, I set up a daily license usage search which calculates the length of _raw for the past day for all the indexes and stores it in a summary index. This search runs at off-peak hours when the system is not being used by other users. That helps me populate my dashboards on demand for the users who want to see this data the next day.

Having raw event size calculated for every event at index size will definitely help me rid myself of those expensive searches that need to be run every night, these searches can be less reliable in case the search head that runs the summary generating search crashes. At index time I create a new field “event_size” using INGEST_EVAL in transforms.conf. The settings used to do this are as below;

If you look closely at the settings;


I have added two new stanzas to the transforms.conf to create the evals for the new fields, orig_time, and event_size.


As we are creating two new fields at ingest time, we add their names as stanzas in fields.conf and make sure these fields are indexed by adding the parameter “INDEXED = true”.


I have updated the TRANSFORMS parameter in the relevant sourcetype. If you notice, the order of the TRANSFORMS stanza actually dictates which transform will be applied first to the data being parsed. In this particular case, the stanza is:

TRANSFORMS = orig-time,time-offset,event-size

In this specific transform the order will be as follows:

  • orig-time will preserve the original parsed time into the orig_time
  • time-offset will update the existing _time field to be offset by 02 hours.
  • event-size will calculate the total length of the event and create a new event_size field.

If you look at the final screenshot (above) closely on the left under “Interesting Fields” you will see that there are two new fields that you can see. These include orig_time and event_size

Now to calculate the total license usage by any measure, you can use your event_size with a | tstats search which will be many folds faster than a regular search.

There can be many other uses for Ingest Time Evals, one of which is listed on the documents page. To find out more, please visit Splunk documentation at


If you want to learn more or have TekStream help with implementing some Splunk use cases, contact us today!

Splunk Phantom Workbooks

By: Joe Wohar | Senior Splunk Consultant


Splunk Phantom is an amazing software used to automate cybersecurity processes, however, many companies do not know that they could also be using Phantom for case management. Arguably the most powerful, yet unknown to many, case management feature of Phantom is the ability to create and use workbooks.

If you’re familiar with Phantom, then you know that Phantom playbooks are repeatable processes that Phantom runs through against events. Phantom workbooks are repeatable, defined processes that analysts run through against events. However, they’re typically only used when an analyst needs to get involved. When an event is determined to be a threat that needs to be investigated by an analyst, the event should be promoted to a case. This can be done with a manual change done on the event (by clicking the toolbox icon button) or by having the conditions specified in a playbook that can then turn the event into a case.

Image 1: A workbook must be selected when converting an event to a case.


One of the biggest advantages of workbooks is that it’s a great way of ensuring that your analysts (new or old) are following the same set of steps when working cases. SOPs help define processes for your analysts to follow, but workbooks put those processes right into the case and make the work easily trackable. Workbooks are made up of 2 trackable components: phases and tasks.



Phases split the investigation into different sections, such as identification, acquisition, analysis, and reporting. Individual SLAs can be set for each phase of a workbook. When SLAs are missed/breached, there is a panel on the Phantom home page for tracking that:

Image 2: Home page SLA breach tracker.


Phases are made up of tasks, which are where the specific steps for investigations are listed.

Image 3: Adding new phases/tasks to a workbook.



Tasks are very customizable, so they can be pretty general with few trackable requirements or be very specific with many tracked steps. First, tasks can have a default owner assigned to them, which could be useful if you want to have a “review” task so that a more experienced analyst can review a newer analyst’s work, however, I think most often you’d want to leave that blank so that tasks can be assigned to the analyst working the case. The description section of the task is where you can describe the specific things that should be done in the task. If you don’t want to track specific steps, you can simply use this section to create a list of steps for analysts to follow. However, if you have very specific steps involved in a task, you may want to use the description just for describing the process and then have the steps listed as actions or playbooks. This brings us to the next part of tasks, adding actions and playbooks.

Actions and playbooks are Phantom automation being added to the human process. The actions and playbooks added to a task are limited to the actions available in your configured apps and the playbooks that you have available in your Phantom instance. Then, when an analyst goes to run the action from the investigation screen, the action is already pulled up and they just need to enter the details.

Image 4: Workbook opened in a case with 2 tasks.


Image 5: Pop-up window from clicking the “run query” action in the workbook.


Running a playbook from a workbook is even simpler. Just click the playbook and click the “Run Playbook” button.

Image 6: Pop-up window from clicking the “Disabled User” playbook in the workbook.


As analysts move through the tasks and complete them, the phase’s tracker will be updated to show completion and whether or not tasks were completed on time and if the phase was completed on time.

Images 7 & 8: First task complete and then both tasks completed.


If you’re not using Phantom for case management, then you’re likely using Phantom to create tickets and add details to them in another software, which is costing you more in hardware and licensing. By using Phantom for case management, you’ll save the cost of another software and its hardware while using software you’ve already bought at no additional cost.

Not sure how to get started with workbooks? Try taking one of your best defined SOPs and make a workbook for it. If you’re not currently a Phantom customer and would like to try it out, you can download the OVA by registering here:


Want to learn more about Phantom workbooks? Contact us today!