Effective Use of Splunk and AWS in the Time of Coronavirus

By: Bruce Johnson | Director, Enterprise Security

Firstly, be safe and be well. The TekStream family has found itself pulling together in ways that transcend remote conference calls and we hope that your respective organizations are able to do the same. We feel very privileged to be in the Splunk ecosystem as uses for Splunk technology are becoming ever more immediate.

To that end, we have seen all of our customers putting emphasis on monitoring remote access. Was any company sizing their network for virtualizing their entire ecosystem overnight? Network access points were sized for pre-determined traffic profiles leveraging pre-determined bandwidth levels for remote access. Those network appliances were configured to support predictable traffic volumes from occasionally remote workers, they weren’t designed to support 100% of all internal access traffic. The impact to operational monitoring of services supporting remote users became the most critical part of your infrastructure overnight.

Likewise, what you were monitoring for security just got hidden in a cloud of chaff. The changes to network traffic have opened you up to new threats that demand immediate attention.

Security Impact

There are several new areas of concern in the context of the current climate:

Your threat surface has changed

Anomalies relative to RDP sessions or escalation of privileges for remotely logged in users used to be a smaller percentage of traffic and might have figured into evaluating potential threat risk. Obviously that is no longer the case. If you’re able to segregate traffic for access to critical systems from traffic that simply needs to be routed or tunneled to other public cloud-provided applications, that would help cut down on the traffic that needs to be monitored but that will require changes to network monitoring and Splunk searches.

Your policies and processes need to be reviewed and revised

Have you published security standards for home networks for remote workers? Do you have policies relative to working in public networks? Do you have adequate personal firewalls in place or standard implementations for users wanting to implement security add-ons for their home networks or work-provided laptops?

Some employees might now be faced with working on home networks which are not adequate to the bandwidth needs of video conferencing and may opt to work from shared public access points (although they might have to make due with working from the Starbucks parking lot as internal access is prohibited). Many do not have secure wireless access points or firewalls on their home networks. Publishing links to your employees on how to implement additional wi-fi security and/or products that are supported for additional security, as well as how to ensure access to critical systems through supported VPN/MFA methods is worth doing even if you have done it before. There is also the potential expansion of access to include personal devices in addition to company-owned devices. They will need to have the same level of security, and you will also need to consider the privacy implications of employee-owned devices connecting to your business network.

Likewise, help desk resources in support of these efforts as well as level1 security analysts monitoring this type of activity might need to be shifted or expanded.

New threats have emerged

Hackers don’t take the day off because they have to work from home and there are several creative threats that take advantage of Coronavirus panic. Hackers are nothing if not nimble. There are several well-publicized attacks which seek to take advantage of users anxious for more information on the progress of the pandemic. The World Health Organization (WHO) and the U.S. Federal Trade Commission (FTC) have publicly warned about impersonators. Thousands of domains are getting registered every day in support of Coronavirus related phishing attacks. Some of them are even targeting hospitals, which takes “unethical” hacking to a brand new low. Additionally, there are new threat lists to consider, for example, RiskIQ is publishing a list of rapidly expanding domains relative to coronavirus.

Stepping up the normal Splunk monitoring for those domains, moving up plans to augment email filtering, setting up a mailbox that Splunk ingests for reported attacks that can be easily forwarded from end-users that suspect a phishing email, or augmenting your Phantom SOAR implementation to highlight automated response to specific phishing attacks are all appropriate in that context.

Operational impact


VPN Monitoring

If you are not currently monitoring VPN usage in Splunk it is relatively straightforward to implement VPN/Firewall data sources and to begin monitoring utilization and health from those appliances. It is useful to monitor network traffic as a whole relative to VPN bandwidth as well as the normal CPU/memory metrics coming from those appliances directly.

If you’re already monitoring VPN traffic and likely you are if you have Splunk, at the very least, you need to alter your thresholds for what constitutes an alert or an anomaly.

The following are examples of dashboards we’ve built to monitor VPN related firewall traffic as well as cpu/memory:

In addition to straightforward monitoring of the environment, expect troubleshooting tickets to increase. Detailed metrics relative to the connectivity errors might need to be monitored more closely or events might be expanded to make troubleshooting more efficient. Below is an example of Palo Alto Splunk dashboards that track VPN errors:

There are several out of the box applications from Splunk for VPN / NGFW sources including but not limited to:

Palo Alto: Includes firewall data that monitors bandwidth across key links. Additionally, Global protect VPN monitoring can help customers with troubleshooting remote access. https://splunkbase.splunk.com/app/491/

Zscaler: Provides visibility into remote access, no matter where the users are connecting from.https://splunkbase.splunk.com/app/3866/

Cisco: Provides equivalent functionality to populate dashboards around remote access and bandwidth on key links.  https://splunkbase.splunk.com/app/1620/

Fortinet: Provides ability to ingest Fortigate Fortinet traffic  https://splunkbase.splunk.com/app/2846/

Nagios: Monitors the network for problems caused by overloaded data links or network connections, also monitors routers, switches and more. https://splunkbase.splunk.com/app/2703/

One of the techniques to consider in response to this spike in volume is to split network traffic on your VPNs to segregate priority or sensitive traffic from traffic that you can pass through to external applications.


Split tunneling can be used to route traffic and it’s being recommended by Microsoft for O365 access. This also effects how VPN traffic and threats are monitored through established tunnels. Obviously, the traffic to internal critical infrastructure and applications would be the priority and all externally routed traffic could be, if not ignored, at least de-prioritized.

Additionally, MFA applications fall into much the same category as monitoring of VPN sources and the same types of use cases apply to monitoring those sources. Below are a subset of relevant application links.

RSA Multifactor Authenticationhttps://splunkbase.splunk.com/app/2958/

Duo Multifactor Authenticationhttps://splunkbase.splunk.com/app/3504/

Okta Multifactor Authentication: https://splunkbase.splunk.com/app/2806/


If you’re familiar with Splunk, you already know that it is typically only a few hours of effort to onboard a new data source and begin leveraging it in the context of searches, dashboards, and alerts.

Engaged Workers

There are some people that are focused in an office environment, then there are the people that work at home with one cup of coffee that can fuel them until they are dragged away from their laptop, then there are the people that have way to much to do around the house to bother with work. It’s nice to know whether people are actually plugged in. A whole new demand for monitoring remote productivity fueling new solution offerings from Splunk. They have developed a set of dashboards under the guise of Remote Work Insights.

Of course, monitoring your VPN and conferencing software is just the beginning and there are a plethora of sources that might be monitored to measure productivity. Often those sources vary by team and responsibilities. The power inherent in Splunk is that each team can be monitored individuality with different measures and aggregated into composite team views at multiple levels, similar to ITSI monitoring of infrastructure layers and components. We are finding a great deal of opportunity in this area and it is expected to be a set of techniques and solutions that will persist well beyond the shared immediate challenges of Coronavirus.

A related use case for VPN monitoring is to track login and logout to confirm that people are actually logging in rather than social distancing on the golf course, but this use case has been less common in practice.

Migrate VPN services to the cloud

Ultimately, when faced with dynamic scaling and provisioning problems, the cloud is your answer. If your VPN infrastructure is taxed, the traffic is now completely unpredictable, and there is no way to scale up your network appliances in the short term, consider moving VPN services to cloud connectivity points. You can move network security to the cloud and consume it just like any other SaaS application. This has the advantage of being instantly scalable up and down (once normal operations resume) as well as being secure. Implementation can be done in parallel to your existing VPN network-based solutions. Virtualizing VPN in AWS is relatively straightforward and it’s certainly something TekStream can help you to accomplish in short order. It has the advantage of scaling and doing so temporarily. There are a variety of options to consider.

AWS Marketplace has VPN appliances you can deploy immediately. This is a good approach if you are already using a commercial-grade VPN like a Cisco ASA or Palo Alto. This will have the least impact on existing users since they can continue to use the same client, just point their connection to a new hostname or IP but it can be a bit pricey.  Some examples of commercial options from the AWS Marketplace are:

Cisco ASA: https://aws.amazon.com/marketplace/pp/B00WH2LGM0

Barracuda Firewall: https://aws.amazon.com/marketplace/pp/B077G9FKK7

Juniper Networks: https://aws.amazon.com/marketplace/pp/B01LYWCGDX


You can use AWS’s managed VPN service. This is a great “middle of the road” compromise if you don’t currently have a VPN.  As a managed service AWS handles a lot of the nuts and bolts and you can get up and connected quickly.  Your users will connect to the AWS VPN which connects to your AWS VPC, (which is connected to your datacenter, network, on-prem resources, etc). As a fully managed client-based VPN you can manage and monitor all your connections from a single console.  AWS VPN is an elastic solution that leverages the cloud to automatically scale based on user demand, without the limitations of a hardware appliance.  It may also allow you to take advantage of additional AWS provided security mechanisms like rotating keys, credentialing, etc. to augment your security practices.

Finally, if you need something quick and have a smaller number of users, you can deploy your own VPN software on an Ec2 instance and “roll-your-own.” While this can be quick and dirty, this can be error-prone, less secure, and introduce a single point of failure, and it has to be manually managed.

Additional Services

There are a whole host of ancillary supporting services which can might need to be expanded for inclusion into Splunk such as Citrix, Webex, Skype, VoiP infrastructure, Teams, etc.. Below is an example of an Australian customer monitoring Video conferencing solutions with Splunk ITSI, but TekStream has been involved to build out monitoring of critical VoiP infrastructure and relate that to multi-channel support mechanisms including web and chat traffic. The point is that all of these channels might have just become critical infrastructure.


Much of the above recommendations can be accomplished in days or weeks. If there is an urgent need to temporarily expand your license to respond to the Coronavirus threat, that might be possible in the short term as well. With uncertainty around the duration of the pandemic, it would seem to warrant an all-out response from infrastructure, to processes and procedures, to operations, and security.

Your business can’t afford to fail. TekStream is here to help if you need us.

The Power of Splunk On-The-Go with Splunk Mobile and Splunk Cloud Gateway

By: Pete Chen | Splunk Practice Team Lead


Splunk can be a powerful tool in cybersecurity, infrastructure monitoring, and forensic investigations. While it’s great to use in the office, after-hour incidents require the ability to have data available immediately. Since most people carry a mobile device, such as a cell phone or a tablet, it’s easy to see how having dashboards and alerts on a mobile device can help bridge the information gap.

Splunk Mobile brings the power of Splunk dashboards to mobile devices, powered by Splunk Cloud Gateway. While Splunk Mobile is installed on a mobile device, Splunk Cloud Gateway feeds the mobile app from Splunk Enterprise. Between the two applications is Splunk’s AWS-hosted Cloud Bridge. Traffic between Splunk Enterprise and the mobile device is protected by TLS 1.2 encryption.

Architecture from Splunk

Splunk Cloud Gateway

Software Download https://splunkbase.splunk.com/app/4250/
Documentation https://docs.splunk.com/Documentation/Gateway

Splunk Cloud Gateway is a standard app found on Splunkbase (link above). It can be installed through the User Interface (UI), or by unpacking the file to <SPLUNK_HOME>/etc/apps/. When installed through the UI, Splunk will prompt for a restart once installation is complete. Otherwise, restart Splunk once the installation package has been unpacked into the Apps folder.

After restart, Splunk Cloud Gateway will appear as an app on Splunk Web. Browse to the app, and these are the pages available in the app:

The first page allows for devices to be manually registered. When Splunk Mobile is opened for the first time (or on a device not registered to another Splunk Cloud Gateway instance), an activation code will appear at the center of the display. That code can be used to register the device on Splunk. The “Device Name” field can be any value, used to identify that particular device. It’s helpful to identify the main user of the device and the type of device.

Skipping over Devices until a device is registered, and putting aside Splunk > AR for another time, the next important section is the “Configure” tab. At the top of the page, all the deployment configurations are listed. The Cloud Gateway ID can be modified through a configuration file to better reflect the environment. A configuration file can be downloaded for a Mobile Device Manager (MDM). This is also where the various products associated with Splunk Connected Experiences can be enabled.

In the Application section, look for Splunk Mobile. Under the Action column, click on Enable. This must be done before a device can be registered.

The App Selection Tab is where apps can be selected, based on each user’s preference, to determine which dashboards are visible through Splunk Mobile. When no apps are selected, all available dashboards are displayed. Select the apps desired by clicking them from the left panel, and they will appear on the right panel. Be sure to click save to commit the changes.

A couple of things to point out in this section.

  • Again, if an app is not selected, all available dashboards to the user will appear on Splunk Mobile.
  • Management of apps is based on the user, not centrally managed. During the registration of a device, a user must log in to authenticate. The apps selected in this page will be the same for all devices registered under this user.
  • Even if apps are specified, all dashboards set with global permissions will still be visible to the user.
  • To eliminate all dashboards and control what is viewable requires setting all dashboards to app-only permissions, and creating a generic app without dashboards. When this app is selected, and after all dashboards are converted to app-only permissions, no dashboards will appear.

The final tab is the dashboard for Splunk Cloud Gateway. This dashboard shows the status of the app, and provides metrics of usage. The top three panels may be the most important when first installing Cloud Gateway. If the service doesn’t seem to be working correctly, these three panels will help in troubleshooting the service.


Splunk Mobile

Google Play Store https://play.google.com/store/apps/details?id=com.splunk.android.alerts
Apple App Store https://apps.apple.com/us/app/splunk-mobile/id1420299852

Installing Splunk Mobile on a mobile device is as simple as going to the app store, and having the device set up the app. Once the app is ready, launching the app will bring up a registration page. On this page, there is a code needed to register the device with Splunk Cloud Gateway. Below is a secondary code. This is used to verify with Cloud Gateway, making sure the device is registered with the correct encryption key.

With the code above, return to Splunk Cloud Gateway, and register the device. Type in the activation code from Splunk Mobile. Enter in a device name, as explained above. Click on “Register” to continue.

Validate the confirmation code displayed in the UI with the code displayed on the device. If the codes don’t match, stop the registration process. If the codes do match, enter credentials for Splunk, and click “Continue”.

At this point, the device is registered with Splunk Cloud Gateway. Validate the device name in the Registered Devices page. Make sure the Device Type, and the Owner matches the device and user. If necessary, “Remove” is available to remove a device from Cloud Gateway.

From a mobile perspective, the initial page displayed is the list of potential alerts.

At the bottom of the screen, tap on “Dashboards” to see the list of dashboards available to the mobile device. Without any additional configuration, all available Splunk dashboards should appear in the list. Click on any dashboard.

As an example, when the Cloud Gateway Status Dashboard is selected, the dashboard opens and allows for a time-selector at the top of the page. The panels available from the UI are displayed in a single column on the mobile device.

Points to Consider

Now that Splunk Mobile and Splunk Cloud Gateway are configured, and ready to be used, here are some points to consider in an Enterprise deployment.

  • When installing on a search head cluster, Splunk Cloud Gateway must be installed on the cluster captain. The captain runs some of the scripts necessary to connect Cloud Gateway to the Spacebridge.
  • All dashboards set with global permissions will appear. To limit visibility, set dashboard permissions to app-only or private.
  • During device registration, the credentials used will determine the dashboards and alerts available to the device. Configuration is user-based, not centrally controlled.
  • Trellis is not a supported feature of Splunk Mobile. Dashboards with panels using trellis will need to be reconfigured.
  • Panel sizing and scaling is not adjustable at this time. Some dashboard re-design may be necessary to tell the best story.
  • Pay special attention to how long dashboards take to load. From a mobile perspective, dashboards will need to load faster for the mobile user.

Want to learn more about Splunk Mobile and Splunk Cloud Gateway? Contact us today!

How to Extract Your PO Numbers Consistently in Oracle’s Forms Recognition AP Project

By: William Phelps | Senior Technical Architect

One of the thornier issues when working with Oracle’s Forms Recognition Accounts Payable (“AP”) project is simply and correctly determining and extracting a correct purchase order number from the invoice image.  This seemingly mundane task is further complicated when the purchase order number is a mere simple string of digits, much like, and sometimes confused with, telephone numbers, serial numbers, shipment numbers, and similar purely numeric strings found on the invoice.

This is a common problem for many companies using the AP Solution project, and it’s a fair bet that if you are reading this article, your company has the same or similar issue.

Let’s note upfront that there is no one magic solution bullet that will fix all extraction problems.  This article is intended as a fine-tuning methodology once very basic solutions and ERP data cleanup has occurred.  It’s at that point, when the easy stuff has been done that any additional techniques should be applied.  (A certified partner can help make these advanced changes with less overall effort and better end results.)

In general terms, the Oracle AP project provides a process called “PO masking” to allow the customer to tell the software about the general characteristics of their PO number structure.  This approach uses somewhat simple regular expressions (or “masks”) to derive potential strings deemed to be viable PO number “candidates” that it encounters while parsing the invoice text.  This kind of generalized setup almost always produces extraneous candidates.  Often it’s further determined by the process that, from this list of candidates that it extracts, some candidates are deemed a better match based on where the string is found in the document.  It places a lower ranking, called “weighting”, on candidates that may be embedded within the body of the invoice, like the case when the PO number is listed within a line description, and instead places a higher “weight” on a wrong value near the page header or top of the invoice.

A somewhat more educated and targeted way to help Forms Recognition get to that right value will involve an additional detailed look at the list of potential candidates.  During this further programmatic inspection, we can try removing or reducing the “weights” of those potential candidates that we think are misses by using true regular expressions in Visual Basic.

For a very simple example, a given operating unit may have only a handful of unique patterns for their PO numbers. Wide, generalized mask definitions intended for multiple operating units will likely result in more misses.

In WFR using the Inspyrus/Solution Accelerator PO header view (“xx_ofr_po_header_v”), the operating unit is available in the view alongside the PO number.  Using this information indirectly, the PO candidate weights can be altered to increase the accuracy of the extraction.

In these cases, the incoming invoice should be coming from a process that is pre-assigning the correct operating unit.  Since we will know the general PO number patterns for each operating unit, the list of extracted potentials can then be whittled down to a very precise list. (The real work is in determining the exact regular expression per operating unit, which is beyond the scope of this post.)

For today’s example,

  • Open the AP Solution project in WFR Designer and edit the script for the Invoices class.
  • On the UserExits script page, add the following function at the very bottom of the sheet. (Be sure to only add custom code in designated or legal areas of the script page for supportability.)

Then, in “UserExitPONUmberPostEvaluate” on the same script sheet, update the subroutine with the PO filtering code below:

Save the project file and try processing those problem vendors and purchase order numbers again.

Variations of this code have been deployed at several customers, resulting in much-improved PO number extraction rates.  This increased extraction success rate translates into less manual correction and increased invoice processing throughput since PO lines can then also be paired with a greater success rate automatically.

As noted earlier, a certified partner can help make these kinds of advanced changes with less overall effort and better end results.

Contact us if this express lane to regular payments sounds like a great idea!