Machine Learning with Splunk: Testing Logistic Regression vs Support Vector Machines (SVM) using the ML Toolkit

By: Brent McKinney | Splunk Consultant

 

If you’ve ever spent time working with Machine Learning, it’s highly likely you’ve come across Logistic Regression and Support Vector Machines (SVMs). These 2 algorithms are amongst the most popular models for binary classification. They both share the same basic goal: Given a sample x, how can we predict a variable of interest y?

For example, let’s say we have a dataset with samples (x) containing the following,

and we want to determine a single variable y: Is this person diabetic or not?

Logistic Regression and SVMs are perfect candidates for this!

The problem now lies in finding the means to test this on a sizeable dataset, where we have hundreds or thousands of samples. Coding machine learning algorithms can become quite a task, even for experienced developers. This is where Splunk comes in!

Splunk’s Machine Learning Toolkit makes testing ML algorithms a breeze. The Machine Learning Toolkit is an app, completely free to download on Splunkbase and allows users to visualize and compare results from ML algorithms quickly, without having to code them.

To stay consistent with our previous example, I will be demonstrating this with a public dataset from Kaggle.com, in the form of a CSV file. This dataset includes real data, that is already labeled and clean. Since our data is properly labeled, this can be will serve as a supervised learning problem. This simply means that our task will learn a function that maps an input (our x features) to an output (our y value – diabetic or not) based on the labeled items. I’ve posted the link to the MLTK app, as well as the dataset used in this example, as sources at the bottom of this page.

To install the MLTK app: Once you’ve downloaded the Machine Learning Toolkit from Splunkbase, log into your Splunk instance, and click the Apps dropdown at the top of the screen. Select “Manage Apps” and then click the button “Install app from file”. From here select Choose File and select the MLTK app folder (no need to untar the file, Splunk will unpack the folder on the server!). Click Upload.

To upload a csv file: You can upload the csv file by clicking Settings>Lookups>Lookup table files>New Lookup Table File. Select MLTK for the app, our csv as the upload file, and give a name with the .csv extension (diabetes.csv). Then go to Settings>Lookups>Lookup definition>New Lookup Definition to define the lookup. We’ll select the MLTK app, “diabetes” for the name, “File-based” for the type, and the csv file for the Lookup file.

Once the Machine Learning Toolkit has been installed, and the dataset file has been uploaded to Splunk, we can get to work.

From your Splunk instance, navigate to the Machine Learning Toolkit app by selecting it from the “App” dropdown menu at the top of the screen.

From here we can see there are several options available, each depending on the problem you are trying to solve. We want to categorize our data into 2 categories: diabetic and not diabetic. So for our example, we will use “Predict Categorical Fields”.

To start, select “Experiments” from the navigation bar, and then “Create New Experiment”. Select the appropriate experiment type, then add a title and description.

 

Once all looks good, select Create.

Now we are brought to the experiment screen. To use our diabetes dataset, we will need to use the SPL inputlookup command in the search bar. Note the search must begin with a | as this is a generating command.

This will return the data we uploaded from the CSV file.

As we can see, there are a few parameters that need to be set. The first being the algorithm we want to use. We will be testing Logistic Regression and SVM. The default is Logistic Regression so we can leave it as-is for now.

The next parameter is “Field to Predict”. This represents the variable we want to discover, y. This list is populated with fields found in our csv file. In our example, our  y variable is “Outcome”, which gives a value of 1 for samples that are diabetic, and a value of 0 for samples that are NOT diabetic.

 

 

The next parameter is “Fields to use for predicting”. As the name implies, these are the variables that make up our feature sample x. The algorithms will use these features to determine our Outcome variable. The more relevant fields we select here, the more accurate our algorithms will be when calculating a result, so in this case we will select all of them.

Once these parameters have been set, all we need to do is decide how we want to split the data into training and testing.

Machine Learning algorithms use the training data to determine a function that most accurately produces the desired output. So to achieve the best accuracy, we want to use a majority of the data for training. Once the algorithm is trained on the dataset, it runs this function on the test data and gives an output based on the samples it saw during training. For this example, I will use 80% for training, and 20% for testing.

(Note, while we want to use as much training data as possible, we must have some test data. If we use 100% of the data for training, then any test data will have already been seen by the algorithm, and therefore not give us any insightful results.)

Now that all of our parameters are set, we are ready to see results!

Select Fit Model to run Logisitic Regression on our data.

Once the algorithm is finished, we are given 3 panels.

The first returns a results table containing our test data. The columns on the right of the bold line show our original x features for each sample. The columns on the left of the bold line show the output that the algorithm predicted for each sample, compared to the actual output in the dataset, for each sample, highlighting the ones it got wrong.

The panel on the bottom left shows the degree of accuracy of the algorithm for our given dataset. From this we can conclude that if we were to give this model a new sample, it would determine whether or not the sample is diabetic or not with a 77% degree of accuracy.

The bottom right panel gives a little more detail, showing how well the algorithm did at predicting each outcome. We can see that for our particular example, it did slightly better at determining samples that were not diabetic, as opposed to samples that were.

Now let us compare this to a SVM model. Considering that we want to use the same dataset and parameters, all we need to do is change the algorithm.

Once that is set, we can select Fit Model to run SVM on our data.

Right away we can see that using Support Vector Machines gives us substantially better results than Logisitic regression. Both algorithms give the same details format, but we can see that using SVM resulted in a 97% accuracy when predicting on our test data, in comparison to LR resulting in 77%.

 

To conclude, Splunk’s Machine Learning Toolkit provides an easy-to-use environment for testing and comparing Machine Learning algorithms. In this demonstration, we used Splunk and Machine Learning to create models to predict whether a given sample is diabetic or not. While this demonstration focused on SVMs and Logistic Regression, there are many more algorithms available in Splunk’s Machine Learning Toolkit to play around with, including Linear Regression, Random Forests, K-means Clustering, and more!

Link to download Machine Learning Toolkit app:

https://splunkbase.splunk.com/app/2890

Link to download dataset used in this example:

https://kaggle.com/johndasilva/diabetes

 

Want to learn more about Machine Learning with Splunk? Contact us today!

Options to Consider for Your Oracle 12c WebCenter Upgrade

By: Brandon Prasnicki | Technical Architect

 

If you search the Oracle knowledgebase on how to upgrade your existing Oracle WebCenter Content (WCC), Imaging, or Portal instance from 11g to 12c, your options are to do an in-place upgrade or to migrate the entire repository using Oracle WebCenter Content supported tools.  However, if an upgrade consists of new hardware (on-premise), new cloud Infrastructure (Oracle Cloud Infrastructure (OCI), Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform etc), upgraded operating systems (Microsoft Windows or Linux) along with database upgrade (Oracle Database 12c), the only supported method is to use these supported migration tools.  To move the content from one machine to the next, this process consists of the following:

  1. Install 12c on the new environment
  2. Create the 12c schemas with RCU
  3. Create and configure the 12c domain
  4. Migrate the WCC configurations with CMU and Archiver
  5. Migrate the WCC content with Archiver

While this is a straightforward approach, the question becomes:  Is this feasible?

The answer to that question is:  It depends.

With any upgrade project, TekStream Solutions evaluates the scope of the upgrade and migration and makes recommendations on the appropriate approach.  Here is a high-level outline of starting points considered during the TekStream QuickStream process:

  1. Is the repository small? This supported methodology is a good approach and alternative for instances that do not hold a lot of content.  We have seen situations for implementations that leverage WCC as a backend for implementations like Portal where the content repository isn’t very large.  For this, the supported methodology is a decent alternative.
  2. Are there opportunities to decommission old Enterprise Content Management Systems? Sometimes there is an opportunity to also mix in and decommission old content repositories. Examples include old shared filesystems not currently managed by any enterprise content management systems (CMS), or even little-used old CMS systems where, depending on the customer license structure, the ROI of rolling into an Oracle WebCenter Content (WCC) instance makes sense during the time of upgrade.  Examples of this include but are not limited to Adobe and Documentum etc.  For this, TekStream utilizes a proprietary utility called “Content Loader” to handle WCC delta migrations, and merge deprecated CMS application content.
  3. Is the repository large? For very large repositories, Tekstream uses a cost-effective approach called the “out of place” in-place upgrade which eliminates the need to migrate the content.  The ‘supported’ Oracle approach simply is not feasible, as repositories with millions of content items would take months and maybe even years to migrate.  Examples of implementations that include large repositories include Digital Asset Management (DAM), Records Management (RM) and even some regular Content Management repositories.   When Oracle states this “out of place” in-place upgrade is not a supported approach, they are strictly referring to all the ‘gothchas’ that can occur.  The support team members at Oracle are not the consultants to handle such an approach.  That is where TekStream solutions come in to guide and implement the upgrade to a successful outcome.
    1. Have we seen ‘gotchas’ in this approach? Certainly.  Every version and situation has its nuances.  TekStream’s QuickStream process digs deeper into identifying unique customer situations to account for during a migration.  Tekstream has proven to handle these challenges and deliver successful implementations.  Our background expertise performing these upgrades before has proven vital to customer success.
    2. Could a customer do this approach by themselves? Honestly, probably not.  We are here to guide you through this approach and avoid the pitfalls that can occur.  We have been through this before, and are here to guide and deliver a successful upgrade.

TekStream Solutions makes sure that the system is migrated, upgraded and in a clean, working, and supported state at the completion of the project.   This approach has proven to save customers a lot of time and money.  TekStream also offers extended support and is an Oracle Managed Services provider to give customers great peace of mind and frees up internal resources for more demanding in-house projects.

Want to learn more about Oracle 12c WebCenter upgrades? Contact us today!

Press Release: TekStream Makes 2019 INC. 5000 List for Fifth Consecutive Year

For the 5th Time, Atlanta-based Technology Company Named One of the Fastest-growing Private Companies in America with Three-Year Sales Growth of 166%

ATLANTA, GA, August 14, 2019– Atlanta-based technology company, TekStream Solutions, is excited to announce that for the fifth time in a row, it has made the Inc. 5000 list of the fastest-growing private companies in America. This prestigious recognition comes again just eight years after Rob Jansen, Judd Robins, and Mark Gannon left major firms and pursued a dream of creating a strategic offering to provide enterprise technology software, services, solutions, and sourcing. Now, they’re a part of an elite group that, over the years, has included companies such as Chobani, Intuit, Microsoft, Oracle, Timberland, Vizio, and Zappos.com.

“Being included in the Inc. 5000 for the fifth straight year is something we are truly proud of as very few organizations in the history of the Inc. 5000 list since 2007 can sustain the consistent and profitable growth year over year needed to be included in this prestigious group of companies,” said Chief Executive Officer, Rob Jansen. “The accelerated growth we are seeing to help clients leverage Cloud-based technologies and Big Data solutions to solve complex business problems has been truly exciting. We are helping our clients take advantage of today’s most advanced recruiting and technology solutions to digitally transform their businesses and address the ever-changing market.”

This year’s Inc. 5000 nomination comes after TekStream has seen a three-year growth of over 166%, and 2019 is already on pace to continue this exceptional growth rate. In addition, the company has added 30% more jobs over the last 12 months.

“Customers continue to invest in ‘Cloud First’ strategies to move their on-premises environments to the cloud, but often struggle with how to get started.  There is a vast market for specialized experts familiar with both legacy systems and newer cloud technology platforms.  Bridging those two worlds to address rapid line of business changes and reducing technology costs are focal points of those strategies. TekStream is well-positioned to continue that thought leadership position over the next several years.” stated Judd Robins, Executive Vice President of Sales.

To qualify for the award, companies had to be privately owned, established in the first quarter of 2015 or earlier, experienced a two-year growth in sales of more than 50 percent, and garnered revenue between $2 million and $300 million in 2018.

“The continued recognition is evidence of our team’s response to client’s recruiting needs across multiple industries and sectors. The growth in hiring demands commercially and federally along with the need to deliver on changing candidate demands have fueled the work we have put into having both outsourced and immediate response contingent recruiting solutions,” stated Mark Gannon, Executive Vice President of Recruitment.

TekStream
We are “The Experts of Business & Digital Transformation”, but more importantly, we understand the challenges facing businesses and the myriad of technology choices and skillsets required in today’s “always on” companies and markets. We help you navigate the mix of transformative enterprise platforms, talent and processes to create future-proof solutions in preparing for tomorrows opportunities…so you don’t have to. TekStream’s IT consulting solutions combined with its specialized IT recruiting expertise helps businesses increase efficiencies, streamline costs and remain competitive in an extremely fast-changing market. For more information about TekStream Solutions, visit www.tekstream.com or email info@tekstream.com.

Integrating Oracle Human Capital Management (HCM) and Content and Experience Cloud (CEC)

By: Greg Becker | Technical Architect

OVERVIEW

During the first phase of a recent project we built an employee file repository for a Healthcare client in the Oracle Cloud Infrastructure – Classic (OCI-C) space. A number of services were used including Oracle Content and Experience Cloud (repository), Oracle Process Cloud Service (for filing the documents in a logic structure), Oracle WebCenter Enterprise Capture (for scanning) and Oracle Database Cloud Service (for custom application tables).

During the second phase of the project our clients had a requirement to automatically update metadata values on their content items stored in the CEC repository. They wanted to trigger a change based on events or updates that occur for an employee record that is stored in Oracle Human Capital Management, for example when an Employee Status changes from Active to Inactive.

Our solution was to use an Oracle Process Cloud Service process to perform the metadata updates when certain values were passed into the process. The reason for updating the metadata is so that accurate searches can be performed by end users. The tricky part of the implementation is how to call the PCS process based on the change. To accomplish this Informatica is used to determine a ‘change’ based on data from the tables within the HCM structure and then pass that change record to a DB table used by the client solution. At that point a database function was developed to action the PCS REST Web Service. The final step of the process was to build a database trigger that called the function.

First you need to do some initial setup to be able to use the APEX libraries as well as create the network ACL to connect to the PCS domain you’re using. You can find this information in various places online. You can either use SOAP or REST web services and we chose REST. If you want to call the web service using SSL (which we did) you’ll have to also create an Oracle wallet.

CODE SNIPPETS

Function Definition:

SOAP Envelope:

Call the Function from a Trigger:

SUMMARY

There are more than one ways to fulfil this customer requirement but these are the pieces that worked well in this case. If you have any additional integration needs between Oracle Human Capital Management and Oracle Content and Experience Cloud please contact TekStream and we’d be happy to assist you.

Iplocation: Simple Explanation for Iplocation Search Command

By: Charles Dills | Splunk Consultant

Iplocation can be used to find some very important information. It is a very simple yet powerful search command that can help with identifying where traffic from a specific IP is coming from.

To start iplocation on its own won’t display any visualizations. What it will do is add a number of additional fields that can be used in your searches that can be added to dashboards, panels, and tables. Below we will use a simple base search using Splunk example data:

From here we will add iplocation to our search, sorting by clientip. As you can see in the below screenshot, this added a few fields that we can use circled in red:

From here we can alter our search with a table to display the information we need. For example, for a company who is based and fully operates out of the US could consider and traffic going outside the us to a foreign country as unauthorized or malicious. Using the iplocation in combination with values, we are able to list out each IP address that is not located inside the US and display each by which country It is located:

The last thing we will do is clean up our table using rename and this can provide a simple way to distinguish where traffic from a specific IP address is coming from:

Want to learn more about iplocation? Contact us today!

Take your Traditional OCR up a notch

By: Greg Moler | Director of Imaging Solutions

While the baseline OCR landscape has not changed much, AWS aims to correct that. Traditional OCR engines are quite limited in what details they can provide. Being able to detect the characters is only half the battle, the ability to get meaningful data out of them becomes the challenge. Traditional OCR follows the ‘what you see is what you get’ mantra, meaning once you run your document through, the blob of seemingly unnavigable text is all you are left with. What if we could enhance this output with other meaningful data elements useful in extraction confidence? What if we could improve the navigation of the traditional OCR block of text?

Enter Textract from AWS. A public web service aimed to improve your traditional OCR experience in an easily scalable, integrable, and low cost package. Textract is built upon an OCR extraction engine that is optimized by AWS’ advanced machine learning. It has been taught how to extract thousands of different types of forms so you don’t have to worry about it. The ‘template’ days are over. It also provides a number of useful advanced features that other engines simply do not offer: confidence ratings, word block identification, word and line object identification, table extraction, and key-value output. Let’s take a quick look at each of these:

  • Confidence Ratings: Ability to intelligently make choices to accept results, or require human intervention based on your own thresholds. Building this into your work flow or product can greatly improve data accuracy
  • Word Blocks: Textract will identify word blocks allowing you to navigate through them to help identify things like address blocks or known blocks of text in your documents. The ability to identify grouped wording rather than sifting through a massive blob of OCR output can help you find the information you are looking for faster
  • Word and Line Objects: Rather than getting a block of text from a traditional OCR engine, having code-navigable objects to parse your documents will greatly improve your efficiency and accuracy. Paired with location data, you can use the returned coordinates to pinpoint where it was extracted from. This becomes useful when you know your data is found in specific areas or ranges of a given document to further improve accuracy and filter out false positives
  • Table Extraction: Using AWS AI-backed extraction technology, Table extraction will intelligently identify and extract tabular data to pipe into whatever your use case may need, allowing you to quickly calculate and navigate these table data elements.
  • Key-value Output: AWS, again using AI-backed extraction technology, will intelligently identify key-value pairs found on the document without having to write custom engines to parse the data programmatically. Optionally, send these key-value pairs to your favorite key-value engine like Splunk or Elasticsearch (Elastic Stack) for easily searchable, trigger-able, and analytical actions for your document’s data.

Contact us today to find out how Textract from AWS can help streamline your OCR based solutions to improve your data’s accuracy!

Tsidx Reduction for Storage Savings

By: Yetunde Awojoodu | Splunk Consultant

Introduction

Tsidx Reduction was introduced in Splunk Enterprise v6.4 to provide users with the option of reducing the size of index files (tsidx files) primarily to save on storage space. The tsidx reduction process transforms full size index files into minified versions which will contain only essential metadata. A few scenarios to consider tsidx reduction include:

  • Consistently running out of disk space or nearing storage limits but not ready to incur additional storage costs
  • Have older data that are not searched regularly
  • Can afford a tradeoff between storage costs and search performance

How it works

Each bucket contains a tsidx file (time series index data) and a journal.gz file (raw data). A tsidx file associates each unique keyword in your data with location references to events, which are stored in the associated rawdata file. This allows for fast full text searches. By default, an indexer retains tsidx files for all its indexed data for as long as it retains the data itself.

When buckets are tsidx reduced, they still contain a smaller version of the tsidx files. The reduction applies mainly to the lexicon of the bucket which is used to find events matching any keywords in the search. The bloom filters, tsidx headers, and metadata files are still left in place. This means that for reduced buckets, search terms will not be checked against the lexicon to see where they occur in the raw data. 

Once a bucket is identified as potentially containing a search term, the entire raw data of the bucket that matches the time range of the search will need to be scanned to find the search term rather than first scanning the lexicon to find a pointer to the term in the raw data. This is where the tradeoff with search performance occurs. If a search hits a reduced bucket, the resulting effect will be slower searches. By reducing tsidx files for older data, you incur little performance hit for most searches while gaining large savings in disk usage.

The process can decrease bucket size by one-third to two-thirds depending on the type of data. For example, a 1GB bucket would decrease in size between 350MB – 700MB. The exact amount depends on the type of data. Data with many unique terms require larger tsidx files. To make a rough estimate of a bucket’s reduction potential, look at the size of its merged_lexicon.lex file. The merged_lexicon.lex file is an indicator of the number of unique terms in a bucket’s data. Buckets with larger lexicon files have tsidx files that reduce to a greater degree.

When a search hits the reduced buckets, a message appears in Splunk Web to warn users of a potential delay in search completion: “Search on most recent data has completed. Expect slower search speeds as we search the minified buckets.” Once you enable tsidx reduction, the indexer begins to look for buckets to reduce. Each indexer reduces one bucket at a time, so performance impact should be minimal.

Benefits

  • Savings in disk usage due to reduced tsidx files
  • Extension of data lifespan by permitting data to be kept longer (and searchable) in Splunk
  • Longer term storage without the need for extra architectural steps like adding S3 archival or rolling to Hadoop.

Configuration

The configuration is pretty straight forward and you can perform a trial by starting with one index and observing the results before taking further action on any other indexes. You will need to specify a reduction age on a per-index basis:

1. On Splunk UI:

  • Go to Settings > Indexes > Select an Index
    Set tsidx reduction policy.

2. Splunk Configuration File:

  • indexes.conf
    [<indexname>]
    enableTsidxReduction = true
    timePeriodInSecBeforeTsidxReduction = <NumberOfSeconds>

The attribute “timePeriodInSecBeforeTsidxReduction” is the amount of time, in seconds, that a bucket can age before it becomes eligible for tsidx reduction. When this time difference is exceeded, a bucket becomes eligible for tsidx reduction. Default Is 604800

To check whether a bucket is reduced, run the dbinspect search command:

| dbinspect index=_internal
The tsidxState field in the results specifies “full” or “mini” for each bucket.

To restore reduced buckets to their original state, refer toSplunk Docs

A few notes

  • Tsidx reduction should be used on old data and not on frequently searched data. You can continue to search across the aged data, if necessary, but such searches will exhibit significantly worse performance. Rare term searches, in particular, will run slowly.
  • A few search commands do not work with reduced buckets. These include ‘tstats’ and ‘typeahead’. Warnings will be included in search.log

Reference Links

https://docs.splunk.com/Documentation/Splunk/7.2.6/Indexer/Reducetsidxdiskusage

https://conf.splunk.com/files/2016/slides/behind-the-magnifying-glass-how-search-works.pdf

https://conf.splunk.com/files/2017/slides/splunk-data-life-cycle-determining-when-and-where-to-roll-data.pdf

Want to learn more about Tsidx Reduction for Storage Savings? Contact us today!

Operating a Splunk Environment with Multiple Deployment Servers

Operating a Splunk Environment with Multiple Deployment Servers

By: Eric Howell | Splunk Consultant

Splunk Environments come in all shapes and sizes, from the small single-server installation managing all of your Splunk needs in one easily-managed box, to the multi-site, extra complex environments scaled out for huge amounts of data and all the bells and whistles to get in-depth visibility and reporting into a wide variety of circumstances as suits functionally any use case you can throw at Splunk. And, of course, everything in between.

For those multi-site, or multi-homed environments, that many data centers require for any range of needs, managing your configurations begins to get complicated between the additional firewall rules, data management stipulations, and any other broad range of issues that might crop up.

Thankfully, Splunk Enterprise allows for your administrative team, or Splunk professional services, to set up a Deployment Server to manage the configurations (bundled into apps) for all of the universal forwarders, so long as they’ve been set up as deployment clients. In a complicated environment, you may find that you need two deployment servers to manage the workload, for any number of reasons. Perhaps you are trying to keep uniform configuration management systems in multiple environments, or perhaps you are aiming to spread the communication load across multiple servers for these deployments. Whatever the use case, setting up two (or more) deployment servers is not the heartache you may be worried about, and the guide below should be ample to get you on the right track.

Multiple Deployment Servers – Appropriate Setup

To set up multiple deployment servers in an environment, you will need to designate one of the Deployment Servers as the “Master” or “Parent” server (DS1). This is likely to be the original deployment server that houses all of the necessary apps, and is likely already serving as deployment server to your environment.

The use case below will allow you to service a multi-site environment where each environment requires the same pool of apps, but is small enough to be serviced by a single deployment server.

  1. Stand up a new box (or repurpose a decommissioned server, as is your prerogative)! Install Splunk on this new server. This will act as your second deployment server (DS2).
  2. The key difference between these servers is that DS2 will actually be a client of the DS1.
  3. Initial set up is minimal, but make sure that this server has any standard configurations the rest of your environment holds, such as an outputs.conf to send its internal logs to the indexer layer, if you are leveraging that functionality.
  • You will create a deployment client app on DS2. You could use a copy of a similar app that resides on one of your heavy forwarders that poll DS1 for configuration management, but you will need to make two key adjustments in deploymentclient.conf:

  • Once this change has been made, the apps that will be pulled down from DS1 will reside in the appropriate location on DS2 to be deployed out to any servers that poll it.
  • Restart Splunk on DS2
  • Next, you will need to navigate to the ForwarderManagement UI on DS1 and create a Server Class for your Slave or ChildDeployment Servers (DS2 in this case)
  • Add all apps to this new server class
    • Allowing Splunk to restart with these apps isfine, as changes made to the originating Deployment Server (DS1) will allow DS2
      to recognize that the apps that it holds have been updated and are ready for
      deployment.
  • Add DS2 to this Server Class
  • Depending on the settings you have configured indeploymentclient.conf on DS2 for its polling period (phoneHomeIntervalInSecs
    attribute), and how many apps there are for it to pull down from DS1, wait an appropriate amount of time (longer than your polling period, and more) and
    verify if the apps have all been deployed.
  • After this, updates made to the apps on DS1 will propagate down to DS2.

Alternative Use Case

If you are planning to leverage multiple deployment servers to service the same group of servers/forwarders, you will want to also copy over the serverclass.conf from DS1. If all server classes have been created through the web ui, the file should be available here:

$SPLUNK_HOME/etc/system/local/serverclass.conf

If this is your intended use case, you will also want to work with your Network Team to place the Deployment Servers behind a loadbalancer. If you do so, you’ll need to modify the following attribute in deploymentclient.conf in your deployment client app that resides on your forwarders to indicate the VLAN:

You will also need to make sure both Deployment Servers generate the same “checksums” so that servers polling in and reaching different DS servers do not redownload the full list of apps with each check-in.

To do so, you will need to modify serverclass.conf on both Deployment Servers to include the following attribute:

This attribute may not be listed by default, so you may need to include it manually. This can be included with the other attributes in your [global] stanza.

Want to learn more about operating a Splunk environment with multiple deployment servers? Contact us today!

AXF 12c Upgrade Patches and FIPSA Components

By: John Schleicher | Sr. Technical Architect

Introduction

This document contains the patch listing that was assembled during a recent Financials Image Processing Solution Accelerator (FIPSA) upgrade where the system was upgraded from 11.1.1.8 (imaging) to 12.2.1.3 release using the standard upgrade process and supplemented by post-upgrade activity to restore the system to full functionality.

The patch listing represents all of the WebLogic server components inclusive of Business Activity Monitoring (BAM)  that were present on the custom solution.   If your system doesn’t include BAM then the additional patches (26404239, 26081565, 28901325) aren’t required.

FIPSA Release

The FIPSA package 12.2.1.3.2 is required for the upgrade as it contains the necessary libraries and archives that are required for the AXF Solution Workspace and Coding form to run in the 12c environment.

Manual Edit

Due to a modification to the central task service engine which affects the SystemAttributes structure a single line edit is required of the InvoiceProcessing.bpel file of the 12.2.1.3.2 FIPSA release.  Presumably, this will be modified by subsequent releases.  Ensure that on line 3411 the reference to task:assigneeUsers/task:id is changed to task:updatedBy/task:id.  This is the least impact solution and may be adjusted in future releases but this has been tested and is working.

Note that active InvoiceProcessing tasks after upgrade cannot use the ‘SaveTask’ AXF action as the old paradigm will be engaged and the process will fault at the noted ‘assigneeUsers’ reference.  It is recommended that the ‘Save Task’ AXF action be disabled via the Imaging Solution Editor to avoid this fault until such time that active workflow instances are no longer present on that baseline.

Patch Listing

Here is an opatch lsinventory listing of the patches applied to the system representing bam, capture, content, soa, and WebLogic:

********************************************************************************

Oracle Interim Patch Installer version 13.9.4.0.0

Copyright (c) 2019, Oracle Corporation.  All rights reserved.

Oracle Home       : /oracle/middleware12c

Central Inventory : /oracle/oraInventory

   from           : /oracle/middleware12c/oraInst.loc

OPatch version    : 13.9.4.0.0

OUI version       : 13.9.3.0.0

Log file location : /oracle/middleware12c/cfgtoollogs/opatch/opatch2019-04-23_10-51-48AM_1.log

OPatch detects the Middleware Home as “/oracle/middleware12c”

Lsinventory Output file location : /oracle/middleware12c/cfgtoollogs/opatch/lsinv/lsinventory2019-04-23_10-51-48AM.txt

——————————————————————————–

Local Machine Information::

Hostname: imaging

ARU platform id: 226

ARU platform description:: Linux x86-64

Interim patches (18) :

Patch  26045997     : applied on Tue Apr 23 10:50:59 MDT 2019

Unique Patch ID:  22112962

Patch description:  “One-off”

   Created on 13 Apr 2018, 23:35:27 hrs UTC

   Bugs fixed:

     26045997

Patch  27133806     : applied on Tue Apr 23 10:41:52 MDT 2019

Unique Patch ID:  22061693

Patch description:  “One-off”

   Created on 27 Mar 2018, 16:59:09 hrs PST8PDT

   Bugs fixed:

     27133806

Patch  25830131     : applied on Tue Apr 23 10:35:35 MDT 2019

Unique Patch ID:  22704908

Patch description:  “One-off”

   Created on 27 Jan 2019, 12:26:12 hrs PST8PDT

   Bugs fixed:

     25830131

   This patch overlays patches:

     28710939

   This patch needs patches:

     28710939

   as prerequisites

Patch  28710939     : applied on Tue Apr 23 10:31:41 MDT 2019

Unique Patch ID:  22540742

Patch description:  “WLS PATCH SET UPDATE 12.2.1.3.190115”

   Created on 21 Dec 2018, 14:25:48 hrs PST8PDT

   Bugs fixed:

     23076695, 23103220, 25387569, 25488428, 25580220, 25665727, 25750303

     25800186, 25987400, 25993295, 26026959, 26080417, 26098043, 26144830

     26145911, 26248394, 26267487, 26268190, 26353793, 26439373, 26473149

     26499391, 26502060, 26547016, 26589850, 26608537, 26624375, 26626528

     26731253, 26806438, 26828499, 26835012, 26929163, 26936500, 26985581

     27055227, 27111664, 27117282, 27118731, 27131483, 27187631, 27213775

     27234961, 27272911, 27284496, 27411153, 27417245, 27445260, 27469756

     27486993, 27516977, 27561226, 27603087, 27617877, 27693510, 27803728

     27819370, 27912485, 27927071, 27928833, 27934864, 27947832, 27948303

     27988175, 28071913, 28103938, 28110087, 28138954, 28140800, 28142116

     28149607, 28166483, 28171852, 28172380, 28311332, 28313163, 28319690

     28360225, 28375173, 28375702, 28409586, 28503638, 28559579, 28594324

     28626991, 28632521

Patch  29620828     : applied on Tue Apr 23 08:57:20 MDT 2019

Unique Patch ID:  22858384

Patch description:  “ADF BUNDLE PATCH 12.2.1.3.0(ID:190404.0959.S)”

   Created on 15 Apr 2019, 17:17:00 hrs PST8PDT

   Bugs fixed:

     23565300, 24416138, 24717021, 25042794, 25802772, 25988251, 26587490

     26674023, 26760848, 26834987, 26957170, 27970267, 28368196, 28811387

     28849860

Patch  29367192     : applied on Tue Apr 23 08:50:38 MDT 2019

Unique Patch ID:  22751712

Patch description:  “One-off”

   Created on 12 Mar 2019, 01:07:01 hrs PST8PDT

   Bugs fixed:

     28843809, 28861250, 28998550, 29259548

   This patch overlays patches:

     28928412

   This patch needs patches:

     28928412

   as prerequisites

Patch  29257258     : applied on Tue Apr 23 08:45:17 MDT 2019

Unique Patch ID:  22807543

Patch description:  “OWEC Bundle Patch 12.2.1.3.190415”

   Created on 16 Apr 2019, 07:02:38 hrs PST8PDT

   Bugs fixed:

     18519793, 18877178, 19712986, 21110827, 21364112, 24702902, 25177136

     25181647, 25693368, 26650230, 27333909, 27412572, 27454558, 27570740

     27578454, 27713280, 27713320, 27839431, 27846706, 28128298, 28179003

     28324896, 28361985, 28373191, 28411455, 28460624, 28517373, 28581435

     28629570, 28705938, 28709611, 28818965, 28878198, 28893677, 28912243

     29197309, 29198801, 29279156, 29285826, 29286452, 29305336, 29305347

     29349853, 29473784, 29620912, 29620944, 29635114

Patch  28901325     : applied on Tue Apr 23 08:36:49 MDT 2019

Unique Patch ID:  22605292

Patch description:  “One-off”

   Created on 30 Nov 2018, 21:05:48 hrs PST8PDT

   Bugs fixed:

     28901325

Patch  26081565     : applied on Tue Apr 23 08:35:28 MDT 2019

Unique Patch ID:  21885885

Patch description:  “One-off”

   Created on 19 Jan 2018, 08:12:44 hrs PST8PDT

   Bugs fixed:

     26081565

Patch  26404239     : applied on Tue Apr 23 08:33:47 MDT 2019

Unique Patch ID:  21885962

Patch description:  “One-off”

   Created on 18 Jan 2018, 21:09:57 hrs PST8PDT

   Bugs fixed:

     26404239

Patch  24950713     : applied on Tue Apr 23 08:24:45 MDT 2019

Unique Patch ID:  22708973

Patch description:  “One-off”

   Created on 29 Jan 2019, 08:18:55 hrs PST8PDT

   Bugs fixed:

     24950713

   This patch overlays patches:

     29142661

   This patch needs patches:

     29142661

   as prerequisites

Patch  29142661     : applied on Wed Apr 17 12:22:50 MDT 2019

Unique Patch ID:  22643444

Patch description:  “SOA Bundle Patch 12.2.1.3.0(ID:181223.0212.0069)”

   Created on 23 Dec 2018, 12:57:19 hrs PST8PDT

   Bugs fixed:

     24922173, 24971871, 25941324, 25980718, 26031784, 26372043, 26385451

     26401629, 26408150, 26416702, 26472963, 26484903, 26498324, 26536677

     26571201, 26573292, 26644038, 26645118, 26669595, 26696469, 26720287

     26739808, 26796979, 26851150, 26868517, 26869494, 26895927, 26935112

     26947728, 26953820, 26957074, 26957183, 26982712, 26997999, 27018879

     27019442, 27024693, 27030883, 27073918, 27078536, 27119541, 27141953

     27150210, 27157900, 27171517, 27210380, 27230444, 27241933, 27247726

     27260565, 27268787, 27311023, 27368311, 27379937, 27411143, 27429480

     27449047, 27486624, 27494478, 27561639, 27627502, 27633270, 27639691

     27640635, 27651368, 27653922, 27656577, 27708766, 27708925, 27715066

     27767587, 27785937, 27832726, 27876754, 27879887, 27880006, 27929443

     27932274, 27940458, 27957338, 28000870, 28034163, 28035648, 28042548

     28053563, 28067002, 28096509, 28163159, 28178811, 28178850, 28265638

     28290635, 28317024, 28324134, 28368230, 28389624, 28392941, 28448109

     28468835, 28597768, 28620247, 28632418, 28702757, 28808901, 28901363

     29005814

Patch  28928412     : applied on Mon Jan 28 13:14:33 MST 2019

Unique Patch ID:  22610612

Patch description:  “WebCenter Content Bundle Patch 12.2.1.3.190115”

   Created on 14 Dec 2018, 02:53:41 hrs PST8PDT

   Bugs fixed:

     16546231, 17278216, 21443677, 23526550, 23567875, 23717512, 24660722

     25051178, 25228941, 25311639, 25357798, 25605764, 25606440, 25801227

     25822038, 25858327, 25885770, 25928125, 25928588, 25979019, 25985875

     26075990, 26105301, 26185222, 26228118, 26283098, 26300787, 26358746

     26415656, 26430590, 26545951, 26574381, 26576630, 26586426, 26596903

     26636302, 26723147, 26732710, 26786056, 26813909, 26820528, 26847632

     26890620, 26893963, 26954901, 27020230, 27065201, 27099662, 27102908

     27119372, 27140730, 27190092, 27190553, 27193483, 27206340, 27233223

     27254464, 27314625, 27319352, 27346199, 27365218, 27383350, 27383732

     27390329, 27396349, 27406356, 27453228, 27457939, 27458003, 27496856

     27502500, 27507189, 27547665, 27574477, 27608152, 27620996, 27648991

     27661839, 27744442, 27771468, 27801161, 27804618, 27814273, 27824132

     27839174, 27877814, 27879502, 27916698, 27921859, 27943295, 27983987

     27984425, 28043459, 28048684, 28098831, 28165088, 28180857, 28185865

     28225141, 28295718, 28302949, 28317851, 28319312, 28378394, 28380642

     28405721, 28425934, 28452764, 28475951, 28481653, 28485796, 28486569

     28556894, 28593461, 28621910, 28635203, 28651169, 28663117, 28704291

     28707740, 28798285, 28872073, 28872314, 28889421, 29011518

Patch  28278427     : applied on Fri Aug 17 08:15:59 MDT 2018

Unique Patch ID:  22374151

Patch description:  “One-off”

   Created on 6 Aug 2018, 05:40:17 hrs PST8PDT

   Bugs fixed:

     28278427

Patch  26355633     : applied on Thu Mar 29 12:51:10 MDT 2018

Unique Patch ID:  21447583

Patch description:  “One-off”

   Created on 1 Aug 2017, 21:40:20 hrs UTC

   Bugs fixed:

     26355633

Patch  26287183     : applied on Thu Mar 29 12:50:58 MDT 2018

Unique Patch ID:  21447582

Patch description:  “One-off”

   Created on 1 Aug 2017, 21:41:27 hrs UTC

   Bugs fixed:

     26287183

Patch  26261906     : applied on Thu Mar 29 12:50:32 MDT 2018

Unique Patch ID:  21344506

Patch description:  “One-off”

   Created on 12 Jun 2017, 23:36:08 hrs UTC

   Bugs fixed:

     25559137, 25232931, 24811916

Patch  26051289     : applied on Thu Mar 29 12:50:26 MDT 2018

Unique Patch ID:  21455037

Patch description:  “One-off”

   Created on 31 Jul 2017, 22:11:57 hrs UTC

   Bugs fixed:

     26051289

Noted Patch Exceptions

The above listing doesn’t leverage the latest bundle patches for SOA nor WebLogic Server as there were overlay patches with dependencies on the bundle that had yet to be released.  Monitor the release of patches 24950713 and 25830131 for inclusion of the latest bundle release.

Conclusion

TekStream has performed the 12.2.1.3 FIPSA upgrade and worked through the issues necessary to restore full functionality on the new baseline.  

Have questions or need assistance with your upgrade? Contact us today!

[pardot-form id=”17931″ title=”Blog – John Schleicher – AXF 12C UPGRADE PATCHES AND FIPSA COMPONENTS”]


[1] Application eXtension Framework