Deep Freeze Your Splunk Data in AWS

/0 Comments/in , , /by

By: Zubair Rauf | Splunk Consultant

 

In today’s day and age, storage has become a commodity, but, even now, reliable high-speed storage comes at a substantial cost. For on-premise Splunk deployments, Splunk recommends RAID 0 or 1+0 disks capable of at least 1200 IOPS and this increases in high-volume environments. Similarly, in bring-your-own-license cloud deployments, customers prefer to use SSD storage with at least 1200 IOPS or more.

Procuring these disks and maintaining them can carry a hefty recurring price tag. Aged data, that no longer needs to be accessed on a daily basis but has to be stored because of corporate governance policies or regulatory requirements, can effectively increase the storage cost for companies if done on these high-performance disks.

This data can securely be moved to Amazon Web Services (AWS) S3, S3 Glacier, or other inexpensive storage options of the Admin’s choosing.

In this blog post, we will dive into a script that we have developed at TekStream which can move buckets from Indexer Clusters to AWS S3 seamlessly, without duplication. It will only move one good copy of the bucket and ignore any duplicates (replicated buckets).

During the process of setting up indexes, Splunk Admins can decide and set data retention on a per-index basis by setting the ‘frozenTimePeriodInSecs’ setting in indexes.conf. This allows the admins to be flexible on their retention levels based on the type of data. Once the data becomes of age, Splunk deletes it or moves it to frozen storage.

Splunk achieves this by referring to the coldToFrozenScript setting in indexes.conf. If a coldToFrozenScript is defined, Splunk will run that script; once it successfully executes without problems, Splunk will go ahead and delete the aged bucket from the indexer.

The dependencies for this script include the following;

–   Python 2.7 – Installed with Splunk

–   AWS CLI tools – with credentials already working.

–   AWS Account, Access Key and Secret Key

–   AWS S3 Bucket

Testing AWS Connectivity

After you have installed AWS CLI and set it up with the Secret Key and Access Key for your account, test connectivity to S3 by using the following command:

Note: Please ensure that the AWS CLI commands are installed under /usr/bin/aws and the AWS account you are using should have read and write access to S3 artifacts.

If AWS CLI commands are set-up correctly, this should return a list of all the S3 buckets in your account.

I have created a bucket titled “splunk-to-s3-frozen-demo”.

Populate the Script with Bucket Name

Once the S3 bucket is ready, you can copy the script to your $SPLUNK_HOME/bin folder. After copying the script, edit it and change the name of your S3 Bucket where you wish to freeze your buckets.

Splunk Index Settings

After you have made the necessary edits to the script, it is time to update the settings on your index in indexes.conf.

Depending on where your index is defined, we need to set the indexes.conf accordingly. On my demo instance, the index is defined in the following location:

In the indexes.conf, my index settings are defined as follows;

 

Note: These settings are only for a test index, that will roll any data off to frozen (or delete if a coldToFrozenScript is not present) after 600 seconds.

Once you have your settings complete in indexes.conf, please restart your Splunk instance. Splunk will read the new settings at restart.

After the restart, I can see my index on the Settings > Indexes page.

Once the index is set up, I use the Add Data Wizard to add some sample data to my index. Ideally, this data should roll over to warm, and the script should be moved to my AWS S3 bucket after 10 minutes.

The remote path on S3 will be set up in the following order:

If you are running this on an indexer cluster, the script will not copy duplicate buckets. It will only copy the first copy of a bucket and ignore the rest. This helps manage storage costs and does not keep multiple copies of the same buckets in S3.

Finally, once the script runs successfully, I can see my frozen Splunk bucket in AWS S3. If you are running this on an indexer cluster, the script will not copy duplicate buckets. It will only copy the first copy of a bucket and ignore the rest. This helps manage storage costs and does not keep multiple copies of the same buckets in S3.

Note: This demo test was done on Splunk Enterprise 8.0 using native Python 2.7.1 that ships with Splunk Enterprise. If you wish to use any other distribution on Python, you will have to modify the script to be compatible.

If there is an error and the bucket does not transfer to S3, or it is not deleted from the source folder, then you can troubleshoot it with the following search:

This search will show you the stdout error that is thrown when the script runs into an error.

To wrap it up, I would highly recommend that you do implement this in a dev/sandbox environment before rolling it out into production. Doing so will ensure that it is robust for your environment and make you comfortable with the set-up.

To learn more about how to set-up AWS CLI Tools for your environment, please refer to the following link; https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

If you have any questions or are interested in getting the script, contact us today!

Sneak Peek Into Our Approach to Migrating Splunk On-Prem to AWS How to Migrate Splunk On-Prem to AWS

/0 Comments/in , , /by

Splunk on AWS offers a special kind of magic. Splunk makes it simple to collect, analyze and act on data of all kinds. AWS applications for Splunk allow users to ingest that data quickly, with visualization tools, dashboards, and alerts. Together, they help organizations see through the noise. When a notable event (such as a potential breach) occurs, you can find and act on it quickly, making the combo a powerful tool for risk management. 

Running Splunk on the cloud gives organization resiliency, with all the advantages of scalability, flexibility, cost-optimization, and security.  Yet migrating to the cloud poses many challenges and implementing the new system alone can be intimidating, costly, and time-consuming if not done correctly. AWS database migration services are available to mitigate the impact of this necessary shift for your business. 

Finding an experienced and expert AWS and combined Splunk managed services partner to help you navigate can ease the process. Here’s a quick look into how to handle the change from Splunk On-Prem to AWS.

How Splunk Licensing Works

Each of your Splunk Enterprise instances require a license, which specifies how many gigabytes per day a given Splunk Enterprise instance can index and which features you have access to. Multiple types of licenses are available, and distributed deployments, consisting of multiple instances, require a few extra steps. 

Choosing the correct Splunk licensing option can be confusing. It requires outlining the types of business problems you wish to solve with Splunk, then estimating how much data usage you will need to perform this work over time.

Finding a Partner with Licensing Expertise

Non-compliance with licensing can lead to overages and penalties. As your advisory partner, TekStream can work with you to ensure that your Splunk and AWS licenses are in order.

TekStream has extensive experience navigating the specifics of complex license structures and contracts. Our Splunk Enterprise consultants will leverage their years of experience to help you assess your needs, accurately estimate your data usage, and determine the optimal license types and quantities for your unique needs.

In addition to Splunk licensing for new implementations, TekStream will also help your organization save money on licensing renewals. We will examine your Splunk usage to date, pinpoint areas where you may be overpaying, and provide you with viable alternatives to reduce your costs without sacrificing efficiency.

The 4 Most Common Licensing Structures

When selecting a licensing structure for Splunk on AWS, there are 4 main options. The best option will vary depending on the organization. Through careful analysis of your current licensing structures and your desired future state, we will work with you to determine the optimal licensing structure.  

Option 1: Migrate Your Existing Perpetual or Term License to AWS

Option 2: Convert Your Current License to Splunk Cloud (which would run on AWS)

Option 3: Convert to a Term or Infrastructure License (if on a Perpetual License)

Option 4: Pay-As-You-Go as part of a 3rd-Party Hosted MSP Solution

Each option has its pros and cons depending on an organization’s goals and usage. A partner can help you select the best option. TekStream’s deep experience overseeing complex data migrations empowers us to act as true consultative partners. We have the experience needed to quickly scope challenges and present solutions for your unique situation.

Take the risk out of your Splunk migration to AWS. We are so confident in our battle-tested strategy and proven database migration process that we guarantee that your database migration will be completed on-time and on-budget (when using TekStream’s Proven Process). We also guarantee optimal and cost-effective license and cloud subscriptions. 

 

Download the
Ultimate Guide

To find out more specifics about our proven process and get an in-depth look into our services, read The Ultimate Guide to Migrating Your Splunk On-Prem to Amazon Web Services. 

Take your Traditional OCR up a notch

/0 Comments/in , /by

By: Greg Moler | Director of Imaging Solutions

While the baseline OCR landscape has not changed much, AWS aims to correct that. Traditional OCR engines are quite limited in what details they can provide. Being able to detect the characters is only half the battle, the ability to get meaningful data out of them becomes the challenge. Traditional OCR follows the ‘what you see is what you get’ mantra, meaning once you run your document through, the blob of seemingly unnavigable text is all you are left with. What if we could enhance this output with other meaningful data elements useful in extraction confidence? What if we could improve the navigation of the traditional OCR block of text?

Enter Textract from AWS. A public web service aimed to improve your traditional OCR experience in an easily scalable, integrable, and low cost package. Textract is built upon an OCR extraction engine that is optimized by AWS’ advanced machine learning. It has been taught how to extract thousands of different types of forms so you don’t have to worry about it. The ‘template’ days are over. It also provides a number of useful advanced features that other engines simply do not offer: confidence ratings, word block identification, word and line object identification, table extraction, and key-value output. Let’s take a quick look at each of these:

  • Confidence Ratings: Ability to intelligently make choices to accept results, or require human intervention based on your own thresholds. Building this into your work flow or product can greatly improve data accuracy
  • Word Blocks: Textract will identify word blocks allowing you to navigate through them to help identify things like address blocks or known blocks of text in your documents. The ability to identify grouped wording rather than sifting through a massive blob of OCR output can help you find the information you are looking for faster
  • Word and Line Objects: Rather than getting a block of text from a traditional OCR engine, having code-navigable objects to parse your documents will greatly improve your efficiency and accuracy. Paired with location data, you can use the returned coordinates to pinpoint where it was extracted from. This becomes useful when you know your data is found in specific areas or ranges of a given document to further improve accuracy and filter out false positives
  • Table Extraction: Using AWS AI-backed extraction technology, Table extraction will intelligently identify and extract tabular data to pipe into whatever your use case may need, allowing you to quickly calculate and navigate these table data elements.
  • Key-value Output: AWS, again using AI-backed extraction technology, will intelligently identify key-value pairs found on the document without having to write custom engines to parse the data programmatically. Optionally, send these key-value pairs to your favorite key-value engine like Splunk or Elasticsearch (Elastic Stack) for easily searchable, trigger-able, and analytical actions for your document’s data.

Contact us today to find out how Textract from AWS can help streamline your OCR based solutions to improve your data’s accuracy!

Considerations for Moving From On-Prem to Cloud

/in , , , /by

Experts today are continually barraged with data about the cloud. It appears to be each different business is using cloud-based programming, leaving those as yet utilizing on-premise arrangements thinking about whether they, as well, should switch. Organizations are rushing to cloud arrangements on the grounds that there are numerous a large advantages than there are with on-prem arrangements. Here are some of the regularly said reasons cloud setups are better.

COST EFFECTIVE

Cloud arrangement suppliers by and large charge some kind of month to month expense.  This rate might be paid every year or month to month and can either be for each client cost or a cost that incorporates a set scope of records. In return for this charge, you’ll have the capacity to set up accounts until the point when you achieve the most extreme, overseeing secret word resets and record evacuations and augmentations utilizing an authoritative gateway.

Rather than depending on CDs or a site download to introduce the product on every gadget, you’ll have programming that is prepared to utilize. Permitting charges are incorporated into the price tag, so your IT group will never again need to stay aware of your product licenses to ensure the greater part of your introduced programming has been obtained.

TECHNICAL SKILLS

With such huge numbers of private companies and new companies in the business world today, technical support is no longer a choice. A SMB for the most part can’t bear the cost of a full-time IT bolster individual, not to mention the high cost of a server chairman. This implies depending on neighborhood organizations to offer help on an as-required premise, which can accompany a heavy for every hour sticker price. Along these lines, the organizations that do have on-introduce programming will regularly depend on remote help, which is outsourced by means of the cloud.

With cloud programming, technical support is generally taken care of by the supplier, regardless of whether by telephone, email, or an assistance work area ticket. These suppliers have the wage base to pay the high pay rates instructed by the present best IT experts, both at the server level and at the client bolster organize. Most smaller organizations essentially couldn’t manage the cost of this kind of skill all the time.

SCALABILITY

Each strategy for success to develop after some time and cloud programming offers the versatility required to deal with that development. At the point when another representative joins its staff, a business utilizing cloud programming can essentially add another client to its record administration. At the point when an organization maximizes its logins, a higher-level record can as a rule be requested with negligible exertion with respect to the business.

Another advantage to cloud arrangements is that they for the most part include new highlights that normal on-premise setups do not include. As clients express an enthusiasm for having the capacity to accomplish more with their product, suppliers include these highlights, making them accessible either consequently or with a discretionary record change. Cloud arrangements are additionally consistently endeavoring to work with other programming applications and these combinations make it simpler for organizations to deal with everything in one place.

AVAILABILITY

The present workforce is progressively portable, telecommuting, inn rooms, coffeeshops, and air terminals. Cloud programming implies that these laborers can get to their documents wherever they are, utilizing a portable workstation, cell phone, or tablet. This implies even while in the midst of some recreation, groups can keep in contact, keeping ventures pushing ahead through the cloud.

A standout amongst other things about cloud arrangements is that experts never again need to make sure to bring records with them when they leave the workplace. An introduction can be conveyed specifically from a client’s cell phone. Applications that handle charging, cost assessing, and venture administration can be gotten to amid gatherings, enabling participants to get the data they require without influencing everybody to hold up until the point when the meeting is finished and everybody has come back to their workplaces.

RELIABILITY

In the event that you’ve at any point endured a server blackout, you know how destructive it can be on an assortment of levels. Your representatives are compelled to either wait around, sitting tight for the circumstance to be settled, or go home for the day and leave your work environment unmanned. On the off chance that this happens over and over again, you’ll utilize customers and even workers, and in addition hurt your well-deserved notoriety as a business that has become a model of togetherness.

Cloud suppliers look at unwavering quality as a vital piece of their plans of action. They make it their central goal to guarantee clients approach the records and applications they require constantly. In the event that a blackout ever happens, many cloud suppliers have worked in reinforcements to assume control, with clients never mindful an issue has occurred. On the off chance that such reinforcement doesn’t exist, a cloud supplier still approaches specialists who can guarantee frameworks are up considerably more rapidly than a SMB could with an on-preface server.

SECURITY

Security is a continuous worry for organizations, with reports of ruptures getting to be plainly typical. Cloud programming guarantees an abnormal state of security, including information encryption and solid secret word prerequisites. These little things will help protect a business’ information, lessening the danger of a rupture that could cost cash and mischief client confide in an organization.

Organizations that store specific data, for example, medicinal records or financial balance data should scan for a cloud supplier that offers these insurances. There are presently cloud suppliers that have some expertise in HIPAA consistence, for example, so a medicinal practice could profit by the authorities on staff at one of those suppliers who can guarantee that wellbeing information stays safe.

DISASTER RECOVERY

What might happen to your business if a cataclysmic event struck your building or server farm? Imagine a scenario where you came in one morning to discover a fire had rendered your workplaces dreadful. Would you be compelled to close everything down for the term or would your representatives have the capacity to begin working promptly?

Cloud programming enables catastrophe to verification your business, guaranteeing your representatives can telecommute or a brief office if for reasons unknown they can’t work in the workplace. Cloud suppliers typically have reinforcement anticipates their own servers to ensure against calamities, so the product and documents you utilize every day will be available regardless of whether an issue strikes one of their server farms. Before you pick a supplier, don’t hesitate to make inquiries about an organization’s catastrophe arrangements to ensure you’ll be effortless.

Thinking about moving from On-Prem to Cloud? Contact us today!

[pardot-form id=”16001″ title=”Blog- Matt Chumley – Considerations for Moving From On-Prem to Cloud”]