ITSI 4.20+ New Features. What’s in the New Releases?
By Tyler Phillips, Senior Splunk Consultant
1. KPI drift detection: Drift detection is a little bit more than adaptive thresholding. Drift detection monitors over a long period of time and detects if your KPIs are slowly drifting towards the threshold you have set. For example, let’s say you have a CPU Utilization KPI on one of your application servers, and you know that it’s a growing application with more users joining every day. You expect a growth of 2% over each month and will need to add a new server in a few years to help with the load. Instead, you are seeing a growth of 10% over each month and will now need a new server to help with processing in just a few months! Not good…
This type of growth never hits your critical threshold of 80% CPU Utilization, so you weren’t alerted. With drift detection, we can set a one-month look-back period with our expected 2% growth rate and be alerted if it ever grows higher than that, letting us know there is a problem. You can then go into your application and investigate. What do you know? There’s a looping calculation that has been growing exponentially in processing power with every additional calculation run each month. Fix the looping calculation, and the CPU Utilization goes back to the expected percentage.
2. Entity Normalization: Duplicate entities can now be detected and shown in the Configuration Issues Detected section. Now you will be able to see duplicate entities that may need to be reconfigured or removed. Keeping ITSI clean is very important these days. This helps maintain Splunk best practices of not having inactive entities using up space in your environment.
3. Adaptive Thresholding for Entities: Entities can now have adaptive thresholds applied, allowing alerts on a specific server with notifications tailored to that entity.
4. Scheduled Maintenance Windows: Now you can regularly schedule maintenance windows for upgrades to servers or other entities so you no longer receive down alerts or high usage alerts for those being worked on. Just go to Configuration then Maintenance Windows and set frequency to recurring with your desired schedule. You can also select specific services and entities to be a part of this maintenance window so as to maintain monitoring on the rest of your environment.
5. Drilldown links for service template and entity types: You now can set a link to be a part of certain services and entities. Making it easier to drilldown directly to your needed information. Just go into your desired service template, select Settings, Service analyzer navigation links, and click Add link. You can place any drilldown or dashboard link you would like to show.
6. More Adaptive Thresholding: Now Adaptive Thresholding can support up to 100,000 KPIs! There was no defined limit that I could find for before this release, but this still signals a huge achievement and is able to handle much larger environments. Let’s say you have three major KPIs of CPU Utilization, Memory Utilization, and Disk Utilization, with adaptive thresholding turned on. With this you could have 33,000+ services all with adaptive thresholding!
7. Service Analyzer Enhancements: This new feature won’t really be seen, but this will help the backend a lot. ITSI has improved error notifications and reduced unnecessary searches running when you have unused browser tabs and auto refreshes in the Service Analyzer. This will reduce search processing and keep your ITSI cluster ready for other searches.
8. Event iQ: You can now use machine learning to group alerts. Event iQ uses ML algorithms to look at field values and correlate them together into notable events. This feature does require Java version 8 or higher. To apply this you go to Configuration, Event Management, and then to Notable Event Aggregation Policies. You can then select an existing policy or create a new one.
9. Simplified Alert Onboarding: Now it is easier than ever to onboard third party alerts from 3-rd party monitoring tools. This is a somewhat expected feature with the Splunk/Cisco acquisition. Cisco also owns AppDynamics which is a very helpful monitoring tool and being able to use these metrics will come in handy. These are stored in Data integrations and as Alert integrations. The alert integrations are already available for AWS CloudTrail, ThousandEyes, Microsoft SCOM, SolarWinds, AppDynamics, and others!
10. Rules Engine Queue Mode: The previous way this was done was a real-time search that ran with the itsirulesengine command collecting events. Now they are sent to a rules queue and processed more efficiently. To use this feature in an on-prem environment the ports 4222 and 4248 must be open between the search head cluster to process between search heads. There is also a new dashboard to monitor the rules engine and its performance.
11. Backup Enhancements: Backups and Restores now include entity types, service analyzers, related macros, saved searches, and saved episode reviews in them. You can also now receive alerts during the restore to know if there are any missing dependencies in the restore.
12. Data Integrations: New data integrations have been added for Content Pack for AppDynamics, Content Pack for ThousandEyes, Azure Cloud Deployment Support, ITSI Content Pack for Cisco Enterprise Networks, Support for Splunk V2 APIs and ITSI Support for IPv6
About the Author
Tyler Phillips is a Splunk Consultant with a Splunk Core Certified Consultant certification. Tyler is also Splunk IT Service Intelligence Certified Admin accredited along with his Splunk Cloud accreditations. Tyler current learning goal is to receive the Splunk O11y Cloud Certified Metrics Use Certification. Tyler has worked on projects customizing Splunk dashboards, Splunk ITSI, TrackMe, and upgrading Splunk environments to 9.x. He has gained experience with a multitude of Splunk items through the Splunk ODS service. In this service, Tyler has learned the ins and outs of troubleshooting and has gained valuable experience in Splunk problem solving. Tyler resides in Georgia and has received his Information Technology degree from Georgia Southern University.