Wrangling Your Splunk Indexer Storage
By: Jon Walthour | Senior Splunk Consultant, Team Lead
I have heard many customers, vexed by overflowing storage attached to their indexers, wonder why warm buckets are not rolling to a separate cold storage filesystem when they’re supposed to. Instead, their hot/warm volume on their indexers fills up and indexers are entering automatic detention because the filesystem has less than minFreeSpace MB available. They’ve reviewed their configuration in indexes.conf. They’ve set up volumes with maxVolumeDataSizeMB set to a value 90-95% the size of the filesystem on which they are storing hot and warm buckets on the indexers. And yet, the problem continues. So, they come to me asking why Splunk is broken.
I tell them Splunk isn’t broken; it just doesn’t work the way they expect it to. To get accurate management of hot-warm storage, define every path by a volume, then use maxVolumeDataSizeMB to have the Volume Manager keep the size under control. Two keys:
- Define every “Path” using a volume. This includes:
- – homePath
- – coldPath
- – summaryHomePath (for report accelerations)
- – tsidxStatsHomePath (for data model accelerations)
- Create three filesystems for classic storage and two filesystems for SmartStore indexing tiers to cover all the bases:
- – /opt/splunk – for Splunk software, logs, temp files, etc.
- – hot/warm volume
- – cold volume (not needed for SmartStore)
The key that usually makes managing hot/warm storage mysterious is that most folks don’t realize that data model accelerations default to “volume:_splunk_summaries/$_index_name/datamodel_summary.” The “_splunk_summaries” volume defaults to “path = $SPLUNK_DB.” The way things get set up usually, SPLUNK_DB gets set to the same base directory as homePath. So, the data model summaries end up as a subdirectory of homePath/. However, the way the volume manager calculates storage is not “df -h,” but more like adding up all the “du -sk” calculations on all the directories where it is defined, usually homePath and coldPath. As such, datamodel summaries do not get added in and Splunk admins find that their hot/warm volumes get filled up and they can’t figure out why. Instead, define everything in the hot-warm with the hot-warm volume to get accurate management. For those settings that cannot use volume settings (e.g., thaweddb and tsidxStatsHomePath), use SPLUNK_DB and set it to a separate filesystem.
Recommendations
- ⁃ bloomHomePath has no default. The indexer stores bloomfilter files for the index inline, inside index bucket directories, using a small amount of storage. So it’s not worth separately defining this path as, by default, gets counted in homePath and coldPath.
- ⁃ $SPLUNK_DB can be left to its default of $SPLUNK_HOME/var/lib/splunk or can be set to a separate filesystem. It does not need to be very large. It is used for tsidxStatsHomePath, which is an indexer-wide setting and, by default, set to $SPLUNK_DB/tsidxstats, and for thawedPath, which in most Splunk deployments is rarely used.
indexes.conf
[default]
homePath=volume:hot-warm/$_index_name/db
coldPath=volume:hot-warm/$_index_name/colddb
thawedPath=$SPLUNK_DB/$_index_name/thaweddb
summaryHomePath=volume:hot-warm/$_index_name/summary
tstatsHomePath=volume:hot-warm/$_index_name/datamodel_summary
[volume:hot-warm]
path = /splunkdata/hot_warm
maxVolumeDataSizeMB = #########
[volume:cold-thawed]
path = /splunkdata/cold_thawed
maxVolumeDataSizeMB = #########
[main]
homePath = volume:hot-warm/defaultdb/db
coldPath = volume:hot-warm/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
summaryHomePath = volume:hot-warm/defaultdb/summary
tstatsHomePath = volume:hot-warm/defaultdb/datamodel_summary
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume
[history]
homePath = volume:hot-warm/historydb/db
coldPath = volume:hot-warm/historydb/colddb
thawedPath = $SPLUNK_DB/historydb/thaweddb
summaryHomePath = volume:hot-warm/historydb/summary
tstatsHomePath = volume:hot-warm/historydb/datamodel_summary
maxDataSize = 10
frozenTimePeriodInSecs = 604800
[summary]
homePath = volume:hot-warm/summarydb/db
coldPath = volume:hot-warm/summarydb/colddb
thawedPath = $SPLUNK_DB/summarydb/thaweddb
summaryHomePath = volume:hot-warm/summarydb/summary
tstatsHomePath = volume:hot-warm/summarydb/datamodel_summary
[_internal]
homePath = volume:hot-warm/_internaldb/db
coldPath = volume:hot-warm/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
summaryHomePath = volume:hot-warm/_internaldb/summary
tstatsHomePath = volume:hot-warm/_internaldb/datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000
[_audit]
homePath = volume:hot-warm/audit/db
coldPath = volume:hot-warm/audit/colddb
thawedPath = $SPLUNK_DB/audit/thaweddb
summaryHomePath = volume:hot-warm/audit/summary
tstatsHomePath = volume:hot-warm/audit/datamodel_summary
[_thefishbucket]
homePath = volume:hot-warm/fishbucket/db
coldPath = volume:hot-warm/fishbucket/colddb
thawedPath = $SPLUNK_DB/fishbucket/thaweddb
summaryHomePath = volume:hot-warm/fishbucket/summary
tstatsHomePath = volume:hot-warm/fishbucket/datamodel_summary
maxDataSize = 500
frozenTimePeriodInSecs = 2419200
# this index has been removed in the 4.1 series, but this stanza must be
# preserved to avoid displaying errors for users that have tweaked the index's
# size/etc parameters in local/indexes.conf.
#
[splunklogger]
homePath = volume:hot-warm/splunklogger/db
coldPath = volume:hot-warm/splunklogger/colddb
thawedPath = $SPLUNK_DB/splunklogger/thaweddb
disabled = true
[_introspection]
homePath = volume:hot-warm/_introspection/db
coldPath = volume:hot-warm/_introspection/colddb
thawedPath = $SPLUNK_DB/_introspection/thaweddb
summaryHomePath = volume:hot-warm/_introspection/summary
tstatsHomePath = volume:hot-warm/_introspection/datamodel_summary
maxDataSize = 1024
frozenTimePeriodInSecs = 1209600
[_telemetry]
homePath = volume:hot-warm/_telemetry/db
coldPath = volume:hot-warm/_telemetry/colddb
thawedPath = $SPLUNK_DB/_telemetry/thaweddb
summaryHomePath = volume:hot-warm/_telemetry/summary
tstatsHomePath = volume:hot-warm/_telemetry/datamodel_summary
maxDataSize = 256
frozenTimePeriodInSecs = 63072000
[_metrics]
homePath = volume:hot-warm/_metrics/db
coldPath = volume:hot-warm/_metrics/colddb
thawedPath = $SPLUNK_DB/_metrics/thaweddb
summaryHomePath = volume:hot-warm/_metrics/summary
tstatsHomePath = volume:hot-warm/_metrics/datamodel_summary
datatype = metric
#14 day retention
frozenTimePeriodInSecs = 1209600
metric.splitByIndexKeys = metric_name
# Internal Use Only: rollup data from the _metrics index.
[_metrics_rollup]
homePath = volume:hot-warm/_metrics_rollup/db
coldPath = volume:hot-warm/_metrics_rollup/colddb
thawedPath = $SPLUNK_DB/_metrics_rollup/thaweddb
summaryHomePath = volume:hot-warm/_metrics_rollup/summary
tstatsHomePath = volume:hot-warm/_metrics_rollup/datamodel_summary
datatype = metric
# 2 year retention
frozenTimePeriodInSecs = 63072000
metric.splitByIndexKeys = metric_name
Want to learn more about managing your Splunk Indexer storage? Contact us today!