Back to the Present: Fixing Incorrect Timestamps in Splunk

  By: Jay Young  | Senior Splunk Consultant

 

It is not uncommon, in large and small Splunk Enterprise environments, to have events with future or past timestamps. With time being a critical component of Splunk, incorrect timestamps can severely impact the hot and warm buckets on the indexers; hot buckets may roll too early, before they meet the set size of the attribute maxDataSize(default size 750mb), creating non-uniform-sized warm buckets.

The excellent news is that Splunk has added IndexAttributes in indexes.conf. These attributes are quarantinePastSecs and quarantineFutureSecs to support the inspection of time at the indexing tier. These two IndexAttributes help quarantine events to better manage the flow of time throughout all indexes.

The quarantine constraints detect future and past events with varying degrees of time as they get indexed. If the indexers encounter events with timestamps that exceed these boundaries, it sends them to a separate hot bucket called the hot quarantine bucket; this bucket is located in the exact location as the primary hot bucket and is identified by “hot_quan_vx_xxx.”

The quarantinePastSecs and quarantineFutureSecs both have default settings in Splunk and should not be altered in the default indexes.conf file; they do allow for control at the individual index level and can be adjusted to fit a single indexes allowable time range.

The two Indexes.conf.IndexAttributes:

  • quarantinePastSecs = <positive integer>
  • quarantineFutureSecs = <positive integer>

 

By default, these two IndexAttributes are set at 30 days for quarantineFutureSecs and 900 Days for quarantinePastSecs. Both IndexAttributes’ accepted values are calculated in seconds. These values determine the range for acceptable future and past events.

Common Issues That Cause Future and Past Timestamps

  1. Improperly configured attributes in props.conf.
  2. Having different timezone events sent to the indexers.
  3. Events are delayed and then get sent to the indexers.
  4. System turned off for extended period with no time server configured on bootup.
  5. When the time changes in the spring and fall, excluding Arizona. 😊

How to Check for Future and Past Timestamps

Example: This is a quick way to identify indexes with future or past timestamps.

Name Action Type App Current Size Max Size Event Count Earliest Event Lastest Event
main Edit Delete Disable Events Search 204 MB 500 GB 22.6k In 3 months In 13 hours

Splunk > Settings > Indexes >

The Lastest Event column in the Splunk Index administration page shown in the example above shows that events will be current time events “in 13 hours.” This means at this time, they are events in the future. For example: if the present time is midnight, then at 1 p.m., the events in the hot quarantine bucket would be eligible to roll to warm buckets as they have passed the present time of now; but until the time passes, the future events will be kept in the Hot Quarantine bucket on the indexer.

The Earliest Event Column is the oldest event currently Indexed in the index.

Example Queries:

  • Query for small environments: index=* earliest=+5m latest=+5y
  • Query for larger environments: index=(name) sourcetype=(name) earliest=+5m latest=+5y

Understanding _indextime vs. _time the Parsed One

In Splunk, there are two different times used. Events in Splunk are not generally received at the same time as indicated in the event timestamp; the difference is usually a few seconds from the indexer arrival time to the event timestamp. The actual arrival time is written to _indextime, and the timestamp embedded in the event is parsed and stored in _time. When searching in Splunk, 99.999% of the time you will be searching against the _time parsed from the event. Future and past timestamps would be tough to get in _indextime and would be a server date and time issue instead of event time issues.

Example Query to find Indexing Latency:

  • index=(name) | eval time=_time | eval indextime=_indextime | eval latency=(indextime-time) | stats count by avg(latency), min(latency), max(latency) by sourcetype

Bucket Naming Examples Normal and Future

In the example below, the lastest timestamp is the newest event in the warm bucket; this can also be described as the last event to get indexed before the hot bucket rolls to warm. The earliest timestamp is the oldest in the warm bucket; this is the first event to go into the newly created hot bucket. When the hot bucket rolls, the warm bucket gets created, then gets appended with epoch timestamps to represent the two values, earliest and lastest.

Epoch timestamp converter: https://www.epochconverter.com/

Example: Normal warm bucket.

EARLIEST TIMESTAMP 1. DST Monday, August 16, 2021 11:42:24 AM GMT-05:00
                                                                                     1.earliest         2.Latest

current warm bucket with epoch time — db_1629961138_1629132144_95
LASTEST TIMESTAMP 2. Thursday, August 26, 2021 1:58:58 AM GMT-05:00

Example: What would happen if a future timestamped event is allowed to roll into a warm bucket?

EARLIEST TIMESTAMP 1. Tuesday, August 24, 2021 11:42:24 AM GMT-05:00 DST
                                                     1.earliest        2.latest

current warm bucket — db_1756166400_1629823344_97
LASTEST TIMESTAMP 2. Tuesday, August 26, 2025 12:00:00 AM GMT-05:00 DST

The above example is a warm bucket with a future timestamp rolled into it. This bucket now has a timestamp that is four years in the future. This future timestamp would cause this warm bucket to not roll into a cold bucket or be removed for four years plus the current retention policy. This type of future bucket would continuously be searched for the next four years, affecting search performance.

Using Splunk without these two IndexAttributes would cause warm buckets to sit on the indexers waiting to age out until the present time pasted the Lastest event timestamp, causing it to be continuously searched from scheduled and ad-hoc searches due to the lastest time being in the future. If this process was allowed to continue, your Splunk indexers could potentially reach the default number of warm buckets (“maxWarmDBCount=300”) and have hundreds of warm buckets sitting on the indexers with data that should have aged out a long time ago.

Indexes.conf indexAttribute reference:

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf

Contact us for more help on managing your Splunk environment!