facebook twitter youtube facebook facebook facebook

E-Mail : info@askmlabs.com

Phone : +1.215.353.8306

Home » , , , , , » Control the index data retention in Splunk - Purge and Claim disk space

Control the index data retention in Splunk - Purge and Claim disk space

Written By askMLabs on Sunday, August 2, 2020 | 12:03 PM

This article is about index data retention of Splunk. Splunk is a too that is used for searching, monitoring, and analyzing machine-generated big data via a Web-style interface. 

If you keep indexing your data, all the indexed data will be stored in index. It will keep increasing your disk space. At some point, you need to think of data retention to save the disk space. The default value of data retention is "188697600" seconds(Apprx 6 Yrs). Keeping the historical data depends on the use case. If you need 6 Yrs worth of historical data, you can keep the settings same and estimate the disk space requirements appropriately.

In our specific case, I don't need 6 years worth of indexed data. Becase I know the specific use case of our data, I can decide the retention of this data. Lets assume, I have to set the retention of the indexed data to 30days.  

30days = 30*24*60*60 seconds = 2592000 seconds

The configuration file where we can set default retention that applies to all the indexes in the Splunk is /opt/splunk/etc/system/default/indexes.conf. And the configuration parameter that controls the retention period is "frozenTimePeriodInSecs"


Following steps should help you setting this parameter.....

Check the disk space....( My splunk indexes are using  /opt/splunk for storing indexed data).


 [root@askmlabs-splunk01 ~]# df -h /opt/splunk
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/data-data
                       493G  416G   52G  90% /opt/splunk
 [root@askmlabs-splunk01 ~]# 



Modify the parameter frozenTimePeriodInSecs in file /opt/splunk/etc/system/default/indexes.conf

NOTE : There are multiple places that this parameter is specified in the indexes.conf file. You need to modify the parameter under the section named  "index specific defaults".

[root@askmlabs-splunk01 default]# diff indexes.conf indexes.conf_bak
42c42
< frozenTimePeriodInSecs = 2592000
---
> frozenTimePeriodInSecs = 188697600
[root@askmlabs-splunk01 default]#


Now restart the Splunk instance to take the value effective....

[root@askmlabs-splunk01 default]# /opt/splunk/bin/splunk restart

Check if the disk space has been reduced by changing the retention....

 [root@askmlabs-splunk01 default]# df -h /opt/splunk
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/data-data
                       493G  224G  244G  48% /opt/splunk
 [root@askmlabs-splunk01 default]#


Conclusion :
Splunk indexed data retention can be controlled using the parameter  frozenTimePeriodInSecs in the configuration file /opt/splunk/etc/system/default/indexes.conf. 

Hope this information helps you. Please post your questions in the comments section.


Thanks
SRI



Share this article :

Related Articles By Category



Post a Comment

Thank you for visiting our site and leaving your valuable comment.

 
Support :
Copyright © 2013. askMLabs - All Rights Reserved
Proudly powered by Blogger