If you keep indexing your data, all the indexed data will be stored in index. It will keep increasing your disk space. At some point, you need to think of data retention to save the disk space. The default value of data retention is "188697600" seconds(Apprx 6 Yrs). Keeping the historical data depends on the use case. If you need 6 Yrs worth of historical data, you can keep the settings same and estimate the disk space requirements appropriately.
In our specific case, I don't need 6 years worth of indexed data. Becase I know the specific use case of our data, I can decide the retention of this data. Lets assume, I have to set the retention of the indexed data to 30days.
30days = 30*24*60*60 seconds = 2592000 seconds
The configuration file where we can set default retention that applies to all the indexes in the Splunk is /opt/splunk/etc/system/default/indexes.conf. And the configuration parameter that controls the retention period is "frozenTimePeriodInSecs".
Following steps should help you setting this parameter.....
Check the disk space....( My splunk indexes are using /opt/splunk for storing indexed data).
[root@askmlabs-splunk01 ~]# df -h /opt/splunk
Filesystem Size Used Avail Use% Mounted on
493G 416G 52G 90% /opt/splunk
Modify the parameter frozenTimePeriodInSecs in file /opt/splunk/etc/system/default/indexes.conf
NOTE : There are multiple places that this parameter is specified in the indexes.conf file. You need to modify the parameter under the section named "index specific defaults".
[root@askmlabs-splunk01 default]# diff indexes.conf indexes.conf_bak42c42< frozenTimePeriodInSecs = 2592000---> frozenTimePeriodInSecs = 188697600[root@askmlabs-splunk01 default]#
Now restart the Splunk instance to take the value effective....
[root@askmlabs-splunk01 default]# /opt/splunk/bin/splunk restart
Check if the disk space has been reduced by changing the retention....
[root@askmlabs-splunk01 default]# df -h /opt/splunkFilesystem Size Used Avail Use% Mounted on/dev/mapper/data-data493G 224G 244G 48% /opt/splunk[root@askmlabs-splunk01 default]#
Splunk indexed data retention can be controlled using the parameter frozenTimePeriodInSecs in the configuration file /opt/splunk/etc/system/default/indexes.conf.
Hope this information helps you. Please post your questions in the comments section.