Elasticsearch indices can quickly fill up with gigabytes of data, especially if you're logging from multiple servers many times a second. To manage data, Elasticsearch
Deleting Using The "Delete By Query" API
Elasticsearch offers a "Delete By Query" API, that will remove all documents matching a query. You can use this to match timestamps greater or less than a certain date, albeit a bit crudely:
POST indexname/_delete_by_query
{
"query": {
"range" : {
"@timestamp" : {
"gte" : "09/02/2020",
"lte" : "11/02/2020",
"format": "dd/MM/yyyy||yyyy"
}
}
}
}
However, this query is really slow. It scales linearly with document size. If you have enough documents that you need to be rotating them to prevent your Elasticsearch instance from bursting into flames, you probably can't delete records this way, and will need to use time-based indices instead.
A Better Method: Time Based Indices
In Elasticsearch, you don't usually use indexes directly. Your dashboards use index patterns, which can match multiple indexes at once. The reason for this is that the indexes themselves can act as groups of data, such as grouping by day or month.
It's much easier to manage and rotate entire indices, so if you had each ingester configured to add the current date to the index name,
index: "indexname-%{+yyyy.MM.dd}"
Of course, this requires you to configure the ingest pipeline to write to the daily index. You'll need to set up your loggers to ingest data in this format.
Once that's done though, you can create a new Index Lifecycle Policy to handle the automatic rollover of data. This option is available under "Stack Management" in the Kibana dashboard.
You can configure multiple phases of index rollover, but for this purpose it's easier to just disable rollover and enable the delete phase, configuring it to remove indices older than X number of days.
Then, to actually apply it to an index template, you'll need to select "Add Policy To Index Template" under "Actions" in the lifecycle policy list.
Select the index pattern you wish to add, and the policy should take effect immediately, and your old indices in the pattern will be deleted.