Schedule Elasticsearch Curator Tasks Natively in Kubernetes
As of Kubernetes 1.21, CronJobs are a stable native Kubernetes feature. Using Kubernetes CronJobs and Elasticsearch Curator, we can schedule periodic changes on an Elasticsearch instance.
Unfortunately, Elasticsearch does not currently publish a native Elasticsearch Curator Docker image. Luckily, however, Bitnami provides such a Docker image, available on DockerHub.
To use the Elasticsearch Curator with a Kubernetes CronJob we need to define two Kubernetes objects, a CronJob and a ConfigMap. We use the ConfigMap to save the configuration of Curator and the task steps we want Curator to take. I have included both the Curator tasks and Curator configuration in the same ConfigMap as separate keys to simplify the setup. However, if it contains sensitive values, the Curator configuration could also be stored as a secret.
An example of the ConfigMap, containing both the Curator configuration and tasks:
apiVersion: v1 kind: ConfigMap metadata: name: curator namespace: logging data: config.yml: | client: hosts: - elasticsearch-master port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False username: password: timeout: 30 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3'] actions.yml: | actions: 1: action: delete_indices description: "Delete indices older than 20 days" options: ignore_empty_list: True filters: - filtertype: age source: creation_date direction: older unit: days unit_count: 20
The CronJob that we need to define will need to mount both the Curator configuration and the task file values stored in our ConfigMap. I have mounted both Curator files in the /curator directory in this example, which is scheduled to run at midnight on the first day of each month:
apiVersion: batch/v1 kind: CronJob metadata: name: curator namespace: logging spec: schedule: "0 0 1 * *" jobTemplate: spec: template: spec: containers: - name: curator image: bitnami/elasticsearch-curator:latest command: ["/bin/sh", "-c"] args: - curator --config /curator/config.yml /curator/actions.yml; volumeMounts: - name: curator mountPath: /curator readOnly: true volumes: - name: curator configMap: name: curator items: - key: "config.yml" path: "config.yml" - key: "actions.yml" path: "actions.yml" restartPolicy: OnFailure
And that is all! The CronJob and ConfigMap are easily customisable to suit whatever you need Curator to achieve in your Kubernetes cluster.
Documentation on Kubernetes CronJobs, and Elasticsearch Curator.