Schedule Elasticsearch Curator Tasks Natively in Kubernetes

As of Kubernetes 1.21, CronJobs are a stable native Kubernetes feature. Using Kubernetes CronJobs and Elasticsearch Curator, we can schedule periodic changes on an Elasticsearch instance.

Unfortunately, Elasticsearch does not publish a native Elasticsearch Curator Docker image. The Bitnami image that was previously available on DockerHub has been removed. You can use the untergeek/curator image from Docker Hub or build your own image by installing curator via pip in a Python base image.

To use the Elasticsearch Curator with a Kubernetes CronJob we need to define two Kubernetes objects, a CronJob and a ConfigMap. We use the ConfigMap to save the configuration of Curator and the task steps we want Curator to take. I have included both the Curator tasks and Curator configuration in the same ConfigMap as separate keys to simplify the setup. However, if it contains sensitive values, the Curator configuration could also be stored as a secret.

An example of the ConfigMap, containing both the Curator configuration and tasks:

apiVersion: v1
kind: ConfigMap
metadata:
    name: curator
    namespace: logging
data:
    config.yml: |
        client:
          hosts:
            - elasticsearch-master
          port: 9200
          url_prefix:
          use_ssl: False
          certificate:
          client_cert:
          client_key:
          ssl_no_validate: False
          username:
          password:
          timeout: 30
          master_only: False

        logging:
          loglevel: INFO
          logfile:
          logformat: default
          blacklist: ['elasticsearch', 'urllib3']
    actions.yml: |
        actions:
          1:
            action: delete_indices
            description: "Delete indices older than 20 days"
            options:
              ignore_empty_list: True
            filters:
              - filtertype: age
                source: creation_date
                direction: older
                unit: days
                unit_count: 20

The CronJob that we need to define will need to mount both the Curator configuration and the task file values stored in our ConfigMap. I have mounted both Curator files in the /curator directory in this example, which is scheduled to run at midnight on the first day of each month:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: curator
  namespace: logging
spec:
  schedule: "0 0 1 * *"
  jobTemplate:
    spec:
      template:
          spec:
            containers:
              - name: curator
                image: untergeek/curator:latest
                command: ["/bin/sh", "-c"]
                args:
                - curator --config /curator/config.yml /curator/actions.yml;
                volumeMounts:
                  - name: curator
                    mountPath: /curator
                    readOnly: true
            volumes:
            - name: curator
              configMap:
                name: curator
                items:
                  - key: "config.yml"
                    path: "config.yml"
                  - key: "actions.yml"
                    path: "actions.yml"
            restartPolicy: OnFailure

And that is all! The CronJob and ConfigMap are easily customisable to suit whatever you need Curator to achieve in your Kubernetes cluster.

Documentation on Kubernetes CronJobs, and Elasticsearch Curator.