How to Run an Initialisation Mechanism Everytime a Container Restarts in a Pod

Table of Contents

a base scenario to test
run some code inside the container itself
- edit the entrypoint
- postStart script
run some code in another container, in the same pod
- the “sidekick” container
- using initContainer
run some code in another pod, using jobs
- using an operator
conclusion
Permalink

a base scenario to test

nginx is often used for tutorials, example and stuffs, let’s use it.

Something very simple and dumb.

    apiVersion: v1
    kind: Pod
    metadata:
      name: testpod
      namespace: default
      labels:
        app.kubernetes.io/name: testpod
    spec:
      volumes:
        - name: data
          hostPath:
            path: /tmp
      containers:
        - image: nginx:latest
          imagePullPolicy: IfNotPresent
          name: maincontainer
          ports:
            - containerPort: 80
              name: http
          livenessProbe:

httpGet:

              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
          readinessProbe:

httpGet:

              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html/init

We will want to initialize the /usr/share/nginx/html/init folder every time the container restarts.

To get access to this service, we have to provide a few resources related to the network.

    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: testsvc
      labels:
        app.kubernetes.io/name: testsvc
    spec:
      ports:
      - name: http
        port: 80
        protocol: TCP
        targetPort: http
      selector:
        app.kubernetes.io/name: testpod
      type: ClusterIP
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        cert-manager.io/cluster-issuer: local
      name: testingress
    spec:
      rules:
      - host: testingress.localtest.me

http:

          paths:
          - backend:
              service:
                name: testsvc
                port:
                  name: http
            path: /
            pathType: Prefix

We can now apply those resources.

Then, see the pod answering a simple get request.

{ curl --fail --silent --show-error --location "http://testingress.localtest.me${url}" | grep "${expected}" ; } 2>&1

<title>Welcome to nginx!</title>
<h1>Welcome to nginx!</h1>

Nothing fancy, but a good start to test our ideas.

Now, let’s focus on how we could initialize the /usr/share/nginx/html/init folder automatically.

run some code inside the container itself

We can run the initialization code inside the container that needs it.

In our case, the initialization script needed a bunch of typescript stuffs. We did not want to pollute the tiny little container with those.

Yet, for the record, to the best of my knowledge, there are two ways of doing this.

edit the entrypoint

You an create your own image with the entrypoint first initializing and then running the old entrypoint.

It requires that you maintain the image though, and possible update the entrypoint if the behavior changes. Also, you might mess up with the signals handlers by running a server in the background, running the initialization script and then put it in the foreground again.

postStart script

I tend to avoid them, because they are hard to debug, but this definitely an option.

run some code in another container, in the same pod

The code of the container that we would like to run should look like this.

- name: init
  image: busybox
  command: ["/bin/sh", "-c"]
  volumeMounts:
    - name: data
      mountPath: /usr/share/nginx/html/init
  args:
    - |
      echo "Waiting for Nginx to start..."
      while ! nc -z localhost 80; do sleep 1; done
      echo "Nginx is up! Initializing..."
      echo "<h1>Welcome! Initialized</h1>" > /usr/share/nginx/html/init/index.html
      echo "Initialization complete."

As far as I could understand, there are three ways to run containers aside the main one.

the “sidekick” container

We could try using another container next to our own. It would wait for the main container to start and then initialize it. I did not actually see the term “sidekick” used in the literature, but I like it.

    apiVersion: v1
    kind: Pod
    metadata:
      name: testpod
      namespace: default
      labels:
        app.kubernetes.io/name: testpod
    spec:
      volumes:
        - name: data
          hostPath:
            path: /tmp
      containers:
        - image: nginx:latest
          imagePullPolicy: IfNotPresent
          name: maincontainer
          ports:
            - containerPort: 80
              name: http
          livenessProbe:

httpGet:

              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
          readinessProbe:

httpGet:

              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html/init
    - name: init
      image: busybox
      command: ["/bin/sh", "-c"]
      volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html/init
      args:
        - |
          echo "Waiting for Nginx to start..."
          while ! nc -z localhost 80; do sleep 1; done
          echo "Nginx is up! Initializing..."
          echo "<h1>Welcome! Initialized</h1>" > /usr/share/nginx/html/init/index.html
          echo "Initialization complete."

This unfortunately results in:

curl: (22) The requested URL returned error: 503

Looking at the resource, we get

kubectl describe pod testpod | gi backoff

    Reason:       CrashLoopBackOff
Warning  BackOff    102s (x6 over 2m35s)  kubelet            Back-off restarting failed container init-sidekick in pod testpod_default(b0f5b871-7095-456a-a384-47922cfca81d)

Indeed, it needs the pod to be up and running, and if we try using a RestartPolicy to Never to avoid needing this.

Now, we get a pod that is partially running.

kubectl describe pod testpod | gi 'Stat\|Reason'|gv "Type"

Status:           Running
    State:          Running
    State:          Terminated
      Reason:       Completed

Listing the pods shows it as:

kubectl get pods testpod

NAME      READY   STATUS     RESTARTS   AGE
testpod   1/2     NotReady   0          17s

Therefore, even though nginx can still be accessed with the port-forward, no svc or ingress will be setup.

Also, the restartPolicy is configured pod-wise. The main container will not restart if need be. It is definitely not what we want.

using initContainer

initContainer have their own restartPolicy¹. Hence the limitation of the “sidekick” container no longer applies.

restartPolicy = Never or OnFailure

This is the classical case of initContainer.

The main container will start only after the initcontainer has run, so this won’t work.

    apiVersion: v1
    kind: Pod
    metadata:
      name: testpod
      namespace: default
      labels:
        app.kubernetes.io/name: testpod
    spec:
      volumes:
        - name: data
          hostPath:
            path: /tmp
      containers:
        - image: nginx:latest
          imagePullPolicy: IfNotPresent
          name: maincontainer
          ports:
            - containerPort: 80
              name: http
          livenessProbe:

httpGet:

              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
          readinessProbe:

httpGet:

              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html/init
  initContainers:
    - name: init
      image: busybox
      command: ["/bin/sh", "-c"]
      volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html/init
      args:
        - |
          echo "Waiting for Nginx to start..."
          while ! nc -z localhost 80; do sleep 1; done
          echo "Nginx is up! Initializing..."
          echo "<h1>Welcome! Initialized</h1>" > /usr/share/nginx/html/init/index.html
          echo "Initialization complete."

As expected, this leads to the pod being stuck in the init phase, subject to the circular dependency initContainer <-> main container.

kubectl get pod testpod

NAME      READY   STATUS     RESTARTS   AGE
testpod   0/1     Init:0/1   0          25s

restartPolicy = Always => sidecar container²

It should do the work.

    apiVersion: v1
    kind: Pod
    metadata:
      name: testpod
      namespace: default
      labels:
        app.kubernetes.io/name: testpod
    spec:
      volumes:
        - name: data
          hostPath:
            path: /tmp
      containers:
        - image: nginx:latest
          imagePullPolicy: IfNotPresent
          name: maincontainer
          ports:
            - containerPort: 80
              name: http
          livenessProbe:

httpGet:

              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
          readinessProbe:

httpGet:

              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html/init
  initContainers:
    - name: init
      image: busybox
      command: ["/bin/sh", "-c"]
      volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html/init
      args:
        - |
          echo "Waiting for Nginx to start..."
          while ! nc -z localhost 80; do sleep 1; done
          echo "Nginx is up! Initializing..."
          echo "<h1>Welcome! Initialized</h1>" > /usr/share/nginx/html/init/index.html
          echo "Initialization complete."
          sleep 3600          
      restartPolicy: Always

<h1>Welcome! Initialized</h1>

It works, but the container remains here for the lifetime of the pod.

kubectl get pod testpod

NAME      READY   STATUS    RESTARTS   AGE
testpod   2/2     Running   0          3m8s

kubectl describe pod testpod | gi 'Stat\|Reason'|gv "Type"

Status:           Running
    State:          Running
    State:          Running

So this is far from ideal.

another issue

initcontainer are run once for the lifetime of the pod. Therefore, even if this worked, it would still not fulfil our need. This can be mitigated using a postStart script that would trigger the initialization in the sidecar, but I intuitively don’t like it.

run some code in another pod, using jobs

If we cannot initialize the pod from the inside, let’s try from the outside. A job might be exactly what we need.

Because this is from the outside, we have to allow the communication between the job and the pod, at least so that the job waits for the pod.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: jobtopod
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: testpod
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: testjob
      ports:
        - protocol: TCP
          port: 80

Then, the code of the job is quite similar to the code of the initcontainer.

apiVersion: batch/v1
kind: Job
metadata:
  name: testjob
  labels:
    app.kubernetes.io/name: testjob
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/name: testjob
    spec:
      volumes:
        - name: data
          hostPath:
            path: /tmp

      containers:
        - name: init
          image: busybox
          command: ["/bin/sh", "-c"]
          args:
            - |
              echo "Waiting for Nginx to be ready..."
              until nc -z testsvc 80; do
                echo "Waiting for Nginx..."
                sleep 2
              done
              echo "Nginx is up! Running initialization..."
              echo "<h1>Welcome! Initialized at $(date)</h1>" > /usr/share/nginx/html/init/index.html
              echo "Initialization complete."              
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html/init
      restartPolicy: Never

Note that I mounted the data volume here, to be able to init some files that will be read by the pod.

We can request the init page. Yeah!

<h1>Welcome! Initialized at Thu Mar 20 10:50:02 UTC 2025</h1>

And the job don’t leaves a dangling pod behind.

kubectl get pods testpod | tail -1
kubectl get pods -l app.kubernetes.io/name=testjob | tail -1

testpod   1/1     Running   0          8m50s
testjob-wkh4n   0/1     Completed   0          8m50s

But that is not enough. When the container restarts, nothing will tell the job to run again.

kubectl exec pods/testpod -- nginx -s stop

Of course, the job won’t start again

kubectl get pods

NAME            READY   STATUS      RESTARTS      AGE
testjob-s5gqn   0/1     Completed   0             3m19s
testpod         1/1     Running     2 (23s ago)   3m19s

And the initialization won’t be updated.

<h1>Welcome! Initialized at Thu Mar 20 10:50:02 UTC 2025</h1>

using an operator

We can try to create a custom operator to trigger this job automatically.

For the sake of this article, I wanted something easy to play with, so I chose shell-operator.

First, I need to setup some policies for the operator to be able to watch the pod and run a job.

kubectl create namespace example-monitor-pods
kubectl create --namespace example-monitor-pods serviceaccount monitor-pods-acc
# the following needs to be deleted and recreated each time you change the clusterrole
kubectl create clusterrole monitor-pods --verb=get,watch,list,delete,create --resource=pods,jobs
kubectl create clusterrolebinding monitor-pods --clusterrole=monitor-pods --serviceaccount=example-monitor-pods:monitor-pods-acc

We will need to put the our job definition inside the operator, so that it can be mounted into the operator.

Let’s put it in a config map.

apiVersion: v1
kind: ConfigMap
metadata:
  name: jobdef
data:
  job.yaml: |
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: testjob
      labels:
        app.kubernetes.io/name: testjob
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/name: testjob
        spec:
          volumes:
            - name: data
              hostPath:
                path: /tmp

          containers:
            - name: init
              image: busybox
              command: ["/bin/sh", "-c"]
              args:
                - |
                  echo "Waiting for Nginx to be ready..."
                  until nc -z testsvc 80; do
                    echo "Waiting for Nginx..."
                    sleep 2
                  done
                  echo "Nginx is up! Running initialization..."
                  echo "<h1>Welcome! Initialized at $(date)</h1>" > /usr/share/nginx/html/init/index.html
                  echo "Initialization complete."
              volumeMounts:
                - name: data
                  mountPath: /usr/share/nginx/html/init
          restartPolicy: Never

Then, we can provide the code of the operator that will recreate a job each time if sees the main container entering the Running state.

I won’t get into the details of the code of the hook here, because the documentation is very well done and explains this better than I could.

The hook has to deal with being called with three kind of events:

with --config, it is expected to return what it wants to watch

configVersion: v1
kubernetes:
- apiVersion: v1
  kind: Pod
  executeHookOnEvent: ["Modified"]
  labelSelector:
    matchLabels:
      app.kubernetes.io/name: testpod
  namespace:
    nameSelector:
      matchNames: ["default"]
  jqFilter: .status.conditions[]|select(.type == "Ready").status
  allowFailure: true

at startup, it is called with the description of all matched objects, into .[0].objects of ${BINDING_CONTEXT_PATH} I identify this situation when .[0].object (singular) is null, as this is the next case
```
pod_name=$(jq -r .[0].object.metadata.name "${BINDING_CONTEXT_PATH}")
if test "${pod_name}" = "null"
then
```
In that case only, I need to get the status in the list of objects provided.
```
pod_name=$(jq -r .[0].objects[0].object.metadata.name "${BINDING_CONTEXT_PATH}")
ready=$(jq -r .[0].objects[0].filterResult "${BINDING_CONTEXT_PATH}")
```
when a matched object changes, it is called with the object described in ~.[0].object~, this is the else part of my condition
```
else
    ready=$(jq -r .[0].filterResult "${BINDING_CONTEXT_PATH}")
```

In 2 or 3, if the container is ready, I want to trigger a new job. Otherwise, do nothing.

if test "${ready}" = "True"
then
    kubectl --namespace default delete --wait -f /jobdef/job.yaml || echo "no job to remove"
    kubectl --namespace default apply -f /jobdef/job.yaml
    echo "Job for pod '${pod_name}' created"
else
    echo "Ready: ${ready}"
fi

Put together, it gives a script a bit hairy, that hides the fact that the logic is pretty simple.

Let’s put that code in a configmap as well (named pods-hook, see ³).

Finally, we have to create a pod to run shell-operator giving it access to both the bash script code and the job definition.

apiVersion: v1
kind: Pod
metadata:
  name: shell-operator
spec:
  volumes:
    - name: pods-hook
      configMap:
        name: pods-hook
        defaultMode: 0777
    - name: jobdef
      configMap:
        name: jobdef
  containers:
  - name: shell-operator
    image: ghcr.io/flant/shell-operator:latest
    imagePullPolicy: Always
    volumeMounts:
      - name: pods-hook
        mountPath: /hooks
      - name: jobdef
        mountPath: /jobdef
  serviceAccountName: monitor-pods-acc

In real life, one would may be provide a custom image with the files in it, but in the scope of this article, I did not want to bother building images.

In the end, after applying this.

We can take a look at the log of this operator to see if it ran the job once started.

kubectl logs --namespace example-monitor-pods pods/shell-operator |gi "shell-operator.hook-manager.hook.executor"|jq -r .msg

job.batch "testjob" deleted
job.batch/testjob created
Job for pod 'testpod' created

And we can see that the job was indeed recently run.

kubectl get pods

NAME            READY   STATUS      RESTARTS      AGE
testjob-9gz29   0/1     Completed   0             22s
testpod         1/1     Running     1 (21m ago)   40m

Now, let’s try to restart the main container and see how it reacts.

kubectl exec pods/testpod -- nginx -s stop

kubectl get pods

NAME            READY   STATUS              RESTARTS     AGE
testjob-kq5pk   0/1     ContainerCreating   0            0s
testpod         1/1     Running             2 (4s ago)   40m

So far so good, let’s take a look at the log of the operator.

job.batch "testjob" deleted
job.batch/testjob created
Job for pod 'testpod' created
Ready: False
job.batch "testjob" deleted
job.batch/testjob created
Job for pod 'testpod' created

It shows that succession of events

operator starting -> detecting the running container -> start a job
main container stopped -> do nothing
main container started again -> start a job

conclusion

From all attempts I made, only the solution operator + job fulfilled exactly my need of initializing my pod EVERYTIME the main container starts. It is however a bit involved, for it needs to configure some rights and start a pod that creates jobs that in turn run the initialization.

On the other hand, this feeling might be linked to the fact this is my first experience writing an operator. May be with time and experience this will feel more natural.

In any case, I liked how kubernetes could adapt to this fancy use case.

Permalink

sidecar containers ignore the pod-level restartPolicy

— https://www.baeldung.com/ops/kubernetes-pod-lifecycle ([2025-03-20 Thu])

↩︎
init container is created with its restartPolicy set to Always

— https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ ([2025-03-20 Thu])

↩︎

apiVersion: v1
kind: ConfigMap
metadata:
  name: pods-hook
data:
  pods-hook.sh: |
    #!/usr/bin/env bash

    if [[ $1 == "--config" ]] ; then
      cat <<EOF
      configVersion: v1
      kubernetes:
      - apiVersion: v1
        kind: Pod
        executeHookOnEvent: ["Modified"]
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: testpod
        namespace:
          nameSelector:
            matchNames: ["default"]
        jqFilter: .status.conditions[]|select(.type == "Ready").status
        allowFailure: true
    EOF
    else
      pod_name=$(jq -r .[0].object.metadata.name "${BINDING_CONTEXT_PATH}")
      if test "${pod_name}" = "null"
      then
         pod_name=$(jq -r .[0].objects[0].object.metadata.name "${BINDING_CONTEXT_PATH}")
         ready=$(jq -r .[0].objects[0].filterResult "${BINDING_CONTEXT_PATH}")
      else
          ready=$(jq -r .[0].filterResult "${BINDING_CONTEXT_PATH}")
      fi
      if test "${ready}" = "True"
      then
          kubectl --namespace default delete --wait -f /jobdef/job.yaml || echo "no job to remove"
          kubectl --namespace default apply -f /jobdef/job.yaml
          echo "Job for pod '${pod_name}' created"
      else
          echo "Ready: ${ready}"
      fi
    fi

↩︎