Eugeny Shtoltc - IT Cloud

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

c3bc2d2e37a6 ubuntu "sleep 10" 10 minutes ago Up 2 seconds keen_sinoussi

Another aspect is when to consider the container dead. By default, this is a process crash. But, by far, the application does not always crash itself in case of an error in order to allow the container to be restarted. For example, a server may be designed incorrectly and try to download the necessary libraries during its startup, but it does not have this opportunity, for example, due to the blocking of requests by the firewall. In such a scenario, the server can wait a long time if an adequate timeout is not specified. In this case, we need to check the functionality. For a web server, this is a response to a specific url, for example:

docker run –rm -d \

–-name = elasticsearch \

–-health-cmd = "curl –silent –fail localhost: 9200 / _cluster / health || exit 1" \

–-health-interval = 5s \

–-health-retries = 12 \

–-health-timeout = 20s \

{image}

For demonstration, we will use the file creation command. If the application has not reached the working state within the allotted time limit (set to 0) (for example, creating a file), then it is marked as working, but before that the specified number of checks is done:

vagrant @ ubuntu: ~ $ sudo docker run \

–d –name healt \

–-health-timeout = 0s \

–-health-interval = 5s \

–-health-retries = 3 \

–-health-cmd = "ls / halth" \

ubuntu bash -c 'sleep 1000'

c0041a8d973e74fe8c96a81b6f48f96756002485c74e51a1bd4b3bc9be0d9ec5

vagrant @ ubuntu: ~ $ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

c0041a8d973e ubuntu "bash -c 'sleep 1000'" 4 seconds ago Up 3 seconds (health: starting) healt

vagrant @ ubuntu: ~ $ sleep 20

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

c0041a8d973e ubuntu "bash -c 'sleep 1000'" 38 seconds ago Up 37 seconds (unhealthy) healt

vagrant @ ubuntu: ~ $ sudo docker rm -f healt

healt

If at least one of the checks worked, then the container is marked as healthy immediately:

vagrant @ ubuntu: ~ $ sudo docker run \

–d –name healt \

–-health-timeout = 0s \

–-health-interval = 5s \

–-health-retries = 3 \

–-health-cmd = "ls / halth" \

ubuntu bash -c 'touch / halth && sleep 1000'

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

160820d11933 ubuntu "bash -c 'touch / hal …" 4 seconds ago Up 2 seconds (health: starting) healt

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

160820d11933 ubuntu "bash -c 'touch / hal …" 6 seconds ago Up 5 seconds (healthy) healt

vagrant @ ubuntu: ~ $ sudo docker rm -f healt

healt

In this case, the checks are repeated all the time at a given interval:

vagrant @ ubuntu: ~ $ sudo docker run \

–d –name healt \

–-health-timeout = 0s \

–-health-interval = 5s \

–-health-retries = 3 \

–-health-cmd = "ls / halth" \

ubuntu bash -c 'touch / halth && sleep 60 && rm -f / halth && sleep 60'

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

8ec3a4abf74b ubuntu "bash -c 'touch / hal …" 7 seconds ago Up 5 seconds (health: starting) healt

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

8ec3a4abf74b ubuntu "bash -c 'touch / hal …" 24 seconds ago Up 22 seconds (healthy) healt

vagrant @ ubuntu: ~ $ sleep 60

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

8ec3a4abf74b ubuntu "bash -c 'touch / hal …" About a minute ago Up About a minute (unhealthy) healt

Kubernetes provides (kubernetes.io/docs/tasks/configure-POD-container/configure-liveness-readiness-probes/) three tools that check the state of a container from outside. They are more important because they serve not only to inform, but also to manage the application life cycle, roll-forward and rollback of updates. Configuring them incorrectly can, and often does, cause the application to malfunction. So, if the liveness test is triggered before the application starts working, Kubernetes will kill the container, not allowing it to rise. Let's consider it in more detail. The liveness probe is used to determine the health of the application, and if the application crashes and does not respond to the liveness probe, Kubernetes reloads the container. As an example, we will take a shell test, due to the simplicity of the demonstration of work, but in practice it should be used only in extreme cases, for example, if the container is started not as a long-lived server, but as a JOB, doing its job and ending its existence, having achieved the result … For server checks, it is better to use HTTP probes, which already have a built-in dedicated proxy and do not require curl in the container and do not depend on external kube-proxy settings. When using databases, you must use a TCP probe, as they usually do not support the HTTP protocol. Let's create a long-lived container at www.katacoda.com/courses/kubernetes/playground:

controlplane $ cat << EOF> liveness.yaml

apiVersion: v1

kind: Pod

metadata:

name: liveness

spec:

containers:

– name: healtcheck

image: alpine: 3.5

args:

– / bin / sh

– -c

– touch / tmp / healthy; sleep 10; rm -rf / tmp / healthy; sleep 60

livenessProbe:

exec:

command:

– cat

– / tmp / healthy

initialDelaySeconds: 15

periodSeconds: 5

EOF

controlplane $ kubectl create -f liveness.yaml

pod / liveness created

controlplane $ kubectl get pods

NAME READY STATUS RESTARTS AGE

liveness 1/1 Running 2 2m11s

controlplane $ kubectl describe pod / liveness | tail -n 10

Type Reason Age From Message

–– – – – –

Normal Scheduled 2m37s default-scheduler Successfully assigned default / liveness to node01

Normal Pulling 2m33s kubelet, node01 Pulling image "alpine: 3.5"

Normal Pulled 2m30s kubelet, node01 Successfully pulled image "alpine: 3.5"

Normal Created 33s (x3 over 2m30s) kubelet, node01 Created container healtcheck

Normal Started 33s (x3 over 2m30s) kubelet, node01 Started container healtcheck

Normal Pulled 33s (x2 over 93s) kubelet, node01 Container image "alpine: 3.5" already present on machine

Warning Unhealthy 3s (x9 over 2m13s) kubelet, node01 Liveness probe failed: cat: can't open '/ tmp / healthy': No such file or directory

Normal Killing 3s (x3 over 2m3s) kubelet, node01 Container healtcheck failed liveness probe, will be restarted

We see that the container is constantly being restarted.

controlplane $ cat << EOF> liveness.yaml

apiVersion: v1

kind: Pod

metadata:

name: liveness

spec:

containers:

– name: healtcheck

image: alpine: 3.5

args:

– / bin / sh

– -c

– touch / tmp / healthy; sleep 30; rm -rf / tmp / healthy; sleep 60

livenessProbe:

exec:

command:

– cat

– / tmp / healthy

initialDelaySeconds: 15

periodSeconds: 5

EOF

controlplane $ kubectl create -f liveness.yaml

pod / liveness created

controlplane $ kubectl get pods

NAME READY STATUS RESTARTS AGE

liveness 1/1 Running 2 2m53s

controlplane $ kubectl describe pod / liveness | tail -n 15

SecretName: default-token-9v5mb

Optional: false

QoS Class: BestEffort

Node-Selectors:

Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s

node.kubernetes.io/unreachable:NoExecute for 300s

Events:

Type Reason Age From Message

–– – – – –

Normal Scheduled 3m44s default-scheduler Successfully assigned default / liveness to node01

Normal Pulled 68s (x3 over 3m35s) kubelet, node01 Container image "alpine: 3.5" already present on machine

Normal Created 68s (x3 over 3m35s) kubelet, node01 Created container healtcheck

Normal Started 68s (x3 over 3m34s) kubelet, node01 Started container healtcheck

Warning Unhealthy 23s (x9 over 3m3s) kubelet, node01 Liveness probe failed: cat: can't open '/ tmp / healthy': No such file or directory

Normal Killing 23s (x3 over 2m53s) kubelet, node01 Container healtcheck failed liveness probe, will be restarted

We also see on cluster events that when cat / tmp / health fails, the container is re-created:

controlplane $ kubectl get events

controlplane $ kubectl get events | grep pod / liveness

13m Normal Scheduled pod / liveness Successfully assigned default / liveness to node01

13m Normal Pulling pod / liveness Pulling image "alpine: 3.5"

13m Normal Pulled pod / liveness Successfully pulled image "alpine: 3.5"

10m Normal Created pod / liveness Created container healtcheck

10m Normal Started pod / liveness Started container healtcheck

10m Warning Unhealthy pod / liveness Liveness probe failed: cat: can't open '/ tmp / healthy': No such file or directory

10m Normal Killing pod / liveness Container healtcheck failed liveness probe, will be restarted

10m Normal Pulled pod / liveness Container image "alpine: 3.5" already present on machine

8m32s Normal Scheduled pod / liveness Successfully assigned default / liveness to node01

4m41s Normal Pulled pod / liveness Container image "alpine: 3.5" already present on machine

4m41s Normal Created pod / liveness Created container healtcheck

4m41s Normal Started pod / liveness Started container healtcheck

2m51s Warning Unhealthy pod / liveness Liveness probe failed: cat: can't open '/ tmp / healthy': No such file or directory

5m11s Normal Killing pod / liveness Container healtcheck failed liveness probe, will be restarted

Let's take a look at RadyNess trial. The availability of this test indicates that the application is ready to accept requests and the service can switch traffic to it:

controlplane $ cat << EOF> readiness.yaml

apiVersion: apps / v1

kind: Deployment

metadata:

name: readiness

spec:

replicas: 2

selector:

matchLabels:

app: readiness

template:

metadata:

labels:

app: readiness

spec:

containers:

– name: readiness

image: python

args:

– / bin / sh

– -c

– sleep 15 && (hostname> health) && python -m http.server 9000

readinessProbe:

exec:

command:

– cat

– / tmp / healthy

initialDelaySeconds: 1

periodSeconds: 5

EOF

controlplane $ kubectl create -f readiness.yaml

deployment.apps / readiness created

controlplane $ kubectl get pods

NAME READY STATUS RESTARTS AGE

readiness-fd8d996dd-cfsdb 0/1 ContainerCreating 0 7s

readiness-fd8d996dd-sj8pl 0/1 ContainerCreating 0 7s

controlplane $ kubectl get pods

NAME READY STATUS RESTARTS AGE

readiness-fd8d996dd-cfsdb 0/1 Running 0 6m29s

readiness-fd8d996dd-sj8pl 0/1 Running 0 6m29s

controlplane $ kubectl exec -it readiness-fd8d996dd-cfsdb – curl localhost: 9000 / health

readiness-fd8d996dd-cfsdb

Our containers work great. Let's add traffic to them:

controlplane $ kubectl expose deploy readiness \

–-type = LoadBalancer \

–-name = readiness \

–-port = 9000 \

–-target-port = 9000

service / readiness exposed

controlplane $ kubectl get svc readiness

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT (S) AGE