When running Docker containers in production ensuring high availability can be challenge. Unlike VMs which run untouched for months or even years, containers are meant to be used for a very short lifespan — typically a few hours or a few days at most. With this constant church, you need a way to ensure that containers that are outdated, vulnerable, or malfunctioning are retired and replaced with new containers. But doing this manually is not scalable, and is prone to human error. The better way is to automate container creation and restarts. There are a few ways you can do this. Before you dive in, take a look at this wiki page with quite a few resources on how to start Docker containers.
Docker provides restart policies for containers. These policies can be configured to restart containers after particular events like failures, or if Docker itself stops. As part of Docker Swarm, you can even restart services automatically.
$ docker run -dit — restart unless-stopped [CONTAINER]
There are four restart policies you can choose from — Off, On-failure, Unless-stopped, and Always. As the terms state, ‘Off’ means that the container won’t be restarted if it fails or stops. ‘On-failure’ ensures the container restarts only in the case of a failure that’s not caused by the user. ‘Unless-stopped’ restarts the container only when any user executes a command to stop the container, not when it fails because of an error. ‘Always’ restarts the container whether the it’s caused by an error, or is executed by a user, or if Docker is restarted.
Restart policies for Docker are a lot simpler than the similar options available for virtual machines in the past. With VMware, the largest provider of VMs, to automatically restart VMs they need to be part of a high availability (HA) cluster. This cluster has both primary and secondary instances of VMs, and as primary instances fail, they are replaced by secondary instances. This is a complex process that consumes more resources as VMs are much larger in size than VMs, and you need to allocate sufficient secondary resources as a backup, which is not economic when running large clusters. Containers, on the other hand, reduce this process to a simple command that’s executed by Docker Engine, and consumes minimal resources.
If you’d like to change the restart policy for an existing you can use the docker update command as follows:
$ docker update — restart=on-failure CONTAINER [CONTAINER…]
This command applies the new restart policy to the container .
However, using this command excessively will lead to configuration drift where your container becomes altered and very different from what it originally was. This leads to ‘works on my machine’ errors, which is the very thing Docker set out to resolve in the first place. So, rather than updating existing containers with new restart policies, you may want to create new containers with the required restart policies and ensure your containers are immutable.
You may choose to use Docker Cloud, the commercial offering from Docker that provides a hosted registry, automated builds and tests, and even deployments to various cloud environments. In this case, the same restart policy applies to your containers with a few minor variations. Rather than the
-restart option, you need to use the
$ docker-cloud service set — autorestart ALWAYS (name or uuid)
This flag provides the same restart policies — Off, On-failure, and Always. The default value for this flag is Off, so if you want your containers to restart you’ll need to manually specify this.
Systemd is a process manager for Linux that’s used to automatically restart services that fail. It an be used to restart Docker containers as well. It lets you configure a container to be restarted on a server reboot.
The Systemd Restart service is used to automatically start containers on various prompts. The options to restart a container with the Restart service are no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, or always. Leaving it at ‘no’ doesn’t restart the container. ‘On-success’ restarts the container when it has a normal exit without errors. ‘On-failure’ restarts the container when it exits with an error code. ‘On-abnormal’ restarts the container when it is stopped due to a timeout. ‘On-abort’ restarts a container when it fails due to an unknown reason. ‘Always’ restarts the container in any of the above scenarios.
From the above options, it’s recommended to use ‘On-failure’ for longterm workloads, and ‘On-abort’ for containers that can stop on their own.
Kubernetes has a restart policy that applies to all containers in a pod. It works at the pod level and is able to control the lifecycle of a pod so that whenever a pod fails it is restarted. This is extremely simple, and all you need to do is specify whether you want a container to be restarted ‘Always’ ‘OnFailure’ or ‘Never. These parameters are similar to Docker’s, but with Kubernetes you need to be aware of the pod phase.
In Kubernetes, the phase of a pod defines where in its lifecycle a pod is. The different phases for a pod are pending, running, succeeded, failed, and unknown. Pending is when containers are still being created in the pod. When the containers are executing tasks, the pod phase becomes running, and when the tasks are completed based on the outcome the pod phase becomes succeeded or failed. Unknown is when there’s a communication issue between the pod and the host.
If a container in a pod fails, the phase of the pod would normally be changed to ‘failed’, but if your restart policy is set to ‘Always’, Kubernetes will restart your container and change the pod phase to ‘running’. In this way Kubernetes takes a more holistic approach to container restarts, looking not just at individual containers, but the pods they’re in.
Restarting containers is an important part of running Dockerized applications. Fortunately, there are many ways to restart containers — using Docker, using Systemd, or using Kubernetes. They all do the same job of restarting containers under various circumstances and give you enough control container restarts. As you choose which container management platform to choose, container restarts is one aspect you should consider.