Created: February 15, 2023
Golang, Docker "FROM scratch," Kubernetes Cronjob, Istio or How to Make Them Friends
Docker has become an indispensable tool for developers and DevOps engineers, making it easier to package, distribute, and deploy applications consistently and efficiently. One of the most popular programming languages for building microservices and scalable applications is Go (Golang). In addition to being an essential tool for developers and DevOps engineers, Docker is often used in conjunction with Kubernetes to manage and orchestrate containerized applications at scale. This is where Istio comes in—it's a service mesh that can be used with Kubernetes to help manage and secure communication between microservices in a containerized application.
In this article, we will discuss how to run a K8s CronJob with the following conditions:
- Our application is written in the GO language
- The Docker image is built on the "FROM scratch" base
auto-injectis enabled for theistio-proxy
Building Docker images for GO applications
Multi-stage builds are a powerful feature in Docker that allows you to build an application in one stage and then copy the compiled binary to a second, minimal stage. This provides several advantages, including:
- Reducing the size of the final image: By using a minimal base image, you can keep the size of the final image as small as possible. This makes it faster to download and deploy and reduces the attack surface of the image.
- Improved security: By using a minimal base image, you can reduce the number of dependencies and potential attack vectors. Additionally, by not copying any build artifacts or temporary files, you can further reduce the attack surface of the image.
- Improved build time: Multi-stage builds allow you to compile the application in one stage and then copy the binary to the final image in a separate stage. This can improve build time, as you can use a faster build environment for the first stage and then use a smaller, more minimal image for the second stage.
The "FROM scratch" base image is a special, minimal base image that contains no files or dependencies. It provides a clean slate for building your application and is often used as the base image for multi-stage builds.
Here's an example of how you can use a multi-stage build to build a Docker image for a Go application:
# Build stage
FROM golang:latest AS build
WORKDIR /app
COPY . .
RUN go build -o myapp
# Final stage
FROM scratch
COPY --from=build /app/myapp /myapp
CMD ["/myapp"]
In this example, we first use the Golang: latest image as the build stage. This image provides a Go environment with all the necessary dependencies to build our application. We then copy the source code to the /app directory and run the go build command to compile the binary.
In the final stage, we use the scratch base image, which provides a minimal, empty environment for our application. We then copy the compiled binary from the build stage using the COPY -- from syntax. Finally, we set the CMD to run the binary, which will be the default command when the image is run as a container.
(This example doesn't show all best practices of building Docker images because it's not a goal of this article. That's why the 'latest' tag is used, etc.)
Using a multi-stage build and the "FROM scratch" base image, we can create small, secure, and efficient Docker images for our Go applications. This approach also provides a consistent and reproducible build environment, making it easier to manage and deploy our applications.
Istio
Istio is an open-source service mesh technology that provides a set of APIs and tools for managing microservices in a cloud-native environment. It is designed to simplify the management of the communication between microservices and to provide a platform for adding new functionality, such as traffic management, security, and telemetry.
Advantages of using Istio:
- Istio helps to manage the complexity of microservices communication by providing a unified control plane.
- It provides built-in security features that help to secure communication between microservices.
- Istio makes it easier to manage and monitor microservices by providing centralized visibility and control over the network.
- Istio provides an API-driven approach to traffic management that makes it easy to manage and control the flow of requests between microservices.
Problems with using Istio in Kubernetes Jobs and docker images built using "FROM scratch"
Kubernetes Jobs (CronJobs) are used to run batch jobs, and when the main container is completed, the Istio proxy sidecar continues to run, which can cause issues with resource usage and security.
To terminate the Istio proxy sidecar, you need to run the following command in your main container:
wget -q --post-data='' -S -O /dev/null http://127.0.0.1:15020/quitquitquit
However, this can be a challenge when the main container is built using the "FROM scratch" Docker image, as it does not include the wget/curl tool, which is required to terminate the Istio proxy sidecar.
In such cases, there are a few alternative options to consider:
- Modify the Docker image to include the wget tool: You can modify the Docker image to include the wget tool by using a base image that includes it or by installing it directly in the Dockerfile. This solution would require modifying the existing Docker image and could potentially impact the size of the image.
- Configure Istio to automatically terminate the proxy: Istio can be configured to automatically terminate the proxy when the main container is completed. This can be done by setting the terminationGracePeriodSeconds property in the Kubernetes Job definition. This solution would require modifying the existing Kubernetes Job definition and may require additional configuration changes in Istio also we need to be sure that our main container is completed.
- Using init containers to copy tools into the main container.
Let's see what the third option looks like.
An init container is a special type of container that runs before the main container in a Pod.
In this case, the init container can be based on the busybox image and have the following steps:
- Copy /bin/sh and /bin/busybox to a volume (e.g. emptyDir) shared between the init container and the main app.
The command for the main container will be set as follows:
set -x; /myapp || EXIT_CODE=$?; /share/busybox wget -q --post-data='' -S -O /dev/null http://127.0.0.1:15020/quitquitquit; exit ${EXIT_CODE}
The above command runs the main application /myapp and sets an exit code stored in the EXIT_CODE variable. When the main application finishes, the Istio proxy sidecar is terminated by sending a wget request to http://127.0.0.1:15020/quitquitquit. The exit code of the main application is then returned to ensure that the correct exit status is passed back to the system.
Here is an example of a Kubernetes CronJob YAML definition that implements the solution:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: my-cronjob
spec:
schedule: "0 0 * * *" # run every day at midnight
jobTemplate:
spec:
template:
spec:
volumes:
- name: share
emptyDir: {}
initContainers:
- name: init
image: busybox:musl
command: ["cp", "/bin/sh", "/bin/busybox", "/share/"]
volumeMounts:
- name: share
mountPath: /share
containers:
- name: main
image: my-main-container-image
command:
- /share/sh
- -c
- set -x; /myapp || CODE=$?; /share/busybox wget -q --post-data='' -S -O /dev/null http://127.0.0.1:15020/quitquitquit; exit ${CODE}
volumeMounts:
- name: share
mountPath: /share
livenessProbe:
tcpSocket:
port: 15020
readinessProbe:
tcpSocket:
port: 15020
restartPolicy: Never
In addition, the above definition includes liveness/readiness probes for the main container to check that the Istio proxy container has started and is ready to serve traffic. Keep in mind that this CronJob spec doesn't have special annotations/labels to enable injecting istio-proxy sidecar.
Conclusion
When using Istio with Kubernetes Jobs, it is important to consider the limitations of the main container's Docker image and the tools that are included. By exploring alternative solutions, such as modifying the Docker image, using a different tool, or configuring Istio, you can ensure that your Kubernetes Jobs run smoothly and securely with Istio.