Kubernetes continuous integration using BuildKit, Buildx and docker registry

At Greeneye we took the decision to be fully kubernetes oriented. This means that we aim to use kubernetes in everything we do:

• Production

• Backend tools

• Continuous Integration (CI)

• Continuous Delivery (CD)

• IoT

In this post we are going to focus on our CI system. In the future, we will consider writing about the rest of the bullets stated above.

Problems

Before diving into details about our problem, we recommend going through these definitions: kubelet; Container Runtime Interface (CRI); Open Container Initiative (OCI); Shim; Docker; Docker Shim; Containerd.

We had few issues, but these 3 were the most critical ones:

• Kubernetes 1.21 - docker shim depreciation.

• Builds are slow.

• Cache doesn’t work properly.

Docker shim depreciation

The kubernetes core team announced the depreciation of docker as a runtime container on Dec 2nd, 2020. After this announcement, the community was in stress:

Fascinating how this docker/docker-shim deprecation has created mass confusion. Probably should have seen it coming.

So many things called "docker". The company, the dev experience, images, container instances.

— Joe Beda (@jbeda) December 3, 2020

The actual difference is this:

https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/

This change created a huge problem. CIs dependent on cloud kubernetes solutions would not have access to docker daemon.

How we used to do it

Before moving to our new CI, we had five different agents:

• “Docker multiplatform” - a Buildx implementation, for building x86/ARM64/ARMv8 images. The agents used a PV with 200GB for caching purposes. They kept on failing for not having enough storage.

• GKE Agents - we store part of our data in Google File Store (NFS), and we wanted to have an agent that had a direct access it it. Dependent on docker.sock.

• DoD agent - a very straightforward agent that builds using docker.sock. Dependent on docker.sock.

• AKS agents temp - works the same as DoD agent, but also supports Kaniko and few more ideas that we tried. Dependent on docker.sock.

• ARM agents - dependent on docker.sock, but runs on our own Jetson Xavier devices.

We had no clear way to deploy these agents. We kept experimenting with them and never put the focus on setting this up.

Caching

Most of our experiments were set because we wanted to improve our build time. We tried to find different solutions for distributed docker layer caching but ended up with nothing!

Understanding our caching problem requires the understanding of two things:

1. How docker works

2. Knowing about our wrong assumptions about building images in a d distributed system

Docker

First, let’s start with how the docker daemon works:

The Docker daemon (dockerd) listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. A daemon can also communicate with other daemons to manage Docker services (source: docker.com)

https://docs.docker.com/get-started/overview/#the-docker-daemon

We are usually used to building docker images either on our own machines or on virtual machine. In these cases, the docker client shares the same machine with the daemon. It might seem legit that the cache is available whenever we get back and use the same machine.

When using a distributed system, our builds occur on different machines/nodes. This raises several issues:

1. Different nodes mean that we have to build N times to have N caches available on each machine. This happens because each node shares its own docker daemon with an agent.

2. Nodes come and go, the whole idea of kubernetes is being able to scale up and down as fast as possible. Thus, most of the local cache we had from previous builds might be completely irrelevant.

3. Even if we decide that we are ok with #2, we are going to experience a storage issue at some point. It is very unlikely that the local storage of the node is big enough to keep so many images, especially when building big docker images like with ML and IoT.

Last but not least, we figured that if we change our CI, we would have to make sure our tests still working as expected. As we run most of them using docker run and the docker daemon is not available anymore we would have to find a good solution for that.

The same goes for CI tests, which usually run MySQL or an equal background service as a dependency. docker-compose triggers the service and requires systemd to be available.

Quick summary before moving on

• Our agents used docker daemon aka Docker outside of Docker (DoD) which is mount from the nodes.

• We tried tools like Kaniko but figured that they were slow.

• Some of our agents used docker build without using BuildKit/Buildx.

• We had an implementation of Docker inside of Docker (DinD) which is a bad practice: https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

Solution

Let's build docker images using BuildKit, Buildx and local-proxy registries!

We have been looking for a solution quite sometime. We tested different tools like Kaniko, Makisu and different ideas we found on Google and such. At the end, we ended up with BuiltKit and Buildx.

BuildKit

BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner. Key features:

• Automatic garbage collection

• Extendable frontend formats

• Concurrent dependency resolution

• Efficient instruction caching

• Build cache import/export

• Nested build job invocations

• Distributable workers

• Multiple output formats

• Pluggable architecture

• Execution without root privileges

(Source: BuildKit)

Buildx

Docker Buildx is a CLI plugin that extends the docker command with the full support of the features provided by Moby BuildKit builder toolkit. It provides the same user experience as docker build with many new features like creating scoped builder instances and building against multiple nodes concurrently. (source: docker/buildx)

Buildx comes with a useful feature: docker buildx supports —secret flag. It enables a safe way to build images while not including secrets in the image’s history.

https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information

Emulator

As stated above, we use Buildx to build x86 images alongside ARM64 images. For that reason, we also need to install a cross platform emulator: https://github.com/tonistiigi/binfmt.

Caching

We decided to split this into several layers:

1. BuildKit daemon will have its own local caching. But, we are not going to be dependent on that.

2. Use —cache-to and —cache-from flags.

3. We are going to reduce network time. As we are using both AKS and GKE together with ACR and GCR, it is important for us to pull images as fast as possible. So, we deploy docker registries served as a proxies.

4. For CI tests we are going to use a local docker registry. That way we keep our CI tests within the same network instead of persisting them somewhere else.

Build

This is how our build system looks like:

ADO agent

ADO agent is our Azure DevOps agent (it’s not a dependency). The agent creates a docker buildx builder and attaches it to the BuildKit daemon. This runs when the agent starts running:

BuildKit daemon
At first, the BuildKit daemon had an emptyDir. We figured that it will consume the node's storage, and we wanted to prevent errors such as:

The node was low on resource: ephemeral-storage. Container node was using 17007192Ki, which exceeds its request of 0.

We set our registries by mounting the following configMap:

ACR, GCR proxies

ACR, GCR proxies mount an SSD. There is an overhead in the sense of storage, but it’s worth it, especially when AKS tries to pull from GCR and GKE from ACR. Also, unfortunately there’s a problem with caching in GCR, so we need to use ACR as a caching registry.

Local registry

We use this to save our CI tests images.

binfmt

This is a DaemonSet that installs the required emulators on all of our cluster’s nodes.

We are also using this to build the ADO agent itself.

Deployment

We deploy everything mentioned in the build and other services through Rancher. The diagram below shows the general idea. Unfortunately, it would take time to explain how this works. Hopefully we will write a follow up about this topic.

Greeneye’s continuous delivery using Rancher

Tests

I mentioned that we used to run our tests with docker run/docker-compose. As we don’t have access to docker daemon nor systemd, we decided to use Podman:

What is Podman? Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System. Containers can either be run as root or in rootless mode. Simply put: alias docker=podman. More details here. (Source: Podman)

Podman runs “out of the box” (although doesn’t support ARM64 at the moment).

So a replacement for a test would just be podman run ... same as with docker and a replacement for a docker-compose would be the following bash script:

It is a bit longer, but at the end it’s more or less the same as a docker-compose.yaml declaration.

What’s next

We are looking how to improve our services, and this seems to show some good result by now:

I was quite shocked when I saw this. The great thing about BuildKit is that it works much better with multi stage Dockerfile. So we ended up with a 50% improvement - without caching!

We still have some issues to figure:

• We would like to run heavy scale tests.

• Should we keep BulidKit’s data in a PV or run a cronjob that cleans it? Maybe registries are more than enough.

• We are using a tool to clean our local registry, but it’s not maintained anymore. So we should figure a better way to do that.

• We should clean our ACR and GCR registries as we keep a lot of old junk.

• We would like to take a look at Github’s pipeline caching. Maybe one day this will be possible in Azure DevOps?

• Project Teleport seems like an interesting idea, although I’m a bit worried about the network.

Interesting projects I saw during this research:

• BuildKit CLI for kubectl

• Kyverno

That’s it for today. If you find this blog post interesting, or you have any farther questions, please feel more than welcome to tweet or DM me on Twitter @shakedko.

Run performance tests on a Real-time system without leaving the office

Here at Greeneye, our system is composed of many processes and jobs responsible for the entire pipeline, from taking pictures in Real-time to deciding whether we want to spray the pictured spot.

The pipeline itself consists of multiple hardware components and physical constraints - from the tractor itself, which carries the entire system, through the speed sensor for detecting and sampling speed, the cameras, nozzles, and more.

Step 1: Get rid of hardware dependencies.

Running performance tests for a hardware-dependent system will require setting up a complicated, expansive, and not-scalable environment.
The second step in our progress was to design our system to be hardware-injected. What is that mean?
Imagine your environment includes a speed sensor. Whenever the speed sensor indicates a positive speed, the system starts working.

Mocking the system’s speed will grant us two essential benefits:

1. Starting the system without having to actually drive a vehicle.

2. Test and benchmark your system at a dynamic speed.

Implementing the hardware-injection will be as described in figure 2.
A configurable variable will determine whether we will use the real or mock hardware. The mocked hardware return-value will be configurable as well.
Both real and mock hardware write their output to the exact same shared-object, it prevents the hardware consumers from being affected by setting real/mock hardware source.

“It does not matter how intelligent you are, if you guess and that guess cannot be backed up by experimental evidence–then it is still a guess.”

-Richard Feynman

One of the most important principles in developing a performant system is the ability to measure and monitor your performance. Metrics, benchmarking and profiling representation will be the best indicators for your development efforts.

In order to make our system testable, we marked few goals:

1. Test the influence of a new detection/classification model on the prediction accuracy.

2. Benchmark every new model.

3. Test every model with a large set of parameters and thresholds, in order to find the set that maximizes our prediction accuracy. i.e: Hypterparameter tunning.

4. Profiling our system’s bottlenecks.

5. Be able to run the tests on a custom independent cloud agent.

Before running into implementation, we encountered two fundamentals issues:

1. The system must be configurable, in order to be able to inject different and multiple sets of configurations without changing the code itself.

2. Our system depends on many hardware components. we must get rid of these dependencies to be able to run performance tests at scale.

Step 2: Make your system configurable

The first step towards running performance tests was converting all configurable variables to be injected from a configuration file.

As described in figure 1, for giving us the ability to define a different and meaningful set of parameters for every test, we use a configuration file and a runtime process to handle edit requests for that configuration file.

figure 1.

figure 2.

Step 3: Simulating the system

Having the ability to set configurable variables at runtime and mock the hardware dependencies, the last piece of the puzzle will be completed by orchestrating the sets of configuration and test their outputs.
Implementing the simulation process will be described in figure 3.
The simulator process loads the desired test configurations. For every test permutation, it will simulate the system and save the relevant output and configuration for a unique location.
After simulating the system for all the different sets of configurations, the only thing left is to post-process and test the output corresponding to the attached configuration.

figure 3.

Step 4: Tests and Visualization

The last part of our process will be handling the system’s output and opening a window into our system’s performance. Having multiple sets of results, including predictions/benchmarks/logs data and the specific configuration that produced those performances, can be leveraged quickly to find the best configuration for our system. 

Here at Greeneye, we use the ClearML platform to analyze, research, and visualize our tests and performances. A cool feature in ClearML is the ability to compare different runs. Figure 4 shows a Comparison between multiple sets of configuration outputs, helping us choose the best detection models for our system. Using the configuration attached for each experiment, we can quickly reproduce the best parameters for our system and use them in our production environment. Another bonus in ClearML is that each run saves not only the output but also the input i.e, the configuration file that produced that results

figure 4

In addition, we used the simulator to efficiently operate a benchmarking test. The simulator sends different speed values for our system and analyzes the performances for every single speed. That helped us discovering our system’s weaknesses, points of failure, and limits. At figure 5, you can see a graph of dropped frames as a function of the tractor’s speed. Thanks to step 2, The tractor is still parking inside the garage, the tests are running on the cloud.

Figure 5.

Summary

Developing a Real-time system will always require testing its performance. To do so, and do it fast, elegant, scalable, and without requiring many resources, we must design our code to be configurable and capable of mocking hardware dependencies. Keeps those in mind will give you one of the best gifts a complex-system engineer can ask- the ability to run your system independently, in a lightweight mode, and configured exactly the way you want it.

Livne Rosenblum,
Tractor Team Lead @ Greeneye Technology

"Trace-viewer to the rescue" - Google's Trace-viewer as a tool for code profiling

Analyzing code performance is a difficult task. There are a lot of code profilers on the market, each with unique characteristics. It’s hard to know which one is best for your needs. Even after choosing one, making it work with your code is not straightforward - it can take time to incorporate a profiler before you start using it. Most times, the profiling itself inhibits performance.

This post will discuss how we utilized Chome’s Trace-viewer, a visualization tool for profiling processes in web pages, to get a fast and easy profiling ability on a real-time embedded system.

The need for visualization

Sometimes when writing a code, one can simply know it works by taking a peek at the logs. In real-time, there is a point where it becomes too complex and runs too fast to spot bugs by watching the output.
Moreover, our system comprises several independent modules that use shared memory and events to work in sync. Therefore, we don’t have a single log file where you can inspect the system’s behavior as a whole.

We had performance issues on a multi-module system. We didn’t know what caused them, where to look, and lacked the tools to find out. When you try to tackle problems that you are unfamiliar with, it is helpful to visualize as much data as possible to identify where those problems are. Visualizing helps you recognize trends and bottlenecks and gives you a deeper understanding of the code. This is true even if you are the writer of the code.

We were at an early stage of development, and we didn’t want to make the investment needed to start working with a full-scale profiler. We needed a profiling method. It had to:

• Show how a large number of events across multiple modules affect each other.

• Be easily incorporated into the codebase

• Have a minimal effect on the runtime itself

Trace-viewer

After some research, we came across Trace-viewer:

Trace-viewer is a tool for generating a timeline of events during a run of a web page. You use it for diagnosing and analyzing performance and behavior. It is a part of chrome and chromium browsers and is accessible via chrome://tracing.

fig.1 - Trace-viewer

Basic functionality:

• The X-axis is time - going down to the microsecond. The Y-axis is the process and\or thread on which the event took place.

• Each event is displayed as a bar on the timeline. It has a beginning, an end, and metadata describing the event.

• You can zoom in\out and traverse over the timeline, measuring event duration and derive conclusions based on how events behave in adjacency to other events.

• You can pair each event with descriptive data.

Trace-viewer is used most commonly by recording events at runtime and offline by uploading JSON formatted text files. Its format is relatively simple - a large JSON object made of lots and smaller objects representing events.
There are two main types of events:

1. A timed event, with a beginning and an end, is represented by two entries of a JSON object, each one with a timestamp. Those entries don’t even have to come in a specific order.

2. A transient event happens at a certain point in time and doesn’t have a length associated with it. This is represented by a single entry of the JSON object.

Once you understand this simple way of describing events, it all comes down to generating an entry with a timestamp whenever you have an event in your code you want to measure. One limitation of using this tool is - events must be nested. If event B starts after event A, it must end before A ends. But it can easily have a workaround with a bit of creativity.

After selecting the tool, there is a new challenge: quickly reporting these events out of our code without polluting it.

Integrating into our system

• We’ve implemented a library that allows the programmer to put measure points throughout the code -
  Each start and end measures make a timed event.

• The system collects the events in batches and sends out a combined JSON message using Redis protocol.
  Having the metrics be sent out in batches made reduce the impact of the profiling process. The profiling process has minimal effect on the whole system performance.

• We paired each event with relevant data such as operating speed, sizes of data structures, etc.

We started off using those measure points to measuring large chunks of our code and worked out way down to shorter, more precise events that are of interest. By the end, we had most of our modules mapped out to events. We could now “record” a run of the system, stop and view the timeline to analyze its performance.

• We could see how specific modules behave when other GPU\CPU heavy operations are happening in parallel.

• We could measure how much time it took, for input, on one side of the system to be output 3-4 modules down the pipeline.

• We could roughly understand how much we utilize the system by measuring the idle parts of the graph.

• We now have a visual proof of assumptions and design choices we made when writing the code.
  Visual tools help to answer questions such a “how much time should we allow an operation before declaring timeout?”; “how many threads should be put to work on a task before suffering overhead?”
  Those questions are suddenly made clear and can be deduced from measuring time intervals on a graph.

From that point on, we could see what was going on in each run at a single glance.

fig.2 - multiple threads

In fig.2, for example, we can see 5 threads receiving jobs and performing tasks.
We can see how certain tasks took longer when run in parallel to other tasks.
We can measure to see if there is a direct proportion between the increase of runtime and the number of tasks run parallel.
we can visually understand how fast we can run with this number of threads before failing to keep up

One of the biggest benefits of visualizing a problem is the ability to detect patterns. A task can usually take ~10ms, and once every 500 times, something unexpected happens, and it suddenly takes 80ms. 80ms might be well within the definition of correct behavior, so you might never detect this anomaly. Using a visualization tool such as this - this anomaly pops out.

Our mind is good at detecting changes:_)

fig.3 - congestion and transient events

In fig.3, we see marked with a red rectangle, a point where hardware strains caused congestion.

Fail points are reported as transient events and are visible as vertical lines. In contrast to timed events, line marking transient events stays the same width no matter how zoomed in or out you go, so they are always visible.

The congestion around that area might be missed if not for a visual aid such as this.

Future work

A tool such as Trace-viewer does not stand alone as a debugging tool. It simply points you in the directions of problems, so you can be focused when using more in-depth tools. You might want to keep improving on what events you are measuring, how many of them, and what data you are sending. With each event, you can use tools to automatically parse those messages and outputs and make a report pointing out anomalies, bugs, behaviors, and trends.

But in an early stage, when all those tools are not yet in place, and you are still learning how your code behaves - a tool such as this can help a lot and deepen your understanding of what is happening under the hood.

Conclusion:

Profiling your code gives you a better understanding of how well it runs. It is advisable to profiling the code as soon as possible. However - It demands considerable resources. In the early stages of development, most of the effort goes into writing new code rather than testing the existing one.

Using tools such as Trace-viewer to visualize performance can bring lots of value for a minimal cost. It helps you see the bigger picture of how different modules work together and helps you make better design and implementation choices.

Nir Erez
Software developer at Greeneye technology

Kubernetes Liveness and Readiness Probes in a Real-Time System

Expectation vs. Reality

Everyone who got to work with Kubernetes, especially as a deployment system, probably noticed the option of `liveness` checks to their system, got very excited for the opportunity to make the system even more robust, but then realized it is not always amazing. Liveness and readiness failures can be confusing and frustrating while trying to understand what causes your system to keep crashing constantly.

The probes Kubernetes offers are necessary but need to be an exact match with your system. It takes time and effort to implement the right for your needs and keep it suitable while the program is changing. As discussed in many blogs, you have to be familiar with the difference between the probes and how they work to get started. One blog post by Colin Breck has helped me understand how to continue working until I reach my goal.

Let’s talk Real-Time

The blog I mentioned before, and many others I could find, introduce implementations for health checks for a web application. In Greeneye Technology, we are developing a Real-Time system that, by definition, is completely different than a web application, with obvious differences in properties and limitations. Most importantly, in a real-time application, we can’t afford to have any of our containers down or stuck for a long time.

Kubernetes has three types of probes: an HTTP GET request on the container’s IP, a TCP connection to the specified container, or running an “exec probe” that as a command inside the container. For a system that has to be efficient in memory and resources, running an internal server specifically for probing is less wanted. Moreover, the realtime

From the description in Kubernetes docs, the “exec command” works as follows:

Command is the command line to execute inside the container, the 
working directory for the command is root ('/') in the container's 
filesystem. The command is simply exec'd, it is not run inside a shell, 
so traditional shell instructions ('|', etc) won't work. To use a 
shell, you need to explicitly call out to that shell. Exit status of 0 
is treated as live/healthy and non-zero is unhealthy.

Now all is left is to decide what command will be running inside the container. This command will decide if the container is healthy or not. The most common example you can find, is the one kubernetes suggests - the container writes (touch) to a file every period time, and the command kubernetes runs is to check if this file exists.

spec:
  containers:
  - name: container-1
    ...
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy

This can be a neat solution, but is it a solution for a real-time system? Definitely not. We can’t afford an additional I\O command from the container side, especially if it needs to be run every few seconds.

So our command for aliveness check has to be: 

1. Fast with no additional I\O operations.

2. Executed every second or even less.

3. Reliable, meaning it won’t fail when the container is actually healthy.

Our solution: Shared memory to the rescue!

Our multi-process system communicates between each process by a shared-memory object, memory saved in a specific place in the system’s memory, where every process can read and write to. Access to an object in the shared memory cost as accessing any variable in the global scope. 

How can shared memory help us with liveness and readiness checks? Exactly as writing to a file, but instead, writing to a shared memory object dedicated to this purpose. Implementing a short program that reads from that memory can be our command to execute inside the container.

The full flow:

Every process (aka container in our program) is responsible for reporting liveness to the shared memory object, every period of time, for example, every cycle in the main loop. Kubernetes runs a short program that does the following:

1. It checks the current state in the shared memory.

2. According to that-the probing command decides to exit with code 0 (healthy) or otherwise (not healthy).
   The decision can be simply comparing the timestamp of the last live report to the timestamp now.

Conclusion

With the exec command type Kubernetes has, there are almost no limits to what we can run as a liveness check. You can use any script or command, as long as it can be executed inside the container, and that’s it!

We can use this necessary feature with shared memory, make the program more robust, without major run time costs, and still be precise about the container’s status.

Make sure you understand the limits of your system and how these checks can beneficial instead of harmful. 

Some tips:

• First, define what it means that a process is alive or ready, and after you understand that, add in each place in the program a liveness report.

• Add logs to the command! They will help you understand quickly where it fails and on which container. You can see if a health check fails by seeing the events in the pod by one of these commands:

- kubectl get events
- kubectl describe pod

• Get to know where these checks can fail and whether there’s a difference between the containers. You can try to find loops or places the process can be stuck for a long time, as the processor won’t report live, and therefore Kubernetes will restart your container.

• Use all the parameters Kubernetes has for health probes, and define them carefully: initialDelaySeconds, periodSeconds, successThreshold and more. Full description here.

Good Luck!

Shelly Bekhor,
Realtime Developer @ Greeneye Technology

Continuous Integration: Injecting secrets to remote clusters with Rancher

We have moved our IoT infrastructure from Azure IoT Edge to Rancher, Fleet and k3s stack. During this migration, which we should talk about in an upcoming post, we had to figure out how to inject secrets when either installing a new cluster; or updating an existing one.

Our clusters are mostly IoT devices - Nvidia’s Jetson Xavier - which are deployed both in the fields and in different locations, such as: our office in Tel Aviv, at my own place, at our freelancer’s place in France and we also move devices from one place to another, especially due to COVID’s lockdowns.

Before taking any action, we had few things in mind:

1. We would like to automatically inject secrets to a new cluster that is being added to the clusters list.

2. We would like to automatically inject secrets to a group of clusters, while a group can be all of them together or part of them depending on the requirements.

3. We would like to manually inject secrets to one or more clusters in case needed.

4. We would like this to be controlled through our CI/CD platform - Azure DevOps.

5. We must keep security in mind. This part is very important to us and we will touch it in depth later on.

In the meanwhile, we have also posted a Github Issue asking for “Best practice to inject a secret to all clusters”. We have partially explained there the steps we took to answer all of our current requirements, and this post will try to give a more clear understanding of how and why we implemented our current solution. Let’s begin!

Automatically inject secrets to a new cluster, Azure DevOps and Security

In order to set this up, we had to follow these steps:

• We have created a new Azure DevOps project and a new user with very restricted access. This user can only run a specific job under this specific project and it cannot change anything within the job nor the project.

• We have generated a personal access tokens (PAT) for this user.

• We have created a Fleet deployment that:

  • Runs on all clusters.

  • Injects an ansible playbook as a ConfigMap

  • Runs ansible with limited user access, which only does a curl request with the deviceInputName to the new Azure DevOps project with the limited PAT mentioned above.

  • We extract deviceInputName by using:
    IP_ADDRESS=$(hostname -is)

  • We run ansible with the following command:

    ansible-playbook -v -t init -i $IP_ADDRESS, /device/playbook.yaml --extra-vars "deviceInputName=$IP_ADDRESS"

    • The reason we use $IP_ADDRESS, is because the IP address is actually the hostname of the machine.

• We have installed rancher CLI in Azure DevOps agents as we had to use it in order to connect to the remote device k3s cluster.

• Once Azure DevOps has started, it will run the following commands while knowing only the deviceInputName which must be escaped!

  • Find the rancher project ID by the cluster name:
    PROJECT=$(rancher login -t $TOKEN $URL < /dev/null 2>/dev/null | grep $DEVICE_INPUT_NAME | grep -i default | awk '{ print $3 }')

  • Login into rancher:
    rancher login -t $TOKEN --context $PROJECT $URL

  • Side note: it would have been nice if we could have gotten the project ID in a nicer way. If you have a suggestion, please let us know in the comments or directly in my Twitter account.

  • We have defined the $TOKEN, $URL, and the rest of the secrets that we are injecting using Azure KeyVault and have integrated it with Azure DevOps’ Library.

  • Now, the rest of the tasks look like this:

Create secrets on remote k3s clusters through Azure DevOps using Rancher CLI

• Once done, the task will update our Telegram channel that everything has been successfully set.

Automatically inject secrets to a group of clusters

The reason I put this here is because I want to emphasize the strength of rancher, k3s, and fleet.

As I have mentioned above, the ansible job runs on all clusters. This works because we are using a specific label that we require when adding a new cluster. We are using the label as we might need to differentiate between secrets and environments at some point.

Then, once we want to update all of our clusters, we only need to update something in our ConfigMap, which can be a “version bump” and push it to git. From there, our helm deployment template will be automatically updated, as we have added this checksum:

  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Manually inject secrets to one or more clusters

At this point we can always decide to run our Azure DevOps job manually by providing it the relevant deviceInputName and by doing that, we can create or update an existing cluster settings without any problem.

Summary

Since we have moved to the Rancher, k3s, and fleet stack we have been very satisfied as this has helped us to be aligned with our entire software and everything is now containerized and k8s friendly.

We are always looking for better ways to make our software more secure, and as we are happy with our solution to inject secrets and configuration from our CI, we are still looking into other options and more than happy to hear other ideas, so if you have any please drop us a comment or tweet about it.