Continuous Integration: Injecting secrets to remote clusters with Rancher

We have moved our IoT infrastructure from Azure IoT Edge to Rancher, Fleet and k3s stack. During this migration, which we should talk about in an upcoming post, we had to figure out how to inject secrets when either installing a new cluster; or updating an existing one.

Rancher Labs

Our clusters are mostly IoT devices - Nvidia’s Jetson Xavier - which are deployed both in the fields and in different locations, such as: our office in Tel Aviv, at my own place, at our freelancer’s place in France and we also move devices from one place to another, especially due to COVID’s lockdowns.

Before taking any action, we had few things in mind:

  1. We would like to automatically inject secrets to a new cluster that is being added to the clusters list.

  2. We would like to automatically inject secrets to a group of clusters, while a group can be all of them together or part of them depending on the requirements.

  3. We would like to manually inject secrets to one or more clusters in case needed.

  4. We would like this to be controlled through our CI/CD platform - Azure DevOps.

  5. We must keep security in mind. This part is very important to us and we will touch it in depth later on.

In the meanwhile, we have also posted a Github Issue asking for “Best practice to inject a secret to all clusters”. We have partially explained there the steps we took to answer all of our current requirements, and this post will try to give a more clear understanding of how and why we implemented our current solution. Let’s begin!

Automatically inject secrets to a new cluster, Azure DevOps and Security

In order to set this up, we had to follow these steps:

  • We have created a new Azure DevOps project and a new user with very restricted access. This user can only run a specific job under this specific project and it cannot change anything within the job nor the project.

  • We have generated a personal access tokens (PAT) for this user.

  • We have created a Fleet deployment that:

    • Runs on all clusters.

    • Injects an ansible playbook as a ConfigMap

    • Runs ansible with limited user access, which only does a curl request with the deviceInputName to the new Azure DevOps project with the limited PAT mentioned above.

    • We extract deviceInputName by using:
      IP_ADDRESS=$(hostname -is)

    • We run ansible with the following command:

      ansible-playbook -v -t init -i $IP_ADDRESS, /device/playbook.yaml --extra-vars "deviceInputName=$IP_ADDRESS"

      • The reason we use $IP_ADDRESS, is because the IP address is actually the hostname of the machine.

  • We have installed rancher CLI in Azure DevOps agents as we had to use it in order to connect to the remote device k3s cluster.

  • Once Azure DevOps has started, it will run the following commands while knowing only the deviceInputName which must be escaped!

    • Find the rancher project ID by the cluster name:
      PROJECT=$(rancher login -t $TOKEN $URL < /dev/null 2>/dev/null | grep $DEVICE_INPUT_NAME | grep -i default | awk '{ print $3 }')

    • Login into rancher:
      rancher login -t $TOKEN --context $PROJECT $URL

    • Side note: it would have been nice if we could have gotten the project ID in a nicer way. If you have a suggestion, please let us know in the comments or directly in my Twitter account.

    • We have defined the $TOKEN, $URL, and the rest of the secrets that we are injecting using Azure KeyVault and have integrated it with Azure DevOps’ Library.

    • Now, the rest of the tasks look like this:

Create secrets on remote k3s clusters through Azure DevOps using Rancher CLI

Create secrets on remote k3s clusters through Azure DevOps using Rancher CLI

  • Once done, the task will update our Telegram channel that everything has been successfully set.

Automatically inject secrets to a group of clusters

The reason I put this here is because I want to emphasize the strength of rancher, k3s, and fleet.

As I have mentioned above, the ansible job runs on all clusters. This works because we are using a specific label that we require when adding a new cluster. We are using the label as we might need to differentiate between secrets and environments at some point.

Then, once we want to update all of our clusters, we only need to update something in our ConfigMap, which can be a “version bump” and push it to git. From there, our helm deployment template will be automatically updated, as we have added this checksum: 

  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Manually inject secrets to one or more clusters

At this point we can always decide to run our Azure DevOps job manually by providing it the relevant deviceInputName and by doing that, we can create or update an existing cluster settings without any problem.

Summary

Since we have moved to the Rancher, k3s, and fleet stack we have been very satisfied as this has helped us to be aligned with our entire software and everything is now containerized and k8s friendly.

We are always looking for better ways to make our software more secure, and as we are happy with our solution to inject secrets and configuration from our CI, we are still looking into other options and more than happy to hear other ideas, so if you have any please drop us a comment or tweet about it.