Open in app

Sign in

Write

Sign in

[IaC] Continous Delivery with Crossplane and FluxCD : EKS cluster example

Nicolas Vogt
14 min readSep 17, 2022

--

Photo by Matteo Modica on Unsplash

Crossplane is an Infrastructure as Code description language and engine. Like Terraform and Pulumi, it comes with connectors to any public Cloud Provider. The philosophy of Crossplane is to manage infrastructures the same way you manage your Kubernetes workloads, avoiding context switching for DevOps teams. In this guide, I will show you how to deploy an EKS cluster on AWS using FluxCD.

Forewords

Described below is the way to implement an automated deployment pipeline of an EKS cluster with a predefined set of applications. For this purpose I will use FluxCD and Crossplane. The first because it offers a more elegant way to handle dependencies than ArgoCD, the latest because the infrastructure description will be in the same language as our kubernetes deployments, and so we can manage both of them with the same tools.

This guide is overly inspired from the work of Vijayan Sarathy. You will find the link to the GitHub repository at the end of this page, altogether with the official AWS documentation. I decided to go a little further into the details to help you filling the gaps.

I followed the part two of Vijayan’s guide, the part one was focused on ArgoCD. I also covered this part, you will find all the guidance here if you are interested.

What is FluxCD?

Flux is a CNCF project offering Continuous Delivery tools for Kubernetes, enabling you to automate developments, from your code repository to production without any manual intervention.

Here are the set of features covered by Flux :

  • Automated deployment (CD)
  • Progressive release (A/B, canary release, .. )
  • Integration with the major container registries and CI workflow providers
  • integration with any Kubernetes and all the associated tooling
  • Multi-tenancy compliant
  • Alerting and notifications

When to apply?

  • To automate infrastructure deployments
  • To avoid context switching, using the same language and tools for Infrastructure as Code and Kubernetes deployments
  • If command line is all what you need and a WebUI is a nice to have but not mandatory

FluxCD versus ArgoCD

FluxCD offers a very convenient feature to handle dependencies between your Kustomizations, for example if you want to deploy applications once an EKS has been created. This use case is a bit tricky with ArgoCD because there is no equivalent to the Flux’s dependsOn directive. You can use ArgoCD syncwaves to reach the same objective, but requires custom scripting to perform an elaborate dependency handling mechanism, better than just a pause container. A native way to do it would have been welcomed.

On the other hand, ArgoCD is more flexible if you intend to manage several projects across multiple teams. This is the reason why I picked it in the first place. To me, it makes sense to separate different domains in different repositories so that each team can have the full ownership of a module / project. The mono-repo approach of FluxCD makes this a little more rigid. But in the example below, we will only instantiate Crossplane compositions, so those can be managed separately and the separation of concerns remains respected.

Also, if you want to have a Webui, then this is quite straightforward, FluxCD does not provide one.

The items above are from my personal experience with Crossplane compositions on both tools. If you want to extend the comparison to a broader list of criterias, check this website.

Before you start

If you intend to follow this guide, you will need the following tools. The links bellow point to the installation procedure for each one of them :

Make sure you have those tools up to date before you start this guide.

Preparation

Setup a Kubernetes cluster

First of all, you need a Kubernetes cluster. Either you are using your own computer and you can simply install kind/ minikube, or you can proceed as I did with the creation of a new cluster on AWS using eksctl. Here is a manifest file to help you with:

mycluster.yaml

Store this file in mycluster.yaml or whatever you want to name it, and type the command :

~ » eksctl create cluster -f mycluster.yaml

The creation will take a moment (approximately 10–15 minutes), and you will end up with one spot node in your EKS cluster.

At the end, update your kubeconfig file

~ » aws eks --update-kubeconfig --name cicd-fluxcrossplane

FluxCD Bootstrap

If you don’t have a repository yet, go to GitHub and create one. FluxCD need one to retrieve the configuration from. Every change made to the repository on a specific branch will be applied to your cluster.

The following command will bootstrap the cluster with Flux, it will create a GitRepository custom resource pointing to the repository you specify in here. A GitHub token will be prompted when you issue the command, follow this documentation to to generate a Read Only token on your repository.

# define your own variables
~ » export CLUSTER_NAME=XXXX
~ » export GITHUB_TOKEN=XXXXX
~ » export GITHUB_USER=XXXX# create a namespace
~ » kubectl create ns flux-system# bootstrap your cluster
~ » flux bootstrap github \
  --components-extra=image-reflector-controller,image-automation-controller \
  --owner=$GITHUB_USER \
  --namespace=flux-system \
  --repository=eksfluxcrossplane \
  --branch=main \
  --path=clusters/$CLUSTER_NAME \
  --personal

In this example, my repository is in GitHub and is called eks-gitops-crossplane-flux, the main branch will be pulled and the clusters/$CLUSTER_NAME will be watched.

Encrypt your AWS credentials with kubeseal

Crossplane will be used to deploy resource in your AWS account, and it requires suitable credentials to do so. The configuration will be made available to your cluster via Git, and you do not want to have your credentials in clear in your code repository. A quick reminder here, base64 is not an encryption, it is an encoding function that anyone can use to encode and decode a string.

We will use kubeseal. Our goal is to generate a master key in your cluster, in a restricted namespace, and use it to encrypt your AWS credentials so that the YAML file stored in GIT is safe and can be shared. Only an authorized list of people and Crossplane will be able to decrypt it.

Kubeseal will be deployed together with Crossplane, but we will prepare the file in advance. I already explained this stage in my precedent article, but i will share with you a quicker way to do it.

~ » helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets~ » helm repo update

Install the helm chart like this :

~ » kubectl create ns sealed-secrets~ » helm install sealed-secrets \ 
      -n sealed-secrets \
      --set-string fullnameOverride=sealed-secrets-controller \
      sealed-secrets/sealed-secrets

You will now have to encrypt your AWS credentials, the ones that crossplane will use to create / update resources. You can use your own credentials but it is not a very good idea if you intend to use Crossplane in production. Following the least privileges principle, you must create an account with the minimum permissions for that purpose.

Then input them in a mycreds.txt file formatted like that :

[default]
aws_access_key_id = XXXXXX
aws_secret_access_key = XXXXX

And encode this file in base64 with the following command :

~ » cat mycreds.txt | base64 | tr -d "\n"

Insert the result string in a kind: Secret resource, replacing the BASE64_STRING string in the following file :

aws-credentials-unsealed.yaml

Generate your sealed yaml file with the following command:

~ » kubeseal --controller-name=sealed-secret-controller --controller-namespace=sealed-secrets --format yaml < aws-credentials-unsealed.yaml > aws-credentials-sealed.yaml

This command will generate your target file aws-credentials-sealed.yaml. Keep this file for a moment, we will use it later. You can then remove the mycreds.txt and aws-credentials-unsealed.yaml or move them to a secure location, but do not push those files to your Git repository.

Now, it is also important to save the master key that helped you to encrypt the file :

~ » kubectl get secret -n sealed-secrets -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealing-master.yaml

Remove the sealed-secrets helm chart, the secret engine will be part of the crossplane deployment :

~ » helm uninstall sealed-secret -n kube-system

Finally, re-apply your sealing-master secrets that was removed together with the helm chart. The future sealed-secrets deployments will reuse this master key :

~ » kubectl apply -f sealing-master.yaml

Make sure you save this master key to someplace safe, in case you need to rebuild your cluster. Otherwise you will not be able to decrypt your files.

Also, you can add this line to your .gitignore file so that it will be excluded from your git commands. This will prevent you from storing the key together with the encrypted file.

~ » echo "sealing-master.yaml" >> .gitignore

Kustomization

So far, we have started a Kubernetes cluster with a GitRepository resource linking it to a GitHub repository. We also generated an encrypted file containing the credentials to use with Crossplane.

Now things are getting serious. Before we will be able to deploy an EKS cluster, we will need those elements :

  • The Crossplane engine, providing a set of resources to automate deployments
  • The AWS provider, APIs to deploy AWS resources (VPC, subnet, …)
  • The Composition, a module containing multiple resources and composing an EKS cluster deployment
  • The XR, a customized instantiation of the composition, also known as a claim

To start with, we will indicate Crossplane where to find its configuration. It will pull the git repository every 30 seconds and trigger actions when a change is detected. Type the following commands :

~ » mkdir -p ./clusters/${CLUSTER_NAME}~ » flux create kustomization crossplane \ 
    --source=flux-system \
    --namespace=flux-system \
    --path=./crossplane \
    --prune=true \
    --validation=client \
    --interval=30s \
    --export > ./clusters/$CLUSTER_NAME/crossplane.yaml

Then create two directories where you will store your configuration :

  • crossplane
  • deploy

The first one will contain the Kustomization manifests with the target namespace, health checks and dependencies, the second the actual resources to deploy.

Deploy the Crossplane engine

The first component to deploy is the crossplane core engine, we will deploy the official helm chart. For the sake of consistency, I will reproduce the naming in the original guide. Create a new file in crossplane/crossplane-helmrelease.yaml with the following content :

crossplane/crossplane-helmrelease.yaml

Create also the following files, in the ./deploy/crossplane-core directory:

Then add those files, commit and push the change to your repository :

~ » git add -A~ » git commit -m "adding crossplane core engine"~ » git push

Watch your Kustomization resources until you see your new resource :

~ » kubectl get kustomization -ANAME                               AGE   READY     STATUS
crossplane                         52s   True      Applied revision: main/eb9ee4e1d61cef251c98973b5daef7aaa1fda8c4
crossplane-helmrelease             52s   Unknown   running health checks with a timeout of 2m0s
flux-system   flux-system                        44m   True    Applied revision: main/eb9ee4e1d61cef251c98973b5daef7aaa1fda8c4

When your resources are all ready you can go to the next section. If it is not, try to run a describe on the kustomization to see the last events.

Deploy an AWS Provider

In the previous section, we have deployed the crossplane engine, now we will deploy the necessary resources for AWS. Crossplane uses providers to fit every use cases. In order to deploy AWS resources, you will need to have the definition of these resources. This is what the provider is intended to provide.

Create the crossplane/crossplane-provider-package.yaml

crossplane-provider-package.yaml

Create also the following files, in the ./deploy/crossplane-aws-provider directory:

Then add those files, commit and push the change to your repository :

~ » git add -A~ » git commit -m "adding crossplane aws provider"~ » git push

Watch your kustomization resources until you see your new resource as READY :

~ » kubectl get kustomization -A --watchNAMESPACE     NAME                               AGE   READY   STATUS
flux-system   crossplane                         30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
lux-system   crossplane-helmrelease             30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   crossplane-provider-package        30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   flux-system                        44m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88

If you want to be sure your package is HEALTHY, use the following command :

~ » kubectl get pkgNAME                                                 INSTALLED   HEALTHY   PACKAGE                                               AGE
provider.pkg.crossplane.io/crossplane-provider-aws   True        True      registry.upbound.io/crossplane/provider-aws:v0.23.0   40m

Create and deploy your own Cluster Composition

A quick reminder before we go any further, Crossplane provide flexibility through what is called a Composition. You can instantiate simple resources with the provider only, because it contains a set of Composition, but when it comes to more complicated or customized deployments you will need to create your own composition. Compositions can be associated as modules, or aggregate sets of resources you can instantiate, also known as claim.

https://crossplane.io/docs/v1.6/media/composition-how-it-works.svg

To build a composition, you need 3 files :

  • A configuration manifest
  • A composition manifest
  • A composite resource definition manifest

So, let’s start building our own composite resource with an EKS Cluster and all its related resources : VPC, subnets, NAT Gateways, …

Make sure you have the following :

  • download the kubectl crossplane set of commands, you will find it here
  • an OCI-compatible registry, you can create one in AWS with Elastic Container Registry (ECR).

Then, create a new directory with the following files :

This file will specify your package’s metadata and the dependencies. We will use the 0.23 version or further of the AWS provider.

This file contains the Composition, ie all the resources that will be created. Two private and two public subnets will be created, will all the necessary elements to make the connection to the outside world. I encourage you to look more into the details if you intend to use it in production, in order to make it fit your environment. The file is quite long, so it will not be printed here. Just follow the link to get the content of the file.

Note that the node instance type (t2.medium) is hardcoded as parameters in the EKS cluster definition. If you want to change it, just edit the last resource at the end of the file.

The last file defines the variables and their boundaries. The allowed values for the region are, in my example, eu-west-2 and eu-west-3. Feel free to change it to fit your need, it also valid for the allowed Kubernetes versions.

Once those files are correctly defined, you can simply build your package with the following command :

~ » kubectl crossplane build configuration

This command will create a .xpkg file in the current directory. This package has to be pushed in a registry in order to make it available to your cluster, so you will need a Container Registry to push your package to. If you don’t have one yet, you can create a public AWS ECR repository, and push your package with the following command :

# Set your variables
~ » export REGISTRY=public.ecr.aws/XXXXX
~ » export REGION=my_region
~ » export PACKAGE=namespace/my_package_name
~ » export VERSION=my_package_version# Login to your registry
~ » aws ecr-public get-login-password --region $REGION | docker login --username AWS --password-stdin $REGISTRY# Push your package
~ » kubectl crossplane push configuration $REGISTRY/$PACKAGE:$VERSION

We have created a new composition, and push it to our container registry. Now we have to deploy it. This is done in FluxCD, we need to create a Kustomization in order to deploy the composition in our Kubernetes cluster.

Create the crossplane/crossplane-configuration-package.yaml

crossplane-configuration-package.yaml

Create the following files, in the ./deploy/crossplane-composition directory:

Then add those files, commit and push the change to your repository :

~ » git add -A~ » git commit -m "adding crossplane configuration package"~ » git push

Watch your Kustomization resources until you see your new resource as READY :

~ » kubectl get kustomization -A --watchNAMESPACE     NAME                               AGE   READY   STATUS
flux-system   crossplane                         30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   crossplane-configuration-package   30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   crossplane-helmrelease             30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   crossplane-provider-package        30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   flux-system                        44m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88

If you want to be sure your package is HEALTHY, use the following command :

~ » kubectl get pkgNAME                                                 INSTALLED   HEALTHY   PACKAGE                                               AGE
provider.pkg.crossplane.io/crossplane-provider-aws   True        True      registry.upbound.io/crossplane/provider-aws:v0.23.0   40mNAME                                                         INSTALLED   HEALTHY   PACKAGE                                                AGE
configuration.pkg.crossplane.io/crossplane-eks-composition   True        True      public.ecr.aws/a3h8w9f7/crossplane/eks-cluster:1.0.1   39m

Deploy an XR Resource

Now it is time to deploy our EKS cluster.

Create the crossplane/crossplane-composite-resources.yaml

crossplane-composite-resources.yaml

In the ./deploy/crossplane-xr directory, edit the following file and specify your own configuration :

Commit and push the change to your repository :

~ » git add -A~ » git commit -m "adding crossplane resources"~ » git push

Watch your Kustomization resources until you see your new resource as READY :

~ » kubectl get kustomization -A --watchNAMESPACE     NAME                               AGE   READY   STATUS
flux-system   crossplane                         30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   crossplane-configuration-package   30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   crossplane-helmrelease             30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   crossplane-provider-package        30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   crossplane-resources               30m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88
flux-system   flux-system                        44m   True    Applied revision: main/f892fcb29b28a11500ccf3c87b1aedb729cedd88

The cluster creation could take few minutes, you can use the following command to know when the cluster is READY :

~ » kubectl get cluster                                                 NAME                            READY   SYNCED   AGE
crossplane-prod-cluster-v294j   True    True     10m

This command will help you to list all the related resources :

~ » kubectl get crossplane

At the end of the cluster creation, a secret should be generated containing the kubeconfig information. Here is how you can retrieve them :

~ » kubectl get secret -n flux-system crossplane-flux-cluster-connection -o yaml | yq '.data.value' | base64 -d > kubeconfig

Then you can connect to your new cluster specifying the location of the kubeconfig file :

~ » kubectl get ns --kubeconfig kubeconfigNAME              STATUS   AGE
default           Active   41m
flux-system       Active   3m42s
kube-node-lease   Active   41m
kube-public       Active   41m
kube-system       Active   41m

With this temporary access, you can update the aws config map to add your account as an administrator. Once this is done, remove the kubeconfig file.

Deploy something in your newly available cluster

Start by initiating a new directory for the applications you want to deploy.

flux create kustomization applications \        
  --source=flux-system \
  --namespace=flux-system \
  --path=./applications \
  --prune=true \
  --validation=client \
  --interval=30s \
  --export > ./clusters/$CLUSTER_NAME/applications.yaml

Create the following file in the applications directory :

The associated file to this kustomization in ./deploy/webapp/ is :

Create the following file in the applications directory :

application-prometheus.yaml

The associated files to this kustomization in ./deploy/monitoring/ are :

Commit and push the change to your repository :

~ » git add -A~ » git commit -m "adding crossplane resources"~ » git push

Watch your Kustomization resources until you see your new resource as READY :

~ » kubectl get kustomization -A --watchNAMESPACE     NAME                               AGE    READY   STATUS
flux-system   application-prometheus             4m2s   False   unable to get 'flux-system/application-webapp' dependency: Kustomization.kustomize.toolkit.fluxcd.io "application-webapp" not found
flux-system   applications                       4m2s   True    Applied revision: main/928087467a4d3fb44b86f7c66e238aec9532c265
flux-system   crossplane                         61m    True    Applied revision: main/928087467a4d3fb44b86f7c66e238aec9532c265
flux-system   crossplane-configuration-package   61m    True    Applied revision: main/928087467a4d3fb44b86f7c66e238aec9532c265
flux-system   crossplane-helmrelease             61m    True    Applied revision: main/928087467a4d3fb44b86f7c66e238aec9532c265
flux-system   crossplane-provider-package        61m    True    Applied revision: main/928087467a4d3fb44b86f7c66e238aec9532c265
flux-system   crossplane-resources               61m    True    Applied revision: main/928087467a4d3fb44b86f7c66e238aec9532c265
flux-system   flux-system                        75m    True    Applied revision: main/928087467a4d3fb44b86f7c66e238aec9532c265

Then connect to the newly created cluster to follow your deployments.

Conclusion

Well I hope this guide helped you to setup a CICD pipeline with Crossplane, I tried to describe all the steps I went through, here are reference links to help you out through your journey:

I hope you enjoyed this guide, feel free to reach me if you need more information regarding this example.

Infrastructure As Code
Aws Eks
Kubernetes Cluster
Crossplane
Flux

--

--

Nicolas Vogt
Nicolas Vogt

Written by Nicolas Vogt

58 Followers
12 Following

Curious, most of the time, eager to learn something new when I’m not

Responses (2)

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams