badhouseplants/badhouseplants-net-old
Archived
4
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Activity

post: Dynamic environments per branch with ArgoCD

Browse Source 
post: add onw todo point
This commit is contained in:
Nikolai Rodionov 2023-02-26 14:29:29 +01:00 committed by Gitea
parent 495279cc86
commit bd3fa6bbab
6 changed files with 130 additions and 35 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
.drone.yml
View File

@ -78,15 +78,15 @@ steps:
- export ARGO_APP_BRANCH=$DRONE_BRANCH - export ARGO_APP_BRANCH=$DRONE_BRANCH
- export ARGO_APP_HOSTNAME="${DRONE_BRANCH}-dev.badhouseplants.net" - export ARGO_APP_HOSTNAME="${DRONE_BRANCH}-dev.badhouseplants.net"
- export ARGO_APP_IMAGE_TAG=$DRONE_COMMIT_SHA - export ARGO_APP_IMAGE_TAG=$DRONE_COMMIT_SHA
- kubectl get -f ./kube/applicationset.yaml -o yaml > /tmp/old_appset.yaml - kubectl get -f ./kube/applicationset.yaml -o yaml > /tmp/appset.yaml
- yq -i "del(.metadata.resourceVersion)" /tmp/old_appset.yaml - yq -i "del(.metadata.resourceVersion)" /tmp/appset.yaml
- yq -i "del(.metadata.generation)" /tmp/old_appset.yaml - yq -i "del(.metadata.generation)" /tmp/appset.yaml
- yq -i "del(.metadata.uid)" /tmp/old_appset.yaml - yq -i "del(.metadata.uid)" /tmp/appset.yaml
- yq -i "del(.status)" /tmp/old_appset.yaml - yq -i "del(.status)" /tmp/appset.yaml
- yq "del(.spec.generators[].list.elements[] | select(.branch == \"$ARGO_APP_BRANCH\"))" /tmp/old_appset.yaml > /tmp/clean_appset.yaml - yq -i "del(.spec.generators[].list.elements[] | select(.branch == \"$ARGO_APP_BRANCH\"))" /tmp/appset.yaml
- envsubst < ./kube/template.yaml > /tmp/elements.yaml - envsubst < ./kube/template.yaml > /tmp/elements.yaml
- yq '.spec.generators[].list.elements += load("/tmp/elements.yaml")' /tmp/clean_appset.yaml > /tmp/new_appset.yaml - yq -i '.spec.generators[].list.elements += load("/tmp/elements.yaml")' /tmp/appset.yaml
- kubectl apply -f /tmp/new_appset.yaml - kubectl apply -f /tmp/appset.yaml
- name: Deploy a main ApplicationSet - name: Deploy a main ApplicationSet
image: alpine/k8s:1.24.10 image: alpine/k8s:1.24.10
@ -103,16 +103,16 @@ steps:
- export ARGO_APP_CHART_VERSION=`cat chart/Chart.yaml | yq '.version'` - export ARGO_APP_CHART_VERSION=`cat chart/Chart.yaml | yq '.version'`
- export ARGO_APP_BRANCH=$DRONE_BRANCH - export ARGO_APP_BRANCH=$DRONE_BRANCH
- export ARGO_APP_IMAGE_TAG=$DRONE_COMMIT_SHA - export ARGO_APP_IMAGE_TAG=$DRONE_COMMIT_SHA
- kubectl get -f ./kube/applicationset.yaml -o yaml > /tmp/old_appset.yaml - kubectl get -f ./kube/applicationset.yaml -o yaml > /tmp/appset.yaml
- yq -i "del(.metadata.resourceVersion)" /tmp/old_appset.yaml - yq -i "del(.metadata.resourceVersion)" /tmp/appset.yaml
- yq -i "del(.metadata.generation)" /tmp/old_appset.yaml - yq -i "del(.metadata.generation)" /tmp/appset.yaml
- yq -i "del(.metadata.uid)" /tmp/old_appset.yaml - yq -i "del(.metadata.uid)" /tmp/appset.yaml
- yq -i "del(.status)" /tmp/old_appset.yaml - yq -i "del(.status)" /tmp/appset.yaml
- yq "del(.spec.generators[].list.elements[] | select(.branch == \"$ARGO_APP_BRANCH\"))" /tmp/old_appset.yaml > /tmp/clean_appset1.yaml - yq -i "del(.spec.generators[].list.elements[] | select(.branch == \"$ARGO_APP_BRANCH\"))" /tmp/appset.yaml
- yq "del(.spec.generators[].list.elements[] | select(.commit_sha == \"$ARGO_APP_IMAGE_TAG\"))" /tmp/clean_appset1.yaml > /tmp/clean_appset.yaml - yq -i "del(.spec.generators[].list.elements[] | select(.commit_sha == \"$ARGO_APP_IMAGE_TAG\"))" /tmp/appset.yaml
- envsubst < ./kube/main.yaml > /tmp/elements.yaml - envsubst < ./kube/main.yaml > /tmp/elements.yaml
- yq '.spec.generators[].list.elements += load("/tmp/elements.yaml")' /tmp/clean_appset.yaml > /tmp/new_appset.yaml - yq -i '.spec.generators[].list.elements += load("/tmp/elements.yaml")' /tmp/appset.yaml
- kubectl apply -f /tmp/new_appset.yaml - kubectl apply -f /tmp/appset.yaml
- name: Sync application - name: Sync application
image: argoproj/argocd image: argoproj/argocd

2
chart/Chart.yaml
View File

@ -2,5 +2,5 @@ apiVersion: v2
name: badhouseplants-net name: badhouseplants-net
description: A Helm chart for Kubernetes description: A Helm chart for Kubernetes
type: application type: application
version: 0.4.0 version: 0.4.2
appVersion: "1.16.0" appVersion: "1.16.0"

2
chart/templates/deployment.yaml
View File

@ -48,7 +48,7 @@ spec:
- name: {{ .Values.hugo.container.name }} - name: {{ .Values.hugo.container.name }}
env: env:
{{- range $key, $value := .Values.hugo.env}} {{- range $key, $value := .Values.hugo.env}}
- key: {{ $key }} - name: {{ $key }}
value: {{ $value }} value: {{ $value }}
{{- end}} {{- end}}
args: args:

2
chart/values.yaml
View File

@ -42,6 +42,8 @@ hugo:
tag: latest tag: latest
baseURL: https://badhouseplants.net/ baseURL: https://badhouseplants.net/
buildDrafts: false buildDrafts: false
env:
HUGO_PARAMS_GITBRANCH: main
istio: istio:
annotations: {} annotations: {}

121
content/posts/argocd-dynamic-environment-per-branch-part-1/index.md
View File

@ -1,11 +1,11 @@
--- ---
title: "Argocd Dynamic Environment Per Branch: Part 1" title: "Dynamic Environment Per Branch with ArgoCD"
date: 2023-02-25T14:00:00+01:00 date: 2023-02-25T14:00:00+01:00
draft: true draft: false
ShowToc: true ShowToc: true
cover: cover:
image: "cover.png" image: "cover.png"
caption: "Argocd Dynamic Environment Per Branch Part 1" caption: "Dynamic Environment Per Branch with ArgoCD"
relative: false relative: false
responsiveImages: false responsiveImages: false
--- ---
@ -39,12 +39,12 @@ Before I can start deploying them, I have to prepare the application for that. A
1. Container must not contain any static content 1. Container must not contain any static content
2. I can't use only latest tags anymore 2. I can't use only latest tags anymore
3. Helm chart has a lot of stuff that's hardcoded 3. Helm chart has a lot of stuff that's hard-coded
4. CI pipelines must be adjusted 4. CI pipelines must be adjusted
5. Deployment process should be rethought 5. Deployment process should be rethought
### Static Container ### Static Container
Static content doesn't play well with dynamic environments. I'd even say, doesn't play at all. So at least I must stop defining hostname for my blog on the build stage. One container should be able to run anywhere with the same result. So I've decided that instedd of putting the generated static content in the container with `nginx` on the build stage, I need to ship a container with source code to `Kubernetes`, generate static there and put it to a container with `nginx`. So before my deployment looked like that: Static content doesn't play well with dynamic environments. I'd even say, doesn't play at all. So at least I must stop defining hostname for my blog on the build stage. One container should be able to run anywhere with the same result. So I've decided that instead of putting the generated static content in the container with `nginx` on the build stage, I need to ship a container with source code to `Kubernetes`, generate static there and put it to a container with `nginx`. So before my deployment looked like that:
```YAML ```YAML
spec: spec:
@ -139,9 +139,9 @@ And also, I'm mounting the `s3-data` volume to the `hugo` container, so it can g
### Helm chart should be more flexible ### Helm chart should be more flexible
I had to find all the values, that should be different between different environments. And turned out, it's not a lot. I had to find all the values, that should be different between different environments. And turned out, it's not a lot.
1. Istio `VirtualServices` hostnames (Or Ingress hostname, if you don't use Istio) 1. `Istio` `VirtualServices` hostnames (Or Ingress hostname, if you don't use `Istio`)
2. Image tag for the container with the source code 2. Image tag for the container with the source code
3. And a hostname that should be passed to hugo as a base URL 3. And a hostname that should be passed to `hugo` as a base URL
4. Preview environments should display pages that are still `drafts` 4. Preview environments should display pages that are still `drafts`
So all of that I've put to `values.yaml` So all of that I've put to `values.yaml`
@ -157,7 +157,7 @@ istio:
``` ```
### CI pipelines ### CI pipelines
Now I need to push a new image on each commit instead of pushing only once the code made it to the main branch, But I also don't want to have something that doesn't work completely in my registry, because I'm self-hosting and ergo I care about storage. So before building and pushing an image, I need to to test it, Now I need to push a new image on each commit instead of pushing only once the code made it to the main branch, But I also don't want to have something that doesn't work completely in my registry, because I'm self-hosting and ergo I care about storage. So before building and pushing an image, I need to test it,
```YAML ```YAML
# --------------------------------------------------------------- # ---------------------------------------------------------------
@ -304,7 +304,7 @@ Since I'm not using latest anymore, I need to add use a new tag every time a new
And the logic that I would like to have in my setup would be And the logic that I would like to have in my setup would be
- In the git repo there is only application set with the main instance only (production) - In the git repo there is only application set with the main instance only (production)
- After a new image is pushed to registry, I'm getting this application set as `yaml` and appending new generator to it. - After a new image is pushed to registry, I'm getting this application set as `yaml` and appending new generator to it.
- Applying a new `ApplicationSet` and syncing application using the `argo` cli tool - Applying a new `ApplicationSet` and syncing application using the `argo` CLI tool
First, let's set environment variables: First, let's set environment variables:
``` ```
@ -348,7 +348,7 @@ And even though it's very ugly, I already like it. Because it works.
I would like to move the whole pipeline logic out of the `.drone.yml` file. But I will do it later. I would like to move the whole pipeline logic out of the `.drone.yml` file. But I will do it later.
After our application set is deployed, we need to update the application the is created by it. I would like to use the `argocd` cli tool for that. to sync one app we need to use selectors, and I'd like to go with labels. So let's first add labels to our `ApplicationSet` After our application set is deployed, we need to update the application the is created by it. I would like to use the `argocd` CLI tool for that. To sync a specific app, we need to use selectors, and I'd like to go with labels. So let's first add labels to our `ApplicationSet`
```YAML ```YAML
... ...
@ -376,11 +376,14 @@ And now let's create a job like that:
- argocd app wait -l app=badhouseplants -l branch=$DRONE_BRANCH - argocd app wait -l app=badhouseplants -l branch=$DRONE_BRANCH
``` ```
And the last step would be to remove an application when branch is removed. It could be easy with `Gitlab` because there you can use `environments` and `triggers` for removing branch *(as I remember)* But with drone it can be harder. Because drone won't be triggered by a removed branch. So I has to be an additional step for the `main` pipeline. And the last step would be to remove an application when branch is removed. It could be easy with `Gitlab` because there you can use `environments` and `triggers` for removing branch *(as I remember)* But with `drone` it seems to be harder. Because `drone` won't be triggered by a removed branch.
Maybe a pull request trigger could be used for that, but I've found another way, which may not be the best, obviously.
I'm always using squash commits that means that after merging a Pull Request the commit will have the same `SHA`. So when merging to the main branch, I can use the commit hash to remove a generator. I've enabled only `fast-forward` merge to the `main` that that means that after merging a Pull Request the commit will have the same `SHA`. So when merging to the main branch, I can use the commit hash to remove a generator. It also means that if I have one commit deployed to several environments, I will remove more that I want. But I don't think that it will be a problem in my case. If you're not a lonely developer, but a team, you may need to choose something else.
So I've created a file `./kube/main-template.yaml`, that looks like that: So I've added a new element to `preview` generator: `commit_sha: $ARGO_APP_IMAGE_TAG`, and then this command will do the trick: `yq -i "del(.spec.generators[].list.elements[] | select(.name == \"$ARGO_APP_BRANCH\"))" /tmp/appset.yaml`
I've created a file `./kube/main-template.yaml`, that looks like that:
```YAML ```YAML
- name: application - name: application
app: badhouseplants app: badhouseplants
@ -418,7 +421,95 @@ And a job:
``` ```
Then I just need to upgrade `./kube/template.yaml`, so it contains `commit_sha: $ARGO_APP_IMAGE_TAG`.
> Also, I've found out that `ArgoCD` won't remove a namespace if it was created by a `SyncPolicy`, so I've added it to the helm chart, and add a new `value` to provide a name. > Also, I've found out that `ArgoCD` won't remove a namespace if it was created by a `SyncPolicy`, so I've added it to the helm chart, and add a new `value` to provide a name.
### And a little bit more
1. Since my storage capacity is a bit limited, I need to care about it. Hence, I can't store all the images there. And I had to come up with a cleaning up solution. Removing images that are older than `X` days, didn't seem to be an option, because in case I'm not pushing to the registry for quite a time, my production image will be gone, and I won't be able to run the blog fast. So I've decided that I only need to store images with tags that still exists in repo as a `commit sha`. If I have a feature branch with 100 commits, I'll have 100 images, but when I squash it before merging, I will be left with only one. So when it's merged to the `main`, I won't have to store 100 images forever. And I've decided to write a script for that.
A `Perl` script. Why Perl? Because I like it and I wanted not to forget it completely. Also, `bash` seems a little bit too primitive for that, compilable languages (`go`, `rust`) seem to be an overkill, `python` I hate. So why not `Perl`?
The initial plan to create a scheduled job that is getting all commit hashes from git, comparing them to docker tags, and if tag with a non-existent commit is found, it's getting removed. But t problem is that `Gitea` *(At the time of writing, I am using `Gitea` 1.18.3)* package registry doesn't have an API to list all tags for an image (or I'm too dummy to find it). So I've decided to use `Drone API`. Getting all commits and all drone builds, comparing builds to commits and for non-existent `SHAs` remove images from the registry. But the problem is that drone doesn't return all the builds, only recent (and again, maybe I couldn't find how to do it). So the scheduled job may not work, if I'm being very productive. So I've added a new step to the job. After syncing an `Argo Application` I'm running [this script](https://git.badhouseplants.net/badhouseplants/badhouseplants-net/src/branch/main/scripts/cleanup.pl):
{{< details "In case you want to read it here:" >}}
```Perl
#!/usr/bin/perl
use strict;
use warnings;
# --------------------------------------
# -- Drone variables
# --------------------------------------
my $drone_url="$ENV{'DRONE_SYSTEM_PROTO'}://$ENV{'DRONE_SYSTEM_HOST'}";
my $drone_project=$ENV{'DRONE_REPO'};
my $drone_api="$drone_url/api/repos/$drone_project/builds";
# --------------------------------------
# -- Gitea variables
# --------------------------------------
my $gitea_url=$ENV{'GITEA_URL'} || 'https://git.badhouseplants.net/api/v1';
my $gitea_org=$ENV{'GITEA_ORG'} || 'badhouseplants';
my $gitea_package=$ENV{'GITEA_PACKAGE'} || 'badhouseplants-net';
my $gitea_api="$gitea_url/packages/$gitea_org/container/$gitea_package";
my $gitea_token=$ENV{'GITEA_TOKEN'};
my $gitea_user=$ENV{'GITEA_USER'} || $ENV{'DRONE_COMMIT_AUTHOR'};
# ---------------------------------------
# -- Get recent builds from drone-ci
# ---------------------------------------
my $builds = "curl -X 'GET' $drone_api -H 'accept: application/json' | jq -r '.[].after'";
my @builds_out = `$builds`;
chomp @builds_out;
# ---------------------------------------
# -- Get a list of all commits + 'latest'
# ---------------------------------------
my $commits = "git log --format=format:%H --all";
my @commits_out = `$commits`;
chomp @commits_out;
push @commits_out, 'latest';
# ---------------------------------------
# -- Compare builds to commits
# -- And remove obsolete imgages from
# -- registry
# ---------------------------------------
foreach my $line (@builds_out)
{
if ( ! grep( /^$line$/, @commits_out ) ) {
my $cmd = "curl -X 'DELETE' -s \"$gitea_api/$line\" -H 'accept: application/json' -u $gitea_user:$gitea_token || true";
print "Removing ${line}\n\n";
my $output = `$cmd`;
print "$output \n";
}
}
```
{{< /details >}}
It's far from being perfect, but it works and I like that I was able to finally use `Perl` somewhere
2. I want to have a manifest that I can apply in case of kind of *disaster recovery*. And it means that `ApplicationSet` should contain enough information to deploy a production instance of my blog right off the bat. But I don't want to keep it up-to-date with every new commit hash. So I've decided to keep pushing `latest` to registry but only on `main` builds. So I can use the latest tag in application set, but in the application life-time I'll keep using `SHA` as tags. The only static hard-coded value in the `ApplicationSet` is a version of the `Helm chart`. And I don't know how to automate it yet. But I'm sure that I will do it somehow.
I know that it's a very common practice to store all `Argo` resource in `git`. But I don't see any sense in storing manifests for temporary environments that can be recreated by clicking a button in `Drone` or by pushing a new commit.
3. Some more static data that I've found later. I've understood that I'm using a badge on the [About page]({{< ref "about" >}}). And it's statically points only to the main branch, that doesn't make a lot of sense on envs built from other branches. But fortunately, *bit ups to devs*, `hugo` can use environment variables for setting up parameters of a site. I've updated the badge, so it looks like that:
```
[![Build Status](https://drone.badhouseplants.net/api/badges/badhouseplants/badhouseplants-net/status.svg?ref=refs/heads/{{< param GitBranch >}})](https://drone.badhouseplants.net/badhouseplants/badhouseplants-net)
```
[![Build Status](https://drone.badhouseplants.net/api/badges/badhouseplants/badhouseplants-net/status.svg?ref=refs/heads/{{< param GitBranch >}})](https://drone.badhouseplants.net/badhouseplants/badhouseplants-net)
And then I'm setting an env var `HUGO_PARAMS_GITBRANCH`. And now badge is looking for its branch.
### What's not done yet
1. I'm using `Minio` as a storage for pictures, and currently all pictures (and other files) are stored in one folder regardless of the environment. I would to have something like that.
- On the first commit to a branch, sync pictures from the main dir to a new one.
- On next commits, if pictures are added, copy them to a new dir only
- When branch is merged, pictures from the branch should be synced to the main dir.
2. Since I don't really have a static content, I can't be 100% sure that content that is generated during the run-time is what I expect to have. So I'd like to add a UI test that is executed after pod with `nginx` is started and is being used as a `startupProbe`. If test is not satisfied by a content, pod is never getting `ready` and traffic will keep going to the older version.
3. A lot of logic that is put to `.drone.yaml` file should be moved out of it. Maybe to scripts, or to `Makefile`. But I don't think it's an important thing for this post, so I've decided not to care about it now.
## Some kind of conclusion
Even though my application is just a simple blog, I still believe that creating dynamic environments is a great idea that should totally replace static dev'n'stages. And it's not only my blog, I've created dynamic envs for. Two biggest pains *as I think* are `Static content` and `Persistent data` (I think, there are more, but these two are most obvious). I've already shown an example how you can handle the first one, and the second is also a big pain in the ass. In my case this data is the one coming from the `Minio` and I'm not doing anything about it, *but I'll write one more post, when it's solved*, other, in my opinion, more obvious example, are databases. You need it to contain all the data that's required for testing, but you also may want it not to be huge, and it most probably should not contain any sensible personal data. So maybe you could stream a database from the production through some kind of anonymizer, clean it up, so it's not too big. And it doesn't sound easy already. But if I'll have to add something like that to my blog once, I'll try to describe it.
Thanks,
Oi!

2
kube/template.yaml
View File

@ -18,3 +18,5 @@
tag: $ARGO_APP_IMAGE_TAG tag: $ARGO_APP_IMAGE_TAG
baseURL: https://$ARGO_APP_HOSTNAME/ baseURL: https://$ARGO_APP_HOSTNAME/
buildDrafts: true buildDrafts: true
env:
HUGO_PARAMS_GITBRANCH: $ARGO_APP_BRANCH