In two previous posts I've described how it's possible to install a couple of applications with [`Applications`]({{< ref "argocd-vs-helmfile-application" >}}) and [`ApplicationSets`]({{< ref "argocd-vs-helmfile-applicationset" >}}), and this one is the last in a row. And here I'm going to install the same applications (`VPA` and `Goldilocks`) with helmfile, and I will tell why I think that it's better than `ArgoCD`
So let's start. Here you can find the [initial config](https://git.badhouseplants.net/allanger/helmfile-vs-argo/src/branch/helmfile-main). Let's see what we got here:
The main file is `/helmfile.yaml`
```YAML
---
{{ readFile "releases.yaml" }}
bases:
- environments.yaml
- repositories.yaml
releases:
helmfiles:
- path: {{.Environment.Name }}/helmfile.yaml
```
You can see several imports here, let's check them one by one:
```YAML
# releases.yaml <- here we will define all the charts that are going to be used by helmfile. It's a templating layer, so we don't have to copy-paste a lot of yaml stuff
---
templates:
# It's supposed to be empty when nothing is installed, but I've decided to show that helmfile also can be used to manage CRDs. With this hooks, you'll be able to install CRDs with helmfile, see diffs when updating and update them while updating releases. But I need to say that I haven't checked how it's wotking on big systems, so consider this thing experimental
It's going to import helmfiles that are not used across all clusters. So if you want to install something to `cluster-2` across, you will add it to the `/cluster2/helmfile.yaml` and sync the main helmfile. I will show an example later.
hook[prepare] logs | + description: VerticalPodAutoscalerCheckpoint is the checkpoint of the internal
hook[prepare] logs | + state of VPA that is used for recovery after recommender's restart.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: 'APIVersion defines the versioned schema of this representation
hook[prepare] logs | + of an object. Servers should convert recognized schemas to the latest
hook[prepare] logs | + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind is a string value representing the REST resource this
hook[prepare] logs | + object represents. Servers may infer this from the endpoint the client
hook[prepare] logs | + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
hook[prepare] logs | + type: string
hook[prepare] logs | + metadata:
hook[prepare] logs | + type: object
hook[prepare] logs | + spec:
hook[prepare] logs | + description: 'Specification of the checkpoint. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.'
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the checkpointed container.
hook[prepare] logs | + type: string
hook[prepare] logs | + vpaObjectName:
hook[prepare] logs | + description: Name of the VPA object that stored VerticalPodAutoscalerCheckpoint
hook[prepare] logs | + object.
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + status:
hook[prepare] logs | + description: Data of the checkpoint.
hook[prepare] logs | + properties:
hook[prepare] logs | + cpuHistogram:
hook[prepare] logs | + description: Checkpoint of histogram for consumption of CPU.
hook[prepare] logs | + properties:
hook[prepare] logs | + bucketWeights:
hook[prepare] logs | + description: Map from bucket index to bucket weight.
hook[prepare] logs | + description: Reference timestamp for samples collected within
hook[prepare] logs | + this histogram.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + totalWeight:
hook[prepare] logs | + description: Sum of samples to be used as denominator for weights
hook[prepare] logs | + from BucketWeights.
hook[prepare] logs | + type: number
hook[prepare] logs | + type: object
hook[prepare] logs | + totalSamplesCount:
hook[prepare] logs | + description: Total number of samples in the histograms.
hook[prepare] logs | + type: integer
hook[prepare] logs | + version:
hook[prepare] logs | + description: Version of the format of the stored data.
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + type: object
hook[prepare] logs | + served: true
hook[prepare] logs | + storage: true
hook[prepare] logs | + - name: v1beta2
hook[prepare] logs | + schema:
hook[prepare] logs | + openAPIV3Schema:
hook[prepare] logs | + description: VerticalPodAutoscalerCheckpoint is the checkpoint of the internal
hook[prepare] logs | + state of VPA that is used for recovery after recommender's restart.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: 'APIVersion defines the versioned schema of this representation
hook[prepare] logs | + of an object. Servers should convert recognized schemas to the latest
hook[prepare] logs | + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind is a string value representing the REST resource this
hook[prepare] logs | + object represents. Servers may infer this from the endpoint the client
hook[prepare] logs | + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
hook[prepare] logs | + type: string
hook[prepare] logs | + metadata:
hook[prepare] logs | + type: object
hook[prepare] logs | + spec:
hook[prepare] logs | + description: 'Specification of the checkpoint. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.'
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the checkpointed container.
hook[prepare] logs | + type: string
hook[prepare] logs | + vpaObjectName:
hook[prepare] logs | + description: Name of the VPA object that stored VerticalPodAutoscalerCheckpoint
hook[prepare] logs | + object.
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + status:
hook[prepare] logs | + description: Data of the checkpoint.
hook[prepare] logs | + properties:
hook[prepare] logs | + cpuHistogram:
hook[prepare] logs | + description: Checkpoint of histogram for consumption of CPU.
hook[prepare] logs | + properties:
hook[prepare] logs | + bucketWeights:
hook[prepare] logs | + description: Map from bucket index to bucket weight.
hook[prepare] logs | + description: VerticalPodAutoscaler is the configuration for a vertical pod
hook[prepare] logs | + autoscaler, which automatically manages pod resources based on historical
hook[prepare] logs | + and real time resource utilization.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: 'APIVersion defines the versioned schema of this representation
hook[prepare] logs | + of an object. Servers should convert recognized schemas to the latest
hook[prepare] logs | + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind is a string value representing the REST resource this
hook[prepare] logs | + object represents. Servers may infer this from the endpoint the client
hook[prepare] logs | + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
hook[prepare] logs | + type: string
hook[prepare] logs | + metadata:
hook[prepare] logs | + type: object
hook[prepare] logs | + spec:
hook[prepare] logs | + description: 'Specification of the behavior of the autoscaler. More info:
hook[prepare] logs | + description: Specifies the minimal amount of resources that
hook[prepare] logs | + will be recommended for the container. The default is
hook[prepare] logs | + no minimum.
hook[prepare] logs | + type: object
hook[prepare] logs | + mode:
hook[prepare] logs | + description: Whether autoscaler is enabled for the container.
hook[prepare] logs | + The default is "Auto".
hook[prepare] logs | + enum:
hook[prepare] logs | + - Auto
hook[prepare] logs | + - "Off"
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + type: object
hook[prepare] logs | + targetRef:
hook[prepare] logs | + description: TargetRef points to the controller managing the set of
hook[prepare] logs | + pods for the autoscaler to control - e.g. Deployment, StatefulSet.
hook[prepare] logs | + VerticalPodAutoscaler can be targeted at controller implementing
hook[prepare] logs | + scale subresource (the pod set is retrieved from the controller's
hook[prepare] logs | + ScaleStatus) or some well known controllers (e.g. for DaemonSet
hook[prepare] logs | + the pod set is read from the controller's spec). If VerticalPodAutoscaler
hook[prepare] logs | + cannot use specified target it will report ConfigUnsupported condition.
hook[prepare] logs | + Note that VerticalPodAutoscaler does not require full implementation
hook[prepare] logs | + of scale subresource - it will not use it to modify the replica
hook[prepare] logs | + count. The only thing retrieved is a label selector matching pods
hook[prepare] logs | + grouped by the target resource.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: API version of the referent
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds"'
hook[prepare] logs | + type: string
hook[prepare] logs | + name:
hook[prepare] logs | + description: 'Name of the referent; More info: http://kubernetes.io/docs/user-guide/identifiers#names'
hook[prepare] logs | + type: string
hook[prepare] logs | + required:
hook[prepare] logs | + - kind
hook[prepare] logs | + - name
hook[prepare] logs | + type: object
hook[prepare] logs | + updatePolicy:
hook[prepare] logs | + description: Describes the rules on how changes are applied to the
hook[prepare] logs | + pods. If not specified, all fields in the `PodUpdatePolicy` are
hook[prepare] logs | + set to their default values.
hook[prepare] logs | + properties:
hook[prepare] logs | + minReplicas:
hook[prepare] logs | + description: Minimal number of replicas which need to be alive
hook[prepare] logs | + for Updater to attempt pod eviction (pending other checks like
hook[prepare] logs | + PDB). Only positive values are allowed. Overrides global '--min-replicas'
hook[prepare] logs | + flag.
hook[prepare] logs | + format: int32
hook[prepare] logs | + type: integer
hook[prepare] logs | + updateMode:
hook[prepare] logs | + description: Controls when autoscaler applies changes to the pod
hook[prepare] logs | + resources. The default is 'Auto'.
hook[prepare] logs | + enum:
hook[prepare] logs | + - "Off"
hook[prepare] logs | + - Initial
hook[prepare] logs | + - Recreate
hook[prepare] logs | + - Auto
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - targetRef
hook[prepare] logs | + type: object
hook[prepare] logs | + status:
hook[prepare] logs | + description: Current information about the autoscaler.
hook[prepare] logs | + properties:
hook[prepare] logs | + conditions:
hook[prepare] logs | + description: Conditions is the set of conditions required for this
hook[prepare] logs | + autoscaler to scale its target, and indicates whether or not those
hook[prepare] logs | + conditions are met.
hook[prepare] logs | + items:
hook[prepare] logs | + description: VerticalPodAutoscalerCondition describes the state
hook[prepare] logs | + of a VerticalPodAutoscaler at a certain point.
hook[prepare] logs | + properties:
hook[prepare] logs | + lastTransitionTime:
hook[prepare] logs | + description: lastTransitionTime is the last time the condition
hook[prepare] logs | + transitioned from one status to another
hook[prepare] logs | + format: date-time
hook[prepare] logs | + type: string
hook[prepare] logs | + message:
hook[prepare] logs | + description: message is a human-readable explanation containing
hook[prepare] logs | + details about the transition
hook[prepare] logs | + type: string
hook[prepare] logs | + reason:
hook[prepare] logs | + description: reason is the reason for the condition's last transition.
hook[prepare] logs | + type: string
hook[prepare] logs | + status:
hook[prepare] logs | + description: status is the status of the condition (True, False,
hook[prepare] logs | + Unknown)
hook[prepare] logs | + type: string
hook[prepare] logs | + type:
hook[prepare] logs | + description: type describes the current condition
hook[prepare] logs | + type: string
hook[prepare] logs | + required:
hook[prepare] logs | + - status
hook[prepare] logs | + - type
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + recommendation:
hook[prepare] logs | + description: The most recently computed amount of resources recommended
hook[prepare] logs | + by the autoscaler for the controlled pods.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerRecommendations:
hook[prepare] logs | + description: Resources recommended by the autoscaler for each
hook[prepare] logs | + container.
hook[prepare] logs | + items:
hook[prepare] logs | + description: RecommendedContainerResources is the recommendation
hook[prepare] logs | + of resources computed by autoscaler for a specific container.
hook[prepare] logs | + Respects the container resource policy if present in the spec.
hook[prepare] logs | + In particular the recommendation is not produced for containers
hook[prepare] logs | + with `ContainerScalingMode` set to 'Off'.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the container.
hook[prepare] logs | + description: Maximum recommended amount of resources. Observes
hook[prepare] logs | + ContainerResourcePolicy. Any resources allocated beyond
hook[prepare] logs | + this value are likely wasted. This value may be larger
hook[prepare] logs | + than the maximum amount of application is actually capable
hook[prepare] logs | + of consuming.
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - target
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + type: object
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - spec
hook[prepare] logs | + type: object
hook[prepare] logs | + served: true
hook[prepare] logs | + storage: true
hook[prepare] logs | + subresources: {}
hook[prepare] logs | + - name: v1beta2
hook[prepare] logs | + schema:
hook[prepare] logs | + openAPIV3Schema:
hook[prepare] logs | + description: VerticalPodAutoscaler is the configuration for a vertical pod
hook[prepare] logs | + autoscaler, which automatically manages pod resources based on historical
hook[prepare] logs | + and real time resource utilization.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: 'APIVersion defines the versioned schema of this representation
hook[prepare] logs | + of an object. Servers should convert recognized schemas to the latest
hook[prepare] logs | + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind is a string value representing the REST resource this
hook[prepare] logs | + object represents. Servers may infer this from the endpoint the client
hook[prepare] logs | + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
hook[prepare] logs | + type: string
hook[prepare] logs | + metadata:
hook[prepare] logs | + type: object
hook[prepare] logs | + spec:
hook[prepare] logs | + description: 'Specification of the behavior of the autoscaler. More info:
hook[prepare] logs | + description: Specifies the minimal amount of resources that
hook[prepare] logs | + will be recommended for the container. The default is
hook[prepare] logs | + no minimum.
hook[prepare] logs | + type: object
hook[prepare] logs | + mode:
hook[prepare] logs | + description: Whether autoscaler is enabled for the container.
hook[prepare] logs | + The default is "Auto".
hook[prepare] logs | + enum:
hook[prepare] logs | + - Auto
hook[prepare] logs | + - "Off"
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + type: object
hook[prepare] logs | + targetRef:
hook[prepare] logs | + description: TargetRef points to the controller managing the set of
hook[prepare] logs | + pods for the autoscaler to control - e.g. Deployment, StatefulSet.
hook[prepare] logs | + VerticalPodAutoscaler can be targeted at controller implementing
hook[prepare] logs | + scale subresource (the pod set is retrieved from the controller's
hook[prepare] logs | + ScaleStatus) or some well known controllers (e.g. for DaemonSet
hook[prepare] logs | + the pod set is read from the controller's spec). If VerticalPodAutoscaler
hook[prepare] logs | + cannot use specified target it will report ConfigUnsupported condition.
hook[prepare] logs | + Note that VerticalPodAutoscaler does not require full implementation
hook[prepare] logs | + of scale subresource - it will not use it to modify the replica
hook[prepare] logs | + count. The only thing retrieved is a label selector matching pods
hook[prepare] logs | + grouped by the target resource.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: API version of the referent
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds"'
hook[prepare] logs | + type: string
hook[prepare] logs | + name:
hook[prepare] logs | + description: 'Name of the referent; More info: http://kubernetes.io/docs/user-guide/identifiers#names'
hook[prepare] logs | + type: string
hook[prepare] logs | + required:
hook[prepare] logs | + - kind
hook[prepare] logs | + - name
hook[prepare] logs | + type: object
hook[prepare] logs | + updatePolicy:
hook[prepare] logs | + description: Describes the rules on how changes are applied to the
hook[prepare] logs | + pods. If not specified, all fields in the `PodUpdatePolicy` are
hook[prepare] logs | + set to their default values.
hook[prepare] logs | + properties:
hook[prepare] logs | + updateMode:
hook[prepare] logs | + description: Controls when autoscaler applies changes to the pod
hook[prepare] logs | + resources. The default is 'Auto'.
hook[prepare] logs | + enum:
hook[prepare] logs | + - "Off"
hook[prepare] logs | + - Initial
hook[prepare] logs | + - Recreate
hook[prepare] logs | + - Auto
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - targetRef
hook[prepare] logs | + type: object
hook[prepare] logs | + status:
hook[prepare] logs | + description: Current information about the autoscaler.
hook[prepare] logs | + properties:
hook[prepare] logs | + conditions:
hook[prepare] logs | + description: Conditions is the set of conditions required for this
hook[prepare] logs | + autoscaler to scale its target, and indicates whether or not those
hook[prepare] logs | + conditions are met.
hook[prepare] logs | + items:
hook[prepare] logs | + description: VerticalPodAutoscalerCondition describes the state
hook[prepare] logs | + of a VerticalPodAutoscaler at a certain point.
hook[prepare] logs | + properties:
hook[prepare] logs | + lastTransitionTime:
hook[prepare] logs | + description: lastTransitionTime is the last time the condition
hook[prepare] logs | + transitioned from one status to another
hook[prepare] logs | + format: date-time
hook[prepare] logs | + type: string
hook[prepare] logs | + message:
hook[prepare] logs | + description: message is a human-readable explanation containing
hook[prepare] logs | + details about the transition
hook[prepare] logs | + type: string
hook[prepare] logs | + reason:
hook[prepare] logs | + description: reason is the reason for the condition's last transition.
hook[prepare] logs | + type: string
hook[prepare] logs | + status:
hook[prepare] logs | + description: status is the status of the condition (True, False,
hook[prepare] logs | + Unknown)
hook[prepare] logs | + type: string
hook[prepare] logs | + type:
hook[prepare] logs | + description: type describes the current condition
hook[prepare] logs | + type: string
hook[prepare] logs | + required:
hook[prepare] logs | + - status
hook[prepare] logs | + - type
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + recommendation:
hook[prepare] logs | + description: The most recently computed amount of resources recommended
hook[prepare] logs | + by the autoscaler for the controlled pods.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerRecommendations:
hook[prepare] logs | + description: Resources recommended by the autoscaler for each
hook[prepare] logs | + container.
hook[prepare] logs | + items:
hook[prepare] logs | + description: RecommendedContainerResources is the recommendation
hook[prepare] logs | + of resources computed by autoscaler for a specific container.
hook[prepare] logs | + Respects the container resource policy if present in the spec.
hook[prepare] logs | + In particular the recommendation is not produced for containers
hook[prepare] logs | + with `ContainerScalingMode` set to 'Off'.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the container.
Yeah, it's huge, but you can see everything that's going to happen. So I'd say it's good.
## Conclusion
It's a short article, because I think the whole setup is super easy, CI is easy too. You still have a full `GitOps` (or almost full) but you also have control. I love this setup and would like to use it for my infrastructure.
With `ArgoCD` I either have a lot of `yaml` to install things, or I have complicated setups with `ApplicationSets` that are most probably very special and won't be reused in other companies. I need to care about how `ArgoCD` will handle a lot of applications that are added there only for diffing. I need additional applications installed in my clusters not only as a part of infrastructure itself, but also as a service that I'm providing other teams with. Because I want to manage applications that are being developed by other teams with `Argo`, so I'm mixing a lot of different kinds of applications here.
Helmfile lets me separate infra from applications. `ArgoCD` can be only provided as a service, and other teams can use, because it's making k8s easier for those who don't need to understand it so deeply. Also, helmfile lets me use helm-secrets to encrypt values. I can do it with Argo too, but then I need to either have a custom `ArgoCD` image, or support a CMP plugin, that will handle SOPS.
You can find an example of PR here: <https://git.badhouseplants.net/allanger/helmfile-vs-argo/pulls/1/files>
> When helmfile is not GitOps?
> To uninstall a helm release, you need to add `isntalled: false` to it. If you just remove a release from helmfile.yaml, it isn't going to be removed. So in such cases it's not GitOps. You can write a hook, that is comparing a previous state of your helmfile to the current one and doing a cleanup, then it's again fully GitOps. But I prefer removing things manually, so to me, it's not a problem. Removing stuff is something that I think should be mostly done by a human being, if it's not a part of your daily work.
