Basics of GitOps With ArgoCD
How do I know if all my desired servers are running and configured correctly?
Do you ever wonder if the servers running on your hardware are configured the way you intend, or if you can easily let others know what is (or should be) running without giving them direct access to your production environment? These are just two of the problems that gitops helps solve.
GitOps
GitOps is a framework that is used to manage the deployment and configuration of infrastructure and servers. This is a declarative framework. As such, the user does not specify the steps that should be taken, but rather they provide a description of the desired state. As suggested by the name, this desired state is stored in a git repository, however any source code version control system would work.
The GitOps framework, consist of three primary components:
- The git repository
- The target target infrastructure
- The reconciliation service
git repository
As stated earlier, the git repository is used to store the desired state of your deployment environment. In most situations GitOps will be used to manage multiple environment. Thus some thought must put into how you separate your definitions so as to make sure the correct configurations are applied to the correct environment. Some options for managing this complexity include:
- Using a git repository for each environment. This increases the mental complexity for the end user since they must remember which repository is associated with which environment. Also promotions from lower to higher environments requires copying/moving definitions from one repository to another. A process that isn’t as simple as creating a pull request. However, this approach allows you to more easily limit who can access configurations files associated with a given environment.
- Using a different branch for each environment. Mentally, this approach is a little less complex since users only need to remember a single git repository. In this situation, the promotion process can be accomplished by merging commits to the appropriate branch (usually via a pull request). One limitation of this approach is that if you need to make changes to multiple environments at once you will need to create a pull request per environment. Additionally, viewing configuration differences for a given resource in different environment becomes rather difficult.
- Using deploying to all environments from the main branch, but using a folder naming structure to specify which environment a configuration is to be associated with. This is the most common approach that is taken by many GitOps practitioners. It avoids the downsides of the other methods mentioned before. However, unlike the “one git repository per environment” approach, you aren’t able to restrict access (this is also a limitation of the branch based approach).
It is worth keeping in mind that the information stored in the git repository is only the desired state. If you want know what is actually running, you will need to examine the resource that has been deployed.
reconciliation server
The job of the reconciliation server is to ensure that desired state that lives in the git repository is what is actually deployed. While not a requirement, this service is usually run within a Kubernetes cluster. The reason for this is that Kubernetes already has its own built in reconciliation loop. Thus many, services will just make use of this feature and add their own logic for how to handle configuration drift. It is important to note that it is not sufficient for the reconciliation process to only happen upon resource creation. But must continually run, and act whenever the created resource has drifted away from its desired state. Such drifts can be caused by human interaction with a resource, like adding a policy to an IAM role, or changing the image that is used in a Kubernetes deployment.
However, tools like OpenTofu (or Terraform) do not perform this continual resource monitoring and
reconciliation process. At least not out of the box. One approach that can be taken to work around
this is to run tofu apply (or terraform apply) on a schedule with a relatively short frequency (~5min).
ArgoCD
Two of the most popular reconciliation servers are ArgoCD and Flux. As I don’t have any experience with Flux I will focus exclusively on ArgoCD.
ArgoCD uses a CRD hierarchy to help you manage your deployed applications. At the bottom of this hierarchy
are your basic Kubernetes resources (any resource that isn’t part of the argoproj.io group). These basic
resources are then grouped into what ArgoCD calls an
Application.
ArgoCD preforms its reconciliations at the Application level, which in turn causes the syncing (reconciliation)
os the composing basic resources. And finally at the top of this structure is the
AppProject.
An AppProject is a logical grouping of similar objects. In a AppProject we can restrict which
environment(s) a member Application can be deployed. Or even what basic Kubernetes resources are
allowed to (or blocked from) be part of the Application.
In addition to using the recommended GitOps workflow, where ArgoCD watches a git repository for changes to the desired state, you can also interact with ArgoCD through its CLI and GUI. As far as I can tell, anything that can be accomplished with the GitOps workflow can be accomplished with the CLI and GUI. As such it is important to set up access control rules to limit what actions different users have when interacting with the CLI or GUI, as these actions won’t have any record of being performed. Unlike when changes happen in your git repository. The GUI, is however very useful for allowing others to view the status of their deployed services without having direct access to your Kubernetes cluster(s).
installation
The installation process for ArgoCD is relatively straight forward, and can be accomplished with either Kustomize or Helm, see the documentation. You can also have ArgoCD manage itself within a GitOps workflow. When doing this I would advise that you pay attention to the following resources;
deployment options
A typical question that is asked is whether you should have a single ArgoCD instance that manages deployments to multiple environments, or one ArgoCD instance per environment. As with most things in life, the answer is that it depends on what your needs are. Having a single instance reduces complexity of the mental map. But can run into performance issues if you have a large number of applications and/or environments to manage. While an instance per environment reduces the amount of work that each instance much do during its sync phase. And this isolation provides a little more security. However, it makes it harder to monitor all instances of a single application that you may have deployed across multiple environments.
Workflow
When using a GitOps workflow, developers don’t have to deviate from working patterns that they are used to. When a new version of a server or other infrastructure component is available and needs to be used by one of your environments, the developer just creates a pull request with the new/updated configuration information. Then once the pull request is approved and merged it will be the responsibility of the reconciliation server to detect the change in desired state and update the current state to match this new desired endpoint.
It is usually best practice to separate your repositories that are used for application code and those that are use for infrastructure. Thus when starting out if a developer makes and update to an application they will need to raise and eventually merge two pull request. The first for the change to the application to itself, and the second for the change to the infrastructure. However, once your organization becomes more comfortable with GitOps and your selected solution, this second PR can be automated as part of your CD process. And then your developers only need to think about your application(s) and providing the best experience to your users/customers. And then you can take this a step further and have your CI process make use of GitOps so that you can bring up a preview version of your application code changes so that reviewers can get a hands on experience to better validate that it works and meets all required change. Then once your application pull request is merged the preview version is deleted thus freeing up resources and any possible associated spend.