- Published on
CICD With Tekton and ArgoCD
- Authors
- Name
- Sunway
- 1 Prerequisite
- 1.1 Install Tekton
- 1.2 Intall ArgCD
- 1.3 Prepare a git repo with helm or kustomize
- 1.4 Secret Configuration
- 2 Build Pipeline
- 2.0 Create SA
- 2.1 Task: Git Clone
- 2.2 Task: Docker Build And Push
- 2.3 Task: Update Image Tag In Helm Chart
- 2.4 Create A Pipeline
- 2.5 Create TriggerTemplate
- 2.6 Create TriggerBinding
- 2.7 Create EventListener
- 3 Create Ingress
- 3.1 Create EventListener Ingress
- 3.2 Create Tekton Dashboard Ingress
- 3.2 Create ArgoCD Dashboard Ingress
- 4 ArgoCD Configuration
- 4.1 Add Others Kubernetes Clusters
- 4.2 Create A Repositories
- 4.3 Create An Application as below
- 5 Github Webhook
- 6 Testing
In this blog, you can implement a ci/cd demo with tekton
adn argocd
- Flow 1: User push or pull_request to a private GitHub repo
- Flow 2: GitHub Repo Webhook send a request to tekton eventListener
- Flow 3: EventListener determines whether it needs to create a pipelineRun based on the triggerTemplate according to the conditions define in filter.
- Flow 4: EventListener create a pipelineRun which contains 3 task: git clone, docker build and push, update helm chart image tag
- Flow 5: Argocd auto sync helm files in GitHub repo after task3(update image tag) execute successfully
- Flow 6: Argocd use the latest image to build a new pod and then delete the old one
1 Prerequisite
1.1 Install Tekton
# https://tekton.dev/docs/installation/pipelines/
# kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml
# Tekton Client RPM install
rpm -Uvh https://github.com/tektoncd/cli/releases/download/v0.38.0/tektoncd-cli-0.38.0_Linux-64bit.rpm
# Tekton Client DEB install
wget https://github.com/tektoncd/cli/releases/download/v0.38.0/tektoncd-cli-0.38.0_Linux-64bit.deb
dpkg -i tektoncd-cli-0.38.0_Linux-64bit.deb && rm -f tektoncd-cli-0.38.0_Linux-64bit.deb
1.2 Intall ArgCD
# https://argo-cd.readthedocs.io/en/stable/getting_started/
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Install ArgoCD CLI
VERSION=$(curl -L -s https://raw.githubusercontent.com/argoproj/argo-cd/stable/VERSION)
wget https://github.com/argoproj/argo-cd/releases/download/v$VERSION/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/bin/argocd
rm argocd-linux-amd64
1.3 Prepare a git repo with helm or kustomize
git clone https://github.com/sunway910/cicd-demo
1.4 Secret Configuration
Create Docker Credential Secret
# If your repo is private, please generate the secret as below
# Tekton task kaniko use this secret to push image to a docker repo
kubectl create secret docker-registry docker-cred \
--docker-server=https://index.docker.io/v1/ \
--docker-username="sunway" \
--docker-password="sunway.run" \
--docker-email="[email protected]"
Create GitHub Credential Secret
# Tekton use this secret to clone and push to a private git repo
# Please delete all configuration like github-cred in this blog if git repo is a public repo
kubectl create secret generic github-cred \
--from-file=id_rsa=/root/.ssh/id_rsa \
--from-file=known_hosts=/root/.ssh/known_hosts
Set argocd server insecure if you want
# set argocd tls: https://argo-cd.readthedocs.io/en/stable/operator-manual/tls/
# set insecure mode if you want
kubectl edit cm -n argocd argocd-cmd-params-cm
# add data in comfigmap as below
data:
server.insecure: "true"
kubectl rollout restart deployment argocd-server -n argocd
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
Create non-admin user for dev-team
2 Build Pipeline
2.0 Create SA
sunway123
is the secret
you should set in your GitHub Repo Webhook
apiVersion: v1
kind: Secret
metadata:
name: github-triggers-secret
type: Opaque
stringData:
secretToken: "sunway123"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: tekton-triggers-github-sa
secrets:
- name: github-triggers-secret
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tekton-triggers-github-minimal
rules:
# EventListeners need to be able to fetch all namespaced resources
- apiGroups: ["triggers.tekton.dev"]
resources:
["eventlisteners", "triggerbindings", "triggertemplates", "triggers"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
# configmaps is needed for updating logging config
resources: ["configmaps"]
verbs: ["get", "list", "watch"]
# Permissions to create resources in associated TriggerTemplates
- apiGroups: ["tekton.dev"]
resources: ["pipelineruns", "pipelineresources", "taskruns"]
verbs: ["create"]
- apiGroups: [""]
resources: ["serviceaccounts"]
verbs: ["impersonate"]
- apiGroups: ["policy"]
resources: ["podsecuritypolicies"]
resourceNames: ["tekton-triggers"]
verbs: ["use"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tekton-triggers-github-binding
subjects:
- kind: ServiceAccount
name: tekton-triggers-github-sa
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: tekton-triggers-github-minimal
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tekton-triggers-github-clusterrole
rules:
- apiGroups: ["triggers.tekton.dev"]
resources: ["clustertriggerbindings", "clusterinterceptors","interceptors"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tekton-triggers-github-clusterbinding
subjects:
- kind: ServiceAccount
name: tekton-triggers-github-sa
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tekton-triggers-github-clusterrole
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: pipeline-account
---
apiVersion: v1
kind: Secret
metadata:
name: kube-api-secret
annotations:
kubernetes.io/service-account.name: pipeline-account
type: kubernetes.io/service-account-token
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pipeline-role
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pipeline-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pipeline-role
subjects:
- kind: ServiceAccount
name: pipeline-account
Use command kubectl apply -f task-git-clone.yaml
to add task: git-clone
, you can also use command tkn hub install task git-clone
to get task from hub and then customize the yaml file kubectl get task git-clone -o yaml > task-git-clone.yaml
by yourself.
2.1 Task: Git Clone
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: git-clone
spec:
workspaces:
- name: output
description: The git repo will be cloned onto the volume backing this workspace
- name: ssh-directory
params:
- name: repo_url
description: git repo url to clone
type: string
- name: revision
description: git revision to checkout (branch, tag, sha, ref…)
type: string
default: main
- name: submodules
description: defines if the resource should initialize and fetch the submodules
type: string
default: "true"
- name: depth
description: performs a shallow clone where only the most recent commit(s) will be fetched
type: string
default: "1"
- name: sslVerify
description: defines if http.sslVerify should be set to true or false in the global git config
type: string
default: "true"
- name: subdirectory
description: subdirectory inside the "output" workspace to clone the git repo into
type: string
default: ""
- name: deleteExisting
description: clean out the contents of the repo's destination directory (if it already exists) before trying to clone the repo there
type: string
default: "false"
results:
- name: commit
description: The precise commit SHA that was fetched by this Task
steps:
- name: clone
image: gcriotekton/pipeline-git-init:latest
securityContext:
runAsUser: 0 # This needs root, and git-init is nonroot by default
script: |
#!/usr/bin/env sh
set -eu
if [[ -d $(workspaces.ssh-directory.path) ]]; then
mkdir -p ~/.ssh
cp $(workspaces.ssh-directory.path)/known_hosts ~/.ssh/known_hosts
cp $(workspaces.ssh-directory.path)/id_rsa ~/.ssh/id_rsa
chmod 700 ~/.ssh
chmod 600 ~/.ssh/*
fi
CHECKOUT_DIR="$(workspaces.output.path)/$(params.subdirectory)"
cleandir() {
# Delete any existing contents of the repo directory if it exists.
#
# We don't just "rm -rf $CHECKOUT_DIR" because $CHECKOUT_DIR might be "/"
# or the root of a mounted volume.
if [[ -d "$CHECKOUT_DIR" ]] ; then
# Delete non-hidden files and directories
rm -rf "$CHECKOUT_DIR"/*
# Delete files and directories starting with . but excluding ..
rm -rf "$CHECKOUT_DIR"/.[!.]*
# Delete files and directories starting with .. plus any other character
rm -rf "$CHECKOUT_DIR"/..?*
fi
}
if [[ "$(params.deleteExisting)" == "true" ]] ; then
cleandir
fi
/ko-app/git-init \
-url "$(params.repo_url)" \
-revision "$(params.revision)" \
-path "$CHECKOUT_DIR" \
-sslVerify="$(params.sslVerify)" \
-submodules="$(params.submodules)" \
-depth="$(params.depth)"
cd "$CHECKOUT_DIR"
git checkout "$(params.revision)"
RESULT_SHA="$(git rev-parse HEAD | tr -d '\n')"
EXIT_CODE="$?"
if [ "$EXIT_CODE" != 0 ]
then
exit $EXIT_CODE
fi
# Make sure we don't add a trailing newline to the result!
echo -n "$RESULT_SHA" > $(results.commit.path)
2.2 Task: Docker Build And Push
Use command kubectl apply -f task-kaniko.yaml
to add task: kaniko-build-and-push
, you can also use command tkn hub install task kaniko
to get task from hub and then customize the yaml file kubectl get task kaniko -o yaml > task-kaniko.yaml
by yourself.
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: kaniko-build-and-push
spec:
workspaces:
- name: source
description: Holds the context and Dockerfile
- name: dockerconfig
description: Includes a docker `config.json`
optional: true
mountPath: /kaniko/.docker
params:
- name: IMAGE
description: Name (reference) of the image to build.
- name: DOCKERFILE
description: Path to the Dockerfile to build.
default: ./Dockerfile
- name: CONTEXT
description: The build context used by Kaniko.
default: ./
- name: EXTRA_ARGS
type: array
default: [--ignore-path=/product_uuid]
- name: BUILDER_IMAGE
description: The image on which builds will run
default: gcr.io/kaniko-project/executor:latest
results:
- name: IMAGE_DIGEST
description: Digest of the image just built.
- name: IMAGE_URL
description: URL of the image just built.
steps:
- name: build-and-push
workingDir: $(workspaces.source.path)
image: $(params.BUILDER_IMAGE)
args:
- $(params.EXTRA_ARGS)
- --dockerfile=$(params.DOCKERFILE)
- --context=$(workspaces.source.path)/$(params.CONTEXT) # The user does not need to care the workspace and the source.
- --destination=$(params.IMAGE)
- --digest-file=$(results.IMAGE_DIGEST.path)
# kaniko assumes it is running as root, which means this example fails on platforms
# that default to run containers as random uid (like OpenShift). Adding this securityContext
# makes it explicit that it needs to run as root.
securityContext:
runAsUser: 0
- name: write-url
image: docker.io/library/bash:5.1.4@sha256:c523c636b722339f41b6a431b44588ab2f762c5de5ec3bd7964420ff982fb1d9
script: |
set -e
image="$(params.IMAGE)"
echo -n "${image}" | tee "$(results.IMAGE_URL.path)"
2.3 Task: Update Image Tag In Helm Chart
Use command kubectl apply -f task-update-image-tag.yaml
to add task: task-update-image-tag
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: update-image-tag
spec:
workspaces:
- name: source
- name: ssh-directory
params:
- name: commit_id
description: result of git commit
- name: project_path
description: which deployment path to update
steps:
- name: update-image-tag
image: alpine/git
script: |
#!/usr/bin/env sh
set -eu
if [[ -d $(workspaces.ssh-directory.path) ]]; then
mkdir -p ~/.ssh
cp $(workspaces.ssh-directory.path)/known_hosts ~/.ssh/known_hosts
cp $(workspaces.ssh-directory.path)/id_rsa ~/.ssh/id_rsa
chmod 700 ~/.ssh
chmod 600 ~/.ssh/*
fi
# git init only required in storageClass like nfs, do not required storageClass like csi..., cause nfs deny git search its parent working tree
# git init
# '[email protected]:sunway910/cicd-demo.git' can be a param
git clone [email protected]:sunway910/cicd-demo.git
git checkout "$(params.revision)"
# cd https://github.com/sunway910/cicd-demo/cicd-demo-helm
cd "$(project_path)"
# update image tag to the latest one
sed -i "s|tag:.*|tag: $(params.commit_id)|" values.yaml || { echo "Failed to update values.yaml"; exit 1; }
git config --global user.email "[email protected]"
git config --global user.name "Tekton Bot"
git add values.yaml
git commit -m "skip ci: update image tag to $(params.commit_id)" || { echo "Failed to commit changes"; exit 1; }
git push || { echo "Failed to push changes"; exit 1; }
securityContext:
runAsUser: 0
2.4 Create A Pipeline
Use command kubectl apply -f pipeline.yaml
to create pipeline: cicd-demo-pipeline
customized pipeline.yaml
file as below:
- tasks
fetch-from-git
: paramsDockerfile
value is$(workspaces.source.path)/Dockerfile
, modify the path if your Dockerfile is not in the root of your git repo. - tasks
update-image-tag
: paramsproject_path
value iscicd-demo-helm
, modify the path if your helm/kustomize directory path.
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: cicd-demo-pipeline
spec:
params:
# clone from which repo and revision
- name: repo_url
type: string
- name: repo_revision
type: string
# push image to which registry and repository
- name: image-registry
default: docker.io/sunway
- name: image-repo-name
type: string
workspaces:
- name: git-source
- name: docker-cred
- name: github-cred
tasks:
- name: fetch-from-git
taskRef:
name: git-clone
params:
- name: repo_url
value: $(params.repo_url)
- name: revision
value: $(params.repo_revision)
- name: deleteExisting
value: "true"
workspaces:
- name: output
workspace: git-source
- name: ssh-directory
workspace: github-cred
- name: build-image
runAfter: [ fetch-from-git ]
taskRef:
name: kaniko-build-and-push
params:
- name: IMAGE
value: $(params.image-registry)/$(params.image-repo-name):$(tasks.fetch-from-git.results.commit)
- name: CONTEXT
value: ""
- name: DOCKERFILE
value: $(workspaces.source.path)/Dockerfile
workspaces:
- name: source
workspace: git-source
- name: dockerconfig
workspace: docker-cred
- name: update-image-tag
runAfter: [ build-image ]
taskRef:
name: update-image-tag
params:
- name: commit_id
value: $(tasks.fetch-from-git.results.commit)
# cd https://github.com/sunway910/cicd-demo/$project_path
- name: project_path
value: cicd-demo-helm
workspaces:
- name: source
workspace: git-source
- name: ssh-directory
workspace: github-cred
2.5 Create TriggerTemplate
Use command kubectl apply -f triggertemplate.yaml
to create triggertemplate: cicd-demo-github-template
customized triggertemplate.yaml
file as below, task update-image-tag
will push image to docker.io/sunway/cicd-demo:$(params.commit_id)
:
- image-registry:
docker.io/sunway
- image-repo-name:
cicd-demo
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
name: cicd-demo-github-template
spec:
params:
- name: gitrevision
- name: gitrepositoryurl
resourceTemplates:
- apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
generateName: tekton-sunway-cicd-demo-
spec:
pipelineRef:
name: cicd-demo-pipeline
taskRunTemplate:
serviceAccountName: pipeline-account
workspaces:
- name: git-source
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
#storageClassName: csi
- name: docker-cred
secret:
defaultMode: 420
items:
- key: .dockerconfigjson
path: config.json
secretName: docker-cred
- name: github-cred
secret:
defaultMode: 420
secretName: github-cred
params:
- name: repo_url
value: $(tt.params.gitrepositoryurl)
- name: repo_revision
value: $(tt.params.gitrevision)
- name: image-registry
value: docker.io/sunway
- name: image-repo-name
value: cicd-demo
2.6 Create TriggerBinding
Use command kubectl apply -f triggerbinding.yaml
to create triggerbinding: github-push-binding
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
name: github-push-binding
spec:
params:
- name: gitrevision
value: $(body.head_commit.id)
- name: gitrepositoryurl
value: $(body.repository.ssh_url)
- name: gitbranch
value: $(body.ref)
2.7 Create EventListener
Use command kubectl apply -f eventlistener.yaml
to create eventlistener: cicd-demo-github-listener
Trigger Condition:
- 1: repo name is equal to
cicd-demo
- 2: commit message does not contain
skip ci
. - 3: push/pull_request on main/master branch
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
name: cicd-demo-github-listener
spec:
serviceAccountName: tekton-triggers-github-sa
triggers:
- name: cicd-demo-github-listener
interceptors:
- ref:
name: "github"
params:
- name: "secretRef"
value:
secretName: github-triggers-secret
secretKey: secretToken
- name: "eventTypes"
value: [ "push", "pull_request" ]
- ref:
# Common Expression Language
name: "cel"
params:
- name: "filter"
# https://github.com/sunway910/cicd-demo
value: "body.repository.name == 'cicd-demo' && !body.head_commit.message.contains('skip ci') && (body.ref == 'refs/heads/main' || body.ref == 'refs/heads/master')"
bindings:
- ref: github-push-binding
template:
ref: cicd-demo-github-template
3 Create Ingress
3.1 Create EventListener Ingress
GitHub Repo Webhook will send request to this ingress: git-listener-cicd-demo.sunway.run
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: pipeline-ingress
spec:
ingressClassName: nginx
rules:
- host: git-listener-cicd-demo.sunway.run
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: cicd-demo-github-listener
port:
number: 8080
3.2 Create Tekton Dashboard Ingress
Access to https://tekton-dashboard.sunway.run
and watch the tasks progress in pipelineRun after push/pull_request to GitHub repo...
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tekton-dashboard-ingress
namespace: tekton-pipelines
spec:
ingressClassName: nginx
rules:
- host: tekton-dashboard.sunway.run
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: tekton-dashboard
port:
number: 9097
3.2 Create ArgoCD Dashboard Ingress
Access to https://argocd.sunway.run
with user:admin
and password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd-ingress
namespace: argocd
spec:
ingressClassName: nginx
rules:
- host: argocd.sunway.run
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: argocd-server
port:
# set port to 443 if you do not disable tls in argocd-server in 1.4 Secret Configuration
number: 80
4 ArgoCD Configuration
4.1 Add Others Kubernetes Clusters
- Step 1: Login to Argo CD Make sure that you have updated the cluster details to the
kubeconfig
file and logged into Argo CD using Argo CD CLI, if not run the following command to log in to Argo CD.
argocd login https://argocd.sunway.run --username admin --password axDaGIswqYaaHxBs
- Step 2: Get the Context of the Cluster
Once you have logged in to Argo CD, you need to find the context of the cluster you need to add to Argo CD.
Run the following command to get the context from the kubeconfig
file
# execute this on others kubernetes cluster and then get context name
kubectl config get-contexts -o name
- Step 3: Add the Cluster To add the cluster to Argo CD, use the context of the running cluster you got from the previous step on the below command
# If you set proxy like cloudflare before others kubernetes api-server, you have to trust the proxy server certificate. Otherwise, you shouldn't set up a proxy on your api-server
argocd cluster add --kubeconfig <path-of-kubeconfig-file> --kube-context string <cluster-context> --name <cluster-name>
This command will create a service account Argo CD-manager
on the cluster you specify in the above command with full cluster privileges, so make sure you have the required permissions on the cluster.
4.2 Create A Repositories
connect to a private repo via ssh as below
4.3 Create An Application as below
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cicd-demo
spec:
destination:
name: ''
namespace: app
server: https://kubernetes.default.svc
source:
path: cicd-demo-helm
repoURL: https://github.com/sunway910/cicd-demo
targetRevision: main
sources: []
project: default
syncPolicy:
automated:
prune: false
selfHeal: false
5 Github Webhook
Set github webhook to tekton eventListener as below
Payload URL: https://git-listener-cicd-demo.sunway.run/github
Content type: application/json
Secret: sunway123
6 Testing
You can see your application after you create your own application at 4.3
Application indicate processing
cause argocd cant check the ingress
health, it is a normal status if you use nginx ingress
or something else. [ ref1, ref2 ]
After you check application is healthy, you can push code to your GitHub repo and then check pipelineRun
on Tekton Dashboard.
ArgoCD will auto sync the image tag
changes in values.yaml
when pipelineRun done.
Argocd will sync every 180 seconds
automatically, but you can also click sync
manually to sync immediately.