- Published on
CICD With Tekton and ArgoCD
- Authors
- Name
- Sunway
- 1 Prerequisite
- 2 Build Pipeline
- 2.0 Create ServiceAccount
- 2.1 Task: Git Clone
- 2.2 Task: Docker Build And Push
- 2.3 Task: Update Image Tag In Helm Chart
- 2.4 Task: Send Notification (Optional)
- 2.5 Create Pipeline
- 2.6 Schedule Pipeline (Optional)
- 2.7 Create TriggerTemplate
- 2.8 Create TriggerBinding
- 2.9 Create EventListener
- 2.10 Create ArgoCD Notification (Optional)
- 3 Create Ingress
- 4 ArgoCD Configuration
- 5 Github Webhook
- 6 Testing
In this blog, you can implement a ci/cd demo with tekton
adn argocd
- Flow 1: User do a push or pull_request operation to a private GitHub repo
- Flow 2: GitHub Repo Webhook send a request to tekton eventListener
- Flow 3: EventListener determines whether it needs to create a pipelineRun based on the triggerTemplate according to the conditions define in filter.
- Flow 4: EventListener create a pipelineRun which contains 4 task:
git clone
,docker build and push
,update helm chart image tag
,send notification
- Flow 5: Argocd auto-sync helm files in GitHub repo after
task3(update helm chart image tag)
run successfully - Flow 6: Argocd use the latest image to build a new pod and then delete the old one
1 Prerequisite
1.1 Install Tekton
# https://tekton.dev/docs/installation/pipelines/
# kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml
# Tekton Client RPM(Redhat/CentOS...) install
rpm -Uvh https://github.com/tektoncd/cli/releases/download/v0.38.0/tektoncd-cli-0.38.0_Linux-64bit.rpm
# Tekton Client DEB(Ubuntu/Debian...) install
wget https://github.com/tektoncd/cli/releases/download/v0.38.0/tektoncd-cli-0.38.0_Linux-64bit.deb
dpkg -i tektoncd-cli-0.38.0_Linux-64bit.deb && rm -f tektoncd-cli-0.38.0_Linux-64bit.deb
1.2 Intall ArgoCD
# https://argo-cd.readthedocs.io/en/stable/getting_started/
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Install ArgoCD CLI in x86_64 Linux
VERSION=$(curl -L -s https://raw.githubusercontent.com/argoproj/argo-cd/stable/VERSION)
wget https://github.com/argoproj/argo-cd/releases/download/v$VERSION/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/bin/argocd
rm argocd-linux-amd64
1.3 Prepare a demo git repo with helm or kustomize
git clone https://github.com/sunway910/cicd-demo
1.4 Secret Configuration
Create Docker Credential Secret
# If your repo is private, please generate the secret as below
# Tekton task kaniko use this secret to push image to a docker repo
kubectl create secret docker-registry docker-cred \
--docker-server=https://index.docker.io/v1/ \
--docker-username="sunway" \
--docker-password="sunway.run" \
--docker-email="[email protected]"
Create GitHub Credential Secret
# Tekton use this secret to clone and push to a private git repo
# Please delete all configuration like github-cred in this blog if git repo is a public repo
kubectl create secret generic github-cred \
--from-file=id_rsa=/root/.ssh/id_rsa \
--from-file=known_hosts=/root/.ssh/known_hosts
Set argocd server insecure if you want
# set argocd tls: https://argo-cd.readthedocs.io/en/stable/operator-manual/tls/
# set insecure mode if you want
kubectl edit cm -n argocd argocd-cmd-params-cm
# add data in comfigmap as below
data:
server.insecure: "true"
kubectl rollout restart deployment argocd-server -n argocd
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
ArgoCD Create non-admin user for dev-team
Tekton pipeline pod security
Note: running TaskRuns and PipelineRuns in the "tekton-pipelines" namespace is discouraged.
- Ref1: Pipeline pod template
- Ref2: Security Standards
2 Build Pipeline
2.0 Create ServiceAccount
sunway123
is the secret
should be set in your GitHub Repo Webhook
sa.yaml
apiVersion: v1
kind: Secret
metadata:
name: github-triggers-secret
type: Opaque
stringData:
secretToken: "sunway123"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: tekton-triggers-github-sa
secrets:
- name: github-triggers-secret
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tekton-triggers-github-minimal
rules:
# EventListeners need to be able to fetch all namespaced resources
- apiGroups: ["triggers.tekton.dev"]
resources:
["eventlisteners", "triggerbindings", "triggertemplates", "triggers"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
# configmaps is needed for updating logging config
resources: ["configmaps"]
verbs: ["get", "list", "watch"]
# Permissions to create resources in associated TriggerTemplates
- apiGroups: ["tekton.dev"]
resources: ["pipelineruns", "pipelineresources", "taskruns"]
verbs: ["create"]
- apiGroups: [""]
resources: ["serviceaccounts"]
verbs: ["impersonate"]
- apiGroups: ["policy"]
resources: ["podsecuritypolicies"]
resourceNames: ["tekton-triggers"]
verbs: ["use"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tekton-triggers-github-binding
subjects:
- kind: ServiceAccount
name: tekton-triggers-github-sa
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: tekton-triggers-github-minimal
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tekton-triggers-github-clusterrole
rules:
- apiGroups: ["triggers.tekton.dev"]
resources: ["clustertriggerbindings", "clusterinterceptors","interceptors"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tekton-triggers-github-clusterbinding
subjects:
- kind: ServiceAccount
name: tekton-triggers-github-sa
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tekton-triggers-github-clusterrole
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: pipeline-account
---
apiVersion: v1
kind: Secret
metadata:
name: kube-api-secret
annotations:
kubernetes.io/service-account.name: pipeline-account
type: kubernetes.io/service-account-token
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pipeline-role
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pipeline-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pipeline-role
subjects:
- kind: ServiceAccount
name: pipeline-account
2.1 Task: Git Clone
Use command kubectl apply -f task-git-clone.yaml
to add task: git-clone
, you can also use command tkn hub install task git-clone
to get task from hub and then customize the yaml file kubectl get task git-clone -o yaml > task-git-clone.yaml
by yourself.
task-git-clone.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: git-clone
labels:
app.kubernetes.io/version: "0.6"
annotations:
tekton.dev/pipelines.minVersion: "0.29.0"
tekton.dev/categories: Git
tekton.dev/tags: git
tekton.dev/displayName: "git clone"
tekton.dev/platforms: "linux/amd64,linux/s390x,linux/ppc64le,linux/arm64"
spec:
description: >-
These Tasks are Git tasks to work with repositories used by other tasks
in your Pipeline.
The git-clone Task will clone a repo from the provided url into the
output Workspace. By default the repo will be cloned into the root of
your Workspace. You can clone into a subdirectory by setting this Task's
subdirectory param. This Task also supports sparse checkouts. To perform
a sparse checkout, pass a list of comma separated directory patterns to
this Task's sparseCheckoutDirectories param.
workspaces:
- name: output
description: The git repo will be cloned onto the volume backing this Workspace.
- name: ssh-directory
optional: true
description: |
A .ssh directory with private key, known_hosts, config, etc. Copied to
the user's home before git commands are executed. Used to authenticate
with the git remote when performing the clone. Binding a Secret to this
Workspace is strongly recommended over other volume types.
- name: basic-auth
optional: true
description: |
A Workspace containing a .gitconfig and .git-credentials file. These
will be copied to the user's home before any git commands are run. Any
other files in this Workspace are ignored. It is strongly recommended
to use ssh-directory over basic-auth whenever possible and to bind a
Secret to this Workspace over other volume types.
- name: ssl-ca-directory
optional: true
description: |
A workspace containing CA certificates, this will be used by Git to
verify the peer with when fetching or pushing over HTTPS.
params:
- name: repo_url
description: Repository URL to clone from.
type: string
- name: revision
description: Revision to checkout. (branch, tag, sha, ref, etc...)
type: string
default: main
- name: refspec
description: Refspec to fetch before checking out revision.
default: ""
- name: submodules
description: Initialize and fetch git submodules.
type: string
default: "true"
- name: depth
description: Perform a shallow clone, fetching only the most recent N commits.
type: string
default: "1"
- name: sslVerify
description: Set the `http.sslVerify` global git config. Setting this to `false` is not advised unless you are sure that you trust your git remote.
type: string
default: "true"
- name: subdirectory
description: Subdirectory inside the `output` Workspace to clone the repo into.
type: string
default: ""
- name: sparseCheckoutDirectories
description: Define the directory patterns to match or exclude when performing a sparse checkout.
type: string
default: ""
- name: deleteExisting
description: Clean out the contents of the destination directory if it already exists before cloning.
type: string
default: "true"
- name: httpProxy
description: HTTP proxy server for non-SSL requests.
type: string
default: ""
- name: httpsProxy
description: HTTPS proxy server for SSL requests.
type: string
default: ""
- name: noProxy
description: Opt out of proxying HTTP/HTTPS requests.
type: string
default: ""
- name: verbose
description: Log the commands that are executed during `git-clone`'s operation.
type: string
default: "true"
- name: gitInitImage
description: The image providing the git-init binary that this Task runs.
type: string
default: "gcriotekton/pipeline-git-init:latest"
- name: userHome
description: |
Absolute path to the user's home directory. Set this explicitly if you are running the image as a non-root user or have overridden
the gitInitImage param with an image containing custom user configuration.
type: string
default: "/tekton/home"
results:
- name: commit
description: The precise commit SHA that was fetched by this Task.
- name: url
description: The precise URL that was fetched by this Task.
- name: pipeline-start-time
description: "The start time of the pipeline"
- name: commit-info
description: "The latest commit info"
- name: done
description: "Is the task works fine"
steps:
- name: clone
securityContext:
runAsUser: 0
image: "$(params.gitInitImage)"
env:
- name: HOME
value: "$(params.userHome)"
- name: PARAM_URL
value: $(params.repo_url)
- name: PARAM_REVISION
value: $(params.revision)
- name: PARAM_REFSPEC
value: $(params.refspec)
- name: PARAM_SUBMODULES
value: $(params.submodules)
- name: PARAM_DEPTH
value: $(params.depth)
- name: PARAM_SSL_VERIFY
value: $(params.sslVerify)
- name: PARAM_SUBDIRECTORY
value: $(params.subdirectory)
- name: PARAM_DELETE_EXISTING
value: $(params.deleteExisting)
- name: PARAM_HTTP_PROXY
value: $(params.httpProxy)
- name: PARAM_HTTPS_PROXY
value: $(params.httpsProxy)
- name: PARAM_NO_PROXY
value: $(params.noProxy)
- name: PARAM_VERBOSE
value: $(params.verbose)
- name: PARAM_SPARSE_CHECKOUT_DIRECTORIES
value: $(params.sparseCheckoutDirectories)
- name: PARAM_USER_HOME
value: $(params.userHome)
- name: WORKSPACE_OUTPUT_PATH
value: $(workspaces.output.path)
- name: WORKSPACE_SSH_DIRECTORY_BOUND
value: $(workspaces.ssh-directory.bound)
- name: WORKSPACE_SSH_DIRECTORY_PATH
value: $(workspaces.ssh-directory.path)
- name: WORKSPACE_BASIC_AUTH_DIRECTORY_BOUND
value: $(workspaces.basic-auth.bound)
- name: WORKSPACE_BASIC_AUTH_DIRECTORY_PATH
value: $(workspaces.basic-auth.path)
- name: WORKSPACE_SSL_CA_DIRECTORY_BOUND
value: $(workspaces.ssl-ca-directory.bound)
- name: WORKSPACE_SSL_CA_DIRECTORY_PATH
value: $(workspaces.ssl-ca-directory.path)
script: |
#!/usr/bin/env sh
set -eu
if [ "${PARAM_VERBOSE}" = "true" ] ; then
set -x
fi
export TZ=UTC-8
TZ='Asia/Shanghai' echo -n "$(date '+Date: %Y-%m-%d Time: %H:%M:%S')" > $(results.pipeline-start-time.path)
if [ "${WORKSPACE_BASIC_AUTH_DIRECTORY_BOUND}" = "true" ] ; then
cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.git-credentials" "${PARAM_USER_HOME}/.git-credentials"
cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.gitconfig" "${PARAM_USER_HOME}/.gitconfig"
chmod 400 "${PARAM_USER_HOME}/.git-credentials"
chmod 400 "${PARAM_USER_HOME}/.gitconfig"
fi
if [ "${WORKSPACE_SSH_DIRECTORY_BOUND}" = "true" ] ; then
cp -R "${WORKSPACE_SSH_DIRECTORY_PATH}" "${PARAM_USER_HOME}"/.ssh
chmod 700 "${PARAM_USER_HOME}"/.ssh
chmod -R 400 "${PARAM_USER_HOME}"/.ssh/*
fi
if [ "${WORKSPACE_SSL_CA_DIRECTORY_BOUND}" = "true" ] ; then
export GIT_SSL_CAPATH="${WORKSPACE_SSL_CA_DIRECTORY_PATH}"
fi
CHECKOUT_DIR="${WORKSPACE_OUTPUT_PATH}/${PARAM_SUBDIRECTORY}"
cleandir() {
# Delete any existing contents of the repo directory if it exists.
#
# We don't just "rm -rf ${CHECKOUT_DIR}" because ${CHECKOUT_DIR} might be "/"
# or the root of a mounted volume.
if [ -d "${CHECKOUT_DIR}" ] ; then
# Delete non-hidden files and directories
# rm -rf "${CHECKOUT_DIR:?}"/ *
rm -rf "${CHECKOUT_DIR:?}/*"
# Delete files and directories starting with . but excluding ..
rm -rf "${CHECKOUT_DIR}"/.[!.]*
# Delete files and directories starting with .. plus any other character
rm -rf "${CHECKOUT_DIR}"/..?*
fi
}
if [ "${PARAM_DELETE_EXISTING}" = "true" ] ; then
cleandir
fi
test -z "${PARAM_HTTP_PROXY}" || export HTTP_PROXY="${PARAM_HTTP_PROXY}"
test -z "${PARAM_HTTPS_PROXY}" || export HTTPS_PROXY="${PARAM_HTTPS_PROXY}"
test -z "${PARAM_NO_PROXY}" || export NO_PROXY="${PARAM_NO_PROXY}"
/ko-app/git-init \
-url="${PARAM_URL}" \
-revision="${PARAM_REVISION}" \
-refspec="${PARAM_REFSPEC}" \
-path="${CHECKOUT_DIR}" \
-sslVerify="${PARAM_SSL_VERIFY}" \
-submodules="${PARAM_SUBMODULES}" \
-depth="${PARAM_DEPTH}" \
-sparseCheckoutDirectories="${PARAM_SPARSE_CHECKOUT_DIRECTORIES}"
cd "${CHECKOUT_DIR}"
git log -1 > $(results.commit-info.path)
RESULT_SHA="$(git rev-parse HEAD | tr -d '\n')"
EXIT_CODE="$?"
if [ "${EXIT_CODE}" != 0 ] ; then
exit "${EXIT_CODE}"
fi
printf "%s" "${RESULT_SHA}" > "$(results.commit.path)"
printf "%s" "${PARAM_URL}" > "$(results.url.path)"
# Make sure we don't add a trailing newline to the result!
echo -n "$RESULT_SHA" > $(results.commit.path)
echo -n "yes" > $(results.done.path)
2.2 Task: Docker Build And Push
Use command kubectl apply -f task-kaniko.yaml
to add task: kaniko-build-and-push
, you can also use command tkn hub install task kaniko
to get task from hub and then customize the yaml file kubectl get task kaniko -o yaml > task-kaniko.yaml
by yourself.
task-kaniko.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: kaniko-build-and-push
spec:
workspaces:
- name: source
description: Holds the context and Dockerfile
- name: dockerconfig
description: Includes a docker `config.json`
optional: true
mountPath: /kaniko/.docker
params:
- name: IMAGE
description: Name (reference) of the image to build.
- name: DOCKERFILE
description: Path to the Dockerfile to build.
default: ./Dockerfile
- name: CONTEXT
description: The build context used by Kaniko.
default: ./
- name: EXTRA_ARGS
type: array
default: [--ignore-path=/product_uuid]
- name: BUILDER_IMAGE
description: The image on which builds will run
default: gcr.io/kaniko-project/executor:latest
results:
- name: IMAGE_DIGEST
description: Digest of the image just built.
- name: IMAGE_URL
description: URL of the image just built.
- name: image-build-end-time
description: "The end time of the build task"
- name: done
description: "Is the task done?"
steps:
- name: init-end-time
image: docker.io/library/bash:5.1.4@sha256:c523c636b722339f41b6a431b44588ab2f762c5de5ec3bd7964420ff982fb1d9
script: |
TZ='Asia/Shanghai' echo -n "$(date '+Date: %Y-%m-%d Time: %H:%M:%S')" > $(results.image-build-end-time.path)
- name: build-and-push
workingDir: $(workspaces.source.path)
image: $(params.BUILDER_IMAGE)
env:
- name: GODEBUG
value: http2client=0
args:
- $(params.EXTRA_ARGS)
- --dockerfile=$(params.DOCKERFILE)
- --context=$(workspaces.source.path)/$(params.CONTEXT) # The user does not need to care the workspace and the source.
- --destination=$(params.IMAGE)
- --digest-file=$(results.IMAGE_DIGEST.path)
# kaniko assumes it is running as root, which means this example fails on platforms
# that default to run containers as random uid (like OpenShift). Adding this securityContext
# makes it explicit that it needs to run as root.
securityContext:
runAsUser: 0
- name: write-url
image: docker.io/library/bash:5.1.4@sha256:c523c636b722339f41b6a431b44588ab2f762c5de5ec3bd7964420ff982fb1d9
script: |
set -e
export TZ=UTC-8
image="$(params.IMAGE)"
echo -n "${image}" | tee "$(results.IMAGE_URL.path)"
TZ='Asia/Hong_Kong' echo -n "$(date '+Date: %Y-%m-%d Time: %H:%M:%S')" > $(results.image-build-end-time.path)
echo -n "yes" > $(results.done.path)
2.3 Task: Update Image Tag In Helm Chart
Use command kubectl apply -f task-update-image-tag.yaml
to add task: task-update-image-tag
task-update-image-tag.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: update-image-tag
spec:
workspaces:
- name: source
- name: ssh-directory
params:
- name: commit_id
description: result of git commit
- name: project_path
type: string
description: which deployment path to update
- name: git_branch
description: git branch to checkout (branch, tag, sha, ref…)
type: string
default: main
results:
- name: pipeline-end-time
- name: done
description: "Is the task done?"
steps:
- name: update-image-tag
image: alpine/git
script: |
#!/usr/bin/env sh
apk add --no-cache bash
export TZ=UTC-8
exec bash << 'EOF'
#!/usr/bin/env bash
set -eu
if [[ -d $(workspaces.ssh-directory.path) ]]; then
mkdir -p ~/.ssh
cp $(workspaces.ssh-directory.path)/known_hosts ~/.ssh/known_hosts
cp $(workspaces.ssh-directory.path)/id_rsa ~/.ssh/id_rsa
chmod 700 ~/.ssh
chmod 600 ~/.ssh/*
fi
# git init only required in network file system pvc, do not required in disk storage pvc, cause nfs deny git search its parent working tree
# git init
git clone [email protected]:sunway910/cicd-demo.git
cd deployment
read -a path_array <<<"$(params.project_path)"
for path in "${path_array[@]}"; do
echo "Update image tag in $path"
cd helm/$path
sed -i "s|tag:.*|tag: $(params.commit_id)|" values.yaml || { echo "Failed to update values.yaml"; exit 1; }
cd -
done
git config --global user.email "[email protected]"
git config --global user.name "Tekton Bot"
git add .
echo "Commit changes: skip ci: update image tag to $(params.commit_id)"
git commit -m "skip ci: update image tag to $(params.commit_id)" || { echo "Failed to commit changes"; exit 1; }
git push || { echo "Failed to push changes"; exit 1; }
TZ='Asia/Shanghai' echo -n "$(date '+Date: %Y-%m-%d Time: %H:%M:%S')" > $(results.pipeline-end-time.path)
echo -n "yes" > $(results.done.path)
EOF
securityContext:
runAsUser: 0
2.4 Task: Send Notification (Optional)
The last task can be used to send notification to slack/email or something else
, here is an example of sending notification to lark
:
Please kindly check your IM Tool Docs
to get the Webhook URL
and Request Body Format
.
Otherwise, you can just skip this task and delete it in your pipeline finally task.
send-notification.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: send-notification-to-lark
labels:
app.kubernetes.io/version: "0.1"
annotations:
tekton.dev/pipelines.minVersion: "0.12.1"
tekton.dev/displayName: "Send message to Lark Group"
tekton.dev/categories: Messaging
tekton.dev/tags: messaging
spec:
description: >-
These tasks post a simple message to a lark group.
This task uses Incoming Webhooks of lark to send the message.
params:
- name: message_type
type: string
default: "interactive"
- name: wide_screen_mode
type: string
default: "true"
- name: pipeline_title
type: string
default: "Demo Pipeline"
- name: pipeline_result
type: string
description: The result of the pipeline, can be "Succeeded" or "Failed"
default: "Succeeded"
- name: commit_info
type: string
default: "Author: Tekton Bot"
- name: pipeline_result_link
type: string
default: "https://tekton-dash.sunway.run/#/pipelineruns"
- name: github_commit_link
type: string
default: "https://github.com/sunway910/cicd-demo"
- name: mention_user_open_id
type: string
default: "all"
- name: pipeline_owner
type: string
default: "ou_xxxxxx"
- name: pipeline_start_time
type: string
default: "2024-01-01 00:00:00"
- name: webhook_url
type: string
default: ""
steps:
- name: lark-webhook
image: docker.io/curlimages/curl:7.70.0@sha256:031df77a11e5edded840bc761a845eab6e3c2edee22669fb8ad6d59484b6a1c4 #tag: 7.70.0
script: |
#!/bin/sh
set -e
# if params.pipeline_result is "Succeeded", then use set content color to green, else set it to red
# if params.pipeline_result is "Succeeded", then set pipeline_result_content to "The pipeline has completed successfully", else set it to "The pipeline has failed"
# The color of the lark webhook card, can be red, orange, yellow, green, blue, purple, gray
echo "pipeline_result: $(params.pipeline_result)"
PIPELINE_RESULT="$(params.pipeline_result)"
if [ "$PIPELINE_RESULT" = "Succeeded" ]; then
pipeline_result_content="The pipeline has completed successfully"
message_color="green"
else
pipeline_result_content="The pipeline has failed"
message_color="red"
fi
echo "Send to webhook_url: $(params.webhook_url)"
message_type="$(params.message_type)"
wide_screen_mode="$(params.wide_screen_mode)"
pipeline_title="$(params.pipeline_title)"
pipeline_start_time="$(params.pipeline_start_time)"
mention_user_open_id="$(params.mention_user_open_id)"
pipeline_owner="$(params.pipeline_owner)"
commit_info=$(echo -n "$(params.commit_info)" | awk 1 ORS='\\n')
pipeline_result_link="$(params.pipeline_result_link)"
github_commit_link="$(params.github_commit_link)"
webhook_url="$(params.webhook_url)"
json_data=$(cat <<EOF
{
"msg_type": "${message_type}",
"card": {
"config": {
"wide_screen_mode": ${wide_screen_mode}
},
"header": {
"title": {
"content": "${pipeline_title}",
"tag": "plain_text"
},
"template": "${message_color}"
},
"elements": [
{
"tag": "div",
"fields": [
{
"is_short": true,
"text": {
"content": "**Start Time:**\n${pipeline_start_time}",
"tag": "lark_md"
}
}
]
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "<at id=${mention_user_open_id}></at> \n <at id=${pipeline_owner}></at> \n ${pipeline_result_content}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Commit Info:**\n ${commit_info}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"content": "Pipeline CI Details",
"tag": "plain_text"
},
"url": "${pipeline_result_link}",
"type": "default"
}
]
},
{
"tag": "hr"
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"content": "Github Commit Details",
"tag": "plain_text"
},
"url": "${github_commit_link}",
"type": "default"
}
]
}
]
}
}
EOF
)
curl -X POST -H "Content-Type: application/json" -d "${json_data}" "$webhook_url"
2.5 Create Pipeline
Use command kubectl apply -f pipeline.yaml
to create pipeline: cicd-demo-pipeline
customized pipeline.yaml
file as below:
- tasks
fetch-from-git
: paramsDockerfile
value is$(workspaces.source.path)/Dockerfile
, modify the path if your Dockerfile is not in the root of your git repo. - tasks
update-image-tag
: paramsproject_path
value iscicd-demo-helm
, modify the path if yourhelm/kustomize
directory path.
pipeline.yaml
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: cicd-demo-pipeline
spec:
params:
# clone from which repo and revision
- name: repo_url
type: string
- name: repo_revision
type: string
- name: git_branch
type: string
# push image to which registry and repository
- name: image-registry
default: docker.io/sunway
- name: image-repo-name
type: string
workspaces:
- name: git-source
- name: docker-cred
- name: github-cred
tasks:
- name: fetch-from-git
onError: continue
taskRef:
name: git-clone
params:
- name: repo_url
value: $(params.repo_url)
- name: revision
value: $(params.repo_revision)
- name: deleteExisting
value: "true"
workspaces:
- name: output
workspace: git-source
- name: ssh-directory
workspace: github-cred
- name: build-image
onError: continue
when:
- input: $(tasks.fetch-from-git.results.done)
operator: in
values: [ "yes" ]
runAfter: [ fetch-from-git ]
taskRef:
name: kaniko-build-and-push
params:
- name: IMAGE
value: $(params.image-registry)/$(params.image-repo-name):$(tasks.fetch-from-git.results.commit)
- name: CONTEXT
value: ""
- name: DOCKERFILE
value: $(workspaces.source.path)/Dockerfile
workspaces:
- name: source
workspace: git-source
- name: dockerconfig
workspace: docker-cred
- name: update-image-tag
onError: continue
when:
- input: $(tasks.build-image.results.done)
operator: in
values: [ "yes" ]
runAfter: [ build-image ]
taskRef:
name: update-image-tag
params:
- name: git_branch
value: $(params.git_branch)
- name: commit_id
value: $(tasks.fetch-from-git.results.commit)
# cd https://github.com/sunway910/cicd-demo/$project_path
- name: project_path
value: cicd-demo-helm
workspaces:
- name: source
workspace: git-source
- name: ssh-directory
workspace: github-cred
# delete finally task if you dont need to send notification to IM
# finally task will not be executed if pipeline_start_time/pipeline_end_time is null
finally:
- name: send-notification-to-lark
taskRef:
name: send-notification-to-lark
params:
- name: webhook_url
value: "https://open.larksuite.com/open-apis/bot/v2/hook/xxxxx"
- name: pipeline_title
value: "Demo Pipeline"
- name: pipeline_result
value: $(tasks.status)
- name: commit_info
value: $(tasks.fetch-from-git.results.commit-info)
- name: pipeline_result_link
value: "https://tekton-dashboard.sunway.run/#/pipelineruns"
- name: github_commit_link
value: https://github.com/sunway910/cicd-demo/commit/$(tasks.fetch-from-git.results.commit)
- name: mention_user_open_id
value: "all"
- name: pipeline_owner
value: "ou_xxx"
- name: pipeline_start_time
value: $(tasks.fetch-from-git.results.pipeline-start-time)
2.6 Schedule Pipeline (Optional)
If you do not want pipeline affects other resources in kubernetes cluster, you can set pipeline run at a specific kubernetes node with label: pipeline=true
:
kubectl label nodes <node-name> pipeline=true
kubectl get cm config-defaults -n tekton-pipelines -o yaml > /tmp/config-defaults-bk.yaml
All of pipelines configuration will extend from configmap config-defaults
and then be scheduled on the node with label pipeline=true
config-default.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: default
app.kubernetes.io/part-of: tekton-pipelines
name: config-defaults
namespace: tekton-pipelines
data:
# kubectl label node node-xxx node-role.kubernetes.io/worker=pipeline
# kubectl taint node node-xxx dedicated=pipeline:NoSchedule
default-pod-template: |
nodeSelector:
node-role.kubernetes.io/worker: pipeline
tolerations:
- key: "dedicated"
operator: "Equal"
value: "pipeline"
effect: "NoSchedule"
_example: |
################################
# #
# EXAMPLE CONFIGURATION #
# #
################################
# This block is not actually functional configuration,
# but serves to illustrate the available configuration
# options and document them in a way that is accessible
# to users that `kubectl edit` this config map.
#
# These sample configuration options may be copied out of
# this example block and unindented to be in the data block
# to actually change the configuration.
# default-timeout-minutes contains the default number of
# minutes to use for TaskRun and PipelineRun, if none is specified.
default-timeout-minutes: "60" # 60 minutes
# default-service-account contains the default service account name
# to use for TaskRun and PipelineRun, if none is specified.
default-service-account: "default"
# default-managed-by-label-value contains the default value given to the
# "app.kubernetes.io/managed-by" label applied to all Pods created for
# TaskRuns. If a user's requested TaskRun specifies another value for this
# label, the user's request supercedes.
default-managed-by-label-value: "tekton-pipelines"
# default-pod-template contains the default pod template to use for
# TaskRun and PipelineRun. If a pod template is specified on the
# PipelineRun, the default-pod-template is merged with that one.
# default-pod-template: |
# default-affinity-assistant-pod-template contains the default pod template
# to use for affinity assistant pods. If a pod template is specified on the
# PipelineRun, the default-affinity-assistant-pod-template is merged with
# that one.
# default-affinity-assistant-pod-template:
# default-cloud-events-sink contains the default CloudEvents sink to be
# used for TaskRun and PipelineRun, when no sink is specified.
# Note that right now it is still not possible to set a PipelineRun or
# TaskRun specific sink, so the default is the only option available.
# If no sink is specified, no CloudEvent is generated
# default-cloud-events-sink:
# default-task-run-workspace-binding contains the default workspace
# configuration provided for any Workspaces that a Task declares
# but that a TaskRun does not explicitly provide.
# default-task-run-workspace-binding: |
# emptyDir: {}
# default-max-matrix-combinations-count contains the default maximum number
# of combinations from a Matrix, if none is specified.
default-max-matrix-combinations-count: "256"
# default-forbidden-env contains comma seperated environment variables that cannot be
# overridden by podTemplate.
default-forbidden-env:
# default-resolver-type contains the default resolver type to be used in the cluster,
# no default-resolver-type is specified by default
default-resolver-type:
# default-imagepullbackoff-timeout contains the default duration to wait
# before requeuing the TaskRun to retry, specifying 0 here is equivalent to fail fast
# possible values could be 1m, 5m, 10s, 1h, etc
# default-imagepullbackoff-timeout: "5m"
# default-maximum-resolution-timeout specifies the default duration used by the
# resolution controller before timing out when exceeded.
# Possible values include "1m", "5m", "10s", "1h", etc.
# Example: default-maximum-resolution-timeout: "1m"
# default-container-resource-requirements allow users to update default resource requirements
# to a init-containers and containers of a pods create by the controller
# Onet: All the resource requirements are applied to init-containers and containers
# only if the existing resource requirements are empty.
# default-container-resource-requirements: |
# place-scripts: # updates resource requirements of a 'place-scripts' container
# requests:
# memory: "64Mi"
# cpu: "250m"
# limits:
# memory: "128Mi"
# cpu: "500m"
#
# prepare: # updates resource requirements of a 'prepare' container
# requests:
# memory: "64Mi"
# cpu: "250m"
# limits:
# memory: "256Mi"
# cpu: "500m"
#
# working-dir-initializer: # updates resource requirements of a 'working-dir-initializer' container
# requests:
# memory: "64Mi"
# cpu: "250m"
# limits:
# memory: "512Mi"
# cpu: "500m"
#
# prefix-scripts: # updates resource requirements of containers which starts with 'scripts-'
# requests:
# memory: "64Mi"
# cpu: "250m"
# limits:
# memory: "128Mi"
# cpu: "500m"
#
# prefix-sidecar-scripts: # updates resource requirements of containers which starts with 'sidecar-scripts-'
# requests:
# memory: "64Mi"
# cpu: "250m"
# limits:
# memory: "128Mi"
# cpu: "500m"
#
# default: # updates resource requirements of init-containers and containers which has empty resource resource requirements
# requests:
# memory: "64Mi"
# cpu: "250m"
# limits:
# memory: "256Mi"
# cpu: "500m"
2.7 Create TriggerTemplate
Use command kubectl apply -f triggertemplate.yaml
to create triggertemplate: cicd-demo-github-template
customized triggertemplate.yaml
file as below, task update-image-tag
will push image to docker.io/sunway/cicd-demo:$(params.commit_id)
:
- image-registry:
docker.io/sunway
- image-repo-name:
cicd-demo
triggerTemplate.yaml
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
name: cicd-demo-github-template
spec:
params:
- name: gitrevision
- name: gitrepositoryurl
- name: gitbranch
resourceTemplates:
- apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
generateName: tekton-sunway-cicd-demo-
spec:
pipelineRef:
name: cicd-demo-pipeline
taskRunTemplate:
serviceAccountName: pipeline-account
workspaces:
- name: git-source
volumeClaimTemplate:
# please make sure a default storage class is available in your cluster
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
#storageClassName: my-custom-sc
- name: docker-cred
secret:
defaultMode: 420
items:
- key: .dockerconfigjson
path: config.json
secretName: docker-cred
- name: github-cred
secret:
defaultMode: 420
secretName: github-cred
params:
- name: repo_url
value: $(tt.params.gitrepositoryurl)
- name: repo_revision
value: $(tt.params.gitrevision)
- name: git_branch
value: $(tt.params.gitbranch)
- name: image-registry
value: docker.io/sunway
- name: image-repo-name
value: cicd-demo
2.8 Create TriggerBinding
Use command kubectl apply -f triggerbinding.yaml
to create triggerbinding: github-push-binding
triggerbinding.yaml
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
name: github-push-binding
spec:
params:
- name: gitrevision
value: $(body.head_commit.id)
- name: gitrepositoryurl
value: $(body.repository.ssh_url)
- name: gitbranch
value: $(body.ref)
2.9 Create EventListener
Use command kubectl apply -f eventlistener.yaml
to create eventlistener: cicd-demo-github-listener
Trigger Condition:
- 1: repo name is equal to
cicd-demo
- 2: commit message does not contain
skip ci
. - 3: push/pull_request on main/master branch
eventlistener.yaml
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
name: cicd-demo-github-listener
spec:
serviceAccountName: tekton-triggers-github-sa
triggers:
- name: cicd-demo-github-listener
interceptors:
- ref:
name: "github"
params:
- name: "secretRef"
value:
secretName: github-triggers-secret
secretKey: secretToken
- name: "eventTypes"
value: [ "push", "pull_request" ]
- ref:
# Common Expression Language
name: "cel"
params:
- name: "filter"
# https://github.com/sunway910/cicd-demo
value: "body.repository.name == 'cicd-demo' && !body.head_commit.message.contains('skip ci') && (body.ref == 'refs/heads/main' || body.ref == 'refs/heads/master')"
bindings:
- ref: github-push-binding
template:
ref: cicd-demo-github-template
2.10 Create ArgoCD Notification (Optional)
use kubectl apply -f ...
to update configmap: argocd-notifications-cm
then: kubectl rollout restart deployment argocd-notifications-controller -n argocd
argocd-notifications-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-notifications-cm
namespace: argocd
data:
service.webhook.team-a-hook: |
url: "https://open.larksuite.com/open-apis/bot/v2/hook/123456"
headers:
- name: Content-Type
value: application/json
service.webhook.team-b-hook: |
url: "https://open.larksuite.com/open-apis/bot/v2/hook/654321"
headers:
- name: Content-Type
value: application/json
context: |
argocdUrl: https://argocd.sunway.run
trigger.on-deployed: | # application is healthy after being synced
- description: Application is synced and healthy. Triggered once per commit.
send: [app-status-change]
when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
trigger.on-sync-running: | # start syncing
- description: Application is being synced
send: [app-status-change]
when: app.status.operationState.phase in ['Running']
trigger.on-sync-succeeded: | # syncing has succeeded
- description: Application syncing has succeeded
send: [app-status-change]
when: app.status.operationState.phase in ['Succeeded']
trigger.on-health-degraded: |
- description: Application has degraded
send: [app-degraded]
when: app.status.health.status == 'Degraded'
trigger.on-sync-failed: |
- description: Application syncing has failed
send: [app-degraded]
when: app.status.operationState.phase in ['Error', 'Failed']
trigger.on-sync-status-unknown: |
- description: Application status is 'Unknown'
send: [app-degraded]
when: app.status.sync.status == 'Unknown'
template.app-status-change: |
webhook:
team-a-hook:
method: POST
body: |
{
"msg_type": "interactive",
"card": {
"config": {
"wide_screen_mode": true
},
"header": {
"title": {
"content": "Prod Application Status Changed",
"tag": "plain_text"
},
"template": "green"
},
"elements": [
{
"tag": "div",
"text": {
"content": "<at id=all></at>",
"tag": "lark_md"
}
},
{
"tag": "div",
"text": {
"content": "**Application Name:** {{.app.metadata.name}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Sync Status:** {{.app.status.operationState.phase}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Application Status:** {{.app.status.health.status}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Time:** {{.app.status.operationState.startedAt}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"content": "Prod ArgoCD Deployment Details",
"tag": "plain_text"
},
"url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
"type": "default"
}
]
}
]
}
}
team-b-hook:
method: POST
body: |
{
"msg_type": "interactive",
"card": {
"config": {
"wide_screen_mode": true
},
"header": {
"title": {
"content": "Prod Application Status Changed",
"tag": "plain_text"
},
"template": "green"
},
"elements": [
{
"tag": "div",
"text": {
"content": "<at id=all></at>",
"tag": "lark_md"
}
},
{
"tag": "div",
"text": {
"content": "**Application Name:** {{.app.metadata.name}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Sync Status:** {{.app.status.operationState.phase}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Application Status:** {{.app.status.health.status}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Time:** {{.app.status.operationState.startedAt}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"content": "Prod ArgoCD Deployment Details",
"tag": "plain_text"
},
"url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
"type": "default"
}
]
}
]
}
}
template.app-degraded: |
webhook:
team-a-hook:
method: POST
body: |
{
"msg_type": "interactive",
"card": {
"config": {
"wide_screen_mode": true
},
"header": {
"title": {
"content": "Alert Prod Application Degraded",
"tag": "plain_text"
},
"template": "red"
},
"elements": [
{
"tag": "div",
"text": {
"content": "<at id=all></at>",
"tag": "lark_md"
}
},
{
"tag": "div",
"text": {
"content": "**Application Name:** {{.app.metadata.name}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Sync Status:** {{.app.status.operationState.phase}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Application Status:** {{.app.status.health.status}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Time:** {{.app.status.operationState.startedAt}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"content": "Prod ArgoCD Deployment Details",
"tag": "plain_text"
},
"url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
"type": "default"
}
]
}
]
}
}
team-b-hook:
method: POST
body: |
{
"msg_type": "interactive",
"card": {
"config": {
"wide_screen_mode": true
},
"header": {
"title": {
"content": "Alert Prod Application Degraded",
"tag": "plain_text"
},
"template": "red"
},
"elements": [
{
"tag": "div",
"text": {
"content": "<at id=all></at>",
"tag": "lark_md"
}
},
{
"tag": "div",
"text": {
"content": "**Application Name:** {{.app.metadata.name}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Sync Status:** {{.app.status.operationState.phase}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Application Status:** {{.app.status.health.status}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "div",
"text": {
"content": "**Time:** {{.app.status.operationState.startedAt}}",
"tag": "lark_md"
}
},
{
"tag": "hr"
},
{
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"content": "Prod ArgoCD Deployment Details",
"tag": "plain_text"
},
"url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
"type": "default"
}
]
}
]
}
}

3 Create Ingress
3.1 Create EventListener Ingress
GitHub Repo Webhook will send request to this ingress: git-listener-cicd-demo.sunway.run
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: pipeline-ingress
spec:
ingressClassName: nginx
rules:
- host: git-listener-cicd-demo.sunway.run
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: cicd-demo-github-listener
port:
number: 8080
3.2 Create Tekton Dashboard Ingress
Access to https://tekton-dashboard.sunway.run
and watch the tasks progress in pipelineRun after push/pull_request
to GitHub repo...
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tekton-dashboard-ingress
namespace: tekton-pipelines
spec:
ingressClassName: nginx
rules:
- host: tekton-dashboard.sunway.run
http:
paths:
- path: /
pathType: Prefix
backend:
service:
# change service backend and port to oauth-proxy if you want to protect your dashboard
name: tekton-dashboard
port:
number: 9097
Optional: How to protect your tekton-dashboard in public network?
3.2 Create ArgoCD Dashboard Ingress
Access to https://argocd.sunway.run
with user:admin
and password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd-ingress
namespace: argocd
spec:
ingressClassName: nginx
rules:
- host: argocd.sunway.run
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: argocd-server
port:
# set port to 443 if you do not disable tls in argocd-server in 1.4 Secret Configuration
number: 80
4 ArgoCD Configuration
4.1 Add Others Kubernetes Clusters

- Step 1: Login to Argo CD Make sure that you have updated the cluster details to the
kubeconfig
file and logged into Argo CD using Argo CD CLI, if not run the following command to log in to Argo CD.
argocd login https://argocd.sunway.run --username admin --password axDaGIswqYaaHxBs
- Step 2: Get the Context of the Cluster
Once you have logged in to Argo CD, you need to find the context of the cluster you need to add to Argo CD.
Run the following command to get the context from the kubeconfig
file
# execute this on others kubernetes cluster and then get context name
kubectl config get-contexts -o name
- Step 3: Add the Cluster To add the cluster to Argo CD, use the context of the running cluster you got from the previous step on the below command
# If you set proxy like cloudflare before others kubernetes api-server, you have to trust the proxy server certificate. Otherwise, you shouldn't set up a proxy on your api-server
argocd cluster add --kubeconfig <path-of-kubeconfig-file> --kube-context string <cluster-context> --name <cluster-name>
This command will create a service account Argo CD-manager
on the cluster you specify in the above command with full cluster privileges, so make sure you have the required permissions on the cluster.
4.2 Create A Git Repository
connect to a private repo via ssh as below

4.3 Create An Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cicd-demo
spec:
destination:
name: ''
namespace: app
server: https://kubernetes.default.svc
source:
path: cicd-demo-helm
repoURL: https://github.com/sunway910/cicd-demo
targetRevision: main
sources: []
project: default
syncPolicy:
automated:
prune: false
selfHeal: false


5 Github Webhook
Set github webhook to tekton eventListener as below
Payload URL: https://git-listener-cicd-demo.sunway.run/github
Content type: application/json
Secret: sunway123

6 Testing
You can see your application after you create your own application at 4.3
Application indicate processing
cause argocd cant check the ingress
health, it is a normal status if you use nginx ingress
or something else. [ ref1, ref2 ]

After you check application is healthy, you can push code to your GitHub repo and then check pipelineRun
on Tekton Dashboard.
ArgoCD will auto sync the image tag
changes in values.yaml
when pipelineRun done.
Argocd will check changes every 180 seconds
, but you can also click sync
manually to sync immediately.