Skip to content

See GitOps Stuff for stuff related to the certificate-based integration, GitLab agent for Kubernetes, Flux and friends.

See Runner on Kubernetes for Runner-specific Kubernetes things. See GitLab Helm Chart + JumpCloud LDAP (and Secrets) for stuff specific to LDAP + GitLab deployed via Helm chart.

Want tips on troubleshooting backup-utility? Me, too! I started writing some out. LMK if you have ideas! 

kubectl exec -it  $(kubectl get pods -lapp=webservice -o name | head -n1 | cut -d"/" -f2)    -- /bin/bash

The GitLab Rails Console is in the Toolbox pod.

Get all Toolbox pods

kubectl get pods -lapp=toolbox -o name | cut -d"/" -f2

Get a Toolbox pod

kubectl get pods -lapp=toolbox -o name | cut -d"/" -f2 | head -n1

Shell into a Toolbox pod

kubectl exec -it  $(kubectl get pods -lapp=toolbox -o name | cut -d"/" -f2 | head -n1) -- /bin/bash

Straight into Rails Console in a Toolbox pod

kubectl exec -it  $(kubectl get pods -lapp=toolbox -o name | cut -d"/" -f2 | head -n1) -- /srv/gitlab/bin/rails  console

Runner via Helm

A brief summary of how I deployed different versions of GitLab Runner via Helm:

 1890  which helm
 1892  brew install helm
 1893  which helm
 1936  helm repo add gitlab https://charts.gitlab.io/
 1937  helm repo update
 2774  helm search repo -l gitlab/gitlab-runner | head
 2777  helm install --namespace cutecats gitlab-runner -f values.yaml gitlab/gitlab-runner --version 0.30.0
 2778  helm upgrade --namespace cutecats -f values.yaml gitlab-runner gitlab/gitlab-runner --version 0.41.1
 2779  helm upgrade --namespace cutecats -f values.yaml gitlab-runner gitlab/gitlab-runner --version 0.41.0
 2780  helm upgrade --namespace cutecats -f values.yaml gitlab-runner gitlab/gitlab-runner --version 0.42.0

See this snippet.

6.11.13 is my goal this time.

values.yaml

gitlabUrl: https://gitlab.carranzas.com
runnerRegistrationToken: 
replicas: 2
runners:
  name:
  serviceAccountName:
  tags:

rbac:
  create: true
  resources: ["*"]
  verbs: ["get", "list", "watch", "create", "patch", "delete", "update"]
  serviceAccountAnnotations:
    eks.amazonaws.com/role-arn: "hello"

Upgrading to a new version

helm upgrade gitlab gitlab/gitlab  --version 5.9.0   -f gke.yml    --set gitlab.migrations.enabled=true

Use incremental logging to avoid problems with logs not all being uploaded.

Docs: Incremental logging architecture Read the technical details.

This changes whether it's rails or redis handling the traces.

Troubleshooting

Windows Runner Nodes

  • Try FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=1 first.
  • Don't test in GKE. EKS should be fine.

attach vs exec

General Kubernetes Stuff

Get logs from a pod that is no more

kubectl logs my-pod --previous # dump pod logs (stdout) for a previous instantiation of a container

Use a different cluster

Switch contexts.

As always, use the kubectl cheatsheet. See Kubectl context and configuration.

These are the four most helpful ones for me on this: 

kubectl config get-contexts                          # display list of contexts
kubectl config current-context                       # display the current-context
kubectl config use-context my-cluster-name           # set the default context to my-cluster-name

kubectl config set-cluster my-cluster-name           # set a cluster entry in the kubeconfig

Set a default namespace

This is from the Kubernetes documentation on setting a namespace preference:

kubectl config set-context --current --namespace=omgbriedev
kubectl config view --minify | grep namespace:

What namespace am I in?

# kubectl config view --minify -o jsonpath='{..namespace}'
# kubectl config view | grep namespace

View information about the relationship between pods, PVCs and namespaces

From this How to get the relationship between the Persistent Volumes and the Pods in Kubernetes post:

kubectl get po -o json --all-namespaces | jq -j '.items[] | "\(.metadata.namespace), \(.metadata.name), \(.spec.volumes[].persistentVolumeClaim.claimName)\n"' | grep -v null

Troubleshooting

BYO Pod

Build your own test pod and test stuff in it:

kubectl run test-pod --image=alpine:latest -it

## While inside the test pod, install curl
apk update
apk add curl openldap-clients

curl -v https://brie.dev

k9s

It's great.

Don't rely on it too much.

Cloud Specific

This is where I'm keeping notes on things that are different between the Kubernetes you get in one cloud vs another.

EKS

You should find that OOM is not enabled by default in EKS.

GKE

Has it's own dashboard

Recovering credentials

Let's assume you already have gcloud working with the right GCP account.

Get the cluster name

gcloud container clusters list

Docs:

🔐 Certificates