Did it on a macbook air as I am away from my Linux workstation at home
I will use some of my selfhosted services in examples, e.g. registry.bouvais.lu for docker registry and git.bouvais.lu for gitea.

1. Create env

First let's install everything that I will use using brew.

% brew install docker
% brew install kind kubectl helm argocd k9s

2. Images

To created the needed dependency, I did a simple 2 images base + jupyter. The first image is a minimal python slim. I then add some

Base is a simple python:3.12-slim-bookworm + tini using a non-root user. Jupyter simply install python dependencies and start the jupyterlab server.

I then build and push them to my registry

% docker build -t registry.bouvais.lu/tenant-base:1.0.0 images/base
% docker push registry.bouvais.lu/tenant-base:1.0.0 

% docker build -t registry.bouvais.lu/tenant-jupyter:1.0.0 images/jupyter
% docker push registry.bouvais.lu/tenant-jupyter:1.0.0

3. Cluster setup

In this section, will create the cluster, create namespaces and shared minio + argo cd namespace.

3.1 Cluster

Now for the actual deployment, I will make a simple local kubernetes cluser with kind.

% kind create cluster --name ctie-exercice
% kubectl cluster-info

% kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable"
% kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml

3.2 Minio

I deploy a unique shared minio instance in a namespace named minio. Obviously it can have its own scaling stategy later if needed but that's out of scope here.

Create minio/manifests.yaml and deploying it:

% kubectl create ns minio
% kubectl apply -f minio/manifests.yaml
% kubectl -n minio get pods
NAME                     READY   STATUS    RESTARTS   AGE
minio-688ffcdbbd-qm47b   1/1     Running   0          3m6s

Can check the connection by forwarding port and goinf to localhost:9001

% kubectl -n minio port-forward svc/minio 9000:9000 9001:9001

3.3 Argo CD

Similar to minio, a single instance in a unique namespace. First lets fetch and run an Argo CD insance.

% kubectl create ns argocd
% kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Similarly, can get admin password + forward to go to Argo CD webui. Obviously getting and using admin credentials is not good, but onece again that's out of scope.

% kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
% kubectl -n argocd port-forward svc/argocd-server 8080:443

4. GitOps

Now let's create Argo CD application set automated with a path in the repo.

First let's add files in gitops/ and run:

% kubectl apply -f gitops/bootstrap/root-app.yaml

At localhost:8080, we can see the root application. Now when we add any directory in format tenant-* with a config.yaml file, it will automatically create a namespace and a SA, deploy a jupyterlab server, add a new bucket to main monio.

[root-app.yaml] (Applied manually)
      │
      ├──► Creates AppProject ("tenants")
      └──► Creates ApplicationSet ("tenants")
                │
                ├──► Scans Git for tenants/*/config.yaml
                │
                └──► Generates Application: tenant-a
                          │
                          ├──► Pulls Helm Chart from: charts/tenant
                          ├──► Applies Values from: tenants/tenant-a/config.yaml
                          └──► Create and deploy to Namespace: tenant-a

3.4 Secrets

Now that we have working automated tenants, they need secrets. I will also use the addon reflector to automatically add secrets to tenants.

Docker Registry

First let's add docker registry credentials so it can pull the built jupyter image.

kubectl create secret docker-registry registry-credentials \
  --docker-server=registry.bouvais.lu \
  --docker-username="" \
  --docker-password=""
  --namespace=argocd

And to automatically make them available to tenant namespaces using reflector. It will propagate registry-credentials secret to namespace in format tenant-something.

kubectl annotate secret registry-credentials -n argocd --overwrite \
  reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"

From here we can access the notebook with:

kubectl port-forward -n tenant-a deployment/jupyter 8888:8888

For this demo, I will stop here. Meaning accessing the notebook using a manual port forwarding. But in reality, this would need a route, automated forward, CA, ect. But that's out of scope again.

Minio Admin

Let's add Minio Admin credentials as a secret too. Similarly propagating them to tenant namespaces.

kubectl create secret generic minio-admin-credentials \
  --from-literal=MINIO_ROOT_USER=something \
  --from-literal=MINIO_ROOT_PASSWORD=something \
  -n argocd

kubectl annotate secret minio-admin-credentials -n argocd --overwrite \
  reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"

Minio User

The tenant chart also automatically create the 2 new buckets and a random credential saved only in the tenant namespace that will use it.

% kubectl get events -n tenant-a --field-selector reason=Completed
LAST SEEN   TYPE     REASON      OBJECT                     MESSAGE
7m6s        Normal   Completed   job/tenant-a-minio-setup   Job completed
% kubectl get secrets -n tenant-a
NAME                      TYPE                             DATA   AGE
s3-credentials            Opaque                           5      90s
% kubectl get secret s3-credentials -n tenant-a -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode
echo ""
xXjvsoRXaEI1GtvVfUrMZOkR

Use Jupyter

Now we should have everything to use the notebook.

Let's go to localhost:8888 and run images/jupyter/test_script.py.

SUCCESS: List items in ref bucket: 1 hello tenant-a
SUCCESS: Write to reference bucket blocked: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.
SUCCESS: Read back from work bucket: hello tenant
SUCCESS: Access to tenant-b blocked: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied.

We are succesfully:

Reading but not writing from bucket ref.
Read and write in work bucket
Cant read nor write in bucket tenant-b

Docker build

Now let's automate docker images. I will go with a simple Action. Gitea has Action like Github.

It is trigger only on tag *.*.*. It then build the base and then jupyter image and push it to the registry using the tag. Jupyter image use the just previously build base.

TODO: une NetworkPolicy limite les communications du notebook aux services nécessaires ; Une pipeline doit : • valider les fichiers de configuration ; • construire les images dans le bon ordre ; • exécuter au moins un test ; • publier les images dans un registre ou simuler clairement cette étape ; • mettre à jour la version utilisée par le déploiement.

Configs

I kept configs minimal, but in real here a list of things that could be added:

A storage limit for Work bucket
A GPU option for the Jupyter