# 0. Introduction

- Did it on a macbook air as I am away from my Linux workstation at home
- I will use some of my selfhosted services in examples, e.g. `registry.bouvais.lu` for docker registry and `git.bouvais.lu` for gitea.

# 1. Create env

First let's install everything that I will use using `brew`.

```
% brew install docker
% brew install kind kubectl helm argocd k9s
```

# 2. Images

To created the needed dependency, I did a simple 2 images base + jupyter.
The first image is a minimal python slim. I then add some 

Base is a simple `python:3.12-slim-bookworm` + tini using a non-root user.
Jupyter simply install python dependencies and start the jupyterlab server.

I then build and push them to my registry

```
% docker build -t registry.bouvais.lu/tenant-base:1.0.0 images/base
% docker push registry.bouvais.lu/tenant-base:1.0.0 

% docker build -t registry.bouvais.lu/tenant-jupyter:1.0.0 images/jupyter
% docker push registry.bouvais.lu/tenant-jupyter:1.0.0 
```

# 3. Cluster setup

In this section, will create the cluster, create namespaces and shared minio + argo cd namespace.

### 3.1 Cluster

Now for the actual deployment, I will make a simple local kubernetes cluser with `kind`.

```
% kind create cluster --name ctie-exercice
% kubectl cluster-info
```

```
% kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable"
% kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml
```

### 3.2 Minio

I deploy a unique shared minio instance in a namespace named minio.
Obviously it can have its own scaling stategy later if needed but that's out of scope here.

Create `minio/manifests.yaml` and deploying it:

```
% kubectl create ns minio
% kubectl apply -f minio/manifests.yaml
% kubectl -n minio get pods
NAME                     READY   STATUS    RESTARTS   AGE
minio-688ffcdbbd-qm47b   1/1     Running   0          3m6s
```

Can check the connection by forwarding port and goinf to localhost:9001

```
% kubectl -n minio port-forward svc/minio 9000:9000 9001:9001
```

### 3.3 Argo CD

Similar to minio, a single instance in a unique namespace.
First lets fetch and run an Argo CD insance.

```
% kubectl create ns argocd
% kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```

Similarly, can get admin password + forward to go to Argo CD webui.
Obviously getting and using admin credentials is not good, but onece again that's out of scope.

```
% kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
% kubectl -n argocd port-forward svc/argocd-server 8080:443
```

# 4. GitOps

Now let's create Argo CD application set automated with a path in the repo.

First let's add files in `gitops/` and run:

```
% kubectl apply -f gitops/bootstrap/root-app.yaml
```

At localhost:8080, we can see the `root` application.
Now when we add any directory in format `tenant-*` with a `config.yaml` file,
it will automatically create a namespace and a SA, deploy a jupyterlab server, add a new bucket to main monio.

```
[root-app.yaml] (Applied manually)
      │
      ├──► Creates AppProject ("tenants")
      └──► Creates ApplicationSet ("tenants")
                │
                ├──► Scans Git for tenants/*/config.yaml
                │
                └──► Generates Application: tenant-a
                          │
                          ├──► Pulls Helm Chart from: charts/tenant
                          ├──► Applies Values from: tenants/tenant-a/config.yaml
                          └──► Create and deploy to Namespace: tenant-a
```

### 3.4 Secrets

Now that we have working automated tenants, they need secrets.
I will also use the addon `reflector` to automatically add secrets to tenants.

#### Docker Registry

First let's add docker registry credentials so it can pull the built jupyter image.

```
kubectl create secret docker-registry registry-credentials \
  --docker-server=registry.bouvais.lu \
  --docker-username="" \
  --docker-password=""
  --namespace=argocd
```

And to automatically make them available to tenant namespaces using reflector.
It will propagate `registry-credentials` secret to namespace in format `tenant-something`.

```
kubectl annotate secret registry-credentials -n argocd --overwrite \
  reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"
```

From here we can access the notebook with:

```
kubectl port-forward -n tenant-a deployment/jupyter 8888:8888
```

For this demo, I will stop here. Meaning accessing the notebook using a manual port forwarding.
But in reality, this would need a route, automated forward, CA, ect. But that's out of scope again.

#### Minio Admin

Let's add Minio Admin credentials as a secret too.
Similarly propagating them to tenant namespaces.

```
kubectl create secret generic minio-admin-credentials \
  --from-literal=MINIO_ROOT_USER=something \
  --from-literal=MINIO_ROOT_PASSWORD=something \
  -n argocd

kubectl annotate secret minio-admin-credentials -n argocd --overwrite \
  reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"
```

#### Minio User

The tenant chart also automatically create the 2 new buckets and a random credential saved only in the 
tenant namespace that will use it.

```
% kubectl get events -n tenant-a --field-selector reason=Completed
LAST SEEN   TYPE     REASON      OBJECT                     MESSAGE
7m6s        Normal   Completed   job/tenant-a-minio-setup   Job completed
% kubectl get secrets -n tenant-a
NAME                      TYPE                             DATA   AGE
s3-credentials            Opaque                           5      90s
% kubectl get secret s3-credentials -n tenant-a -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode
echo ""
xXjvsoRXaEI1GtvVfUrMZOkR
```

# Use Jupyter

Now we should have everything to use the notebook.

Let's go to localhost:8888 and run `images/jupyter/test_script.py`.

```
SUCCESS: List items in ref bucket: 1 hello tenant-a
SUCCESS: Write to reference bucket blocked: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.
SUCCESS: Read back from work bucket: hello tenant
SUCCESS: Access to tenant-b blocked: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied.
```

We are succesfully:
- Reading but not writing from bucket ref.
- Read and write in work bucket
- Cant read nor write in bucket tenant-b

# Docker build

Now let's automate docker images.
I will go with a simple Action. Gitea has Action like Github.

It is trigger only on tag `*.*.*`. It then build the base and then jupyter image and push it to the registry
using the tag. Jupyter image use the just previously build base.

TODO:
une NetworkPolicy limite les communications du notebook aux services nécessaires ;
Une pipeline doit :
 • valider les fichiers de configuration ;
 • construire les images dans le bon ordre ;
 • exécuter au moins un test ;
 • publier les images dans un registre ou simuler clairement cette étape ;
 • mettre à jour la version utilisée par le déploiement.

# Configs

I kept configs minimal, but in real here a list of things that could be added:
- A storage limit for Work bucket
- A GPU option for the Jupyter
-