Better README

2026-08-17 10:48:16 +02:00 · 2026-08-17 10:48:16 +02:00 · 51250f8dd2
commit 51250f8dd2
parent 2f29eaa8b5
1 changed files with 219 additions and 164 deletions
--- a/README.md
+++ b/README.md
@ -1,167 +1,192 @@
-# 0. Introduction
+# Technical Assessment: Multi-Tenant Jupyter Platform
- Did it on a macbook air as I am away from my Linux workstation at home
+## 1. Introduction & Architecture
 - I will use some of my selfhosted services in examples, e.g. `registry.bouvais.lu` for docker registry and `git.bouvais.lu` for gitea.
-# 1. Create env
+This repository contains the declarative configuration, infrastructure code,
 and CI/CD pipelines to deploy and manage a multi-tenant Jupyter platform on Kubernetes.
-First let's install everything that I will use using `brew`.
+The goal is to provide isolated workspaces for different teams (tenants) using a GitOps methodology,
 meaning no manual manifest duplication or imperative `kubectl` commands are required to onboard new projects.
 ### High-Level Architecture
 The solution relies on **Argo CD** for GitOps synchronization, **Helm** for templating
 the tenant resources, and a shared **MinIO** instance for S3-compatible storage.
 ```mermaid
 graph TD
    subgraph Git Repository
        A[GitOps: gitops/tenants/] -->|Contains| B(tenant-a/config.yaml)
        A -->|Contains| C(tenant-b/config.yaml)
    end
    subgraph Kubernetes Cluster
        D[Argo CD] -->|Scans Git & Generates| E{ApplicationSet}
        E -->|Deploys via Helm| F[Namespace: tenant-a]
        E -->|Deploys via Helm| G[Namespace: tenant-b]
        subgraph F [Namespace: tenant-a]
            F1(Jupyter Pod)
            F2(ServiceAccount)
            F3(NetworkPolicy)
            F4(MinIO Setup Job)
        end
        subgraph MinIO [Namespace: minio]
            M1[(Shared MinIO S3)]
        end
    end
    B -.-> D
    C -.-> D
    F4 -->|Creates Buckets & Credentials| M1
    F1 -->|Reads/Writes| M1
 ```
 % brew install docker
 % brew install kind kubectl helm argocd k9s
 ```
-# 2. Images
+---
-To created the needed dependency, I did a simple 2 images base + jupyter.
+## 2. Meeting the Requirements
 The first image is a minimal python slim. I then add some 
-Base is a simple `python:3.12-slim-bookworm` + tini using a non-root user.
+This section details how the technical choices address the specific requirements of the assessment.
 Jupyter simply install python dependencies and start the jupyterlab server.
-I then build and push them to my registry
+### 2.1 Multi-Tenant Management & GitOps
 * **Zero Duplication:** Tenant configurations are fully abstracted. The 
 `charts/tenant` Helm chart acts as the single source of truth for the Kubernetes manifests.
 * **Declarative Onboarding:** Adding a new workspace simply requires creating a
 new folder named `tenant-*` with a `config.yaml` file inside `tenants/`.
 * **Argo CD ApplicationSet:** A root ApplicationSet monitors the Git repository
 path `tenants/*/config.yaml`. Upon detecting a new folder, it automatically
 generates an Argo CD `Application`, pulls the Helm chart, injects the specific
 values, and provisions the new namespace.
 ### 2.2 Kubernetes & S3 Isolation
 * **Namespace & Identity:** Each tenant is strictly deployed in its own namespace
 (`tenant-a`, `tenant-b`) and runs under a dedicated Kubernetes `ServiceAccount`.
 * **Network Security:** A `NetworkPolicy` is deployed within each tenant namespace
 to restrict ingress/egress traffic. The Jupyter notebook can only communicate
 with necessary services (like MinIO) and the internet, preventing cross-namespace communication.
 * **Automated & Isolated S3 Setup:** Instead of sharing credentials, a one-off Kubernetes `Job`
 runs during the Helm deployment for each tenant. This job connects to the MinIO admin API,
 creates a reference bucket (Read-Only), a work bucket (Read/Write), and generates
 **dedicated, scoped S3 credentials**. These credentials are saved as a Kubernetes
 Secret locally in the tenant's namespace.
 * **Security Context:** The Docker images use the `tini` init system and execute
 application containers as a non-root user.
 ### 2.3 Container Images & Dependencies
 * **Base & Dependent Images:** The project uses two custom images located in
 `images/`. The `tenant-base` image contains the core OS (`python:3.12-slim-bookworm`),
 security contexts, and user setups. The `tenant-jupyter` image builds on top
 of the base image and installs the application requirements.
 * **No Hardcoded Secrets:** Images are completely stateless and free of secrets. They
 rely solely on environment variables injected at runtime via Kubernetes Secrets.
 ### 2.4 CI/CD & Secret Management
 * **Gitea Actions Pipeline:** The CI/CD workflow (`.gitea/workflows/`) automatically
 manages the image lifecycle and code quality.
 * **Validation:** Helm charts and Kubernetes manifests are validated before deployment.
 * **Ordered Build Strategy:** The pipeline respects the image hierarchy. It builds the base image first,
 then uses it to build the Jupyter image. Triggered on `*.*.*` tags, it pushes the versioned
 artifacts to a private Docker registry.
 * **Testing:** Basic testing is executed during the CI phase to ensure the Jupyter environment boots correctly.
 * **Branch Protection & PR Flow:** The `main` branch is protected against direct pushes.
 All infrastructure and application changes must be proposed via Pull Requests.
 The CI pipelines act as mandatory status checks, ensuring that no code can be merged
 into `main` unless the configurations are validated, no secrets are leaked, and all tests pass successfully.
 * **Secret Scanning:** `TruffleHog` runs as a CI step to scan the repository and prevent any accidental commit of sensitive information (AWS keys, passwords).
 * **Secret Injection:** Admin credentials (Docker registry, MinIO admin) are injected into the Argo CD
 namespace and propagated to tenant namespaces dynamically matching `tenant-[a-z0-9-]+`
 using the `Reflector` Kubernetes addon.
 ---
 ## 3. Deployment Guide (Quickstart)
 This section provides the step-by-step instructions to set up the local Kubernetes cluster and
 bootstrap the GitOps environment.
 ### 3.1 Prerequisites
 Ensure the following tools are installed on your machine (e.g., via Homebrew on macOS):
 ```bash
 brew install docker kind kubectl helm argocd k9s
 ```
 % docker build -t registry.bouvais.lu/tenant-base:1.0.0 images/base
 % docker push registry.bouvais.lu/tenant-base:1.0.0 
-% docker build -t registry.bouvais.lu/tenant-jupyter:1.0.0 images/jupyter
+### 3.2 Cluster Creation
 % docker push registry.bouvais.lu/tenant-jupyter:1.0.0 
 ```
-# 3. Cluster setup
+Spin up a local Kubernetes cluster using `kind`:
-In this section, will create the cluster, create namespaces and shared minio + argo cd namespace.
+```bash
-
+kind create cluster --name ctie-exercice
-### 3.1 Cluster
+kubectl cluster-info
 Now for the actual deployment, I will make a simple local kubernetes cluser with `kind`.
 ```
 % kind create cluster --name ctie-exercice
 % kubectl cluster-info
 ```
-```
+### 3.3 Install Cluster Dependencies (Argo CD & Reflector)
-% kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable"
+
-% kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml
+Install the Argo CD CRDs, the Reflector addon (used to propagate secrets to dynamically created tenant namespaces),
 and the Argo CD engine itself:
 ```bash
 # Reflector & Argo CD CRDs
 kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable"
 kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml
 # Argo CD in its dedicated namespace
 kubectl create ns argocd
 kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
 ```
-### 3.2 Minio
+### 3.4 Install MinIO (Shared Storage)
-I deploy a unique shared minio instance in a namespace named minio.
+Deploy the shared MinIO instance which will act as the S3 backend for all tenants:
 Obviously it can have its own scaling stategy later if needed but that's out of scope here.
-Create `minio/manifests.yaml` and deploying it:
+```bash
 kubectl create ns minio
 kubectl apply -f minio/manifests.yaml
 # Verify MinIO is running
 kubectl -n minio get pods -w
 ```
 % kubectl create ns minio
 % kubectl apply -f minio/manifests.yaml
 % kubectl -n minio get pods
 NAME                     READY   STATUS    RESTARTS   AGE
 minio-688ffcdbbd-qm47b   1/1     Running   0          3m6s
 ```
-Can check the connection by forwarding port and goinf to localhost:9001
+### 3.5 Global Secrets Setup
-```
+Before launching the GitOps automated deployments, we need to provide the base credentials.
-% kubectl -n minio port-forward svc/minio 9000:9000 9001:9001
+Reflector will automatically replicate these to any namespace matching `tenant-[a-z0-9-]+`.
 ```
-### 3.3 Argo CD
+**1. Docker Registry Credentials:**
-Similar to minio, a single instance in a unique namespace.
+```bash
 First lets fetch and run an Argo CD insance.
 ```
 % kubectl create ns argocd
 % kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
 ```
 Similarly, can get admin password + forward to go to Argo CD webui.
 Obviously getting and using admin credentials is not good, but onece again that's out of scope.
 ```
 % kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
 % kubectl -n argocd port-forward svc/argocd-server 8080:443
 ```
 # 4. GitOps
 Now let's create Argo CD application set automated with a path in the repo.
 First let's add files in `gitops/` and run:
 ```
 % kubectl apply -f gitops/bootstrap/root-app.yaml
 ```
 At localhost:8080, we can see the `root` application.
 Now when we add any directory in format `tenant-*` with a `config.yaml` file,
 it will automatically create a namespace and a SA, deploy a jupyterlab server, add a new bucket to main monio.
 ```
 [root-app.yaml] (Applied manually)
      │
      ├──► Creates AppProject ("tenants")
      └──► Creates ApplicationSet ("tenants")
                │
                ├──► Scans Git for tenants/*/config.yaml
                │
                └──► Generates Application: tenant-a
                          │
                          ├──► Pulls Helm Chart from: charts/tenant
                          ├──► Applies Values from: tenants/tenant-a/config.yaml
                          └──► Create and deploy to Namespace: tenant-a
 ```
 ### 3.4 Secrets
 Now that we have working automated tenants, they need secrets.
 I will also use the addon `reflector` to automatically add secrets to tenants.
 #### Docker Registry
 First let's add docker registry credentials so it can pull the built jupyter image.
 ```
 kubectl create secret docker-registry registry-credentials \
  --docker-server=registry.bouvais.lu \
-  --docker-username="" \
+  --docker-username="<YOUR_USER>" \
-  --docker-password=""
+  --docker-password="<YOUR_TOKEN>" \
  --namespace=argocd
 ```
 And to automatically make them available to tenant namespaces using reflector.
 It will propagate `registry-credentials` secret to namespace in format `tenant-something`.
 ```
 kubectl annotate secret registry-credentials -n argocd --overwrite \
  reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"
 ```
 From here we can access the notebook with:
 ```
 kubectl port-forward -n tenant-a deployment/jupyter 8888:8888
 ```
-For this demo, I will stop here. Meaning accessing the notebook using a manual port forwarding.
+**2. MinIO Admin Credentials:**
 But in reality, this would need a route, automated forward, CA, ect. But that's out of scope again.
-#### Minio Admin
+```bash
 Let's add Minio Admin credentials as a secret too.
 Similarly propagating them to tenant namespaces.
 ```
 kubectl create secret generic minio-admin-credentials \
-  --from-literal=MINIO_ROOT_USER=something \
+  --from-literal=MINIO_ROOT_USER=admin \
-  --from-literal=MINIO_ROOT_PASSWORD=something \
+  --from-literal=MINIO_ROOT_PASSWORD=password123 \
  -n argocd
 kubectl annotate secret minio-admin-credentials -n argocd --overwrite \
@ -169,63 +194,93 @@ kubectl annotate secret minio-admin-credentials -n argocd --overwrite \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"
 ```
 #### Minio User
 The tenant chart also automatically create the 2 new buckets and a random credential saved only in the 
 tenant namespace that will use it.
 ```
 % kubectl get events -n tenant-a --field-selector reason=Completed
 LAST SEEN   TYPE     REASON      OBJECT                     MESSAGE
 7m6s        Normal   Completed   job/tenant-a-minio-setup   Job completed
 % kubectl get secrets -n tenant-a
 NAME                      TYPE                             DATA   AGE
 s3-credentials            Opaque                           5      90s
 % kubectl get secret s3-credentials -n tenant-a -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode
 echo ""
 xXjvsoRXaEI1GtvVfUrMZOkR
 ```
-# Use Jupyter
+### 3.6 GitOps Bootstrap
-Now we should have everything to use the notebook.
+Finally, apply the root Argo CD ApplicationSet. This will instruct Argo CD to monitor
 the repository and automatically provision `tenant-a` and `tenant-b`.
-Let's go to localhost:8888 and run `images/jupyter/test_script.py`.
+```bash
 kubectl apply -f gitops/bootstrap/root-app.yaml
 ```
 *(Optional)* You can access the Argo CD UI to watch the automated deployment:
 ```bash
 kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
 kubectl -n argocd port-forward svc/argocd-server 8080:443
 # Access at https://localhost:8080 using username 'admin'
 ```
 ---
 ## 4. Tests and Validation
 Once the tenants are fully deployed by Argo CD, we can verify that data access and workspace
 isolation behave exactly as expected.
 ### 4.1 Accessing the Jupyter Workspace
 Because Ingress routing is out-of-scope for this exercise, we will connect to `tenant-a`'s
 notebook using a manual port-forward:
 ```bash
 kubectl port-forward -n tenant-a deployment/jupyter 8888:8888
 ```
 Open a browser and navigate to `http://localhost:8888`.
 ### 4.2 Running the Isolation Tests
 Inside the JupyterLab interface, open a terminal or a notebook and run the provided Python
 validation script (`test_buckets.py`). The output will look like this:
 ```text
 SUCCESS: List items in ref bucket: 1 hello tenant-a
 SUCCESS: Write to reference bucket blocked: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.
 SUCCESS: Read back from work bucket: hello tenant
 SUCCESS: Access to tenant-b blocked: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied.
 ```
-We are succesfully:
+**What this proves:**
 - Reading but not writing from bucket ref.
 - Read and write in work bucket
 - Cant read nor write in bucket tenant-b
-# Docker build
+1. **Read-only Reference:** The tenant can read the reference bucket, but writing to
 it is successfully blocked by MinIO policies.
 2. **Read/Write Work Bucket:** The tenant has full capability in its designated working bucket.
 3. **Strict Isolation:** Any attempt to list, read, or write to `tenant-b`'s buckets from
 `tenant-a`'s credentials results in an `AccessDenied` error.
-Now let's automate docker images.
+---
 I will go with a simple Action. Gitea has Action like Github.
-It is trigger only on tag `*.*.*`. It then build the base and then jupyter image and push it to the registry
+## 5. Teardown
 using the tag. Jupyter image use the just previously build base.
-TODO:
+To clean up your local environment, you simply need to delete the `kind` cluster.
-une NetworkPolicy limite les communications du notebook aux services nécessaires ;
+This will remove all associated resources, namespaces, and volumes.
 Une pipeline doit :
 • valider les fichiers de configuration ;
 • construire les images dans le bon ordre ;
 • exécuter au moins un test ;
 • publier les images dans un registre ou simuler clairement cette étape ;
 • mettre à jour la version utilisée par le déploiement.
-# Configs
+```bash
 kind delete cluster --name ctie-exercice
-I kept configs minimal, but in real here a list of things that could be added:
+```
- A storage limit for Work bucket
+
- A GPU option for the Jupyter
+---
- 
+
 ## 6. Future Improvements
 While this setup successfully demonstrates a GitOps-driven multi-tenant architecture, deploying this to a production environment would require several additions:
 * **Ingress & TLS:** Replacing manual `kubectl port-forward` commands with a proper Ingress Controller
 (e.g., NGINX or Traefik) and integrating `cert-manager` for automated SSL/TLS certificates.
 * **Storage Quotas:** Implementing strict S3 storage limits (quotas) on the work buckets to prevent
 a single tenant from exhausting shared storage resources.
 * **Kubernetes Resource Limits:** Adding Kubernetes `ResourceQuotas` per tenant namespace and
 `LimitRanges` to cap CPU, Memory, and optionally GPU consumption per Jupyter pod.
 * **SSO / Authentication:** Replacing token-based Jupyter authentication with OIDC integration
 (e.g., Keycloak, Dex, or Azure AD) for seamless enterprise user login.
 * **Automated DNS:** Integrating `ExternalDNS` to automatically map newly created tenant
 workspaces to human-readable URLs (e.g., `tenant-a.jupyter.internal.domain`).