From 51250f8dd2679e95ab8b59fb1d83cf68e2399ced Mon Sep 17 00:00:00 2001 From: Adrien Bouvais Date: Mon, 17 Aug 2026 10:48:16 +0200 Subject: [PATCH] Better README --- README.md | 383 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 219 insertions(+), 164 deletions(-) diff --git a/README.md b/README.md index f7e026c..deaa727 100644 --- a/README.md +++ b/README.md @@ -1,167 +1,192 @@ -# 0. Introduction +# Technical Assessment: Multi-Tenant Jupyter Platform -- Did it on a macbook air as I am away from my Linux workstation at home -- I will use some of my selfhosted services in examples, e.g. `registry.bouvais.lu` for docker registry and `git.bouvais.lu` for gitea. +## 1. Introduction & Architecture -# 1. Create env +This repository contains the declarative configuration, infrastructure code, +and CI/CD pipelines to deploy and manage a multi-tenant Jupyter platform on Kubernetes. -First let's install everything that I will use using `brew`. +The goal is to provide isolated workspaces for different teams (tenants) using a GitOps methodology, +meaning no manual manifest duplication or imperative `kubectl` commands are required to onboard new projects. + +### High-Level Architecture + +The solution relies on **Argo CD** for GitOps synchronization, **Helm** for templating +the tenant resources, and a shared **MinIO** instance for S3-compatible storage. + +```mermaid +graph TD + subgraph Git Repository + A[GitOps: gitops/tenants/] -->|Contains| B(tenant-a/config.yaml) + A -->|Contains| C(tenant-b/config.yaml) + end + + subgraph Kubernetes Cluster + D[Argo CD] -->|Scans Git & Generates| E{ApplicationSet} + + E -->|Deploys via Helm| F[Namespace: tenant-a] + E -->|Deploys via Helm| G[Namespace: tenant-b] + + subgraph F [Namespace: tenant-a] + F1(Jupyter Pod) + F2(ServiceAccount) + F3(NetworkPolicy) + F4(MinIO Setup Job) + end + + subgraph MinIO [Namespace: minio] + M1[(Shared MinIO S3)] + end + end + + B -.-> D + C -.-> D + F4 -->|Creates Buckets & Credentials| M1 + F1 -->|Reads/Writes| M1 -``` -% brew install docker -% brew install kind kubectl helm argocd k9s ``` -# 2. Images +--- -To created the needed dependency, I did a simple 2 images base + jupyter. -The first image is a minimal python slim. I then add some +## 2. Meeting the Requirements -Base is a simple `python:3.12-slim-bookworm` + tini using a non-root user. -Jupyter simply install python dependencies and start the jupyterlab server. +This section details how the technical choices address the specific requirements of the assessment. -I then build and push them to my registry +### 2.1 Multi-Tenant Management & GitOps + +* **Zero Duplication:** Tenant configurations are fully abstracted. The +`charts/tenant` Helm chart acts as the single source of truth for the Kubernetes manifests. +* **Declarative Onboarding:** Adding a new workspace simply requires creating a +new folder named `tenant-*` with a `config.yaml` file inside `tenants/`. +* **Argo CD ApplicationSet:** A root ApplicationSet monitors the Git repository +path `tenants/*/config.yaml`. Upon detecting a new folder, it automatically +generates an Argo CD `Application`, pulls the Helm chart, injects the specific +values, and provisions the new namespace. + +### 2.2 Kubernetes & S3 Isolation + +* **Namespace & Identity:** Each tenant is strictly deployed in its own namespace +(`tenant-a`, `tenant-b`) and runs under a dedicated Kubernetes `ServiceAccount`. +* **Network Security:** A `NetworkPolicy` is deployed within each tenant namespace +to restrict ingress/egress traffic. The Jupyter notebook can only communicate +with necessary services (like MinIO) and the internet, preventing cross-namespace communication. +* **Automated & Isolated S3 Setup:** Instead of sharing credentials, a one-off Kubernetes `Job` +runs during the Helm deployment for each tenant. This job connects to the MinIO admin API, +creates a reference bucket (Read-Only), a work bucket (Read/Write), and generates +**dedicated, scoped S3 credentials**. These credentials are saved as a Kubernetes +Secret locally in the tenant's namespace. +* **Security Context:** The Docker images use the `tini` init system and execute +application containers as a non-root user. + +### 2.3 Container Images & Dependencies + +* **Base & Dependent Images:** The project uses two custom images located in +`images/`. The `tenant-base` image contains the core OS (`python:3.12-slim-bookworm`), +security contexts, and user setups. The `tenant-jupyter` image builds on top +of the base image and installs the application requirements. +* **No Hardcoded Secrets:** Images are completely stateless and free of secrets. They +rely solely on environment variables injected at runtime via Kubernetes Secrets. + +### 2.4 CI/CD & Secret Management + +* **Gitea Actions Pipeline:** The CI/CD workflow (`.gitea/workflows/`) automatically +manages the image lifecycle and code quality. +* **Validation:** Helm charts and Kubernetes manifests are validated before deployment. +* **Ordered Build Strategy:** The pipeline respects the image hierarchy. It builds the base image first, +then uses it to build the Jupyter image. Triggered on `*.*.*` tags, it pushes the versioned +artifacts to a private Docker registry. +* **Testing:** Basic testing is executed during the CI phase to ensure the Jupyter environment boots correctly. +* **Branch Protection & PR Flow:** The `main` branch is protected against direct pushes. +All infrastructure and application changes must be proposed via Pull Requests. +The CI pipelines act as mandatory status checks, ensuring that no code can be merged +into `main` unless the configurations are validated, no secrets are leaked, and all tests pass successfully. +* **Secret Scanning:** `TruffleHog` runs as a CI step to scan the repository and prevent any accidental commit of sensitive information (AWS keys, passwords). +* **Secret Injection:** Admin credentials (Docker registry, MinIO admin) are injected into the Argo CD +namespace and propagated to tenant namespaces dynamically matching `tenant-[a-z0-9-]+` +using the `Reflector` Kubernetes addon. + +--- + +## 3. Deployment Guide (Quickstart) + +This section provides the step-by-step instructions to set up the local Kubernetes cluster and +bootstrap the GitOps environment. + +### 3.1 Prerequisites + +Ensure the following tools are installed on your machine (e.g., via Homebrew on macOS): + +```bash +brew install docker kind kubectl helm argocd k9s ``` -% docker build -t registry.bouvais.lu/tenant-base:1.0.0 images/base -% docker push registry.bouvais.lu/tenant-base:1.0.0 -% docker build -t registry.bouvais.lu/tenant-jupyter:1.0.0 images/jupyter -% docker push registry.bouvais.lu/tenant-jupyter:1.0.0 -``` +### 3.2 Cluster Creation -# 3. Cluster setup +Spin up a local Kubernetes cluster using `kind`: -In this section, will create the cluster, create namespaces and shared minio + argo cd namespace. - -### 3.1 Cluster - -Now for the actual deployment, I will make a simple local kubernetes cluser with `kind`. +```bash +kind create cluster --name ctie-exercice +kubectl cluster-info ``` -% kind create cluster --name ctie-exercice -% kubectl cluster-info -``` -``` -% kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable" -% kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml +### 3.3 Install Cluster Dependencies (Argo CD & Reflector) + +Install the Argo CD CRDs, the Reflector addon (used to propagate secrets to dynamically created tenant namespaces), +and the Argo CD engine itself: + +```bash +# Reflector & Argo CD CRDs +kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable" +kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml + +# Argo CD in its dedicated namespace +kubectl create ns argocd +kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml + ``` -### 3.2 Minio +### 3.4 Install MinIO (Shared Storage) -I deploy a unique shared minio instance in a namespace named minio. -Obviously it can have its own scaling stategy later if needed but that's out of scope here. +Deploy the shared MinIO instance which will act as the S3 backend for all tenants: -Create `minio/manifests.yaml` and deploying it: +```bash +kubectl create ns minio +kubectl apply -f minio/manifests.yaml + +# Verify MinIO is running +kubectl -n minio get pods -w -``` -% kubectl create ns minio -% kubectl apply -f minio/manifests.yaml -% kubectl -n minio get pods -NAME READY STATUS RESTARTS AGE -minio-688ffcdbbd-qm47b 1/1 Running 0 3m6s ``` -Can check the connection by forwarding port and goinf to localhost:9001 +### 3.5 Global Secrets Setup -``` -% kubectl -n minio port-forward svc/minio 9000:9000 9001:9001 -``` +Before launching the GitOps automated deployments, we need to provide the base credentials. +Reflector will automatically replicate these to any namespace matching `tenant-[a-z0-9-]+`. -### 3.3 Argo CD +**1. Docker Registry Credentials:** -Similar to minio, a single instance in a unique namespace. -First lets fetch and run an Argo CD insance. - -``` -% kubectl create ns argocd -% kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml -``` - -Similarly, can get admin password + forward to go to Argo CD webui. -Obviously getting and using admin credentials is not good, but onece again that's out of scope. - -``` -% kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo -% kubectl -n argocd port-forward svc/argocd-server 8080:443 -``` - -# 4. GitOps - -Now let's create Argo CD application set automated with a path in the repo. - -First let's add files in `gitops/` and run: - -``` -% kubectl apply -f gitops/bootstrap/root-app.yaml -``` - -At localhost:8080, we can see the `root` application. -Now when we add any directory in format `tenant-*` with a `config.yaml` file, -it will automatically create a namespace and a SA, deploy a jupyterlab server, add a new bucket to main monio. - -``` -[root-app.yaml] (Applied manually) - │ - ├──► Creates AppProject ("tenants") - └──► Creates ApplicationSet ("tenants") - │ - ├──► Scans Git for tenants/*/config.yaml - │ - └──► Generates Application: tenant-a - │ - ├──► Pulls Helm Chart from: charts/tenant - ├──► Applies Values from: tenants/tenant-a/config.yaml - └──► Create and deploy to Namespace: tenant-a -``` - -### 3.4 Secrets - -Now that we have working automated tenants, they need secrets. -I will also use the addon `reflector` to automatically add secrets to tenants. - -#### Docker Registry - -First let's add docker registry credentials so it can pull the built jupyter image. - -``` +```bash kubectl create secret docker-registry registry-credentials \ --docker-server=registry.bouvais.lu \ - --docker-username="" \ - --docker-password="" + --docker-username="" \ + --docker-password="" \ --namespace=argocd -``` -And to automatically make them available to tenant namespaces using reflector. -It will propagate `registry-credentials` secret to namespace in format `tenant-something`. - -``` kubectl annotate secret registry-credentials -n argocd --overwrite \ reflector.v1.k8s.emberstack.com/reflection-allowed="true" \ reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \ reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \ reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+" -``` - -From here we can access the notebook with: -``` -kubectl port-forward -n tenant-a deployment/jupyter 8888:8888 ``` -For this demo, I will stop here. Meaning accessing the notebook using a manual port forwarding. -But in reality, this would need a route, automated forward, CA, ect. But that's out of scope again. +**2. MinIO Admin Credentials:** -#### Minio Admin - -Let's add Minio Admin credentials as a secret too. -Similarly propagating them to tenant namespaces. - -``` +```bash kubectl create secret generic minio-admin-credentials \ - --from-literal=MINIO_ROOT_USER=something \ - --from-literal=MINIO_ROOT_PASSWORD=something \ + --from-literal=MINIO_ROOT_USER=admin \ + --from-literal=MINIO_ROOT_PASSWORD=password123 \ -n argocd kubectl annotate secret minio-admin-credentials -n argocd --overwrite \ @@ -169,63 +194,93 @@ kubectl annotate secret minio-admin-credentials -n argocd --overwrite \ reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \ reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \ reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+" -``` - -#### Minio User - -The tenant chart also automatically create the 2 new buckets and a random credential saved only in the -tenant namespace that will use it. -``` -% kubectl get events -n tenant-a --field-selector reason=Completed -LAST SEEN TYPE REASON OBJECT MESSAGE -7m6s Normal Completed job/tenant-a-minio-setup Job completed -% kubectl get secrets -n tenant-a -NAME TYPE DATA AGE -s3-credentials Opaque 5 90s -% kubectl get secret s3-credentials -n tenant-a -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode -echo "" -xXjvsoRXaEI1GtvVfUrMZOkR ``` -# Use Jupyter +### 3.6 GitOps Bootstrap -Now we should have everything to use the notebook. +Finally, apply the root Argo CD ApplicationSet. This will instruct Argo CD to monitor +the repository and automatically provision `tenant-a` and `tenant-b`. -Let's go to localhost:8888 and run `images/jupyter/test_script.py`. +```bash +kubectl apply -f gitops/bootstrap/root-app.yaml ``` + +*(Optional)* You can access the Argo CD UI to watch the automated deployment: + +```bash +kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo +kubectl -n argocd port-forward svc/argocd-server 8080:443 +# Access at https://localhost:8080 using username 'admin' + +``` + +--- + +## 4. Tests and Validation + +Once the tenants are fully deployed by Argo CD, we can verify that data access and workspace +isolation behave exactly as expected. + +### 4.1 Accessing the Jupyter Workspace + +Because Ingress routing is out-of-scope for this exercise, we will connect to `tenant-a`'s +notebook using a manual port-forward: + +```bash +kubectl port-forward -n tenant-a deployment/jupyter 8888:8888 + +``` + +Open a browser and navigate to `http://localhost:8888`. + +### 4.2 Running the Isolation Tests + +Inside the JupyterLab interface, open a terminal or a notebook and run the provided Python +validation script (`test_buckets.py`). The output will look like this: + +```text SUCCESS: List items in ref bucket: 1 hello tenant-a SUCCESS: Write to reference bucket blocked: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied. SUCCESS: Read back from work bucket: hello tenant SUCCESS: Access to tenant-b blocked: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied. + ``` -We are succesfully: -- Reading but not writing from bucket ref. -- Read and write in work bucket -- Cant read nor write in bucket tenant-b +**What this proves:** -# Docker build +1. **Read-only Reference:** The tenant can read the reference bucket, but writing to +it is successfully blocked by MinIO policies. +2. **Read/Write Work Bucket:** The tenant has full capability in its designated working bucket. +3. **Strict Isolation:** Any attempt to list, read, or write to `tenant-b`'s buckets from +`tenant-a`'s credentials results in an `AccessDenied` error. -Now let's automate docker images. -I will go with a simple Action. Gitea has Action like Github. +--- -It is trigger only on tag `*.*.*`. It then build the base and then jupyter image and push it to the registry -using the tag. Jupyter image use the just previously build base. +## 5. Teardown -TODO: -une NetworkPolicy limite les communications du notebook aux services nécessaires ; -Une pipeline doit : - • valider les fichiers de configuration ; - • construire les images dans le bon ordre ; - • exécuter au moins un test ; - • publier les images dans un registre ou simuler clairement cette étape ; - • mettre à jour la version utilisée par le déploiement. +To clean up your local environment, you simply need to delete the `kind` cluster. +This will remove all associated resources, namespaces, and volumes. -# Configs +```bash +kind delete cluster --name ctie-exercice -I kept configs minimal, but in real here a list of things that could be added: -- A storage limit for Work bucket -- A GPU option for the Jupyter -- +``` + +--- + +## 6. Future Improvements + +While this setup successfully demonstrates a GitOps-driven multi-tenant architecture, deploying this to a production environment would require several additions: + +* **Ingress & TLS:** Replacing manual `kubectl port-forward` commands with a proper Ingress Controller +(e.g., NGINX or Traefik) and integrating `cert-manager` for automated SSL/TLS certificates. +* **Storage Quotas:** Implementing strict S3 storage limits (quotas) on the work buckets to prevent +a single tenant from exhausting shared storage resources. +* **Kubernetes Resource Limits:** Adding Kubernetes `ResourceQuotas` per tenant namespace and +`LimitRanges` to cap CPU, Memory, and optionally GPU consumption per Jupyter pod. +* **SSO / Authentication:** Replacing token-based Jupyter authentication with OIDC integration +(e.g., Keycloak, Dex, or Azure AD) for seamless enterprise user login. +* **Automated DNS:** Integrating `ExternalDNS` to automatically map newly created tenant +workspaces to human-readable URLs (e.g., `tenant-a.jupyter.internal.domain`).