Better README
This commit is contained in:
Adrien Bouvais
parent
2f29eaa8b5
commit
51250f8dd2
383
README.md
383
README.md
@ -1,167 +1,192 @@
|
||||
# 0. Introduction
|
||||
# Technical Assessment: Multi-Tenant Jupyter Platform
|
||||
|
||||
- Did it on a macbook air as I am away from my Linux workstation at home
|
||||
- I will use some of my selfhosted services in examples, e.g. `registry.bouvais.lu` for docker registry and `git.bouvais.lu` for gitea.
|
||||
## 1. Introduction & Architecture
|
||||
|
||||
# 1. Create env
|
||||
This repository contains the declarative configuration, infrastructure code,
|
||||
and CI/CD pipelines to deploy and manage a multi-tenant Jupyter platform on Kubernetes.
|
||||
|
||||
First let's install everything that I will use using `brew`.
|
||||
The goal is to provide isolated workspaces for different teams (tenants) using a GitOps methodology,
|
||||
meaning no manual manifest duplication or imperative `kubectl` commands are required to onboard new projects.
|
||||
|
||||
### High-Level Architecture
|
||||
|
||||
The solution relies on **Argo CD** for GitOps synchronization, **Helm** for templating
|
||||
the tenant resources, and a shared **MinIO** instance for S3-compatible storage.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Git Repository
|
||||
A[GitOps: gitops/tenants/] -->|Contains| B(tenant-a/config.yaml)
|
||||
A -->|Contains| C(tenant-b/config.yaml)
|
||||
end
|
||||
|
||||
subgraph Kubernetes Cluster
|
||||
D[Argo CD] -->|Scans Git & Generates| E{ApplicationSet}
|
||||
|
||||
E -->|Deploys via Helm| F[Namespace: tenant-a]
|
||||
E -->|Deploys via Helm| G[Namespace: tenant-b]
|
||||
|
||||
subgraph F [Namespace: tenant-a]
|
||||
F1(Jupyter Pod)
|
||||
F2(ServiceAccount)
|
||||
F3(NetworkPolicy)
|
||||
F4(MinIO Setup Job)
|
||||
end
|
||||
|
||||
subgraph MinIO [Namespace: minio]
|
||||
M1[(Shared MinIO S3)]
|
||||
end
|
||||
end
|
||||
|
||||
B -.-> D
|
||||
C -.-> D
|
||||
F4 -->|Creates Buckets & Credentials| M1
|
||||
F1 -->|Reads/Writes| M1
|
||||
|
||||
```
|
||||
% brew install docker
|
||||
% brew install kind kubectl helm argocd k9s
|
||||
```
|
||||
|
||||
# 2. Images
|
||||
---
|
||||
|
||||
To created the needed dependency, I did a simple 2 images base + jupyter.
|
||||
The first image is a minimal python slim. I then add some
|
||||
## 2. Meeting the Requirements
|
||||
|
||||
Base is a simple `python:3.12-slim-bookworm` + tini using a non-root user.
|
||||
Jupyter simply install python dependencies and start the jupyterlab server.
|
||||
This section details how the technical choices address the specific requirements of the assessment.
|
||||
|
||||
I then build and push them to my registry
|
||||
### 2.1 Multi-Tenant Management & GitOps
|
||||
|
||||
* **Zero Duplication:** Tenant configurations are fully abstracted. The
|
||||
`charts/tenant` Helm chart acts as the single source of truth for the Kubernetes manifests.
|
||||
* **Declarative Onboarding:** Adding a new workspace simply requires creating a
|
||||
new folder named `tenant-*` with a `config.yaml` file inside `tenants/`.
|
||||
* **Argo CD ApplicationSet:** A root ApplicationSet monitors the Git repository
|
||||
path `tenants/*/config.yaml`. Upon detecting a new folder, it automatically
|
||||
generates an Argo CD `Application`, pulls the Helm chart, injects the specific
|
||||
values, and provisions the new namespace.
|
||||
|
||||
### 2.2 Kubernetes & S3 Isolation
|
||||
|
||||
* **Namespace & Identity:** Each tenant is strictly deployed in its own namespace
|
||||
(`tenant-a`, `tenant-b`) and runs under a dedicated Kubernetes `ServiceAccount`.
|
||||
* **Network Security:** A `NetworkPolicy` is deployed within each tenant namespace
|
||||
to restrict ingress/egress traffic. The Jupyter notebook can only communicate
|
||||
with necessary services (like MinIO) and the internet, preventing cross-namespace communication.
|
||||
* **Automated & Isolated S3 Setup:** Instead of sharing credentials, a one-off Kubernetes `Job`
|
||||
runs during the Helm deployment for each tenant. This job connects to the MinIO admin API,
|
||||
creates a reference bucket (Read-Only), a work bucket (Read/Write), and generates
|
||||
**dedicated, scoped S3 credentials**. These credentials are saved as a Kubernetes
|
||||
Secret locally in the tenant's namespace.
|
||||
* **Security Context:** The Docker images use the `tini` init system and execute
|
||||
application containers as a non-root user.
|
||||
|
||||
### 2.3 Container Images & Dependencies
|
||||
|
||||
* **Base & Dependent Images:** The project uses two custom images located in
|
||||
`images/`. The `tenant-base` image contains the core OS (`python:3.12-slim-bookworm`),
|
||||
security contexts, and user setups. The `tenant-jupyter` image builds on top
|
||||
of the base image and installs the application requirements.
|
||||
* **No Hardcoded Secrets:** Images are completely stateless and free of secrets. They
|
||||
rely solely on environment variables injected at runtime via Kubernetes Secrets.
|
||||
|
||||
### 2.4 CI/CD & Secret Management
|
||||
|
||||
* **Gitea Actions Pipeline:** The CI/CD workflow (`.gitea/workflows/`) automatically
|
||||
manages the image lifecycle and code quality.
|
||||
* **Validation:** Helm charts and Kubernetes manifests are validated before deployment.
|
||||
* **Ordered Build Strategy:** The pipeline respects the image hierarchy. It builds the base image first,
|
||||
then uses it to build the Jupyter image. Triggered on `*.*.*` tags, it pushes the versioned
|
||||
artifacts to a private Docker registry.
|
||||
* **Testing:** Basic testing is executed during the CI phase to ensure the Jupyter environment boots correctly.
|
||||
* **Branch Protection & PR Flow:** The `main` branch is protected against direct pushes.
|
||||
All infrastructure and application changes must be proposed via Pull Requests.
|
||||
The CI pipelines act as mandatory status checks, ensuring that no code can be merged
|
||||
into `main` unless the configurations are validated, no secrets are leaked, and all tests pass successfully.
|
||||
* **Secret Scanning:** `TruffleHog` runs as a CI step to scan the repository and prevent any accidental commit of sensitive information (AWS keys, passwords).
|
||||
* **Secret Injection:** Admin credentials (Docker registry, MinIO admin) are injected into the Argo CD
|
||||
namespace and propagated to tenant namespaces dynamically matching `tenant-[a-z0-9-]+`
|
||||
using the `Reflector` Kubernetes addon.
|
||||
|
||||
---
|
||||
|
||||
## 3. Deployment Guide (Quickstart)
|
||||
|
||||
This section provides the step-by-step instructions to set up the local Kubernetes cluster and
|
||||
bootstrap the GitOps environment.
|
||||
|
||||
### 3.1 Prerequisites
|
||||
|
||||
Ensure the following tools are installed on your machine (e.g., via Homebrew on macOS):
|
||||
|
||||
```bash
|
||||
brew install docker kind kubectl helm argocd k9s
|
||||
|
||||
```
|
||||
% docker build -t registry.bouvais.lu/tenant-base:1.0.0 images/base
|
||||
% docker push registry.bouvais.lu/tenant-base:1.0.0
|
||||
|
||||
% docker build -t registry.bouvais.lu/tenant-jupyter:1.0.0 images/jupyter
|
||||
% docker push registry.bouvais.lu/tenant-jupyter:1.0.0
|
||||
```
|
||||
### 3.2 Cluster Creation
|
||||
|
||||
# 3. Cluster setup
|
||||
Spin up a local Kubernetes cluster using `kind`:
|
||||
|
||||
In this section, will create the cluster, create namespaces and shared minio + argo cd namespace.
|
||||
|
||||
### 3.1 Cluster
|
||||
|
||||
Now for the actual deployment, I will make a simple local kubernetes cluser with `kind`.
|
||||
```bash
|
||||
kind create cluster --name ctie-exercice
|
||||
kubectl cluster-info
|
||||
|
||||
```
|
||||
% kind create cluster --name ctie-exercice
|
||||
% kubectl cluster-info
|
||||
```
|
||||
|
||||
```
|
||||
% kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable"
|
||||
% kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml
|
||||
### 3.3 Install Cluster Dependencies (Argo CD & Reflector)
|
||||
|
||||
Install the Argo CD CRDs, the Reflector addon (used to propagate secrets to dynamically created tenant namespaces),
|
||||
and the Argo CD engine itself:
|
||||
|
||||
```bash
|
||||
# Reflector & Argo CD CRDs
|
||||
kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable"
|
||||
kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml
|
||||
|
||||
# Argo CD in its dedicated namespace
|
||||
kubectl create ns argocd
|
||||
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
|
||||
|
||||
```
|
||||
|
||||
### 3.2 Minio
|
||||
### 3.4 Install MinIO (Shared Storage)
|
||||
|
||||
I deploy a unique shared minio instance in a namespace named minio.
|
||||
Obviously it can have its own scaling stategy later if needed but that's out of scope here.
|
||||
Deploy the shared MinIO instance which will act as the S3 backend for all tenants:
|
||||
|
||||
Create `minio/manifests.yaml` and deploying it:
|
||||
```bash
|
||||
kubectl create ns minio
|
||||
kubectl apply -f minio/manifests.yaml
|
||||
|
||||
# Verify MinIO is running
|
||||
kubectl -n minio get pods -w
|
||||
|
||||
```
|
||||
% kubectl create ns minio
|
||||
% kubectl apply -f minio/manifests.yaml
|
||||
% kubectl -n minio get pods
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
minio-688ffcdbbd-qm47b 1/1 Running 0 3m6s
|
||||
```
|
||||
|
||||
Can check the connection by forwarding port and goinf to localhost:9001
|
||||
### 3.5 Global Secrets Setup
|
||||
|
||||
```
|
||||
% kubectl -n minio port-forward svc/minio 9000:9000 9001:9001
|
||||
```
|
||||
Before launching the GitOps automated deployments, we need to provide the base credentials.
|
||||
Reflector will automatically replicate these to any namespace matching `tenant-[a-z0-9-]+`.
|
||||
|
||||
### 3.3 Argo CD
|
||||
**1. Docker Registry Credentials:**
|
||||
|
||||
Similar to minio, a single instance in a unique namespace.
|
||||
First lets fetch and run an Argo CD insance.
|
||||
|
||||
```
|
||||
% kubectl create ns argocd
|
||||
% kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
|
||||
```
|
||||
|
||||
Similarly, can get admin password + forward to go to Argo CD webui.
|
||||
Obviously getting and using admin credentials is not good, but onece again that's out of scope.
|
||||
|
||||
```
|
||||
% kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
|
||||
% kubectl -n argocd port-forward svc/argocd-server 8080:443
|
||||
```
|
||||
|
||||
# 4. GitOps
|
||||
|
||||
Now let's create Argo CD application set automated with a path in the repo.
|
||||
|
||||
First let's add files in `gitops/` and run:
|
||||
|
||||
```
|
||||
% kubectl apply -f gitops/bootstrap/root-app.yaml
|
||||
```
|
||||
|
||||
At localhost:8080, we can see the `root` application.
|
||||
Now when we add any directory in format `tenant-*` with a `config.yaml` file,
|
||||
it will automatically create a namespace and a SA, deploy a jupyterlab server, add a new bucket to main monio.
|
||||
|
||||
```
|
||||
[root-app.yaml] (Applied manually)
|
||||
│
|
||||
├──► Creates AppProject ("tenants")
|
||||
└──► Creates ApplicationSet ("tenants")
|
||||
│
|
||||
├──► Scans Git for tenants/*/config.yaml
|
||||
│
|
||||
└──► Generates Application: tenant-a
|
||||
│
|
||||
├──► Pulls Helm Chart from: charts/tenant
|
||||
├──► Applies Values from: tenants/tenant-a/config.yaml
|
||||
└──► Create and deploy to Namespace: tenant-a
|
||||
```
|
||||
|
||||
### 3.4 Secrets
|
||||
|
||||
Now that we have working automated tenants, they need secrets.
|
||||
I will also use the addon `reflector` to automatically add secrets to tenants.
|
||||
|
||||
#### Docker Registry
|
||||
|
||||
First let's add docker registry credentials so it can pull the built jupyter image.
|
||||
|
||||
```
|
||||
```bash
|
||||
kubectl create secret docker-registry registry-credentials \
|
||||
--docker-server=registry.bouvais.lu \
|
||||
--docker-username="" \
|
||||
--docker-password=""
|
||||
--docker-username="<YOUR_USER>" \
|
||||
--docker-password="<YOUR_TOKEN>" \
|
||||
--namespace=argocd
|
||||
```
|
||||
|
||||
And to automatically make them available to tenant namespaces using reflector.
|
||||
It will propagate `registry-credentials` secret to namespace in format `tenant-something`.
|
||||
|
||||
```
|
||||
kubectl annotate secret registry-credentials -n argocd --overwrite \
|
||||
reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
|
||||
reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
|
||||
reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
|
||||
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"
|
||||
```
|
||||
|
||||
From here we can access the notebook with:
|
||||
|
||||
```
|
||||
kubectl port-forward -n tenant-a deployment/jupyter 8888:8888
|
||||
```
|
||||
|
||||
For this demo, I will stop here. Meaning accessing the notebook using a manual port forwarding.
|
||||
But in reality, this would need a route, automated forward, CA, ect. But that's out of scope again.
|
||||
**2. MinIO Admin Credentials:**
|
||||
|
||||
#### Minio Admin
|
||||
|
||||
Let's add Minio Admin credentials as a secret too.
|
||||
Similarly propagating them to tenant namespaces.
|
||||
|
||||
```
|
||||
```bash
|
||||
kubectl create secret generic minio-admin-credentials \
|
||||
--from-literal=MINIO_ROOT_USER=something \
|
||||
--from-literal=MINIO_ROOT_PASSWORD=something \
|
||||
--from-literal=MINIO_ROOT_USER=admin \
|
||||
--from-literal=MINIO_ROOT_PASSWORD=password123 \
|
||||
-n argocd
|
||||
|
||||
kubectl annotate secret minio-admin-credentials -n argocd --overwrite \
|
||||
@ -169,63 +194,93 @@ kubectl annotate secret minio-admin-credentials -n argocd --overwrite \
|
||||
reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
|
||||
reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
|
||||
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"
|
||||
```
|
||||
|
||||
#### Minio User
|
||||
|
||||
The tenant chart also automatically create the 2 new buckets and a random credential saved only in the
|
||||
tenant namespace that will use it.
|
||||
|
||||
```
|
||||
% kubectl get events -n tenant-a --field-selector reason=Completed
|
||||
LAST SEEN TYPE REASON OBJECT MESSAGE
|
||||
7m6s Normal Completed job/tenant-a-minio-setup Job completed
|
||||
% kubectl get secrets -n tenant-a
|
||||
NAME TYPE DATA AGE
|
||||
s3-credentials Opaque 5 90s
|
||||
% kubectl get secret s3-credentials -n tenant-a -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode
|
||||
echo ""
|
||||
xXjvsoRXaEI1GtvVfUrMZOkR
|
||||
```
|
||||
|
||||
# Use Jupyter
|
||||
### 3.6 GitOps Bootstrap
|
||||
|
||||
Now we should have everything to use the notebook.
|
||||
Finally, apply the root Argo CD ApplicationSet. This will instruct Argo CD to monitor
|
||||
the repository and automatically provision `tenant-a` and `tenant-b`.
|
||||
|
||||
Let's go to localhost:8888 and run `images/jupyter/test_script.py`.
|
||||
```bash
|
||||
kubectl apply -f gitops/bootstrap/root-app.yaml
|
||||
|
||||
```
|
||||
|
||||
*(Optional)* You can access the Argo CD UI to watch the automated deployment:
|
||||
|
||||
```bash
|
||||
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
|
||||
kubectl -n argocd port-forward svc/argocd-server 8080:443
|
||||
# Access at https://localhost:8080 using username 'admin'
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Tests and Validation
|
||||
|
||||
Once the tenants are fully deployed by Argo CD, we can verify that data access and workspace
|
||||
isolation behave exactly as expected.
|
||||
|
||||
### 4.1 Accessing the Jupyter Workspace
|
||||
|
||||
Because Ingress routing is out-of-scope for this exercise, we will connect to `tenant-a`'s
|
||||
notebook using a manual port-forward:
|
||||
|
||||
```bash
|
||||
kubectl port-forward -n tenant-a deployment/jupyter 8888:8888
|
||||
|
||||
```
|
||||
|
||||
Open a browser and navigate to `http://localhost:8888`.
|
||||
|
||||
### 4.2 Running the Isolation Tests
|
||||
|
||||
Inside the JupyterLab interface, open a terminal or a notebook and run the provided Python
|
||||
validation script (`test_buckets.py`). The output will look like this:
|
||||
|
||||
```text
|
||||
SUCCESS: List items in ref bucket: 1 hello tenant-a
|
||||
SUCCESS: Write to reference bucket blocked: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.
|
||||
SUCCESS: Read back from work bucket: hello tenant
|
||||
SUCCESS: Access to tenant-b blocked: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied.
|
||||
|
||||
```
|
||||
|
||||
We are succesfully:
|
||||
- Reading but not writing from bucket ref.
|
||||
- Read and write in work bucket
|
||||
- Cant read nor write in bucket tenant-b
|
||||
**What this proves:**
|
||||
|
||||
# Docker build
|
||||
1. **Read-only Reference:** The tenant can read the reference bucket, but writing to
|
||||
it is successfully blocked by MinIO policies.
|
||||
2. **Read/Write Work Bucket:** The tenant has full capability in its designated working bucket.
|
||||
3. **Strict Isolation:** Any attempt to list, read, or write to `tenant-b`'s buckets from
|
||||
`tenant-a`'s credentials results in an `AccessDenied` error.
|
||||
|
||||
Now let's automate docker images.
|
||||
I will go with a simple Action. Gitea has Action like Github.
|
||||
---
|
||||
|
||||
It is trigger only on tag `*.*.*`. It then build the base and then jupyter image and push it to the registry
|
||||
using the tag. Jupyter image use the just previously build base.
|
||||
## 5. Teardown
|
||||
|
||||
TODO:
|
||||
une NetworkPolicy limite les communications du notebook aux services nécessaires ;
|
||||
Une pipeline doit :
|
||||
• valider les fichiers de configuration ;
|
||||
• construire les images dans le bon ordre ;
|
||||
• exécuter au moins un test ;
|
||||
• publier les images dans un registre ou simuler clairement cette étape ;
|
||||
• mettre à jour la version utilisée par le déploiement.
|
||||
To clean up your local environment, you simply need to delete the `kind` cluster.
|
||||
This will remove all associated resources, namespaces, and volumes.
|
||||
|
||||
# Configs
|
||||
```bash
|
||||
kind delete cluster --name ctie-exercice
|
||||
|
||||
I kept configs minimal, but in real here a list of things that could be added:
|
||||
- A storage limit for Work bucket
|
||||
- A GPU option for the Jupyter
|
||||
-
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Future Improvements
|
||||
|
||||
While this setup successfully demonstrates a GitOps-driven multi-tenant architecture, deploying this to a production environment would require several additions:
|
||||
|
||||
* **Ingress & TLS:** Replacing manual `kubectl port-forward` commands with a proper Ingress Controller
|
||||
(e.g., NGINX or Traefik) and integrating `cert-manager` for automated SSL/TLS certificates.
|
||||
* **Storage Quotas:** Implementing strict S3 storage limits (quotas) on the work buckets to prevent
|
||||
a single tenant from exhausting shared storage resources.
|
||||
* **Kubernetes Resource Limits:** Adding Kubernetes `ResourceQuotas` per tenant namespace and
|
||||
`LimitRanges` to cap CPU, Memory, and optionally GPU consumption per Jupyter pod.
|
||||
* **SSO / Authentication:** Replacing token-based Jupyter authentication with OIDC integration
|
||||
(e.g., Keycloak, Dex, or Azure AD) for seamless enterprise user login.
|
||||
* **Automated DNS:** Integrating `ExternalDNS` to automatically map newly created tenant
|
||||
workspaces to human-readable URLs (e.g., `tenant-a.jupyter.internal.domain`).
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user