ctie-exercice/README.md

# Technical Assessment: Multi-Tenant Jupyter Platform

## 1. Introduction & Architecture

This repository contains the declarative configuration, infrastructure code,
and CI/CD pipelines to deploy and manage a multi-tenant Jupyter platform on Kubernetes.

The goal is to provide isolated workspaces for different teams (tenants) using a GitOps methodology,
meaning no manual manifest duplication or imperative `kubectl` commands are required to onboard new projects.

### High-Level Architecture

The solution relies on **Argo CD** for GitOps synchronization, **Helm** for templating
the tenant resources, and a shared **MinIO** instance for S3-compatible storage.

```mermaid
graph TD
    subgraph Git Repository
        A[GitOps: tenants/] -->|Contains| B(tenant-a/config.yaml)
        A -->|Contains| C(tenant-b/config.yaml)
    end

    subgraph Kubernetes Cluster
        D[Argo CD] -->|Scans Git & Generates| E{ApplicationSet}
        
        E -->|Deploys via Helm| F[Namespace: tenant-a]
        E -->|Deploys via Helm| G[Namespace: tenant-b]

        subgraph argo [Namespace: argocd]
            D(Argo CD)
            E(ApplicationSet)
        end
        
        subgraph F [Namespace: tenant-a]
            F1(Jupyter Pod)
            F2(ServiceAccount)
            F3(NetworkPolicy)
        end
        
        subgraph MinIO [Namespace: minio]
            M1[(Shared MinIO S3)]
            M2(MinIO Setup Job)
        end
    end

    B -.-> D
    C -.-> D
    M2 -->|Creates Buckets & Credentials| M1
    M2 -->|Provisions Scoped Secret| F
    F1 -->|Reads/Writes| M1

```

---

## 2. Requirements

This section details how the technical choices address the specific requirements of the assessment.

### 2.1 Multi-Tenant Management & GitOps

* **Zero Duplication:** Tenant configurations are fully abstracted. The 
`charts/tenant` Helm chart acts as the single source of truth for the Kubernetes manifests.
* **Declarative Onboarding:** Adding a new workspace simply requires creating a
new folder named `tenant-*` with a `config.yaml` file inside `tenants/`.
* **Argo CD ApplicationSet:** A root ApplicationSet monitors the Git repository
path `tenants/*/config.yaml`. Upon detecting a new folder, it automatically
generates an Argo CD `Application`, pulls the Helm chart, injects the specific
values, and provisions the new namespace.
* **Version Promotion:** Image builds and pushes happen automatically on every tag push, but promoting a tenant to a new version is a deliberate action: it requires updating the image tag value in that tenant's config.yaml and merging a PR. This keeps each tenant's rollout independent and auditable, rather than pushing every tenant to the newest image the moment it's built.

### 2.2 Kubernetes & S3 Isolation

* **Namespace & Identity:** Each tenant is strictly deployed in its own namespace
(`tenant-a`, `tenant-b`) and runs under a dedicated Kubernetes `ServiceAccount`.
* **Network Security:** A `NetworkPolicy` is deployed within each tenant namespace
to restrict ingress/egress traffic. The Jupyter notebook can only communicate
with necessary services, preventing cross-namespace communication.
* **Automated & Isolated S3 Setup:** To avoid exposing admin credentials outside the storage infrastructure, a Kubernetes Job runs inside the minio namespace during tenant provisioning. It connects to the MinIO admin API locally, creates a read-only reference bucket and a read/write work bucket, and generates dedicated, tenant-scoped S3 credentials. The Job then writes those credentials directly into the target tenant namespace as a Secret, so only the scoped credentials (never the MinIO admin credentials) ever leave the minio namespace.
* **Security Context:** The Docker images use the `tini` init system and execute
application containers as a non-root user.
* **Shared MinIO Instance:** A single MinIO deployment backs all tenants rather than one instance per tenant. This was an intentional choice: it keeps operational overhead low and leaves the door open for shared reference datasets across tenants. Isolation is enforced at the credential and bucket-policy level regardless of the sared backend, as demonstrated in section 4.2.

### 2.3 Container Images & Dependencies

* **Base & Dependent Images:** The project uses two custom images located in
`images/`. The `tenant/base` image contains the core OS (`python:3.12-slim-bookworm`),
security contexts, and user setups. The `tenant/jupyter` image builds on top
of the base image and installs the application requirements.
* **No Hardcoded Secrets:** Images are completely stateless and free of secrets. They
rely solely on environment variables injected at runtime via Kubernetes Secrets.

### 2.4 CI/CD & Secret Management

* **Gitea Actions Pipeline:** The CI/CD workflow (`.gitea/workflows/`) automatically
manages the image lifecycle and code quality.
* **Validation:** Helm charts and tenant configs are validated before deployment.
* **Ordered Build Strategy:** The pipeline respects the image hierarchy. It builds the base image first,
then uses it to build the Jupyter image. Triggered on `*.*.*` tags, it pushes the versioned
artifacts to a private Docker registry.
* **Testing:** Basic testing is executed during the CI phase to ensure the Jupyter environment boots correctly.
* **Branch Protection & PR Flow:** The `main` branch is protected against direct pushes.
All infrastructure and application changes must be proposed via Pull Requests.
The CI pipelines act as mandatory status checks, ensuring that no code can be merged
into `main` unless the configurations are validated, no secrets are leaked, and all tests pass successfully.
* **Secret Scanning:** `TruffleHog` runs as a CI step to scan the repository and prevent any accidental commit of sensitive information (AWS keys, passwords).
* **Secret Injection:** Docker registry credentials are injected into the argocd namespace and
propagated to tenant namespaces dynamically matching tenant-[a-z0-9-]+ using
the Reflector Kubernetes addon. MinIO admin credentials remain strictly contained
within the minio namespace and are never reflected to tenant environments.

---

## 3. Deployment Guide (Quickstart)

This section provides the step-by-step instructions to set up the local Kubernetes cluster and
bootstrap the GitOps environment.

*Note: You will not be able to use this repo directly as it require credentials for
my selfhosted gitea and docker registry.*

### 3.1 Prerequisites

Ensure the following tools are installed on your machine (e.g., via Homebrew on macOS):

```bash
brew install docker kind kubectl helm argocd k9s

```

### 3.2 Cluster Creation

Spin up a local Kubernetes cluster using `kind`:

```bash
kind create cluster --name ctie-exercice
kubectl cluster-info

```

### 3.3 Install Cluster Dependencies (Argo CD & Reflector)

Install the Argo CD CRDs, the Reflector addon (used to propagate secrets to dynamically created tenant namespaces),
and the Argo CD engine itself:

```bash
# Reflector & Argo CD CRDs
kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable"
kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml

# Argo CD in its dedicated namespace
kubectl create ns argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

```

### 3.4 Install MinIO (Shared Storage)

Deploy the shared MinIO instance which will act as the S3 backend for all tenants:

```bash
kubectl create ns minio
kubectl apply -f minio/manifests.yaml

# Verify MinIO is running
kubectl -n minio get pods -w

```

### 3.5 Global Secrets Setup

Before launching the GitOps automated deployments, we need to provide the base credentials.
Reflector will automatically replicate these to any namespace matching `tenant-[a-z0-9-]+`.

**1. Docker Registry Credentials:**

```bash
kubectl create secret docker-registry registry-credentials \
  --docker-server=registry.bouvais.lu \
  --docker-username="<YOUR_USER>" \
  --docker-password="<YOUR_TOKEN>" \
  --namespace=argocd

kubectl annotate secret registry-credentials -n argocd --overwrite \
  reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"

```

**2. MinIO Admin Credentials:**

```bash
kubectl create secret generic minio-admin-credentials \
  --from-literal=MINIO_ROOT_USER=admin \
  --from-literal=MINIO_ROOT_PASSWORD=password123 \
  -n minio

```

### 3.6 GitOps Bootstrap

Finally, apply the root Argo CD ApplicationSet. This will instruct Argo CD to monitor
the repository and automatically provision `tenant-a` and `tenant-b`.

```bash
kubectl apply -f gitops/bootstrap/root-app.yaml

```

*(Optional)* You can access the Argo CD UI to watch the automated deployment:

```bash
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
kubectl -n argocd port-forward svc/argocd-server 8080:443
# Access at https://localhost:8080 using username 'admin'

```

---

## 4. Tests and Validation

Once the tenants are fully deployed by Argo CD, we can verify that data access and workspace
isolation behave exactly as expected.

### 4.1 Accessing the Jupyter Workspace

Because Ingress routing is out-of-scope for this exercise, we will connect to `tenant-a`'s
notebook using a manual port-forward:

```bash
kubectl port-forward -n tenant-a deployment/jupyter 8888:8888

```

Open a browser and navigate to `http://localhost:8888`.

### 4.2 Running the Isolation Tests

Inside the JupyterLab interface, open a terminal or a notebook and run the provided Python
validation script (`test_buckets.py`). The output will look like this:

```text
SUCCESS: List items in ref bucket: 1 hello tenant-a
SUCCESS: Write to reference bucket blocked: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.
SUCCESS: Read back from work bucket: hello tenant
SUCCESS: Access to tenant-b blocked: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied.

```

**What this proves:**

1. **Read-only Reference:** The tenant can read the reference bucket, but writing to
it is successfully blocked by MinIO policies.
2. **Read/Write Work Bucket:** The tenant has full capability in its designated working bucket.
3. **Strict Isolation:** Any attempt to list, read, or write to `tenant-b`'s buckets from
`tenant-a`'s credentials results in an `AccessDenied` error.

---

## 5. Teardown

To clean up your local environment, you simply need to delete the `kind` cluster.
This will remove all associated resources, namespaces, and volumes.

```bash
kind delete cluster --name ctie-exercice

```

---

## 6. Future Improvements

While this setup successfully demonstrates a GitOps-driven multi-tenant architecture, deploying this to a production environment would require several additions:

* **Ingress & TLS:** Replacing manual `kubectl port-forward` commands with a proper Ingress Controller
(e.g., NGINX or Traefik) and integrating `cert-manager` for automated SSL/TLS certificates.
* **Storage Quotas:** Implementing strict S3 storage limits (quotas) on the work buckets to prevent
a single tenant from exhausting shared storage resources.
* **Kubernetes Resource Limits:** Adding Kubernetes `ResourceQuotas` per tenant namespace and
`LimitRanges` to cap CPU, Memory, and optionally GPU consumption per Jupyter pod.
* **SSO / Authentication:** Replacing token-based Jupyter authentication with OIDC integration
(e.g., Keycloak, Dex, or Azure AD) for seamless enterprise user login.
* **Automated DNS:** Integrating `ExternalDNS` to automatically map newly created tenant
workspaces to human-readable URLs (e.g., `tenant-a.jupyter.internal.domain`).