This repository contains the declarative configuration, infrastructure code, and CI/CD pipelines to deploy and manage a multi-tenant Jupyter platform on Kubernetes.

The goal is to provide isolated workspaces for different teams (tenants) using a GitOps methodology, meaning no manual manifest duplication or imperative kubectl commands are required to onboard new projects.

High-Level Architecture

The solution relies on Argo CD for GitOps synchronization, Helm for templating the tenant resources, and a shared MinIO instance for S3-compatible storage.

graph TD
    subgraph Git Repository
        A[GitOps: tenants/] -->|Contains| B(tenant-a/config.yaml)
        A -->|Contains| C(tenant-b/config.yaml)
    end

    subgraph Kubernetes Cluster
        D[Argo CD] -->|Scans Git & Generates| E{ApplicationSet}
        
        E -->|Deploys via Helm| F[Namespace: tenant-a]
        E -->|Deploys via Helm| G[Namespace: tenant-b]

        subgraph argo [Namespace: argocd]
            D(Argo CD)
            E(ApplicationSet)
        end
        
        subgraph F [Namespace: tenant-a]
            F1(Jupyter Pod)
            F2(ServiceAccount)
            F3(NetworkPolicy)
        end
        
        subgraph MinIO [Namespace: minio]
            M1[(Shared MinIO S3)]
            M2(MinIO Setup Job)
        end
    end

    B -.-> D
    C -.-> D
    M2 -->|Creates Buckets & Credentials| M1
    M2 -->|Provisions Scoped Secret| F
    F1 -->|Reads/Writes| M1

2. Requirements

This section details how the technical choices address the specific requirements of the assessment.

2.1 Multi-Tenant Management & GitOps

Zero Duplication: Tenant configurations are fully abstracted. The charts/tenant Helm chart acts as the single source of truth for the Kubernetes manifests.
Declarative Onboarding: Adding a new workspace simply requires creating a new folder named tenant-* with a config.yaml file inside tenants/.
Argo CD ApplicationSet: A root ApplicationSet monitors the Git repository path tenants/*/config.yaml. Upon detecting a new folder, it automatically generates an Argo CD Application, pulls the Helm chart, injects the specific values, and provisions the new namespace.

2.2 Kubernetes & S3 Isolation

Namespace & Identity: Each tenant is strictly deployed in its own namespace (tenant-a, tenant-b) and runs under a dedicated Kubernetes ServiceAccount.
Network Security: A NetworkPolicy is deployed within each tenant namespace to restrict ingress/egress traffic. The Jupyter notebook can only communicate with necessary services, preventing cross-namespace communication.
Automated & Isolated S3 Setup: To prevent exposing admin credentials outside the storage infrastructure, a Kubernetes Job runs directly within the minio namespace during tenant provisioning. This job connects to the MinIO admin API locally, creates a reference bucket (Read-Only), a work bucket (Read/Write), and generates dedicated, scoped S3 credentials. It then creates the resulting secret containing only tenant-scoped credentials directly in the target tenant namespace.
dedicated, scoped S3 credentials. These credentials are saved as a Kubernetes Secret locally in the tenant's namespace.
Security Context: The Docker images use the tini init system and execute application containers as a non-root user.

2.3 Container Images & Dependencies

Base & Dependent Images: The project uses two custom images located in images/. The tenant/base image contains the core OS (python:3.12-slim-bookworm), security contexts, and user setups. The tenant/jupyter image builds on top of the base image and installs the application requirements.
No Hardcoded Secrets: Images are completely stateless and free of secrets. They rely solely on environment variables injected at runtime via Kubernetes Secrets.

2.4 CI/CD & Secret Management

Gitea Actions Pipeline: The CI/CD workflow (.gitea/workflows/) automatically manages the image lifecycle and code quality.
Validation: Helm charts and tenant configs are validated before deployment.
Ordered Build Strategy: The pipeline respects the image hierarchy. It builds the base image first, then uses it to build the Jupyter image. Triggered on *.*.* tags, it pushes the versioned artifacts to a private Docker registry.
Testing: Basic testing is executed during the CI phase to ensure the Jupyter environment boots correctly.
Branch Protection & PR Flow: The main branch is protected against direct pushes. All infrastructure and application changes must be proposed via Pull Requests. The CI pipelines act as mandatory status checks, ensuring that no code can be merged into main unless the configurations are validated, no secrets are leaked, and all tests pass successfully.
Secret Scanning: TruffleHog runs as a CI step to scan the repository and prevent any accidental commit of sensitive information (AWS keys, passwords).
Secret Injection: Docker registry credentials are injected into the argocd namespace and propagated to tenant namespaces dynamically matching tenant-[a-z0-9-]+ using the Reflector Kubernetes addon. MinIO admin credentials remain strictly contained within the minio namespace and are never reflected to tenant environments.

3. Deployment Guide (Quickstart)

This section provides the step-by-step instructions to set up the local Kubernetes cluster and bootstrap the GitOps environment.

Note: You will not be able to use this repo directly as it require credentials for my selfhosted gitea and docker registry.

3.1 Prerequisites

Ensure the following tools are installed on your machine (e.g., via Homebrew on macOS):

brew install docker kind kubectl helm argocd k9s

3.2 Cluster Creation

Spin up a local Kubernetes cluster using kind:

kind create cluster --name ctie-exercice
kubectl cluster-info

3.3 Install Cluster Dependencies (Argo CD & Reflector)

Install the Argo CD CRDs, the Reflector addon (used to propagate secrets to dynamically created tenant namespaces), and the Argo CD engine itself:

# Reflector & Argo CD CRDs
kubectl apply --server-side -k "https://github.com/argoproj/argo-cd/manifests/crds?ref=stable"
kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml

# Argo CD in its dedicated namespace
kubectl create ns argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

3.4 Install MinIO (Shared Storage)

Deploy the shared MinIO instance which will act as the S3 backend for all tenants:

kubectl create ns minio
kubectl apply -f minio/manifests.yaml

# Verify MinIO is running
kubectl -n minio get pods -w

3.5 Global Secrets Setup

Before launching the GitOps automated deployments, we need to provide the base credentials. Reflector will automatically replicate these to any namespace matching tenant-[a-z0-9-]+.

1. Docker Registry Credentials:

kubectl create secret docker-registry registry-credentials \
  --docker-server=registry.bouvais.lu \
  --docker-username="<YOUR_USER>" \
  --docker-password="<YOUR_TOKEN>" \
  --namespace=argocd

kubectl annotate secret registry-credentials -n argocd --overwrite \
  reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"

2. MinIO Admin Credentials:

kubectl create secret generic minio-admin-credentials \
  --from-literal=MINIO_ROOT_USER=admin \
  --from-literal=MINIO_ROOT_PASSWORD=password123 \
  -n minio

3.6 GitOps Bootstrap

Finally, apply the root Argo CD ApplicationSet. This will instruct Argo CD to monitor the repository and automatically provision tenant-a and tenant-b.

kubectl apply -f gitops/bootstrap/root-app.yaml

(Optional) You can access the Argo CD UI to watch the automated deployment:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
kubectl -n argocd port-forward svc/argocd-server 8080:443
# Access at https://localhost:8080 using username 'admin'

4. Tests and Validation

Once the tenants are fully deployed by Argo CD, we can verify that data access and workspace isolation behave exactly as expected.

4.1 Accessing the Jupyter Workspace

Because Ingress routing is out-of-scope for this exercise, we will connect to tenant-a's notebook using a manual port-forward:

kubectl port-forward -n tenant-a deployment/jupyter 8888:8888

Open a browser and navigate to http://localhost:8888.

4.2 Running the Isolation Tests

Inside the JupyterLab interface, open a terminal or a notebook and run the provided Python validation script (test_buckets.py). The output will look like this:

SUCCESS: List items in ref bucket: 1 hello tenant-a
SUCCESS: Write to reference bucket blocked: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.
SUCCESS: Read back from work bucket: hello tenant
SUCCESS: Access to tenant-b blocked: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied.

What this proves:

Read-only Reference: The tenant can read the reference bucket, but writing to it is successfully blocked by MinIO policies.
Read/Write Work Bucket: The tenant has full capability in its designated working bucket.
Strict Isolation: Any attempt to list, read, or write to tenant-b's buckets from tenant-a's credentials results in an AccessDenied error.

5. Teardown

To clean up your local environment, you simply need to delete the kind cluster. This will remove all associated resources, namespaces, and volumes.

kind delete cluster --name ctie-exercice

6. Future Improvements

While this setup successfully demonstrates a GitOps-driven multi-tenant architecture, deploying this to a production environment would require several additions:

Ingress & TLS: Replacing manual kubectl port-forward commands with a proper Ingress Controller (e.g., NGINX or Traefik) and integrating cert-manager for automated SSL/TLS certificates.
Storage Quotas: Implementing strict S3 storage limits (quotas) on the work buckets to prevent a single tenant from exhausting shared storage resources.
Kubernetes Resource Limits: Adding Kubernetes ResourceQuotas per tenant namespace and LimitRanges to cap CPU, Memory, and optionally GPU consumption per Jupyter pod.
SSO / Authentication: Replacing token-based Jupyter authentication with OIDC integration (e.g., Keycloak, Dex, or Azure AD) for seamless enterprise user login.
Automated DNS: Integrating ExternalDNS to automatically map newly created tenant workspaces to human-readable URLs (e.g., tenant-a.jupyter.internal.domain).