Removed tenant in path

Reviewed-on: #5

README.md

@@ -16,7 +16,7 @@ the tenant resources, and a shared **MinIO** instance for S3-compatible storage.
 ```mermaid
 graph TD
     subgraph Git Repository
-        A[GitOps: gitops/tenants/] -->|Contains| B(tenant-a/config.yaml)
+        A[GitOps: tenants/] -->|Contains| B(tenant-a/config.yaml)
         A -->|Contains| C(tenant-b/config.yaml)
     end
 
@@ -26,28 +26,34 @@ graph TD
         E -->|Deploys via Helm| F[Namespace: tenant-a]
         E -->|Deploys via Helm| G[Namespace: tenant-b]
 
+        subgraph argo [Namespace: argocd]
+            D(Argo CD)
+            E(ApplicationSet)
+        end
+        
         subgraph F [Namespace: tenant-a]
             F1(Jupyter Pod)
             F2(ServiceAccount)
             F3(NetworkPolicy)
-            F4(MinIO Setup Job)
         end
         
         subgraph MinIO [Namespace: minio]
             M1[(Shared MinIO S3)]
+            M2(MinIO Setup Job)
         end
     end
 
     B -.-> D
     C -.-> D
-    F4 -->|Creates Buckets & Credentials| M1
+    M2 -->|Creates Buckets & Credentials| M1
+    M2 -->|Provisions Scoped Secret| F
     F1 -->|Reads/Writes| M1
 
 ```
 
 ---
 
-## 2. Meeting the Requirements
+## 2. Requirements
 
 This section details how the technical choices address the specific requirements of the assessment.
 
@@ -69,9 +75,11 @@ values, and provisions the new namespace.
 * **Network Security:** A `NetworkPolicy` is deployed within each tenant namespace
 to restrict ingress/egress traffic. The Jupyter notebook can only communicate
 with necessary services (like MinIO) and the internet, preventing cross-namespace communication.
-* **Automated & Isolated S3 Setup:** Instead of sharing credentials, a one-off Kubernetes `Job`
+**Automated & Isolated S3 Setup:** To prevent exposing admin credentials outside the storage
-runs during the Helm deployment for each tenant. This job connects to the MinIO admin API,
+infrastructure, a Kubernetes Job runs directly within the minio namespace during tenant provisioning.
-creates a reference bucket (Read-Only), a work bucket (Read/Write), and generates
+This job connects to the MinIO admin API locally, creates a reference bucket (Read-Only), a work bucket (Read/Write),
+and generates dedicated, scoped S3 credentials. It then creates the resulting secret containing only
+tenant-scoped credentials directly in the target tenant namespace.
 **dedicated, scoped S3 credentials**. These credentials are saved as a Kubernetes
 Secret locally in the tenant's namespace.
 * **Security Context:** The Docker images use the `tini` init system and execute
@@ -80,8 +88,8 @@ application containers as a non-root user.
 ### 2.3 Container Images & Dependencies
 
 * **Base & Dependent Images:** The project uses two custom images located in
-`images/`. The `tenant-base` image contains the core OS (`python:3.12-slim-bookworm`),
+`images/`. The `tenant/base` image contains the core OS (`python:3.12-slim-bookworm`),
-security contexts, and user setups. The `tenant-jupyter` image builds on top
+security contexts, and user setups. The `tenant/jupyter` image builds on top
 of the base image and installs the application requirements.
 * **No Hardcoded Secrets:** Images are completely stateless and free of secrets. They
 rely solely on environment variables injected at runtime via Kubernetes Secrets.
@@ -90,7 +98,7 @@ rely solely on environment variables injected at runtime via Kubernetes Secrets.
 
 * **Gitea Actions Pipeline:** The CI/CD workflow (`.gitea/workflows/`) automatically
 manages the image lifecycle and code quality.
-* **Validation:** Helm charts and Kubernetes manifests are validated before deployment.
+* **Validation:** Helm charts and tenant configs are validated before deployment.
 * **Ordered Build Strategy:** The pipeline respects the image hierarchy. It builds the base image first,
 then uses it to build the Jupyter image. Triggered on `*.*.*` tags, it pushes the versioned
 artifacts to a private Docker registry.
@@ -100,9 +108,10 @@ All infrastructure and application changes must be proposed via Pull Requests.
 The CI pipelines act as mandatory status checks, ensuring that no code can be merged
 into `main` unless the configurations are validated, no secrets are leaked, and all tests pass successfully.
 * **Secret Scanning:** `TruffleHog` runs as a CI step to scan the repository and prevent any accidental commit of sensitive information (AWS keys, passwords).
-* **Secret Injection:** Admin credentials (Docker registry, MinIO admin) are injected into the Argo CD
+* **Secret Injection:** Docker registry credentials are injected into the argocd namespace and
-namespace and propagated to tenant namespaces dynamically matching `tenant-[a-z0-9-]+`
+propagated to tenant namespaces dynamically matching tenant-[a-z0-9-]+ using
-using the `Reflector` Kubernetes addon.
+the Reflector Kubernetes addon. MinIO admin credentials remain strictly contained
+within the minio namespace and are never reflected to tenant environments.
 
 ---
 
@@ -111,6 +120,9 @@ using the `Reflector` Kubernetes addon.
 This section provides the step-by-step instructions to set up the local Kubernetes cluster and
 bootstrap the GitOps environment.
 
+*Note: You will not be able to use this repo directly as it require credentials for
+my selfhosted gitea and docker registry.*
+
 ### 3.1 Prerequisites
 
 Ensure the following tools are installed on your machine (e.g., via Homebrew on macOS):
@@ -176,8 +188,8 @@ kubectl create secret docker-registry registry-credentials \
 kubectl annotate secret registry-credentials -n argocd --overwrite \
   reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
   reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
-  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
+  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+,minio" \
-  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"
+  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+,minio"
 
 ```
 
@@ -187,13 +199,7 @@ kubectl annotate secret registry-credentials -n argocd --overwrite \
 kubectl create secret generic minio-admin-credentials \
   --from-literal=MINIO_ROOT_USER=admin \
   --from-literal=MINIO_ROOT_PASSWORD=password123 \
-  -n argocd
+  -n minio
-
-kubectl annotate secret minio-admin-credentials -n argocd --overwrite \
-  reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
-  reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
-  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-[a-z0-9-]+" \
-  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-[a-z0-9-]+"
 
 ```
 

charts/tenant/templates/deployment.yaml

@@ -6,8 +6,6 @@ metadata:
   labels:
     app: jupyter
     tenant: 'tenant-{{.Values.name}}'
-  annotations:
-    argocd.argoproj.io/sync-wave: "1"
 spec:
   replicas: 1
   selector:
@@ -29,7 +27,7 @@ spec:
           imagePullPolicy: IfNotPresent
           envFrom:
             - secretRef:
-                name: s3-credentials
+                name: 'tenant-{{.Values.name}}-s3-credentials'
           resources:
             requests:
               cpu: {{ .Values.resources.requests.cpu }}

charts/tenant/templates/minio-setup.yaml

@@ -2,99 +2,94 @@ apiVersion: batch/v1
 kind: Job
 metadata:
   name: 'tenant-{{ .Values.name }}-minio-setup'
-  namespace: {{ .Release.Namespace }}
+  namespace: minio
   annotations:
+    argocd.argoproj.io/sync-wave: "-1"
     argocd.argoproj.io/hook: Sync
     argocd.argoproj.io/hook-delete-policy: HookSucceeded
 spec:
   ttlSecondsAfterFinished: 100
   template:
     spec:
+      serviceAccountName: minio-provisioner
       restartPolicy: OnFailure
+      imagePullSecrets:
+        - name: registry-credentials
       containers:
-        - name: mc
+        - name: setup
-          image: minio/mc:RELEASE.2024-03-03T00-13-08Z
+          image: registry.bouvais.lu/minio-runner:1.0.0
-          env:
+          command: ["/bin/sh", "-c"]
-            # 1. MinIO Admin Credentials (Replicated from argocd namespace)
+          args:
-            - name: MINIO_ADMIN_USER
-              valueFrom:
-                secretKeyRef:
-                  name: minio-admin-credentials
-                  key: MINIO_ROOT_USER
-            - name: MINIO_ADMIN_PASS
-              valueFrom:
-                secretKeyRef:
-                  name: minio-admin-credentials
-                  key: MINIO_ROOT_PASSWORD
-
-            # 2. Randomly Generated Tenant Credentials
-            - name: TENANT_USER
-              valueFrom:
-                secretKeyRef:
-                  name: s3-credentials
-                  key: MINIO_ACCESS_KEY_ID
-            - name: TENANT_PASS
-              valueFrom:
-                secretKeyRef:
-                  name: s3-credentials
-                  key: MINIO_SECRET_ACCESS_KEY
-            - name: BUCKET_REF
-              valueFrom:
-                secretKeyRef:
-                  name: s3-credentials
-                  key: BUCKET_REF
-            - name: BUCKET_WORK
-              valueFrom:
-                secretKeyRef:
-                  name: s3-credentials
-                  key: BUCKET_WORK
-          command:
-            - /bin/sh
-            - -c
             - |
-              # Fail immediately if any command fails
               set -e
 
-              # Connect to MinIO using quotes to protect special characters in passwords
+              # Generate password
-              mc alias set minioadmin http://minio.minio.svc.cluster.local:9000 "$MINIO_ADMIN_USER" "$MINIO_ADMIN_PASS"
+              SCOPED_SECRET=$(tr -dc 'a-zA-Z0-9' < /dev/urandom | head -c 32)
+
+              # Configure MinIO alias
+              mc alias set local http://minio.minio.svc:9000 \
+                "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
 
               # Create buckets
-              mc mb --ignore-existing minioadmin/$BUCKET_REF
+              mc mb -p local/ref-tenant-{{ .Values.name }} local/work-tenant-{{ .Values.name }}
-              mc mb --ignore-existing minioadmin/$BUCKET_WORK
 
-              echo "hello tenant-{{ .Values.name }}" | mc pipe minioadmin/$BUCKET_REF/hello.txt
+              # Create user with generated credentials
+              mc admin user add local tenant-{{ .Values.name }}-user "$SCOPED_SECRET"
 
-              # Create tenant user (REMOVED || true)
+              # Add demo file
-              mc admin user add minioadmin "$TENANT_USER" "$TENANT_PASS"
+              echo "hello tenant-{{ .Values.name }}" | mc pipe local/ref-tenant-{{ .Values.name }}/hello.txt
 
-              # Apply IAM isolation policy
+              # Inline policy generation to avoid missing mount issues
               cat <<EOF > /tmp/policy.json
               {
                 "Version": "2012-10-17",
                 "Statement": [
                   {
+                    "Sid": "ReadOnlyRefBucket",
                     "Effect": "Allow",
-                    "Action": ["s3:GetBucketLocation", "s3:ListBucket"],
+                    "Action": [
+                      "s3:GetBucketLocation",
+                      "s3:ListBucket",
+                      "s3:GetObject"
+                    ],
                     "Resource": [
-                      "arn:aws:s3:::$BUCKET_REF",
+                      "arn:aws:s3:::ref-tenant-{{ .Values.name }}",
-                      "arn:aws:s3:::$BUCKET_WORK"
+                      "arn:aws:s3:::ref-tenant-{{ .Values.name }}/*"
                     ]
                   },
                   {
+                    "Sid": "ReadWriteWorkBucket",
                     "Effect": "Allow",
-                    "Action": ["s3:GetObject"],
+                    "Action": [
-                    "Resource": ["arn:aws:s3:::$BUCKET_REF/*"]
+                      "s3:*"
-                  },
+                    ],
-                  {
+                    "Resource": [
-                    "Effect": "Allow",
+                      "arn:aws:s3:::work-tenant-{{ .Values.name }}",
-                    "Action": ["s3:*"],
+                      "arn:aws:s3:::work-tenant-{{ .Values.name }}/*"
-                    "Resource": ["arn:aws:s3:::$BUCKET_WORK/*"]
+                    ]
                   }
                 ]
               }
               EOF
+              # Apply policy and attach to user
+              mc admin policy create local tenant-{{ .Values.name }}-policy /tmp/policy.json
+              mc admin policy attach local tenant-{{ .Values.name }}-policy --user tenant-{{ .Values.name }}-user
 
-              mc admin policy create minioadmin tenant-{{ .Values.name }}-policy /tmp/policy.json || mc admin policy update minioadmin tenant-{{ .Values.name }}-policy /tmp/policy.json
+              # Create target secret with matching credentials
-              # Good enough for now, would need better error handeling
+              kubectl create secret generic tenant-{{ .Values.name }}-s3-credentials \
-              mc admin policy attach minioadmin tenant-{{ .Values.name }}-policy --user="$TENANT_USER" || echo "Policy already attached, skipping."
+                --from-literal=S3_ENDPOINT="http://minio.minio.svc.cluster.local:9000" \
-              echo "MinIO setup completed for tenant-{{ .Values.name }}"
+                --from-literal=MINIO_ACCESS_KEY_ID=tenant-{{ .Values.name }}-user \
+                --from-literal=MINIO_SECRET_ACCESS_KEY="$SCOPED_SECRET" \
+                --from-literal=BUCKET_REF="ref-tenant-{{ .Values.name }}" \
+                --from-literal=BUCKET_WORK="work-tenant-{{ .Values.name }}" \
+                -n minio --dry-run=client -o yaml | kubectl apply -f -
+
+              # Annotate for Emberstack Reflector
+              kubectl annotate secret tenant-{{ .Values.name }}-s3-credentials -n minio --overwrite \
+                reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
+                reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true" \
+                reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="tenant-{{ .Values.name }}" \
+                reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="tenant-{{ .Values.name }}"
+          envFrom:
+            - secretRef:
+                name: minio-admin-credentials

charts/tenant/templates/networkpolicy.yaml

@@ -15,15 +15,14 @@ spec:
     - Ingress
     - Egress
 
-  # Entrée : Autorise le trafic vers le Notebook Jupyter sur le port 8888
+  # Entrée
   ingress:
     - ports:
         - protocol: TCP
           port: 8888
 
-  # Sortie : Restreint les communications aux composants strictement nécessaires
+  # Sortie
   egress:
-    # 1. Résolution DNS dans kube-system
     - to:
         - namespaceSelector:
             matchLabels:
@@ -34,7 +33,7 @@ spec:
         - protocol: TCP
           port: 53
 
-    # 2. Flux vers l'API MinIO (Stockage S3)
+    # 2. Vers MinIO
     - to:
         - namespaceSelector:
             matchLabels:

charts/tenant/templates/secret-s3.yaml

@@ -10,7 +10,8 @@ type: Opaque
 stringData:
   MINIO_ACCESS_KEY_ID: 'tenant-{{ .Values.name }}-user'
   # If the secret exist, keep. Otherwise create a new random one. (To prevent a new password at each sync).
-  MINIO_SECRET_ACCESS_KEY: {{ if and $existingSecret $existingSecret.data (hasKey $existingSecret.data "AWS_SECRET_ACCESS_KEY") }}{{ index $existingSecret.data "AWS_SECRET_ACCESS_KEY" | b64dec }}{{ else }}{{ randAlphaNum 24 }}{{ end }}
+  MINIO_SECRET_ACCESS_KEY: {{ if and $existingSecret $existingSecret.data (hasKey $existingSecret.data "MINIO_SECRET_ACCESS_KEY") }}{{ index $existingSecret.data "MINIO_SECRET_ACCESS_KEY" | b64dec }}{{ else }}{{ randAlphaNum 24 }}{{ end }}
   S3_ENDPOINT: "http://minio.minio.svc.cluster.local:9000"
   BUCKET_REF: "ref-tenant-{{ .Values.name }}"
   BUCKET_WORK: "work-tenant-{{ .Values.name }}"
+

gitops/bootstrap/root-app.yaml

@@ -8,7 +8,7 @@ spec:
   source:
     repoURL: https://git.bouvais.lu/adrien/ctie-exercice
     targetRevision: main
-    path: gitops/tenants
+    path: gitops
     directory:
       recurse: true
   destination:

gitops/tenants/project.yaml

@@ -4,12 +4,18 @@ metadata:
   name: tenants
   namespace: argocd
 spec:
+  # Ensure only pull from here
   sourceRepos:
     - https://git.bouvais.lu/adrien/ctie-exercice
+
+  # So this namespace can only affect other tenant and minio (not kube-system or argocd)
   destinations:
     - server: https://kubernetes.default.svc
       namespace: 'tenant-*'
+    - server: https://kubernetes.default.svc
+      namespace: 'minio'
 
+  # This is necessary to create namespace
   clusterResourceWhitelist:
     - group: ''
       kind: Namespace

images/minio-setup-runner/dockerfile

@@ -0,0 +1,13 @@
+FROM alpine:3.20
+
+RUN apk add --no-cache curl bash openssl \
+    && curl -fsSL -o /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc \
+    && chmod +x /usr/local/bin/mc \
+    && curl -fsSL -o /usr/local/bin/kubectl "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" \
+    && chmod +x /usr/local/bin/kubectl \
+    && adduser -D -u 1000 runner
+
+USER runner
+WORKDIR /home/runner
+
+ENTRYPOINT ["/bin/bash"]

minio/manifests.yaml

@@ -1,3 +1,33 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: minio-provisioner
+  namespace: minio
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: secret-writer
+  namespace: minio
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["create", "get", "update", "patch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: minio-provisioner-secret-writer
+  namespace: minio
+subjects:
+  - kind: ServiceAccount
+    name: minio-provisioner
+    namespace: minio
+roleRef:
+  kind: Role
+  name: secret-writer
+  apiGroup: rbac.authorization.k8s.io
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -13,7 +43,7 @@ spec:
     spec:
       containers:
         - name: minio
-          image: minio/minio:latest
+          image: quay.io/minio/minio:latest
           args: ["server", "/data", "--console-address", ":9001"]
           env:
             - name: MINIO_ROOT_USER