Kubernetes

1. Install JuiceFS (ref: https://juicefs.com/docs/csi/getting_started).

   # Add the JuiceFS repository.
   helm repo add juicefs https://juicedata.github.io/charts/
   # Update the repository.
   helm repo update
   # Install JuiceFS CSI Driver.
   # JuiceFS needs certain permission on cluster, and hence needs
   # to be installed in kube-system namespace.
   helm install juicefs-csi-driver juicefs/juicefs-csi-driver -n kube-system

2. Wait for the respective services and containers to deploy. Use kubectl to check the deployment status of pods

   kubectl get pods -n kube-system -l app.kubernetes.io/name=juicefs-csi-driver

3. JuiceFS needs a database for storing metadata. Since CubeAPM already uses a database server (MySQL or PostgreSQL), same database server can be used for JuiceFS as well.

   MySQL

   CREATE DATABASE cubeapm_logs_archive_meta CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
   
   CREATE USER 'cubeapm_logs_archive_user'@'%' IDENTIFIED BY 'cubeapm_logs_archive_pass';
   
   GRANT ALL PRIVILEGES ON cubeapm_logs_archive_meta.* TO 'cubeapm_logs_archive_user'@'%';
   
   FLUSH PRIVILEGES;

   PostgreSQL

   CREATE DATABASE cubeapm_logs_archive_meta;
   
   CREATE USER cubeapm_logs_archive_user WITH PASSWORD 'cubeapm_logs_archive_pass';
   
   GRANT ALL PRIVILEGES ON DATABASE cubeapm_logs_archive_meta TO cubeapm_logs_archive_user;
   
   GRANT ALL ON SCHEMA public TO cubeapm_logs_archive_user;
   
   \c cubeapm_logs_archive_meta
   
   GRANT ALL ON SCHEMA public TO cubeapm_logs_archive_user;

4. Create object storage bucket for storing archive logs data. Name the bucket as cubeapm-logs-archive

5. (Applicable to GCP only) Create a service account and give the (ObjectAdmin and Storage Admin) permission to access the gcp bucket from pods running inside kubernetes cluster.

6. (Applicable to GCP only) Generate a (JSON) key of the service account and using that service account key create a kubernetes secret in kube-system and namespace where cubeapm is running.

   kubectl create secret generic gc-secret \
     --from-file=application_default_credentials.json=./application_default_credentials.json \
     -n <namespace>  # Match your namespace

7. Create Kubernetes Secret configuration file as cubeapm-logs-archive-secret.yaml. Replace the values with actual values.

   AWS

   apiVersion: v1
   kind: Secret
   metadata:
     name: cubeapm-logs-archive-secret
   stringData:
     name: logsarchive
     # database_string
     # MySQL: mysql://cubeapm_logs_archive_user:cubeapm_logs_archive_pass@tcp(localhost:3306)/cubeapm_logs_archive_meta
     # PostgreSQL: postgres://cubeapm_logs_archive_user:cubeapm_logs_archive_pass@localhost:5432/cubeapm_logs_archive_meta
     metaurl: <database_string>
     storage: s3
     # bucket_url: https://cubeapm-logs-archive.s3.ap-south-1.amazonaws.com
     bucket: <bucket_url>
     access-key: <ACCESS_KEY>
     secret-key: <SECRET_KEY>
     # To create the file system in mount pod, add more juicefs format parameters to format-options.
     # format-options: trash-days=1,block-size=4096

   GCP

   apiVersion: v1
   kind: Secret
   metadata:
     name: cubeapm-logs-archive-secret
   stringData:
     name: logsarchive
     # database_string
     # MySQL: mysql://cubeapm_logs_archive_user:cubeapm_logs_archive_pass@tcp(localhost:3306)/cubeapm_logs_archive_meta
     # PostgreSQL: postgres://cubeapm_logs_archive_user:cubeapm_logs_archive_pass@localhost:5432/cubeapm_logs_archive_meta
     metaurl: <database_string>
     storage: gs
     # bucket_url: gs://cubeapm-logs-archive
     bucket: <bucket_url>
     # GCS auth: Use service account JSON keyfile
     configs: "{gc-secret: /root/.config/gcloud}"
     envs: "{GOOGLE_APPLICATION_CREDENTIALS: /root/.config/gcloud/application_default_credentials.json}"

8. Apply secret to cluster

   kubectl apply -f cubeapm-logs-archive-secret.yaml -n <namespace>

9. Create Kubernetes StorageClass configuration file as cubeapm-logs-archive-sc.yaml. Replace the values with actual values.

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: cubeapm-logs-archive-sc
   provisioner: csi.juicefs.com
   reclaimPolicy: Retain
   parameters:
     csi.storage.k8s.io/provisioner-secret-name: cubeapm-logs-archive-secret
     csi.storage.k8s.io/provisioner-secret-namespace: <namespace>
     csi.storage.k8s.io/node-publish-secret-name: cubeapm-logs-archive-secret
     csi.storage.k8s.io/node-publish-secret-namespace: <namespace>

10. Apply storage class to cluster

    kubectl apply -f cubeapm-logs-archive-sc.yaml -n <namespace>

11. Update your CubeAPM values.yaml file. Set configVars.logs.archive.enabled as true and run helm upgrade

12. Check for JuiceFS CSI driver mount pod in kube-system namespace. It follows specific pattern as juicefs-<node-name>-<pvc-id>

    kubectl get pods -n kube-system