Persistent Storage

Don’t need persistent storage? Skip to Provider Installation.

This guide shows how to enable persistent storage on your Akash provider using Rook-Ceph. The default configuration targets NVMe drives and the beta3 storage class.

Time: 45-60 minutes

Prerequisites

Before starting, ensure you have:

Hardware Requirements

See Hardware Requirements - Persistent Storage for detailed specifications:

Recommended (default in this guide): NVMe drives — minimum 2 NVMe SSDs across all nodes
Alternative: 4 SATA/SAS SSDs across all nodes (use beta2 / osdsPerDevice: "1" instead)
Drives must be:
- Dedicated exclusively to persistent storage
- Unformatted (no partitions or filesystems)
- NOT used for OS or ephemeral storage
Recommended: Distribute across multiple nodes for redundancy

Network Requirements

Minimum: 10 GbE NIC cards for storage nodes
Recommended: 25 GbE or faster for better performance

Ceph Requirements

For production use:

Minimum 3 OSDs for redundancy and high availability
Minimum 2 Ceph managers
Minimum 3 Ceph monitors
Minimum 60 GB disk space at /var/lib/ceph/ (each monitor requires 60 GB)

OSDs per drive:

HDD: 1 OSD max
SSD: 1 OSD max
NVMe: 2 OSDs max (default in this guide)

Important: Do NOT run multiple OSDs on a single SAS/SATA drive. NVMe drives can achieve improved performance with 2 OSDs per drive.

STEP 1 - Identify Storage Nodes and Devices

List Available Nodes

kubectl get nodes

Check Available Drives

SSH into each potential storage node and list unformatted drives:

lsblk -f

Look for drives with no FSTYPE (unformatted). Example NVMe output:

NAME        FSTYPE LABEL UUID                                 MOUNTPOINT
nvme0n1
nvme1n1
nvme2n1
nvme0n2   ext4         a1b2c3d4-5678-90ab-cdef-1234567890ab /

In this example, nvme0n1, nvme1n1, and nvme2n1 are unformatted and can be used for Ceph. Do not use the OS disk (here nvme0n2).

Wipe Drives (if needed)

If drives have existing partitions or filesystems, wipe them:

# WARNING: This destroys all data on the drive!
sudo wipefs -a /dev/nvme0n1
sudo wipefs -a /dev/nvme1n1

STEP 2 - Install Rook-Ceph Operator

Run from a control plane node:

Add Rook Helm Repository

helm repo add rook-release https://charts.rook.io/release
helm repo update

Create Rook-Ceph Namespace

kubectl create namespace rook-ceph

Install Operator

Standard installation (default kubelet path /var/lib/kubelet):

helm install rook-ceph-operator rook-release/rook-ceph \
  --namespace rook-ceph \
  --version 1.18.7 \
  --wait \
  --timeout 10m

Custom kubelet path (if you configured a custom path in Kubernetes setup):

If you configured a custom kubelet directory (e.g., /data/kubelet), you need to set the CSI kubelet directory:

helm install rook-ceph-operator rook-release/rook-ceph \
  --namespace rook-ceph \
  --version 1.18.7 \
  --set csi.kubeletDirPath="/data/kubelet" \
  --wait \
  --timeout 10m

Verify Operator

kubectl -n rook-ceph get pods

Expected output:

NAME                                 READY   STATUS    RESTARTS   AGE
rook-ceph-operator-xxx               1/1     Running   0          2m
rook-discover-xxx                    1/1     Running   0          2m
rook-discover-yyy                    1/1     Running   0          2m

STEP 3 - Deploy Ceph Cluster

Create Cluster Configuration

Create a file with your storage node and device information. The defaults below are for NVMe (beta3):

cat > rook-ceph-cluster.values.yml << 'EOF'
operatorNamespace: rook-ceph

configOverride: |
  [global]
  osd_pool_default_pg_autoscale_mode = on
  osd_pool_default_size = 3
  osd_pool_default_min_size = 2

cephClusterSpec:
  dataDirHostPath: /var/lib/rook  # Change if using custom mount point

  mon:
    count: 3

  mgr:
    count: 2

  storage:
    useAllNodes: false
    useAllDevices: false
    deviceFilter: "^nvme[0-9]+n1$"  # All NVMe namespaces nvme0n1, nvme1n1, …
    config:
      osdsPerDevice: "2"            # NVMe default; use "1" for SATA/SAS SSD or HDD
    nodes:
    - name: "node1"                 # Replace with your actual node names
    - name: "node2"
    - name: "node3"

cephBlockPools:
  - name: akash-deployments
    spec:
      failureDomain: host
      replicated:
        size: 3
      parameters:
        min_size: "2"
        bulk: "true"
    storageClass:
      enabled: true
      name: beta3                # NVMe storage class (default)
      isDefault: true
      reclaimPolicy: Delete
      allowVolumeExpansion: true
      parameters:
        imageFormat: "2"
        imageFeatures: layering
        csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
        csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
        csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
        csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
        csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
        csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
        csi.storage.k8s.io/fstype: ext4

# Do not create default Ceph file systems or object stores
cephFileSystems:
cephObjectStores:

# Spawn rook-ceph-tools for troubleshooting
toolbox:
  enabled: true
EOF

Important Configuration:

dataDirHostPath: Default is /var/lib/rook
- If using a custom mount point (e.g., RAID array at /data), change to /data/rook
- This directory stores Ceph monitor and manager data (not OSD data)
deviceFilter: Must match your data drives only (never the OS disk)
- NVMe (default): "^nvme[0-9]+n1$" — matches nvme0n1, nvme1n1, nvme10n1, etc.
- Do not use "^nvme[01]n1" — that is a single-character class (0 or 1 only), so drives like nvme2n1 are skipped and you get far fewer OSDs than expected
- SATA/SAS SSD or HDD: "^sd[a-z]+$" (adjust to your device names)
osdsPerDevice:
- NVMe (default): "2"
- HDD/SSD: "1"
nodes: Replace with your actual storage node names from kubectl get nodes
storageClass name:
- NVMe (default): beta3
- SSD: beta2
- HDD: beta1

Install Cluster

helm install rook-ceph-cluster rook-release/rook-ceph-cluster \
  --namespace rook-ceph \
  --version 1.18.7 \
  -f rook-ceph-cluster.values.yml

This will take 5-10 minutes to deploy.

STEP 4 - Verify Ceph Cluster

Check Cluster Status

kubectl -n rook-ceph get cephcluster

Expected output:

NAME        DATADIRHOSTPATH   MONCOUNT   AGE   PHASE   MESSAGE                        HEALTH
rook-ceph   /var/lib/rook     3          5m    Ready   Cluster created successfully   HEALTH_OK

Wait until PHASE is Ready and HEALTH is HEALTH_OK.

Check OSDs

kubectl -n rook-ceph get pods -l app=rook-ceph-osd

You should see OSD pods running on your storage nodes. With osdsPerDevice: "2", expect two OSDs per NVMe drive matched by deviceFilter.

Check Ceph Status

Use the Ceph toolbox to check cluster health:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status

Expected output:

  cluster:
    id:     a1b2c3d4-5678-90ab-cdef-1234567890ab
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c
    mgr: a(active), standbys: b
    osd: 6 osds: 6 up, 6 in

(OSD count depends on how many NVMe drives match the filter × osdsPerDevice.)

Check Storage Class

kubectl get storageclass

Expected output:

NAME                 PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   AGE
beta3 (default)      rook-ceph.rbd.csi.ceph.com   Delete          Immediate           5m

Label Storage Class for Akash

Label your storage class so Akash can recognize it. With the defaults in this guide that is beta3:

kubectl label sc beta3 akash.network=true

If you used beta1 or beta2 instead, label that class name.

Verify the label was applied:

kubectl get sc beta3 --show-labels

You should see akash.network=true in the labels.

STEP 5 - Test Persistent Storage

Create a test PVC to verify storage is working:

cat > test-pvc.yaml << 'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: beta3
EOF

kubectl apply -f test-pvc.yaml

Verify PVC

kubectl get pvc test-pvc

Expected output:

NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-pvc   Bound    pvc-a1b2c3d4-5678-90ab-cdef-1234567890ab   1Gi        RWO            beta3          10s

Status should be Bound.

Cleanup

kubectl delete pvc test-pvc

Troubleshooting

Check Operator Logs

kubectl -n rook-ceph logs -l app=rook-ceph-operator

Check Ceph Logs

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd tree
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd status

Common Issues

Fewer OSDs than expected (e.g. only ~6 when you have more NVMe drives):

Check deviceFilter. "^nvme[01]n1" only matches nvme0n1 and nvme1n1. Use "^nvme[0-9]+n1$" for all nvmeXn1 data namespaces.
Confirm with lsblk -f which devices should be claimed, then ceph osd tree for what Ceph actually took.

OSDs not starting:

Verify drives are unformatted: lsblk -f
Check deviceFilter matches your drives
Review OSD pod logs: kubectl -n rook-ceph logs <osd-pod-name>

HEALTH_WARN:

Check ceph status for specific warnings
Common warnings during initial setup are normal and will resolve

No PVC binding:

Verify storage class exists: kubectl get sc
Check CSI provisioner is running: kubectl -n rook-ceph get pods -l app=csi-rbdplugin-provisioner

Next Steps

Your persistent storage is now ready!

→ Provider Installation - Install the Akash provider

Optional enhancements:

Provider installation – STEP 9 (TLS) - Required for all providers: cert-manager and Gateway TLS
IP Leases - Enable static IPs

Note: You’ll need to configure storage classes in the inventory operator during provider installation to advertise persistent storage capabilities (beta3 for this default NVMe setup). This is covered in the Provider Installation guide.

Prerequisites

Hardware Requirements

Network Requirements

Ceph Requirements

STEP 1 - Identify Storage Nodes and Devices

List Available Nodes

Check Available Drives

Wipe Drives (if needed)

STEP 2 - Install Rook-Ceph Operator

Add Rook Helm Repository

Create Rook-Ceph Namespace

Install Operator

Verify Operator

STEP 3 - Deploy Ceph Cluster

Create Cluster Configuration

Install Cluster

STEP 4 - Verify Ceph Cluster

Check Cluster Status

Check OSDs

Check Ceph Status

Check Storage Class

Label Storage Class for Akash

STEP 5 - Test Persistent Storage

Verify PVC

Cleanup

Troubleshooting

Check Operator Logs

Check Ceph Logs

Common Issues

Next Steps

Additional Resources