Background
For a variety of reasons, users may wish to mount a persistent volume on two or more pods spanning multiple availability zones. One such use case is to make data stored outside of IRIS available to both mirror members in case of failover.
Unfortunately the built-in storage classes in most Kubernetes implementations (whether cloud or on-prem) do not provide this capability:
- Does not support access mode "ReadWriteMany"
- Does not support being mounted on more than one pod at a time
- Does not support access across availability zones
However, some Kubernetes add-ons (both provider and third-party) do provide this capability. The one we'll be looking at in this article is Amazon Elastic File System (EFS).
Overview
In this article we will:
- Create a Kubernetes cluster on Amazon EKS (Elastic Kubernetes Service)
- Use EFS to create a persistent volume of type ReadWriteMany
- Use IKO to deploy an IRIS failover mirror spanning two availability zones
- Mount the persistent volume on both mirror members
- Demonstrate that both mirror members have read/write access to the volume
Steps
The following steps were all carried out using AWS CloudShell. Please note that InterSystems is not responsible for any costs incurred in the following examples.
We will be using region "us-east-2" and availability zones "us-east-2b" and "us-east-2c".
Create Kubernetes Cluster
export AWS_REGION=us-east-2
export CLUSTER=sample
eksctl create cluster \
--name $CLUSTER \
--region $AWS_REGION \
--zones us-east-2b,us-east-2c \
--node-type m5.2xlarge \
--nodes 3
Configure EBS and EFS
export AWS_ID=$(aws sts get-caller-identity --query Account --output text)
export EBS_ROLE=AmazonEKS_EBS_CSI_DriverRole_$CLUSTER
export EFS_ROLE=AmazonEKS_EFS_CSI_DriverRole_$CLUSTER
eksctl utils associate-iam-oidc-provider \
--cluster $CLUSTER \
--region $AWS_REGION \
--approve
aws eks create-addon \
--addon-name aws-ebs-csi-driver \
--cluster-name $CLUSTER \
--region $AWS_REGION \
--service-account-role-arn arn:aws:iam::${AWS_ID}:role/${EBS_ROLE} \
--configuration-values '{"defaultStorageClass":{"enabled":true}}'
eksctl create addon \
--name aws-efs-csi-driver \
--cluster $CLUSTER \
--region=$AWS_REGION \
--service-account-role-arn arn:aws:iam::$AWS_ID:role/$EFS_ROLE \
--force
eksctl create addon \
--name=eks-pod-identity-agent \
--cluster=$CLUSTER
export ADDONS=$(aws eks list-addons --cluster-name $CLUSTER --query addons[] --output text)
for ADDON in $ADDONS; do
eksctl update addon \
--name $ADDON \
--cluster $CLUSTER \
--region $AWS_REGION
done
eksctl create iamserviceaccount \
--name ebs-csi-controller-sa \
--namespace kube-system \
--cluster $CLUSTER \
--region $AWS_REGION \
--role-name $EBS_ROLE \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--approve \
--override-existing-serviceaccounts
eksctl create iamserviceaccount \
--name efs-csi-controller-sa \
--namespace kube-system \
--cluster $CLUSTER \
--region $AWS_REGION \
--role-name $EFS_ROLE \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy \
--approve \
--override-existing-serviceaccounts
Configure Security and Ingress
We create a Security Group and configure ingress to EFS port 2049 (NFS):
export VPC_ID=$(aws eks describe-cluster --name $CLUSTER --query "cluster.resourcesVpcConfig.vpcId" --output text)
export SG=$(aws ec2 create-security-group \
--description efs-sample-sg \
--group-name efs-sg \
--vpc-id $VPC_ID \
--query "GroupId" \
--output text)
export VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids $VPC_ID --query "Vpcs[].CidrBlock" --output text)
aws ec2 authorize-security-group-ingress \
--group-id $SG \
--protocol tcp \
--port 2049 \
--cidr $VPC_CIDR
Create a File System
The File System routes traffic from the PersistentVolume in each zone to the shared file store.
export FS_ID=$(aws efs create-file-system \
--region $AWS_REGION \
--performance-mode generalPurpose \
--query 'FileSystemId' \
--output text)
Each File System needs an Access Point. We set user and group to 51773 ("irisowner") and provide access to the entire volume ("/"). Note that changing ownership requires root access by EFS ("Uid=0,Gid=0"):
export ACCESS_POINT=$(aws efs create-access-point \
--file-system-id $FS_ID \
--root-directory "Path=/,CreationInfo={OwnerUid=51773,OwnerGid=51773,Permissions=777}" \
--posix-user "Uid=0,Gid=0" \
--tags Key=Name,Value=east-users \
--query "AccessPointId" \
--output text)
Each File System also needs a Mount Target in the subnet of each availability zone. Each Mount Target has an IP address that routes to the local PersistentVolume:
export SUBNET_IDS=$(aws eks describe-cluster --name $CLUSTER --query "cluster.resourcesVpcConfig.subnetIds" --output text)
for SUBNET_ID in $SUBNET_IDS; do
aws efs create-mount-target \
--file-system-id $FS_ID \
--subnet-id $SUBNET_ID \
--security-group $SG
done
Create a StorageClass
Add the following to a file named "efs-sc.yaml":
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: efs-sc
provisioner: efs.csi.aws.com
Now create the storage class:
kubectl apply -f efs-sc.yaml
Create a PersistentVolume
Determine the Volume Handle for the File System:
echo $FS_ID::$ACCESS_POINT
fs-0e67f9ac9a3ba51cd::fsap-02c3ed5dc9233394f // <-- example only, do not use
Add the following to a file named "efs-pv.yaml". Replace the volumeHandle field below with your own:
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-pv
spec:
capacity:
storage: 5Gi
csi:
driver: efs.csi.aws.com
volumeHandle: fs-0e67f9ac9a3ba51cd::fsap-02c3ed5dc9233394f
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
volumeMode: Filesystem
Now create the persistent volume:
kubectl apply -f efs-pv.yaml
Create a PersistentVolumeClaim
Add the following to a file named "efs-pvc.yaml":
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: efs-pvc
namespace: default
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-sc
resources:
requests:
storage: 5Gi
Now create the persistent volume claim:
kubectl apply -f efs-pvc.yaml
Install IKO
Install and run IKO:
helm install sample iris_operator_amd-3.8.42.100/chart/iris-operator
See IKO documentation for additional information on how to download and configure IKO.
Create an IrisCluster
Add the following to a file named iris-efs-demo.yaml:
apiVersion: intersystems.com/v1beta1
kind: IrisCluster
metadata:
name: sample
spec:
storageClassName: iris-ssd-storageclass
licenseKeySecret:
name: iris-key-secret
imagePullSecrets:
- name: dockerhub-secret
volumes:
- name: efs-volume
persistentVolumeClaim:
claimName: efs-pvc
topology:
data:
image: containers.intersystems.com/intersystems/iris:2025.2
preferredZones: ["us-east-2a","us-east-2b"]
mirrored: true
volumeMounts:
- name: efs-volume
mountPath: "/mnt/nfs"
Notes:
- The mirror spans both availability zones in our cluster
- See IKO documentation for information on how to configure an IrisCluster
Now create the IrisCluster:
kubectl apply -f iris-efs-demo.yaml
Soon after that you should see the IrisCluster is up and running:
$ kubectl get pod,pv,pvc
NAME READY STATUS RESTARTS AGE
pod/sample-data-0-0 1/1 Running 0 9m34s
pod/sample-data-0-1 1/1 Running 0 91s
NAME CAPACITY ACCESS MODES STATUS CLAIM STORAGECLASS
pvc-bbdb986fba54 5Gi RWX Bound efs-pvc efs-sc
pvc-9f5cce1010a3 4Gi RWO Bound iris-data-sample-data-0-0 iris-ssd-storageclass
pvc-5e27165fbe5b 4Gi RWO Bound iris-data-sample-data-0-1 iris-ssd-storageclass
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
efs-pvc Bound pvc-bbdb986fba54 5Gi RWX efs-sc
iris-data-sample-data-0-0 Bound pvc-9f5cce1010a3 4Gi RWO iris-ssd-storageclass
iris-data-sample-data-0-1 Bound pvc-5e27165fbe5b 4Gi RWO iris-ssd-storageclass
We can also (by joining the output of "kubectl get pod" with "kubectl get node") see that the mirror members reside in different availability zones:
sample-data-0-0 ip-192-168-18-38.us-east-2.compute.internal us-east-2b
sample-data-0-1 ip-192-168-52-17.us-east-2.compute.internal us-east-2c
Test the shared volume
We can create files on the shared volume on each pod:
kubectl exec sample-data-0-0 -- touch /mnt/nfs/primary.txt
kubectl exec sample-data-0-1 -- touch /mnt/nfs/backup.txt
And then observe that files are visible from both pods:
$ kubectl exec sample-data-0-0 -- ls /mnt/nfs
primary.txt
backup.txt
$ kubectl exec sample-data-0-1 -- ls /mnt/nfs
primary.txt
backup.txt
Cleanup
Delete IrisCluster deployment
kubectl delete -f iris-efs-demo.yaml --ignore-not-found
helm uninstall sample --ignore-not-found
Delete Persistent Volumes
kubectl delete pvc efs-pvc iris-data-sample-data-0-0 iris-data-sample-data-0-1 --ignore-not-found
Note that deleting PersistentVolumeClaim triggers deletion of the corresponding PersistentVolume.
Delete EFS resources
export ACCESS_POINTS=$(aws efs describe-access-points --file-system-id $FS_ID --query "AccessPoints[].AccessPointId" --output text)
for ACCESS_POINT in $ACCESS_POINTS; do
aws efs delete-access-point \
--access-point-id $ACCESS_POINT
done
export MOUNT_TARGETS=$(aws efs describe-mount-targets --file-system-id $FS_ID --query "MountTargets[].MountTargetId" --output text)
for MOUNT_TARGET in $MOUNT_TARGETS; do
aws efs delete-mount-target \
--mount-target-id $MOUNT_TARGET
done
aws efs delete-file-system --file-system-id $FS_ID
Delete more resources
aws iam detach-role-policy \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--role-name $EBS_ROLE
aws iam delete-role \
--role-name $EBS_ROLE
aws iam detach-role-policy \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy \
--role-name $EFS_ROLE
aws iam delete-role \
--role-name $EFS_ROLE
ADDONS=$(eksctl get addon --cluster $CLUSTER --region $AWS_REGION --output json | grep Name | cut -d '"' -f4 | xargs echo)
for ADDON in $ADDONS; do
aws eks delete-addon \
--addon-name $ADDON \
--cluster-name $CLUSTER
done
aws ec2 revoke-security-group-ingress \
--group-id $SG
aws ec2 delete-security-group \
--group-id $SG
Delete Kubernetes Cluster
eksctl delete cluster --name $CLUSTER
Conclusion
We demonstrated how Amazon EFS can be used to mount read/write volumes on pods residing in different availability zones. Several other solutions are available both for AWS and for other cloud providers. As you can see, their configuration can be highly esoteric and vendor-specific, but once working can be reliable and effective.
.jpg)
