Background

For a variety of reasons, users may wish to mount a persistent volume on two or more pods spanning multiple availability zones. One such use case is to make data stored outside of IRIS available to both mirror members in case of failover.

Unfortunately the built-in storage classes in most Kubernetes implementations (whether cloud or on-prem) do not provide this capability:

• Does not support access mode "ReadWriteMany"
• Does not support being mounted on more than one pod at a time
• Does not support access across availability zones

However, some Kubernetes add-ons (both provider and third-party) do provide this capability. The one we'll be looking at in this article is Google FIlestore.

Overview

In this article we will:

• Create a Kubernetes cluster on GKE (Google Kubernetes Engine)
• Use Google Filestore to create a persistent volume of type ReadWriteMany
• Use IKO to deploy an IRIS failover mirror spanning two availability zones
• Mount the persistent volume on both mirror members
• Demonstrate that both mirror members have read/write access to the volume

Steps

The following steps were all carried out using Google Cloud Shell. Please note that InterSystems is not responsible for any costs incurred in the following examples.

We will be using region "us-east1" and availability zones "us-east1-b" and "us-east1-c".

Create a Network

Google Filestore requires creation of a custom network and subnet:

Create a Subnet

Enable Google Services

Create Addresses for VPC Peering

Create VPC Peering

Create Kubernetes Cluster

gcloud container clusters create sample-cluster \
    --project=$GOOGLE_CLOUD_PROJECT \
    --region us-east1 \
    --node-locations us-east1-b,us-east1-c \
    --machine-type t2d-standard-2
    --num-nodes=3 \
    --release-channel rapid \
    --enable-ip-alias \
    --network=vpc-multi-region \
    --subnetwork=subnet-us-east1 \
    --cluster-secondary-range-name=pods \
    --services-secondary-range-name=services \
    --shielded-secure-boot \
    --shielded-integrity-monitoring \
    --addons=GcpFilestoreCsiDrive

Create a StorageClass

Google provides a number of storage classes, but none of them will work with our custom network, so we must create our own.  Add the following to a file named gke-filestore-sc.yaml:

Note that the value under key "toppology.gke.io/zone" is a region, not a zone.

Now create the storage class:

Create a PersistentVolumeClaim

Add the following to a file named gke-filestore-pvc.yaml:

Note that 1TB is the minimum size allowed.

Now create the persistent volume claim:

Install IKO

Download IKO.  In the Helm chart, edit file values.yaml, and enable "useIrisFsGroup". This ensures that the volume will be writable by "irisowner":

Install and run IKO:

See IKO documentation for additional information on how to download and configure IKO.

Create an IrisCluster

Add the following to a file named iris-filestore-demo.yaml:

Notes:

• If we hadn't enabled useIrisFsGroup, we would have to create an init container to make the volume writable by "irisowner"
• The mirror spans both availability zones in our cluster
• See IKO documentation for information on how to configure an IrisCluster

Now create the IrisCluster:

Note that it may take some time for the shared volume to be allocated, and you may see errors such as the following in the events log:

A better way to confirm that allocation is underway is to look directly at the filestore instance:

 Note the status of "CREATING"; deployment won't proceed until the state reaches "READY". This may take between five and ten minutes. Soon after that you should see the IrisCluster is up and running:

We can also (by joining the output of "kubectl get pod" with "kubectl get node") see that the mirror members reside in different availability zones:

Test the shared volume

We can create files on the shared volume on each pod:

And then observe that files are visible from both pods:

Cleanup

Delete IrisCluster deployment

Delete Persistent Volumes

Note that deleting PersistentVolumeClaim triggers deletion of the corresponding PersistentVolume.

Delete Kubernetes Cluster

Delete Google Filestore resources

gcloud services vpc-peerings delete --network=vpc-multi-region --quiet
gcloud compute addresses delete google-service-range --global --quiet
gcloud services disable servicenetworking.googleapis.com file.googleapis.com --quiet
gcloud compute networks subnets delete subnet-us-east1 --quiet
gcloud compute networks delete vpc-multi-region --quiet

Conclusion

We demonstrated how Google Filestore can be used to mount read/write volumes on pods residing in different availability zones.  Several other solutions are available both for GKE and for other cloud providers.  As you can see, their configuration can be highly esoteric and vendor-specific, but once working can be reliable and effective.