Encrypt an existing Kubernetes Persistent Volume running on AWS EKS
Photo by Alex Motoc on Unsplash
As of writing this article, AWS does not have server-side encryption enabled by default for EBS volumes and due to this, we sometimes end up creating unencrypted volume(s). A few months ago, AWS introduced a feature to enable server-side encryption on newly created volumes by default, but even this option needs to be explicitly turned on.
You can enable auto server-side encryption by visiting the Settings page available on the EC2 dashboard. Any volume created hereafter will use the default encryption key selected during the enable process for encrypting data at the server-side unless another is specified in the request.
EBS Encryption
As mentioned earlier, this option will only encrypt the newly launched volume so our existing Kubernetes persistent volume is still unencrypted. Let’s learn how we can encrypt an existing K8s persistent volume without losing any data.
To follow along you will need an EKS cluster. I have mine set up with the managed node group but you can also use an unmanaged node group.
EKS Cluster
EKS Node Group
StatefulSet Application
Let’s start with launching a simple nginx StatefulSet application.
Using the above YAML file I launched the nginx stateful application.
Nginx Stateful Application
Let’s also confirm if PV and PVC are successfully created.
Persistent Volume Claim
Persistent Volume
Next, let’s confirm if the EBS volume created is unencrypted.
Unencrypted EBS Volume
Perfect. The EBS volume created by K8s is unencrypted. You can match the EBS volume name with the Persistence Volume name in Kubernetes to be completely sure that we have selected the correct volume.
Now that we have an unencrypted volume let’s start the actual task of encrypting this volume without losing any data.
Encrypting Volume
First of all, let's scale down the StatefulSet to 0 to maintain data integrity.
kubectl scale sts nginx --replicas 0Note: If you have used Helm for managing k8s resources then you will have to update the replica using the Helm. Doing it directly via kubectl might not work as Helm might notice the change and scale back your resources.
Once the StatefulSet has been scaled down to zero we need to navigate to Volumes within AWS EC2 using the management console.
EBS Volumes
Once you are here, select the correct EBS volume and create a snapshot of it by clicking on the Actions dropdown followed by Create snapshot option. For easy identification, I’ll be using the Persistent Volume name as my snapshot name.
EBS Create Snapshot
Alright, once the snapshot is ready we will use the Copy snapshot feature to create an encrypted snapshot. Select the unencrypted snapshot and click on the Actions dropdown followed by the Copy snapshot option.
Encrypted Snapshot
Once your encrypted snapshot is ready we need to create a volume using it so select the encrypted snapshot and click on the Actions dropdown, then Create volume.
Note: When creating the encrypted volume make sure to launch it in the same Availability Zone as your unencrypted volume is.
Create Encrypted Volume 1
Create Encrypted Volume 2
So now you should have two EBS volumes: an unencrypted one and the encrypted one that we created just now.
Encrypted EBS Volume
Alright, let’s export the existing PV and PVC resources in YAML format.
# pvc
kubectl get pvc www-nginx-0 -o yaml > pvc.yaml
# pv
kubectl get pv pvc-8804b445-540c-4a42-ba0a-3395e0b2a7f2 -o yaml > pv.yamlNow, that we have exported PV and PVC definitions in YAML format and have encrypted snapshot and volume in place let’s delete the existing PVC resource from the Kubernetes cluster. This will also delete the PV resource and the unencrypted EBS volume that is associated with it.
kubectl delete pvc www-nginx-0Next, we need to modify the exported PV and PVC YAML files to attach the encrypted volume to the Stateful application.
Note: Please make sure to replace the region, availability zone, volume id, etc with your own values before deploying the PV and PVC resources.
# Create a persistent volume
kubectl apply -f pv.yaml
# Create persistent volume claim
kubectl apply -f pvc.yamlWe can now scale back our Stateful application to 1 replica and confirm whether the transition was successful.
kubectl scale sts nginx --replicas 1Once the scaling is complete, the pod must come online if you followed all the above steps correctly.
Nginx Pod
Let’s also confirm if the encrypted volume is attached to the node or not.
Attached Encrypted EBS Volume
Looking at the data in the volume details tab like the Volume state, Size, Attached Instances, etc we can confirm that the encrypted volume is attached to the EKS worker node and is in use.
Before we leave, please check if you are able to access the data and if you want to you can delete both the unencrypted and encrypted snapshots now.
Congratulations! Pat yourself on the back. You have successfully transitioned Kubernetes managed unencrypted volume to an encrypted volume without losing any data.
Covering the basics
-
In Kubernetes, a persistent volume (PV) is a storage location provisioned either manually by the administrator or dynamically using the Storage Class whereas a persistent volume claim (PVC) is a request to use a persistent volume for storing data by a pod.
-
Persistent Volume is a block of storage provisioned by either the administrator manually or via Storage Class on the fly to be attached to the pod for long-term data storage. The lifecycle of persistent volume is independent of the pod to which the PV is attached. This means you need to explicitly delete the PV even after the pod is deleted.
-
Using the persistent volume reclaim policy you are able to control what should happen to the volume after the associated claim is deleted. You can set the reclaim policy to Delete, Retain or Recycle.
