Integrating Amazon EFS with Lambda
Before diving into the integration part let’s have a brief overview of Amazon EFS and Lambda.
EFS is an abbreviation for Elastic File System. It is a highly reliable NFS-based regional storage service, with all data stored durably across multiple Availability Zones. Unlike EBS which is block-based storage, EFS is file-based storage. Moreover, it even supports POSIX which means you can use secure or lock your files using the traditional way.
Lambda is the very famous AWS serverless compute service which is used by many companies ranging from startup to enterprise to run their compute workloads. The beauty of lambda is you don’t have to manage any servers yourself for running your workload. The heavy lifting is managed by AWS seamlessly in the background.
Before we start with the implementation part let’s understand the benefit of this feature. Prior to the release of this feature, we used to have only 512 MB of ephemeral storage per function to store and process data but with this announcement, you can now store and process unlimited data.
Creating VPC
I’m using a two-tier VPC for this lab in which Lambda and EFS will be launched. We need at minimum two private subnets and a NAT gateway so that our Lambda function can download a file from the internet. You can use the below Terraform code to quickly create a two-tier VPC for yourself if you don’t have one.
Let’s now dive into the integration part.
Creating an Elastic File System
Before we can create an EFS we need a security group so let’s create one.
I’m using default VPC for this implementation and make sure you add NFS rule for inbound traffic. I have restricted the traffic source to VPC CIDR.
Head to EFS service available under Storage section.
Step 1: Network Access
Select appropriate VPC from the dropdown.
Select the private subnets you created earlier.
Make sure to select the correct security group we created above.
Step 2: File System Settings
You can go with the default settings to change them as per your need.
Step 3: Client Access
If you want to secure EFS endpoint you can create an IAM policy to describe how EFS is accessed by a resource.
Add access point so that our lambda function can associate itself to this EFS storage.
Step 4: Review and Create
Confirm all the inputs supplied in the earlier steps and create EFS.
Note: EFS will take around 5–10 minute to get provisioned
Creating a Lambda Function
Step 1: IAM Role
We need to create an IAM role for our Lambda function to assume so that it can make required API calls.
Visit the IAM dashboard.
Click on Roles present in the left panel.
Select Lambda under the use case.
Next, select AWSLambdaBasicExecutionRole, AWSLambdaVPCAccessExecutionRole and AmazonElasticFileSystemClientFullAccess policies.
You can tag your role if required else please go ahead and create the role.
Step 2: Create a Security Group
We need port 80 open so that our Lambda function can download files from the internet to confirm if we are able to store file bigger than 512MB.
Step 3: Lambda Function
I have used Python as runtime because I am familiar with it and IAM role created above.
Step 3: Associating VPC
Once the function is created we need to associate our Lambda function to the same VPC which is associated with our EFS.
Note: It will take few minutes to update the function.
Step 4: Attaching EFS
Use the dropdown to select the EFS volume we created earlier. Note: If you don’t see your EFS volume in the list make sure you are in the same region as your EFS volume is in.
Select access point created while launching the EFS volume.
Finally, provide the mount path of your choice. Your lambda function will be able to store and process data on this path.
Step 5: Python Code
We are now ready to test whether our lambda function is able to download files larger than 512MB.
Note: You will have to increase the default 3 seconds timeout for your Lambda function to allow it to download the file. I have set it to 5 minutes
I tested the Lambda function to download a file of 1 GB both via EFS and without EFS.
Without EFS
With EFS
Voila! 🎉 We have successfully integrated EFS with Lambda.
