Create AWS Lambda Layer for Python

Overview

AWS Lambda is one of the most powerful services offered by AWS. I have used it in a few of my personal projects and it never fails to surprise me with the pleathora of features it has.

I use boto3, the Python SDK for AWS, to perform start/stop on my EC2 instances during work hours and for a few report generation process where I just upload the CSV file to S3 and it is processed by lambda and delieverd via SNS. The biggest problem I always come across is the limited number of python packages natively available in Lambda. Since I use pandas very often across my projects, instead of zipping it with the code, I created a lambda layer so that I can use it across all my lambda functions.

Lambda layers provide a convenient way to package libraries and other dependencies that you can use with your Lambda functions. Using layers reduces the size of uploaded deployment archives and makes it faster to deploy your code. You can read more about it here:
[+] https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

Setting up EC2 instance

Once the instance is initialized, connect to it using SSH. We now have to update the linux packages and download the required python version if necessary. Lambda currently supports Python3.7, Python3.8 and Python3.9.

1. Updating Linux packages:

1

$ sudo yum update && sudo yum upgrade -y

2. Check Python3 version currently installed on the instance:

1
2

$ python3 --version
Python 3.7.10

In my instance, I had version 3.7.10 installed but I needed version 3.8. First you need to make sure if amazon-linux-extras currently installed on the instance. You can do it using the command:

1
2

$ which amazon-linux-extras
/usr/bin/amazon-linux-extras

You can look for available python versions in amazon-linux-extras

1
2

$ amazon-linux-extras | grep -i python
 44  python3.8                available    [ =stable ]

Enable the python3.8:

1

sudo amazon-linux-extras enable python3.8

3. You can now install the python3.8 version on your EC2 instance:

1

sudo yum install python38 -y

Confirm that the version of python3.8 and pip3.8 installed on the instance:

1
2
3
4

$ python3.8 --version
Python 3.8.5
$ pip3.8 --version
pip 21.0.1 from /usr/lib/python3.8/site-packages/pip (python 3.8)

4. Enable virtual environments.

1
2

$ directory='lambda_layer'
$ mkdir $directory && cd $directory

The python library should follow the directory structure provided in the below documentation. Hence, we will set up a similar virtual environment:

[+] https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-compile

5. Create a python virutal environment named “python” using venv

1

$ python3.8 -m venv python

6. Activate the virtual environment

1

$ source python/bin/activate

7. Install pandas using pip3.8

1

$ pip3.8 install pandas

8. Deactivate the virtual envirnoment

1

$ deactivate

9. Zip the library directory which contains the site-packages

1

$ zip -r panda_lambda_layer.zip python/lib

10. To make sure your zip file is accurate, it should look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

$ unzip -l panda_lambda_layer.zip | less
---truncated---
Archive:  panda_lambda_layer.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  10-10-2022 07:54   python/lib/
        0  10-10-2022 07:54   python/lib/python3.8/
        0  10-10-2022 07:54   python/lib/python3.8/site-packages/
      126  10-10-2022 07:54   python/lib/python3.8/site-packages/easy_install.py
        0  10-10-2022 07:54   python/lib/python3.8/site-packages/pkg_resources/
   108570  10-10-2022 07:54   python/lib/python3.8/site-packages/pkg_resources/__init__.py
      396  10-10-2022 07:54   python/lib/python3.8/site-packages/pkg_resources/py2_warn.py
      558  10-10-2022 07:54   python/lib/python3.8/site-packages/pkg_resources/py31compat.py
        0  10-10-2022 07:54   python/lib/python3.8/site-packages/pkg_resources/_vendor/
        0  10-10-2022 07:54   python/lib/python3.8/site-packages/pkg_resources/_vendor/__init__.py
    24701  10-10-2022 07:54   python/lib/python3.8/site-packages/pkg_resources/_vendor/appdirs.py
   232055  10-10-2022 07:54   python/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py
    30098  10-10-2022 07:54   python/lib/python3.8/site-packages/pkg_resources/_vendor/six.py
---truncated--- 

The zip file panda_lambda_layer.zip thatt needs to be uploaded as the Lambda Layer.

Publishing the Lambda Layer

You can publish a Lambda Layer using the following ways.

• Upload your .zip file to s3 bucket and create a Lambda Layer using S3 URL:

1

$ aws s3 cp <path_the_zip_file> s3://<bucket_name>

You need to make sure that your ec2 instance has necessary permissions to put objects to your S3 bucket.

Create a Lambda Layer using the URL for the zip file stored in S3 using the steps outlined here:
[+] https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-create

• If your instance has the necessary IAM role to publish Layers to Lambda, you can directly publish it using AWS CLI.

1
2
3
4

$ aws lambda publish-layer-version --layer-name <layer_name> --description "My Pandas layer"  \ 
--license-info "MIT" --content S3Bucket=<bucket name>,S3Key=<file name> \
 --compatible-runtimes python3.8
  --compatible-architectures "x86_64" 

Hope this helps!