Using moviepy, scipy and numpy in amazon lambda

by

I was also following your first link and managed to import numpy and pandas in a Lambda function this way (on Windows):

  1. Started a (free-tier) t2.micro EC2 instance with 64-bit Amazon Linux AMI 2015.09.1 and used Putty to SSH in.

  2. Tried the same commands you used and the one recommended by the Amazon article:

    sudo yum -y update
sudo yum -y upgrade
sudo yum -y groupinstall "Development Tools"
sudo yum -y install blas --enablerepo=epel
sudo yum -y install lapack --enablerepo=epel
sudo yum -y install Cython --enablerepo=epel
sudo yum install python27-devel python27-pip gcc

  3. Created the virtual environment:

    virtualenv ~/env
source ~/env/bin/activate

  4. Installed the packages:

    sudo ~/env/bin/pip2.7 install numpy
sudo ~/env/bin/pip2.7 install pandas

  5. Then, using WinSCP, I logged in and downloaded everything (except _markerlib, pip*, pkg_resources, setuptools* and easyinstall*) from /home/ec2-user/env/lib/python2.7/dist-packages, and everything from /home/ec2-user/env/lib64/python2.7/site-packages from the EC2 instance.

  6. I put all these folders and files into one zip, along with the .py file containing the Lambda function.
    illustration of all files copied

  7. Because this .zip is larger than 10 MB, I created an S3 bucket to store the file. I copied the link of the file from there and pasted at “Upload a .ZIP from Amazon S3” at the Lambda function.

  8. The EC2 instance can be shut down, it’s not needed any more.

With this, I could import numpy and pandas. I’m not familiar with moviepy, but scipy might already be tricky as Lambda has a limit for unzipped deployment package size at 262 144 000 bytes. I’m afraid numpy and scipy together are already over that.

More Related Contents:

  1. Use AWS Glue Python with NumPy and Pandas Python Packages
  2. Using psycopg2 with Lambda to Update Redshift (Python)
  3. Cannot use Requests-Module on AWS Lambda
  4. Using python Logging with AWS Lambda
  5. Pandas in AWS lambda gives numpy error
  6. Import libraries in lambda layers
  7. Python request in AWS Lambda timing out
  8. AWS error from Python: No module named lambda_function
  9. Tesseract OCR on AWS Lambda via virtualenv
  10. Fitting empirical distribution to theoretical ones with Scipy (Python)?
  11. ValueError: setting an array element with a sequence
  12. Numpy `logical_or` for more than two arguments
  13. ImportError: numpy.core.multiarray failed to import
  14. How do you use the ellipsis slicing syntax in Python?
  15. Fast replacement of values in a numpy array
  16. Concatenate two NumPy arrays vertically
  17. How to add items into a numpy array
  18. multithreaded blas in python/numpy
  19. shuffling/permutating a DataFrame in pandas
  20. SQL-like window functions in PANDAS: Row Numbering in Python Pandas Dataframe
  21. numpy array assignment problem
  22. Python Numpy TypeError: ufunc ‘isfinite’ not supported for the input types
  23. RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  24. Find index where elements change value numpy
  25. Why is np.where faster than pd.apply
  26. Why are frequency values rounded in signal using FFT?
  27. 2d convolution using python and numpy
  28. Converting a 2D numpy array to a structured array
  29. Large Performance difference when summing ints vs floats in Cython vs NumPy
  30. Shift rows of a numpy array independently

Leave a Comment