Use AWS Glue Python with NumPy and Pandas Python Packages

by

You can check latest python packages installed using this script as glue job

import logging
import pip
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

if __name__ == '__main__':
    logger.info(pip._internal.main(['list']))

As of 30-Jun-2020 Glue as has these python packages pre-installed. So numpy and pandas is covered.

awscli 1.16.242
boto3 1.9.203
botocore 1.12.232
certifi 2020.4.5.1
chardet 3.0.4
colorama 0.3.9
docutils 0.15.2
idna 2.8
jmespath 0.9.4
numpy 1.16.2
pandas 0.24.2
pip 20.0.2
pyasn1 0.4.8
PyGreSQL 5.0.6
python-dateutil 2.8.1
pytz 2019.3
PyYAML 5.2
requests 2.22.0
rsa 3.4.2
s3transfer 0.2.1
scikit-learn 0.20.3
scipy 1.2.1
setuptools 45.1.0
six 1.14.0
urllib3 1.25.8
virtualenv 16.7.9
wheel 0.34.2

You can install additional packages in glue-python if they are present in the requirements.txt used to build the attaching .whl. The whl file gets collected and installed before your script is kicked-off. I would also suggest you to look into Sagemaker Processing which is more easier for python based jobs. Unlike serveless instance for glue-python shell, you are not limited to 16gb limit there.

More Related Contents:

  1. Using moviepy, scipy and numpy in amazon lambda
  2. Using psycopg2 with Lambda to Update Redshift (Python)
  3. Cannot use Requests-Module on AWS Lambda
  4. Using python Logging with AWS Lambda
  5. Pandas in AWS lambda gives numpy error
  6. Import libraries in lambda layers
  7. Python request in AWS Lambda timing out
  8. AWS error from Python: No module named lambda_function
  9. Tesseract OCR on AWS Lambda via virtualenv
  10. How do I select rows from a DataFrame based on column values?
  11. Pandas Melt Function
  12. Pandas read in table without headers
  13. Constructing pandas DataFrame from values in variables gives “ValueError: If using all scalar values, you must pass an index”
  14. Python Pandas: Check if string in one column is contained in string of another column in the same row
  15. First non-null value per row from a list of Pandas columns
  16. Pandas: Filtering multiple conditions
  17. Pandas: Subindexing dataframes: Copies vs views
  18. Get column index from column name in python pandas
  19. Plot bar graph from Pandas DataFrame
  20. pandas DataFrame output end of csv
  21. Comparing floats in a pandas column
  22. Pandas sparse dataFrame to sparse matrix, without generating a dense matrix in memory
  23. Dictionary column in pandas dataframe
  24. How do I split a string into several columns in a dataframe with pandas Python?
  25. Pandas: combining header rows of a multiIndex DataFrame
  26. pandas groupby and join lists
  27. Selecting pandas column by location
  28. Row-wise average for a subset of columns with missing values
  29. Plot multiple columns of pandas DataFrame using Seaborn
  30. Pandas: Creating aggregated column in DataFrame

Leave a Comment