How to submit Spark jobs to EMR cluster from Airflow?

by

While it may not directly address your particular query, broadly, here are some ways you can trigger spark-submit on (remote) EMR via Airflow

  1. Use Apache Livy

    • This solution is actually independent of remote server, i.e., EMR
    • Here’s an example
    • The downside is that Livy is in early stages and its API appears incomplete and wonky to me

  2. Use EmrSteps API

    • Dependent on remote system: EMR
    • Robust, but since it is inherently async, you will also need an EmrStepSensor (alongside EmrAddStepsOperator)
    • On a single EMR cluster, you cannot have more than one steps running simultaneously (although some hacky workarounds exist)

  3. Use SSHHook / SSHOperator

    • Again independent of remote system
    • Comparatively easier to get started with
    • If your spark-submit command involves a lot of arguments, building that command (programmatically) can become cumbersome

EDIT-1

There seems to be another straightforward way

  1. Specifying remote master-IP

Useful links

More Related Contents:

  1. Deploying to multiple AWS accounts with Terraform?
  2. Terraform: state management for multi-tenancy
  3. Initial setup of terraform backend using terraform
  4. CloudFront + S3 Website: “The specified key does not exist” when an implicit index document should be displayed
  5. AWS Elastic Beanstalk, running a cronjob
  6. Redshift. Convert comma delimited values into rows
  7. AWS Lambda Scheduled Tasks
  8. Add Keypair to existing EC2 instance
  9. Amazon products API – Looking for basic overview and information
  10. How to get Elastic Beanstalk nginx-backed proxy server to auto-redirect from HTTP to HTTPS?
  11. connecting AWS SAM Local with dynamodb in docker
  12. Append data to an S3 object
  13. Serving a multitude of static sites from a wildcard domain in AWS
  14. How to look for updated rows when using AWS Glue?
  15. Proper access policy for Amazon Elastic Search Cluster
  16. Is there a way to list all resources in AWS
  17. How list Amazon S3 bucket contents by modified date?
  18. Fetching AWS instance metadata from within Docker container?
  19. What is difference between Lightsail and EC2? [closed]
  20. Does an Application Load Balancer support WebSockets?
  21. Sending html content in AWS SNS(Simple Notification Service) emails notifications
  22. How do you search an amazon s3 bucket?
  23. Specify log group for an AWS lambda?
  24. SSL CERTIFICATE_VERIFY_FAILED in aws cli
  25. Can’t push image to Amazon ECR – fails with “no basic auth credentials”
  26. AWS S3 CLI – Could not connect to the endpoint URL
  27. AWS CloudFront redirecting to S3 bucket
  28. How can I join tables in AWS DynamoDB?
  29. Cloudfront and EC2
  30. AWS S3 gracefully handle 403 after getSignedUrl expired

Leave a Comment