Initial setup of terraform backend using terraform
Answer recommended by AWS Collective
terraform
Use pre-installed Terraform plugins instead of downloading them with terraform init
You can use pre-installed plugins by either putting the plugins in the same directory as the terraform binary or by setting the -plugin-dir flag. It’s also possible to build a bundle of every provider you need automatically using the terraform-bundle tool. I run Terraform in our CI pipeline in a Docker container so have a … Read more
How to submit Spark jobs to EMR cluster from Airflow?
While it may not directly address your particular query, broadly, here are some ways you can trigger spark-submit on (remote) EMR via Airflow Use Apache Livy This solution is actually independent of remote server, i.e., EMR Here’s an example The downside is that Livy is in early stages and its API appears incomplete and wonky … Read more