A slight improvement over the existing answer is to use a second “numbers” table that enumerates all of the possible list lengths and then use a cross join to make the query more compact. Redshift does not have a straightforward method for creating a numbers table that I am aware of, but we can use … Read more
Lambda functions connected to a VPC public subnet cannot typically access the internet. To access the internet from a public subnet you need a public IP or you need to route via a NAT that itself has a public IP. You also need an Internet Gateway (IGW). However: Lambda functions do not, and cannot, have … Read more
I just tested this and it works: aws s3 –recursive mv s3://<bucketname>/<folder_name_from> s3://<bucket>/<folder_name_to>
A lot has changed in Docker since this question was asked, so here’s an attempt at an updated answer. First, specifically with AWS credentials on containers already running inside of the cloud, using IAM roles as Vor suggests is a really good option. If you can do that, then add one more plus one to … Read more
I’ve created a video tutorial for this. Just check: Connect to Amazon EC2 file directory using FileZilla and SFTP, Video Tutorial Summary of above video tutorial: Edit (Preferences) > Settings > Connection > SFTP, Click “Add key file” Browse to the location of your .pem file and select it. A message box will appear asking … Read more
While it may not directly address your particular query, broadly, here are some ways you can trigger spark-submit on (remote) EMR via Airflow Use Apache Livy This solution is actually independent of remote server, i.e., EMR Here’s an example The downside is that Livy is in early stages and its API appears incomplete and wonky … Read more
This error message means you failed to authenticate. These are common reasons that can cause that: Trying to connect with the wrong key. Are you sure this instance is using this keypair? Trying to connect with the wrong username. ubuntu is the username for the ubuntu based AWS distribution, but on some others it’s ec2-user … Read more
Update per 20-9-2016 Amazon finally made this easy using the Lambda Proxy integration. This allows your Lambda function to return proper HTTP codes and headers: let response = { statusCode: ‘400’, body: JSON.stringify({ error: ‘you messed up!’ }), headers: { ‘Content-Type’: ‘application/json’, } }; context.succeed(response); Say goodbye request/response mapping in the API Gateway! Option 2 … Read more
Q: Can I use certificates on Amazon EC2 instances or on my own servers? No. At this time, certificates provided by ACM can only be used with specific AWS services. Q: With which AWS services can I use certificates provided by ACM? You can use ACM with the following AWS services: • Elastic Load Balancing … Read more
You can set a bucket policy as detailed in this blog post: http://ariejan.net/2010/12/24/public-readable-amazon-s3-bucket-policy/ As per @robbyt’s suggestion, create a bucket policy with the following JSON: { “Version”: “2008-10-17”, “Statement”: [ { “Sid”: “AllowPublicRead”, “Effect”: “Allow”, “Principal”: { “AWS”: “*” }, “Action”: [ “s3:GetObject” ], “Resource”: [ “arn:aws:s3:::bucket/*” ] } ] } Important: replace bucket in … Read more