How to look for updated rows when using AWS Glue?

by

You can use the query to find the updated records by filtering data at source JDBC database as shown below example. I have passed date as an argument so that for each run I can fetch only latest values from mysql database in this example.

query= "(select ab.id,ab.name,ab.date1,bb.tStartDate from test.test12 ab join test.test34 bb on ab.id=bb.id where ab.date1>'" + args['start_date'] + "') as testresult"

datasource0 = spark.read.format("jdbc").option("url", "jdbc:mysql://host.test.us-east-2.rds.amazonaws.com:3306/test").option("driver", "com.mysql.jdbc.Driver").option("dbtable", query).option("user", "test").option("password", "Password1234").load()

More Related Contents:

  1. AWS Glue executor memory limit
  2. AWS Glue output file name
  3. Why do we need private subnet in VPC?
  4. Amazon AWS Route 53 Hosted Zone does not work
  5. Is there a way to change the http status codes returned by Amazon API Gateway?
  6. How to submit Spark jobs to EMR cluster from Airflow?
  7. Connect to Amazon EC2 file directory using Filezilla and SFTP
  8. How long should I wait after applying an AWS IAM policy before it is valid?
  9. AWS EFS vs EBS vs S3 (differences & when to use?) [closed]
  10. Getting json body in aws Lambda via API gateway
  11. Multiple Cloudfront Origins with Behavior Path Redirection
  12. Signature expired: is now earlier than error : InvalidSignatureException
  13. Deploying react-redux app in AWS S3
  14. How to change permission recursively to folder with AWS s3 or AWS s3api
  15. aws billing information using aws java sdk
  16. The AWS Access Key Id does not exist in our records
  17. AWS Cognito: Best practice to handle same user (with same email address) signing in from different identity providers (Google, Facebook)
  18. Difference between upload() and putObject() for uploading a file to S3?
  19. What is the difference between scan and query in dynamodb? When use scan / query?
  20. AWS sts assume role in one command
  21. Celery with Amazon SQS
  22. Declare a variable in RedShift
  23. Adding AWS Lambda with VPC configuration causes timeout when accessing S3
  24. AWS Difference between a snapshot and AMI
  25. How do I delete a versioned bucket in AWS S3 using the CLI?
  26. How to change content type of Amazon S3 Objects
  27. connect to AWS IoT using web socket with Cognito authenticated users
  28. Include custom fonts in AWS Lambda
  29. Get last modified object from S3 using AWS CLI
  30. Initial setup of terraform backend using terraform

Leave a Comment