Defining AWS glue jobs as Infrastructure-as-Code
How to develop ETL workflows as Python AWS Glue jobs with AWS CDK and enable local development
Apr 23, 20234 min read514
Command Palette
Search for a command to run...
Series
How to Data Engineering on AWS
In this series I will describe best practices and delve into some technical concepts of Data Engineering on AWS
Use your own Python packages in Glue jobs
Many data engineering use cases require you to repeat some ETL logic on different (database) tables or event streams. It is advised to separate the ETL workflows for those tables in separate Glue jobs for multiple reasons: Keeping your ETL runs per ...
Apr 23, 20234 min read1.2KData Engineering on AWS: Best Practices Overview
This blog post contains a listing of best-practices for data engineering on AWS. I will try to update this post regularly with new insights and best practices. Please note that this is not an exhaustive list. Am I missing an important one? Please let...
Apr 23, 20234 min read101