Efficiently Obtaining Large Amounts of Data from External APIs with Python
Optimizing Python API Interactions for High-Volume Data
Dec 10, 20237 min read322
Command Palette
Search for a command to run...
#data-engineering
Articles tagged with #data-engineering
Data Engineering on AWS: Best Practices Overview
This blog post contains a listing of best-practices for data engineering on AWS. I will try to update this post regularly with new insights and best practices. Please note that this is not an exhaustive list. Am I missing an important one? Please let...
Apr 23, 20234 min read101Use your own Python packages in Glue jobs
Many data engineering use cases require you to repeat some ETL logic on different (database) tables or event streams. It is advised to separate the ETL workflows for those tables in separate Glue jobs for multiple reasons: Keeping your ETL runs per ...
Apr 23, 20234 min read1.2KDefining AWS glue jobs as Infrastructure-as-Code
How to develop ETL workflows as Python AWS Glue jobs with AWS CDK and enable local development
Apr 23, 20234 min read514