Martijn Sturm's blog
Tag

data-engineering

#data-engineering

More content

Read more stories on Hashnode

Articles with this tag

Efficiently Obtaining Large Amounts of Data from External APIs with Python

Dec 10, 20237 min read227 views

Optimizing Python API Interactions for High-Volume Data · For some analytics or machine learning use cases, you could be dependent on data from external...

Efficiently Obtaining Large Amounts of Data from External APIs with Python

Data Engineering on AWS: Best Practices Overview

Apr 23, 20234 min read81 views

This blog post contains a listing of best-practices for data engineering on AWS. I will try to update this post regularly with new insights and best...

Use your own Python packages in Glue jobs

Apr 23, 20234 min read937 views

Many data engineering use cases require you to repeat some ETL logic on different (database) tables or event streams. It is advised to separate the...

Defining AWS glue jobs as Infrastructure-as-Code

Apr 23, 20234 min read445 views

How to develop ETL workflows as Python AWS Glue jobs with AWS CDK and enable local development · Using Infrastructure-as-Code (IaC) for deployment of...