Over the past years, “traditional” ETL development has morphed into data engineering, which has a more disciplined software engineering approach. One of the benefits of having a more code-based approach in data pipelines is that it has become easier to build metadata driven pipelines. What does this mean exactly?
Say for example you need to copy a set of tables from a source database into data lake storage. If you have 50 source tables, you don’t want to create 50 distinct pipelines manually. It’s not only very time-consuming (and boring), but it’s also hard to maintain. Suppose you need to