Data movement is a fundamental piece of a data engineer’s duties, and recently I’ve been thinking about the art of data movement. What are some of the most important pieces that a data engineer needs to think about when confronted with data ingestion? There is of course data exporting as well, and in that case, similar thought processes should be undertaken. If you’re exporting data for consumption, then you also need to be thinking about how someone will consume that data.
An important stance as a data engineer with any experience knows is that we’re always questioning whether data is