The refinement of roles and titles in the ‘Big Data’ world continues to evolve. Previously I blogged about the many roles in the data pipeline engineering world, now Maxime Beauchemin adds to the context with an article on “The Rise of the Data Engineer”.
In many scenarios I think the ETL concepts of old are dead. As I’ve blogged about previous, its more a ELT pipeline, since all raw data is worthy of keeping.
I’d say the below comment is particularly key, especially if you’ve got an efficient Continuous Delivery pipeline:
Just like software engineers, data engineers should be constantly looking to automate their workloads and building abstraction that allow them to climb the complexity ladder.