The Product-Minded Data Engineer
Rethinking the Data Engineering Role
In a world where AI is center stage and is threatening the jobs of software engineers and data engineers, it challenges us all to rethink what the role may look like in 1 to 5 years, or if it will even exist at all.
Like most punters, I don’t think the role is going anywhere, I do, however, think it’s going to change, but let's ignore the AI hype cycle for a moment and consider the accountability of the data engineer, where their primary focus is ingestion, transformation and serving quality data to consumers—so the business can drive effective outcomes.
The focus of a traditional data engineer is to:
Build and maintain data pipelines.
Transform & model data.
Data quality.
Collaborate primarily within engineering teams.
Write documentation for other data engineers.
Have limited interaction with stakeholders.
There is nothing inherently wrong with the traditional data engineering, and probably looks like most data engineering roles advertised online today. Look at the job below, it pays $900 a day.
But what if we paused for a moment and considered if we are unintentionally limiting the data engineering role. What if we approached data engineering in similar ways that software engineering has with product minded engineers? What if we move past data engineers being viewed as code monkeys. What if data engineers applied a product mindset to not just the end customer-facing data product, but in everything they do from ingestion, to consumption?
The Product-minded Data Engineer
A product-minded data engineer brings curiosity and strategic thinking to the data team. The product-minded data engineer is the most curious data engineer on the data team.
The product-minded data engineer is always asking questions and wanting to know why a pipeline is required, how the data will be used and what decisions it’ll drive.
The product-minded engineer is not a new role to the data team like the analytics engineer, but a mindset shift for existing data engineers. The product-minded engineer collaborates extremely close with the product manager.
The product-minded data engineer cares about the overall success of the data product and is involved throughout all phases of the data product lifecycle. The traditional data engineer shares some similarities, but there are distinct differences and characteristics in their approach.
Characteristics of product-minded data engineer:
#1 Product Curiosity —Always curious about the final data product and the decisions it drives. They are curious to know the business side and ask “why” often. They ask lots of questions, but the right questions.
🧠 Mindset: Can this pipeline be turned into a pattern other teams can use? I wonder if other teams could use this data product, is this data answering the right questions?
#2 Thinking Beyond Pipelines — When you focus only on what’s in front of you, you miss the big picture. As Tony Fadell said in his book Build, as an IC1, you need to look up, as well as look around. Look past your immediate deadline and all the way down to your ultimate goal.
🧠 Mindset: What do I have to do next? Is what I am working on contributing to team or company goals?#3 Strong Ownership — Treats data products and its engineering components as long-lived assets that require ongoing evolution and improvement. Understand the difference between a data project and a data product.
🧠 Mindset: What features, documentation, or data quality checks can I add to make this data pipeline more robust and reusable over time?#4 Outcome-Oriented — Strong focus on the outcome of the data product, but also aware of the overall vision. They are really good at connecting the “why” with “how”.
🧠 Mindset: Is this work directly contributing to a measurable business outcome, or am I just executing tasks?
#5 Continuous Feedback Loop — Thinks iteratively. Designs data product in consultation with data consumers, gathers feedback after they’ve been delivered, and keeps refining the data product based on actual usage and collected evidence.
🧠 Mindset : If the data product fails to drive decisions, what did I miss? Was the problem poorly scoped, or did I misunderstand the stakeholder’s needs? Can I get feedback and improve?
#6 Outside-in Approach — Historically, data teams and engineers solve problems in silos (the what) and assume business knowledge. Data teams need to move to an inside-out approach and focus on the why, then the what.
🧠 Mindset : Am I building this because it’s technically interesting, or because it solves a real business problem?#7 Customer-Centricity — This leads us to the most critical characteristic. Customer-centricity. Being custom-centric is all about prioritizing the needs of the customer over all else. Being customer-centric is all about humanizing and showing empathy.
🧠 Customer-Centric Mindset
Being customer-centric is all about prioritizing the needs of the customer and putting the customer at the center of all decisions. It goes beyond just taking requirements and executing, it’s about understanding and anticipating customer needs and wants.
Throughout the data product lifecycle, the data engineer needs to consider what other customers (data engineers, ML engineers, analytics) or new customers are created as an outcome of creating a data product. The illilities2 of a data product are a big part of what makes it a data product.
Product-minded engineer vs. the product owner
The product-minded data engineer isn’t a replacement for the product owner, or product manager. The roles complement each other & are intended to work together closely.
Conclusion
Whilst the industry remains hyper-focused on the technical aspect of data products, we’re missing the critical conversation of what human transformation is required to make this shift real, and what changes data engineers need to make.
The product-minded data engineer is not a new role, it’s a mindset, it’s a mental model that must be used when approaching problems. Mindsets are not developed overnight, they have to be developed overtime, and like developing muscles in the gym, you have to work on it daily.
If you’re not approaching your data iniatives with a product mindset, you’re likely developing data projects not data products
If the product-minded data engineer piques your interest, then hit the subscribe button below. I’m going to be covering more of the product-minded data engineer in the following posts to cover why it’s different from the analytics engineer and why they complement each other and some techniques you can apply yourself.
An IC (Individual Contributor) is a professional who focuses on executing tasks and delivering results without direct management responsibilities. They contribute expertise, skills, and work output individually rather than leading a team.
The term "illities" refers to the non-functional requirements of a data product—qualities that describe how a data product operates rather than what it does.