(or 2010s to be more precise) big-data boom brought the emergence of specialization in data roles. What used to be solely described as “Business Intelligence Engineer” was further broken down into Business Intelligence Engineers/Analysts, Data Engineers/Analysts, Data Scientists etc. The reason for this? The abundance of data, and the multidisciplinary responsibilities that come with it, which could not be tamed by one generic job description. So, there was a need to break it down to smaller pieces because of the variety of day-to-day tasks. Approaching the end of 2025 though, are we now going back to more generalized data roles?
The Rise of the Data Generalist
Let’s take it from the start. What do I mean by Data Generalists? If you Google “generalist definition”, it gives you the following definition:
“A person competent in several different fields or activities”
Take the above definition and apply it to the data sector. The more experience I get in the data field, the greater is the extent that I see an increase in demand for data generalists.
Nowadays, a data engineer is not only expected to know how to implement data pipelines in order to transfer data from point A to point B. You expect them to know how to spin up cloud resources, implement CI/CD pipelines and best practices, and also develop AI/ML models. That means that cloud, DevOps and machine learning engineering are all part of the modern data engineer’s tech stack now.
Similarly, a data scientist doesn’t just develop models in a notebook that will never end up somewhere in production. They have to know how to work in production and serve the AI/ML models by possibly using containers or APIs. That is an overlap of data science, machine learning engineering, and cloud all over again.
So, you see where this is going? What could be the reasons that these roles are nowadays getting all mixed up and overlapped with each other? Why are data roles more demanding now and the tech stack required includes multiple disciplines? Is this indeed the era where the data generalist is on the rise?
My personal opinion to why data generalists are now flourishing is due to the 3 main reasons:
- Emergence of Cloud Services
- Explosion of Startup Companies
- Evolution of Artificial Intelligence Tools
Let’s evaluate.
Emergence of Cloud Services
Cloud services have come a long way since 2010, bringing everything to a single platform. AWS, Google and Azure are making it much easier and accessible now for professionals to have access to resources and services that can be used to deploy applications. This means some of the over-specified roles, that operated these functions, are now offloaded to the cloud providers and the data professionals stick to data side of things.
For example, if you use a Platform as a Service (PaaS) data warehouse, you don’t need to worry about the virtual machine it runs on, the operating system, updates etc. A data engineer can immediately take over database administrator or system engineer tasks without too much burden on their day to day tasks. Instead of having 2-3 people maintaining the data warehouse, 1 is enough. That also means that the data engineer needs to have an understanding of infrastructure and database administration on top of the usual data engineering tasks.
The way that the industry is evolving, with more Software as a Service (SaaS) products being developed (such as Databricks, Snowflake and Fabric), I think that this trend is going to be the new norm. These products now make it easy for a data professional to handle the whole end-to-end data pipeline from a single platform. Of course, this comes with a price.
Explosion of Startup Companies
Startups are increasingly critical and economical driving forces for each country. An astonishing number of over 150 million startups exist worldwide, as reported in this study, with about 50 million new business launching each year. Of these, there are more than 1,200 unicorn startups worldwide. Based on these figures, no one can argue with us living in an era of startup dominance.
Say you have an idea that you want to turn into a startup company, what type of people are you looking to surround yourself with? Are you going for people with a niche expertise on data or individuals with more generic knowledge that know how to navigate around the whole end-to-end data pipeline? I would think it’s the latter one.
Deep expertise is good for multinational companies where you get to work on very specific things everyday but being a data generalist is your passport to startups. At least, that’s what I noticed from my experience.
Artificial Intelligence Tools
November 2022 – a month in the history books for the technology world where everything changed. The release of ChatGPT. ChatGPT brought the revolution in the AI world. From that day, every day is different in the tech sector. The impact on the industry? Huge. AI tools being released every day, each with its own strengths and weaknesses.
Long gone are the days where in order to write a piece of code or to gain some knowledge you had to go to stack overflow and read whether anyone had a similar issue with you in the past and has solved it. This was the way that things used to be in order to start developing a solution. Now, every data professional writes code with an AI buddy all day long. AI can answer questions, make you work more efficiently but also get a relatively easy head start on things you have never done before. Of course it still makes mistakes, but if you prompt it correctly and ask the right questions you get amazing help from it.
How is this related to data generalists? Nowadays, if you know the right questions for ChatGPT or Gemini or Copilot (or whatever other AI exists out there) you can do things incredibly fast. So if a data engineer wants to get a quick overview of how to develop a linear regression model, AI can help. If a data scientist wants help in creating a cloud resource, AI can help.
This is how this industry is developing and where things are heading. This is also the reason why I think if you are a good data generalist these days and you know how to ask the right questions, you can achieve anything. The expertise will come later, depending on the repetition of a task and the errors you encounter on the way.
Conclusion
We are living in a time where the data landscape evolves at an incredible pace. Each day brings new challenges and new tools to learn. Yet, I believe that focusing on the bigger picture and developing as a data generalist will be the key to long-term success.
By nailing the fundamentals and understanding the architecture of the entire data pipeline end-to-end, you position yourself as someone who will remain highly demanded in the future. In many ways, the industry seems to be shifting back towards valuing versatile data generalists over narrowly specialized roles.
Of course, this is just my opinion—but I’d love to hear yours.
Source link
#Generalist #AllAround #Type #Data #Professional
