Past is Prologue: How Conversational Analytics Is Changing Data Work

— We’ve Been Down This Road

Many who have come before have bemoaned the analytics dashboard. Dashboards may contain a lot of information but not much in the way of insight. They may answer the question someone had yesterday but not the question they have today — and certainly not with the granularity needed.

The evolution of generative AI will change dashboarding and reporting in general. I want to discuss how I think generative AI will change the work of data professionals and improve the experience of gaining insights for the end user. I also want to discuss the pitfalls that may come as a result of the shift — and how to avoid them.

The New Paradigm: Conversational Analytics

In any contemplated future of how the work of data professionals will change, shaping insights within the business context will remain the primary requirement. Dashboards may still serve as the starting point for gaining insights — a visual representation of context that allows the user to proceed with additional questions through a chat interface within the dashboard. Or the user could start with a simple chat interface.

In that case, the user would be guided with context through other means; for instance, they could be prompted with a list of questions that others in the same department have previously asked.

Context-Giving as a New (But Also Old) Data Discipline

In either of these scenarios — whether starting questions from a dashboard interface or chat alone — the data professional is responsible for implementing the context-giving: orienting the user to the type of data the business has and the types of questions that may generate the insights the user is interested in. The data professional will frame how the question is answered, which models and metrics should be referenced, what kind of return represents good or bad performance, and how the data should be visualized. They may also include possible follow-up questions the user might want to ask.

As an example of context-giving behind the scenes, a user may ask, “What is the ROI for the individual products this client has?” The prompt engineering created by the data professional would direct that the question be answered by:

• Referencing the primary utilization model,
• Benchmarking against clients within the same industry, and
• Defaulting to a bar chart when discrete categorical data is the output.

Maybe not all data professionals will become what is essentially a prompt engineer, but this will need to be a skill set on the data team.

To do the fun work of allowing users to safely self-serve — by providing underlying guardrails — the data professional must focus on where many data teams have fallen short: clear documentation of dimensions and metrics and documentation of how key methodologies for metrics have changed over time.

The data work required to prepare for the capabilities that generative AI will bring to self-service analytics has to begin upstream with the foundational work that has often been de-prioritized in service of faster shipping — creating well-documented artifacts in a central location. In this way, conversational analytics is bringing data teams back to basics.

Recommendations Become a Built-In Feature

Providing recommendations for decision-making should also be a basic function of the data team. The ability to recommend next steps will become a built-in feature of conversational analytics — but one deserving of the most oversight. As discussed above, current dashboarding methodology may not provide insights; moreover, dashboards cannot recommend an action to be taken.

The data professional will be at the forefront of putting recommendations into production through conversational analytics. However, determining what those recommendations should be will be a collaborative effort among many departments in the business.

The data professional will partner with subject matter experts to understand what type of business context should inform the recommended next step.

As an example, the user may ask, “Why has there been an increase in the utilization of the chronic care product by this client this year?” To understand the why, after discussing with the right product and marketing teams, the data team may put in place requirements for the model to reference any population changes for the client and any marketing materials that went out for the particular program. The model may then reference those sources again to recommend a next step such as:

“The chronic care campaign effectively targeted a growing population of this client. Sleep management is emerging as a key concern, so we recommend sending a targeted communication after the new year.”

From Dashboard Builders to AI Managers

The process of giving context — and the user being able to ask a question and end up with not only an insight but a thoughtful recommendation — shows how flexible this process can and should become. As the user experience becomes more flexible and less tied to the rigidity of static dashboards or reports, the use of dashboards will decrease.

Fewer dashboards will be created, and more dashboards will be retired — meaning less maintenance required by the data team. There will be fewer ad hoc requests for specific reports because generative AI will be able to answer those questions. However, there will be more requests to verify the accuracy of AI’s answers and more incident reports of unexpected or unhelpful outputs generated by AI.

The work of the data team may shift from building dashboards and answering ad hoc questions that serve reporting needs to ensuring that the answers given by conversational analytics tools are accurate and meaningful to the end user.

Earlier, I used the ROI question as an example of how AI can surface insights quickly. In that same scenario, the data team’s work includes verifying that the ROI AI answer always aligns with the latest metric definitions and business rules.

The data team will need to build infrastructure to monitor the output and accuracy of generative AI and continually build in tests as the company allows AI to answer more questions.

Pitfalls and Implementation Strategy

The increasing responsibility that will be given leads me to what I believe can be a pitfall in this world of generative AI for providing self-service analytics: an approach that is not tightly scoped or nuanced.

Almost every tool we currently use on our data team now has a compelling AI offering — including our data warehouse and our business intelligence tool — and they can essentially be turned on with the click of a button. Sometimes they can even yield helpful answers. However, without that product mindset brought to these tools by the data team, they are generally not helpful and often inaccurate.

Imagine if, in the chronic care example, AI began recommending outreach campaigns without checking whether the client’s population health data.

As always, there is tension between building fast — in this case, clicking on conversational analytics in those data tools you already know and love — and building with intent to future-proof these designs.

The company will need to decide what reporting first makes sense to offload to generative AI. To do this well, implementation will need to be done in a phased approach. Perhaps sales reporting comes first because those questions generate the most volume, or perhaps it’s ROI questions because they are the most urgent.

Back to Basics, Forward to Recommendations

To take full advantage of these new capabilities, the data team has to return to understanding and documenting company history as displayed in data modeling and the semantic layer in order to give full context for insights and recommendations. As discussed above, we need to encode our understanding of metrics like ROI and design how we want to provide recommendations — such as when to recommend a type of communication.

The data role has always been collaborative but will now be collaborative in a different way. It will not be primarily requirements gathering for dashboards or advanced machine learning but requirements gathering for generative AI insights and recommendation outputs.

The value proposition of the company has to be encoded in the prompt design. This is an essential but difficult task, which is why I advocate for a thoughtful, phased approach to using generative AI in reporting — even for tools that make it very easy to “put AI in production.”

I’m excited for and invested in the day when the chatbot becomes the primary reporting tool.