Insights

In this segment of “Office Optional with Larry English,” Larry talks about how the wrong data can result in false AI results.

In brief:

• AI creates false insights from real but flawed data. Using outdated, incomplete, or biased datasets leads to confident but incorrect business decisions.
• Data governance at scale is impossible. With over 900 apps per organization, companies can't clean and govern every data point.
• Focus on output governance, not input governance. Test AI tool outputs before production rather than trying to perfect all data sources.

Apple found itself in hot water when news broke that the Apple Card was offering lower credit limits to women, regardless of their financial profiles. The fault lies with the AI algorithm used to analyze creditworthiness. The training data for the algorithm included historical lending data that reflected gendered biases.

The lesson here is straightforward: AI makes mistakes. Sometimes AI hallucinates, filling in gaps in data to provide a “made up” answer — a phenomenon where the model generates plausible but incorrect information. But sometimes AI also generates misinformation and false insights. It provides an answer based on real data, but data that is outdated, incomplete or inaccurate.

For example, say you ask AI to pull average salaries for a role at your company. For some reason, the 2020 salary document was opened more recently than the document for 2024. The AI tool thinks the 2020 document is the most relevant, so it uses that data to provide an answer. It’s a false AI insight — AI provided an answer based on real data, just not the correct data.

As leaders increasingly turn to AI for help with data-driven decision-making, it’s important to understand how false insights occur, how to prevent them and what this means for data governance.

How Does Too Much Data Lead To False AI Insights?

It’s no secret that many organizations are drowning in data. On average, organizations leverage nearly 900 apps, with each modern business application collecting and running on a staggering amount of data.

This creates a critical challenge: AI requires masses of clean data to produce trustworthy, valuable results. Companies certainly have the volume of necessary data. But where most organizations fall short is in the quality of their data. It’s impossible to clean up all their data and properly govern each data point — there’s simply too much of it.

Consider the possibilities for fake insights from AI when employees start using a tool like Microsoft Copilot. Copilot scans all the documents across the entire system for answers to questions, crafting a response based on what seems relevant. It’s not always right. Copilot could pull data from an outdated document from a long-gone employee — not exactly a relevant or trusted source of information.

CEO: C-suite news, analysis, and advice for top decision makers right to your inbox.

What’s more, with new tools such as Microsoft Fabric, a cloud-based, end-to-end data analytics platform, employees are more empowered than ever before to access and act on data. While this creates massive opportunities for organizations, it also multiplies the potential for exposing AI to ungoverned, unmanaged and inaccurate data.

It’s a catch-22. Governing every piece of data isn’t feasible but letting AI access ungoverned data leads to unreliable results. And restricting AI to only well-governed data may limit its usefulness.

So what’s the solution? How can leaders harness the power of AI and ensure AI doesn’t just produce misleading insights? What’s needed is a new mindset around governance.

Prevent AI Misinformation With A New Output Governance Mindset

The age of AI requires a new governance mindset. What’s out: governing all the individual data points. What’s in: governing the outputs of AI tools through end-to-end testing strategies. This change in approach will allow organizations to encourage innovation and take advantage of AI while also mitigating the risks of fake insights leading to poor data-driven decision-making.

Big picture, this new governance framework allows teams to access a broad array of data — including raw or ungoverned data — to build automation tools. But before the tool is brought to production, it must go through a governance checkpoint to evaluate the model and its outputs using standard test cases. The scale and speed with which these innovations occur require that the testing framework leverage automation to keep up. Skipping this governance checkpoint essentially means letting people create powerful and untested tools for decision-making, which could be disastrous to an organization’s future success.

In addition to a governance checkpoint, each AI tool should be closely monitored during its first 90 days of deployment. This period requires proactive monitoring, with a plan to transition to reactive monitoring once the team gains confidence in the tool’s performance.

Proactive monitoring involves direct human oversight — reviewing logs, evaluating test cases, and using AI-based guardrails to observe the tool’s behavior in real time. Once the tool has demonstrated reliability, the team can shift to reactive monitoring, which relies on other AI systems to detect anomalies and trigger alerts when potentially unacceptable behavior occurs.

Good output governance means using AI to help govern AI. Think of it like this: the AI doing the actual work — like analytics — is the adult in the room, capable of complex reasoning. The AIs that monitor it are more like kids: They don’t always get the big picture, but they’re great at shouting, “Hey! That’s not OK!” when something clearly breaks the rules.

Another tactic to prevent AI misinformation and inspire confidence in the output from AI is to require AI tools to include annotations in their responses. With every factual question an employee asks of an AI tool, it should list where it’s pulling the data from. Employees can quickly scan the annotations and decide if the data sources are trustworthy and make sense. (Needless to say, annotations are most appropriate for AI tools intended for internal use.)

AI requires masses of data to work correctly. Organizations have no shortage of data, but most struggle applying data governance to their thousands upon thousands of data points. The solution isn’t to lock data away or just let AI loose on ungoverned data.

Rather, leaders need to reconsider their governance mindset, putting in place a robust end-to-end testing strategy for any new AI tools to ensure the outputs are accurate and decrease the likelihood of AI producing false insights, leading to poor decision-making.

By shifting their mindset from data governance to output governance, organizations can unlock AI’s potential — without falling victim to AI misinformation.

[cta bg="purple" button="View Webinar" link="https://centricconsulting.com/ub_nat_ai_agents_wbnr/"]In our on-demand webinar, our AI expert explains the business case for using AI agents and demonstrates the power of these supercharged LLMs in action.[/cta]

Are you ready to explore how artificial intelligence can fit into your business but aren't sure where to start? Our AI experts can guide you through the entire process, from planning to implementation. Talk to an expert