Artificial intelligence is rapidly reshaping healthcare, from predictive analytics to clinical decision support. Yet its promise comes with a critical caveat: AI systems are only as reliable and equitable as the data and decisions that shape them. Without intentional oversight, these tools risk reinforcing—rather than reducing—longstanding disparities in care delivery.
Members of the Senior Executive Healthcare Think Tank bring deep expertise across technology, policy, patient experience and equity. They believe addressing bias in clinical AI is not just a technical challenge but a leadership responsibility.
A recent analysis from Kaiser Family Foundation found that AI can exacerbate disparities when models are trained on biased or incomplete data, with studies linking AI use to longer wait times, underdiagnosis and poorer predictive performance for Black and Hispanic patients. At the same time, the research notes that AI could help reduce disparities if it is intentionally designed with representative data, transparency and ongoing oversight—reinforcing the dual reality leaders now face.
Against this backdrop, healthcare leaders must rethink how AI is designed, validated and governed. The following insights from Think Tank members offer a roadmap for ensuring clinical AI improves outcomes for all patients—not just a subset.
Treat Bias as a Patient Safety Risk
Tirumala Ashish Kumar Manne, Principal Cloud Architect at Optum, emphasizes that bias in AI should be framed not as a technical flaw but as a direct threat to patient safety and equity. With nearly 11 years in healthcare technology, his work focuses on building secure, scalable and AI-enabled systems that support better outcomes.
“Clinical AI succeeds only when it reflects the realities of the populations it serves,” Manne says. “Leaders must treat bias as a patient-safety and equity risk, not just a data issue.”
He stresses the need for rigorous validation across demographic and socioeconomic variables. “Models should be validated across race, age, sex, language, geography, insurance status, disability and socioeconomic risk, with attention to false positives, false negatives, calibration and outcomes,” he explains.
Manne also highlights a critical blind spot: proxy variables. “Proxy labels such as cost or prior utilization must be challenged, because they may reflect unequal access rather than true need,” he says.
To mitigate these risks, he advocates for continuous oversight. “AI should be locally tested, clinically governed, continuously monitored and supported by clinician oversight, auditability and clear pause criteria when it creates unsafe or inequitable outcomes.”
“Bias does not start with the model. It starts with research design—and upstream, with the decision-makers in the room.”
Understand That Bias Starts With Leadership Decisions
Donna Mitchell, CEO of Mitchell Universal Network, brings nearly five decades of cross-industry experience to the issue, including work in healthcare, aviation and telecommunications. She argues that bias originates long before a model is trained.
“Bias does not start with the model. It starts with research design—and upstream, with the decision-makers in the room,” Mitchell says.
She points to real-world consequences of flawed design choices. “Healthcare algorithms have under-referred Black patients because cost was used as a proxy for need. The bias was the upstream choice, not the model,” she explains.
Mitchell emphasizes that leadership composition and accountability are central to solving the problem. “Every model inherits the lens of whoever framed it,” she says. “Leaders address this by changing who is in the room—and by asking three questions: Who controls AI strategy? Whose data is missing? Who is accountable when inequitable outcomes occur?”
Ultimately, Mitchell notes that “bias is a leadership decision before it is a model decision.”
“Poorly labeled and unstructured data can work against identifying important distinctions.”
Build and Maintain Truly Representative Data
Mark Francis, Founder and CEO of CaregiverZone, focuses on the foundational role of data quality and diversity. With experience spanning AWS, Intel and digital health ventures, he has seen firsthand how data gaps translate into biased outcomes.
“Leaders should proactively seek to obtain and build the broadest and most diverse dataset possible,” Francis says.
But diversity alone is not enough. “Data must be thoroughly and accurately labeled to identify all sources of diversity,” he explains. “Poorly labeled and unstructured data can work against identifying important distinctions.”
He also points to the importance of precision in how AI systems are guided. “Be very clear in prompting to ensure the important differences among population groups are not being generalized,” he says.
Even with strong data practices, human oversight remains essential. “Always provide human clinical oversight to assess responses, broaden datasets and tune models to proactively prevent bias from emerging or re-emerging.”
Move From Black Box to Observable Systems
Mahendran Chinnaiah, a Digital Healthcare Architect for a major U.S. healthcare and pharmacy services firm, argues that transparency and observability are essential to preventing bias.
“To prevent AI from widening health disparities, leaders must move beyond black box models and prioritize algorithmic observability,” Chinnaiah says.
He warns that bias often enters through historical data. “Bias often sneaks in through historical data that reflects existing inequalities,” he explains. “We cannot simply set and forget these tools.”
Instead, he advocates for real-time monitoring. “We must build an architecture that monitors for concept drift in real time,” he says.
Chinnaiah also argues the importance of representative data sourcing and governance. “A non-negotiable step is auditing datasets for diverse populations and social determinants of health,” he notes. “We need a human-in-the-loop governance framework where clinical experts can override suggestions that deviate from equitable care.”
His conclusion is clear: “Bias isn’t just a data problem—it’s a governance challenge. We improve equity by making the ‘plumbing’ of AI transparent and accountable.”
Strengthen Governance and Local Validation
Sriharsha Chavali, an Enterprise Technology Leader at The Aspen Group, highlights how bias often reflects existing gaps in healthcare delivery systems.
“AI does not create bias on its own; it learns from the data and workflows already in place,” Chavali says.
He shares real-world examples. “Lab feeds from major systems often look complete, but rural hospitals have thinner records, and patients with inconsistent access to care have real gaps,” he explains.
These disparities can propagate into AI systems if left unchecked. “If historical billing data shows low-income patients being written off earlier, the model absorbs those patterns unless leaders intervene,” he says.
To counteract this, Chavali recommends rigorous evaluation practices. “Leaders need subgroup performance reviews, local validation and continuous monitoring,” he says.
He also reinforces the role of clinicians. “AI should support clinician judgment, not replace it,” he adds. “The real responsibility lies in ensuring governance is as rigorous as the model itself.”
Embed Ethics, Accountability and Oversight
Jordan W. Henry, Founder and Chief AI Ethicist at Veritas AI Consulting, frames bias mitigation as an ethical imperative requiring structured governance.
“Healthcare leaders can address AI bias by ensuring diverse, representative training data from underrepresented groups to avoid embedding historical inequities,” Henry says.
He stresses the importance of proactive evaluation. “Leaders should conduct audits with fairness metrics across subgroups and immediately pull biased tools like contaminated drugs,” he explains.
Transparency is another cornerstone. “Adopt transparent, explainable models with TRIPOD+AI reporting and equity impact assessments,” he says.
Henry also emphasizes inclusive design. “Engage multidisciplinary teams of clinicians, ethicists and patients throughout design, training, silent testing and monitoring,” he notes.
Finally, he calls for leadership accountability. “Board-level governance with human-in-the-loop oversight completes the approach,” he says. “These steps turn AI into an equity tool that narrows rather than widens disparities.”
Practical Steps to Build More Equitable Clinical AI
- Treat bias as a patient safety issue. Evaluate AI performance across diverse populations and establish clear intervention thresholds.
- Address bias at the leadership level. Ensure diverse decision-making teams and define accountability for AI outcomes.
- Invest in representative, well-labeled data. Prioritize diversity and accuracy in datasets to avoid systemic blind spots.
- Implement continuous monitoring and observability. Track model performance in real time to detect drift and emerging bias.
- Validate locally and govern rigorously. Test models in real-world settings and maintain strong oversight frameworks.
- Embed ethics and accountability into AI strategy. Use fairness metrics, transparency standards and board-level governance to guide implementation.
Leading the Future of Fair and Responsible AI
Clinical AI holds enormous potential to improve outcomes, streamline operations and expand access to care. But without deliberate action, it can just as easily reinforce the very disparities it aims to solve.
The insights from the Senior Executive Healthcare Think Tank make one point clear: Eliminating bias in AI is not a one-time fix but an ongoing leadership commitment. By prioritizing equity in design, governance and monitoring, healthcare organizations can ensure that AI becomes a force for better, fairer care across every population they serve.
