Overview

This project supported by multiple medical professional organizations and spanning the private and public sectors utilized an interactive web-based database of test cases vetted by a global community of physicians and medical students to utilize collective intelligence to improve diagnostic accuracy.

Department

Department of Medicine

Status/Stage of Development

Completed

Funding Sources

Funding and support for the Human Diagnosis Project nonprofit arm came from The Gordon and Betty Moore Foundation.

National/Policy Context

• While diagnosis is central to the practice of medicine, misdiagnosis is prevalent for common conditions, leading to potential morbidity and mortality. Traditionally, diagnosis is performed by an individual practitioner assessing the patient and arriving at a conclusion on his/her own.
• The National Academy of Medicine champions collaborative, team-based diagnosis (e.g. team rounds, case conferences, tumor boards) as a method to reduce diagnostic errors.
• There is little evidence on the relative benefit of diagnosing with group-based approaches, though the ability of collective intelligence (See Glossary) in outperforming individuals has gained traction in politics, business, and economics.
  • Collective intelligence facilitated by software is streamlined for medicine since it has the potential to offer superior performance with little coordination.
• The existing evidence on collective intelligence in general clinical diagnosis is limited to single-center studies and binary decisions; However, most diagnostic processes require combining many pieces of information together and considering multiple diagnoses.

Local/Organizational Context

The Human Diagnostic Project (See Tools/Products for more) is a worldwide effort lead by the global medical community, combining public, private, and social sectors

Patient Population Served and Payor Information

• Self-reported Human Dx profiles categorized each participant by specialty: internal medicine, surgery, other specialties, and medical students (See Figure 1).

Fig 1: Breakdown of sample participants.

Leadership

Dr. Barnett lead the project, holding full access to all the data in the study and taking full responsibility for the integrity of data and its analysis.

Tools or Products Developed

• Human Diagnosis Project (Human Dx): A multinational medical project where physicians and medical students solve teaching cases to study whether collective intelligence is associated with improving diagnostic accuracy across multiple specialties.
  • Human Dx creates teaching cases from users’ own clinical practices with key elements of history, physical, and diagnostic tests to identify the intended differential diagnoses.
  • Respondents independently generate ranked diagnoses and are notified if they are correct, then rate cases for clarity and teaching value.

Fig 2: An example of what a case in the Human Dx looks like

Tech Involved

• Desktop computer
• Electronic medical record
• Smartphone Application

Team Members Involved

• Physicians
• Software Engineers/ IT
• Support Staff

Workflow Steps

• When they are not seeing patients, physicians and medical students of all specialties can use the Human Diagnosis Project to diagnose test cases and help contribute to the project of using collective intelligence to improve diagnosis accuracy.
• Based on the plan they pay for (basic or enterprise-level), they have access to different amounts of case studies and resources to help.
  • Personalized content and training for each medical professional
  • Analysis of data (e.g. EHR) to guide personalized training
  • Automated peer review
  • 24/7 phone and staff support
  • Concierge and support for reporting admin
  • Customized content integrated with existing medical education and initiatives
  • Smart credentialing of medical professionals

 

Outcomes

• During this project, 1572 cases were studied by at least 10 respondents.
• Number of cases solved: Among all participants, 748 users (36.2%) solved only one case each, 580 users (28.0%) solved 5 or more cases each, and 22 users (1.1%) solved more than 100 cases each.
• Diagnostic accuracy: The diagnostic accuracy of all users was 62.5%, the accuracy of individual residents/fellows was 65.5%, the accuracy of attending physicians was 63.9%, and the accuracy of medical students was 55.8%.
• Collective intelligence was associated with greater diagnostic accuracy: individual accuracy was 62.5% while accuracy for a group of nine physicians was 85.6%.
  • When comparing individuals with groups of individuals, groups of two scored 12.5% higher and groups of five scored 17.8% higher than individuals alone. (See Fig 3). Across subgroups of specialty, collective intelligence from increasing group size was still associated with improved diagnostic accuracy across all cases.

Fig 3: Diagnostic Accuracy increases as the number of people in a group increases.

Future Outcomes

• In the future, researchers would focus on assessing the impact of the intervention on patient care.

Benefits

• By providing the opportunity to pool many physicians’ diagnoses together, Human Dx is a scalable approach to improve diagnostic accuracy in future real life cases. Since collective intelligence doesn’t require software facilitation, it is broadly applicable, especially in low-resource settings.
• A collective intelligence approach could provide valuable diagnostic assistance for clinicians in low-income areas that struggle with human capital in healthcare (having less specialized care providers) with higher rates of diagnostic error.
• Increasing diagnostic accuracy benefits patients by increasing standards of care because they are receiving the correct treatment for the diseases they have.

Intervention-Specific Challenges

• Users who contribute to Human Dx may not be a representative sample of the medical community, therefore preventing conclusions from being generalizable to all practices and fields of medicine.
  • While the data set included the largest number of attending physicians among other studies of collective intelligence, trainees composed nearly 80% of the population.
  • Nonetheless, the diagnostic error rate in the data set was within the range identified in previous studies of diagnostic error.
• Human Dx was not designed specifically to assess collective intelligence so its outcomes are not directly transferable.
• There is potential for inaccurate diagnostic solution assessment when using manual review, and specialty labels are often misattributed to different cases.
  • This could create bias in solution assessment and the specialty match between subspecialists and case specialty.
  • These issues won’t affect the relative performance between individuals to groups.
• The breadth of cases might not be representative of the types of cases usually encountered in practice, or those that are more amenable to improvement using collective intelligence.

Glossary

• Collective intelligence: Groups of individuals acting independently or collectively in ways that are more effective and efficient (e.g. pooling their knowledge together to form a better conclusion or diagnoses than if an individual was acting alone).
  • It refers a group’s capability to collaborate and achieve goals an individual would not be able to accomplish alone.

Sources

• Barnett ML, Boddupalli D, Nundy S, Bates DW. Comparative Accuracy of Diagnosis by Collective Intelligence of Multiple Physicians vs Individual Physicians. JAMA Network Open. 2019 Mar 1;2(3):e190096. doi: 10.1001/jamanetworkopen.2019.0096. PubMed PMID: 30821822.
• Collective Intelligence, Metta Center. (2014, August 16).
• The Human Diagnosis Project. (n.d.).