Dear AI Vendors: This Is What We Need

Evaluating vendors’ AI solutions within an AI governance framework is of critical importance in a healthcare environment. Vendor-provided tools must be safe, effective and ethical in order to provide the best outcomes for all patients. As part of a governance structure there must be an effective system for understanding and evaluating risks associated with AI solutions in clinical use. This approach defines four categories of risk and a grading system to assess the risk in each category. These categories are 1) correctness and transparency, 2) fairness and equity, 3) integrated workflow and 4) safety and privacy. By highlighting the areas of risk, the healthcare system can make more informed decisions on which tools to use, and the solution/deployment teams can develop risk mitigations appropriate to the tools and/or workflows. Establishing a robust governance framework to interact with vendor products will allow healthcare organizations to harness the benefits of commercial AI systems while minimizing risk and maintaining public trust.