Experiences in ML Scaling, ML Project Delivery in Healthcare

机器学习扩展与医疗领域ML项目交付的经验

By John P Desmond, AI Trend Editor

作者：John P Desmond，AI Trend 编辑

Experiences with AI and machine learning at CVS Health and St. Luke’s Health System in Boise, Idaho, are having practical benefits to the two organizations.

在CVS Health和爱达荷州博伊西的St. Luke's Health System应用AI和机器学习的经验，正为这两个组织带来切实的收益。

CVS Health is learning how to scale AI applications using machine learning, especially through the house of machine learning operations (MLOps) tools, according to Nels Lindahl, director of Clinical Decision Systems, speaking in a virtual session at the recent Ai4 Conference held virtually recently.

CVS Health正在学习如何利用机器学习扩展AI应用，尤其是通过机器学习运维（MLOps）工具套件，据CVS Health临床决策系统总监Nels Lindahl在近期线上举办的Ai4会议虚拟会议上所言。

And St. Luke’s Health Center put a COVID-19 prediction program, a supply chain purchase engine and a demand-based staffing application into initial production using AI and machine learning, said Dr. Justin Smith, senior director of advanced analytics at St. Luke’s, also at a recent Ai4 virtual conference session.

St. Luke's Health Center则已将新冠预测程序、供应链采购引擎和基于需求的人员排班应用投入初步生产，均使用AI和机器学习技术，St. Luke's高级分析高级总监Justin Smith博士在近期Ai4虚拟会议上也做了分享。

“We are at an MLOps tipping point, where ML has a growing production footprint, with adoption picking up pace and awareness and understanding at an all-time high,” stated Lindahl. “ML tech can now deliver; people are seeing real use cases in the wild and having them grow; it’s real.”

Lindahl表示：“我们正处于MLOps的临界点，机器学习在生产环境中的足迹不断扩大，采用速度加快，认知和理解程度也达到了历史最高水平。ML技术现在能够交付成果；人们正在现实中看到真实的使用案例并使其成长；这是实实在在的。”

The three primary “ecosystems” for building out an AI footprint are Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP), he said. Developers are using open source tools and some they develop on their own to deliver value to their organizations.

构建AI版图的三大主要“生态系统”是亚马逊云服务（AWS）、微软Azure和谷歌云平台（GCP），他说。开发者正在使用开源工具以及一些自行开发的工具来为其组织创造价值。

He highly recommended having an ML strategy, with executive sponsor, defined budget resources, and a repeatable process to ensure the same approach can be replicated throughout an enterprise. “You need a narrative that helps you push things forward. When you can deliver on a use case consistently, you want to be ready to go,” he said.

他强烈建议制定ML战略，包括高管发起人、明确的预算资源和可重复的流程，以确保相同的方法可以在整个企业中复制。“你需要一个叙事来推动事情向前发展。当你能够持续地交付一个用例时，你需要准备好付诸实施，”他说。

In his professional role today, Lindahl is directing IT application development. He has delivered several complex process automation projects within the clinical decision space of pharmacy benefit management.

在目前的专业角色中，Lindahl负责指导IT应用开发。他已在药房福利管理的临床决策领域交付了多个复杂的流程自动化项目。

Tracking Popular API Services on GitHub

追踪GitHub上受欢迎的API服务

In April, he was tracking 19 external API services on axes of scale versus maturity, with for example AWS Enterprise Search and GCP Vision API rated high on the scale and maturity axes, so in the upper right quadrant. He recently updated his chart and is now tracking 49 external services. “The number of APIs out there is growing every day,” he stated.

今年4月，他在规模与成熟度的坐标轴上追踪了19项外部API服务，例如AWS Enterprise Search和GCP Vision API在规模和成熟度轴上都获得了高评分，因此位于右上象限。他最近更新了图表，现在追踪49项外部服务。“可用的API数量每天都在增长，”他说。

The external services are becoming more specific, so that instead of one large vision API, it is breaking down into more specializations. “The exact thing you want to do is probably closer to something you can put into production from your API, ” he said. “The AI is out there and ready to go.”

外部服务正变得更加专业化，因此不再是一个大型的视觉API，而是分解为更多的专业化方向。“你想做的确切事情，可能已经接近于你可以从API投入生产的东西，”他说。“AI已经存在并且可以随时使用。”

He emphasized again the need for a development organization to have an overall ML strategy. “Just because you can go out and get an API, does not mean that is the right thing to do for your organization. It might be cool technology that is amazing, but there might be no return on that investment for your customer,” he said, adding, “Part of your ML strategy must be about purpose, replication, and reuse. Those are going to be at the heart of getting value back for the organization.”

他再次强调了开发组织需要拥有整体ML战略。“仅仅因为你可以去获取一个API，并不意味着这对你的组织来说是正确的选择。它可能是令人惊叹的酷炫技术，但对你的客户来说可能没有投资回报，”他说，并补充道，“你的ML战略的一部分必须关乎目标、复制和重用。这些将是组织获得价值回报的核心。”

He recommended looking at the GitHub Trending page, to see the popularity of different services. He tracks them over time, using indicators such as “stars,” which means people who have indicated a continuing interest, like a bookmark. In April, for example, TensorFlow registered over 154,000 stars and Pytorch was at 47,000; those attracted the most interest.

他建议查看GitHub Trending页面，以了解不同服务的受欢迎程度。他通过“star”等指标来追踪它们随时间的变化，“star”意味着表示持续关注的人，类似于书签。例如，今年4月，TensorFlow注册了超过154，000个star，PyTorch有47，000个；这些吸引了最多的关注。

Lindahl is also tracking MLOps: machine learning operations tools that support practices to maintain ML models in production reliably. His tracking of MLOps on GitHub shows 10,000 stars for Kubeflow, a machine learning toolkit for Kubernetes, which is an open source container orchestration platform. What surprised Lindahl more was the rate of growth for MLReef, an open source MLOps platform with a focus on collaboration. “The number of people who downloaded it and are actively using it is going up really fast,” he said. He is researching it to see if he can understand what is driving the increased use.

Lindahl还在追踪MLOps：支持实践以可靠地维护生产环境中ML模型的机器学习运维工具。他在GitHub上对MLOps的追踪显示Kubeflow（一个用于Kubernetes的机器学习工具包，Kubernetes是一个开源容器编排平台）有10，000个star。更让Lindahl惊讶的是MLReef的增长速度，这是一个专注于协作的开源MLOps平台。“下载并积极使用它的人数增长非常快，”他说。他正在研究它，看看能否理解是什么推动了使用的增长。

Three ML Projects at St. Luke’s Delivering Business Value in New Ways

St. Luke's的三个ML项目以新方式交付商业价值

The emphasis of Dr. Smith at St. Luke’s was more on current delivered projects, and less on their technical underpinnings. The St. Luke’s complex has eight medical centers serving about one million patients. Dr. Smith described three projects.

St. Luke's的Smith博士的重点更多放在当前已交付的项目上，而非其技术基础。St. Luke's综合系统拥有八个医疗中心，服务约一百万患者。Smith博士描述了三个项目。

The first was an application to predict the rate of spread of COVID-19, so that the hospital could do some better planning. “We were receiving all kinds of wild forecasts for what to expect,” he said. “We wanted to know if we could predict how many patients we would have in our ICU and on general hospital floors.”

第一个是预测新冠传播速度的应用，以便医院能够进行更好的规划。“我们收到了各种关于预期的疯狂预测，”他说。“我们想知道我们是否能预测ICU和普通医院楼层将有多少患者。”

The difficulty was that the forecasts were so different. Idaho had not experienced a wave of infections that happened in New York, for example. The team decided to document what they did know, then produce from that data two sets of forecasts: a short-term forecast for one or two weeks, and a medium-term forecast of up to 30 days. They used a new technique, XGBoost, an open source software library popular in applied machine learning for structuring tabular data. It implements gradient-boosted decision trees, which builds a regression tree in a stepwise fashion, measuring the error of each step and correcting it in the next.

困难在于预测结果差异太大。例如，爱达荷州没有经历纽约发生的那波感染浪潮。团队决定记录他们所知道的信息，然后从这些数据中生成两组预测：一到两周的短期预测和最多30天的中期预测。他们使用了一种新技术XGBoost，这是一个在结构化表格数据的应用机器学习中流行的开源软件库。它实现了梯度提升决策树，逐步构建回归树，测量每一步的误差并在下一步中进行修正。

“It got us to within five or six patients a month out,” Dr Smith said, using variables including the inpatient census and the positivity rate from their own health system. “We controlled 90% of the testing, so we had good data on positivity rates,” Dr. Smith said. “We showed with strong accuracy whether we would be increasing or decreasing the patient census.”

Smith博士说：“这让我们在一个月内将误差控制在五六个患者以内，”使用的变量包括住院患者人数和他们自己健康系统的阳性率。“我们控制了90%的检测，所以我们在阳性率方面有良好的数据，”Smith博士说。“我们以很强的准确性展示了我们的住院患者人数将增加还是减少。”

For the supply chain, the team was asked if they could develop a “purchasing engine” that could achieve savings with more optimal purchasing from the 100 vendors and 400,000 products in the supply chain. Certain volumes of purchase qualify for better pricing, posing an optimization problem. “With such levels of complexity, it’s too large for a human. You can’t solve it on Excel,” Dr. Smith said.

对于供应链，团队被问到他们是否能开发一个“采购引擎”，通过从供应链中的100家供应商和400，000种产品中进行更优采购来实现节省。某些采购量有资格获得更好的定价，这构成了一个优化问题。“在如此复杂的程度下，这对人类来说太大了。你无法在Excel上解决它，”Smith博士说。

Breaking Analyst Habit of Sticking with Familiar Vendors

打破分析师固守熟悉供应商的习惯

An examination of the practices of hospital analysts involved in purchasing, showed that many operated in familiar territory, often choosing vendors they knew and had dealt with in the past. Using advanced analytics, the team wrote algorithms that generated “billions” of scenarios. Providing a view of that data was challenging. “We don’t want to show everything or too little either,” Dr. Smith said, The team took the approach of eliminating the non-viable scenarios, sticking to spending within existing agreements, and to spend the required volume with each vendor to get the best price.

对参与采购的医院分析师实践的检查显示，许多人习惯于在熟悉的领域操作，经常选择他们认识并曾打交道过的供应商。利用高级分析，团队编写了生成“数十亿”种场景的算法。如何展示这些数据是个挑战。“我们不想展示所有内容，但也不想展示太少，”Smith博士说。团队采取了消除不可行情景的方法，坚持在现有协议范围内支出，并确保从每个供应商处采购所需数量以获得最佳价格。

The team also tried to minimize the number of transitions, in which a product would be purchased from a different vendor. The analysis allowed the team to select opportunities representing the lowest number of transitions and the highest potential for savings across the entire health system. “It’s very powerful,” Dr. Smith said. “It’s being rolled out across the enterprise.”

团队还试图最小化供应商转换次数，即产品从不同供应商处采购的情况。分析使团队能够选择代表整个医疗系统内最少转换次数和最高节省潜力的机会。“这非常强大，”Smith博士说。“它正在整个企业范围内推广。”

The new system will require renegotiated contracts with a number of suppliers, so Dr. Smith expects it will take several years to become fully implemented.

新系统将需要与许多供应商重新谈判合同，因此Smith博士预计需要几年时间才能完全实施。

The third project was around demand-based staffing. “We don’t want to be overstaffed and we don’t want to be understaffed,” he said. “We want to optimize our staffing to match historical patient demands.” While COVID did “wild things” to the patient census in 2020, that data was included. The system tried to avoid high labor costs associated with “on-call” roles, and also having to send medical professionals home when demand is low.

第三个项目是关于基于需求的人员排班。“我们不想人员过剩，也不想人员不足，”他说。“我们希望优化人员排班以匹配历史患者需求。”尽管新冠疫情在2020年对住院患者人数造成了“剧烈”影响，但该数据仍被纳入考虑。该系统试图避免与“待命”角色相关的高劳动力成本，以及在需求低时不得不让医疗专业人员回家的情形。

“The solution we created, which is still rolling out, is based on algorithms that offer an optimized schedule that matches staff and census according to predicted demand based on the last three years of data, including 2020,” he said. The system was focused on nurses and their support staff to start. Using a visual picture of the forecasts via Power BI, the business analytics service from Microsoft, human professionals were able to adjust the recommended staffing based on “front-line knowledge.” That would be, for example, knowing there is a football game, a music festival and a rodeo all in the same weekend, something the machine learning algorithm might not pick up.

“我们创建的解决方案仍在推广中，它基于算法，根据过去三年（包括2020年）的数据，提供匹配人员和住院人数的优化排班表，以预测需求，”他说。该系统最初专注于护士及其支持人员。通过Power BI（微软的商业分析服务）的预测可视化，专业人员能够根据“一线知识”调整推荐的排班。例如，知道某个周末同时有足球比赛、音乐节和牛仔竞技活动，这是机器学习算法可能无法捕捉到的。

“The output might be that we need six RNs to cover the 7 am to 7 pm shift, but just four for the 9 am to 5 pm shift,” Dr. Smith said. The system is beginning to be deployed and can project weekly staffing levels needed for four to six weeks out, he said. “It’s very dynamic, and senior executives can look and see how we are doing in hospital staff,” Dr. Smith said.

“输出可能是我们早上7点到晚上7点的班次需要6名注册护士，但上午9点到下午5点的班次只需要4名，”Smith博士说。该系统已开始部署，可以预测未来四到六周所需的每周人员排班水平，他说。“这非常动态，高级管理人员可以查看并了解我们在医院人员排班方面的表现，”Smith博士说。

Learn more at the recent Ai4 Conference, at the GitHub Trending page, at Nels Lindahl and at Dr. Justin Smith.

欲了解更多信息，请关注近期Ai4会议、GitHub Trending页面、Nels Lindahl以及Justin Smith博士。