TerpAI Exercise

RADIOLOGY

The purpose of this project is to streamline the integration of artificial intelligence in radiology education and diagnostic workflows. Despite significant advances in medical image analysis in recent years, many of the latest models are never applied in clinical settings because standalone research-grade models (e.g. state-of-the-art cancer classification and organ segmentation) do not easily interface with existing medical image viewers. Last semester, a CMSC435 team built a web-based viewer that allows users to load a folder of DICOM images from local storage, upload a custom ML model to Google Cloud, and run cloud-based inference. The latest version of our code and documentation is available on GitLab.

This is where 435 students will pick up. The first task is to add support for quantitatively evaluating model predictions. This will involve understanding how the existing codebase works. The team will add support for loading and visualizing ground truth segmentation masks on top of model predictions (and highlighting the differences between the two), and compute standard segmentation metrics like DICE score. Given a folder of DICOM images, associated ground truth masks and predicted masks, a typical user should be able to easily load and view the raw data, model predictions, ground truth, and compute standard metrics.

Although allowing practitioners to visualize model predictions is valuable as a diagnostic tool, researchers need to be able to use these insights to improve their models. Physicians spend considerable time looking at and analyzing CT scans. However, this effort largely goes to waste after physicians make a diagnosis. We want to extend the current web interface to allow experienced radiologists to edit AI model predictions to help state-of-the-art models learn from their mistakes using active learning. Concretely, active learning allows human annotators to improve model performance with partial feedback by modifying or augmenting the model predictions or ground truth. For medical image segmentation, building a robust active learning pipeline requires high quality data annotation tools to quickly add new segmentation masks or edit existing model predictions. Once we make these modifications to the annotations, we should have some way to save predictions for integrating into an external ML training pipeline (re-training ML models is strictly out of scope).

This is a research project. As much as any software this semester, what we build in this project chases a moving target. This is not a one-and-done exercise. Our success is tied with how easily a scientist can use the tool to quickly and accurately modify existing segmentation masks and annotate new masks if required. Some experience with computer vision or graphics is a plus. Teams should be able to deal with ambiguity.

A successful project will be able to extend existing functionality by loading raw images, ground truth masks, model predictions and computing a DICE score to measure prediction quality. Once loaded, the raw DICOM images and annotations will need to be rendered. This rendering should be interactive, allowing for users to toggle an overlay of the raw image, ground truth segmentations (if available) and model predictions (if available). Further, a successful project should create an intuitive annotation interface to facilitate active learning. This annotation process should be interactive and fast, allowing users to visualize updated segmentation masks as they make edits. Lastly, users should be able to save the annotations locally and resume annotating at a later time. We hope to submit these results as a conference publication.

Our virtual client is captured in the chatbot in your CMSC435 agent at TerpAI.umd.edu.