Author name cluster

Berkman Sahiner

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

4 papers

2 author rows

JBHI Journal 2025 Journal Article

Bias Amplification to Facilitate the Systematic Evaluation of Bias Mitigation Methods

Alexis Burgon
Yuhang Zhang
Nicholas Petrick
Berkman Sahiner
Kenny H. Cha
Ravi K. Samala

The future of artificial intelligence (AI) safety is expected to include bias mitigation methods from development to application. The complexity and integration of these methods could grow in conjunction with advances in AI and human-AI interactions. Numerous methods are being proposed to mitigate bias, but without a structured way to compare their strengths and weaknesses. In this work, we present two approaches to systematically amplify subgroup performance bias. These approaches allow for the evaluation and comparison of the effectiveness of bias mitigation methods on AI models by varying the degrees of bias, and can be applied to any classification model. We used these approaches to compare four off-the-shelf bias mitigation methods. Both amplification approaches promote the development of learning shortcuts in which the model forms associations between patient attributes and AI output. We demonstrate these approaches in a case study, evaluating bias in the determination of COVID status from chest x-rays. The maximum achieved increase in performance bias, measured as a difference in predicted prevalence, was 72% and 32% for bias between subgroups related to patient sex and race, respectively. These changes in predicted prevalence were not accompanied by substantial changes in the differences in subgroup area under the receiver operating characteristic curves, indicating that the increased bias is due to the formation of learning shortcuts, not a difference in ability to distinguish positive and negative patients between subgroups.

Details DOI

NeurIPS Conference 2023 Conference Paper

Knowledge-based in silico models and dataset for the comparative evaluation of mammography AI for a range of breast characteristics, lesion conspicuities and doses

Elena Sizikova
Niloufar Saharkhiz
Diksha Sharma
Miguel Lago
Berkman Sahiner
Jana Delfino
Aldo Badano

To generate evidence regarding the safety and efficacy of artificial intelligence (AI) enabled medical devices, AI models need to be evaluated on a diverse population of patient cases, some of which may not be readily available. We propose an evaluation approach for testing medical imaging AI models that relies on in silico imaging pipelines in which stochastic digital models of human anatomy (in object space) with and without pathology are imaged using a digital replica imaging acquisition system to generate realistic synthetic image datasets. Here, we release M-SYNTH, a dataset of cohorts with four breast fibroglandular density distributions imaged at different exposure levels using Monte Carlo x-ray simulations with the publicly available Virtual Imaging Clinical Trial for Regulatory Evaluation (VICTRE) toolkit. We utilize the synthetic dataset to analyze AI model performance and find that model performance decreases with increasing breast density and increases with higher mass density, as expected. As exposure levels decrease, AI model performance drops with the highest performance achieved at exposure levels lower than the nominal recommended dose for the breast type.

PDF Details

UAI Conference 2022 Conference Paper

Sequential algorithmic modification with test data reuse

Jean Feng
Gene Pennello
Nicholas Petrick
Berkman Sahiner
Romain Pirracchio
Alexej Gossmann

After initial release of a machine learning algorithm, the model can be fine-tuned by retraining on subsequently gathered data, adding newly discovered features, or more. Each modification introduces a risk of deteriorating performance and must be validated on a test dataset. It may not always be practical to assemble a new dataset for testing each modification, especially when most modifications are minor or are implemented in rapid succession. Recent work has shown how one can repeatedly test modifications on the same dataset and protect against overfitting by (i) discretizing test results along a grid and (ii) applying a Bonferroni correction to adjust for the total number of modifications considered by an adaptive developer. However, the standard Bonferroni correction is overly conservative when most modifications are beneficial and/or highly correlated. This work investigates more powerful approaches using alpha-recycling and sequentially-rejective graphical procedures (SRGPs). We introduce two novel extensions that account for correlation between adaptively chosen algorithmic modifications: the first leverages the correlation between consecutive modifications using flexible fixed sequence tests, and the second leverages the correlation between the proposed modifications and those generated by a hypothetical prespecified model updating procedure. In empirical analyses, both SRGPs control the error rate of approving deleterious modifications and approve significantly more beneficial modifications than previous approaches.

Details

JBHI Journal 2019 Journal Article

3-D Convolutional Neural Networks for Automatic Detection of Pulmonary Nodules in Chest CT

Aria Pezeshk
Sardar Hamidian
Nicholas Petrick
Berkman Sahiner

Deep two-dimensional (2-D) convolutional neural networks (CNNs) have been remarkably successful in producing record-breaking results in a variety of computer vision tasks. It is possible to extend CNNs to three dimensions using 3-D kernels to make them suitable for volumetric medical imaging data such as CT or MRI, but this increases the processing time as well as the required number of training samples (due to the higher number of parameters that need to be learned). In this paper, we address both of these issues for a 3-D CNN implementation through the development of a two-stage computer-aided detection system for automatic detection of pulmonary nodules. The first stage consists of a 3-D fully convolutional network for fast screening and generation of candidate suspicious regions. The second stage consists of an ensemble of 3-D CNNs trained using extensive transformations applied to both the positive and negative patches to augment the training set. To enable the second stage classifiers to learn differently, they are trained on false positive patches obtained from the screening model using different thresholds on their associated scores as well as different augmentation types. The networks in the second stage are averaged together to produce the final classification score for each candidate patch. Using this procedure, our overall nodule detection system called DeepMed is fast and can achieve 91% sensitivity at 2 false positives per scan on cases from the LIDC dataset.

Details DOI