Pediatric Bone Age Challenge

Organized by RSNA.organizing.committee - Current server time: Nov. 23, 2017, 11:50 p.m. UTC

Current

Test
Oct. 7, 2017, midnight UTC

Next

Leaderboard
Sept. 1, 2017, midnight UTC

Goal

Develop an algorithm which can most accurately determine skeletal age on a validation set of pediatric hand radiographs.

Background

The Pediatric Bone Age Challenge will utilize three skeletal age datasets acquired from Stanford Children’s Hospital and Colorado Children’s Hospital. A training set of hand radiographs and corresponding skeletal ages will be provided to the participants.

Description

Artificial intelligence (AI) algorithms have existed for decades, and have recently been propelled to the forefront of medical imaging research. Two factors are primarily responsible for this revolution: very powerful computer hardware available at relatively low cost, and the recognition that a certain types of algorithms are particularly well suited to image analysis. The latter discovery was made possible through the IMAGENET competition and represents the power of a fundamental transformation in research mechanics.

Currently, most research studies collect data, perform analysis, and publish results. The same researchers may continue to augment and expand the data set and perform subsequent analysis, with resulting publications. The data for each study is held quite closely, and outside of multi-center trials is rarely shared amongst institutions. Competitions are fundamentally different model of research: research data is made available to the public, usually with a baseline performance metric. Groups around the world are invited to analyze the data, and create algorithms to beat performance of the prior generation.

This model is so compelling in the rapidly growing data science community that a startup called Kaggle, which hosts ML datasets, competitions and algorithms was recently purchased by Google.

Purpose

RSNA Mission Statement: The RSNA promotes excellence in patient care and health care delivery through education, research and technologic innovation.

ML competitions align with the research and technologic innovation aspects of the RSNA mission.

Timeline

The availability of training and test data sets will adhere to the following schedule:

  • Training Data Available: Aug. 5
    Training Phase: Aug. 5 - Sept. 1
    Leaderboard Data Available: Sept. 1
    Leaderboard Phase: Sept. 1 - Oct. 7
    Test Data Available: Oct. 7
    Submission of Test Results: Oct. 7 - Oct. 15
    Review and Confirmation of Results and Notification of Awardees: Oct. 15
    Acknowledgment of Awardees at RSNA Annual Meeting: Monday, Nov. 27

Participants should submit a csv file with the case ID and the predicted age in months. A prediction should be provided for all cases in each phase.  A zip file containing this csv file as well as a text file describing the algorithm is uploaded as the participant's submission. An example test submission is available here

The primary evaluation measure is Mean Absolute Distance (MAD), calculated as the mean of the absolute values of the difference between the model estimates and those of the reference standard bone age.

In case of ties, the concordance correlation coefficient (CCC) will be used to break ties.

Terms and Conditions

  1. Entrants agree to hold harmless the organizers and their institutions, as well as RSNA and its staff and contractors for any costs, harm or damage incurred in the course of participating.
  2. Entrants retain all rights to algorithms and associated intellectual property they develop in the course of participating in the challenge.
  3. Entrants may make identified or pseudonymous submissions. A valid email must be associated with each submission and will be used to communicate with entrants.
  4. Results are to be submitted as a zip folder containing a single .csv file of submission data and an MS-Word or text document describing your research methods.
  5. Entrants may make multiple submissions up to the deadline date. The MedICI system will acknowledge successful submissions and provide error messages to help entrants make any fixes needed in submitted data sets.
  6. A data board will be made public on the day the challenge ends (Oct. 15).
  7. Entrants are required to submit a brief statement (2-3 paragraphs) describing the methods they used, including the size of the data set required for prediction. These statements may be used by the organizing committee in developing general publications about the challenge.
  8. Entrants may use only publicly available models trained on publicly available data sets. Entrants may not use other external data sets.
  9. Entrants may re-annotate images in the training set, but may not have human observers rate and evaluate the test data set.
  10. Entrants may use transfer learning. However, these models must be trained only on publicly available data (eg, GoogleNet).
  11. The availability of training and test data sets will adhere to the following schedule:

    Training Data Available: Aug. 5 
    Training Phase: Aug. 5 - Sept. 1
    Leaderboard Data Available: Sept. 1
    Leaderboard Phase: Sept. 1 - Oct. 7
    Test Data Available: Oct. 7
    Submission of Test Results: Oct. 7 - Oct. 15
    Review and Confirmation of Results and Notification of Awardees: Oct. 15
    Acknowledgment of Awardees at RSNA Annual Meeting: Monday, Nov. 27

  12. Awardees may be offered the opportunity to participate in research publications led by the Organizing Committee. Entrants may publish other research papers publications based on their participation in the Bone Age Challenge only with approval of the Organizing Committee.
  13. (added 9/20/17) 

    We ask that you personally abide by codes of honor and ethical behavior in your participation. Only one registration is allowed per participant and compliance will be monitored. During the Test phase, starting on October 7th, 2017, registration to the site will be closed and only existing participants/teams will be able to submit results (maximum of 3 submissions). Team creation will also be disabled during the Test phase. We view this competition as a unique opportunity to further contribute to the field of machine learning in medical imaging and we appreciate your most legitimate efforts to make this activity a great success.

 

Conflict of Interest:

Organizers of the challenge and groups that provided the data sets used are not allowed to participate as entrants in the challenge. 

Questions:

Questions about the terms and conditions of the challenge should be directed to the RSNA Informatics Department: informatics@rsna.org.

Organizers and Major Contributors:

The Radiological Society of North America (RSNA) Radiology Informatics Committee (RIC) Pediatric Bone Age Machine Learning Challenge Organizing Committee: 

  • Kathy Andriole, Massachusetts General Hospital

  • Brad Erickson, Mayo Clinic

  • Adam Flanders, Thomas Jefferson University

  • Safwan Halabi, Stanford University

  • Jayashree Kalpathy-Cramer, Massachusetts General Hospital

  • Marc Kohli, University of California - San Francisco

  • Luciano Prevedello, The Ohio State University

Data sets used in the Pediatric Bone Age Challenge have been contributed by Stanford University, the University of Colorado and the University of California - Los Angeles. 

The MedICI platform (built CodaLab) used for the challenge is provided by Jayashree Kalpathy-Cramer, supported through NIH grants (U24CA180927) and a contract from Leidos.

The challenge is coordinated by the RSNA. Questions about the challenge can be directed to the RSNA Informatics Department, informatics@rsna.org.

 

All Leaderboard Results

 

Dear RSNA Bone Age Challenge Participants:

Thank you all so much for your patience as we wrap up the first annual RSNA Machine Learning Challenge. With close to 300 participants from all over the world, this challenge exceeded all expectations. Despite some bumps along the road, we are grateful to all of the organizers and participants for their patience and engagement.

The dataset authors are also appreciative of all the efforts and extend their congratulations to all the participants.

Dataset Author Congratulatory Letter

Without further adieu...

CONGRATULATIONS TO THE RSNA BONE AGE CHALLENGE TOP 10!

User
MAD
CCC
s8t
4.265
0.991
ianpan
4.35
0.991
felipe.kitamura
4.382
0.99
hhthodberg
4.505
0.99
leonchen
4.527
0.991
amiper
4.555
0.99
rsnahandchallenge
4.561
0.991
grin
4.802
0.99
lbicfigraz
4.881
0.989
jcrayan
4.907
0.989

 

The Challenge winners will be officially notified in the next few days and will be invited to join us for the Challenge panel at the RSNA Annual Meeting in Chicago next month.  We hope that other participants will be able to join us for this session.  

We will continue to keep the challenge site open. We are working on making all the results available. Also, we will be creating a data use agreement and citing instructions for those of you that would like to use the dataset for future research efforts.  

Sincerely,

RSNA Bone Age Challenge Organizers

Training

Start: Aug. 5, 2017, midnight

Leaderboard

Start: Sept. 1, 2017, midnight

Test

Start: Oct. 7, 2017, midnight

Competition Ends

Oct. 17, 2018, 3:55 a.m.

You must be logged in to participate in competitions.

Sign In