Pediatric Bone Age Challenge

Organized by RSNA.organizing.committee - Current server time: Sept. 25, 2017, 4:58 p.m. UTC

Previous

Training
Aug. 5, 2017, midnight UTC

Current

Leaderboard
Sept. 1, 2017, midnight UTC

Next

Test
Oct. 7, 2017, midnight UTC

Goal

Develop an algorithm which can most accurately determine skeletal age on a validation set of pediatric hand radiographs.

Background

The Pediatric Bone Age Challenge will utilize three skeletal age datasets acquired from Stanford Children’s Hospital, Colorado Children’s Hospital and the University of California Los Angeles. A training set of hand radiographs and corresponding skeletal ages will be provided to the participants.

Description

Artificial intelligence (AI) algorithms have existed for decades, and have recently been propelled to the forefront of medical imaging research. Two factors are primarily responsible for this revolution: very powerful computer hardware available at relatively low cost, and the recognition that a certain types of algorithms are particularly well suited to image analysis. The latter discovery was made possible through the IMAGENET competition and represents the power of a fundamental transformation in research mechanics.

Currently, most research studies collect data, perform analysis, and publish results. The same researchers may continue to augment and expand the data set and perform subsequent analysis, with resulting publications. The data for each study is held quite closely, and outside of multi-center trials is rarely shared amongst institutions. Competitions are fundamentally different model of research: research data is made available to the public, usually with a baseline performance metric. Groups around the world are invited to analyze the data, and create algorithms to beat performance of the prior generation.

This model is so compelling in the rapidly growing data science community that a startup called Kaggle, which hosts ML datasets, competitions and algorithms was recently purchased by Google.

Purpose

RSNA Mission Statement: The RSNA promotes excellence in patient care and health care delivery through education, research and technologic innovation.

ML competitions align with the research and technologic innovation aspects of the RSNA mission.

Timeline

The availability of training and test data sets will adhere to the following schedule:

  • Training Data Available: Aug. 5
    Training Phase: Aug. 5 - Sept. 1
    Leaderboard Data Available: Sept. 1
    Leaderboard Phase: Sept. 1 - Oct. 7
    Test Data Available: Oct. 7
    Submission of Test Results: Oct. 7 - Oct. 15
    Review and Confirmation of Results and Notification of Awardees: Oct. 15
    Acknowledgment of Awardees at RSNA Annual Meeting: Monday, Nov. 27

Participants should submit a csv file with the case ID and the predicted age in months. A prediction should be provided for all cases in each phase.  A zip file containing this csv file as well as a text file describing the algorithm is uploaded as the participant's submission. An example test submission is available here

The primary evaluation measure is Mean Absolute Distance (MAD), calculated as the mean of the absolute values of the difference between the model estimates and those of the reference standard bone age.

In case of ties, the concordance correlation coefficient (CCC) will be used to break ties.

Terms and Conditions

  1. Entrants agree to hold harmless the organizers and their institutions, as well as RSNA and its staff and contractors for any costs, harm or damage incurred in the course of participating.
  2. Entrants retain all rights to algorithms and associated intellectual property they develop in the course of participating in the challenge.
  3. Entrants may make identified or pseudonymous submissions. A valid email must be associated with each submission and will be used to communicate with entrants.
  4. Results are to be submitted as a zip folder containing a single .csv file of submission data and an MS-Word or text document describing your research methods.
  5. Entrants may make multiple submissions up to the deadline date. The MedICI system will acknowledge successful submissions and provide error messages to help entrants make any fixes needed in submitted data sets.
  6. A data board will be made public on the day the challenge ends (Oct. 15).
  7. Entrants are required to submit a brief statement (2-3 paragraphs) describing the methods they used, including the size of the data set required for prediction. These statements may be used by the organizing committee in developing general publications about the challenge.
  8. Entrants may use only publicly available models trained on publicly available data sets. Entrants may not use other external data sets.
  9. Entrants may re-annotate images in the training set, but may not have human observers rate and evaluate the test data set.
  10. Entrants may use transfer learning. However, these models must be trained only on publicly available data (eg, GoogleNet).
  11. The availability of training and test data sets will adhere to the following schedule:

    Training Data Available: Aug. 5 
    Training Phase: Aug. 5 - Sept. 1
    Leaderboard Data Available: Sept. 1
    Leaderboard Phase: Sept. 1 - Oct. 7
    Test Data Available: Oct. 7
    Submission of Test Results: Oct. 7 - Oct. 15
    Review and Confirmation of Results and Notification of Awardees: Oct. 15
    Acknowledgment of Awardees at RSNA Annual Meeting: Monday, Nov. 27

  12. Awardees may be offered the opportunity to participate in research publications led by the Organizing Committee. Entrants may publish other research papers publications based on their participation in the Bone Age Challenge only with approval of the Organizing Committee.
  13. (added 9/20/17) 

    We ask that you personally abide by codes of honor and ethical behavior in your participation. Only one registration is allowed per participant and compliance will be monitored. During the Test phase, starting on October 7th, 2017, registration to the site will be closed and only existing participants/teams will be able to submit results (maximum of 3 submissions). Team creation will also be disabled during the Test phase. We view this competition as a unique opportunity to further contribute to the field of machine learning in medical imaging and we appreciate your most legitimate efforts to make this activity a great success.

 

Conflict of Interest:

Organizers of the challenge and groups that provided the data sets used are not allowed to participate as entrants in the challenge. 

Questions:

Questions about the terms and conditions of the challenge should be directed to the RSNA Informatics Department: informatics@rsna.org.

Organizers and Major Contributors:

The Radiological Society of North America (RSNA) Radiology Informatics Committee (RIC) Pediatric Bone Age Machine Learning Challenge Organizing Committee: 

  • Kathy Andriole, Massachusetts General Hospital

  • Brad Erickson, Mayo Clinic

  • Adam Flanders, Thomas Jefferson University

  • Safwan Halabi, Stanford University

  • Jayashree Kalpathy-Cramer, Massachusetts General Hospital

  • Marc Kohli, University of California - San Francisco

  • Luciano Prevedello, The Ohio State University

Data sets used in the Pediatric Bone Age Challenge have been contributed by Stanford University, the University of Colorado and the University of California - Los Angeles. 

The MedICI platform (built CodaLab) used for the challenge is provided by Jayashree Kalpathy-Cramer, supported through NIH grants (U24CA180927) and a contract from Leidos.

The challenge is coordinated by the RSNA. Questions about the challenge can be directed to the RSNA Informatics Department, informatics@rsna.org.

 

Training

Start: Aug. 5, 2017, midnight

Leaderboard

Start: Sept. 1, 2017, midnight

Test

Start: Oct. 7, 2017, midnight

Competition Ends

Oct. 15, 2017, midnight

You must be logged in to participate in competitions.

Sign In