The ACM Conference on Health, Inference, and Learning (CHIL) solicits work across a variety of disciplines, including machine learning, statistics, epidemiology, health policy, operations, and economics. CHIL 2020 invites submissions touching on topics focused on relevant problems affecting health. Specifically, authors are invited to submit 8-10 page papers (with unlimited pages for references) to each of the tracks described below.
To ensure that all submissions to CHIL are reviewed by a knowledgeable and appropriate set of reviewers, the conference is divided into tracks and areas of interest. Authors will select exactly one primary track and area of interest when they register their submissions, in addition to one or more sub-disciplines.
Track chairs will oversee the reviewing process. In case you are not sure which track your submission fits under, feel free to contact the track or PC chairs for clarification. The PC Chairs reserve the right to move submissions between tracks and/or areas of interest if the PC believes that a submission has been misclassified.
- Submissions due – January 13, 2020 (11:59 pm AoE)
- Notification of Acceptance – Feb 12, 2020 (11:59 pm AoE)
- Camera Ready Due – March 6, 2020 (11:59 pm AoE)
- Conference Date – April 2-4, 2020
- Track 1: Machine Learning: Models, Algorithms, Inference, and Estimation
- Track 2: Applications: Investigation, Evaluation, and Interpretation
- Track 3: Policy: Impact, Economics, and Society
- Track 4: Practice: Deployments, Systems, and Datasets
These are called topics in the submission form. Authors should select one or more discipline(s) in machine learning for health (ML4H) from the following list when submitting their paper: benchmark datasets, distribution shift, transfer learning, population health, social networks, scaleable ML4H systems, natural language processing (NLP), computer vision, time series, bias/fairness, causality, *-omics, wearable-data, etc. Peer reviewers are assigned according to expertise in the sub-discipline(s) selected, so please choose your relevant topics carefully.
Work submitted to ACM CHIL will be reviewed by 3 reviewers within the broader field of machine learning for healthcare. Reviewers will be asked to primarily judge the work according to four criteria:
- Relevance: All submissions are expected to be relevant to health. Concretely, this means that the problem is well-placed into the relevant themes for the conference;
- Quality: The overall submission quality will be measured by the clarity, validity, comprehensiveness, and depth of scientific exploration, including the summarization of relevant field(s) and placement of the work within them;
- Novelty/Sophistication: Excellent submissions will display novel contributions in their problem domain, and demonstrate some form of sophistication.
- Suitability to Track: We will instruct reviewers to gauge whether works submitted are best suited to the track, or should be moved elsewhere.
Final decisions will be made in accordance to reviewer’s overall judgement, along with their subjective ratings of confidence/expertise, and according to our own editorial judgement.
Submission Format and Guidelines
Submissions should be made via the online submission system: https://chil2020.hotcrp.com/. At least one author of each accepted paper is required to register for, attend, and present the work at the conference in order for the paper to appear in the conference proceedings in the ACM Digital Library.
Length and Formatting
Submitted papers must be 8-10 pages (including all figures and tables), plus unlimited pages for references. Additional supplementary materials (e.g., appendices) can be submitted with their main manuscript. Reviewers will not be required to read the supplementary materials.
Papers should be formatted using the ACM Master Article Template and the reference format indicated therein. For LaTeX users, choose format=sigconf. An overleaf template is included here. ACM also makes a Word template available. Authors do not need to include terms, keywords, or other front matter in their submissions. Papers that are neither in ACM format or exceeding the specified page length, may be rejected without review.
Submissions to the main conference are considered archival and will appear in the published proceedings of the conference if accepted. Author notification of acceptance will be provided towards the end of February 2020.
The review process is double-blind. Please submit completely anonymized drafts. Please do not include any identifying information, and refrain from citing the authors’ own prior work in anything other than third-person. Violations to this policy may result in rejection without review.
Conference organizers and reviewers are required to maintain confidentiality of submitted material. Upon acceptance, the titles, authorship, and abstracts of papers will be released prior to the conference.
For accepted papers, authors will need to provide the following camera-ready materials by March 6:
- Metadata for the eRights system
- Submit forms for approval
- Final versions of papers via FTP
- Dual Submission Policy
You may not submit papers that are identical, or substantially similar to versions that are currently under review at another conference or journal, have been previously published, or have been accepted for publication. Submissions to the main conference are considered archival and will appear in the published proceedings of the conference if accepted.
An exception to this rule is extensions of workshop papers that have previously appeared in non-archival venues, such as workshops, arXiv, or similar without formal proceedings. These works may be submitted as-is or in an extended form. CHIL also welcomes full paper submission that extend previously published short papers or abstracts, so long as the previously published version does not exceed 4 pages in length. Note that the submission should not cite the workshop/report and preserve anonymity in the submitted manuscript.
ACM CHIL is committed to open science and ensuring our proceedings are freely available. The conference will make use of the ‘ACM Authorizer “Open Access” Service’ and ‘ACM OpenTOC Service’, allowing unrestricted access to individual papers as well as the overall proceedings, see here for more details.
ACM CHIL abides by ethics guidelines provided here: ACM Ethics guidelines.
Track 1: Machine Learning: Models, Algorithms, Inference, and Estimation
- Dr. Shakir Mohamed
- Dr. Sanmi Koyejo
- Dr. Adrian Dalca
- Dr. Uri Shalit
- Irene Chen
Advances in machine learning are critical for a better understanding of health. This track seeks contributions in modeling, inference, and estimation in health-focused or health-inspired settings. We welcome submissions that develop novel methods and algorithms, introduce relevant machine learning tasks, or identify challenges with prevalent approaches. Submissions focused more on health applications, for example establishing baselines or suggesting new evaluation metrics for assessing algorithmic advances are encouraged to submit to Track 2 instead.
While submissions should address problems relevant to health, the contributions themselves are not required to be directly applied to health. For example, authors may use synthetic datasets and experiments to demonstrate the properties of algorithms.
Authors may consider one or more machine learning sub-discipline(s) from the following list:
- Bayesian learning
- Causal inference
- Computer vision
- Deep learning architectures
- Evaluation methods.
- Knowledge graphs
- Natural language processing
- Reinforcement Learning
- Representation learning
- Robust learning
- Structured learning
- Supervised learning
- Survival analysis
- Time series
- Transfer learning
- Unsupervised learning
- Algorithmic Fairness
Authors may also consider sub-disciplines not listed here.
Upon submission, authors will select one or more relevant sub-discipline(s). Peer reviewers for a paper will be experts in the sub-discipline(s) selected upon its submission, so please select your relevant disciplines judiciously.
- Shalit, Uri, Fredrik D. Johansson, and David Sontag. “Estimating individual treatment effect: generalization bounds and algorithms.” Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
- Choi, Edward, et al. “MiME: Multilevel medical embedding of electronic health records for predictive healthcare.” Advances in Neural Information Processing Systems. 2018.
- McDermott, Matthew BA, et al. “Semi-supervised biomedical translation with cycle Wasserstein regression GANs.” Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
- Futoma, Joseph, Sanjay Hariharan, and Katherine Heller. “Learning to detect sepsis with a multitask Gaussian process RNN classifier.” Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR.org, 2017.
Track 2: Applications: Investigation, Evaluation, and Interpretation
- Dr. Tristan Naumann
- Dr. Andrew Beam
- Dr. Joyce Ho
- Matthew McDermott
- Dr. Shalmali Joshi
The goal of this track is to highlight works applying robust methods, models, or practices to identify, characterize, audit, evaluate, or benchmark systems. Whereas the goal of Track 1 is to select papers that show significant technical novelty, submit your work here if the contribution is more focused on solving a carefully motivated problem grounded in applications. Introducing a new method is not prohibited by any means for this track, but the focus should be on methods which are designed to work particularly robustly (e.g., fail gracefully in practice), scale particularly well either in terms of computational runtime or data required, work across real-world data modalities and systems, etc. Contributions will be evaluated for technical rigor, robustness, and comprehensiveness.
All areas of machine learning and all kinds of data within healthcare are amenable to this track. An example set of topics of interest and exemplar papers is shown below. These examples are by no means exhaustive and are meant as illustration and motivation.
Careful examinations of the robustness of ML systems to real-world dataset shift, adversarial shift, or on minority subpopulations.
- Nestor, Bret, et al. “Feature Robustness in Non-Stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks.” Proceedings of Machine Learning for Healthcare 2019 (MLHC ’19), 2019, https://www.mlforhc.org/s/Nestor.pdf.
- Finlayson, Samuel G., et al. “Adversarial attacks on medical machine learning.” Science 363.6433 (2019): 1287-1289.
Investigations into model performance on minority subpopulations, and the implications thereof.
- Boag, Willie, et al. “Racial Disparities and Mistrust in End-of-Life Care.” Machine Learning for Healthcare Conference. 2018. https://www.mlforhc.org/s/2.pdf
- Chen, Irene Y., Peter Szolovits, and Marzyeh Ghassemi. “Can AI Help Reduce Disparities in General Medical and Mental Health Care?.” AMA journal of ethics 21.2 (2019): 167-179. https://journalofethics.ama-assn.org/article/can-ai-help-reduce-disparities-general-medical-and-mental-health-care/2019-02
Scalable, safe machine learning / inference in clinical environments.
- Henderson, Jette, et al. “Phenotype instance verification and evaluation tool (PIVET): A scaled phenotype evidence generation framework using web-based medical literature.” Journal of medical Internet research 20.5 (2018): e164.
New tools or comprehensive benchmarks for machine learning for healthcare.
- Wang, Shirly, et al. “MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III.” Machine Learning for Healthcare, 2019.
Development of Scalable Systems for Processing Data in Practice (demonstrating, e.g., concern for multi-modality, runtime, robustness, etc., as guided by a clinical use case):
- Xu, Yanbo, et al. “Raim: Recurrent attentive and intensive model of multimodal patient monitoring data.” Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2018.
Bridging the deployment gap.
- Tonekaboni, Sana, et al. “What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use.” Machine Learning for Healthcare (2019)
Track 3: Policy: Impact, Economics, and Society
- Dr. Laura Rosella
- Dr. Ziad Obermeyer
- Dr. Avi Goldfarb
- Dr. Tom Pollard
- Dr. Rajesh Ranganath
- Dr. Rumi Chunara
Algorithms do not exist in the digital world alone: indeed, they often explicitly take aim at important social outcomes. This track considers issues at the intersection of algorithms and the societies they seek to impact. This track welcomes theoretical, methodological, and applied contributions for understanding and accounting for fairness, accountability, and transparency of algorithmic systems and for societal applications including mitigating discrimination, inequality, public health, health systems, policy applications, and other societal impacts from the deployment of such systems in real-world contexts. Given the societal implications of this area of focus, it includes work using data that fall out of traditional clinical data streams and includes health and non-health data sources including demographic data, online data streams, environmental and climate data. This includes the development of machine learning methods relevant to policy and public health, or new methods for working with data related for broader societal applications.
We welcome papers from various sub-disciplines (see list below). Paper submissions must indicate at least one area of interest (see list below) and at least one sub-discipline upon abstract registration.
Methods for combining non-clinical and clinical data; Understanding includes detecting and measuring how and which forms of bias are manifested in datasets and models; determining how algorithmic systems may introduce, exacerbate, or reduce inequities, discrimination and unjust outcomes; measuring the efficacy of existing techniques for explaining and interpreting automated decisions; evaluating perceptions of fairness and algorithmic bias. Accounting includes the governance of the design, development and deployment of algorithmic systems, which takes into consideration all stakeholders and interactions with socio-technical systems. Development of multi-level machine learning models (e.g. combining individual and population-level information); Mitigating includes introducing techniques for data collection and analysis and processing that measure, incorporate and acknowledge the selection bias and discrimination that may be present in datasets and models; formalizing fairness objectives based on notions from the social sciences, law, and humanistic studies; building socio-technical systems which incorporate these insights to minimize harm on historically disadvantaged communities and empower them; introducing methods for decision validation, correction and participation in co-designing algorithmic systems. Methods for generating spatial or temporal features relevant to health from noisy point observations.
- System design for implementation of ML at scale in healthcare: methods and techniques for evaluating computer systems within an existing regulatory framework, methods for establishing new regulatory guidelines, and tools for enabling adoption of ML within large healthcare organizations. Examples include evaluation of black-boxes and bias compared to a legal standard, and complementary intangible capital including training and processes.
- Methods to audit, measure, and evaluate fairness and bias: methods and techniques to check and measure the fairness (or unfairness) of existing computing systems and to assess associated risks. Examples include metrics and formal testing procedures to evaluate fairness, quantify the risk of fairness violations, or explicitly show tradeoffs.
- Examination of public health and policy and implications of machine learning in existing computing systems. Examples include the explanation of black-boxes, counterfactual and what-if reasoning.
- Methods for combining non-clinical and clinical data for population health applications.
- Methods involving human factors and humans-in-the-loop: methods and techniques that center on the human-machine relationship. Examples include visual analytics for fairness exploration, cognitive evaluation of explanations, and systems that combine human and algorithmic elements.
- Bolukbasi T, Chang K-W, Zou J, Saligrama V, Kalai A. 2016. Quantifying and reducing stereotypes in word embeddings. arXiv:1606.06121 [cs.CL]
- Chu KH, Colditz J, Malik M, Yates T, Primack B Identifying Key Target Audiences for Public Health Campaigns: Leveraging Machine Learning in the Case of Hookah Tobacco Smoking J Med Internet Res 2019;21(7):e12443
- Dranove, David, Chris Forman, Avi Goldfarb, and Shane Greenstein. 2014. “The Trillion Dollar Conundrum: Complementarities and Health Information Technology.” American Economic Journal: Economic Policy, 6 (4): 239-70.
- Neill, D.B., Moore, A.W., Sabhnani, M. and Daniel, K., 2005, August. Detection of emerging space-time clusters. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining (pp. 218-227). ACM.
Track 4: Practice: Deployments, Systems, and Datasets
- Dr. Leo Celi
- Dr. Stephanie Hyland
- Dr. Danielle Belgrave
- Dr. Katherine Heller
- Dr. Alistair Johnson
The transformation of healthcare through computational approaches is dependent on understanding how to empirically evaluate these systems, widely sharing tools for conducting research, and publicly accessible data allowing fair comparison of methods. This track seeks descriptions of the implementation or evaluation of informatics-based studies, computer software which has direct utility for medical researchers, and new datasets which support healthcare research.
Informatics based studies should primarily focus on evaluating these systems in clinical practice. Examples include applications of predictive modeling , deployment of a clinical decision support system , or evaluation of the impact of digital user interface modifications on routine practice .
Computer software submissions should describe the intended use for the software, justify the need for the software, and provide executable examples for other researchers. Software submissions should directly support a healthcare application. Examples include code for summarizing the demographics of a study cohort , deriving meaningful clinical concepts from electronic health records , and natural language processing tools specifically designed for clinical text [6, 7]. All computer software submissions must be open source and released under a suitable open source license. Computer software should adhere to best practices in software development where possible, including the use of unit tests, continuous integration, and diligent documentation of component design and purpose .
Descriptions of databases to support biomedical or health research are welcome. Dataset publications should focus on helping others reuse the data, rather than demonstrating any new insights or techniques. Datasets should include a full detailed description including the methods used to collect the data, the structure of records in the data, technical analyses supporting the quality of the data, and executable code demonstrating the use of the data. In terms of scope, we welcome datasets both large and small, so long as there is potential for a direct healthcare application. Datasets should be publicly available in an appropriate data repository with reasonable mechanisms for providing external researchers with access. Examples of suitable data repositories include but are not limited to Dryad, FigShare, PhysioNet, Synapse, or a university established data repository.
Evaluation of deployed applications, systems and software
- 1. Corey KM, Kashyap S, Lorenzi E, Lagoo-Deenadayalan SA, Heller K, Whalen K, Balu S, Heflin MT, McDonald SR, Swaminathan M, Sendak M. Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study. PLoS medicine. 2018 Nov 27;15(11):e1002701.
- 2. Henry, Katharine, et al. “Can septic shock be identified early? Evaluating performance of A targeted real-time early warning score (TREWScore) for septic shock in a community hospital: global and subpopulation performance.” D15. CRITICAL CARE: DO WE HAVE A CRYSTAL BALL? PREDICTING CLINICAL DETERIORATION AND OUTCOME IN CRITICALLY ILL PATIENTS. American Thoracic Society, 2017. A7016-A7016.
Openly available computer software that supports healthcare research
- 3. Ghassemi M, Pushkarna M, Wexler J, Johnson J, Varghese P. ClinicalVis: Supporting Clinical Task-Focused Design Evaluation. arXiv preprint arXiv:1810.05798. 2018 Oct 13.
- 4. Pollard TJ, Johnson AE, Raffa JD, Mark RG. tableone: An open source Python package for producing summary statistics for research papers. JAMIA Open. 2018 May 23;1(1):26-31.
- 5. Johnson AE, Stone DJ, Celi LA, Pollard TJ. The MIMIC Code Repository: enabling reproducibility in critical care research. Journal of the American Medical Informatics Association. 2017 Sep 27;25(1):32-9.
- 6. Peng Y, Wang X, Lu L, Bagheri M, Summers R, Lu Z. NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. AMIA Summits on Translational Science Proceedings. 2018;2018:188.
- 7. Wilson G, Aruliah DA, Brown CT, Hong NP, Davis M, Guy RT, Haddock SH, Huff KD, Mitchell IM, Plumbley MD, Waugh B. Best practices for scientific computing. PLoS biology. 2014 Jan 7;12(1):e1001745.
Publicly available medical, clinical, or otherwise health-related datasets
- 8. Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, Marklund H, Haghgoo B, Ball R, Shpanskaya K, Seekins J. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901.07031. 2019 Jan 21.
Call for Tutorials
- Irene Chen
- Dr. Andrew Beam
ACM CHIL 2019 will feature a select number of tutorials on topics related to health, inference, and learning. We will consider any topic, provided that the proposal makes a strong argument that the tutorial is important for the ACM CHIL community. Tutorials should be of interest to a substantial portion of the community and should represent a sufficiently mature area of research or practice.
Note that an ACM CHIL tutorial should not focus exclusively on the results or tools of the presenters or their organizations. Instead, a tutorial should provide a balanced overview of an area of research.
Each accepted tutorial will be approximately two hours long. We anticipate that there will be four to six tutorials, with sets of two or three running parallel.
Proposals should be no more than four pages in 12 point font submitted in PDF format. Each proposal should be clearly structured to provide the following information:
- Abstract (up to 250 words)
- Description and outline: What material will the tutorial cover and in what depth? Please provide a detailed outline.
- Goals: What are the objectives of the tutorial? What is the benefit to attendees? Why is this tutorial important to include at ACM CHIL?
- Target audience: What is the target audience? What background should attendees have?
- Presenters: Who are the presenters? Please provide names, affiliations, email addresses, and short bios for each presenter. Bios should cover the presenters’ expertise related to the topic of the tutorial. If there are multiple presenters, please describe how the time will be divided between them. All presenters listed in the proposal are expected to attend.
- A list of the most important references that will be covered.
- Previous tutorials: Has the tutorial (or a similar/highly related tutorial) been presented at another venue previously? If so, please list the dates and venues, and describe the similarities and differences between the previous tutorials and the proposed tutorial. If available, please include URLs for slides and video recordings.
- Links to video recordings of the presenters’ previous talks (optional, but extremely helpful)
The Tutorial Chairs welcome any questions at our email addresses: firstname.lastname@example.org and email@example.com. Tutorials will be considered on a rolling basis, so we encourage applicants to submit as early as possible.
Doctoral Symposium Call for PhD Students
- Dr Leo Anthony Celi
- Matthew McDermott
The 2020 ACM Conference on Health, Inference, and Learning (CHIL) is excited to announce our inaugural Doctoral Symposium. This event, targeted towards late-stage PhD students or recently graduated postdoctoral researchers working in machine learning for health or biomedicine, offers attendees an unparalleled opportunity for feedback, engagement with a community of peers and experienced researchers, stimulating discussion, and the possibility for new collaborations. The Doctoral Symposium will be held April 2, 2020, and consist of:
- Presentations by attending students on their thesis work
- Engaging discussions with other attendees and experienced researchers
- Individual meetings with researchers throughout the overall conference
- Peer feedback
- A current PhD student expected to graduate within 2 years or a current postdoctoral researcher who graduated no more than 1 year ago.
- Primary thesis research area within the machine learning and health-related fields
The application should consist of a single PDF file containing, in order:
- A CV, including a list of selected publications, if applicable.
- A one-page thesis summary, describing the overall goal, progress to date, and future plans.
- A one-page personal statement, outlining why you want to attend the Symposium and what you would add to the event in discussions, presentations, etc. Please be specific, outlining research problems you hope to discuss at the event, the kinds of experts you’d most hope to interact with and what kinds of questions you’d ask them, how you’d benefit from and drive positive interactions with other attendees, etc.
- A brief statement from your PhD advisor indicating that they support your attending the event.
You may use whatever format you wish, provided the thesis summary and personal statement are readable, with reasonable font sizes and margins.
Applications will be judged holistically according to clarity, completeness, evidence of significant research background and progress on ongoing future directions, our evaluation of how impactful attending the event will be on your future research directions as well as how impactful your attendance will be on other attendees.
Submit Your Application
To submit an application, fill out this form: https://forms.gle/9zAXAgSKtnoyBm8f7 by 11:59 p.m. February 23, 2020, AoE.
- Submission Deadline: February 23, 2020
- Notification Date: March 1, 2020
- Symposium Date: April 2, 2020
Contact and organization
Please contact firstname.lastname@example.org with “Doctoral Symposium” in the subject line for any questions.