The AHLI Conference on Health, Inference, and Learning (CHIL) solicits work across a variety of disciplines at the intersection of machine learning and healthcare. CHIL 2024 invites submissions focused on artificial intelligence and machine learning (AI/ML) techniques that address challenges in health, which we view broadly as including clinical healthcare, public health, population health, and beyond.
Specifically, authors are invited to submit 8-10 page papers (with unlimited pages for references) to one of 3 possible tracks: Models and Methods, Applications and Practice, or Impact and Society. Each track is described in detail below. Authors will select exactly one primary track when they register each submission, in addition to one or more sub-disciplines. Appropriate track and sub-discipline selection will ensure that each submission is reviewed by a knowledgeable set of reviewers. Track Chairs will oversee the reviewing process. In case you are not sure which track your submission fits under, feel free to contact the Track or Proceedings Chairs for clarification. The Proceedings Chairs reserve the right to move submissions between tracks if they believe that a submission has been misclassified.
Important Dates (all times in EST)
- Submissions due: Feb 5, 2024 at 11:59pm
- Bidding opens for reviewers: Feb 6, 2024 at 11:59pm
- Bidding closes for reviewers: Feb 9, 2024 at 11:59pm
- Reviews assigned: Feb 12, 2024 by 11:59pm
- Reviews due: Feb 26, 2024 at 11:59pm
- Reviews released: Mar 4, 2024 by 11:59pm
- Author/Reviewer discussion period: Mar 10-21, 2024
- Meta-review period: Mar 22-31, 2024
- Author notification: Apr 3, 2024 by 11:59pm
- CHIL conference: June 27-28, 2024
Submission Tracks
- Track 1 - Models and Methods: Algorithms, Inference, and Estimation
- Track 2 - Applications and Practice: Investigation, Evaluation, Interpretations, and Deployment
- Track 3 - Policy: Impact and Society: Policy, Public Health, Social Outcomes, and Economics
Evaluation
Works submitted to CHIL will be reviewed by at least 3 reviewers. Reviewers will be asked to primarily judge the work according to the following criteria:
Relevance: Is the submission relevant to health, broadly construed? Does the problem addressed fall into the domains of machine learning and healthcare?
Quality: Is the submission technically sound? Are claims well supported by theoretical analysis or experimental results? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work? Is the work complete rather than a work in progress?
Originality: Are the tasks, methods and results novel? Is it clear how this work differs from previous contributions? Is related work adequately cited to provide context? Does the submission contribute unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
Clarity: Is the submission clearly written? Is it well-organized? Does it adequately provide enough information for readers to reproduce experiments or results?
Significance: Is the contribution of the work important? Are other researchers or practitioners likely to use the ideas or build on them? Does the work advance the state of the art in a demonstrable way?
Final decisions will be made by Track and Proceedings Chairs, taking into account reviewer comments, ratings of confidence and expertise, and our own editorial judgment. Reviewers will be able to recommend that submissions change tracks or flag submissions for ethical issues, relevance and suitability concerns.
Submission Format and Guidelines
Submission Site
Submissions should be made via the online submission system (Coming soon!). At least one author of each accepted paper is required to register for, attend, and present the work at the conference in order for the paper to appear in the conference proceedings.
Length and Formatting
Submitted papers must be 8-10 pages (including all figures and tables). Unlimited additional pages can be used for references and additional supplementary materials (e.g. appendices). Reviewers will not be required to read the supplementary materials.
Required Sections
Similar to last year, two sections will be required: 1) Data and Code Availability, and 2) Institutional Review Board (IRB).
Data and Code Availability: This initial paragraph is required. Briefly state what data you use (including citations if appropriate) and whether the data are available to other researchers. If you are not sharing code, you must explicitly state that you are not making your code available. If you are making your code available, then at the time of submission for review, please include your code as supplemental material or as a code repository link; in either case, your code must be anonymized. If your paper is accepted, then you should de-anonymize your code for the camera-ready version of the paper. If you do not include this data and code availability statement for your paper, or you provide code that is not anonymized at the time of submission, then your paper will be desk-rejected. Your experiments later could refer to this initial data and code availability statement if it is helpful (e.g., to avoid restating what data you use).
Institutional Review Board (IRB): This endmatter section is required. If your research requires IRB approval or has been designated by your IRB as Not Human Subject Research, then for the cameraready version of the paper, you must provide IRB information (and at the time of submission for review, you can say that this IRB information will be provided if the paper is accepted). If your research does not require IRB approval, then you must state this to be the case. This section does not count toward the paper page limit.
Archival Submissions
Submissions to the main conference are considered archival and will appear in the published proceedings of the conference, if accepted. Author notification of acceptance will be provided by the listed date under Important Dates.
Preprint Submission Policy
Submissions to preprint servers (such as ArXiv or MedRxiv) are allowed while the papers are under review. While reviewers will be encouraged not to search for the papers, you accept that uploading the paper may make your identity known.
Peer Review
The review process is mutually anonymous (aka “double blind”). Your submitted paper, as well as any supporting text or revisions provided during the discussion period, should be completely anonymized (including links to code repositories such as Github). Please do not include any identifying information, and refrain from citing the authors’ own prior work in anything other than third-person. Violations of this anonymity policy at any stage before final manuscript acceptance decisions may result in rejection without further review.
Conference organizers and reviewers are required to maintain confidentiality of submitted material. Upon acceptance, the titles, authorship, and abstracts of papers will be released prior to the conference.
You may not submit papers that are identical, or substantially similar to versions that are currently under review at another conference or journal, have been previously published, or have been accepted for publication. Submissions to the main conference are considered archival and will appear in the published proceedings of the conference if accepted.
An exception to this rule is extensions of workshop papers that have previously appeared in non-archival venues, such as workshops, arXiv, or similar without formal proceedings. These works may be submitted as-is or in an extended form, though they must follow our manuscript formatting guidelines. CHIL also welcomes full paper submissions that extend previously published short papers or abstracts, so long as the previously published version does not exceed 4 pages in length. Note that the submission should not cite the workshop/report and preserve anonymity in the submitted manuscript.
Upon submission, authors will select one or more relevant sub-discipline(s). Peer reviewers for a paper will be experts in the sub-discipline(s) selected upon its submission.
Open Access
CHIL is committed to open science and ensuring our proceedings are freely available.
Responsible and Ethical Research
Computer software submissions should include an anonymized code link or code attached as supplementary material, licensing information, and provide documentation to facilitate use and reproducibility (e.g., package versions, README, intended use, and execution examples that facilitate execution by other researchers).
Submissions that include analysis on public datasets need to include appropriate citations and data sequestration protocols, including train/validation/test splits, where appropriate. Submissions that include analysis of non-public datasets need to additionally include information about data source, collection sites, subject demographics and subgroups statistics, data acquisition protocols, informed consent, IRB and any other information supporting evidence of adherence to data collection and release protocols.
Authors should discuss ethical implications and responsible uses of their work.
Coming Soon
Models and Methods:
Algorithms, Inference, and Estimation
Algorithms, Inference, and Estimation
Advances in machine learning are critical for a better understanding of health. This track seeks technical contributions in modeling, inference, and estimation in health-focused or health-inspired settings. We welcome submissions that develop novel methods and algorithms, introduce relevant machine learning tasks, identify challenges with prevalent approaches, or learn from multiple sources of data (e.g. non-clinical and clinical data).
Our focus on health is broadly construed, including clinical healthcare, public health, and population health. While submissions should be primarily motivated by problems relevant to health, the contributions themselves are not required to be directly applied to real health data. For example, authors may use synthetic datasets to demonstrate properties of their proposed algorithms.
We welcome submissions from many perspectives, including but not limited to supervised learning, unsupervised learning, reinforcement learning, causal inference, representation learning, survival analysis, domain adaptation or generalization, interpretability, robustness, and algorithmic fairness. All kinds of health-relevant data types are in scope, including tabular health records, time series, text, images, videos, knowledge graphs, and more. We welcome all kinds of methodologies, from deep learning to probabilistic modeling to rigorous theory and beyond.
Examples
Janizek, J. D., Erion, G., DeGrave, A. J., and Lee, S. " An adversarial approach for the robust classification of pneumonia from chest radiographs." Proceedings of the ACM Conference on Health, Inference, and Learning (CHIL), 2020.
Choi, E., Xiao, C., Stewart, W. F., and Sun, J. "MiME: Multilevel medical embedding of electronic health records for predictive healthcare." Advances in Neural Information Processing Systems. 2018.
Shalit, U., Johansson, F.D., and Sontag, D. "Estimating individual treatment effect: generalization bounds and algorithms." Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Futoma, J., Hariharan, S., and Heller, K. "Learning to detect sepsis with a multitask Gaussian process RNN classifier." Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Applications and Practice:
Investigation, Evaluation, Interpretation, and Deployment
Investigation, Evaluation, Interpretation, and Deployment
The goal of this track is to highlight works applying robust methods, models, or practices to identify, characterize, audit, evaluate, or benchmark ML approaches to healthcare problems. Additionally, we welcome unique deployments and datasets used to empirically evaluate these systems are necessary and important to advancing practice. Whereas the goal of Track 1 is to select papers that show significant algorithmic novelty, submit your work here if the contribution is describing an emerging or established innovative application of ML in healthcare. Areas of interest include but are not limited to:
- Datasets and simulation frameworks for addressing gaps in ML healthcare applications
- Tools and platforms that facilitate integration of AI algorithms and deployment for healthcare applications
- Innovative ML-based approaches to solving a practical problems grounded in a healthcare application
- Surveys, benchmarks, evaluations and best practices of using ML in healthcare
- Emerging applications of AI in healthcare
Introducing a new method is not prohibited by any means for this track, but the focus should be on the extent of how the proposed ideas contribute to addressing a practical limitation (e.g., robustness, computational scalability, improved performance). We encourage submissions in both more traditional clinical areas (e.g., electronic health records (EHR), medical image analysis), as well as in emerging fields (e.g., remote and telehealth medicine, integration of omics).
Examples
Johnson AE, Stone DJ, Celi LA, Pollard TJ. "The MIMIC Code Repository: enabling reproducibility in critical care research." Journal of the American Medical Informatics Association. 2017 Sep 27;25(1):32-9.
Wang, Xiaosong, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. "Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106. 2017.
Nestor B, McDermott MB, Boag W, Berner G, Naumann T, Hughes MC, Goldenberg A, Ghassemi M. "Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks." In Machine Learning for Healthcare Conference, pp. 381-405. PMLR, 2019.
Cha, Kenny H., Nicholas Petrick, Aria Pezeshk, Christian G. Graff, Diksha Sharma, Andreu Badal, and Berkman Sahiner. "Evaluation of data augmentation via synthetic images for improved breast mass detection on mammograms using deep learning." Journal of Medical Imaging 7, no. 1 (2020): 012703-012703.
Impact and Society:
Policy, Public Health, and Social Outcomes
Policy, Public Health, and Social Outcomes
Algorithms do not exist in a vacuum: instead, they often explicitly aim for important social outcomes. This track considers issues at the intersection of algorithms and the societies they seek to impact, specifically for health. Submissions could include methodological contributions such as algorithmic development and performance evaluation for policy and public health applications, large-scale or challenging data collection, combining clinical and non-clinical data, as well as detecting and measuring bias. Submissions could also include impact-oriented research such as determining how algorithmic systems for health may introduce, exacerbate, or reduce inequities and inequalities, discrimination, and unjust outcomes, as well as evaluating the economic implications of these systems. We invite submissions tackling the responsible design of AI applications for healthcare and public health. System design for the implementation of such applications at scale is also welcome, which often requires balancing various tradeoffs in decision-making. Submissions related to understanding barriers to the deployment and adoption of algorithmic systems for societal-level health applications are also of interest. In addressing these problems, insights from social sciences, law, clinical medicine, and the humanities can be crucial.
Examples
Obermeyer, Ziad, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. "Dissecting racial bias in an algorithm used to manage the health of populations." Science 366.6464 (2019): 447-453.
Zink, Anna, and Sherri Rose. "Fair regression for health care spending." Biometrics 76.3 (2020): 973-982.
Yang, Wanqian, Lars Lorch, Moritz Graule, Himabindu Lakkaraju, and Finale Doshi-Velez. "Incorporating interpretable output constraints in Bayesian neural networks." Advances in Neural Information Processing Systems 33 (2020): 12721-12731.
Pierson, Emma, David M. Cutler, Jure Leskovec, Sendhil Mullainathan, and Ziad Obermeyer. "An algorithmic approach to reducing unexplained pain disparities in underserved populations." Nature Medicine 27.1 (2021): 136-140.
Seyyed-Kalantari, Laleh, Haoran Zhang, Matthew BA McDermott, Irene Y. Chen, and Marzyeh Ghassemi. "Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations." Nature medicine 27.12 (2021): 2176-2182.
