The AHLI Conference on Health, Inference, and Learning (CHIL) solicits work across a variety of disciplines at the intersection of machine learning and healthcare. CHIL 2024 invites submissions focused on artificial intelligence and machine learning (AI/ML) techniques that address challenges in health, which we view broadly as including clinical healthcare, public health, population health, and beyond.
Specifically, authors are invited to submit 8-10 page papers (with unlimited pages for references) to one of 3 possible tracks: Models and Methods, Applications and Practice, or Impact and Society. Each track is described in detail below. Authors will select exactly one primary track when they register each submission, in addition to one or more sub-disciplines. Appropriate track and sub-discipline selection will ensure that each submission is reviewed by a knowledgeable set of reviewers. Track Chairs will oversee the reviewing process. In case you are not sure which track your submission fits under, feel free to contact the Track or Proceedings Chairs for clarification. The Proceedings Chairs reserve the right to move submissions between tracks if they believe that a submission has been misclassified.
Important Dates (all times are anywhere on Earth, AoE)
- Submissions due: Feb 16, 2024
- Bidding opens for reviewers: Feb 17, 2024
- Bidding closes for reviewers: Tue Feb 20, 2024
- Papers assigned to reviewers: Wed Feb 21, 2024
- Reviews due: Wed Mar 6, 2024
- Author response period: Mar 12-19, 2024
- Author / reviewer discussion period: Mar 19-26, 2024
- Decision notification: Apr 3, 2024
- CHIL conference: June 27-28, 2024
Submission Tracks
- Track 1 - Models and Methods: Algorithms, Inference, and Estimation
- Track 2 - Applications and Practice: Investigation, Evaluation, Interpretations, and Deployment
- Track 3 - Impact and Society: Policy, Public Health, Social Outcomes, and Economics
Evaluation
Works submitted to CHIL will be reviewed by at least 3 reviewers. Reviewers will be asked to primarily judge the work according to the following criteria:
Relevance: Is the submission relevant to health, broadly construed? Does the problem addressed fall into the domains of machine learning and healthcare?
Quality: Is the submission technically sound? Are claims well supported by theoretical analysis or experimental results? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work? Is the work complete rather than a work in progress?
Originality: Are the tasks, methods and results novel? Is it clear how this work differs from previous contributions? Is related work adequately cited to provide context? Does the submission contribute unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
Clarity: Is the submission clearly written? Is it well-organized? Does it adequately provide enough information for readers to reproduce experiments or results?
Significance: Is the contribution of the work important? Are other researchers or practitioners likely to use the ideas or build on them? Does the work advance the state of the art in a demonstrable way?
Final decisions will be made by Track and Proceedings Chairs, taking into account reviewer comments, ratings of confidence and expertise, and our own editorial judgment. Reviewers will be able to recommend that submissions change tracks or flag submissions for ethical issues, relevance and suitability concerns.
Submission Format and Guidelines
Submission Site
Submissions should be made via the online submission system: https://openreview.net/group?id=chilconference.org/CHIL/2024/Conference#tab-recent-activity. At least one author of each accepted paper is required to register for, attend, and present the work at the conference in order for the paper to appear in the conference proceedings.
Length and Formatting
Submitted papers must be 8-10 pages (including all figures and tables). Unlimited additional pages can be used for references and additional supplementary materials (e.g. appendices). Reviewers will not be required to read the supplementary materials.
Authors are required to use the LaTeX template: Overleaf
Required Sections
Similar to last year, two sections will be required: 1) Data and Code Availability, and 2) Institutional Review Board (IRB).
Data and Code Availability: This initial paragraph is required. Briefly state what data you use (including citations if appropriate) and whether the data are available to other researchers. If you are not sharing code, you must explicitly state that you are not making your code available. If you are making your code available, then at the time of submission for review, please include your code as supplemental material or as a code repository link; in either case, your code must be anonymized. If your paper is accepted, then you should de-anonymize your code for the camera-ready version of the paper. If you do not include this data and code availability statement for your paper, or you provide code that is not anonymized at the time of submission, then your paper will be desk-rejected. Your experiments later could refer to this initial data and code availability statement if it is helpful (e.g., to avoid restating what data you use).
Institutional Review Board (IRB): This endmatter section is required. If your research requires IRB approval or has been designated by your IRB as Not Human Subject Research, then for the cameraready version of the paper, you must provide IRB information (and at the time of submission for review, you can say that this IRB information will be provided if the paper is accepted). If your research does not require IRB approval, then you must state this to be the case. This section does not count toward the paper page limit.
Archival Submissions
Submissions to the main conference are considered archival and will appear in the published proceedings of the conference, if accepted. Author notification of acceptance will be provided by the listed date under Important Dates.
Preprint Submission Policy
Submissions to preprint servers (such as ArXiv or MedRxiv) are allowed while the papers are under review. While reviewers will be encouraged not to search for the papers, you accept that uploading the paper may make your identity known.
Peer Review
The review process is mutually anonymous (aka “double blind”). Your submitted paper, as well as any supporting text or revisions provided during the discussion period, should be completely anonymized (including links to code repositories such as Github). Please do not include any identifying information, and refrain from citing the authors’ own prior work in anything other than third-person. Violations of this anonymity policy at any stage before final manuscript acceptance decisions may result in rejection without further review.
Conference organizers and reviewers are required to maintain confidentiality of submitted material. Upon acceptance, the titles, authorship, and abstracts of papers will be released prior to the conference.
You may not submit papers that are identical, or substantially similar to versions that are currently under review at another conference or journal, have been previously published, or have been accepted for publication. Submissions to the main conference are considered archival and will appear in the published proceedings of the conference if accepted.
An exception to this rule is extensions of workshop papers that have previously appeared in non-archival venues, such as workshops, arXiv, or similar without formal proceedings. These works may be submitted as-is or in an extended form, though they must follow our manuscript formatting guidelines. CHIL also welcomes full paper submissions that extend previously published short papers or abstracts, so long as the previously published version does not exceed 4 pages in length. Note that the submission should not cite the workshop/report and preserve anonymity in the submitted manuscript.
Upon submission, authors will select one or more relevant sub-discipline(s). Peer reviewers for a paper will be experts in the sub-discipline(s) selected upon its submission.
Note: Senior Area Chairs (AC) are prohibited from submitting manuscripts to their respective track. Area Chairs (AC) who plan to submit papers to the track they were assigned to need to notify the Track Senior Area Chair (AC) within 24 hours of submission.
Open Access
CHIL is committed to open science and ensuring our proceedings are freely available.
Responsible and Ethical Research
Computer software submissions should include an anonymized code link or code attached as supplementary material, licensing information, and provide documentation to facilitate use and reproducibility (e.g., package versions, README, intended use, and execution examples that facilitate execution by other researchers).
Submissions that include analysis on public datasets need to include appropriate citations and data sequestration protocols, including train/validation/test splits, where appropriate. Submissions that include analysis of non-public datasets need to additionally include information about data source, collection sites, subject demographics and subgroups statistics, data acquisition protocols, informed consent, IRB and any other information supporting evidence of adherence to data collection and release protocols.
Authors should discuss ethical implications and responsible uses of their work.
Reviewing for CHIL
Reviewing is a critical service in any research community, and we highly respect the expertise and contributions of our reviewers. Every submission deserves thoughtful, constructive feedback that:
- Selects quality work to be highlighted at CHIL; and
- Helps authors improve their work, either for CHIL or a future venue.
To deliver high-quality reviews, you are expected to participate in four phases of review: Bidding; Assignment; Review; Discussion. This guide is here to help you through each of these steps. Your insights and feedback make a big difference in our community and in the field of healthcare and machine learning.
Timeline
To deliver high-quality reviews, you are expected to participate in the four phases of review:
- Bidding
- Skim abstracts
- Suggest >10 submissions that you feel qualified to review
- Time commitment: ~1 hour
- Assignment
- Skim your assigned papers and immediately report:
- Major formatting issues
- Anonymity or Conflict of Interest issues
- Papers that you are not comfortable reviewing
- Workload: 2-5 papers per reviewer
- Time commitment: ~10 minutes per paper
- Skim your assigned papers and immediately report:
- Review
- Deliver a thoughtful, timely review for each assigned paper
- Workload: 2-5 papers per reviewer
- Time commitment: ~2-5 hours per paper
- Discussion
- Provide comments that respond to author feedback, other reviewers, and chairs
- Workload: 2-5 papers per reviewer
- Time commitment: ~1-2 hours per paper
Phase 1: Bidding
After the submission deadline, you will be invited to "bid" for your preferred papers in OpenReview, based on titles and abstracts. Bidding instructions will be provided via email. Please bid promptly and generously!
Phase 2: Assignment
After the bidding period closes, you will be formally assigned 2-5 papers to review. We ask you to promptly skim your papers to ensure:
- no violations of required formatting rules (page limits, margins, etc)
- no violations of anonymity (author names, institution names, github links, etc)
- that you have sufficient expertise to review the paper
If you feel that you cannot offer an informed opinion about the quality of the paper due to expertise mismatch, please write to your assigned Area Chair on OpenReview. Area Chairs will do their best to ensure that each submission has the most competent reviewers available in the pool.
Phase 3: Review
You will be asked to complete thoughtful, constructive reviews for all assigned papers. Please ensure that your reviews are completed before the deadline, and sooner if possible. For each paper, you will fill out a form on OpenReview, similar to the form below. To help us to ensure consistency and quality, all reviews are subject to internal checks that may be manual or automated.
Review format
- Summary of the paper
- Summarize *your* understanding of the paper. Stick to the facts: ideally, the authors should agree with everything written here.
- Strengths
- Identify the promising aspects of the work.
- Weaknesses
- Every paper that does not meet the bar for publication is the scaffolding upon which a better research idea can be built. If you believe the work is insufficient, help the authors see where they can take their work and how.
- If you are asking for more experiments, clearly explain why and outline what new information the experiment might offer.
- Questions for the authors
- Communicate what additional information would help you to evaluate the study.
- Be explicit about how responses your questions might change your score for the paper. Prioritize questions that might lead to big potential score changes.
Emergency reviewing
We will likely be seeking emergency reviewers for papers that do not receive all reviews by the deadline. Emergency reviewers will be sent a maximum of 3 papers and will need to write their reviews in a short time frame. Emergency review sign-up will be indicated in the reviewer sign-up form.
General advice for preparing reviews
Please strive to be timely, polite, and constructive, submitting reviews that you yourself would be happy to receive as an author. Be sure to review the paper, not the authors.
When making statements, use phrases like “the paper proposes” rather than “the authors propose”. This makes your review less personal and separates critiques of the submission from critiques of the authors.
External resources
If you would like feedback on a review, we recommend asking a mentor or colleague. When doing so, take care not breach confidentiality. Some helpful resources include:
- Dennett's 4 rules for successful critical feedback. Especially, you should not give criticism without first acknowledging strengths and what you have learned from the submission.
- A blogpost with advice for reviewers from ACL 2017
- Common mistakes of reviewers.
- Useful advice on structuring reviews, collated by Matthew McDermott.
Track specific advice for preparing reviews for a CHIL submission
- Track 1: it is acceptable for a paper to use synthetic data to evaluate a proposed method. Not every paper must touch real health data, though all methods should be primarily motivated by health applications and the realism of the synthetic data is fair to critique
- Track 2: the contribution of this track should be either more focused on solving a carefully motivated problem grounded in applications or on deployments or datasets that enable exploration and evaluation of applications
- Track 3: meaningful contributions to this track can include a broader scope of contribution beyond algorithmic development. Innovative and impactful use of existing techniques is encouraged
Phase 4: Discussion
During the discussion period, you will be expected to participate in discussions on OpenReview by reading the authors’ responses and comments from other reviewers, adding additional comments from your perspective, and updating your review accordingly.
We expect brief but thoughtful engagement from all reviewers here. For some papers, this would involve several iterations of feedback-response. A simplistic response of “I have read the authors’ response and I chose to keep my score unchanged” is not sufficient, because it does not provide detailed reasoning about what weaknesses are still salient and why the response is not sufficient. Please engage meaningfully!
Track Chairs will work with reviewers to try to reach a consensus decision about each paper. In the event that consensus is not reached, Track Chairs make final decisions about acceptance.
Models and Methods:
Algorithms, Inference, and Estimation
Algorithms, Inference, and Estimation
Advances in machine learning are critical for a better understanding of health. This track seeks technical contributions in modeling, inference, and estimation in health-focused or health-inspired settings. We welcome submissions that develop novel methods and algorithms, introduce relevant machine learning tasks, identify challenges with prevalent approaches, or learn from multiple sources of data (e.g. non-clinical and clinical data).
Our focus on health is broadly construed, including clinical healthcare, public health, and population health. While submissions should be primarily motivated by problems relevant to health, the contributions themselves are not required to be directly applied to real health data. For example, authors may use synthetic datasets to demonstrate properties of their proposed algorithms.
We welcome submissions from many perspectives, including but not limited to supervised learning, unsupervised learning, reinforcement learning, causal inference, representation learning, survival analysis, domain adaptation or generalization, interpretability, robustness, and algorithmic fairness. All kinds of health-relevant data types are in scope, including tabular health records, time series, text, images, videos, knowledge graphs, and more. We welcome all kinds of methodologies, from deep learning to probabilistic modeling to rigorous theory and beyond.
Examples
Tang, S., Dunnmon, J. A., Qu, L., Saab, K. K., Baykaner, T., Lee-Messer, C., Rubin, D. L. Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Models. Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2023.
Zhu, J., Darnell, G., Kumar, A., Zhao, D., Li, B., Nguyen, X., Ren, S. Y. PhysioMTL: Personalizing Physiological Patterns using Optimal Transport Multi-Task Regression. Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2022.
Choi, E., Xiao, C., Stewart, W. F., Sun, J. MiME: Multilevel medical embedding of electronic health records for predictive healthcare. Advances in Neural Information Processing Systems (NeurIPS), 2018.
Shalit, U., Johansson, F.D., Sontag, D. Estimating individual treatment effect: generalization bounds and algorithms. Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Futoma, J., Hariharan, S., Heller, K. Learning to detect sepsis with a multitask Gaussian process RNN classifier. Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Applications and Practice:
Investigation, Evaluation, Interpretation, and Deployment
Investigation, Evaluation, Interpretation, and Deployment
The goal of this track is to highlight works applying robust methods, models, or practices to identify, characterize, audit, evaluate, or benchmark ML approaches to healthcare problems. Additionally, we welcome unique deployments and datasets used to empirically evaluate these systems are necessary and important to advancing practice. Whereas the goal of Track 1 is to select papers that show significant algorithmic novelty, submit your work here if the contribution is describing an emerging or established innovative application of ML in healthcare. Areas of interest include but are not limited to:
- Datasets and simulation frameworks for addressing gaps in ML healthcare applications
- Tools and platforms that facilitate integration of AI algorithms and deployment for healthcare applications
- Innovative ML-based approaches to solving a practical problems grounded in a healthcare application
- Surveys, benchmarks, evaluations and best practices of using ML in healthcare
- Emerging applications of AI in healthcare
Introducing a new method is not prohibited by any means for this track, but the focus should be on the extent of how the proposed ideas contribute to addressing a practical limitation (e.g., robustness, computational scalability, improved performance). We encourage submissions in both more traditional clinical areas (e.g., electronic health records (EHR), medical image analysis), as well as in emerging fields (e.g., remote and telehealth medicine, integration of omics).
Examples
Kinyanjui, N.M., Johansson, F.D. ADCB: An Alzheimer’s disease simulator for benchmarking observational estimators of causal effects. Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2022.
Zhou, H., Chen, Y., Lipton, Z. Evaluating Model Performance in Medical Datasets Over Time. Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2023.
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R. M. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Nestor B., McDermott, M.B., Boag W., Berner, G., Naumann, T., Hughes, M. C, Goldenberg, A., and Ghassemi, M. Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks. Machine Learning for Healthcare Conference (MLHC), 2019.
Cha, K. H., Petrick, N., Pezeshk, A., Graff, C. G., Sharma, D., Badal, A., Berkman Sahiner, B. Evaluation of data augmentation via synthetic images for improved breast mass detection on mammograms using deep learning. Journal of Medical Imaging (2020).
Impact and Society:
Policy, Public Health, and Social Outcomes
Policy, Public Health, and Social Outcomes
Algorithms do not exist in a vacuum: instead, they often explicitly aim for important social outcomes. This track considers issues at the intersection of algorithms and the societies they seek to impact, specifically for health. Submissions could include methodological contributions such as algorithmic development and performance evaluation for policy and public health applications, large-scale or challenging data collection, combining clinical and non-clinical data, as well as detecting and measuring bias. Submissions could also include impact-oriented research such as determining how algorithmic systems for health may introduce, exacerbate, or reduce inequities and inequalities, discrimination, and unjust outcomes, as well as evaluating the economic implications of these systems. We invite submissions tackling the responsible design of AI applications for healthcare and public health. System design for the implementation of such applications at scale is also welcome, which often requires balancing various tradeoffs in decision-making. Submissions related to understanding barriers to the deployment and adoption of algorithmic systems for societal-level health applications are also of interest. In addressing these problems, insights from social sciences, law, clinical medicine, and the humanities can be crucial.
Examples
Merrill, M., Safranchik, E., Kolbeinsson, A., Gade, P., Ramirez, E., Schmidt, L., Foschini, L., Althoff, T. Homekit2020: A Benchmark for Time Series Classification on a Large Mobile Sensing Datset with Laboratory Tested Ground Truth of Influenza Infections. Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2023.
Zhang, H., Dullerud, N., Roth, K., Oakden-Rainer, Pfohl, S., Ghassemi, M. Improving the Fairness of Chest X-ray Classifiers. Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2022. [Software]
Pierson, E., Cutler, D. M., Leskovec, J., Mullainathan, S., Ziad, O. An algorithmic approach to reducing unexplained pain disparities in underserved populations. Nature Medicine, 2021.
Seyyed-Kalantari, L., Zhang, H., McDermott M. A. B.,Chen, I. Y., Ghassemi, M. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in underserved patient populations. Nature Medicine, 2021.
Yang, W., Lorch, L., Graule, M., Lakkaraju, H., Doshi-Velez, F. Incorporating interpretable output constraints in Bayesian neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Obermeyer, Z., Powers, B., Vogeli, C., Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 2019.