The AHLI Conference on Health, Inference, and Learning (CHIL) solicits work across a variety of disciplines, including machine learning, statistics, epidemiology, health policy, operations, and economics. CHIL 2023 invites submissions focussed on significant problems affecting health. Specifically, authors are invited to submit 8-10 page papers (with unlimited pages for references) to any of the tracks described below.
To ensure that all submissions to CHIL are reviewed by a knowledgeable and appropriate set of reviewers, the conference is divided into tracks and areas of interest. Authors will select exactly one primary track and area of interest when they register their submissions, in addition to one or more sub-disciplines.
Track Chairs will oversee the reviewing process. In case you are not sure which track your submission fits under, feel free to contact the Track or Proceedings Chairs for clarification. The Proceedings Chairs reserve the right to move submissions between tracks and/or areas of interest if the Proceedings believes that a submission has been misclassified.
Important Dates
- Submissions due:
Feb 3rd, 11:59 ESTFeb 15th, 11:59 EST - Author/Reviewer Discussion period:
March 10th - March 24thMarch 20th - April 3rd - Author notification:
April 3rdApril 10th - CHIL conference: June 22-24
Submission Tracks
- Track 1: Models and Methods
- Track 2: Applications and Practice
- Track 3: Policy: Impact and Society
Evaluation
Works submitted to CHIL will be reviewed by at least 3 reviewers. Reviewers will be asked to primarily judge the work according to the following criteria:
Relevance: All submissions to CHIL are expected to be relevant to health. Concretely, this means that the problem is well-placed into the relevant themes for the conference. Reviewers will be able to recommend that submissions change tracks or flag submissions that are not suitable for the venue as a whole.
Quality: Is the submission technically sound? Are claims well supported by theoretical analysis or experimental results? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? Is it clear how this work differs from previous contributions? Is related work adequately cited?
Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note: a superbly written paper provides enough information for an expert reader to reproduce its results.)
Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
Final decisions will be made by Track and Proceedings Chairs, taking into account reviewer comments, ratings of confidence and expertise, and our own editorial judgment.
Submission Format and Guidelines
Submission Site
Submissions should be made via OpenReview: https://openreview.net/group?id=chilconference.org/CHIL/2023/Conference . At least one author of each accepted paper is required to register for, attend, and present the work at the conference in order for the paper to appear in the conference proceedings.
Length and Formatting
Submitted papers must be 8-10 pages (including all figures and tables), plus unlimited pages for references. Additional supplementary materials (e.g., appendices) can be submitted with their main manuscript. Reviewers will not be required to read the supplementary materials.
Authors are required to use the LaTeX template: Download or Overleaf.
Required Sections
Similar to last year, two sections will be required: 1) Data and Code Availability, and 2) Institutional Review Board (IRB).
Data and Code Availability: This initial paragraph is required. Briefly state what data you use (including citations if appropriate) and whether the data are available to other researchers. If you are not sharing code, you must explicitly state that you are not making your code available. If you are making your code available, then at the time of submission for review, please include your code as supplemental material or as a code repository link; in either case, your code must be anonymized. If your paper is accepted, then you should de-anonymize your code for the camera-ready version of the paper. If you do not include this data and code availability statement for your paper, or you provide code that is not anonymized at the time of submission, then your paper will be desk-rejected. Your experiments later could refer to this initial data and code availability statement if it is helpful (e.g., to avoid restating what data you use).
Institutional Review Board (IRB): This endmatter section is required. If your research requires IRB approval or has been designated by your IRB as Not Human Subject Research, then for the cameraready version of the paper, you must provide IRB information (and at the time of submission for review, you can say that this IRB information will be provided if the paper is accepted). If your research does not require IRB approval, then you must state this to be the case. This section does not count toward the paper page limit.
Archival Submissions
Submissions to the main conference are considered archival and will appear in the published proceedings of the conference if accepted. Author notification of acceptance will be provided towards April 2023.
Preprint Submission Policy
Submissions to preprint servers (such as ArXiv or MedRxiv) are allowed while the papers are under review. While reviewers will be encouraged not to search for the papers, you accept that uploading the paper may make your identity known.
Peer Review
The review process is mutually anonymous. Please submit completely anonymized drafts. Please do not include any identifying information, and refrain from citing the authors’ own prior work in anything other than third-person. Violations to this policy may result in rejection without review.
Conference organizers and reviewers are required to maintain confidentiality of submitted material. Upon acceptance, the titles, authorship, and abstracts of papers will be released prior to the conference.
You may not submit papers that are identical, or substantially similar to versions that are currently under review at another conference or journal, have been previously published, or have been accepted for publication. Submissions to the main conference are considered archival and will appear in the published proceedings of the conference if accepted.
An exception to this rule is extensions of workshop papers that have previously appeared in non-archival venues, such as workshops, arXiv, or similar without formal proceedings. These works may be submitted as-is or in an extended form. CHIL also welcomes full paper submission that extend previously published short papers or abstracts, so long as the previously published version does not exceed 4 pages in length. Note that the submission should not cite the workshop/report and preserve anonymity in the submitted manuscript.
Open Access
CHIL is committed to open science and ensuring our proceedings are freely available.
Reviewing for CHIL 2023
Reviewing is a critical service in any research community, especially for the vibrant and growing community on machine learning for health that we are all part of. Every submission deserves thoughtful, constructive feedback that both (1) identifies quality work worth highlighting at this venue and (2) helps authors improve their work for either this venue or a future publication.
Here we provide instructions for reviewers to help us achieve these goals, across the 4 distinct phases of reviewing (Bidding, Assignment, Review, and Discussion).
Jump to: Timeline and Workload - bidding - assignment - reviewing - discussion
Timeline and Workload
To deliver high-quality reviews, you (a CHIL reviewer) are expected to participate in the 4 distinct phases of review:
Bidding period (2/4 - 2/7)- Skim abstracts and suggest >10 submissions you are interested in / qualified for
- Estimated time commitment: 1 hour or less
- Skim each of your assigned papers and report immediately
- Formatting violations
- Anonymity or Conflict of Interest violations
- Topics you are not qualified to review
- Workload: 2-5 papers per reviewer
- Estimated time commitment: 10 minutes per paper
- Deliver thoughtful review comments in timely fashion
- Workload: 2-5 papers per reviewer
- Estimated time commitment: 2-5 hours per paper, spread out asynchronously
- Provide comments that respond to author feedback, other reviewers, and chairs
- Workload: 2-5 papers per reviewer
- Estimated time commitment: 1-2 hours per paper, spread out asynchronously
Bidding Period Instructions
After all submissions are received on Feb. 3, you (a CHIL reviewer) will have the opportunity to review submitted titles/abstracts within OpenReview and indicate which papers you are most qualified for and excited about. Bidding instructions will be provided via email.
Please bid promptly (by 2/7) and bid generously.
Assignment Period Instructions
By 2/17, you (a CHIL reviewer) will be formally assigned 2-5 papers to review for CHIL. We ask for each reviewer to promptly (within 10 days) skim their assigned papers to ensure:
- no violations of required formatting rules (page limits, margins, etc)
- no violations of anonymity (author names, institution names, github links, etc)
- sufficient expertise to review the paper
If you feel that you cannot offer an informed opinion about the quality of the paper due to expertise mismatch, please write to your assigned Area Chair on OpenReview ASAP. Because of the diverse interests and finite availability of the research community, we cannot always reassign reviewers, but chairs will do our best to make sure each submission has the most competent reviewers available in the pool.
Reviewing Period Instructions
Between 2/17 and 3/14, you will be asked to complete thoughtful, constructive reviews for all assigned papers. Please make sure to complete your reviews by March 14th, 11:59 EST (earlier preferred). For each paper, you’ll fill out a form on OpenReview, similar to the form below:
Review format
- Summary of the paper
- Summarize *your* understanding of the paper. Stick to the facts: ideally, the authors should agree with everything written here.
- Strengths
- Identify the promising aspects of the work.
- Weaknesses
- Every paper that does not meet the bar for publication is the scaffolding upon which a better research idea can be built. If you believe the work is insufficient, help the authors see where they can take their work and how.
- If you are asking for more experiments, elucidate clearly why you are recommending the authors consider an experiment as well as what new information your suggested experiment might shed on the proposed method.
- Questions for the authors
- Communicate your potential misunderstandings of the work, to help authors craft a helpful and engaging response.
- Be explicit about how responses to each of your questions might change your score for the paper. Prioritize questions that would lead to big potential score changes.
Emergency Reviewing
To accommodate last minute emergencies, we will be seeking emergency reviewers for papers who do not receive all reviews by the deadline. Emergency reviewers will be sent a maximum of 3 papers by 3/15, and will need to write their reviews in a short time frame (between 3/15 and 3/18). Emergency review signup will be indicated in the reviewer signup form.
General Advice for preparing reviews
Follow the golden rule: be on time, provide polite and constructive reviews that you yourself would be happy to receive as an author. Be sure to review the paper, not the authors. When making statements about the paper, use phrases like “the paper proposes” rather than “the authors propose”. This makes your review less personal and separates critiques of the submission from critiques of the authors.
External resources:
- Remember Dennett's 4 rules for successful critical feedback
- Especially, you should not give criticism without first acknowledging strengths and what you have learned from the submission.
- ACL 2017 blogpost on advice for reviewers
- Mistakes reviewers make has some common errata to keep an eye out for
- Matthew McDermott has some useful advice on structuring reviews.
If you are a junior reviewer, there is no harm in asking a senior mentor or colleague to provide you with feedback on your review (though do not breach confidentiality).
Track specific advice for preparing reviews for a CHIL submission
- Track 1: it is acceptable for a paper to use synthetic data to evaluate a proposed method. Not every paper must touch real health data, though all methods should be primarily motivated by health applications and the realism of the synthetic data is fair to critique
- Track 2: the contribution of this track should be either more focused on solving a carefully motivated problem grounded in applications or on deployments or datasets that enable exploration and evaluation of applications
- Track 3: meaningful contributions to this track can include a broader scope of contribution beyond algorithmic development. Innovative and impactful use of existing techniques is encouraged
Discussion and Consensus-Building Period Instructions
Between 3/20 and 4/3, you (a CHIL reviewer) will be expected to participate in discussions on OpenReview by reading the authors’ response and comments from other reviewers, adding additional comments from your perspective, and updating your review accordingly.
We expect brief but thoughtful engagement from all reviewers here. For some papers, this would involve several iterations of feedback-response. A simplistic response of “I have read the authors’ response and I chose to keep my score unchanged” is not sufficient, because it does not provide detailed reasoning about what weaknesses are still salient and why the response is not sufficient. Please engage meaningfully!
Track Chairs will work with reviewers to try to reach a consensus decision about each paper by 4/3. In the event that consensus is not reached, Track Chairs make final decisions about acceptance.
Models and Methods:
Algorithms, Inference, and Estimation
Algorithms, Inference, and Estimation
Track Chairs
- Dr. Michael Hughes, Tufts University
- Dr. Yuyin Zhou, Univ. of California - Santa Cruz
- Dr. Jean Feng, Univ. of California - San Francisco
- Dr. Rahul Krishnan, University of Toronto
- Dr. Samantha Kleinberg, Stevens Institute of Technology
Description
Advances in machine learning are critical for a better understanding of health. This track seeks technical contributions in modeling, inference, and estimation in health-focused or health-inspired settings. We welcome submissions that develop novel methods and algorithms, introduce relevant machine learning tasks, identify challenges with prevalent approaches, or learn from multiple sources of data (e.g. non-clinical and clinical data).
Our focus on health is broadly construed, including clinical healthcare, public health, and population health. While submissions should be primarily motivated by problems relevant to health, the contributions themselves are not required to be directly applied to real health data. For example, authors may use synthetic datasets to demonstrate properties of their proposed algorithms.
We welcome submissions from many perspectives, including but not limited to supervised learning, unsupervised learning, reinforcement learning, causal inference, representation learning, survival analysis, domain adaptation or generalization, interpretability, robustness, and algorithmic fairness. All kinds of health-relevant data types are in scope, including tabular health records, time series, text, images, videos, knowledge graphs, and more. We welcome all kinds of methodologies, from deep learning to probabilistic modeling to rigorous theory and beyond.
Upon submission, authors will select one or more relevant sub-discipline(s). Peer reviewers for a paper will be experts in the sub-discipline(s) selected upon its submission, so please select your relevant disciplines judiciously.
Examples
Janizek, Joseph D., et al. "An adversarial approach for the robust classification of pneumonia from chest radiographs." Proceedings of the ACM Conference on Health, Inference, and Learning 2020.
Choi, Edward, et al. "MiME: Multilevel medical embedding of electronic health records for predictive healthcare." Advances in Neural Information Processing Systems. 2018.
McDermott, Matthew BA, et al. "Semi-supervised biomedical translation with cycle Wasserstein regression GANs." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
Shalit, Uri, Fredrik D. Johansson, and David Sontag. "Estimating individual treatment effect: generalization bounds and algorithms." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
Futoma, Joseph, Sanjay Hariharan, and Katherine Heller. "Learning to detect sepsis with a multitask Gaussian process RNN classifier." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
Neill, D.B., Moore, A.W., Sabhnani, M. and Daniel, K. "Detection of emerging space-time clusters." In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining (pp. 218-227). ACM
Applications and Practice:
Investigation, Evaluation, Interpretation, and Deployment
Investigation, Evaluation, Interpretation, and Deployment
Track Chairs
- Dr. Lifang He
- Dr. Tom Pollard
- Dr. Carl Yang
- Dr. Yu Zhang
Description
The goal of this track is to highlight works applying robust methods, models, or practices to identify, characterize, audit, evaluate, or benchmark systems. Additionally, unique deployments and datasets used to empirically evaluate these systems are necessary and important to advancing practice. Whereas the goal of Track 1 is to select papers that show significant technical novelty, submit your work here if the contribution is either more focused on solving a carefully motivated problem grounded in applications or on deployments or datasets that enable exploration and evaluation of applications. Introducing a new method is not prohibited by any means for this track, but the focus should be on methods which are designed to work particularly robustly (e.g., fail gracefully in practice), scale particularly well either in terms of computational runtime or data required, work across real-world data modalities and systems, etc. Contributions will be evaluated for technical rigor, robustness, and comprehensivity. We encourage applications and practice in both traditional and emerging clinical areas (e.g., models in electronic health records as well as applications in emerging fields such as remote and telehealth, integration of omics, etc.)
Computer software submissions should describe the intended use for the software, justify the need for the software, and provide executable examples for other researchers. Software submissions should directly support a healthcare application. Examples include code for summarizing the demographics of a study cohort, deriving meaningful clinical concepts from electronic health records, and natural language processing tools specifically designed for clinical text. All computer software submissions must be open source and released under a suitable open source license. Computer software should adhere to best practices in software development where possible, including the use of unit tests, continuous integration, and diligent documentation of component design and purpose.
We welcome submissions from a wide variety of perspectives, including but not limited to: examination of robustness of ML systems to real-world dataset shift or adversarial shift, scalable and safe machine learning/inference in clinical environments, new ML tools or comprehensive benchmarks for healthcare, development of scalable systems for processing data in practice, bridging the deployment gap, remote, wearable, and telehealth, data or software packages.
Upon submission, authors will select one or more relevant sub-discipline(s). Peer reviewers for a paper will be experts in the sub-discipline(s) selected upon its submission.
Examples
Pollard TJ, Johnson AE, Raffa JD, Mark RG. "tableone: An open source Python package for producing summary statistics for research papers." JAMIA Open. 2018 May 23;1(1):26-31.
Johnson AE, Stone DJ, Celi LA, Pollard TJ. "The MIMIC Code Repository: enabling reproducibility in critical care research." Journal of the American Medical Informatics Association. 2017 Sep 27;25(1):32-9.
Chen, Irene Y., Peter Szolovits, and Marzyeh Ghassemi. "Can AI help reduce disparities in general medical and mental health care?." AMA journal of ethics 21.2 (2019): 167-179.
Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, Marklund H, Haghgoo B, Ball R, Shpanskaya K, Seekins J. "Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison." arXiv preprint arXiv:1901.07031. 2019 Jan 21.
Wilson G, Aruliah DA, Brown CT, Hong NP, Davis M, Guy RT, Haddock SH, Huff KD, Mitchell IM, Plumbley MD, Waugh B. "Best practices for scientific computing." PLoS biology. 2014 Jan 7;12(1):e1001745.
Wei Q, Wang Z, Hong H, Chi Z, Feng DD, Grunstein R, Gordon C. "A residual based attention model for eeg based sleep staging." IEEE Journal of Biomedical and Health Informatics. 2020 Mar 3.
Nestor B, McDermott MB, Boag W, Berner G, Naumann T, Hughes MC, Goldenberg A, Ghassemi M. "Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks." In Machine Learning for Healthcare Conference, pp. 381-405. PMLR, 2019.
Finlayson SG, Bowers JD, Ito J, Zittrain JL, Beam AL, Kohane IS. "Adversarial attacks on medical machine learning." Science 363, no. 6433 (2019): 1287-1289.
Boag W, Suresh H, Celi LA, Szolovits P, Ghassemi M. "Racial disparities and mistrust in end-of-life care." In Machine Learning for Healthcare Conference, pp. 587-602. PMLR, 2018.
Henderson J, Ke J, Ho JC, Ghosh J, Wallace BC. "Phenotype instance verification and evaluation tool (PIVET): A scaled phenotype evidence generation framework using web-based medical literature." Journal of medical Internet research 20.5 (2018): e164.
Wang S, McDermott MB, Chauhan G, Ghassemi M, Hughes MC, Naumann T. "Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii." In Proceedings of the ACM conference on health, inference, and learning, pp. 222-235. 2020.
Xu Y, Biswal S, Deshpande SR, Maher KO, Sun J. "Raim: Recurrent attentive and intensive model of multimodal patient monitoring data." In Proceedings of the 24th ACM SIGKDD international conference on Knowledge Discovery & Data Mining, pp. 2565-2573, 2018.
Tonekaboni S, Joshi S, McCradden MD, Goldenberg A. "What clinicians want: contextualizing explainable machine learning for clinical end use." In Machine Learning for Healthcare Conference, pp. 359-380. PMLR, 2019.
Impact and Society:
Policy, Public Health, and Social Outcomes
Policy, Public Health, and Social Outcomes
Track Chairs
- Dr. Sanja Šćepanović
- Dr. Stephen Pfohl
- Dr. Dimitrios Spathis
Description
Algorithms do not exist in a vacuum: instead, they often explicitly aim for important social outcomes. This track considers issues at the intersection of algorithms and the societies they seek to impact, specifically for health. Submissions could include methodological contributions such as algorithmic development and performance evaluation for policy and public health applications, large-scale or challenging data collection, combining clinical and non-clinical data, as well as detecting and measuring bias. Submissions could also include impact-oriented research such as determining how algorithmic systems for health may introduce, exacerbate, or reduce inequities and inequalities, discrimination, and unjust outcomes, as well as evaluating the economic implications of these systems. In other words, we invite submissions tackling the responsible design of AI applications for healthcare and public health. System design for the implementation of such applications at scale is also welcome, which often requires balancing various tradeoffs in decision-making. Submissions related to understanding barriers to the deployment and adoption of algorithmic systems for societal-level health applications are also of interest. In addressing these problems, insights from social sciences, law, clinical medicine, and the humanities can be crucial.
We welcome papers from but not limited to following areas of interest: responsible AI for health, fairness, equity, ethics and justice, policy, public health, and societal impact of algorithms, interpretability, system design for implementation of ML at scale, regulatory frameworks, tools for the adoption of ML, evaluation of bias in legal and/or health contexts, and human-algorithm interaction.
Upon submission, authors will select one or more relevant sub-discipline(s). Peer reviewers for a paper will be experts in the sub-discipline(s) selected upon its submission.
Examples
Bolukbasi, Tolga, et al. "Quantifying and reducing stereotypes in word embeddings." arXiv preprint arXiv:1606.06121 (2016).
Obermeyer, Ziad, et al. "Dissecting racial bias in an algorithm used to manage the health of populations." Science 366.6464 (2019): 447-453.
Kleinberg, Jon, and Sendhil Mullainathan. "Simplicity creates inequity: implications for fairness, stereotypes, and interpretability." Proceedings of the 2019 ACM Conference on Economics and Computation. 2019.
Zink, Anna, and Sherri Rose. "Fair regression for health care spending." Biometrics 76.3 (2020): 973-982.
Yang, Wanqian, et al. "Incorporating interpretable output constraints in Bayesian neural networks." Advances in Neural Information Processing Systems 33 (2020): 12721-12731.
Bhatt, Umang, et al. "Explainable machine learning in deployment." Proceedings of the 2020 conference on fairness, accountability, and transparency. 2020.
Pierson, Emma, et al. "An algorithmic approach to reducing unexplained pain disparities in underserved populations." Nature Medicine 27.1 (2021): 136-140.
Seyyed-Kalantari, Laleh, et al. "Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations." Nature medicine 27.12 (2021): 2176-2182.
Panch, Trishan, Heather Mattie, and Leo Anthony Celi. "The “inconvenient truth” about AI in healthcare." NPJ digital medicine 2.1 (2019): 1-3.