Call for Artifact Submissions
The artifact evaluation track aims to review, promote, share, and catalog the research artifacts of accepted software engineering papers. Authors of papers accepted to the Technical/SEIP/NIER/SEET/SEIS tracks can submit an artifact for the Artifacts Available and Artifacts Evaluated badges. Authors of any prior SE work (published at ICSE or elsewhere) are also invited to submit their work for the Results Validated (replicated or reproduced) badges. Definitions for all badges can be found on ACM Artifact Review and Badging Version 1.1.
New this year: our primary goal will be to help authors make their artifacts available and reusable. To this end, we both strongly encourage authors to provide a clean image (Docker or similar) as part of their artifacts for any software components (see the preparation instructions), and will prioritize awarding the corresponding two badges, described below. To ensure that all submitted artifacts can be brought up to the standard of reusable, which requires high-quality documentation and structure, we will enable PC/Author discussions for the entire review period.
Available: Author-created artifacts relevant to this paper have been placed on a publicly accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided.
Reusable: The artifacts associated with the research are found to be complete, exercisable, and include appropriate evidence of verification and validation. In addition, they are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.
Dec 29, 2023: Artifact abstract deadline.
Jan 4, 2024: Artifact submissions deadline.
Jan 5 - Jan 23, 2024: review period (PC/authors discussion).
Jan 27, 2024: Notifications.
Best Artifact Awards
There will be two ICSE 2024 Best Artifact Awards to recognize the effort of authors creating and sharing outstanding research artifacts.
Submission for Evaluated and Available Badges
Only authors of papers accepted to the 2024 Technical/SEIP/NIER/SEET/SEIS tracks can submit candidate reusable or available artifacts. By the submission deadline, register your research artifact at the ICSE 2024 HotCRP site by submitting a 2 pages (max) abstract in PDF format describing your artifact. For the evaluated and available badges, authors must offer “download information” showing how reviewers can access and execute (if appropriate) their artifact. Authors must perform the following steps to submit an artifact:
Prepare the artifact
Make the artifact available
Document the artifact
Submit the artifact
1. Prepare the artifact
Both executable and non-executable artifacts may be submitted.
Executable artifacts consist of a tool or software system. For these artifacts, authors should prepare an installation package so that the tool can be installed and run in the evaluator’s environment. Following the instructions below, provide enough associated instruction, code, and data such that an average CS professional could build, install, and run the code within a reasonable time-frame. If installation and configuration requires more than 30 minutes, the artifact is unlikely to be accepted on practical grounds, simply because the PC will not have sufficient time to evaluate it.
When preparing executable packages for submission, we recommend vetting the artifact on a clean machine to confirm that it can be setup in a reasonable time frame. We strongly encourage authors to consider using a Docker (or VirtualBox VM) image for this process. Besides providing a clean environment to assess the installation instructions, the resulting image can be submitted as part of the artifact to allow quick replication. In particular, if the artifact contains or requires the use of a special tool or any other non-trivial piece of software, the authors must provide a VirtualBox VM image or a Docker container image with a working environment containing the artifact and all the necessary tools.
Non-executable artifacts only contain data and documents that can be used with a simple text editor, a PDF viewer, or some other common tool (e.g., a spreadsheet program in its basic configuration). These artifacts can be submitted as a single, optionally compressed package file (e.g., a tar, zip, or tar.gz file).
2. Make the artifact available
The authors need to make the packaged artifact available so that the PC can access it.
Artifacts must be made available via an archival repository, such as Software Heritage (see their submission guide), which provides long-term availability of software source code. Other often used solutions, more focused on long-term data archival, include Figshare and Zenodo. Please note that platforms that do not guarantee long-term archival, which presently includes GitHub, do not qualify.
3. Document the artifact
The authors need to write and submit documentation explaining how to obtain, unpack, and use their artifact in detail. The artifact submission must only describe the technicalities of the artifacts and uses of the artifact that are not already described in the paper. To this end, the submission should include the following documents (in Markdown, plain text, or PDF format) in a single archive file:
A README file describing what the artifact does, where it can be obtained, and providing clear instructions for how to repeat/replicate/reproduce the main results presented in the paper. Artifacts which focus on data should cover aspects related to understanding the context, data provenance, ethical and legal statements (as long as relevant), and storage requirements. Executable artifacts which focus on software should cover aspects relevant to how to install and use the software (and be accompanied by a small example).
A REQUIREMENTS file, for executable artifacts. This file should cover requirements on both the hardware (e.g., performance, storage or non-commodity peripherals) and software environments (e.g., Docker, VM, and operating system; package dependencies, etc.) for executing the software. Any deviation from standard environments needs to be reasonably justified.
An INSTALL file with installation instructions for executable artifacts. These instructions should include notes illustrating a basic usage example or a method to test the installation. For instance, it may describe what command to run and what output to expect to confirm that the code is installed and operational.
A STATUS file stating what kind of badge(s) the authors are applying for as well as the reasons why the authors believe that the artifact deserves that badge(s).
A LICENSE file describing the distribution rights. For submissions aiming for the Available badge, the license needs to ensure public availability. In the spirit of the ICSE Open Science Policy, we recommend adopting an open source license for executable artifacts and open data license for non-executable artifacts.
A copy of the accepted paper in pdf format.
4. Submit the artifact
By the abstract submission deadline (see important dates), register your research artifact at the ICSE 2024 HotCRP site by submitting an abstract describing your artifact. The abstract should include the paper title, the purpose of the research artifact, the badge(s) you are claiming, and the technology skills assumed by the reviewer evaluating the artifact. Please also mention if running your artifact requires any specific Operating Systems or other, unusual environments.
The PC may contact the authors, via the submission system, during the entire review period to request clarifications on the basic installation and start-up procedures or to resolve simple installation problems. Reviewers will be encouraged to attempt to execute submitted software artifacts early on, to minimize the time spent iterating on making the artifact functional and in turn provide enough time to ensure that all artifacts can be made reusable. Given the short review time available, the authors are expected to respond within a 72-hour period. Authors may update their research artifact after submission only for changes requested by reviewers during this time. Information on this phase is provided in the Submission and Reviewing Guidelines.
By submitting your artifact to an ACM Publication, you are hereby acknowledging that you and your co-authors are subject to all ACM Publications Policies, including ACM’s new Publications Policy on Research Involving Human Participants and Subjects. Alleged violations of this policy or any ACM Publications Policy will be investigated by ACM and may result in a full retraction of your paper, in addition to other potential penalties, as per ACM Publications Policy.
Please ensure that you and your co-authors obtain an ORCID ID, so you can complete the publishing process for your accepted paper. ACM has been involved in ORCID from the start and we have recently made a commitment to collect ORCID IDs from all of our published authors. The collection process has started and will roll out as a requirement throughout 2022. We are committed to improve author discoverability, ensure proper attribution and contribute to ongoing community efforts around name normalization; your ORCID ID will help in these efforts.
Further information will be constantly made available on the website https://conf.researchr.org/track/icse-2024/icse-2024-artifact-evaluation.
Please do not hesitate to contact the chairs for any questions.