Call for Artifacts

Authors of accepted papers in the research, RE@Next! and industry tracks of RE’22 are invited to submit an artifact to the RE Artifact Evaluation (AE) Track. Research papers with artifacts receive a “Badge” on the front page of their paper in the proceedings.

An artifact is considered as any dataset, tool, script, experimental protocol, codebook, or other executable or non-executable object produced by the research, or used in the research.

The track aims to promote and celebrate open science. Given the emerging nature of open-science and AE tracks in software engineering research, we encourage discussion and patience as a community when reviewing the submissions. More accepted artifacts are better, as long as the review process transforms those submissions into an acceptable state.

The review process is single-blind, so reviewers know the authors’ identity, but authors do not know the reviewers’ identity.

The Badges

There are two badges: Available and Reusable. Available is awarded to publicly accessible artifacts with a DOI, with minimal documentation. Reusable is awarded to well-documented artifacts that facilitate reuse and replication.

The two badges build on each other. That is, an artifact that receives the Reusable badge needs to also fulfill the criteria for Available, unless there are specific confidentiality issues that the authors need to explain.

Badges are loosely based on the ACM badges. We deliberately removed the badges in the Validated category this year, as we believe that the focus in the RE community should be on encouraging sharing of artifacts.

Artifacts Available	Artifacts Reusable
Open to RE’22 Submissions	Open to RE’22 Submissions
The artifacts associated with this research are permanently available for retrieval. Author-created artifacts relevant to this article have been placed on a publically accessible archival repository (such as Zenodo or FigShare). A DOI is provided via these archival repositories and is referenced in both the articles and the artifacts.	The artifacts associated with the research are documented, exercisable ( if applicable), and complete. The artifacts are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated.

Eligibility and Evaluation Criteria

Artifacts Available - Available Badge

Artifacts are hosted online.
Artifacts contain a README.md file containing the following sections and content:

“Summary of Artifact” - Why does this artifact exist and what does it do?
“Authors Information” - List all authors and how to cite work that uses this artifact.
“Artifact Location” - Describing under which DOI/URL the artifact can be obtained.

The organisation hosting the URL plans to maintain it for the foreseeable future.

Artifacts contain a LICENSE.md file containing the license used for the artifact. The license must be a proper open-source license.

Artifacts Reusable - Reusable Badge

Artifacts fulfill ALL the criteria for “Artifacts Available”.

Artifacts contain a README.md file containing the following sections and content:

“Summary of Artifact” - Why does this artifact exist and what does it do?
“Author Information” - List all authors and how to cite work that uses this artifact.
“Description of Artifact” - Describe each of the files, including what was not included.
“System Requirements” (Only for automated analyses or tools) - required system, programs, etc. to run the artifact.
“Installation Instructions” (Only for automated analyses or tools) - how to go from nothing to a running artifact.

For automated analyses or tools, there is an expectation that the submitted artifacts can be run on any machine. In cases where this is not possible, it is the responsibility of the authors to provide virtual environments from which to run the artifacts. For example, Python Virtual envs, Docker envs, VirtualBox VMs, etc.

Maximum reasonable installation time: 60 minutes.

If your installation time is longer than 60 minutes, you must make this clear in your Installation section and offer an explanation. Some scripts take a long time to produce results. In these cases, the authors must provide a minimum working example and the expected output. This can be done via a smaller dataset, intermediate script data saved by the authors, a truncated script, etc.

“Usage Instructions” - An explanation how the artifact can be used.

For automated analyses or tools, this should include instructions on how to interact with the tool, API documentation, and all the information that enables other subjects to reuse the artifact.
For non-executable artifacts, as, e.g., interview guides, protocols, codebooks, data collected from qualitative studies, or datasets in general, this should include explanations on how the artifacts can be reused by other researchers or practitioners.

“Steps to Reproduce” (Only for automated analyses or tools) - Instructions on how to generate the results presented in the article.

All known deviations from results presented in the article must be explicitly outlined (e.g., when a table or figure is not produced, or the produced results are different from the results presented in the paper).

Maximum reasonable reproduction time: 60 minutes

If reproduction time is longer, the authors must provide intermediate results that can be used to facilitate reproduction.

Summary: if you have developed a novel tool, or you are submitting scripts for analysis, follow the guidelines for Available, make sure your artifacts can be run by other subjects, and document the process for reuse and replication of your results. If you are submitting non-executable artifacts, make sure to clearly document them, and illustrate reuse scenarios.

What to Submit:

Authors are required to submit just an Abstract, in the appropriate EasyChair field (no submission file required):

The Abstract describes your submission. Required sections:
Requested Badges (requested badges must include qualification justification)
Artifact Location (link to an upload of your artifact online: Zenodo, FigShare, Github, institutional repo, etc.) If you like Github for open source collaboration, we strongly recommend making a release and integrating with Zenodo to create a permanent archive that is citable. See here for instructions. When creating the artifact, authors shall make sure that the requirements of the Call for Artifacts are fulfilled.
Pre-Print Location (reviewers need to check details in the pre-print)

Please make sure that your shared artifact contains the sections and content described in the Call for Artifacts.

How to Submit:

The review process will be conducted via the RE’22 AE Track EasyChair. Please submit at this link:

The review process has two primary objectives: encourage improvement of artifacts through proper documentation, and verification that the artifacts meet the aforementioned badge criteria. For this reason, the RE Artifact Evaluation Track review process is more of a discussion, and less of a traditional conference review.

The review process will take place via Google Documents, for early review, and via EasyChair for the final review. Each submission will consist of a textual Abstract including information about the artifact. For each submission, the track chairs will create an associated Early Review Document where reviewers will interact with the authors to fix minor issues.

The entire review process is conducted over a two-week period. During this time, the reviewers will check the submitted artifacts against the badge guidelines. Reviewers are encouraged to start the review process early, as it can take time for reviewers and authors to sort out unforeseen issues in the artifacts. If reviewers encounter issues, or simply need clarifications, they will communicate via the Early Review Document. Authors must reply as soon as possible to ensure a timely review process.

Once the reviewer has checked all badge guidelines, and feels there is no more reasonable improvement that can be made by the authors, they will then submit their final review through EasyChair. We recommend a clear statement such as “Recommended Badges: Available” near the end of the full review. Additional information in the review may include a summary of the artifact and compliments regarding the artifact itself.