SAS 2022 Artifacts

Call for Artifacts

As in previous years, we encourage authors to submit any artifacts and evaluations presented in the paper. The goal of the artifact submissions is to strengthen our field’s scientific approach to evaluations and reproducibility of results. Artifact submissions also serve as a stepping stone for future research, allowing researchers to rapidly build upon existing tools, and better evaluate or contrast existing work.

Important Dates:

• Artifact Submission: May 18, 2022 AoE

Are you running late in preparing your artifact? No worries! Deadline is extended to the 25th!

• Artifact Notification: July 15, 2022 AoE (along with the paper notification)

Criteria

This year, as last year, each artifact can be awarded up to three badges:

Validated

Artifacts which can reasonably reproduce the experimental claims in the paper. This is the minimum expectation from a submitted artifact.

Extensible

Artifacts which allow easy addition of new capabilities. To be eligible for an Extensible badge, the artifact must be bundled with the source code and pass the requirements for the Validated badge. Reviewers may attempt to tweak the source code and recompile, in order to evaluate how easy it is to build upon the tool. Please note that the artifact submission is optional. Submitting source code with the artifact is also optional. However, artifacts without source code will only be eligible for the Validated badge.

Available

Artifacts which obtain at least a Validated badge, will obtain the Available badge if the authors upload the submitted version of the artifact to the Zenodo SAS 2022 public repository. If your artifact received the Validated badge during the artifact evaluation, you can add your artifact to Zenodo SAS 2022 community by publishing your artifact here. The title and set of authors of your Zenodo submission must match with your SAS submission. The files you need to add are your artifact (virtual machine or container image) and the step-by-step instructions. After you publish your artifact, we will accept it to the community if the submission is OK, and we will award you the Available badge.

Packaging

Authors can package their artifact either as a virtual machine image or a Docker container image, and provide step-by-step instructions for loading/running the artifact.

• Virtual machine image: The VM image must be bootable and contain all the necessary libraries installed. Please ensure that the VM image can be processed with VirtualBox. While preparing your artifact, please make it as lightweight as possible. Consider installing ssh in the VM to make it reachable even when virtual screen isn’t working (common problem when changing host hardware).
• Container image: Authors can provide a zipped version of a container image containing the experiments, along with all dependencies. Please ensure that the zipped container image can be loaded using the docker load command. Please opt for container images only if your experiments can be executed and validated from the command line.
• Step-by-step instructions: This should clearly explain how to reproduce the results that support your paper’s conclusions. We encourage the authors to include easy-to-run scripts. Note that a reviewer may be evaluating your artifact on weaker hardware compared to your setup, so try to parameterize your scripts to allow running smaller versions of your experiments. Also, you should explain how to interpret the outputs of the artifact. If possible, please provide an estimate of the execution times for the experiments. Lastly, if you are providing the source code, try to provide an outline of how the code is structured.

Submission

Please follow the instructions below to submit your artifact:

• Package the VM/container image and the instruction document into a single compressed archive file using zip or gzip. Use your paper number for the name of the archive file.
• Upload the archive file to well-known storage service such as Dropbox or Google Drive and get the sharable link of it.
• Run a checksum function on the archive file. Create a text file that contains the link to the archive file and the checksum result.
• Submit the text file via the specific submission page on EasyChair.

Badges

The SAS Artifact Evaluation badges were designed by Arpita Biswas and Suvam Mukherjee, and are available for download from GitHub.