The 31st IEEE International Requirements Engineering Conference (RE’23) will have an artifact evaluation track (AE). The AE track aims to foster reusability in the requirements engineering field. Through the AE track, researchers can actively contribute to open science in software engineering research.

An artifact includes (but not limited to) any dataset, tool, script, experimental protocol, codebook, or other executable or non-executable object produced by or used in the research.

Call for Artifacts

Gain more visibility, and get acknowledged for your contribution to the RE community!

Authors of accepted papers in RE’23 (Research, RE@Next!, and Industrial Innovation tracks) are encouraged to submit their artifact for evaluation. Research papers with accepted artifacts WILL receive a “Badge” on the front page of their paper in the proceedings.

NEW! We (re-)implement existing solutions for various reasons all the time! Get credit for your invested time and effort.

The AE track calls for a “revamp of existing artifacts” from the RE literature. Irrespective of having an accepted paper in RE’23, artifacts can be submitted for evaluation as long as they are derived from papers published previously in past editions of RE-related venues (e.g., IEEE RE conference, REFSQ conference or Requirements Engineering Journal). The submitted artifact can be an implementation/reimplementation or an upgrade of an existing artifact.

The motivation is to enlarge the set of shared artifacts within the RE community by including also those artifacts of state-of-the-art papers whose authors did not originally share any artifact (or did but the artifact was updated to meet the current research demands). Accepted artifacts in this case WILL NOT receive a “Badge”. Accepted artifacts in this case WILL NOT receive a “Badge”. Instead, the authors must submit a 2-page abstract describing the artifact. Upon the acceptance of the artifact, this abstract will appear in the proceedings.

Following the tradition in RE, ALL accepted artifacts will be presented during the conference.

Best Artifact Award

All accepted artifacts, whether they are associated with RE’23 or previous related papers or not, will compete for the best artifact award. The goal of the award is to recognize the effort of authors creating and sharing outstanding research artifacts. The best artifact will be selected by the program committee during the review process.

Dates
Tracks
You're viewing the program in a time zone which is different from your device's time zone change time zone

Wed 6 Sep

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

10:45 - 12:15
User RequirementsRE@Next! Papers / Journal-First / Research Papers at f128
Chair(s): Frank Houdek Mercedes-Benz AG
10:45
30m
Talk
User Driven Functionality Deletion for Mobile Apps
Research Papers
A: Maleknaz Nayebi Polytechnique Montréal, A: Konstantin Kuznetsov DFKI, Germany, A: Andreas Zeller CISPA Helmholtz Center for Information Security, A: Guenther Ruhe University of Calgary
11:15
30m
Talk
A Quality Framework for Measuring User Experience in Industrial IoT Systems
Journal-First
A: Adam Trendowicz , A: Eduard C. Groen Fraunhofer IESE, A: Jens Henningsen Fraunhofer IESE, A: Julien Siebert Fraunhofer IESE, A: Nedo Alexander Bartels Fraunhofer IESE, A: Sven Storck Fraunhofer IESE, A: Thomas Kuhn
11:45
30m
Paper
Boosting GUI Prototyping with Diffusion Models
RE@Next! Papers
A: Jialiang Wei IMT Mines Alès, A: Anne-Lise Courbis IMT Mines Alès, A: Thomas Lambolais IMT Mines Alès, A: Binbin Xu IMT Mines Alès, A: Pierre Louis Bernard University of Montpellier, A: Gerard Dray IMT Mines Alès
Pre-print
10:45 - 12:15
Formal RequirementsJournal-First / Industrial Innovation Papers / RE@Next! Papers at f142
Chair(s): Paola Spoletini Kennesaw State University
10:45
30m
Paper
Requirements Analysis of Variability Constraints in a Configurable Flight Software System
Industrial Innovation Papers
A: Chin Khor Iowa State University, A: Robyn Lutz Iowa State University
Pre-print
11:15
30m
Talk
The Role of Formalism in System Requirements
Journal-First
A: Jean-Michel Bruel IRIT (CNRS and Univ. of Toulouse)
Link to publication DOI
11:45
30m
Research paper
Understanding Fairness Requirements for ML-based Software
RE@Next! Papers
A: Luciano Baresi Politecnico di Milano, A: Chiara Criscuolo Politecnico di Milano, A: Carlo Ghezzi Politecnico di Milano
File Attached
13:45 - 15:15
Panel: Are we ready for the future? - Redefining RE to strengthen and sustain our community in the light of emerging topics and challenges Research Papers at b305
Chair(s): Anne Hess Fraunhofer IESE
13:45
90m
Panel
Panel: Are we ready for the future? - Redefining RE to strengthen and sustain our community in the light of emerging topics and challenges
Research Papers
M: Anne Hess Fraunhofer IESE, P: Jane Cleland-Huang University of Notre Dame, P: Bashar Nuseibeh The Open University (UK) & Lero (Ireland), P: Alistair Sutcliffe University of Manchester, P: Marcus Trapp , P: Irit Hadar University of Haifa
13:45 - 15:15
SafetyArtifacts / RE@Next! Papers / Journal-First at f128
Chair(s): Jaelson Castro Universidade Federal de Pernambuco
13:45
30m
Talk
Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System
Journal-First
A: Markus Borg CodeScene, A: Jens Henriksson Semcon, dept. Software and Emerging Tech, Gothenburg, A: Kasper Socha RISE Research Institutes of Sweden, A: Olof Lennartsson , A: Elias Sonnsjö , A: Thanh Bui RISE Research Institutes of Sweden , A: Piotr Tomaszewski RISE Research Institutes of Sweden, A: Sankar Raman Sathyamoorthy QRTECH, A: Sebastian Brink Combitech, A: Mahshid Helali Moghadam Scania R&D
Link to publication DOI File Attached
14:15
30m
Paper
ARCADE: a Framework for Integrated Management of Safety Assurance Information
RE@Next! Papers
A: Camilo Almendra Universidade Federal do Ceará, A: Carla Silva Universidade Federal de Pernambuco
Pre-print
14:45
30m
Talk
ILLOD Replication Package: An Open-Source Framework for Abbreviation-Expansion Pair Detection and Term Consolidation in RequirementsBest Artifact
Artifacts
A: Hussein Hasso Fraunhofer FKIE, A: Katharina Großer University of Koblenz, A: Iliass Aymaz Fraunhofer FKIE, A: Hanna Geppert Fraunhofer FKIE, A: Jan Jürjens University of Koblenz-Landau
DOI Pre-print File Attached
13:45 - 15:15
SustainabilityIndustrial Innovation Papers / RE@Next! Papers at f142
Chair(s): Martin Glinz University of Zurich
13:45
30m
Paper
Digital Design - Shaping a Sustainable Digital Future Requires a New Holistic Design Approach
Industrial Innovation Papers
A: Dominik Birkmeier MaibornWolff, A: Martina Beck MaibornWolff
14:15
30m
Paper
Application of the Sustainability Awareness Framework in Agile Software Development
Industrial Innovation Papers
A: Peter Bambazek Johannes Kepler University Linz, A: Iris Groher Johannes Kepler University, Linz, A: Norbert Seyff University of Applied Sciences and Arts Northwestern Switzerland FHNW
File Attached
14:45
30m
Paper
Requirements Engineering Knowledge as a Foundation for a Sustainability-Aware Scrum Framework
RE@Next! Papers
A: Peter Bambazek Johannes Kepler University Linz, A: Iris Groher Johannes Kepler University, Linz, A: Norbert Seyff University of Applied Sciences and Arts Northwestern Switzerland FHNW
File Attached
15:45 - 17:15
Agile & User StoriesResearch Papers / RE@Next! Papers / Journal-First at b305
Chair(s): Jean-Michel Bruel IRIT (CNRS and Univ. of Toulouse)
15:45
30m
Paper
Quantifying Requirements Technical Debt: A Systematic Mapping Study and a Conceptual Model
Research Papers
A: Judith Perera University of Auckland, A: Ewan Tempero University of Auckland, A: Yu-Cheng Tu , A: Kelly Blincoe University of Auckland
Pre-print Media Attached File Attached
16:15
30m
Talk
An Impact-Driven Approach to Predict User Stories Instability
Journal-First
A: Yarden Levy Ben-Gurion University of the Negev, A: Roni Stern , A: Arnon Sturm Ben-Gurion University of the Negev, A: Argaman Mordoch , A: Yuval Yarom
File Attached
16:45
30m
Paper
Handling Uncertainty in the Development of Conversational Assistants with RELAXed User Stories
RE@Next! Papers
A: Mariana Bonito Universidade Nova de Lisboa, A: João Araújo Universidade Nova de Lisboa - Portugal, A: Joao Magalhaes Universidade NOVA de Lisboa
15:45 - 17:15
15:45
30m
Paper
A Requirements-Driven Platform for Validating Field Operations of Small Uncrewed Aerial VehiclesArtifact Available
Research Papers
A: Ankit Agrawal Saint Louis University, Missouri, A: Bohan Zhang Saint Louis University, Missouri, A: Yashaswini Shivalingaiah Saint Louis University, Missouri, A: Michael Vierhauser University of Innsbruck, A: Jane Cleland-Huang University of Notre Dame
Pre-print Media Attached File Attached
16:15
30m
Paper
Leveraging Natural Language Processing for a Consistency Checking Toolchain of Automotive Requirements
Industrial Innovation Papers
A: Vincent Bertram RWTH Aachen University, A: Hendrik Kausch RWTH Aachen University, A: Evgeny Kusmenko RWTH Aachen University, A: Haron Nqiri RWTH Aachen University, A: Bernhard Rumpe RWTH Aachen University, A: Constantin Venhoff
16:45
30m
Paper
Managing Human Factors in Automated Vehicle Development: Towards Challenges and Practices
RE@Next! Papers
A: Amna Pir Muhammad , A: Eric Knauss Chalmers | University of Gothenburg, A: Jonas Bärgman Chalmers University of Technology, A: Alessia Knauss Veoneer Research, Sweden
15:45 - 16:45
Security & PrivacyJournal-First / Research Papers at f142
Chair(s): Seok-Won Lee Ajou University
15:45
30m
Paper
Mobile Application Privacy Risk Assessments from User-authored ScenariosArtifact Available
Research Papers
A: Tianjian Huang Carnegie Mellon University, A: Vaishnavi Kaulagi Carnegie Mellon University, A: Mitra Bokaei Hosseini University of Texas at San Antonio, A: Travis Breaux Carnegie Mellon University
Pre-print
16:15
30m
Talk
The Importance of Security is in the Eye of the Beholder: Cultural, Organizational, and Personal Factors Affecting the Implementation of Security by Design
Journal-First
A: Renana Arizon-Peretz University of Haifa, A: Irit Hadar University of Haifa, A: Gil Luria University of Haifa
17:30 - 18:00

Thu 7 Sep

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

10:45 - 12:15
Human AspectsJournal-First / RE@Next! Papers at b305
Chair(s): João Araújo Universidade Nova de Lisboa - Portugal
10:45
30m
Talk
Requirements Engineering Framework for Human-centered Artificial Intelligence Software Systems
Journal-First
A: Khlood Ahmad Deakin University, A: Mohamed Abdelrazek Deakin University, Australia, A: Chetan Arora Monash University, A: Arbind Agrahari Baniya Deakin University, A: Muneera Bano Swinburne University of Technology, Melbourne, A: John Grundy Monash University
Pre-print
11:15
30m
Paper
Multi-Modal Emotion Recognition for Enhanced Requirements Engineering: A Novel Approach
RE@Next! Papers
A: Ben Cheng School of Information Technology, Deakin University, A: Chetan Arora Monash University, A: Xiao Liu School of Information Technology, Deakin University, A: Thuong Hoang School of Information Technology, Deakin University, A: Yi Wang School of Information Technology, Deakin University, A: John Grundy Monash University
Pre-print
11:45
30m
Paper
Accountable Design for Individual, Societal, and Regulated Values in the UAV Domain
RE@Next! Papers
A: Agnieszka Marczak-Czajka , A: Jarek Nabrzyski University of Notre Dame, A: Jane Cleland-Huang University of Notre Dame
10:45 - 12:15
State of the PracticeIndustrial Innovation Papers / Journal-First / Research Papers at f128
Chair(s): Tong Li Beijing University of Technology
10:45
30m
Paper
Standardising and Coaching Requirements Engineering Methods at Fresenius Medical Care​
Industrial Innovation Papers
A: Jonas Esch Schaeffler Engineering GmbH, A: David Richter Fresenius Medical Care
File Attached
11:15
30m
Talk
The State-of-Practice in Requirements Specification: An Extended Interview Study at 12 Companies
Journal-First
A: Xavier Franch Universitat Politècnica de Catalunya, A: Cristina Palomares Polytechnic University of Catalonia, A: Carme Quer Universitat Politècnica de Catalunya, A: Panagiota Chatzipetrou , A: Tony Gorschek Blekinge Institute of Technology / DocEngineering
File Attached
11:45
30m
Paper
Eliciting the Double-edged Impact of Digitalisation: a Case Study in Rural AreasArtifact ReusableArtifact Available
Research Papers
A: Alessio Ferrari CNR-ISTI, A: Fabio Lepore University of Pisa, A: Livia Ortolani University of Pisa, A: Gianluca Brunori University of Pisa
Pre-print File Attached
10:45 - 12:15
AI & GPTResearch Papers / RE@Next! Papers at f142
Chair(s): Mehrdad Sabetzadeh University of Ottawa
10:45
30m
Paper
Prompting Creative Requirements via Traceable and Adversarial Examples in Deep LearningArtifact Available
Research Papers
A: Hemanth Gudaparthi Governors State University, A: Nan Niu University of Cincinnati, A: Boyang Wang University of Cincinnati, A: Tanmay Bhowmik Mississippi State University, A: Hui Liu Beijing Institute of Technology, A: Jianzhang Zhang , A: Juha Savolainen Danfoss, A: Glen Horton University of Cincinnati, A: Sean Crowe University of Cincinnati, A: Thomas Scherz University of Cincinnati, A: Lisa Haitz University of Cincinnati
11:15
30m
Paper
Zero-shot Learning for Named Entity Recognition in Software Specification DocumentsArtifact Available
Research Papers
A: Souvick Das , A: Novarun Deb Assistant Professor, Indian Institute of Information Technology, Vadodara, A: Agostino Cortesi Università Ca' Foscari Venezia, A: Nabendu Chaki
11:45
30m
Paper
Inconsistency Detection in Natural Language Requirements using ChatGPT: a Preliminary Evaluation}
RE@Next! Papers
A: Alessandro Fantechi University of Florence, A: Stefania Gnesi Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" , A: Lucia Passaro University of Pisa, A: Laura Semini Università di Pisa - Dipartimento di Informatica
File Attached
13:45 - 15:15
TraceabilityIndustrial Innovation Papers / Research Papers at b305
Chair(s): Jane Cleland-Huang University of Notre Dame
13:45
30m
Paper
All Eyes on Traceability: An Interview Study on Industry Practices and Eye Tracking PotentialArtifact Available
Research Papers
A: Maike Ahrens Leibniz Universität Hannover, A: Lukas Nagel Leibniz University Hannover
File Attached
14:15
30m
Talk
Taming Cross-Tool Traceability in the Wild
Industrial Innovation Papers
A: Cosmina-Cristina Ratiu Johannes Kepler University Linz, A: Christoph Mayr-Dorn JOHANNES KEPLER UNIVERSITY LINZ, A: Wesley Assunção North Carolina State University, A: Alexander Egyed Johannes Kepler University Linz
Pre-print Media Attached
14:45
30m
Paper
Traceability Evaluation in Requirements Engineering According to Automotive SPICE
Industrial Innovation Papers
A: Vishakha Rathod Technische Hochschule Ingolstadt, A: Thomas Cebulla AUDI AG, A: Stefan Kugele Technische Hochschule Ingolstadt
13:45 - 15:15
ModelingRE@Next! Papers / Research Papers at f142
Chair(s): Gunter Mussbacher McGill University
13:45
30m
Paper
Analysis and optimisation of SPL products using goal models
Research Papers
A: Inmaculada Ayala Universidad de Málaga, A: Mercedes Amor Universidad de Málaga, A: Lidia Fuentes Universidad de Málaga, Spain
Pre-print File Attached
14:15
30m
Paper
Discovering runtime requirements from user interactions: ideas and preliminary studies
RE@Next! Papers
A: Tong Li Beijing University of Technology, A: Xinran Zhang Beijing University of Technology, A: Yiting Wang
Media Attached File Attached
14:45
30m
Paper
Visualizations for User-supported State Space Exploration of Goal Models
RE@Next! Papers
A: Yesugen Baatartogtokh Smith College, A: Irene Foster Smith College, A: Alicia M. Grubb Smith College
Pre-print
15:45 - 17:15
Panel: Requirements Engineering and Large Language Models: Best of Friends or Worst of Enemies?Research Papers at b305
Chair(s): Neil Ernst University of Victoria
15:45
90m
Panel
Panel: Requirements Engineering and Large Language Models: Best of Friends or Worst of Enemies?
Research Papers
M: Neil Ernst University of Victoria, P: Fatma Başak Aydemir Boğaziçi University, P: Alessio Ferrari CNR-ISTI, P: Markus Borg CodeScene, P: Henning Wachsmuth Leibniz Universität Hannover, P: Walid Maalej University of Hamburg
Pre-print
15:45 - 17:15
Experimentation & EvaluationResearch Papers / RE@Next! Papers at f128
Chair(s): Travis Breaux Carnegie Mellon University
15:45
30m
Paper
An Experiment on the Effects of using Color to Visualize Requirements Analysis TasksArtifact ReusableArtifact Available
Research Papers
A: Yesugen Baatartogtokh Smith College, A: Irene Foster Smith College, A: Alicia M. Grubb Smith College
Pre-print
16:15
30m
Paper
Can Videos be Used to Communicate Non-functional Requirements? An Early Empirical Investigation
RE@Next! Papers
A: Sylwia Kopczyńska Poznan University of Technology, A: Jianwei Shi Leibniz University Hannover
16:45
30m
Paper
A Comparative Evaluation of Requirement Template SystemsArtifact Available
Research Papers
A: Katharina Großer University of Koblenz, A: Marina Rukavitsyna University of Koblenz, A: Jan Jürjens University of Koblenz-Landau
DOI Pre-print File Attached

Fri 8 Sep

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

10:45 - 12:15
RegulationsResearch Papers / RE@Next! Papers at b305
Chair(s): Grischa Liebel Reykjavik University
10:45
30m
Paper
A Transformer-based Approach for Abstractive Summarization of Requirements from Obligations in Software Engineering ContractsArtifact Available
Research Papers
A: Chirag Jain TCS Research, A: Preethu Rose Anish TCS Research, A: Amrita Singh TCS Research, A: Smita Ghaisas TCS Research
File Attached
11:15
30m
Paper
ML-based Compliance Verification of Data Processing Agreements against GDPR
Research Papers
A: Orlando Amaral University of Luxembourg, A: Sallam Abualhaija University of Luxembourg, A: Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland
Pre-print File Attached
11:45
30m
Paper
Towards Legal Contract Formalization with Controlled Natural Language Templates
RE@Next! Papers
A: Regan Meloche University of Ottawa, A: Daniel Amyot University of Ottawa, A: John Mylopoulos University of Ottawa
Pre-print
10:45 - 12:15
DataResearch Papers / Industrial Innovation Papers at f128
Chair(s): Sylwia Kopczyńska Poznan University of Technology
10:45
30m
Paper
A Data-Driven Approach for Finding Requirements Relevant Feedback from TikTok and YouTube
Research Papers
A: Manish Sihag University of Victoria, A: Ze Shi Li University of Victoria, A: Amanda Dash University of Victoria, A: Nowshin Nawar Arony University of Victoria, A: Kezia Devathasan University of Victoria, A: Neil Ernst University of Victoria, A: Alexandra Albu University of Victoria, A: Daniela Damian University of Victoria
11:15
30m
Paper
Synthesized data quality requirements and roadmap for improving reusability of in-situ marine data
Research Papers
A: Ngoc-Thanh Nguyen Western Norway University of Applied Sciences, A: Keila Lima Western Norway University of Applied Sciences, A: Astrid Marie Skalvik University of Bergen, Norway, A: Rogardt Heldal , A: Eric Knauss Chalmers | University of Gothenburg, A: Tosin Daniel Oyetoyan , A: Patrizio Pelliccione Gran Sasso Science Institute, L'Aquila, Italy, A: Camilla Sætre University of Bergen
File Attached
11:45
30m
Paper
Shaping a GAIA-X Data Ecosystem through Innovation Modeling
Industrial Innovation Papers
A: Ali Shakeri German Aerospace Center (DLR), A: Oliver Klemp German Aerospace Center (DLR), A: Bernd Westphal German Aerospace Center (DLR)
File Attached
10:45 - 12:15
NL ProcessingJournal-First / Industrial Innovation Papers at f142
Chair(s): Chetan Arora Monash University
10:45
30m
Paper
RClassify: Combining NLP and ML to Classify Rules from Requirements Specifications Documents
Industrial Innovation Papers
A: Asha Rajbhoj TCS Research, A: Padmalata Nistala , A: Ajim Pathan TCS Research, A: Piyush Kulkarni TCS Research, A: Vinay Kulkarni Tata Consultancy Services Research
11:15
30m
Paper
Requirements Classification for Smart Allocation: A Case Study in the Railway Industry
Industrial Innovation Papers
A: Sarmad Bashir RISE Research Institutes of Sweden, A: Muhammad Abbas RISE Research Institutes of Sweden, A: Alessio Ferrari CNR-ISTI, A: Mehrdad Saadatmand RISE Research Institutes of Sweden, A: Pernilla Lindberg Alstom
DOI Pre-print
11:45
30m
Talk
Enhanced Abbreviation-Expansion Pair Detection for Glossary Term Extraction
Journal-First
A: Hussein Hasso Fraunhofer FKIE, A: Katharina Großer University of Koblenz, A: Iliass Aymaz Fraunhofer FKIE, A: Hanna Geppert Fraunhofer FKIE, A: Jan Jürjens University of Koblenz-Landau
Link to publication DOI Pre-print File Attached
13:45 - 14:45
Most Influential PaperResearch Papers at b305
Chair(s): Daniel Amyot University of Ottawa
13:45
60m
Talk
User feedback in the AppStore: An empirical study
Research Papers
A: Dennis Pagano CQSE, A: Walid Maalej University of Hamburg
DOI Pre-print

Not scheduled yet

Not scheduled yet
Awards
Best Artifact
Artifacts

Eligibility and Evaluation Criteria

The purpose of this section is to communicate submission expectations to authors and reviewing guidelines for reviewers. Failure to meet these guidelines does not automatically mean rejection and adhering fully to these guidelines does not automatically mean acceptance. Ambiguity is certain to exist, so academic knowledge and skills must be used to fully consider the eligibility of submissions, and scientific integrity is key to a successful and amicable process.

The Badges

Like the previous edition of RE, there will be two badges: Available and Reusable. Available is awarded to publicly accessible artifacts with a DOI, with minimal documentation that ensures the runnability of the artifact. Reusable is awarded to well-documented artifacts that, facilitate reuse and replication. We deliberately removed the badges in the Validated category to focus on the current needs of the RE community, available and reusable artifacts.

The two badges build on each other. That is, an artifact that receives the Reusable badge needs to also fulfil the criteria for Available. We encourage the authors to apply to both badges. Exceptional cases due to confidentiality issues must be clearly explained by the authors.

Available Reusable
An artifact gets this badge only when it is permanently available for retrieval. The authors must place the artifact on a publicly accessible archival repository (such as Zenodo or FigShare). A DOI for the artifact is provided via these archival repositories and is referenced in both the article and artifact. An artifact is well-documented, exercisable, complete, and includes appropriate evidence of verification. The artifact with this badge should facilitate reuse and repurpose. Norms and standards of the research community for this artifact type should be strictly adhered to.

Submission Instructions for Authors

Applying for the badge “Available”

  • The artifact must be hosted online, considering the following criteria:
    • The URL to access the artifact is immutable (cannot be altered by the author). [Tip: Use Zenodo or FigShare, and avoid services like Dropbox, Google Drive, One Drive, and institutional websites, as they can easily change URLs and the data behind them.]
    • The artifact has a Digital Object Identifier (DOI) redirecting to the immutable URL. [Tip: If your artifact is on GitHub, follow these instructions to get a DOI for your code.]
  • The artifact must contain a README.md file summarizing the following content:
    • “Summary of Artifact” – Describe what the artifact does, the expected inputs and outputs, and the motivation for developing the artifact
    • “Authors Information” – List all authors and how to cite a work that uses this artifact Note: The AE track will employ a single-blind review. No need for the authors to anonymize their submissions.
    • “Artifact Location” – Describe at which URL (or DOI) the artifact can be obtained

  • The artifact must contain a LICENSE.md file showing the license of the artifact. The license should be a proper open-source license. If there exists a license file under a different name, the LICENSE.md file must point to the actual license.
  • Anyone must be able to access the artifact, without the need for registration.

Applying for the badge “Reusable”

  • Authors are strongly recommended to ask their colleagues to test the usability of their artifact on a fresh environment before submitting it.

  • In almost all cases, the artifact must fulfil ALL the criteria for the “Available” badge listed above. Note that if confidentiality issues prevent the authors from publicly sharing the artifact, the “Reusable” badge can still be awarded. However, a clear statement of the motivations for not sharing the artifact publicly shall be provided in the README.md file.
  • The artifact must contain an extended README.md file explaining the following content:
    • Same fields explained for the "Available" badge.
    • “Description of Artifact” - Describe each of the files in the artifact.
    • “System Requirements” (For automated analyses or tools) – state the required system, programs, and libraries needed to successfully run the artifact
    • “Installation Instructions” (For automated analyses or tools) – explain in detail how to run the artifact from scratch.
    • For automated analyses or tools, there is an expectation that the submitted artifacts can be run on any machine. In cases where this is not possible, it is the responsibility of the authors to provide virtual environments from which to run the artifacts. For example, Python Virtual environments, Docker envs, VirtualBox VMs, etc.
    • The artifact must be runnable within a maximum time of 60 minutes. If your installation time is longer than 60 minutes, you must make this clear in your Installation section and offer an explanation. Some scripts take a long time to produce results. In these cases, the authors must provide a minimum working example and the expected output. This can be done via a smaller dataset, intermediate script data saved by the authors, a truncated script, etc.
    • “Usage Instructions” – Explain (preferably with a running example) how the artifact can be used
      • For automated analyses or tools, this should include instructions on how to interact with the tool, API documentation, and all the information that enables other subjects to reuse the artifact.
      • For non-executable artifacts, as, e.g., interview guides, protocols, codebooks, data collected from qualitative studies, or datasets in general, this should include explanations on how the artifacts can be reused by other researchers or practitioners.
    • “Steps to Reproduce” (For automated analyses or tools) – provide instructions on how to generate the results presented in the paper. Known deviations from results presented in the paper should be explicitly outlined (e.g., when a table or figure is not produced, or the produced results are different from the results presented in the paper). The anticipated time for reproducing the results should not exceed 60 minutes. Otherwise, if reproduction time is longer, the authors must provide intermediate results that can be used to facilitate reproduction.

No Badge - A Revamp of existing artifacts

  • The artifact must fulfil ALL the criteria for either “Available” or “Reusable”.
  • Authors of updated artifacts can be the same as the ones created the original artifact or others.

What to submit

In the abstract field of EasyChair, submit text describing:

  • What badge is being applied for (even for previous submissions where the paper does not get a badge)
  • Why the badge is appropriate
  • Link to the repository with a readme page
  • Whether the submission is for an old or new paper
  • A link to the PDF or a supplementary material attachment of the PDF for the relevant old or new paper

For those submitting artifacts for older papers: also submit a 2-page summary describing how the artifact builds on the previous paper, or a previous artifact. This should be in IEEE format and submitted as a paper in easychair along with the information described above in the abstract field. Accepted 2-page summaries will appear in the proceedings (note: artifact with papers accepted in RE’23 already have a paper in the proceedings, so do not need this additional document)

How to submit

The review process will be conducted via the RE’23 AE Track EasyChair. Please submit at this link. Make sure you select “RE’23 Artifacts”.

After the submission, and before the notification date, the reviewers will interact with the authors using the Early Review Document of the artifact, and authors should be prepared to quickly reply to the reviewers. They may ask for updates to the artifact or clarifications. The goal is to allow the authors to fix minor issues and fully comply with the criteria of the AE Track.

The review process in the AE track involves thorough discussions to improve the accessibility and reusability of the artifact. Reviewers and authors will patiently work together to achieve this goal. The AE track will employ a single-blind review.

The review process has two primary objectives: encourage improvement of artifacts through proper documentation, and verification that the artifacts meet the aforementioned badge criteria. For this reason, the AE Track review is more of a a discussion, and less of a traditional conference review.

The review process will take place via Google Documents, for early review, and via EasyChair for the final review. Each submission will consist of a textual Abstract including information about the artifact. For each submission, the track chairs will create an associated Early Review Document where reviewers will interact with the authors to fix minor issues.

The entire review process is conducted over a two-week period. During this time, the reviewers will check the submitted artifacts against the badge guidelines. Reviewers are encouraged to start the review process early, as it can take time for reviewers and authors to sort out unforeseen issues in the artifacts. If reviewers encounter issues, or simply need clarifications, they will communicate via the Early Review Document. Authors must reply as soon as possible to ensure a timely review process.

Once the reviewers have checked all badge guidelines, and feel there is no more reasonable improvement that can be made by the authors, they will then submit their final review through EasyChair. We recommend a clear statement such as “Recommended Badges: Available” near the end of the full review. We expect meaningful reviews that help the authors improve their submissions. Such a review includes (but not limited to): (i) a summary of the artifact, its purpose, inputs, and outputs from the reviewers’ perspective, (ii) strengths and weakness; (iii) possible potential use of the artifact in RE tasks; and (iv) reasons for accepting or rejecting the artifact.