Seminar: Replicability of Research Artifacts

This seminar will be held in English. However, if all participants are proficient in German, we can switch the language.

Contents

In this seminar, students will learn basic skills on how to conduct replicable scientific research, which will be useful in their studies and further academic career. They will learn how to perform a replicability study, document their work, write a report, and present their findings.

In particular, this seminar will follow the ACM classification on availability and replicability of artifacts published at major software engineering or programming language conferences such as ICSE, ESEC/FSE, ASE, MSR, ISSTA, PLDI, TACAS, POPL, OOPSLA, and ICFP. Artifact evaluation in the software engineering community was intoduced in 2011 at ESEC/FSE and since then has been added to many conferences in order to increase awareness for the importance of artifacts created by researchers. However, a recent study has shown that the criterias and processes are still heterogenious and further standardization is needed (check the study on this video).

Students will perform a replicability study on a given set of already published artifacts. Their main task is to verify if the artifacts are still available, read the publications, follow the documentation of the artifacts to replicate the results, and report the findings.

The seminar consists of 3 phases.

1. Reading lectures - four sessions (one per week) in which students will read and discuss relevant publications on artifacts evaluation to gain knowledge on the topic and relevant skills to perform the replicability.
2. Replicability - independent work in which each student is given a set of publications with artifacts. Students will document their independent work in a lab report.
3. Report and presentation - students will write a final report based on their experimental work documented in the lab report from phase 2. Additionally, they will prepare a 10 minutes presentation.

Mandatory sessions

The seminar will be comprised of mandatory sessions. In the remainder of the semester, students will work on their research topic independently, and refer to their mentors when necessary.

Attendance is mandatory for all students registered to the seminar.

The sessions are on the the following days (these are preliminary; confirmed dates by the beginning of summer semester):

• Meeting 1: Organizational matters
• Meeting 2: Reading lecture 1
• Meeting 3: Reading lecture 2
• Meeting 4: Reading lecture 3
• Meeting 5: Reading lecture 4
• Meeting 6: Online block seminar

During phase two there will be one optional Q&A session for students.

Registration is handled individually for the seminar. To apply please send a short paragraph on your motivation to attend the seminar. There is no need to make up a motivation here. "I need a seminar in my credit sheet" is a perfectly reasonable motivation. 

Send this paragraph also with your information (Matrikelnummer specifically) to
benjamin.hermanntu-dortmundde

You will then be registered to LSF.