(No) Influence of CI on the Commit Activity in GitHub Projects

GitHub vs. Travis


A core goal of Continuous Integration (CI) is to make small incremental changes to software projects, which are integrated frequently into a mainline repository or branch. This paper presents an empirical study that investigates if developers adjust their commit activity towards the above-mentioned goal after projects start using CI. We analyzed the commit and merge activity in 93 GitHub projects that introduced the hosted CI system Travis CI, but have previously been developed for at least one year before introducing CI. In our analysis, we only found one non-negligible effect, an increased merge ratio, meaning that there were more merging commits in relation to all commits after the projects started using Travis CI. This effect has also been reported in related work. However, we observed the same effect in a random sample of 60 GitHub projects not using CI. Thus, it is unlikely that the effect is caused by the introduction of CI alone. We conclude that: (1) in our sample of projects, the introduction of CI did not lead to major changes in developers’ commit activity, and (2) it is important to compare the commit activity to a baseline before attributing an effect to a treatment that may not be the cause for the observed effect.


(No) Influence of Continuous Integration on the Commit Activity in GitHub Projects.
Sebastian Baltes, Jascha Knack, Daniel Anastasiou, Ralf Tymann, and Stephan Diehl.
4th International Workshop on Software Analytics (SWAN 2018).
Acceptance rate: 64% (7/11).
Best paper award
Preprint arXiv Slides BibTeX

Supplementary Material

(No) Influence of Continuous Integration on the Commit Activity in GitHub Projects — Supplementary Material.
Sebastian Baltes.

(No) Influence of Continuous Integration on the Commit Activity in GitHub Projects — Dataset.
Sebastian Baltes and Jascha Knack.

These datasets are licensed under the Creative Commons Attribution 4.0 International License.

More Information

The git-log-extractor and git-log-parser tools we developed for this project are available on GitHub.

The documents distributed on this website have been provided by the contributing authors by means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author’s copyright and the provided license. Not CC licensed works may not be reposted without the explicit permission of the copyright holder.