How Developers Locate Performance Bugs

sketch3 sketch1 sketch2


Background: Performance bugs can lead to severe issues regarding computation efficiency, power consumption, and user experience. Locating these bugs is a difficult task because developers have to judge for every costly operation whether runtime is consumed necessarily or unnecessarily. Objective: We wanted to investigate how developers, when locating performance bugs, navigate through the code, understand the program, and communicate the detected issues.

Method: We performed a qualitative user study observing twelve developers trying to fix documented performance bugs in two open source projects. The developers worked with a profiling and analysis tool that visually depicts runtime information in a list representation and embedded into the source code view.

Results: We identified typical navigation strategies developers used for pinpointing the bug, for instance, following method calls based on runtime consumption. The integration of visualization and code helped developers to understand the bug. Sketches visualizing data structures and algorithms turned out to be valuable for externalizing and communicating the comprehension process for complex bugs.

Conclusion: Fixing a performance bug is a code comprehension and navigation problem. Flexible navigation features based on executed methods and a close integration of source code and performance information support the process.


Navigate, Understand, Communicate: How Developers Locate Performance Bugs.
Sebastian Baltes, Oliver Moseler, Fabian Beck, and Stephan Diehl.
Proceedings of the 9th International Symposium on Empirical Software Engineering and Measurement (ESEM 2015).
Acceptance rate: 25% (20/81).
Preprint arXiv Slides BibTeX

In Situ Understanding Of Performance Bottlenecks Through Visually Augmented Code.
Fabian Beck, Oliver Moseler, Stephan Diehl, Gunter Daniel Rey.
Proceedings of the 21st International Conference on Program Comprehension (ICPC 2013).

Supplementary Material

  1. Tutorial: We provide the slides ( PDF) and the video ( MP4) we used in the tutorial phase of our study.

  2. Locating Bugs: We also provide supplementary material for each research question. The advices we prepared for each bug in case a team got stuck can be found here ( ZI­­­P). The questions we asked after each bug fixing session can be found on the slides ( PDF).

    • RQ1: Navigating and Understanding

      • RQ1.1: How was information from the profiling tool or other parts of the IDE used to locate the performance bug? Cross-case analysis (in German) ( XLSX)

      • RQ1.2: Is the in-situ visualization of the profiling data beneficial compared to a traditional list representation? Cross-case analysis (in German) ( XLSX)

      • RQ1.3: What navigation strategies do developers pursue to locate a specific performance bug? Interaction logs ( ZI­­­P) • Navigation visualizations ( ZI­­­P) • Screen recordings for Bug 3 (without audio because of confidentiality) ( Group 1, Group 2, Group 3, Group 4, Group 5, Group 6)

    • RQ2: Understanding and Communicating

  3. Questionnaire: The questionnaire that the participants filled out at the end of the study can be found here ( PDF).

Cite the dataset as:

How Developers Locate Performance Bugs — Supplementary Material.
Sebastian Baltes, Oliver Moseler, Fabian Beck, and Stephan Diehl.

The dataset is licensed under the Creative Commons Attribution 4.0 International License.

Bugs Used In The Study

  1. Apache Commons Collections:

  2. Guava (Google Core Libraries for Java):

    • #1155: Performance of ImmutableSet.contains.

The documents distributed on this website have been provided by the contributing authors by means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author’s copyright and the provided license. Not CC licensed works may not be reposted without the explicit permission of the copyright holder.