Issues in Sampling Software Developers


Background: Reaching out to professional software developers is a crucial part of empirical software engineering research. One important method to investigate the state of practice is survey research. As drawing a random sample of professional software developers for a survey is rarely possible, researchers rely on various sampling strategies.

Objective: We report on our experience with different sampling strategies we employed, highlight ethical issues, and motivate the need to maintain a collection of key demographics about software developers to ease the assessment of the external validity of studies.

Method: Our report is based on data from two studies we conducted in the past.

Results: Contacting developers over public media proved to be the most effective and efficient sampling strategy. However, we not only describe the perspective of researchers who are interested in reaching goals like a large number of participants or a high response rate, but we also shed light onto ethical implications of different sampling strategies. We present one specific ethical guideline and point to debates in other research communities to start a discussion in the software engineering research community about which sampling strategies should be considered ethical.


Worse Than Spam: Issues In Sampling Software Developers.
Sebastian Baltes and Stephan Diehl.
Proceedings of the 10th International Symposium on Empirical Software Engineering and Measurement (ESEM 2016).
Acceptance rate: 37% (23/61).
Preprint arXiv Slides BibTeX

The documents distributed on this website have been provided by the contributing authors by means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author’s copyright and the provided license. Not CC licensed works may not be reposted without the explicit permission of the copyright holder.