The acceptance of empirical studies in software engineering and their contributions to increasing knowledge is continuously growing. The analytical research paradigm is not sufficient for investigating complex real life issues, involving humans and their interactions with technology. However, the overall share of empirical studies is negligibly small in computer science research; Sjøberg et al. (2005), found 103 experiments in 5,453 articles Ramesh et al. (2004) and identified less than 2% experiments with human subjects, and only 0.16% field studies among 628 articles. Further, existing work on empirical research methodology in software engineering has a strong focus on experimental research; the earliest by Moher and Schneider (1981), Basili et al. (1986), the first methodology handbook by Wohlin et al. (2000), and promoted by Tichy (1998). All have a tendency towards quantitative approaches, although also qualitative approaches are discussed during the later years, e.g. by Seaman (1999). There exist guidelines for experiments’ conduct (Kitchenham et al. 2002; Wohlin et al. 2000) and reporting (Jedlitschka and Pfahl 2005), measurements (Basili and Weiss 1984; Fenton and Pfleeger 1996; van Solingen and Berghout 1999), and systematic reviews (Kitchenham 2007), while only little is written on case studies in software engineering (Höst and Runeson 2007; Kitchenham et al. 1995; Wohlin et al. 2003) and qualitative methods (Dittrich 2007; Seaman 1999; Sim et al. 2001). Recently, a comprehensive view of empirical research issues for software engineering has been presented, edited by Shull et al. (2008).
The term “case study” appears every now and then in the title of software engineering research papers. However, the presented studies range from very ambitious and well organized studies in the field, to small toy examples that claim to be case studies. Additionally, there are different taxonomies used to classify research. The term case study is used in parallel with terms like field study and observational study, each focusing on a particular aspect of the research methodology. For example, Lethbridge et al. use field studies as the most general term (Lethbridge et al. 2005), while Easterbrook et al. (2008) call case studies one of five “classes of research methods”. Zelkowitz and Wallace propose a terminology that is somewhat different from what is used in other fields, and categorize project monitoring, case study and field study as observational methods (Zelkowitz and Wallace 1998). This plethora of terms causes confusion and problems when trying to aggregate multiple empirical studies.
The case study methodology is well suited for many kinds of software engineering research, as the objects of study are contemporary phenomena, which are hard to study in isolation. Case studies do not generate the same results on e.g. causal relationships as controlled experiments do, but they provide deeper understanding of the phenomena under study. As they are different from analytical and controlled empirical studies, case studies have been criticized for being of less value, impossible to generalize from, being biased by researchers etc. This critique can be met by applying proper research methodology practices as well as reconsidering that knowledge is more than statistical significance (Flyvbjerg 2007; Lee 1989). However, the research community has to learn more about the case study methodology in order to review and judge it properly.
Case study methodology handbooks are superfluously available in e.g. social sciences (Robson 2002; Stake 1995; Yin 2003) which literature also has been used in software engineering. In the field of information systems (IS) research, the case study methodology is also much more mature than in software engineering. For example, Benbasat et al. provide a brief overview of case study research in information systems (Benbasat et al. 1987), Lee analyzes case studies from a positivistic perspective (Lee 1989) and Klein and Myers do the same from an interpretive perspective (Klein and Myers 1999).
It is relevant to raise the question: what is specific for software engineering that motivates specialized research methodology? In addition to the specifics of the examples, the characteristics of software engineering objects of study are different from social science and also to some extent from information systems. The study objects are 1) private corporations or units of public agencies developing software rather than public agencies or private corporations using software systems; 2) project oriented rather than line or function oriented; and 3) the studied work is advanced engineering work conducted by highly educated people rather than routine work. Additionally, the software engineering research community has a pragmatic and result-oriented view on research methodology, rather than a philosophical stand, as noticed by Seaman (1999).
The purpose of this paper is to provide guidance for the researcher conducting case studies, for reviewers of case study manuscripts and for readers of case study papers. It is synthesized from general methodology handbooks, mainly from the social science field, as well as literature from the information systems field, and adapted to software engineering needs. Existing literature on software engineering case studies is of course included as well. The underlying analysis is done by structuring the information according to a general case study research process (presented in Section 2.4). Where different recommendations or terms appear, the ones considered most suited for the software engineering domain are selected, based on the authors’ experience on conducting case studies and reading case study reports. Links to data sources are given by regular references. Specifically, checklists for researchers and readers are derived through a systematic analysis of existing checklists (Höst and Runeson 2007), and later evaluated by PhD students as well as by members of the International Software Engineering Research Network and updated accordingly.
This paper does not provide absolute statements for what is considered a “good” case study in software engineering. Rather it focuses on a set of issues that all contribute to the quality of the research. The minimum requirement for each issue must be judged in its context, and will most probably evolve over time. This is similar to the principles by Klein and Myers for IS case studies (Klein and Myers 1999), “it is incumbent upon authors, reviewers, and exercise their judgment and discretion in deciding whether, how and which of the principles should be applied”. We do neither assess the current status of case study research in software engineering. This is worth a study on its own, similar to the systematic review on experiments by Sjøberg et al. (2005). Further, examples are used both to illustrate good practices and lack thereof.
This paper is outlined as follows. We first define a set of terms in the field of empirical research, which we use throughout the paper (Section 2.1), set case study research into the context of other research methodologies (Section 2.2) and discuss the motivations for software engineering case studies (Section 2.3). We define a case study research process (Section 2.4) and terminology (Section 2.5), which are used for the rest of the paper. Section 3 discusses the design of a case study and planning for data collection. Section 4 describes the process of data collection. In Section 5 issues on data analysis are treated, and reporting is discussed in Section 6. Section 7 discusses reading and reviewing case study report, and Section 8 summarizes the paper. Checklists for conducting and reading case study research are linked to each step in the case study process, and summarized in the Appendix.