# Detailed Information about the Database Search

## Search Criteria
The search criteria used in this systematic literature review includes the year of publication, the language and the type of literature [^1]'[^2].

### Publication Date
To determine from which year to search it can be helpful to know the date of the first article dealing with the specific topic [^3]. The first department dealing exclusively with Chaos Engineering was established at Netflix in 2015 [^4]. Therefore, all publications from 2015 onwards are included in this systematic literature review (SLR).

### Language
As Netflix is an American company only publications in English are examined. This corresponds to the standard of specialist publications and allows the majority of the available literature to be taken into account.

### Literature Type
Databases often do not take into account the frequently overlooked gray literature [^1]. However, for a complete SLR it is necessary to consider gray literature as well [^3]. Gray literature is not uniformly defined in the software engineering industry. This can lead to misunderstandings about the scope of the SLR. Therefore, this SLR uses the definition of gray literature according to the Prague definition [^5]. This states: “Grey literature stands for manifold document types produced on all levels of government, academics, business and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserved by library holdings or institutional repositories, but not controlled by commercial publishers (...)” [^6].
In order to comply with the Prague definition blog entries, online posts and websites are not recognized as grey literature in this SLR and are therefore not taken into account. Similarly, seminar papers, master's theses and bachelor's theses are excluded, while dissertations are included. Books are classified as so-called white literature in the context of an SLR in the field of software engineering and are therefore included in this paper [^5].

## Screening
In the second step "screening" the remaining literature is checked for relevance based on the abstract and content. Literature that does not contain the search terms “Chaos Engineering” or “chaos experiment(s)” is excluded. In addition to the criteria defined in the section "Search Criteria" the following further exclusion criteria are applied in the screening:
- Data set
- Document no longer available
- No access possible
- Cannot be found
- Patent
- Podcast material
- Podcast presentation
- Security risk when downloading
- Video material
- Webinar
- Citation

It is also checked whether the search terms appear in the correct context. The reviewed articles must refer to the process of Chaos Engineering originating from the software industry. Articles that deal with the search terms in a different context or only mention them casually and once are also discarded. From the 335 database search results 107 remain after the screening process.


[^1]: {Borrego, Maura; Foster, Margaret J.; Froyd, Jeffrey E. (2014): Systematic Literature Reviews in Engineering Education and Other Developing Interdisciplinary Fields, in: Journal of Engineering Education, 103. vol., 2014, no. 1, p. 45–76, DOI: 10.1002/jee.20038}
[^2]: {Fink, Arlene (2014): Conducting research literature reviews: From the internet to paper, 4. ed., Thousand Oaks (SAGE), 2014, ISBN: 9781452259499}
[^3]: {Kitchenham, Barbara; Charters, Stuart M. (2007): Guidelines for performing Systematic Literature Reviews in Software Engineering, Keele University and Durham University Joint Report, 2007}
[^4]: {Rosenthal, Casey; Jones, Nora (Ed.) (2020): Chaos Engineering: System Resiliency in Practice, 1. vol., Beijing et al. (O'Reilly) 2020, ISBN: 9781492043867}
[^5]: {Kitchenham, Barbara; Madeyski, Lech; Budgen, David (2023): How Should Software Engineering Secondary Studies Include Grey Material?, in: IEEE Transactions on Software Engineering, 49. vol., 2023, no. 2, p. 872–882, DOI: 10.1109/TSE.2022.3165938}
[^6]: {Schöpfel, Joachim (2010): Towards a Prague Definition of Grey Literature, in: Twelfth International Conference on Grey Literature : Transparency in Grey Literature, Grey Tech Approaches to High Tech Issues, Prague, Czech Republic, 2010, p. 11–26, retrieved from: https://opengrey.eu/item/display/10068/700015-2/}