High Level Theory

Towards a Theory of Replications, Open Science, and Reproducibility

The literature on the reproducibility crisis presents several putative causes for the proliferation of irreproducible results, including p-hacking and publication bias. Without a theory of reproducibility, however, it is difficult to determine whether these putative causes can explain most irreproducible results. Drawing from an historically-informed conception of science that is open and collaborative, we identify the components of an idealized experiment and analyze these components as a precursor to develop such a theory. Openness, we suggest, has long been intuitively proposed as a solution to irreproducibility. However, this intuition has not been validated in a theoretical framework. We use probabilistic arguments and examine how openness of experimental components relates to reproducibility of results. We show that there are some impediments to obtaining reproducible results that precede many of the causes often cited in literature on the reproducibility crisis. That is, even if, for example, p-hacking never took place, publication bias was absent, and other erroneous individual- or system-level practices were corrected, reproducibility is not ensured.

SciTS Presentation: Toward a theory of scientific discovery and reproducibility

Foxholes, Not Silos

It has become de riguer, beyond even cliché, for academics to pejoratively describe the state of the Academy in terms of silos. The point of the metaphor is to decry narrow minded overspecialization and to encourage disciplinary specialists to interact more with specialists outside of their discipline. But in general, the silo metaphor for the modern university is worse than wrong. It is sterile. I propose a more accurate and useful metaphor for our situation, a framing that leads more naturally to possible solutions: foxholes.

Metaphors matter. Framing a subject can either hamper or facilitate discussion, shaping our ability to understand an issue and to deal with it. Statistician George E. P. Box’s observation about statistical models applies equally well to metaphors: “All models are wrong, but some are useful.” A useful metaphor illuminates, prodding us to deeper understanding, hopefully even suggesting solutions. In other words, good metaphors stimulate our imagination, rubbing imagery against reality with an eye to improving the latter.

I maintain that the academic “siloization” metaphor (I have heard otherwise literate people use that term) is neither accurate nor useful if we want to understand and improve the Academy. The foxhole image leads more naturally to useful analogies. The foxhole metaphor suggests solutions. Foxholes provide different perspectives, which might help us understand where the threats really lie. Most significantly, soldiers in foxholes are comrades, not enemies.

SciTS Presentation: Foxholes, not Silos

Transdisciplinary Knowledge Producing Teams (KPTs): Typology, features, and communication skills

Transdisciplinary Knowledge Producing Teams (TDKPTs) are groups of stakeholder participants tasked with producing knowledge across disciplinary, sectoral, and ecological boundaries. This paper accesses literature from the Science-of-Team-Science (SciTS), complexity theory, and systems theory to construct a typology of the features of TDKPTs. First, we conduct a descriptive analysis of features of TDKPTs from a systems perspective. We distinguish between interactive systemic complexities (arising from interpersonal interactions in TD team settings) and structural systemic complexities (arising from characteristics inherent to the make of the team). We analyze the characteristics of TDKPTs (such as complex problem solving, stakeholder involvement, methodological pluralism), elucidate how they parallel various complex adaptive systems factors (such as non-linearity, open systems, interactions between systems components), and present case examples of both structural and interactive system complexities. Additionally, we identify concrete skill building aspects needed for TDKPTs to be successful. We align each skill building foci with specific outcomes related to supporting effective communication in TDKPTs.

As TDKPT members are required to engage in theoretical, epistemological, and methodological reflections to elucidate the dynamic nature of TD knowledge producing teams, understanding how conflict, dissonance, and reciprocal interdependencies contribute to knowledge generation are key areas of team effectiveness and call for continual research and inquiry. This paper provides a framework by which team functioning can be considered and enhanced within TDKPTs. The features of TDKPTs developed and described in this paper inform the development of the requisite skill set for better team communication and functioning. This offering attracts researchers of TD teams and TD team members alike to reconsider the development and study of TDKPTs.