The importance of standardised approaches for experimental research

The replication crisis is both the result of suboptimal theorizing and the consequence of problematic statistics. Psychological science suffers from "parallel" and "mini" theories, as well as analogies that stand-in for well-defined theoretical frameworks. This tendency leads to the frequent reporting of intriguing yet isolated findings, which exist outside of established theories. This practice not only overlooks existing theories but also hinders the integration of knowledge within a comprehensive theoretical framework. Kuhn (1996) suggested that science progresses in phases. He calls the initial stage of young disciplines protoscience. At this stage, disciplines gravitate towards exploratory rather than theory-driven research. His characterization of protoscience perfectly matches the current state of psychological science. According to Kuhn, the maturation of a field requires the development of a research paradigm. A paradigm serves as cohesive framework that unifies theory, measurement instruments, values, and underlying metaphysical assumptions. A paradigm is shared and agreed upon by the research community. While psychology offers such theories they are not widely used. I believe this is the result of the ubiquitous demand to make theoretical contributions. In contrast, paradigmatic research implies that scientists examine auxiliary assumptions while leaving the broader theoretical framework untouched. Rather than changing the theoretical framework, paradigmatic science strives for highly accurate measures and relatively small specifications of theoretical assumptions in the form of auxiliary assumptions. We will discuss whether psychology can adopt a paradigm and how changes in the research and review process can support this change.

Challenges in the relationship between theory and observation, such as the theory-ladenness of observations, can be interpreted as a call for general methods to evaluate theories independently of observational data. In this presentation, I will discuss such methods, with a focus on theories about mechanisms of higher cognitive functions. Central to the discussion is the definition of a 'mechanism' as a set of entities with specific properties that lead to systematic activity. I will explore the following key points as problematic: Theories based on proposing novel, dedicated mechanisms (i.e., introducing new entities), theories based on concepts that are challenging to link to any tangible entity, and theories that are only specified at a singular level of abstraction (i.e., overlooking entities at lower levels). All three aspects relate to the principle of parsimony, which favors theories requiring fewer assumptions. However, this discussion aims to refine the definition of 'assumption,' to reduce ambiguity in applying the parsimony concept, thereby enhancing its utility in evaluating theories in cognitive sciences independently of observational data. (This talk was part of the symposium "Towards a Sustainable Experimental Psychology: Theory, Replicability, Behavior" organized by Lukas Thürmer.)

Psychological science entered a crisis when a lack of reproducibility, outright fraud, and the discovery of the prevalence of questionable research practices converged. Since then, funders, scientists, and the public have proposed or even demanded open science as a remedy to increase the accessibility, transparency, and reproducibility/replicability of psychological research. While these practices are generally perceived as positive and laudable, they often require increased effort on the part of the researchers, at least if pursued in a meaningful way (cf. FAIR principles). At ZPID, our mission is to promote open science practices by providing free-of-charge tools that reduce the burden on researchers while ensuring quality standards. One part of my talk will cover some of these tools (e.g., P-TOS, PreReg, PsychArchives), their adoption, and also their weaknesses. Another part will look beyond ZPID to developments in the scientific publication system, such as Registered Reports and PCI Registered Reports. While I will occasionally present empirical data, this talk will mainly reflect my subjective perspective as a former researcher who has moved into a research infrastructure position at an Open Science Institute. This talk was part of the symposium "Towards a Sustainable Experimental Psychology: Theory, Replicability, Behavior" chaired by Lukas Thürmer.

In 2000, the APA’s Board of Scientific Affairs announced the Decade of Behavior (DoB) to promote the importance and support for the study of human action (Kendall, 2000). Nevertheless, the proportion of published studies that include measures of behavior has declined steadily (e.g., Baumeister Vohs, & Funder, 2007, Banks, Woznyj, & Mansfield, 2021). In this talk, we will provide a personal account on why and how we are enthusiastic about studying behavior. Specifically, we will (a) discuss why studying behavior yields uniquely informative and relevant insights, (b) provide a practical definition of what constitutes a behavioral measure, and (c) present a program of experimental research on intergroup criticism where behavioral measures have allowed us to make unique predictions and yield surprising findings. We will discuss how to increase actual behavioral research.