Fraud = scientifc misconduct.

• Falsifying or fabricating data.
• This is intentional, not accidental.
• Puts all science under a bad light.
• Markedly different from QRPs (next).

• Diederik Stapel, social psychologist. Suspended in 2011. Fabricating and manipulating data.
• Marc Hauser, psychologist at Harvard. Resigned in 2011. Scientific misconduct.
• Jens Förster, social psychologist. Resigned in 2017. Data tampering.
  
  
  

Today we don’t talk about fraud explicitly.

We talk about something much harder to identify and erradicate:

Questionable research practices (QRPs).

Term coined by John, Loewenstein, & Prelec (2012).
See also Simmons, Nelson, & Simonsohn (2011).

• Not necessarily fraud.
• Includes the (ab)use of actually acceptable research practices.
• Problem with QRPs:
  
  • Introduce bias (typically, in favor of the researcher’s intentions…).
  • Inflated power at the cost of inflated Type I error probability (\(\gg 5\%\)).
  • Results not replicable.

(John et al., 2012; Schimmack, 2015).

• Omit some DVs.
• Omit some conditions.
• Peeking: Sequential testing — Look and decide:
  • \(p > .05\): Collect more.
  • \(p < .05\): Stop.
• Only report \(p<.05\) results.
• \(p\)-hacking: E.g.,
  • Exclusion of outliers depending on whether \(p<.05\).
  • \(p = .054 \longrightarrow p = .05\).
• HARKing (Kerr, 1998): Convert exploratory results into research questions.
• …

• Researchers have a multitude of decisions to make (experiment design, data collection, analyses performed); Wicherts et al. (2016), Steegen, Tuerlinckx, Gelman, & Vanpaemel (2016).
• It is very possible to manipulate results in favor of one’s interests.
• This is now known as researcher’s degrees of freedom (Simmons et al., 2011).
• Consequence: Inflated false positive findings (Ioannidis, 2005).

From Bem (2004):

“(…) [L]et us (…) become intimately familiar with (…) the data. Examine them from every angle. Analyze the sexes separately. Make up new composite indices. If a datum suggests a new hypothesis, try to find further evidence for it elsewhere in the data. If you see dim traces of interesting patterns, try to reorganize the data to bring them into bolder relief. If there are participants you don’t like, or trials, observers, or interviewers who gave you anomalous results, drop them (temporarily). Go on a fishing expedition for something– anything– interesting.”

This is not OK unless the exploration is explicity stated.

Daryl Bem is the author of the famous 2011 precognition paper
(data used in Part 2 of today’s workshop).

Prof. Brian Wansink at Cornell University.

His description of the efforts of a visiting Ph.D student:

I gave her a data set of a self-funded, failed study which had null results (…). I said, “This cost us a lot of time and our own money to collect. There’s got to be something here we can salvage because it’s a cool (rich & unique) data set.” I had three ideas for potential Plan B, C, & D directions (since Plan A had failed). I told her what the analyses should be and what the tables should look like. I then asked her if she wanted to do them.

Every day she came back with puzzling new results, and every day we would scratch our heads, ask “Why,” and come up with another way to reanalyze the data with yet another set of plausible hypotheses. Eventually we started discovering solutions that held up regardless of how we pressure-tested them. I outlined the first paper, and she wrote it up (…). This happened with a second paper, and then a third paper (which was one that was based on her own discovery while digging through the data).

This isn’t being creative or thinking outside the box.

This is QRPing.

• He was severely criticized, his work was scrutinized (e.g., van der Zee, Anaya, & Brown, 2017).
• Over 100 (!!) errors in a set of four papers…
• Has now 40 (!!) publications retracted (as of July 2019).
• After a year-long internal investigation, he was forced to resign.

Yes.

• Martinson, Anderson, & Vries (2005): “Scientists behaving badly”.
• Fanelli (2009): Meta-analysis shows evidence of science misconduct.
• John et al. (2012): Evidence for QRPs in psychology.
• Mobley, Linder, Braeuer, Ellis, & Zwelling (2013): Reported evidence of pressure to find significant results.
• Agnoli, Wicherts, Veldkamp, Albiero, & Cubelli (2017): Evidence of QRPs, now in Italy.
• Fraser, Parker, Nakagawa, Barnett, & Fidler (2018): In other fields of science.
  
  

Interestingly, science misconduct has been a longtime concern (see Babbage, 1830).

And for the sake of balance:
There are also some voices against this description of the current state of affairs (e.g., Fiedler & Schwarz, 2016).

Well, maybe not (yet).

Here’s an interesting preprint (from July 2019!) from a Japanese research group (Kyushu University):

Ikeda, A., Xu, H., Fuji, N., Zhu, S., & Yamada, Y. (2019). Questionable research practices following pre-registration [Preprint]. doi: 10.31234/osf.io/b8pw9

It is strongly related to incentives (Nosek, Spies, & Motyl, 2012; Schönbrodt, 2015).

• “Publish or perish”:
  Publish a lot, at highly prestigious journals.
  • Journals only publish a fraction of all manuscripts.
  • Journals don’t like publishing null findings…
• Get tenured.
• Get research grant.
• Fame (prizes, press coverage, …).
• …
  
  
  

But, very importantly, it also happens in spite of the researcher’s best intentions.

• Deficient statistics education (yes, statisticians need to acknowledge this!…).
• Perpetuating traditions in the field.

Until very recently (Makel, Plucker, & Hegarty, 2012).

• Very low rate of replications in Psychology (estimated ~1%).
• Until 2012, majority of replications were actually successful!!
• However, in most cases both the original and replication studies shared authorship…
• Conflict of interest?…

• The Marshmallow Test (Watts, Duncan, & Quan, 2018)
• Ego depletion (Friese, Loschelder, Gieseler, Frankenbach, & Inzlicht, 2019; Hagger et al., 2016; Vadillo, Gold, & Osman, 2018)
• Power posing (Ranehill et al., 2015)
• The Stanford Prison Experiment (Griggs, 2014; Reicher & Haslam, 2006)
• The facial feedback hypothesis (Wagenmakers et al., 2016)
• Newborn babies’ imitation (Oostenbroek et al., 2016)
• The blocking effect (Maes et al., 2016)
• The stereotype threat (Flore, Mulder, & Wicherts, 2019)
• The facial expression (Gendron, Crivelli, & Barrett, 2018)
• ESP, of course! (Galak, LeBoeuf, Nelson, & Simmons, 2012; Ritchie, Wiseman, & French, 2012)
• The Mozart Effect (McKelvie & Low, 2002; Steele, Bass, & Crook, 1999)
• …

http://www.tylervigen.com/spurious-correlations

See here.

• Gelman blogged about an impressive timeline about the replication crisis.
• The whole blog post is worth reading for many reasons, including Gelman’s criticism over criticism!
  (versus Susan Fiske’s position).

See also this impressive dynamic plot:
https://psyborgs.github.io/projects/replication-in-psychology/

Probability of an effect at least as extreme as the one we observed, given that \(\mathcal{H}_0\) is true.

\[\fbox{$ p\text{-value} = P\left(X_\text{obs} \text{ or more extreme}|\mathcal{H}_0\right) $}\]

The definition is simple enough, right?…

Consider the following statement (Falk & Greenbaum, 1995; Gigerenzer, Krauss, & Vitouch, 2004; Haller & Kraus, 2002; Oakes, 1986):

Suppose you have a treatment that you suspect may alter performance on a certain task. You compare the means of your control and experimental groups (say, 20 subjects in each sample). Furthermore, suppose you use a simple independent means \(t\)-test and your result is significant (\(t = 2.7\), \(df = 18\), \(p = .01\)). Please mark each of the statements below as “true” or “false.” False means that the statement does not follow logically from the above premises. Also note that several or none of the statements may be correct.

Try it!: rebrand.ly/pvalue

All statements are incorrect.

But how did students and teachers perceive these statements?

This was in 2004. But things did not improve since…

This paper expands Goodman (2008) and elaborates on 25 misinterpretations.

Special issue with 43 (!!) papers (Wasserstein, Schirm, & Lazar, 2019).

Moving to a world beyond “\(p<.05\)”

• Confidence intervals (CIs) have been often advocated as the best inferential alternative to NHST.
• Recall, for example the Wilkinson Task Force (Wilkinson & Task Force on Statistical Inference, 1999):

“(…) it is hard to imagine a situation in which a dichotomous accept–reject decision is better than reporting an actual \(p\) value or, better still, a confidence interval.”

• But, are CIs really a better alternative?

See, for instance, Hoekstra, Morey, Rouder, & Wagenmakers (2014).

A (say) 95% CI is a numerical interval found through a procedure that, if repeated across a series of hypothetical data, leads to an interval covering the true parameter 95% of the times.

• A CI indicates a property of the performance of the procedure used to compute it:
  How is the procedure expected to perform in the long run?
• A CI for a parameter is constructed around the parameter’s estimate.
• However, a CI does not (really not!) directly indicate a property of the parameter being estimated!
  
  

Confused?
So is the vast majority of psychologists…

From Hoekstra et al. (2014), mimicking the \(p\) value study by Gigerenzer et al. (2004).

Try it!: rebrand.ly/confint

All statements are incorrect.

But how did students and teachers perceive these statements?

“If we were to repeat the experiment over and over, then 95% of the time the confidence intervals contain the true mean.”

How informative is this?!

Mental note:
Remember this when interpreting Bayesian credible intervals in part 2 of today’s workshop!

For completeness, not everyone agrees with the Hoekstra study (García-Pérez & Alcalá-Quintana, 2016; Miller & Ulrich, 2016; see also a reply by Morey, Hoekstra, Rouder, & Wagenmakers, 2016).

“The Basic and Applied Social Psychology (BASP) (…) emphasized that the null hypothesis significance testing procedure (NHSTP) is invalid (…). From now on, BASP is banning the NHSTP.”

Did it actually work? For a reflection, see Fricker, Burke, Han, & Woodall (2019).

(…) I will encourage authors to provide replication syntax and data through public repositories. Moreover, I will encourage the journal to focus on a manuscript’s research design and the author’s justification thereof, rather than the results, with the aim of ensuring that transparent studies that explore a research question with equipoise, will be published.

Editorial (Harrington et al., 2019).

“(…) a requirement to replace \(p\) values with estimates of effects or association and 95% confidence intervals”

Among many many, advices,

• Do not focus on \(p\) values.
• Report effect sizes.
• Report power analyses.
• Check model assumptions.

“Novice researchers err either by overgeneralizing their results or, equally unfortunately, by overparticularizing.”

Six principles:

1. \(p\)-values can indicate how incompatible the data are with a specified statistical model.
2. \(p\)-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a \(p\)-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A \(p\)-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a \(p\)-value does not provide a good measure of evidence regarding a model or hypothesis.

This is an editorial of a special issue consisting of 43 (!!) papers.

Main ideas:

• “Don’t” is not enough – Some what to do advices are provided.
• However… Don’t say “statistically significant” – Just don’t.

“(…) it is time to stop using the term “statistically significant” entirely. Nor should variants such as “significantly different,” “\(p < 0.05\),” and “nonsignificant” survive, whether expressed in words, by asterisks in a table, or in some other way."

But:

“Despite the limitations of \(p\)-values (…), however, we are not recommending that the calculation and use of continuous \(p\)-values be discontinued. Where \(p\)-values are used, they should be reported as continuous quantities (e.g., \(p = 0.08\)). They should also be described in language stating what the value means in the scientific context.”

• There is no unique “do”:

“What you will NOT find in this issue is one solution that majestically replaces the outsized role that statistical significance has come to play.”

• Accept uncertainty (I cannot stress this enough!).
  Be thoughtful, open, and modest.

• Editorial, educational, and other institutional practices will have to change.
  This includes: Journals, funding agencies, education, career system.

• Value replicability, open materials and data, and reliable practices (which all take time) over “publish or perish”.

• Debates in blogs, Twitter, and journals can be fierce.
• Criticism should be part of science, of course.
• It’s not bullying to criticize, of course, in particular, with grounded reasons (vide Wansink).
• But sometimes criticism gets too carried away, IMHO.
  
  

NYT, 2017

(Interestingly: A recent comeback in Psychological Science.)

Most likely, each of us has some skeleton’s in their scientific closets.

We’ve all fallen prey to one or more of the problems mentioned today.

Full disclosure:

I have too!!

So:

No one is better than anyone.

Or in the words of Brian Nosek (as quoted here):

“We’re not here to be right. We’re here to get it right.”

• Replications projects
• Registered reports
• Preregistrations
• Education
• …

(But we can talk about it too!…)

Today we focus on statistics.

After the break:

Gentle introduction to Bayesian statistics.