Psychological Research Methods

10 Foundational Questions that Educated People Should be able to Ask and Answer when Evaluating Evidence*

Adapted from Neil Lutsky (NITOP 2007)

1. What do the numbers show?
2. How representative are the numbers?
3. Compared to what?
4. Is the outcome statistically significant (and what does it mean if it is, or isn't)?
5. What is the effect size?
6. Are the results of a single study or of a literature?
7. What is the research design (e.g., correlational or experimental)?
8. How were the variables operationalized?
9. Who is in the sample?
10. What was controlled (and what was not controlled)?

*Behavioral Science Statistics helps students to ask and answer questions 1 - 5. Psychological Research Methods helps students to ask and answer questions 6 - 10. Experimental Psychology helps students to generalize their ability to ask and answer all 10 questions.

Quotes

"The modern world is a noxious enviornment for those of us bothered by logical error. People may have become no worse at reasoning, but they now have so many more opportunities to show off how bad they are."

-- Jamie Whyte, philosopher, 2005

"After all, the ultimate goal of all research is not objectivity, but truth."

-- Helene Deutsch, psychiatrist, 1944-45

"The anxieties of the physicists and mathematicians over randomness are akin to worrying about atomic clocks drifting by .01 seconds per century, whereas social scientists typically operate at a level of precision more appropriate for an alarm clock that gains or loses 6 hour a week."

-- Robert P. Abelson, psychologist, 1995

"Students need to do complex writing that requires systematic thinking, organization, and integration of references. They must also be able to express opposing views and show how they have used evidence to come to their conclusions."

--Akyea & Sandoval, educators, 2004

"I will never know the experiences of others, but I can know my own, and I can approximate theirs by entering their world. This approximation marks the tragic, perpetually inadequate aspect of social research."

-- Shulamit Reinharz, sociologist, 1984

Rap song inspired by "Thinking and deciding" attributed to Deborah Frisch, Ph.D.

I woke up and had a revelation Correlation does not imply causation. The situation is quite terrible When you've got a confounding variable. Another reason to think carefully: Does B cause A or does A cause B? Popper said "I wonder why? Einstein's theory lived but Freud's has died. It seems to me the difference is That Einstein's could be falsified." Suppose one day I say to you, "Hey listen up: 'If p, then q!'" And then I tell you: "p is true." Could you be sure that q's true too? And what if I'd said: "q is true." Could you conclude that p's true too?" [Not you again!] That's right! It's me! Is it "not q" if it's "not p?" [Please go away!] One more to do! Is it "not p" if it's "not q?" Peter Wason is the name Of a man who liked card games. But unlike me and unlike you He played with four, not fifty-two. Letters on the front Numbers on the back. "A" and "B" and "2" and "3" Are all that you can see. A rule has been proposed. You don't know if it's true. "A card that has a vowel Has an even number too."

Which cards must you inspect To verify the rule? [Why not check them all?] No, that would not be cool! You've got to check the A (Don't bother with the B) The 2 can't give you trouble But watch out for the 3! Doc Wason's back He's got more tricks Here's a game Called 2-4-6. Your job is to name sets of three To test different hypotheses. [3-6-9?] Yes! [1-2-3?] Well that works too. [I'm smart as you!] [6-8-10?] Way to go, my friend! [3-2-1?] No! Please try again. [10-12-14?] That's peachy keen! [2-3-4?] Oy, how many more? [1-2-10?] Yes! We're near the end! [5-6-8?] Right! You're doing great! I'm ready, now. I can guess your rule. I'm the one that you cannot fool. The rule you use for sets of three: Increasing numerosity!