PSY400 Research Methods and Analysis 4: Scientific Research Strategies

PSY400 Research Methods and Analysis 4

Dr Joshua Adie University of the Sunshine Coast | CRICOS Provider Number: 01595DUniSC

Week 1 Experimental Design Principles

University of the Sunshine Coast | CRICOS Provider Number: 01595D

Reading - covers undertaking these techniques in SPSS

• Field, A. (2018) Discovering statistics using IBS SPSS statistics
• Chapter 1
• Chapter 2
• Chapter 3

Workshop content

1. Course overview
2. Hypothesis testing approach (NHST)
3. The fallibility of chasing p
4. Effect size estimation
5. How do we do "Science"
6. The process of scientific research
7. Multivariate designs

University of the Sunshine Coast | CRICOS Provider Number: 01595D

Course overview

• Course designed to cover advanced and complex research designs and the accompanying statistical analyses
• Primarily focusing on multivariate level designs and analyses
• Will also explore qualitative design and analysis as well as Bayesian approaches
• Assessment:
• To demonstrate your ability to develop a multivariate design and appropriate selection of analytic techniques
• To demonstrate your ability and knowledge in the use and application of multivariate data analysis
• To demonstrate your knowledge of alternate designs and techniques beyond NHST, including qualitative and Bayesian approaches

3 forms of assessment to assess these competencies

1. Written report with a multivariate design and analysis proposed (cannot be your own thesis or thesis topic)
2. In class examination of your use of SPSS to perform statistical analyses
3. End of semester examination of your knowledge of course content.

University of the Sunshine Coast | CRICOS Provider Number: 01595D

Schematic to your training in design and analysis

Quantitative methods

Qualitative methods

NHST (null hypothesis significance testing) approaches

Alternatives to NHST

Bayesian models

Thematic analysis

Comparing Samples Single DV

Comparing Samples Multiple DVS

Relationships between variables

Bayes Theorem

Single IV designs

Two IV designs

Multivariate designs

Single predictor

Multiple predictors

Odds Ratio

Textual analysis

Pearson's r

Partial correlation

One sample t-test

Chi-square goodness of fit

2 way ANOVA

MANOVA

independent sample t-test

Chi-square test of contingencies

2 way ANCOVA

MANCOVA

Hierarchical multiple regression

Sensitivity / Specificity

paired sample t-test

Two way repeated measures ANOVA

Repeated measures MANOVA

Stepwise multiple regression

Regional Operating Curves (ROC)

One way ANOVA

Forward/Backward multiple regression

Logistic regression

Exploratory Factor Analysis

Discriminant Function Analysis

Confirmatory Factor Analysis

100 LEVEL

200 LEVEL

300 LEVEL

400 LEVEL

Spearman-Rho correlation

Standard multiple regression

Likelihood Ratio

NVIVO analysis

One way repeated measures ANOVA

One way ANCOVA

Structural Equation Modelling

University of the Sunshine Coast | CRICOS Provider Number: 01595D

Conditional probability

Hypothesis testing approach (NHST - Null Hypothesis Significance Testing)

• NHST is the most commonly used inferential procedure

. NHST - enables inferences about a larger population to be made from a smaller sample drawn from the larger population

• The NHST process:

1. create a hypothesis (prediction) regarding the population of interest (H1)
2. alternate to the experimental hypothesis is the null hypothesis (H0 )
3. collect data from a sample drawn from the population of interest and test the prediction (hypothesis) made
4. If the sample is consistent with the prediction, then we have evidence in support of the hypothesis (H1). If the sample is inconsistent with the prediction, then we reject the hypothesis (H1 ) and accept the null hypothesis (H0 ) in the interim
5. The decision point of whether to accept or reject the hypothesis H1 is based on the probability of being correct or incorrect in our decision based on the sample (for Psych - aim for a 95% probability of being correct - ie a = . 05 ). Remember, this doesn't mean the hypothesis is true - rather the probability of it being correct is 95%, but 5% of the time we might actually be wrong.

University of the Sunshine Coast | CRICOS Provider Number: 01595D

The fallibility of chasing p

. The basic premise of NHST approach - devise an experimental hypothesis and test the probability of the experimental hypothesis being correct compared to the null hypothesis.

• The probability estimate is based on two key notions:

1. The magnitude (size) of the difference or relationship
2. The variance of scores for each of the variables being compared

. However, the exact magnitude and variance levels required to attain p. < . 05 is variable: . The cause of this variability is Statistical Power -the power to detect a difference/relationship between variables (Power = the statistical probability that a test wil correctly reject a false null hypothesis). Sample size is main driver of power - the greater the sample size the smaller the magnitude of difference (and larger the variances) detect . Solution - estimate the Effect Size of the study. . Effect Size is a metric designed to quantify the magnitude of the differences observed following removal of sample size as an effect on the probability of the difference detected. As n increases the magnitude of the difference detected decreases - effect size removes the influence of sample size and effectively computes an adjusted estimate of the magnitude of the effect detected.

Large Effect Size p. <. 05 Medium Smal Smal Sample Size (n) Large

University of the Sunshine Coast | CRICOS Provider Number: 01595D

The fallibility of chasing p continued

. Once the ES for a comparison of 2 means is known, then one can calculate the degree of overlap between the distributions of the 2 samples (see Zakzanis, 2001) . This computation assists in understanding the meaningfulness of the significant differences detected. . For example, assume we are comparing the means of 2 different groups. Using Cohens d as our estimate of effect size we can compute the exact d obtained for our comparison. As a rule of thumb, for d a small ES is 0.2, a medium ES is 0.5, and a large ES is 0.8.

d = 0.2 small ES d = 0.5 medium ES %% SD 1 SD + X1 X2 X X2 84.3% overlap 66.6% overlap d = 0.8 large ES 1% SD X1 X2 52.6% overlap d = 3.4 extremely large ES X X 4.7% overlap

University of the Sunshine Coast | CRICOS Provider Number: 01595D

Effect size estimation

• NHST assesses the probability of whether the data supports the experimental hypothesis over the null hypothesis. . The mathematical computation of this probability is the statistical techniques we use to determine this probability. The precise statistical technique we use depends on the type of data we are examining and the study design we used to test the hypothesis(es). . Ultimately, our mathematical computation of probability results in our seeking to find a probability of 95% or better certainty in the experimental hypothesis - so we select p. < . 05 as our cut-off for statistical significance. . But statistical significance can be misleading as reaching statistical significance depends on the statistical power available - with the statistical power available to any comparison being a function of the sample size. Therefore, NHST using only p. value as the basis for deciding whether a difference is meaningful is fraught with risk.
• That something is statistically significant does not mean that the difference is meaningful in the wider population.
• Effect size is important as it allows the measurement of the potential meaningfulness of a statistically significant finding. Effect size allows us to compute the degree of overlap in the distributions between our samples - which helps in the interpretation the meaningfulness of a statistical difference.
• There are 3 general classes of effect size estimation methods:
1. Confidence intervals
2. ES in standardised units of difference (e.g. Cohens d, Glass g', Hedges g)
3. Variance-accounted-for statistics (e.g. r2, eta-squared [n2], partial-eta squared [n2 ], omega-squared [GJ2], Cramer's V).

. We will discuss these across the semester. University of the Sunshine Coast | CRICOS Provider Number: 01595D

How do we do "Science"

Psychology is a science - in that truth is found by external reference - i.e. empirically, from external data. The scientific approach comprised 3 central assumptions:

1. There is a "world" to be understood . The external world actually exists and is not a creation of our own minds
2. Laws in the external world explain the events that take place there . Behaviour does not occur randomly, but occurs for distinct reasons and those reasons may be discovered
3. The means for finding the explanatory laws in the world is empirical (meaning through our senses)

From these 3 central assumptions come the scientific method, which has 3 major features:

1. Science is empirical
2. Science is repeatable and public
3. Science is explanatory, predictive, and theoretical . A theory explains behaviour by postulating why the behaviour occurs, usually stating the conditions under which the behaviour happens. . An obvious test of theory is to let it predict behaviour in a situation other than the setting where the theory was developed (experiment). If the theory adequately predicts behaviour, then the theory is strengthened.

University of the Sunshine Coast | CRICOS Provider Number: 01595D

Scientific research strategies

Technique

Features

Strengths

Weaknesses

Observational methods

Naturalistic

Non-intrusive and "natural"

• Realistic situation
• Possible intrusion
• Observer bias
• Lack of control
• No causal inference

Case study (& n=1 research)

Interactive

• In-depth study
• Observer bias
• Realistic situation
• Clearly intrusive
• Subject bias
• No causal inference

Self-report

Highly personal

• In-depth study
• Subjectivity
• Observer bias
• Lack of generality
• No causal inference

Survey

Structured

• Objective
• Purely descriptive
• No causal inference

Correlational methods

Structured testing

• Relationships Widely applicable
• Possible confounding
• No causal inference

Experimental methods

Quasi-experimental

Cannot manipulate independent variable

• Only way to do some experiments (ethical)
• Loss of control
• Possible confounding

Experiment

Full control of all variables

• Economy
• Causal inference
• Artificial Possible lack of generality

University of the Sunshine Coast | CRICOS Provider Number: 01595D