Evaluate Research
Experimental Methodology
Experimental Design
Define the aim of a study
- The aim of a study is what theory or idea is being tested
- Usually takes the form of 'the effect of x on y' where x is the independent variable and y is the dependent variable
- The effect of studying on test scores
- The effect of alcohol on driving precision
State a research and null hypothesis of a study (HL only)
- Null hypothesis - there is no difference or change between the control and experimental conditions
- Research hypothesis - there is a difference between the control and experimental conditions
- Can be in the form of a directional or non-directional
- Directional - predict the direction of the effect of the independent variable on the dependent variable
- Studying more will increase test scores
- Non-directional - only predict that there there will be an effect but not in which direction
- Amount of study time will affect test scores
State the independent and dependent variable in an experiment
- Independent variable - what you manipulate
- Dependent variable - what you measure
- How many mistakes drunk driver makes
State operational definitions of variables
- A variable is operating as a measure for some phenomenon
- IQ could be a variable that is operating as a definition (and measure) of intelligence
Describe potential confounding variables
- Confounding variables are any factors other than the independent variable that could have affected the dependent variable
- In the drunk driving study, driving ability and experience would affect how many mistakes the driver makes
- In the amount of study time test scores study, intelligence would affect test scores as well as study time
- Carryover effects - in a repeated measures (within subjects) design where participants act as their own control, they may perform better or worse in the experimental condition because of practice
- In the IQ test example, test scores are likely to increase from one condition to the next because they have become familiar with the test
Explain the controls needed for an experiment
- Maturation - the dependent variable can be affected simply by participants aging
- I'm testing how video games affect IQ in 2nd graders
- They could improve because their brains are maturing and are cognitively more able to tackle IQ tests
- We would need a matched pairs design where we control for existing cognitive ability then have one group play video games and the other not and compare the test scores
- Contamination -
- Placebo effect - the belief that something is affecting our behavior can be enough to effect the change
- This is why when anti-depressants are tested there is always a placebo control group
- Counterbalancing - to control for carryover effects we would have some drivers get drunk first and then drive while others would drive sober first and then drink and drive
Explain effects of participant and researcher expectations and bias
- Demand characteristics - when the participant guesses what you are investigating and modifies his behavior to either help or hinder you from gaining positive evidence
- If the drivers figure out I'm testing alcohol's effect on driving ability, some may help me out by trying really hard not to make mistakes while others may drive extra recklessly
- Either way we only want to test for alcohol's effect so we can use a single blind method and not tell the participants what we are investigating and hope they do not figure it out
- Expectancy effect - either the researcher or the participant expects a certain outcome from the experiment and unconsciously affects the results
- A researcher investigating aggression in children may be more inclined to accept ambiguous acts as aggressive
- Can be resolved by making it double-blind and using independent observers
Explain the use of single-, double- and triple blind techniques
- Single blind - participant doesn't know which treatment they are given
- Double bind - neither participant nor researcher knows which treatment participant is given
- Triple blind - participant, researcher and statistician do not know which treatment the participant was given
Discuss the strengths and limitations of experimental design
Independent samples (Between Subjects)
- 2 separate groups of individuals
- Each group is participates in a different condition (experimental or control)
Strengths
- Each participant is only given one treatment so there are no carryover effects
Limitations
- Cannot account for individual differences
- Requires more people and therefore usually costs more
Repeated measures (Within Subjects)
- 1 group of individuals
- All individuals participate in all conditions
Strengths
- Less participants needed
- Accounts for individual differences because each participant acts as their own control
Limitations
- Effects of repeated testing (e.g. practice) can confound the results
Matched Pairs
- A form of between subjects that attempts to control some of the individual differences by matching participants' characteristics in the experimental group to the control group
- Can be age, gender, socioeconomic background, intelligence, culture, personality type
Strengths
- Accounts for individual differences that repeated measures does not
- No carryover effects
Limitations
- There are some individual differences that are problematic to control for
Single Participant
- Usually natural or quasi-experiments where the researcher cannot manipulate the independent variable usually because it would be unethical to do so
- Putting a tamping iron through someone's head vs. it occurring naturally with Phineas Gage
Strengths
- Allows the study of unique individuals
- Can elucidate phenomena that would otherwise be unethical to intentionally cause
Limitations
- Lack of control over variables for natural experiments
- Can be difficult to find a control participant to compare to
- Cannot extrapolate the results to the entire population because the sample size is too small
Sampling procedures
Discuss sampling techniques appropriate to quantitative research
Random
- You randomly choose participants based on some algorithm like their area code
Opportunity
- Most common type of sample where you choose people for your study based on their willingness to participate
- Most studies use university students who either have to participate for course credit or receive some money for their cooperation
Systematic
- Type of random sample but you have taken measures to make sure that every individual in your target population has an equal chance of being chosen
Stratified
- When you have a sample composed of diverse individuals it is ideal to stratify the sample
- Say I have a sample of high school students aged 14-18 with different ethnicities, cultural backgrounds and genders
- Ideally the sample should be divided into homogeneous subcategories called stratums
- One stratum would 17-year-old boys, caucasian, who grew up in an individualistic country (preferably the same country)
- Another stratum could be 17-year-old boys only or caucasians only
- Can reduce variability of scores within the stratum
Discuss how participants are allocated to experimental and control groups
Random Allocation
- Random allocation can be generated by a computer
- Using other criteria like the first 10 who arrive go in the experimental group is invalid because there may be characteristics of the participants that make them show up early for the experiment that could confound the results
Matched Pairs
- Relevant characteristic is matched in the experimental group and control group
- Raine et al., (1997) study used a control group matching for age, sex and where relevant schizophrenia
Explain the concept of representative sampling
- The sample chosen should accurately represent the characteristics of the target population
- It makes no sense to only use university students for a study as they do not accurately represent the population
- Some characteristics of university students like intelligence, age and socioeconomic background could skew the results
Evaluation of research
Discuss the concepts of internal and external validity
- Generally there is always a trade-off between internal and external validity
- Laboratory experiments tend to have high internal validity but lack some form of external validity
- Field experiments tend to have high external validity (though only for that specific setting) but lack internal validity because of the nature of the real world
- You can't create a world where participants do not know that everything is under control like The Truman Show, though it would be perfect for psychological research
Internal Validity
- Experiment must be valid in that it measures what it claims to measure
- Experiment must be reliable in that the same measuring tool was used for all participants
External Validity
- Refers to the extent that results from the experiment can be generalized to the general population, different conditions and settings
- Small sample sizes threaten external validity especially if they are not representative of the population
- Ecological validity refers to the problem of generalizing results from an experimental settings to real world situations
- It is also argued that even if you conduct an experiment in one real-world setting does not mean that they generalize to all real-world settings
- Solution is to conduct the same experiment in different real-world settings and hope for converging results
Discuss conditions that increase a study's reliability
- Counterbalancing in a within subjects design to reduce carryover effects
- Using matched pairs to establish controls
- Replication increases validity and reliability
- Cross-cultural verification to account for individualism/collectivism and power distance
- Using different genders
Apply descriptive statistics to analyse data
Measures of Central Tendency
- Mean - average of all the scores
- used for ratio and interval data
- Median - middle value of all the scores
- Mode - value that appears most
Measures of Dispersion
- Range - maximum value minus the minimum value
- Standard deviation - how much scores vary
Distinguish between levels of measurement
Nominal
- Lowest level of measurement
- Surveying beer preferences among college students
- Data is just stated
Ordinal
- Likert scale
- Data can be put in order from least to greatest
- People in a race are given places 1st, 2nd, 3rd
- But we don't know how much faster 1st place was compared to 3rd place
- 1st place, 2nd place etc are ordinal data
- Allows for some comparision but not by how much in meaningful terms
- Rating happiness on a Likert scale
- Person who rates 5 compared to 10 we cannot say that person is twice as happy, only that 10 is happier than 5
Interval/Ratio
- Like ordinal data but now we have meaningful units to compare
- Test scores are interval/ratio data
- If you get 5/10 correct and someone else gets 7/10 correct we can say that person did 20% better than you
- The difference between interval and ratio data is that ratio data has an absolute zero whereas interval data does not
Apply appropriate graphing techniques to represent data
- Bar charts for comparing means
- Histograms for showing frequency of nominal data
- Scatter plots for correlational data (ordinal or interval/ratio)
Apply an appropriately chosen statistical test
Parametric
- Tests for interval/ratio data
Comparing Means
- t-test for independent samples
- t-test for repeated measures
Relationship
Non-Parametric
- Tests for ordinal data and below
Comparing Means
- Mann-Whitney U - Independent measures
- Wilcoxon t-test - Repeated measures
- Sign test - Matched pairs
Relationship
- Spearman's rho for ordinal data
- if at least one of the two variables is ordinal then use Spearman's
- chi-squared for nominal data
Additional Resources
