Merge branch 'main' of http://github.com/langcog/experimentology

009-design.qmd

@@ -48,13 +48,13 @@ In this section, we'll discuss some key dimensions on which experiments vary: 1)
 
 The classical "design of experiments" framework has as its goal to separate observed variability in the dependent measure into 1) variability due to the manipulation(s) and (2) other variability, including measurement error and participant-level variation. This framework maps nicely onto the statistical framework described in Chapters [-@sec-estimation] -- [-@sec-models]. In essence, this framework models the distribution of the measure using the condition structure of our experiment as the predictor. 
 
-Different experimental designs will allow us to estimate condition effects more and less effectively. Recall in @sec-estimation, we estimated the effect of our manipulation by a simple subtraction: $\beta = \theta_{T} - \theta_{C}$ (where $\beta$ is the effect estimate, and $\theta$s indicate the estimates for each condition, treatment $T$ and control $C$). This logic works just fine also if there are two distinct treatments in a three condition experiment: each treatment can be compared to control separately.  For treatment 1, $\beta_{T_1} = \theta_{T_2} - \theta_{C}$ and $\beta_{T_2} = \theta_{T_2} - \theta_{C}$. That logic is going to get more complicated if we have more than one distinct factor of interest, though. Let's look at a simple example. 
+Different experimental designs will allow us to estimate condition effects more and less effectively. Recall in @sec-estimation, we estimated the effect of our manipulation by a simple subtraction: $\beta = \theta_{T} - \theta_{C}$ (where $\beta$ is the effect estimate, and $\theta$s indicate the estimates for each condition, treatment $T$ and control $C$). This logic works just fine also if there are two distinct treatments in a three condition experiment: each treatment can be compared to control separately.  For treatment 1, $\beta_{T_1} = \theta_{T_1} - \theta_{C}$ and $\beta_{T_2} = \theta_{T_2} - \theta_{C}$. That logic is going to get more complicated if we have more than one distinct factor of interest, though. Let's look at a simple example. 
 
 ![The 2x2 crossed design used in @young2007](images/design/young2007-design2.png){#fig-design-young-design .column-margin}
 
 Let's look at an example. @young2007 were interested in how moral judgments depend on both the beliefs of actors and the outcomes of their actions. They presented participants with vignettes in which they learned, for example, that Grace visits a chemical factory with her friend and goes to the coffee break room, where she sees a white powder that she puts in her friend's coffee. They then manipulated both Grace's *beliefs* and the *outcomes* of her action following the schema in @fig-design-young-design. Participants (N=10) used a four-point Likert scale to rate whether the actions were morally forbidden (1) or permissible (4). 
 
-Young et al.'s design has two factors -- belief and outcome -- each with two levels (neutral and negative, noted as $B$ and $-B$ for belief and $0$ and $-O$ for outcome).^[Neither of these is necessarily a "control" condition: the goal is simply to compare these two levels of the factor -- negative and neutral -- to estimate the effect due to the factor.] These factors are **fully crossed**: each level of each factor is combined with each level of each other. That means that we can estimate a number of effects of interest. The experimental data are shown in @fig-design-young-data. 
+Young et al.'s design has two factors -- belief and outcome -- each with two levels (neutral and negative, noted as $B$ and $-B$ for belief and $O$ and $-O$ for outcome).^[Neither of these is necessarily a "control" condition: the goal is simply to compare these two levels of the factor -- negative and neutral -- to estimate the effect due to the factor.] These factors are **fully crossed**: each level of each factor is combined with each level of each other. That means that we can estimate a number of effects of interest. The experimental data are shown in @fig-design-young-data. 
 
 ![Moral permissability as a function of belief and outcome. Results from @young2007, annotated with the estimated effects. Simple effects measure differences between the individual conditions and the neutral belief, neutral outcome condition. The interaction measures the difference between the predicted sum of the two simple effects and the actual observed data for the negative belief, negative outcome condition.](images/design/young2007-data2.png){#fig-design-young-data .margin-caption}
 
@@ -167,7 +167,7 @@ The implication of these examples is clear: experimenters need to take care in b
 
 1. To maximize generality, use samples of experimental items -- words, pictures, or vignettes -- that are comparable in size to your samples of participants.
 2. When replicating an experiment, consider taking a new sample of items as well as a new sample of participants. It's more work to draft new items, but it will lead to more robust conclusions. 
-3. When experimental items are sampled random from a broader population, use a statistical model that includes this sampling process (e.g., mixed effects models with random intercepts for items from @sec-models). 
+3. When experimental items are sampled at random from a broader population, use a statistical model that includes this sampling process (e.g., mixed effects models with random intercepts for items from @sec-models). 
 :::
 
 One variation on the repeated measures, between-participants design is a specific version where the measure is administered both before (pre-) and after (post-) intervention, as in @fig-design-pre-post). This design is sometimes known as a **pre-post** design. It is extremely common in cases where the intervention is larger-scale and harder to give within-participants, such as in a field experiment where a policy or curriculum is given to one sample and not to another. The pre measurements can be used to subtract participant-level variability out and recover a more precise estimate of the treatment effect. Recall that our treatment effect in a pure between participants design is $\beta = \theta_{T} - \theta_{C}$. In a pre-post design, we can do better by computing $\beta = (\theta_{T_{post}} - \theta_{T_{pre}}) - (\theta_{C_{post}} - \theta_{C_{pre}}$. This equation says "how much more did the treatment group go up than the control group?^[This estimate is sometimes called a "difference in differences" and is very widely used in the field of econometrics, both in experimental and quasi-experimental cases [@cunningham2021].]
@@ -275,7 +275,7 @@ Suppose you were designing an experiment of this sort and you wanted to follow o
 
 ![Confounding order and condition assignment means that you can't make an inference about the link between money and happiness.](images/design/money1.png){#fig-design-money1 .column-margin}
 
-If you think your experimental design might have a confound, you should think about ways to remove it. For example, **counterbalancing** order across participants is a very safe choice. Some participants get \$100 first and others get \$1000 first. That way, you are guaranteed that the order of conditions will have no effect of the confound on your average effect. The effect of this counterbalancing is that it "snips" the causal dependency between condition assignment and later time. We notate this on our causal diagram with a scissors icon (@fig-design-money2).^[In practice, counterbalancing is like adding an additional factor to your factorial design! But because the factor is a **nuisance factor** -- basically, one we don't care about -- we don't discuss it as a true condition manipulation. Despite that, it's a good practice to check for effects of these sorts of nuisance factors in your preliminary analysis. Even though your average effect won't be biased by it, it introduces variation that you might want to understand to interpret other effects and plan news studies.] Time can still have an effect on happiness, but the effect is independent from the effect of condition and hence your experiment can still yield an unbiased estimate of the condition effect. 
+If you think your experimental design might have a confound, you should think about ways to remove it. For example, **counterbalancing** order across participants is a very safe choice. Some participants get \$100 first and others get \$1000 first. That way, you are guaranteed that the order of conditions will have no effect of the confound on your average effect. The effect of this counterbalancing is that it "snips" the causal dependency between condition assignment and later time. We notate this on our causal diagram with a scissors icon (@fig-design-money2).^[In practice, counterbalancing is like adding an additional factor to your factorial design! But because the factor is a **nuisance factor** -- basically, one we don't care about -- we don't discuss it as a true condition manipulation. Despite that, it's a good practice to check for effects of these sorts of nuisance factors in your preliminary analysis. Even though your average effect won't be biased by it, it introduces variation that you might want to understand to interpret other effects and plan new studies.] Time can still have an effect on happiness, but the effect is independent from the effect of condition and hence your experiment can still yield an unbiased estimate of the condition effect. 
 
 ![Confounding between a specific condition and the time at which it's administered can be removed by counterbalancing or randomization of order.](images/design/money2.png){#fig-design-money2 .column-margin}