Multi-Arm Experiment

Learning about what works within a field often takes time, money and effort. Multi-arm designs can make this process somewhat more efficient: multiple treatments are assigned in parallel and compared against the same control group, which reduces sample size and hence costs.

Suppose that we want to learn whether there is racial discrimination in the labor market by conducting an experiment. Companies from our population of size \(30\) are randomly assigned to receive a résumé from a white, black or Latino candidate. Companies are assigned to one of these three conditions with equal probabilities, and résumés only vary race and are otherwise identical. We define our outcome of interest as the difference in callbacks between experimental conditions.

In settings of multiple treatment arms, we could do a number of pairwise comparisons: across treatments and each treatment against control.

Design Declaration

  • Model: We specify a population of size \(N\) where a unit \(i\) has a potential outcome, \(Y_i(Z = 0)\), when it remains untreated and \(m\) \((m = 1, 2, ..., M)\) potential outcomes defined according to the treatment that it receives. The effect of each treatment on the outcome of unit \(i\) is equal to the difference in the potential outcome under treatment condition \(m\) and the control condition: \(Y_i(Z = m) -Y_i(Z = 0)\).

  • Inquiry: We are interested in the effect of each treatment arm with respect to a comparison group.

  • Data strategy: We randomly assign \(k/N\) units to each of the three treatment arms.

  • Answer strategy: We fit a linear regression model with individual indicators for each of the treatments as covariates. The average treatment effect is equal to the regression coefficients, which are computed by subtracting the mean of the comparison group from the mean of each treatment group.

N  <-  30
outcome_means  <-  c(0.5, 1, 2)
sd_i  <-  1
outcome_sds  <-  c(0, 0, 0)

population <- declare_population(N = N, u_1 = rnorm(N, 0, outcome_sds[1L]), 
    u_2 = rnorm(N, 0, outcome_sds[2L]), u_3 = rnorm(N, 0, outcome_sds[3L]), 
    u = rnorm(N) * sd_i)
potential_outcomes <- declare_potential_outcomes(formula = Y ~ (outcome_means[1] + 
    u_1) * (Z == "1") + (outcome_means[2] + u_2) * (Z == "2") + 
    (outcome_means[3] + u_3) * (Z == "3") + u, conditions = c("1", 
"2", "3"), assignment_variables = Z)
estimand <- declare_estimands(ate_Y_2_1 = mean(Y_Z_2 - Y_Z_1), ate_Y_3_1 = mean(Y_Z_3 - 
    Y_Z_1), ate_Y_3_2 = mean(Y_Z_3 - Y_Z_2))
assignment <- declare_assignment(num_arms = 3, conditions = c("1", "2", "3"
), assignment_variable = Z)
reveal_Y <- declare_reveal(assignment_variables = Z)
estimator <- declare_estimator(handler = function(data) {
    estimates <- rbind.data.frame(ate_Y_2_1 = difference_in_means(formula = Y ~ 
        Z, data = data, condition1 = "1", condition2 = "2"), 
        ate_Y_3_1 = difference_in_means(formula = Y ~ Z, data = data, 
            condition1 = "1", condition2 = "3"), ate_Y_3_2 = difference_in_means(formula = Y ~ 
            Z, data = data, condition1 = "2", condition2 = "3"))
    estimates$estimator_label <- c("DIM (Z_2 - Z_1)", "DIM (Z_3 - Z_1)", 
    "DIM (Z_3 - Z_2)")
    estimates$estimand_label <- rownames(estimates)
    estimates$estimate <- estimates$coefficients
    estimates$term <- NULL
    return(estimates)
})
multi_arm_design <- population + potential_outcomes + assignment + 
    reveal_Y + estimand + estimator
diagnosis <- diagnose_design(multi_arm_design)
Estimand Label Estimator Label N Sims Bias RMSE Power Coverage Mean Estimate SD Estimate Mean Se Mean Estimand
ate_Y_2_1 DIM (Z_2 - Z_1) 500 -0.01 0.42 0.16 0.95 0.49 0.42 0.44 0.50
(0.02) (0.01) (0.01) (0.01) (0.02) (0.01) (0.00) (0.00)
ate_Y_3_1 DIM (Z_3 - Z_1) 500 0.00 0.44 0.90 0.95 1.50 0.45 0.45 1.50
(0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.00) (0.00)
ate_Y_3_2 DIM (Z_3 - Z_2) 500 0.01 0.42 0.56 0.96 1.01 0.42 0.45 1.00
(0.02) (0.02) (0.02) (0.01) (0.02) (0.02) (0.00) (0.00)

The diagnosis of our design indicates that the regression provides unbiased estimates of the average treatment effect (ATE) in each arm. These estimates, however, are not so precise; The estimated standard deviation is large yielding wide confidence intervals that contain the true value of the ATEs more than 95% of the time.

Using the Multi-Arm Designer

In R, you can generate a multi-arm design using the template function multi_arm_designer() in the DesignLibrary package by running the following lines, which load the package:

library(DesignLibrary)

We can then create specific designs by defining values for each argument. For example, we can create a design called my_multi_arm_design where N, m_arms, and outcome_means set to 80, 4, and c(-0.2, 0.2, 0.1), respectively, and other parameters use default values. To do so, we run the lines below.

my_multi_arm_design <- multi_arm_designer(N = 80, m_arms = 4, outcome_means = c(-0.2, 0.2, 0.1))

You can see more details on the multi_arm_designer() function, its arguments, and default values, by running the following line of code:

??multi_arm_designer