Random assignment

Random assignment in LimeSurvey is fairly straightforward, with one important twist.

We accomplish random assignment by adding a “question” of type Equation to the survey, then defining the equation to generate a random number. That random number becomes the condition number.

The twist comes from the fact that every time LimeSurvey evaluates the equation, it will create a different random number. Moreover, LimeSurvey evaluates each question at least twice: once on initial display, and again on validation before saving the results to the database. To overcome this limitation, we define an equation that generates a random number if it has already not been generated. If it has already been generated, that same random number is simply re-assigned to the question:

  1. Add a new question to the survey.
  2. Give it a unique ‘code’ — this will be the variable name in the dataset, and will also be how we refer to the random number later in the survey. Let’s say we choose ‘rand2’ as the code for this question.
  3. Set the question type to ‘Equation.’ This is critical for the result to be available for use in later questions.
  4. Set “Always hide this question” (under Advanced settings) to “yes” so the random number will not be displayed to the survey respondent.
  5. Enter the equation. If the code is set to ‘rand2’ and we want three conditions, the equation would be:

Here’s what’s going on with that equation: the curly braces indicate that what is between them should be evaluated by LimeSurvey as an equation. (Note that there cannot be a space between the braces and the content within, though there can be spaces elsewhere within the equation.) The equation itself consists of an if() function; this function takes the form ‘if(condition,result_if_true,result_if_false)’ — if the -condition- evaluates as true, the function returns ‘result_if_true’; otherwise it returns ‘result_if_false.’

In our case, the condition evaluates whether -rand2- (i.e., this question) has an answer. It will not have an answer the first time LimeSurvey encounters it, so the if() function returns the second argument.  This second argument is itself a function, -rand(1,3)-, that returns a random integer between 1 and 3.

On subsequent encounters with this question, ‘rand2’ will not be empty, and so the if() function will return the third argument, which is simply whatever ‘rand2’ was assigned when LimeSurvey first encountered the question. Thus, once assigned, the random number (i.e., condition) never changes.

Recruiting subjects from mTurk

The basic approach to recruiting subjects from Amazon’s Mechanical Turk (“mTurk”) is that subjects (“workers” in mTurk lingo) are given a link in mTurk to your survey, which is hosted on the Political Cognition Lab’s Limesurvey implementation. Once they complete the study, Limesurvey gives them a unique code, which they then enter back into mTurk to confirm that they completed the study. You can then use that code to confirm that workers completed the study before you pay them in mTurk.

There is an excellent tutorial on setting up mTurk to recruit research subjects:
Experiments using mechanical turk part 1
Experiments using mechanical turk part 2

Randomizing order of blocks of questions in Limesurvey

Update: With Limesurvey 2, you can now randomize the presentation of groups directly. When you set up a group, just include a string in the “Group Randomization” field — all groups that share a value for this field will be presented in random order.

The current version of Limesurvey (1.92+) allows the order of individual questions to be randomized, but does not allow for randomization of the order of entire blocks.  (Version 2.0 does have this function; when that release is stable we will upgrade.) In the mean time, primitive block randomization can be implemented by repeating relevant blocks and tying the display of the repeated blocks (i.e., their “relevance” in Limesurvey-lingo) to a random assignment variable.
Continue reading