Our Blog

We can't shut up.

You are here: Home Our Blog Scientists for Change: NOI's "Experiments in Online Advocacy" rocks!
April 4 2011

Scientists for Change: NOI's "Experiments in Online Advocacy" rocks!

We take a look at the New Organizing Institute's impressive new report, "Experiments in Online Advocacy: A collection of email experiments 2009-2011." After two years and over five million emails, the results are in.

Our friends at the New Organizing Institute just published a great, in-depth research paper titled "Experiments in Online Advocacy" that is one of the first published pieces we've seen that applies meticulous experiment design to some of the thorny challenges around effective online advocacy technique.

NOI researchers Nirmal Mankani and Ethan Roeder partnered up with a several national-scale progressive organizations, including Health Care for America Now, GetEqual, Firedoglake and TrueMajority, to run a number of different experiments related to topics such as:

  • Format of sender name in online advocacy emails
  • Formatting of advocacy emails
  • Time of day for sending advocacy emails
  • Plan-making
  • Video embed formats
  • Combining email and phone calls to prompt action

Before we dive into the results (which we know you're dying for), we wanted to take a quick moment to give NOI a shout-out for their thorough, disciplined methodology.  In our sector, data is too often the plural of anecdote, and it's thus doubly refreshing to see a research report that pays close attention to experiment design, repeated trials, and statistical methodology. Nirmal and Ethan: you've raised the bar!

The Findings

One challenge, though, with scientific rigor is that it can take a while to get around to the results. Here's a quick cheat-sheet, but really, give yourself the hour it will take to read the whole thing. You won't regret it.  (Important disclaimer: your mileage can and will vary.  Do your own testing to find out what works for you.)

1. Varying the email sender name improved email performance slightly for Health Care for America Now (HCAN).

In this test, HCAN found that sending a series of emails with a different sender name slightly improved both open and click-through rates, and that this improvement persisted even when the experimental recipients were merged back into the normal email stream.

2. Sending emails with just a person's name and no organizational identifier led to a slight increase in open rates for GetEQUAL.

GetEQUAL found that sending emails from "Jon Stahl" instead of "Jon Stahl, Groundwire" or "Groundwire" led to a slight increase in open rates, but didn't seem to affect click-through rates.

3. Several variations of email layout/format didn't produce a consistent impact on click-through rate, nor a significant impact on action rate for HCAN.

HCAN tested three layouts for the same advocacy email: one that was almost entirely plain-text, one with a minimal HCAN logo up top, and one with a "take action" call-out box (à la MoveOn). They couldn't find any significant performance difference between these layouts.

4. Including links to a video as an image + link in the body text worked slightly better than linking to the video in a call-out box or as plain-text for GetEQUAL.

GetEQUAL tested three ways of linking to a video from their advocacy email: one with a still from the image inlined into the message text, one with just a plain-text hyperlink and one with the video screenshot in a "MoveOn-style" callout box. The inline image seemed to work the best overall, but the results weren't perfectly consistent in all tests.

5. Asking folks to plan a time + date to make an advocacy phone call produced a slight positive impact on patch-through rates for an anonymous organization, but no impact for FireDogLake.

NOI worked with two organizations to test the psychological concept of "plan-making."  In these experiments, they asked activists to make a specific plan (time and date) to make their phone calls to legislators, then compared their performance to folks who were just asked to call. In one experiment, they found a slight positive impact from the plan-making tactic, but were not able to repeat the finding in a second experiment with a second organization.

6. HCAN was not able to determine an optimal time of day to send advocacy emails.  TrueMajority found a slight bump in open-rates at 3PM, but no significant impact on click-through rates.

Attempting to answer the age-old question, "When is the best time of day to send email," NOI was not able to find any clear recommendation. This experiment was a great example of the importance of running many tests before drawing conclusions.

7. Appending emails to voter file records was not a cost-effective approach to activist recruitment for Families USA.

No surprise here. Spamming the voter file is an expensive way to recruit activists. Not to mention it's unethical.

8. NOI found that combining a phone call and an email produced much higher action rates than just an email.

Finally, NOI did some great experiments with multi-channel mobilization. They found that calling AND emailing activists was much more effective than just emailing them. This seems pretty intuitive, but it's nice to see some hard numbers to back it up.

Comments

Add Yours…

You can add a comment by filling out the form below. Plain text formatting. Comments are moderated.