Knowing When to Test – and How to Do It Effectively
If You’re Not Testing, You’re Not Learning
Since many of our new clients have come from somewhere else, we’ve had the opportunity to see what others have done. We can address their biggest concerns and produce the annual net income increases they need.
Three problems are alarmingly common:
- Most organizations are not doing enough testing. Nonprofits must test to grow their lists, renew and upgrade existing donors, and generate revenue.
- The most important investment decisions are not getting the attention they deserve. Most testing is small in scope and potential. And these typical “tweak tests” almost never deliver powerful growth or save a program in decline. These nonprofits are missing the chance to explore their biggest marketing questions. As a result, they lack the directional guidance they need to answer fundamental investment decisions. Questions like: “Will we bring in more loyal donors and more reliable long-term net income by investing the next increment of money in recruiting canvass, event, Facebook, or direct mail donors?
- The third dirty little secret about testing is … the vast majority of email “tests” we have seen from others are not statistically significant.
Usually, due to the small number of respondents in these email campaigns or flaws in their test design, these so-called “winning ideas” are anecdotal at best. The perceived differences in response rates might be due to chance. Nonprofits should not rely upon them for decisions about what to do in the future. There is a strong possibility the “losing idea” could actually have been as effective or more. Worse, organizations then believe that they should never do X in the future because Y was better. Any statistician worth their salt will tell you that’s not much better than flipping a coin.
Make Sure to Test the Most Important Things
All too often, the Big Things – that can change the impact of an organization – are not being adequately explored.
The Big Things not getting the attention they deserve fall into two categories:
- The fundraising elements that have already been discovered to be the most determinative of success for similar organizations; and
- The emerging technologies and more far-reaching improvements that have the potential of being the biggest game-changers.
In either case, if these kinds of ideas are never tested, the mere knowledge of them will do little good for you.
This is exactly why Avalon has an in-house analytics team that is fluent in both statistics and fundraising. Our specialty is helping clients decide what to test, and how to interpret findings accurately – for smarter, faster, and more strategic innovation.