Ask the Experts: Predictive Modeling Roundtable
Modeling and its efficacy have been under careful review recently, including by Avalon! To see what our nonprofit friends are seeing, I reached out to three industry experts to learn how they utilize predictive modeling to enhance their direct marketing programs.
Here’s the roster of folks who provided input:
- Jeff Huberty, Senior Vice President, Target MarkeTeam, on behalf of the Humane Society of the United States (HSUS)
- Sam Putnam, Senior Data Analyst at the Human Rights Campaign (HRC)
- Mac Mirabile, Director, Analytics at the World Wildlife Fund (WWF)
I sought these experts’ insights on modeling best practices, how and when to model, and how modeling can boost a program’s performance. Before we get into the roundtable, here are some common themes that emerged:
- Modeling tends to work best for larger organizations that have larger universes of names to model. This has been my experience, as well.
- Nonprofits most often use modeling for reinstatement, lapsed donor panels in acquisition, and telemarketing. Appeals, sustainers, and upgrades can also be effectively modeled. With large or disparate universes, modeling can boost your results, especially if they are marginal.
- Some organizations use modeling as a more efficient selection tool if performance results are similar to RFM (recency-frequency-monetary value).
- Modeling does not universally outperform RFM in head-to-head tests—results vary by the program, goals, and size of campaign. Testing will confirm what is best for your program.
In short, modeling is not an all-or-nothing tool—it can be used in a variety of ways to gain insight and yield gains. And now, on to the Roundtable!
Describe how your organization uses predictive modeling in direct response fundraising and what program areas you focus on.
JEFF: HSUS uses predictive modeling extensively and across programs. We used models in appeals, acquisition, reinstatement, lapsed, sustainer invite telemarketing, and mid-level upgrading. Additionally, there are models specific to non-premium giving, which is a priority for us, and we’re looking into further modeling for planned giving and online initiatives.
SAM: HRC uses modeling for direct response—primarily with direct mail and telemarketing. In acquisition and reinstatement, we use a “direct mail responsiveness” model for house names (non-donor supporters) and lapsed names. We model telemarketing responsiveness for house and lapsed, as well as appeals. And we model to target planned giving prospects.
MAC: At WWF, we use modeling for selections of all our campaigns, except a monthly renewal campaign.
How do you make the decision to use modeling, and how often do you model?
SAM: If there’s a performance need—to improve results—or if there’s a large volume of data available and we need to identify the best candidates for marketing out of a large pool, we model. Then we validate models on recent campaign data before rolling out. Roughly speaking, we model quarterly.
MAC: We model all our weekly mail drops. This stemmed from testing modeling against RFM selecting, and the results were clear: For WWF, modeling is a superior way to select the mail targets.
JEFF: The decision to model is based on multiple factors, including program goals, universe targets, schedule, and efficiency. In appeals, the decision may vary, given the timing of the campaign—because modeling can add more lead time to the schedule, and the priority may be to include more recent hotline names. We frequently model reinstatement and lapsed campaigns, as the results exceed traditional segmentation. When conducting a head-to-head test against RFM segmentation, donor modeling improved cost-per-dollar-raised by 20%, while also allowing for volumes to increase by 30%.
Do you model in-house? With outside partners? Both?
MAC: We develop our house models exclusively in-house and have tested some external models within our acquisition program.
JEFF: HSUS models through its agency with a tool called SmartSelect, which includes a collection of models to optimize existing donor mail campaigns. We also create specialized models used for other program areas, such as mid-level upgrades and sustainer invites. HSUS also engages with partners like Wiland and Epsilon for acquisition mailings.
SAM: We model in-house, and we have used outside partners, as well. Outside partner models are often general HRC-responsiveness models that need to be paired with media-preference models. As an example, Catalist built a model that predicted overall likelihood to donate to HRC and then paired that with its media “swim lane” model to help identify those who would also be direct mail responsive.
For house file models, what information do you use to build the model, aside from donation history (e.g., demographic appends, consumer data)?
SAM: We use contact history (contacted previously, how many contacts, ever refused telemarketing, etc.); demographics (if predictive, often age and sometimes income); supporter actions—history, frequency, type of actions, online vs. offline, etc.; and other appended commercial and publicly available data.
MAC: Donation history and relevant transformations are the most predictive variables for WWF. We also do demographic appends and other household-level interest, consumer, and nonprofit behavior appends.
JEFF: While demographic information is not technically appended to the database, we sometimes use it to enhance the model selection capabilities through external licensed data.
What is your overall goal when using modeling, and how do you evaluate its success?
JEFF: Our modeling goals depend on the program and campaign—net revenue, investment levels, quantity, and more can all be factors. Success in modeling is relevant metrics for a campaign above traditional segmentation. Additionally, HSUS uses modeling to help build efficiencies in processes (e.g., ease of selection, strategic insight, budgeting), even if actual results are similar to traditional segmentation.
MAC: Our goal is always to iteratively improve on the prior model’s results and optimally select names to maximize the long-term financial position of WWF. We measure success by how much better the model performs over existing champion models.
SAM: The usual goal: to identify the best prospects for marketing efforts. HRC is also working on models to identify other things, like the optimal ask string level based on gift history, etc. To evaluate success, we usually break model scores into deciles and then measure the performance of each decile. And we usually measure performance based on response rate or productivity.
Have you done head-to-head tests related to modeling and between models?
SAM: Yes, we’ve done head-to-head tests, and we will often include several models in data sets. We are more likely to test different models and choose the best before implementing in production, or we may use one model (like a third-party model) as an independent variable (in our in-house model), so long as the models do not “cross talk.” We also use RFM for segmentation, both as its own piece of data in a data set and also as an independent variable in a logistic regression. In terms of methodology, as a first step, we often review recent complete campaigns to see how models (would have) performed.
MAC: We tend to use head-to-head models on the back end, where we retain the scores for various models (e.g., decision tree, neural network, regression, etc.) and compare their rank-ordering performance. And in our tests against RFM, our initial models (10 years ago) beat RFM by 30 to 40%. Since then, we have iteratively improved our models with additional techniques, model weighting, additional variable transformations, imputations, and data appends—all of which continue to deliver strong results in a negative growth channel (direct response marketing).
JEFF: We have conducted testing using a “yours/mine/ours” methodology that identifies the overlap between the model and traditional segmentation and evaluates the performance of the unique names for each group. As noted above, testing has shown modeling to significantly outperform RFM in reinstatement and lapsed.
Other comments? Results you’d like to share?
SAM: One of the challenges we had for our planned giving program was to identify who, from our database of 6 million members and supporters, would be good prospects to target. Using look-alike modeling, we identified a pool of prospects for a survey mailing. The mailing got a 5% response rate, of which 21% indicated that they plan (or are willing to consider) to include HRC in their planned giving. Some of the best-performing segments were donors we found who had given very consistently for many years, though at relatively low dollar levels.
Modeling can help gain marginal lifts, which can be helpful for campaigns with low response rates. For instance, we built a telemarketing model to predict likelihood to pledge, in which the higher scores resulted in a 23.5% lift in pledge rate.
JEFF: In my experience, it’s important to note that while modeling can provide successful, effective results, if you understand your program and what drives giving, the more complex models may not always be necessary. Also, larger organizations tend to find modeling more effective than organizations with smaller file sizes.
And finally, sometimes each selection method could produce similar results, and the decision to use modeling is made on the ease of implementation (model vs. RFM), which will depend on the organization and its agency.