Addressing Non-Bias Response Rate in Conversion Studies

This information is taken from a presentation by the Explore Minnesota Research staff at a 1999 Conference.

The Problem: Non-Response Bias

Survey respondents provide different responses than would have been provided by sample members who did not respond

Example: Travelers tend to respond to surveys from travel organizations that serviced them; Non-travelers tend not to respond

Conversion Rates:
Respondents and Non-Respondents

Conversion Rates Graph

Alternatives to Addressing Non-response bias

  1. Intensive follow-up with a small number of sample members who did not respond *
  2. Regression line of cumulative conversion over cumulative response rate (must record survey response dates!)*
  3. Best Case /Worst Case Scenario -- Easiest

* Requires resources that may exceed the scope of in-house research projects

Option 1: Intensive follow-up with small number of sample members who did not respond*

  • Typically by mail, only moderating non-response bias
  • Typically only looking for difference between respondents and non-respondents
  • When done by phone, can play a more useful role in extrapolating to non-respondents

* Requires resources that may exceed the scope of in-house research projects

Conversion Rates:
Mail and Follow-up Phone Respondents

Conversion Rates of Mail and Follow-up Phone Respondents
*Conservative Approach: refusals=did not travel

Option 2: Regression line of cumulative conversion over cumulative response rate*

  • Must record survey response dates on all survey responses
  • Uses statistical procedure to calculate regression equation for extrapolating

* Requires resources that may exceed the scope of in-house research projects

Conversion Rates and Response Rates SS - 1997
Regular Minnesota Sample

Conversion Rates and Response Rates SS - 1997

Conversion by Response

Option 3: Best Case/Worst Case Scenario -- Easiest

  • Best case = conversion rate of respondents holds up for non-respondents
  • Worst case = none of the non-respondents took a trip
  • Probable case = somewhere in between (midpoint)

Conversion Rates:
Best, Worst and Most Probable Cases

Best, Worst , and Most Probable Cases for Conversion Rates

Extrapolating Conversion Rates

Additional Considerations:

  • There is no substitute for a high response rate (50% minimum if possible before alternative methods are applied)
  • May want to combine results from more than one method
  • Or may want to simply use results from other methods as reality checks