How to Conduct an Advertising Effectiveness Conversion Study

The Challenge: To create a model for a conversion study using nothing more than high school math and common, off-the-shelf spreadsheet software (e.g., Microsoft Excel) that could be used by Minnesota communities.

The following instructions for measuring the effectiveness of advertising in bringing people to a destination are illustrated by data gathered on the Ely, MN area using this method. All the data for this study was gathered via mail survey. The survey used to collect data in the Ely area is provided as an example and template. View the survey form. (23 KB .pdf)

Audience: Primarily intended for tourism associations doing marketing for a variety of businesses.

Step 1: Identify the Issues

The issues identified for the Ely area were:

1. Effectiveness of TV and outdoor magazines to determine if they are worth the investment
2. Measure the use of area services
3. Measure the ability of promotion to attract new customers

Step 2: Determine How the Issues Will Be Measured

In the Ely study, the effectiveness measure was Return On Investment (ROI).
Return = $ spent by tourists adjusted for influence of information

• Dollars spent in area
• proportion of travelers influenced by information

Make sure that 1 or 2 questions on the survey will elicit data appropriate to each issue.

Step 3: Write the Survey (or modify the Ely Survey)

Print our survey form (23 KB .pdf) and customize it to fit your needs.

Step 4: Send Out a Sample Mailing

Choose a small group of people to test the survey. Evaluate the responses and correct any ambiguous questions.

Step 5: Determine Your Sample

In determining the size of the group to receive the survey (sample) there are a few rules of thumb:

• Your goal should be a 50% response rate to your survey.
• You must have at least 400 surveys for significant findings
• Some of the surveys you mail will be returned as undeliverable
• Therefore, the sample size should be at least 825.*

Once your size is determined, you must select your sample. You must use a scientifically selected random sample to create your survey group.

You will most likely have to do 3 or 4 mailings to reach the 50% response rate.

* If you don't have 800+ inquiries from visitors, simply send a mailing to everyone. This is called a census. Your minimum response rate should still be 50%.

5A: Comparing different groups or populations

If you want to compare a subset of your population (i.e., you want to know how people responding to television ads responded compared to the population as a whole), use a second survey to measure the conversion rate of that sub-population.

Examples of comparisons from the Ely Area Survey follow.

Conversion Rates by Population

5B: Comparing your community to the other communities in the state.

If you would like to be able to evaluate the effectiveness of advertising in your community in relation to other communities, use the same questions on your survey that other communities have used.

Average Party Size and Number of Trips Taken in Ely Area, MN Compared to NLTA

 

Average Number of Nights in Area

 

Origin of Travelers to Ely Area, MN Compared to the Northern Lights Tourism Alliance Area

Step 6: Sending Out the Survey

You must expect to send out multiple mailings to reach the 50% response rate goal. To send out multiple mailings you will need to keep track of who in your sample, has returned their survey to you. There are many ways to do this. Here is the method used in the Ely Area survey.

1. Assign each person in the sample a number
2. Write the number on the survey being mailed to that person
3. Mark off the names associated with the numbers on the incoming surveys
4. Send out the next mailing only to those whose survey has not been received

The mailings should be sent in 10 day increments. Here is the schedule we used along with the cumulative response rate for each mailing:

Number of Days Elapsed	Mailings	Response Rate After Mailing
0	1st Mailing	23%
10	2nd Mailing	39%
20	3rd Mailing	48%
30	4th Mailing (postcard)	51%

You may also want to consider sending an abbreviated survey as your last mailing. Send only 2 or 3 questions total on a self-addressed, stamped postcard. You can view our postcard (10 KB .pdf).

Step 7: Gathering the Data

Enter your survey responses into a spreadsheet. A standard program such as Microsoft Excel or Quattro Pro is all that is needed.

Simply create a column for each question on the survey and put the responses for each survey in a separate row.

TIP: we found it helpful in analyzing the data to have an extra column for marking who had answered in the multiple choice questions (e.g., What facilities did you use?). That way when we went to tally the total number of responses for that question, we only had to look at one column and not 16 different columns.

Step 8: Analyzing Your Data

In analyzing your data there are usually only two kinds of numbers you will generate:

1. Averages
2. Percentages

Both require that you analyze the data for the question based on the number of people who responded to that question, not the number of people who sent in surveys.

Step 9: Extrapolating From Your Data

Once you have analyzed your data, you know that 35% of the people in your survey that reached you with inquiries actually came to visit. You can assume that at least 35% of your total inquiries have come to visit, but what about the 49% of your sample that did not respond? That's 49% of your total inquiries that are unrepresented in the data.

The people that did not respond are not as likely to have traveled to your area as the people who did respond. However, that does not mean that none of them came.

To find a good estimate of how many people actually visited follow these steps you can follow any of three methods. The simplest method follows.

All examples are based on the data in the Ely Area Survey:

Conversion Rate of those who replied: 35%
Response Rate: 51%
Total Number of Inquires: 29,949.

1. Find the Best Case Scenario

Take the conversion rate from the respondents and assume that the conversion rate will hold for the non-respondents as well

e.g.:
If the conversion rate of those who responded is 35%, then in the "Best Case Scenario" that figure holds and 35% of both the respondents and non-respondents visited

2. Find the Worst Case Scenario

Take the conversion rate from your data (i.e., 35% of the respondents visited the area) and assume that no one else came.

e.g.:
If the conversion rate of those who responded is 35%, and no one else visited, then the conversion rate of the whole is 35% of the 51% response rate

35% x 51% = 17.85% Conversion Rate for all

3. Find the Most Probable Scenario

It is unlikely that either the Best or Worst Case Scenarios are true. The following is the most accurate method for finding the reality that lies in between the two extremes.

Add your best and worst case scenario conversion rates and divide by 2. The average of the two numbers is the best estimate you are going to get without resorting to statistics. (NOTE)

e.g.:
35% + 17.85% = 52.85%
52.85 / 2 =26.43%

Example:

Examples: Extrapolating From Your Data

The following are examples of extrapolations from the Ely Area Survey Data:

 

Number of Groups Visiting Ely Area Who Were Influenced by Information

 

Extrapolated Total Expenditures of Visitors in the Ely Area

 

 

Visitor Expenditures in the Ely Area

 

Inquiries Generated for Information on the Ely Area

 

Return on Advertising Investment (per dollar spent)

 

---

NOTE: In trials, this method was not more than 5% off and was as often an underestimate as an overestimate. To see more options for extrapolating you data with a non-response bias see the Non-Bias Response Rate for Extrapolating Conversion Rates in Mail Survey Research (from the Minnesota State Office of Tourism).

BACK to "Find the Most Probable Scenario"

 

Based on research done in the Ely Area of Minnesota.