Frequently asked questions about election forecasting by Interdisciplinary Scientific Research
Survey, performance feedback, and prizes
Privacy, anonymity, and confidentiality
Respondents and sampling
How the forecasts will be used
About Interdisciplinary Scientific Research and commissioning a forecast
(Note: Many of these questions and answers apply to prospective respondents for our 2008 general election surveys.)
Questions about the survey, performance feedback, and prizes
Q. What kinds of questions are in the survey?
A. You will be asked to predict the outcomes of statewide races and ballot measures in the general election. Other questions focus on the candidates and measures you support, and your general views of politics.
Q. How long will the survey take?
A. It takes about 10 minutes to complete the survey online.
Q. What should I do if I feel too uncertain to make a prediction for a particular race or measure?
A. It is important for you to give a prediction for each race and ballot measure, even if you just have to give your best guess. Your guess is likely to be more accurate than that of a non-expert. The survey will not continue unless you give a prediction for each contest on that page.
Q. What do I get if I participate in the survey?
A. After the election, we will send you a scorecard that shows how accurate your predictions are and how your performance compares with that of the other experts who participate. Prizes will also be awarded based on accuracy. The respondent who predicts election outcomes most accurately will win a $100 gift certificate to a major online retailer (Amazon, eBay, or Target) of his or her choice. Other gift certificate prizes will also be awarded according to respondents' prediction accuracy (see below).
|Prediction accuracy rank
| Q. How is the prediction accuracy
A. The primary accuracy score is simply the percentage of election winners (offices and ballot measures) correctly predicted. Ties are likely to be common, so ties will be split by a secondary accuracy score, which is the average (mean) difference between predicted and actual margins of victory (with lower scores indicating greater accuracy).
Q. What should I do if I need to close my web browser window before completing the survey? How can I finish the survey at another time?
A. Before closing your browser window, make sure to copy the link or URL for the current page of the survey. Save this link and return to it when you are ready to complete the survey. If you weren't able to save this link, please contact Devon Brewer by telephone at 206-985-2398 or by e-mail at email@example.com for instructions on how to continue.
Q. Can I complete the survey some other way than online?
A. Yes, you can complete the survey by phone, fax, or e-mail if you wish. These options might not allow you to respond anonymously, although your responses and information will still be keep strictly confidential (see questions about privacy, anonymity, and confidentiality below). If you want to respond in one of these other ways, please contact Devon Brewer by telephone at 206-985-2398 or by e-mail at firstname.lastname@example.org to make the arrangements.
Q. What can I do if I want to update my predictions after I complete the survey?
A. Once you complete the survey, you won't be able to re-enter it. However, three weeks before the election, you will be invited to participate in a very brief second survey. In this survey, you can give your updated predictions. If you participate in the second survey, your accuracy score for determining prizes will be based on the survey in which your predictions are more accurate. In other words, participating in the second survey gives you a chance to improve your accuracy score.
Questions about privacy, anonymity, and confidentiality
Q. Are my responses and information confidential?
A. All responses and other information you provide are kept strictly confidential for the purposes of producing election forecasts only and will not be shared with, sold to, or traded with any other organization or individual (see the terms and conditions for more information). Also, forecasts and reports will based on aggregated responses only, and individual expert respondents will never be identified, even indirectly.
Q. Can I participate anonymously?
A. Yes, you can participate anonymously.
Q. If I participate anonymously, how will I get my performance feedback and prize?
A. There are two ways you can receive feedback about your performance and learn whether you won a prize. The first way is to give us an e-mail address (for example, with Yahoo, Gmail, or Hotmail) for which you have registered anonymously or with fictitious information. Then we will send our correspondence to you at that e-mail address.
Another way is to record the ID number we give you at the end of the survey. You would then keep this number and return to our election forecasting web site after the general election to query our online database to receive your performance feedback and learn the amount of your prize. Because the prizes are gift certificates to online retailers, the only thing you need to redeem a prize is the gift certificate number -- which you would find by querying our online database with your ID number. If you choose this option, however, you must remember to visit our site and enter the correct ID number. Otherwise, we would have no way to contact you. Please note if you choose to participate anonymously in this way that you will not have an opportunity to improve your accuracy score by completing the second brief survey.
Q. Do you track respondents' IP addresses?
A. No, we do not track respondents' IP addresses. In fact, a few days after sending our initial e-mail invitation message to experts, we send another message to ask experts to complete the survey if they have not already done so, because we have no way to know which respondents have already completed the survey.
Q. To access the survey via the link you sent me by e-mail, I also need to enter the unique one-time password you sent. Can you connect that password to my e-mail address and identity?
A. No, after we send experts the invitation to participate by e-mail, we permanently delete from our records the password that had been assigned to you. No connection remains between your password and your e-mail address and identity. Furthermore, for added protection, we delete all entered passwords from the survey data file.
Q. How do respondents indicate their informed consent to participate?
A. As noted on the first page of the online survey, continuing further with the survey is how respondents indicate their informed consent.
Q. What other terms and conditions are there?
A. Click here for other details on the terms and conditions of participation and survey data.
Questions about respondents and sampling
Q. Are you surveying experts only?
A. Yes. Prior research indicates that non-experts tend to have less accurate predictions than experts about election outcomes.
Q. How do you identify experts?
A. We invite to participate those individuals who have publicly demonstrated experience in studying, reporting on, or participating in statewide politics. Specifically, we recruit expert political observers in academia, the media, the state legislature, and lobbyists to state government.
Q. I would like to forward the e-mail invitation to someone else so they can participate. Will that work?
A. No, the link and unique password to access the survey that we sent you will only work one time, and they are intended for your use only. However, if there are types of people you think should be surveyed, please let us know in the survey where you are asked for your comments and suggestions. Depending on the advice we receive, we can modify our selection of experts in the future.
Questions about the forecasting method
Q. How can you make accurate forecasts if you don't have a random or probability sample of respondents?
A. Our aim is not to sample voters to describe their opinions, but to sample experts who have good knowledge of the election contests as well as voters' opinions and voting tendencies. Knowledge of political races and ballot measures, and of the electorate and its voting patterns, is unevenly distributed in the community. Individuals whose jobs involve, in part, assessing the political landscape tend to be quite accurate in predicting election outcomes when their predictions are aggregated. Collecting predictions from random samples of voters in a state would needlessly increase the cost of producing a forecast and reduce its accuracy.
Q. Aren't people's predictions of election outcomes biased toward their own preferences? That is, don't people's predictions reflect wishful thinking (also known as desirability bias or the false consensus effect)?
A. There is a tendency for people to predict election outcomes consistent with their own voting preferences. However, this bias is less prominent in expert political observers. By including a diverse sample of experts, who have different roles in observing politics and different political orientations and values, we can further minimize the influence of such biases, as they will cancel each other. Also, we provide respondents monetary and other incentives for accuracy. Moreover, with our analytic procedures, we can detect when this bias is strong and adjust our forecast accordingly.
Q. Why don't you simply conduct a political poll instead?
A. There are two main reasons we don't conduct polls. First, scientific polls, regardless of methodology, are much more expensive than our forecasting approach. Second, our approach is as accurate as polling carried out 7-8 weeks before an election. Political polls provide snapshots of public opinion at varying points before an election; our forecasts, however, are true projections of election outcomes.
Q. Why don't you create an election prediction market instead?
A. There are several reasons why prediction markets are not a good option for forecasting most statewide election races and ballot measures. First, prediction markets involving real money wagers are, in most circumstances, illegal in the U.S. Many knowledgeable experts might not participate in prediction markets without some sort of monetary incentive. Second, the prediction markets that do exist for statewide races (some governor and US Senate races) are small and illiquid, meaning that they are likely to be quite inaccurate and unattractive to most potential bettors. States with smaller populations are particularly unsuitable for election prediction markets. Third, prediction markets' forecasts are effectively weighted by the size of bettors' wagers, not by their knowledge or probable accuracy. Many different factors are involved in the size of bettors' wagers (such as amount available to wager, aggressiveness, risk aversion, etc.). Fourth, bettors in prediction markets, like other markets, do not make their predictions independently of each other, and are susceptible to a "herd" mentality, diminishing the value of individual bettors' insights. Thus, even if prediction markets were a feasible alternative, it is likely that our approach will still be more accurate.
Q. What techniques do you use to produce election forecasts from experts' predictions?
A. Our proprietary analytic procedures are modified from standard information aggregation techniques. In brief, our methods involve synthesizing respondents' predictions into overall collective judgments of predicted winners and margins of victory. To do so, we weight respondents' predictions by individual respondents' estimated accuracy (based on responses to a variety of questions in the survey and other information). Our methods allow a statistical determination of how reliable the forecasts are as representations of the aggregated opinions of expert political observers in the state. Moreover, with our procedures, we can assess different types of respondent bias and make corresponding adjustments in forecasts, as necessary.
Q. Will you report on the accuracy of your forecasts?
A. Yes. We have posted and distributed a report that includes a full evaluation of the accuracy of our forecasts for the 2008 general election in South Dakota, as well as a comparison with polls and other predictions.
Questions about how the forecasts will be used
Q. What will you do with my predictions and other survey responses?
A. Your responses will be used to forecast election outcomes. This information may be used in scientific research articles and reports. Forecasts and reports will based on aggregated responses only.
Q. When will you make your forecasts available?
In 2008, all forecasts were made public for free 10 days before the general election.
Questions about Interdisciplinary Scientific Research
Q. What is Interdisciplinary Scientific Research?
A. Interdisciplinary Scientific Research is a nonpartisan, nonpolitical, scientific research firm.
Q. What is Interdisciplinary Scientific Research's interest in forecasting elections in my state?
A. We intend to provide forecasts for elections that have historically had few or no systematic forecasts and to refine innovative election forecasting methods.
Q. What experience do you have in maintaining privacy, confidentiality, and anonymity for research participants?
A. We have conducted research on sensitive topics for federal agencies, universities, and other research partners since 1988 without a single breach of confidential information (see our Projects and Publications pages for more information on our past work). Maintaining the confidentiality of your information is our highest responsibility and concern.
Q. I would like you to make a forecast for an election. Can you do it?
A. Please discuss your interests with us so we can determine whether our approach might be useful. Our approach works best for state and local elections, particularly those with multiple races and/or ballot measures. Contact Devon Brewer by telephone at 206-985-2398 or by e-mail at email@example.com.
Q. How much would it cost to commission an election forecast?
The price for a comprehensive set of election forecasts, like those for the 2008 general election in South Dakota is about 1/5th the cost of a comparable interviewer-administered poll or about 1/3rd the cost of a comparable automated (interactive voice response) poll. These comparisons are based on the lowest price scientific polls of each type in the industry.
Q. I have another question that isn't listed here. How can I get an answer to my question?
A. Send us an e-mail message at firstname.lastname@example.org, or call us at 206-985-2398.
Copyright © 2008 Interdisciplinary Scientific Research, LLC. All rights reserved worldwide.