3 1 Smith, T W (2013) Survey-Research Paradigms Old and New

Vista previa del material en texto

International Journal of Public Opinion Research Vol. 25 No. 2 2013
� The Author 2012. Published by Oxford University Press on behalf of The World Association
for Public Opinion Research. All rights reserved.
doi:10.1093/ijpor/eds040 Advance Access publication 12 December 2012
Survey-Research Paradigms Old and New
Tom W. Smith
NORC at the University of Chicago
Abstract
Information gathering underwent a dramatic transformation in the 1930s with the rise
of survey research. Now, a growing number of critics are arguing that survey research’s
time has passed and it should and will be replaced by a combination of data mining
from the Internet and the use of various administrative records. The promise and
reliability of the new techniques are assessed, and although they show promise, they
are not at present scientifically superior to survey research for most uses.
A paradigm shift occurred almost 80 years ago in the mid-1930s when Gallup,
Roper, Crossley, and a handful of other innovators pioneered the public
opinion poll (Brick, 2011; Converse, 1987; Groves, 2011b). Before the
advent of polling, politicians, journalists, social scientists, and others had
turned to various sources to measure public opinion and other aspects of
society. These included tracking election returns, the outcomes of referenda,
crowd counts, straw polls, compilations of editorials and news articles collected
by such publications as Public Opinion (taken over by Literary Digest in 1906),
studies of letters to the editor, and, as George Gallup (1957) noted in 1957,
such other evidence as ‘‘letters to congressmen, the lobbying of pressure
groups, and the reports of political henchmen . . . . ’’1 These alternatives were
All correspondence concerning this article should be addressed to Tom W. Smith, NORC at the University
of Chicago, 1155 East 60th Street., Chicago, IL 60637.
1 In general see Albig, 1939 & 1956; Gallup and Rae, 1940; Robinson, 1932. On straw polls (Converse,
1987; Kernell, 2000; Robinson, 1932), on counts of letters, telegrams, etc. (Dexter, 1956; Herbst, 1993;
Kriesberg, 1945), on counting crowds (Herbst, 1993; Splichal, 1999), on informants, especially local party
workers (Gallup and Rae, 1940; Herbst, 1993; Holli, 2002; Robinson, 1932), on newspapers (Albig, 1956;
Herbst, 1993; Splichal, 1999) and on letters to the editor (Gray and Brown, 1970; Wahl-Jorgensen, 2001;
2002).
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
 at Pontificia U
niversidad C
atÃ
³lica de C
hile on O
ctober 3, 2013
http://ijpor.oxfordjournals.org/
D
ow
nloaded from
 
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
http://ijpor.oxfordjournals.org/
supplanted by the polls, and soon, public opinion and poll results became
considered to be almost synonymous with each other. The advent of polling
was a complete game changer. As Elmo Wilson (1947), a researcher at Roper
and other organizations, remarked in 1947, ‘‘25 years ago the possibility of
measuring public opinion with any degree of precision was at least as remote
from public consciousness as the atomic bomb.’’
Now, a rising chorus is asserting that polls are passé, a growingly anti-
quated relic of the past century. They claim that public opinion, consumer
behaviors, and other sociopolitical outcomes can be better measured (less ex-
pensively, more quickly, more easily) by the analysis of Internet usage in
general and of social media in particular, by the data mining of administrative
databases (including the merging of disparate information sources through
such techniques as data fusion), or by a combination of these two alternatives
to traditional surveys. The low cost of mined data is often touted in contrast
to the high and rising cost of traditional survey research (Brick, 2011; Groves,
2011b; O’Connor, B., Balasubramanyan, R., Routledge, B. R., & Smith, N. A.,
2010; Schaeffer and Dykema, 2011). Another attractive aspect of social media
and other Internet analysis is its speed (O’Connor, B., Balasubramanyan, R.,
Routledge, B. R., & Smith, N. A., 2010). For example, on January 20, 2012,
Politico (Van Dongen, 2012) published daily counts covering January 12-18
of ‘‘Facebook chatter’’ about the Republican presidential candidates based
on more than a million ‘‘status updates, postings, and comments.’’
Others emphasize the ease of amassing Internet data: that it can be
compiled routinely, even automatically (Hillygus, 2011; O’Connor,
B., Balasubramanyan, R., Routledge, B. R., & Smith, N. A., 2010; Savage
and Burrows, 2007). As Myron Gutmann (2011), Assistant Director of the
National Science Foundation, has pointed out, there is the promising ‘‘poten-
tial of existing data being more easily accessed. These might be administrative
data collected by federal and state agencies or commercial or opportunistic
data sets, collected for a particular purpose or, like the social media sites,
obtained in the course of business or captured from a device.’’
For example, Mike Savage and Roger Burrows (2007), British sociologists,
argue that the ‘‘glory days’’ of survey research ‘‘are in the past.’’ They write,
‘‘It is unlikely, we suggest, that in the future the sample survey will be a
particularly important research tool . . . . ’’ They believe surveys will be
replaced by ‘‘digital data generated routinely as by-products of their own
transactions . . . . ’’2 Similarly, Carnegie Mellon computer scientist Brendan
O’Connor and others in ‘‘From Tweets to Polls: Linking Text Sentiments
to Public Opinion Time Series’’ (2010) have argued that ‘‘expensive and
2 For follow-ups to Savage and Burrows (2007), see Compton (2007), Savage and Burrows (2009), and
Webber (2009).
S U R V E Y - R E S E A R C H P A R A D I G M S O L D A N D N E W 219
Ignacio
Resaltado
Ignacio
Resaltado
time-intensive polling can be supplemented or supplanted with simple-to-
gather text data that is generated from online social networking.’’
Others are not so ready to consign surveys to the dustbin of history, but
also see them as losing ground to newer techniques. D. Sunshine Hillygus
(2011), a political scientist at Duke, states, ‘‘There has been a noticeable
decline in the prominence of polls in election politics and scholarship.
In forecasting elections, statistical models and prediction markets appear to
beviable alternatives to polling predictions, especially early in the campaign.
In understanding voting behavior, surveys are increasingly replaced by experi-
mental designs or alternative measures of attitudes and behaviors.’’
A star exhibit for the ‘‘alternative measures’’ of electoral behavior is the
research on the 2009 German national election by Andranik Tumasjan and
others (2010) at the Technical University of Munich. They concluded that
‘‘the mere number of tweets mentioning a political party can be considered a
plausible reflection of the vote share and its predictive power even comes close
to traditional election polls.’’ They reported a mean absolute error (MAE) for
vote share of 1.65% from their tweets’ analysis versus an average MAE of
1.14% for six polls.
But their success has not been replicated in a number of studies of U.S.
elections. Analysis using tweets, Facebook likes, and other social media and
studies of Internet searches of races involving the U.S. Senate, governors, and
other contests have typically found mixed to poor results. For example, Lui,
Metaxas, and Mustafaraj (2011) found that Google Trends3 ‘‘was, in general,
not a good predictor of both the 2008 and 2010 elections, as compared to
incumbency, the NYT polls, and even chance.’’ Evaluating Twitter chatter
volume and sentiment analysis of the U.S. election in 2010, Gayo-Avello,
Metaxas, and Mustafaraj (2011) found that ‘‘the mean average error (MAE)
was rather high: 17.1% for Twitter volume and 7.6% for the sentiment ana-
lysis.4 By comparison, MAE for professional polling services is typically about
2-3%.’’ Similarly, Olson and Bunnett (2011) found that Facebook likes5 in
2010 explained 13% of the variation in Senate races, but were ‘‘very weak’’
predictors of the results of gubernatorial races, and for the U.S. House races,
there was ‘‘a slight negative correlation.’’
Estimates of other political matters beside election outcomes have also
been problematic. Green (2011) reported that Twitter followings were
3 Google Trends is a sampling of Google Web searches that computes how many searches have been done
for selected terms relative to the total number of searches done on Google.
4 Volume counts tally how many tweets use certain selected terms such as candidate names. Sentiment
analysis using standard algorithms automatically classifies the tweets as positive or negative toward the
indexed target.
5 Facebook usage has been analyzed in various ways. Some focus on only specific pages (e.g. candidates in
a particular election) versus overall Facebook activity. The component examined most frequently was posted
‘‘likes,’’ but other elements studied include links to external Web sites, within Facebook links, and other
posts.
I N T E R N A T I O N A L J O U R N A L O F P U B L I C O P I N I O N R E S E A R C H220
‘‘a leading indicator of polling’’ early in the Republican presidential contest,
but no statistical analysis was presented to back up this assertion. O’Connor,
B., Balasubramanyan, R., Routledge, B. R., & Smith, N. A., (2010) found that
sentiment ratios on Twitter in 2008 did not correlate with pre-election polls.
As a roundtable discussion of political blogging during the 2008 U.S. presi-
dential campaign indicated, ‘‘Studying these blogs is valuable, but not the
equivalent of studying the body politic nor the political process as a whole’’
(Permutter, 2008).
Success outside politics has been on balance better for the social media and
Internet tallies. These include predicting Best Motion Picture Oscar winners
(Bothos, Apostolou, & Mentzas, 2010), box-office sales for movies (Asur and
Huberman, 2010), initial unemployment claims filed for (Wolfers, 2011),
various retail sales figures (Choi and Varian (2009), and estimates of corrup-
tion (Saiz and Simonsohn, 2007). However, although scoring some notable
results, outcomes have been far from perfect.
Thus far, the new alternatives to survey research in general and public opin-
ion research in particular have not established their superiority either scientific-
ally or empirically. There are many limitations of the new alternatives to surveys.
First, there is the coverage issue (Berube, 2011). Most people do not
regularly use any social media, and many have no access to, or make little
use of, the Internet.
Second, the coverage is not representative of the general population. The
digital divide is still wide. As Couper (2011) has observed, ‘‘While Facebook
users are of interest in their own right, few would suggest that those who use
Facebook are representative of the entire population of any particular country
(and it’s often difficult to attach a country to such users or even properly
identify who such users are).’’ For example, in 2011, 45% of U.S. Facebook
users were aged �25 years.
Third, much social media activity is heavily concentrated in a few thumbs.
For example, among tweets about the 2009 German election, ‘‘Only 4% of all
users accounted for more than 40% of the messages’’ (Tumasjan, A., Sprenger
T. O., Sandner P. G., & Welpe I. M., 2010). Not only is social media activity
skewed, but it also can be intentionally distorted. Gayo-Avello, Metaxas, and
Mustafaraj (2011) note that social media counts ‘‘allow manipulation by spam-
mers and propagandists.’’
Fourth, many Internet data–mining techniques are relatively untested and
clearly have not yet been optimized (Hong, J. I., Zhang, J., & Zimmerman, J.,
2011). This includes the selection of search lexicons, polarity algorithms, and
sentiment analysis; filtering techniques to isolate target populations of interest
(e.g., by age, residence); and smoothing, rolling averages, and aggregation pro-
cedures. For example, the automatic tabulating of tweets, blogs, and other text
in sentiment analysis is crude and often has low reliability (Gayo-Avello,
Metaxas, & Mustafaraj, 2011). Other keyword counts also have problems
S U R V E Y - R E S E A R C H P A R A D I G M S O L D A N D N E W 221
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
due to incomplete lexicons, words with multiple meanings, and ambiguous
terms (Saiz and Simonsohn, 2007). Greater use and more attention to valid-
ation will of course improve and refine these techniques.
Fifth, while much information about Internet and social media usage is
available, much more is restricted by the operators to protect privacy, for
business reasons, or otherwise. For example, Couper (2011) notes that it is
not possible to use Facebook users as a sample frame.
Sixth, documentation of Internet data is often sparse. It is often not clearly
indicated how they are compiled and exactly what they count. For example,
Google Trends uses a fluctuating sampling strategy that is both not explained
and creates variability in measurements over time (Lui, Metaxas, and
Mustafaraj, 2011). Other products are limited in what they provide. Google
Trends does not report hard counts, but only the relative level of searches
from a base point. It also excludes many low-incidence events.
Seventh, the Internet is a rapidly expanding and ever-changing phenom-
enon, and that changeability makes its use and especially its consistent use
over time particularly difficult (Murphy, 2012). Companies, hardware, soft-
ware, platforms, Web sites, and so forth are continually changing, and research
designs using these must continually adapt. At the minimum, this creates
technical challenges. At the maximum, it seriously questions whether valid
reliable comparisons can be made over time.
Eighth, much of the new Internet analysis is, by necessity, restricted to
what routinely appears there and in accessible databases. Often, they do not
include crucial measures of theoretical and empirical importance (Golder and
Macy, 2012). In Groves’ (2011a, b) nomenclature, the Internet-based ‘‘organic
data’’ are not ‘‘designed data’’ and almost always lacks the scope and depth
that a well-designed survey created toexplore a specific research question can
include. Designed research can more completely articulate a full model for
testing hypotheses. Crompton (2008) argues that in contrast to what is possible
with most Internet data, ‘‘the capacity of survey research to simultaneously
explore the impact of a number of different variables on a particular phenom-
enon and the relationships between different variables should not be
underestimated.’’
Ninth, there is the flip side of the limited-access issue, the invasion of
privacy danger. If the veil of inaccessibility is raised, won’t we let in scam-
mers, identity thieves, and worse along with benign researchers? If the door to
data is swung open, who knows who will come in and what will be taken out?
While the push and pull between sharing and privacy continues to move back
and forth, there is a reasonable probability that privacy concerns are growing
and that access will become more restricted in the future (Couper, 2011;
Hong, J. I., Zhang, J., & Zimmerman, J., 2011).
As with using social media and other Internet data, using administrative
records has both many advantages and serious impediments. Administrative
I N T E R N A T I O N A L J O U R N A L O F P U B L I C O P I N I O N R E S E A R C H222
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
data can be useful both independently and when combined with survey data.
Although administrative data are sometimes portrayed as official and definitive
and as representing actual situations and behaviors rather than merely the
recalled and reported information that is collected in surveys, their advantage
over survey data is neither certain nor automatic.
First, there are serious access issues. Much valuable administrative data
are either inaccessible to researchers in general or restricted to a narrow band
of users, such as only by government statistical agencies. Access to individual
administrative records is (and should be) more limited than for anonymous
Internet-use data. However, if access is severely limited and even scholars for
legitimate research applications cannot acquire and use the data, it really does
not matter whether such potentially valuable, but inaccessible, information
exists.
Second, administrative records are variable in quality. Although some
records are carefully collected and checked, this is far from universal. All
records contain some error, and some records are poorly organized, out of
date, badly documented, and with little or no quality-control procedures. For
example, Brick (2011) notes that administrative records may not have ‘‘the
same level of quality as is needed for sampling purposes.’’
Third, administrative records are collected for specific programmatic or
commercial purposes and not to address specific research questions. As a
result, they often lack the needed information to address the research ques-
tions under investigation. For example, voter registration records indicate who
is registered to vote and who voted in particular elections. However, they
contain little other information about the voters. For example, they do not
include voters’ demographics, who they voted for, or any attitudinal informa-
tion. They tell us who voted, but little about who the ‘‘whos’’ are and nothing
about their motivations, reasons, or many other matters.
Of course, administrative records do not have to stand on their own. Their
value can be enhanced by combining together with other data sources. First,
different administrative records can be merged together. This has been done
in a number of European countries where various governmental databases have
been linked together to create a much fuller profile of individuals (Smith,
2011b). However, often there are serious technical, legal, and ethical barriers
to such merging (Stockwell, 2011). Also, while good from a data analysis
perspective, this raises both serious privacy concerns and access issues.
Second, survey data and administrative data can be linked together. Usually
administrative data are appended to survey data (but the data exchanges could go
in the opposite direction). Such data linkages have been done for decades and are
fairly common and often valuable (Blumberg & Cynamon, 1999; Calderwood &
Lessof, 2009; Hewat, 2011; Huynh, Rupp, & Sears, 2000; Moore & Marquis,
1987; Pedace & Bates, 2000; Rhodes & Fung, 2004). However, linking individuals
across lists is often challenging and error prone. Even when a correct linkage is
S U R V E Y - R E S E A R C H P A R A D I G M S O L D A N D N E W 223
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
Ignacio
Resaltado
made, results may be problematic because the timing of survey and administrative
data are out of sync and because of errors in either source.
Although the increased volume of computerized and individual-level data
has facilitated this approach, that has not eliminated the problems of mis- and
non-linkage and of underlying data errors, nor are synthetic merging proced-
ures such as data fusion a simple answer (Van der Putten &Kok, 2010). While
this emerging technique can combine different data sources and create usable
new information, the product is more of a simulation than actual hard data.
Overall, administrative records can be valuable research tools and have
even greater potential in the future (Card, 2011), but as Brick (2011) notes,
‘‘Despite their promise, the use of administrative records has not lived up to
its potential, at least not in US applications [and] (t)he future use of admin-
istrative records, while still promising, faces many challenges.’’
But of course, perhaps things do not need to go as far as the complete data
acquisition and analysis transformation discussed earlier in the text. Although
well short of a paradigm shift, there are important mode and sample shifts to
access panels and river-sampling studies on the Internet. These have already
become ubiquitous in market research and are widely used in other areas
as well (Brick, 2011). There is a continuum of research designs running from
1) traditional surveys using such samples and modes as random digit dialing
(RDD) and computer-assisted telephone interviewing (CATI) or address-based
sampling and face-to-face or mixed modes to 2) probability-based Internet survey
such as Knowledge Networks or e-mail–list samples of more limited target popu-
lations such as employees of a particular company or students at certain uni-
versities to 3) non-probability Internet samples using access panels and/or river
sampling to 4) the data mining of Internet activity (e.g., searches, sites, social
media) and/or administrative records without any interviewing and often with-
out any sampling (i.e., the entire population is analyzed). While useful for certain
purposes, the less extreme, but more common, shift to non-probability, Internet
surveys also has serious problems. The AAPOR Task Force on Online Panels
(Baker et al., 2010) recommended that ‘‘Researchers should avoid
non-probability online panels when one of the research objectives is to accurately
estimate population values.’’ They concluded that ‘‘Non-probability samples are
generally less accurate than probability samples.’’ Similarly, Smyth’s (2012)
methodological review of Web surveys concluded that they were not yet
‘‘ready for prime time.’’ Also, Yeager et al. (2011) found Internet non-probability
studies were inferior to both non-Internet probability samples and Internet-based
probability samples. Similarly, Brick (2011) notes that ‘‘No generally accepted
method of sampling from the Web has been established at this time.’’
However, despite the shortcomings and uncertainties about the new
sources of information, they should not simply be ignored. Social media are
an important and growing form of social interaction and should be studied in
depth. However, thestudy of these social media should not be equated to the
I N T E R N A T I O N A L J O U R N A L O F P U B L I C O P I N I O N R E S E A R C H224
analysis of general public opinion or of society in toto (Bhutta, 2012). They
should be studied for what they are and not for what they are not. They
provide information about public opinion and are a source of information
about many other societal aspects, but are generally more limited than stand-
ard surveys in capturing the full range and complexity of public opinion and
measuring society’s values and behaviors.
Similarly, analysis of Internet searches and Web sites and the traffic to same
can also be valuable and can illuminate many research topics. However, they
will not be the best source for answering all research questions. New technol-
ogies, especially new communication technologies, are important to study in
general, and in particular for the formation and measurement of public opinion
and other social phenomenon. As Splichal (1999) has noted, ‘‘a new technology
of forming and expressing opinions may commonly be associated with changes
in the nature of public opinion.’’ He goes on to indicate that just as the rise
of newspapers changed public opinion in the past, ‘‘computer mediated
communication may again affect the formation of public opinion and its
nature’’ in the future. This development is important to study, but it does
not mean that using the ‘‘new technology’’ can simply replace existing and
more reliable means of measuring public opinion and other matters.
Administrative databases can also be useful for sampling some populations
(especially of course in countries with population registers) and can be used to
check and supplement survey-based data when record linkage can be reliably
carried out. They can be especially valuable for contextualizing data (Smith,
2011a). However, the idea that using administrative databases alone could
replace survey research is clearly misguided.
The most promising future is not one that abandons survey research, but
which augments traditional forms of sampling and interviewing with supple-
mental data from various administrative sources and combines survey-based
individual-level information with aggregate-level data from censuses, admin-
istrative records, and other surveys (Smith & Kim, 2009). Multilevel analysis
combining individual-level and aggregate-level data from various sources is
especially promising (Smith, 2011a, b). Couper (2011) notes that ‘‘these rich
data sources will add much to our understanding of public opinion, but will
not replace surveys in the near future.’’ As Robert Groves (2011a), Director of
the U.S. Census, has observed, ‘‘The combination of designed data [from
surveys] with organic data [from the Internet and other automatic sources]
is the ticket to the future.’’
Examples of hybrid, old-school, new-tech blends include recruiting
samples from Facebook users and combining survey responses with data
from their online pages (Baresch, B., Knight L., Harp D., Yaschur C.,
2011), time-series analysis combining measures of Web activity with data
from real-world shopping studies (Stacey, Pauwels, & Lackman, 2012), aug-
menting individual-level survey-based political data with aggregate-level
S U R V E Y - R E S E A R C H P A R A D I G M S O L D A N D N E W 225
information from administrative records on voter registration and electoral
results (Smith, 2011b), and cross-validating analysis of online discourse on a
topic, media-content analysis, and survey-based studies of the same issue.
So we should not shift the paradigm and jettison survey research, but we
should boost standard surveys of public opinion and other topics with auxil-
iary data from alternative sources, supplement general population surveys with
valuable, but more limited, studies analyzing social media and other Internet
usages, and use the study designs and sources that are best suited for assessing
the many different research questions that we seek to answer.
References
Albig, W. (1939). Public opinion. New York: McGraw-Hill.
Albig, W. (1956). Modern public opinion. New York: McGraw-Hill.
Asur, S., & Huberman, B. A. (2010). Predicting the future with social media.
Unpublished report, Social Computing Lab, HP Lab. doi:10.1109/
WI-IAT.2010.63.
Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J.
M., Dillman, D., Frankel, M. R., Garland, P., Groves, R. M., Kennedy, C.,
Krosnick, J., Lavrakas, P. J., Lee, S., Link, M, Piekarski, L., Rao, K.,
Thomas, R. K., & Zahs, D. (2010). AAPOR report on online panels. Public
Opinion Quarterly, 74, 711–781. doi:10.1093/poq/nfq048.
Baresch, B., Knight, L., Harp, D., & Yaschur, C. (2011). Friends who choose your
news: An analysis of content links on Facebook. Paper presented to the International
Symposium on Online Journalism, Austin.
Berube, D. M. (2011). Researching social media in the 21st century (White paper for
Gutmann, Myron P.). Rebuilding the mosaic: Fostering research in the social,
behavioral, and economic sciences at the national science foundation in the next
decade. Arlington, VA: NSF.
Bhutta, C. B. (2012). Not by the Book: Facebook as a sampling frame. Sociological
Methods & Research, 41, 57–88. doi:10.1177/0049124112440795.
Blumberg, S. J., & Cynamon, M. L. (1999). Misreporting medicaid enrollment: Results
of three studies linking telephone surveys to state administrative records. Proceedings of
the 7th Conference on Health Survey Research Methodology. DHHS Publication
No. 01-1013. Hyattsville, MD.
Bothos, E., Apostolou, D., & Mentzas, G. (2010). Using social media to predict
events with agent-based markets. IEEE Intelligent Systems, 50–58. doi:10.1109/
MIS.2010.152.
Brick, J. M. (2011). The future of survey sampling. Public Opinion Quarterly, 75,
872–888. doi:10.1093/poq/nfr045.
Calderwood, L., & Lessof, C. (2009). Enhancing longitudinal surveys by linking to
administrative data. In P. Lynn (Ed.), Methodology of longitudinal surveys.
Chichester: John Wiley & Sons.
Card, D., Chetty, R., Feldstein, M., & Saez, E. (2011). Expanding access to admin-
istrative data for research in the United States (White paper for Gutmann, Myron
I N T E R N A T I O N A L J O U R N A L O F P U B L I C O P I N I O N R E S E A R C H226
P.). Rebuilding the mosaic: Fostering research in the social, behavioral, and eco-
nomic sciences at the national science foundation in the next decade. Arlington, VA:
NSF.
Choi, H., & Varian, Hal. (2009). Predicting the Present with Google Trends.
Unpublished paper, Google Inc. doi:10.2139/ssrn.1659302.
Converse, J. M. (1987). Survey research in the United States: Roots and emergence,
1890-1960. Berkeley, CA: University of California Press.
Couper, M. P. (2011). The future of modes of data collection. Public Opinion
Quarterly, 75, 889–908. doi:10.1093/poq/nfr046.
Crompton, R. (2008). Forty years of sociology: Some comments. Sociology, 42,
1218–1227. doi:10.1177/0038038508096942.
Dexter, L. A. (1956). What do congressmen hear: The mail. Public Opinion Quarterly,
20, 16–27. doi:10.1086/266594.
Gallup, G. (1957). The changing climate of public opinion research. Public Opinion
Quarterly, 21, 23–27. doi:10.1086/266682.
Gallup, G., & Rae, S. F. (1940). The pulse of democracy: The public–opinion poll
and how it works. Westport, CT: Greenwood press.
Gayo-Avello, D., Metaxas, P., & Mustafaraj, E. (2011). Limits of electoral predictions
using twitter. Paper presented at ICWSM, Barcelona.
Golder, S., & Macy, M. (2012). Social science with social media. ASA Footnotes, 40, 5.
Gray, D. L., & Brown, T. (1970). Letters to the editor: Hazy reflections of public
opinion. Journalism Quarterly, 47, 450–456. doi:10.1177/107769907004700302.
Green, Z. (2011). Twitter predicts polls. Retrieved from http://140elect.com.
Groves, R. M. (2011a). ‘Designated Data’ and ‘Organic Data’. The Director’s Blog.
Retrieved from http://blogs.census.gov.
Groves,R. M. (2011b). Three eras of survey research. Public Opinion Quarterly, 75,
861–871. doi:10.1093/poq/nfr057.
Gutmann, M. P. (2011). Rebuilding the mosaic: Fostering research in the social,
behavioral, and economic sciences at the National Science Foundation in the
next decade. Arlington, VA: NSF.
Herbst, S. (1993). Numbered voice: How opinion polling has shaped American
politics. Chicago: University of Chicago Press.
Hewat, C. (2011, May). Using education databases to improve the quality of the LFS.
Paper presented to the 6th Workshop on LFS Methodology, Wiesbaden.
Hillygus, D. S. (2011). The evolution of election polling in the United States. Public
Opinion Quarterly, 75, 962–981. doi:10.1093/poq/nfr054.
Holli, M. G. (2002). The wizard of Washington: Emil Hurja, Franklin Roosevelt, and
the birth of public opinion polling. New York: Palgrave.
Hong, J. I., Zhang, J., & Zimmerman, J. (2011). Understanding human behavior
atlarge scales through mobile devices (White paper for, M. P.Gutmann).
Rebuilding the mosaic: Fostering research in the social, behavioral, and economic sciences
at the National Science Foundation in the next decade. Arlington, VA: NSF.
Huynh, M., Rupp, K., & Sears, J. (2000). The Assessment of Survey of Income and
Program Participation (SIPP) benefit data using longitudinal administrative records
(SIPP Report No. 238) on line at http://www2.census.gov/prod2/sipp/wp/
SIPP_WP_238.pdf.
S U R V E Y - R E S E A R C H P A R A D I G M S O L D A N D N E W 227
Kernell, S. (2000). Life before polls: Ohio politicians predict the 1828 Presidential
vote. Political Science and Politics, 33, 569–574. doi:10.2307/420860.
Kriesberg, M. (1945). What Congressmen and administrators think of the polls. Public
Opinion Quarterly, 9, 333–337.
Lui, C., Metaxas, P. T., & Mustafaraj, E. (2011, March). On the predictability of the
U.S. elections through search volume activity. Paper presented to the e-Society
Conference, Avila, Spain.
Moore, J. C., & Marquis, K. H. (2010). Using administrative record data to evaluate
the quality of survey estimates, research report series, survey methodology 2010-10.
Washington, DC: US Census Bureau. (Original work published 1987).
Murphy, J. (2012, August). Déjà vu too: Statisticians don’t like non-probability.
Survey Post. Retrieved from https://blogs.rti.org/surveypost/2012/08/09/
facebook-and-surveys-thoughts-on-going-not-by-the-book/.
O’Connor, B., Balasubramanyan, R., Routledge, B. R., & Smith, N. A. (2010). From
tweets to polls: Linking text sentiments to public opinion time series. Tepper
School of Business Paper 559. Carnegie Mellon University.
Olson, S., & Bunnett, W. (2011). Facebook predicts electoral victory, or not?
Retrieved from http://trilogyinteractive.com.
Pedace, R., & Bates, N. (2000). Using administrative records to assess earnings
reporting error in the survey of income and program participation. Journal of
Economic and Social Measurement, 26, 173–192.
Permutter, D. D. (2008). Political blogging and campaign 2008: A roundtable.
International Journal of Press/Politics, 13, 160–170. doi:10.1177/1940161208315742.
Rhodes, A.E., & Fung, K. (2004). Self-reported use of mental health services versus
administrative records: Care to recall? International Journal of Methods in Psychiatric
Research, 13, 165–175. doi:10.1002/mpr.172.
Robinson, C. E. (1932). Straw votes: A study of political prediction. New York:
Columbia University Press.
Saiz, A., & Simonsohn, U. (2007). Downloading wisdom from online crowds.
Unpublished report, Wharton School, University of Pennsylvania. doi: 10.2139/
ssrn.990021.
Savage, M., & Burrows, R. (2007). The coming crisis of empirical sociology.
Sociology, 41, 885–899. doi:10.1177/0038038507080443.
Savage, M., & Burrows, R. (2009). some further reflections on the coming crisis of
empirical sociology. Sociology, 43, 762–771. doi:10.1177/0038038509105420.
Schaeffer, N. C., & Dykema, J. (2011). Questions for surveys: Current trends and
future direction. Public Opinion Quarterly, 75, 909–961. doi:10.1093/poq/nfr048.
Smith, T. W. (2011a). Contextualizing social-science data (White paper for M. P.
Gutmann). Rebuilding the Mosaic: Fostering research in the social, behavioral, and eco-
nomic sciences at the national science foundation in the next decade. Arlington, VA: NSF.
Smith, T. W. (2011b). The report on the international workshop on using multi-level
data from sample frames, auxiliary databases, paradata, and related sources to detect
and adjust for nonresponse bias in surveys. International Journal of Public Opinion
Research, 23, 389–402. doi:10.1093/ijpor/edr035.
Smith, T. W., & Kim, J. (2009). An assessment of the multi-level integrated database
approach (MIDA) (GSS Methodological Report No. 116). Chicago: NORC.
I N T E R N A T I O N A L J O U R N A L O F P U B L I C O P I N I O N R E S E A R C H228
https://blogs.rti.org/surveypost/2012/08/09/facebook-and-surveys-thoughts-on-going-not-by-the-book/
https://blogs.rti.org/surveypost/2012/08/09/facebook-and-surveys-thoughts-on-going-not-by-the-book/
http://trilogyinteractive.com
Smyth, J. D. (2012, January). On the frontier of Web surveys. Paper presented to
NORC, Chicago.
Splichal, S. (1999). Public opinion: Developments and controversies in the twentieth cen-
tury. Lanham, MD: Rowman & Littlefield.
Stacey, E. C., Pauwels, K. H., & Lackman, A. (2012). Beyond likes and tweets: How
conversation content drives store and site traffic. Unpublished report, The Center
for Measureable Marketing. Retrieved from http://pages.stern.nyu.edu/�atakos/
centerresearch/likesandtweets.pdf.
Stockwell, I. (2011). The role of data and analytical methods in public discovery and
implementation (White paper for M. P. Gutmann). Rebuilding the mosaic: Fostering
research in the social, behavioral, and economic sciences at the national science founda-
tion in the next decade. Arlington, VA: NSF.
Tumasjan, A., Sprenger, T. O., Sandner, P. G., & Welpe, I. M. (2010). Predicting
elections with twitter: What 140 characters reveal about political sentiment.
Proceedings of the Fourth International AAAI Conference on Weblogs and Social
Media. Washington, DC, USA.
Van der Putten, P., & Kok, J. N. (2010). Using data fusion to enrich customer databases
with survey data for database marketing. In J. Casillas & F. J. M. Lopez (Eds.),
Marketing intelligent systems using soft computing. Springer studies in fuzziness and soft
computing, (Vol. 258, pp. 113–130). doi:10.1007/978-3-642-15606-9_11.
Van Dongen, R. (2012, January 20). Paul beats Mitt in Facebook chatter. Retrieved
from http://political.com.
Wahl-Jorgensen, K. (2002). The construction of the public in letters to the editor:
Deliberative democracy and the idiom of insanity. Journalism, 3, 183–204.
doi:10.1177/146488490200300203.
Wahl-Jorgensen, K. (2001). Letters to the editor as a forum for public deliberation:
modes of publicity and democratic debate. Critical Studies in Media Communications,
18, 303–320. doi:10.1080/07393180128085.
Webber, R. (2009). Response to ‘The Coming Crisis of Empirical Sociology’: An
outlineof the research potential of administrative and transactional data. Sociology,
43, 169–178. doi:10.1177/0038038508099104.
Wilson, E. C. (1947). The Measurement of Public Opinion. Annals of the American
Academy of Political and Social Science, 250, 121–129. doi:0.1177/00027162472
5000117.
Wolfers, J. (2011, May 25). Google’s new correlation mining tool: It works! Retrieved
from http://www.freakonomics.com.
Yeager, D. S., Krosnick, J. A., Chang, L., Javitz, H. S., Levendusky, M. S.,
Simpser, A., & Wang, R. (2011). Comparing the accuracy of RDD telephone sur-
veys and Internet surveys conducted with probability and non-probability samples.
Public Opinion Quarterly, 75, 709–747. doi:10.1093/poq/nfr020.
S U R V E Y - R E S E A R C H P A R A D I G M S O L D A N D N E W 229
http://pages.stern.nyu.edu/atakos/centerresearch/likesandtweets.pdf
http://pages.stern.nyu.edu/atakos/centerresearch/likesandtweets.pdfhttp://pages.stern.nyu.edu/atakos/centerresearch/likesandtweets.pdf
http://political.com
http://www.freakonomics.com