What’s missing from the Edelman/Technorati blogger study

Edelman and Technorati have published the final report on their blogging survey, titled “Public RelationSHIPS: Communications in the Age of Personal Media” (PDF). There are two introductory pieces written by Richard Edelman and Peter Hirshberg (Technorati’s Executive VP), and there’s an interesting comparison between the “old” and “new” PR school.

The page dedicated to the survey’s methodology has an incomplete list of limitations. Here’s what’s missing:

  • A warning that the survey is not a typical “scientific survey”; it has used self-selection and convenience sampling, not random sampling. The results cannot be generalized to a larger population; they are relevant only for the respondents to the survey.
  • There’s no such thing as “margin of error” for a survey using non probabilistic sampling. Saying that “the calculated margin of error for this survey is 3.4% based on the sample size” is nonsense.
  • The size of the sample: (at least) 31000 participants, and an unknown number of people who found the survey questionnaire via links posted on various blogs (sources: Richard Edelman’s presentation on October 6, 2005, and the survey results)
  • The response rate, which is very low: less than 3% (821 respondents).

Disappointing, especially since the problem of survey’s limitations has been discussed in the blogosphere. A timeline of these discussions is available here:

Edelman/Technorati blogger survey – aide-memoire

Updates:

  • Kamy Huyse thinks that “some of the comments especially are interesting to read“. Indeed. But the study has no analysis of the responses to open-ended questions.
  • Buldog Reporter’s Daily ‘Dog has a neutral article about the survey; no word about survey’s limitations.

Note: The initial title of this posting was “Edelman & Technorati blogger study released — what’s missing from its methodology“. The entry has been edited.