wiki.communitydata.science Open in urlscan Pro
2001:4b98:dc0:41:216:3eff:fe22:a35  Public Scan

Form analysis
1 forms found in the DOM

/index.php

<form id="searchform" class="mw-search" action="/index.php">
  <input type="hidden" name="title" value=" Special:Search">
  <div class="input-group">
    <input name="search" placeholder="Search CommunityData" aria-label="Search CommunityData" autocapitalize="sentences" title="Search CommunityData [alt-shift-f]" accesskey="f" id="searchInput" class="form-control">
    <div class="input-group-append">
      <button value="Go" id="searchGoButton" name="go" type="submit" class="search-btn searchGoButton" aria-label="Go to page" title="Go to a page with this exact name if it exists"></button>
    </div>
  </div>
</form>

Text Content

 * Not logged in
 * Talk
 * Contributions
 * Create account
 * Log in


Navigation
Main page
Research Blog
Teaching
People
Publications
Projects
Resources
Wiki Functions
Recent changes
Help
Licensing
Page
Discussion
Edit
View history




COVID-19 DIGITAL OBSERVATORY

From CommunityData
Jump to:navigation, search

This project and project page are under active development.


Microscopy image of the virus which causes COVID-19.

This page documents a digital observatory project that aims to collect,
aggregate, distribute, and document public social data from digital
communication platforms in relation to the 2019–20 coronavirus pandemic. The
primary goal is to build on existing data collection efforts to make data
analysis possible by a wider range of social, health, and computational
scientists. The project is being coordinated by the Community Data Science
Collective and Pushshift.


CONTENTS

 * 1 Overview and objectives
 * 2 Stay connected
 * 3 Resources
   * 3.1 Code
     * 3.1.1 Keywords
   * 3.2 Data
     * 3.2.1 Search Engine Results Pages (SERP) Data
     * 3.2.2 Wikipedia data
 * 4 Get Help Using Data
 * 5 Contribute
 * 6 Related projects


OVERVIEW AND OBJECTIVES[EDIT]

As people struggle to make sense of the COVID-19 pandemic, many turn to social
media and social computing systems to share information, to understand what's
happening, and to find new ways to support one another. As scholars, scientists,
technologists, and concerned members of the public, we are building a digital
observatory to understand where and how people are talking about
COVID-19-related topics. The observatory collects, aggregates, and distributes
social data related to how people are responding to the ongoing public health
crisis of COVID-19. The public datasets and freely licensed tools created
through this project will allow researchers, practitioners, and public health
officials to more efficiently understand and act to improve these crucial
sources of information during crises.

The public data we are focused on is available on public webpages and in public
APIs but requires technical skills and computational resources that are less
widely distributed than the ability to analyze data. In particular, we are
attempting to make datasets that researchers can download and analyze on
personal computers.

Everything here is a work in progress as we get the project running, create
communication channels, and start releasing datasets. Learn how you can stay
connected, use our resources as we produce them, and get involved below.


STAY CONNECTED[EDIT]

Subscribe to our low traffic announcement mailing list. You can fill out the
form on the list website or email covid19-announce-request@communitydata.science
with the word 'help' in the subject or body (don't include the quotes). You will
get back a message with instructions.

The email list will contain occasional updates, information about new datasets,
partnerships, and so on. We will not use the list or email addresses for other
purposes.


RESOURCES[EDIT]

The digital observatory data, code, and other resources will exist in a few
locations, all linked from this page. More details on the different datasets and
sources follow below.

Our initial releases should provide a good starting point for investigating
social computing and social media content related COVID-19. We're currently
releasing three types of material: code, keywords, and data.


CODE[EDIT]

For code used to produce the data and get started with analysis we have a github
repository where almost everything lives. If you want to get involved or start
using our work please clone the repository! You'll find example analysis scripts
that walk through downloading data directly into something like R and producing
some minimal analysis to help you get started.

The code used to generate the search engine results pages (SERP) data come from
Nick Vincent's SERP scraping project.

KEYWORDS[EDIT]

We currently use and provide three different types of keywords and search terms:

 * Article names/topics from Wikipedia's WikiProject Covid-19
 * Wikidata entities generated via the "Main items" described by Wikidata's
   WikiProject COVID-19
 * Top 25 daily trending search terms from Google and Bing.

We also provide translations of keywords into many languages by collecting
translations of labels from Wikidata related to the COVID-19 pandemic. This is
done by passing keywords and trending Google "related searches" to the Wikidata
search API. The resulting Wikidata items are tagged with labels and aliases in
many languages. We hope this provides a useful starting point for searches to
discover pandemic related social information in languages beyond English. Code
for this part of the project, including examples for loading the data in Python
and R, is under keywords in our git repository. Similarly, resultant data is
under keywords/csv on our server.


DATA[EDIT]

The best way to find the data is to visit
https://covid19.communitydata.science/datasets/. The search_results directory
contains compressed raw data generated by Nick Vincent's SERP scraping project.
The wikipedia directory has view counts and revision histories for Wikipedia
pages of COVID-19-related articles in .json and .tsv format. The keywords
directory has .csv files with COVID-19 related keywords translated into many
languages and associated Wikidata item identifiers.

SEARCH ENGINE RESULTS PAGES (SERP) DATA[EDIT]

The SERP data in our initial data release includes the first search result page
from Google and Bing for a variety of COVID-19 related terms gathered from
Google Trends and Google and Bing's autocomplete "search suggestions."
Specifically, using a set of six "stem keywords" about COVID-19 and online
communities ("coronavirus", "coronavirus reddit", coronavirus wiki", "covid 19",
"covid 19 reddit", and "covid 19 wiki"), we collect related keywords from Google
Trends (using open source software[1]) and autocomplete suggestions from Google
and Bing (using open source software[2]). In addition to COVID-19 keywords, we
also collect SERP data for the top daily trending queries. Currently, the SERP
data collection process does not specify location in its searches. Consequently,
the default location used is the location of our machine, at Northwestern
University's Evanston campus. We are working on collecting SERP data with
location specified beyond the Chicago area (aka other 'localized' content).

The SERP data is released as a series of compressed archives (7z), one archive
per day, that follow the naming convention covid_search_data-[YYYYMMDD].7z.
Within these compressed archives, there is a folder for each device emulated in
the data collection (currently two: Chrome on Windows and iPhone X) which
contains all of the respective SERP data. Per each device subdirectory, SERP
data itself is organized into folders that are titled by the URL of the search
query (e.g. 'https---www.google.com-search?q=Krispy Kreme'), and each SERP
folder contains three data files:

 * a PNG screenshot of the full first page of results,
 * an mhtml "snapshot" (https://github.com/puppeteer/puppeteer/issues/3658),
 * and a json file with a variety of metadata (e.g. date, the device emulated)
   and a list of every link (<a>) element in the page with its coordinates (top,
   left, bottom, right) in pixels.

WIKIPEDIA DATA[EDIT]

Our initial release provides exhaustive edit and pageview data for the list of
English Wikipedia articles covered by WikiProject Covid-19. Please note that the
edit JSON data of revisions include the full text of every revision made to
articles in English Wikipedia's Wikiproject COVID-19. They are highly compressed
and and expand to more than 20GB of data. Depending on the computer you use, it
may not work to load them into memory all at once for analysis.

Each are updated daily and we are working to add historical data from all other
language Wikipedia editions.


GET HELP USING DATA[EDIT]

As we develop data collection resources and datasets, we will also provide
simple example analysis scripts to demonstrate how you might access, import, and
analyze small subsets of the data we produce. For instance, take a look at the
"example analysis" subdirectory of the wikipedia section of our Github project.

We plan to develop tutorials and demos for the use of the data we release and
particularly welcome contributions that help make these resources more easily
usable by others. In some cases, the data sources are quite large and might not
be suitable for analysis on your personal computer. Wherever possible, we'll try
to build examples that only ingest a small subset of data and/or point you to
useful tools to help make larger scale analyses feasible or easier.

Tips for working with SERP data


CONTRIBUTE[EDIT]

We are eager for collaborators and committed to working openly.

 * Want to contribute to this wiki? Consider creating an account and be bold!
 * Want to contribute to our datasets and/or analysis code? Clone our github
   repository and pitch in with pull requests.

In terms of conduct, we expect all contributors adhere to the Contributor
Covenant.


RELATED PROJECTS[EDIT]

This is an incomplete list of related projects, several of which have additional
and more comprehensive lists of related projects. Please add more!

 * Social media for public health (Johns Hopkins University + University of
   Maryland + GWU). Includes a lengthy list of resources and related projects.
 * COVID-19 Containment measures data from the COVID-19 Forecasting project
   (University of Oxford).
 * Health GeoLab Collaborative listing of (mostly geospatial) data resources
   related to COVID-19.
 * COVID-19 Infodemics Observatory analyzing Twitter data. Created by the
   Laboratory of the FBK research unit.
 * Archive team listing of archive sites related to the Coronavirus
 * Repository with data on COVID-19 from the NYC Department of Health and Mental
   Hygiene (DOHMH)
 * The Citizens and Technology Lab is tracking COVID-related posts on the Reddit
   front page (Cornell and J. Nathan Mattias).
 * General statistics about COVID-19 editing in Wikipedia projects (Diego Saez
   from the Wikimedia Foundation)
 * The New York Times data files with cumulative counts of coronavirus cases in
   the United States, at the state and county level, over time.

Retrieved from
"https://wiki.communitydata.science/index.php?title=COVID-19_Digital_Observatory&oldid=227513"

Tools
What links here
Related changes
Special pages
Printable version
Permanent link
Page information
This page was last edited on 22 February 2022, at 00:20.
Content is available under Attribution-Share Alike 3.0 Unported unless otherwise
noted.
Privacy policy
About CommunityData
Disclaimers