Web-scraping for Campaigns
As the primaries approach, I am experiencing a mix of angst, FOMO, and excitement. One of my largest concerns is that progressive campaigns are stuck in a sort of antiquated but nonetheless entrenched workflow. Google Sheets reign in metric reporting. Here I want to present one use case (of a few more to come) where R can be leveraged by your data team.
In this post I show you how to scrape the most recent polling data from FiveThirtyEight. FiveThirtyEight aggregates this data in an available way. This can allow you as a Data Manager to provide a useful report to your Media Manager.
As always, please feel free to contact me on Twitter @josiahparry if you have any questions or want to discuss this further.
Polling use case
A very important metric to keep track of is how your candidate is polling. Are they gaining a lead in the polls or falling behind? This data is often reported via traditional news organizations or some other mediums. The supposed demi-God and mythical pollster Nate Silver’s organization FiveThirtyEight does a wonderful job aggregating polls. Their page National 2020 Democratic Presidential Primary Polls has a table of the most recent polls from many different pollsters.
In this use case we will acquire this data by web scraping using rvest. We will also go over ways to programatically save polls results to a text file. Saving polling results can allow you present a long term view of your candidate’s growth during the quarter.
Understanding rvest
This use case will provide a cursory overview of the package rvest. To learn more go here.
Web scraping is the process of extracting data from a website. Websites are written in HTML and CSS. There are a few aspects of these languages that are used in web scraping that is important to know. HTML is written in a series of what are call tags. A tag is a set of characters wrapped in angle brackets—i.e. <img>.
With CSS (cascading style sheets), web developers can give unique identifiers to a tag. Classes can also be assigned to a tag. Think of these as group. With web scraping we can specify a particular part of a website by it’s HTML tag and perhaps it’s class or ID. rvest provides a large set of functions to make this simpler.
Example
For this example we will be scraping FiveThirtyEight’s aggregated poll table. The table can be found at https://projects.fivethirtyeight.com/2020-primaries/democratic/national/.
Before we begin, we must always prepare our workspace. Mise en place.
library(rvest)## Loading required package: xml2library(tidyverse)## ── Attaching packages ──────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──## ✔ ggplot2 3.1.0 ✔ purrr 0.3.0.9000
## ✔ tibble 2.1.1 ✔ dplyr 0.7.8
## ✔ tidyr 0.8.2 ✔ stringr 1.4.0
## ✔ readr 1.2.1 ✔ forcats 0.3.0## ── Conflicts ─────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ readr::guess_encoding() masks rvest::guess_encoding()
## ✖ dplyr::lag() masks stats::lag()
## ✖ purrr::pluck() masks rvest::pluck()The first thing we will have to do is specify what page we will be scraping from. html_session() will simulate a session in an html browser. By providing a URL to html_session() we will then be able to access the underlying code of that page. Create an object called session by providing the FiveThirtyEight URL to html_session().
session <- html_session("https://projects.fivethirtyeight.com/2020-primaries/democratic/national/")The next and most important step is to identify which piece of HTML code contains the table. The easiest way to do this is to open up the webpage in Chrome and open up the Inspect Elements view (on Mac - ⌘ + Shift + C). Now that this is open, click the select element button at the top left corner of the inspection pane. Now hover over the table.
You will see that the HTML element is highlighted. We can see that it is a table tag. Additionally we see that there are two different classes polls-table and tracker. To specify a class we put a preceding . to the class name—i.e. .class-name. If there are multiple classes we just append the second class name to it—i.e. .first-class.second-class. Be aware that these selectors can be quite finicky and be a bit difficult to figure out. You might need to do some googling or playing around with the selector.
To actually access the content of this HTML element, we must specify the element using the proper selector. html_node() will be used to do this. Provide the html session and the CSS selector to html_node() to extract the HTML element.
session %>%
html_node(".polls-table.tracker")## {xml_node}
## <table class="polls-table tracker">
## [1] <thead class="hide-mobile" id="table-header"><tr>\n<th class="new">< ...
## [2] <tbody>\n<tr class="visible-row" data-id="97723">\n<!-- Shared--><td ...Here we see that this returns on object of class xml_node. This object returns some HTML code but it is still not entirely workable. Since this is an HTML table we want to extract we can use the handy html_table(). Note that if this wasn’t a table but rather text, you can use html_text().
session %>%
html_node(".polls-table.tracker") %>%
html_table()Take note of the extremely informative error. It appears we might have to deal with mismatching columns.
session %>%
html_node(".polls-table.tracker") %>%
html_table(fill = TRUE) %>%
head()## Dates Pollster Sample
## 1 • Jun 6-10, 2019503 RV Jun 6-10, 2019 A-Quinnipiac University
## 2 • Jun 3-9, 201917,012 LV Jun 3-9, 2019 B-Morning Consult
## 3 • May 29-Jun 5, 20192,271 RV May 29-Jun 5, 2019 B+Ipsos
## 4 • May 29-Jun 5, 20192,525 A May 29-Jun 5, 2019 B+Ipsos
## 5 • Jun 2-4, 2019500 RV Jun 2-4, 2019 BYouGov
## 6 • Jun 2-4, 2019550 A Jun 2-4, 2019 BYouGov
## Sample Biden Sanders Harris Warren O'Rourke Buttigieg Booker Klobuchar
## 1 503 RV 30% 19% 7% 15% 3% 8% 1%
## 2 17,012 LV 37% 19% 7% 11% 4% 7% 3%
## 3 2,271 RV 31% 14% 6% 9% 3% 5% 2%
## 4 2,525 A 30% 15% 6% 8% 4% 5% 2%
## 5 500 RV 27% 15% 9% 12% 2% 10% 2%
## 6 550 A 27% 16% 8% 11% 2% 9% 2%
## Castro Yang Gillibrand Hickenlooper Gabbard Delaney Inslee Ryan Bullock
## 1 1% 0% 1% 0% 0% 0% 0% 0% 1%
## 2 2% 1% 1% 1% 1% 1% 1% 1% 1%
## 3 2% 1% 1% 0% 1% 1% 0% 0% 1%
## 4 1% 0% 1% 0% 1% 1% 0% 0% 1%
## 5 1% 0% 1% 0% 1% 1% 1% 0% 0%
## 6 1% 0% 1% 0% 1% 1% 1% 0% 0%
## de Blasio Bennet Williamson Gravel Swalwell Moulton Messam H. Clinton
## 1 0% 0% 0% 0% 0% 0% 0% 0%
## 2 1% 0% 1% 0% 0% 0%
## 3 0% 1% 0% 0% 0% 0% 0% 0%
## 4 0% 1% 0% 0% 0% 0% 0% 0%
## 5 1% 2% 0% 0% 0% 0% 0% 0%
## 6 1% 2% 0% 0% 0% 0% 0% 0%
## Bloomberg M. Obama Brown Kerry Abrams Holder McAuliffe Winfrey Ojeda
## 1
## 2
## 3
## 4
## 5
## 6
## Trump Cuomo Avenatti Kennedy Patrick Zuckerberg Pelosi Garcetti Newsom
## 1
## 2
## 3
## 4
## 5
## 6
## Steyer Schultz Kaine Johnson Kucinich Lee Scott Sinema Warner NA
## 1
## 2
## 3
## 4
## 5
## 6
## NA
## 1 Biden30%Sanders19%Warren15%Buttigieg8%Harris7%O'Rourke3%Booker1%Klobuchar1%Yang1%Ryan1%Gillibrand0%Castro0%Gabbard0%Inslee0%Hickenlooper0%Delaney0%Williamson0%Messam0%Swalwell0%Moulton0%Bennet0%Bullock0%de Blasio0%Gravel0%
## 2 Biden37%Sanders19%Warren11%Buttigieg7%Harris7%O'Rourke4%Booker3%Klobuchar2%Bennet1%Bullock1%Castro1%Delaney1%Gabbard1%Gillibrand1%Hickenlooper1%Inslee1%Yang1%Ryan1%de Blasio0%Moulton0%Swalwell0%Williamson0%
## 3 Biden31%Sanders14%Warren9%Harris6%Buttigieg5%O'Rourke3%Booker2%Klobuchar2%Castro1%Gabbard1%Hickenlooper1%Yang1%Ryan1%de Blasio1%Gillibrand0%Bullock0%Inslee0%Delaney0%Williamson0%Messam0%Swalwell0%Moulton0%Bennet0%Gravel0%
## 4 Biden30%Sanders15%Warren8%Harris6%Buttigieg5%O'Rourke4%Booker2%Klobuchar1%Gabbard1%Hickenlooper1%Yang1%Ryan1%de Blasio1%Castro0%Gillibrand0%Bullock0%Inslee0%Delaney0%Williamson0%Messam0%Swalwell0%Moulton0%Bennet0%Gravel0%
## 5 Biden27%Sanders15%Warren12%Buttigieg10%Harris9%Booker2%de Blasio2%O'Rourke2%Bullock1%Delaney1%Gabbard1%Hickenlooper1%Klobuchar1%Yang1%Bennet0%Castro0%Gillibrand0%Gravel0%Inslee0%Messam0%Moulton0%Ryan0%Swalwell0%Williamson0%
## 6 Biden27%Sanders16%Warren11%Buttigieg9%Harris8%Booker2%de Blasio2%O'Rourke2%Bullock1%Delaney1%Gabbard1%Hickenlooper1%Klobuchar1%Yang1%Bennet0%Castro0%Gillibrand0%Gravel0%Inslee0%Messam0%Moulton0%Ryan0%Swalwell0%Williamson0%This is much better! But based on visual inspection the column headers are not properly matched. There are a few things that need to be sorted out: there are two date columns, there are commas and percents where numeric columns should be, the column headers are a little messy, and the table isn’t a tibble (this is just personal preference).
We will handle the final two issues first as they are easiest to deal with. The function clean_names() from janitor will handle the column headers, and as_tibble() will coerce the data.frame into a proper tibble. Save this semi-clean tibble into an object called polls.
polls <- session %>%
html_node(".polls-table.tracker") %>%
html_table(fill = TRUE) %>%
janitor::clean_names() %>%
as_tibble()
polls## # A tibble: 116 x 59
## x dates pollster sample sample_2 biden sanders harris warren
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 • Jun … Jun 6-1… A-Qui… 503 RV 30% 19% 7%
## 2 • Jun … Jun 3-9… B-Mor… 17,012 LV 37% 19% 7%
## 3 • May … May 29-… B+Ips… 2,271 RV 31% 14% 6%
## 4 • May … May 29-… B+Ips… 2,525 A 30% 15% 6%
## 5 • Jun … Jun 2-4… BYouG… 500 RV 27% 15% 9%
## 6 • Jun … Jun 2-4… BYouG… 550 A 27% 16% 8%
## 7 • Jun … Jun 1-2… C+Har… 431 RV 35% 16% 4%
## 8 • May … May 27-… B-Mor… 16,587 LV 38% 19% 7%
## 9 • May … May 28-… A-CNN… 412 RV 32% 18% 8%
## 10 • May … May 29-… C+Har… 471 RV 36% 17% 8%
## # … with 106 more rows, and 50 more variables: o_rourke <chr>,
## # buttigieg <chr>, booker <chr>, klobuchar <chr>, castro <chr>,
## # yang <chr>, gillibrand <chr>, hickenlooper <chr>, gabbard <chr>,
## # delaney <chr>, inslee <chr>, ryan <chr>, bullock <chr>,
## # de_blasio <chr>, bennet <chr>, williamson <chr>, gravel <chr>,
## # swalwell <chr>, moulton <chr>, messam <chr>, h_clinton <chr>,
## # bloomberg <chr>, m_obama <chr>, brown <chr>, kerry <chr>,
## # abrams <chr>, holder <chr>, mc_auliffe <chr>, winfrey <chr>,
## # ojeda <chr>, trump <chr>, cuomo <chr>, avenatti <chr>, kennedy <chr>,
## # patrick <chr>, zuckerberg <chr>, pelosi <chr>, garcetti <chr>,
## # newsom <chr>, steyer <chr>, schultz <chr>, kaine <chr>, johnson <chr>,
## # kucinich <chr>, lee <chr>, scott <chr>, sinema <chr>, warner <chr>,
## # na <chr>, na_2 <chr>We want to shift over the column names to the right just once. Unfortunately there is no elegant way to do this (that I am aware of). We can see that the first column is completely useless so that can be removed. Once that column is removed we can reset the names this way they will be well aligned.
We will start by creating a vector of the original column names.
col_names <- names(polls)
col_names## [1] "x" "dates" "pollster" "sample"
## [5] "sample_2" "biden" "sanders" "harris"
## [9] "warren" "o_rourke" "buttigieg" "booker"
## [13] "klobuchar" "castro" "yang" "gillibrand"
## [17] "hickenlooper" "gabbard" "delaney" "inslee"
## [21] "ryan" "bullock" "de_blasio" "bennet"
## [25] "williamson" "gravel" "swalwell" "moulton"
## [29] "messam" "h_clinton" "bloomberg" "m_obama"
## [33] "brown" "kerry" "abrams" "holder"
## [37] "mc_auliffe" "winfrey" "ojeda" "trump"
## [41] "cuomo" "avenatti" "kennedy" "patrick"
## [45] "zuckerberg" "pelosi" "garcetti" "newsom"
## [49] "steyer" "schultz" "kaine" "johnson"
## [53] "kucinich" "lee" "scott" "sinema"
## [57] "warner" "na" "na_2"Unfortunately this also presents another issue. Once a column is deselected, there will be one more column name than column. So we will need to select all but the last element of the original names. We will create a vector called new_names.
# identify the integer number of the last column
last_col <- length(col_names) - 1
# create a vector which will be used for the new names
new_names <- col_names[1:last_col]Now we can try implementing the hacky solution. Here we will deselect the first column and reset the names using setNames(). Following, we will use the mutate_at() variant to remove the percent sign from every candidate column and coerce them into integer columns. Here we will specify which variables to not mutate at within vars().
polls %>%
select(-1) %>%
setNames(new_names)%>%
select(-1) %>%
mutate_at(vars(-c("dates", "pollster", "sample", "sample_2")),
~as.integer(str_remove(., "%")))## Warning in (structure(function (..., .x = ..1, .y = ..2, . = ..1) : NAs
## introduced by coercion## # A tibble: 116 x 57
## dates pollster sample sample_2 biden sanders harris warren o_rourke
## <chr> <chr> <chr> <chr> <int> <int> <int> <int> <int>
## 1 Jun … A-Quinn… 503 RV 30 19 7 15 3
## 2 Jun … B-Morni… 17,012 LV 37 19 7 11 4
## 3 May … B+Ipsos 2,271 RV 31 14 6 9 3
## 4 May … B+Ipsos 2,525 A 30 15 6 8 4
## 5 Jun … BYouGov 500 RV 27 15 9 12 2
## 6 Jun … BYouGov 550 A 27 16 8 11 2
## 7 Jun … C+Harri… 431 RV 35 16 4 5 4
## 8 May … B-Morni… 16,587 LV 38 19 7 10 4
## 9 May … A-CNN/S… 412 RV 32 18 8 7 5
## 10 May … C+Harri… 471 RV 36 17 8 5 4
## # … with 106 more rows, and 48 more variables: buttigieg <int>,
## # booker <int>, klobuchar <int>, castro <int>, yang <int>,
## # gillibrand <int>, hickenlooper <int>, gabbard <int>, delaney <int>,
## # inslee <int>, ryan <int>, bullock <int>, de_blasio <int>,
## # bennet <int>, williamson <int>, gravel <int>, swalwell <int>,
## # moulton <int>, messam <int>, h_clinton <int>, bloomberg <int>,
## # m_obama <int>, brown <int>, kerry <int>, abrams <int>, holder <int>,
## # mc_auliffe <int>, winfrey <int>, ojeda <int>, trump <int>,
## # cuomo <int>, avenatti <int>, kennedy <int>, patrick <int>,
## # zuckerberg <int>, pelosi <int>, garcetti <int>, newsom <int>,
## # steyer <int>, schultz <int>, kaine <int>, johnson <int>,
## # kucinich <int>, lee <int>, scott <int>, sinema <int>, warner <int>,
## # na <int>Now we must tidy the data. We will use tidyr::gather() to transform the data from wide to long. In short, gather takes the column headers (the key argument) and creates a new variable from the values of the columns (the value argument). In this case, we will create a new column called candidate from the column headers and a second column called points which are a candidates polling percentage. Next we deselect any columns that we do not want to be gathered.
polls %>%
select(-1) %>%
setNames(new_names)%>%
select(-1) %>%
mutate_at(vars(-c("dates", "pollster", "sample", "sample_2")),
~as.integer(str_remove(., "%"))) %>%
gather(candidate, points, -dates, -pollster, -sample, -sample_2)## Warning in (structure(function (..., .x = ..1, .y = ..2, . = ..1) : NAs
## introduced by coercion## # A tibble: 6,148 x 6
## dates pollster sample sample_2 candidate points
## <chr> <chr> <chr> <chr> <chr> <int>
## 1 Jun 6-10, 2019 A-Quinnipiac Univers… 503 RV biden 30
## 2 Jun 3-9, 2019 B-Morning Consult 17,012 LV biden 37
## 3 May 29-Jun 5, 20… B+Ipsos 2,271 RV biden 31
## 4 May 29-Jun 5, 20… B+Ipsos 2,525 A biden 30
## 5 Jun 2-4, 2019 BYouGov 500 RV biden 27
## 6 Jun 2-4, 2019 BYouGov 550 A biden 27
## 7 Jun 1-2, 2019 C+HarrisX 431 RV biden 35
## 8 May 27-Jun 2, 20… B-Morning Consult 16,587 LV biden 38
## 9 May 28-31, 2019 A-CNN/SSRS 412 RV biden 32
## 10 May 29-30, 2019 C+Harris Interactive 471 RV biden 36
## # … with 6,138 more rowsThere are a few more house-keeping things that need to be done to improve this data set. sample_2 is rather uninformative. On the FiveThirtyEight website there is a key which describes what these values represent (A = ADULTS, RV = REGISTERED VOTERS, V = VOTERS, LV = LIKELY VOTERS). This should be specified in our data set. In addition the sample column ought to be cast into an integer column. And finally, those messy dates will need to be cleaned. My approach to this requires creating a function to handle this cleaning. First, the simple stuff.
To do the first two above steps, we will continue our function chain and save it to a new variable polls_tidy.
polls_tidy <- polls %>%
select(-1) %>%
setNames(new_names)%>%
select(-1) %>%
mutate_at(vars(-c("dates", "pollster", "sample", "sample_2")),
~as.integer(str_remove(., "%"))) %>%
gather(candidate, points, -dates, -pollster, -sample, -sample_2) %>%
mutate(sample_2 = case_when(
sample_2 == "RV" ~ "Registered Voters",
sample_2 == "LV" ~ "Likely Voters",
sample_2 == "A" ~ "Adults",
sample_2 == "V" ~ "Voters"
),
sample = as.integer(str_remove(sample, ",")))## Warning in (structure(function (..., .x = ..1, .y = ..2, . = ..1) : NAs
## introduced by coercionpolls_tidy## # A tibble: 6,148 x 6
## dates pollster sample sample_2 candidate points
## <chr> <chr> <int> <chr> <chr> <int>
## 1 Jun 6-10, 2019 A-Quinnipiac Univ… 503 Registered Vo… biden 30
## 2 Jun 3-9, 2019 B-Morning Consult 17012 Likely Voters biden 37
## 3 May 29-Jun 5,… B+Ipsos 2271 Registered Vo… biden 31
## 4 May 29-Jun 5,… B+Ipsos 2525 Adults biden 30
## 5 Jun 2-4, 2019 BYouGov 500 Registered Vo… biden 27
## 6 Jun 2-4, 2019 BYouGov 550 Adults biden 27
## 7 Jun 1-2, 2019 C+HarrisX 431 Registered Vo… biden 35
## 8 May 27-Jun 2,… B-Morning Consult 16587 Likely Voters biden 38
## 9 May 28-31, 20… A-CNN/SSRS 412 Registered Vo… biden 32
## 10 May 29-30, 20… C+Harris Interact… 471 Registered Vo… biden 36
## # … with 6,138 more rowsDate cleaning
Next we must work to clean the date field. I find that when working with a messy column, creating a single function which handles the cleaning is one of the most effective approaches. Here we will create a function which takes a value provided from the dates field and return a cleaned date. There are two unique cases I identified. There are poll dates which occurred during a single month, or a poll that spanned two months. The dates are separated by a single hyphen -. If we split the date at - we will either receive two elements with a month indicated or one month with a day and a day number. In the latter case we will have to carry over the month. Then the year can be appended to it and parsed as a date using the lubridate package. For more on lubridate visit here.
The function will only return one date at a time. The two arguments will be date and .return to indicate whether the first or second date should be provided. The internals of this function rely heavily on the stringr package (see R for Data Science Chapter 14). switch() at the end of the function determines which date should be returned (see Advanced R Chapter 5).
clean_date <- function(date, .return = "first") {
# take date and split at the comma to get the year and the month-day combo
date_split <- str_split(date, ",") %>%
# remove from list / coerce to vector
unlist() %>%
# remove extra white space
str_trim()
# extract the year
date_year <- date_split[2]
# split the month day portion and coerce to vector
dates <- unlist(str_split(date_split[1], "-"))
# paste the month day and year together then parse as date using `mdy()`
first_date <- paste(dates[1], date_year) %>%
lubridate::mdy()
second_date <- ifelse(!str_detect(dates[2], "[A-z]+"),
yes = paste(str_extract(dates[1], "[A-z]+"),
dates[2],
date_year),
no = paste(dates[2], date_year)) %>%
lubridate::mdy()
switch(.return,
first = return(first_date),
second = return(second_date)
)
}
# test on a date
clean_date(polls_tidy$dates[10], .return = "first")## [1] "2019-05-29"clean_date(polls_tidy$dates[10], .return = "second")## [1] "2019-05-30"We can use this new function to create two new columns poll_start and poll_end using mutate(). Following this we can deselect the original dates column, remove any observations missing a points value, remove duplicates using distinct(), and save this to polls_clean.
polls_clean <- polls_tidy %>%
mutate(poll_start = clean_date(dates, "first"),
poll_end = clean_date(dates, "second")) %>%
select(-dates) %>%
filter(!is.na(points)) %>%
distinct()
polls_clean## # A tibble: 1,955 x 7
## pollster sample sample_2 candidate points poll_start poll_end
## <chr> <int> <chr> <chr> <int> <date> <date>
## 1 A-Quinnipiac … 503 Registered… biden 30 2019-06-06 2019-06-10
## 2 B-Morning Con… 17012 Likely Vot… biden 37 2019-06-06 2019-06-10
## 3 B+Ipsos 2271 Registered… biden 31 2019-06-06 2019-06-10
## 4 B+Ipsos 2525 Adults biden 30 2019-06-06 2019-06-10
## 5 BYouGov 500 Registered… biden 27 2019-06-06 2019-06-10
## 6 BYouGov 550 Adults biden 27 2019-06-06 2019-06-10
## 7 C+HarrisX 431 Registered… biden 35 2019-06-06 2019-06-10
## 8 B-Morning Con… 16587 Likely Vot… biden 38 2019-06-06 2019-06-10
## 9 A-CNN/SSRS 412 Registered… biden 32 2019-06-06 2019-06-10
## 10 C+Harris Inte… 471 Registered… biden 36 2019-06-06 2019-06-10
## # … with 1,945 more rowsVisualization
The cleaned data can be aggregated and visualized.
avg_polls <- polls_clean %>%
group_by(candidate) %>%
summarise(avg_points = mean(points, na.rm = TRUE),
min_points = min(points, na.rm = TRUE),
max_points = max(points, na.rm = TRUE),
n_polls = n() - sum(is.na(points))) %>% # identify how many polls candidate is in
# remove candidates who appear in 50 or fewer polls: i.e. HRC
filter(n_polls > 50) %>%
arrange(-avg_points)
avg_polls## # A tibble: 16 x 5
## candidate avg_points min_points max_points n_polls
## <chr> <dbl> <dbl> <dbl> <int>
## 1 biden 32.1 9 66 108
## 2 sanders 19.4 4 42 110
## 3 harris 8.62 2 38 110
## 4 warren 8.05 2 43 109
## 5 o_rourke 6.50 2 21 105
## 6 buttigieg 4.88 0 21 80
## 7 booker 3.30 0 9 105
## 8 klobuchar 1.77 0 5 94
## 9 yang 1.13 0 3 62
## 10 castro 1.10 0 12 97
## 11 gillibrand 0.929 0 9 98
## 12 hickenlooper 0.728 0 2 81
## 13 gabbard 0.723 0 3 83
## 14 delaney 0.530 0 8 83
## 15 bullock 0.509 0 1 55
## 16 inslee 0.429 0 2 77avg_polls %>%
mutate(candidate = fct_reorder(candidate, avg_points)) %>%
ggplot(aes(candidate, avg_points)) +
geom_col() +
theme_minimal() +
coord_flip() +
labs(title = "Polls Standings", x = "", y = "%")
Creating historic polling data
It may become useful to have a running history of how candidates have been polling. We can use R to write a csv file of the data from FiveThirtyEight. However, what happens when the polls update? How we can we keep the previous data and the new data? We will work through an example using a combination of bind_rows() and distinct(). I want to emphasize that this is not a good practice if you need to scale to hundreds of thousand of rows. This works in this case as the data are inherently small.
To start, I have created a sample dataset which contains 80% of these polls (maybe less by the time you do this!). Note that is probably best to version control this or have multiple copies as a failsafe.
The approach we will take is to read in the historic polls data set and bind rows with the polls_clean data we have scraped. Next we remove duplicate rows using distinct().
old_polls <- read_csv("https://raw.githubusercontent.com/JosiahParry/r-4-campaigns/master/data/polls.csv")## Parsed with column specification:
## cols(
## pollster = col_character(),
## sample = col_double(),
## sample_2 = col_character(),
## candidate = col_character(),
## points = col_double(),
## poll_start = col_date(format = ""),
## poll_end = col_date(format = "")
## )old_polls## # A tibble: 1,564 x 7
## pollster sample sample_2 candidate points poll_start poll_end
## <chr> <dbl> <chr> <chr> <dbl> <date> <date>
## 1 C+HarrisX 370 Registered… klobuchar 2 2019-06-06 2019-06-10
## 2 C+HarrisX 448 Registered… gillibra… 1 2019-06-06 2019-06-10
## 3 B-Morning Con… 11627 Likely Vot… harris 13 2019-06-06 2019-06-10
## 4 B-Morning Con… 699 Registered… delaney 0 2019-06-06 2019-06-10
## 5 C+HarrisX 743 Registered… williams… 1 2019-06-06 2019-06-10
## 6 A-Quinnipiac … 559 Registered… gabbard 0 2019-06-06 2019-06-10
## 7 B-Morning Con… 14250 Likely Vot… gillibra… 2 2019-06-06 2019-06-10
## 8 A-Quinnipiac … 559 Registered… gillibra… 0 2019-06-06 2019-06-10
## 9 B-Morning Con… 14250 Likely Vot… harris 6 2019-06-06 2019-06-10
## 10 A+Monmouth Un… 330 Registered… warren 8 2019-06-06 2019-06-10
## # … with 1,554 more rowsupdated_polls <- bind_rows(old_polls, polls_clean) %>%
distinct()
updated_polls## # A tibble: 1,955 x 7
## pollster sample sample_2 candidate points poll_start poll_end
## <chr> <dbl> <chr> <chr> <dbl> <date> <date>
## 1 C+HarrisX 370 Registered… klobuchar 2 2019-06-06 2019-06-10
## 2 C+HarrisX 448 Registered… gillibra… 1 2019-06-06 2019-06-10
## 3 B-Morning Con… 11627 Likely Vot… harris 13 2019-06-06 2019-06-10
## 4 B-Morning Con… 699 Registered… delaney 0 2019-06-06 2019-06-10
## 5 C+HarrisX 743 Registered… williams… 1 2019-06-06 2019-06-10
## 6 A-Quinnipiac … 559 Registered… gabbard 0 2019-06-06 2019-06-10
## 7 B-Morning Con… 14250 Likely Vot… gillibra… 2 2019-06-06 2019-06-10
## 8 A-Quinnipiac … 559 Registered… gillibra… 0 2019-06-06 2019-06-10
## 9 B-Morning Con… 14250 Likely Vot… harris 6 2019-06-06 2019-06-10
## 10 A+Monmouth Un… 330 Registered… warren 8 2019-06-06 2019-06-10
## # … with 1,945 more rowsNow you have a cleaned data set which has been integrated with the recently scraped data. Write this to a csv using write_csv() for later use.