Scrape candidates from District 1

Next we scrape data on candidates in the first district of North Carolina.

The code for the following should go in the scrape-nc.R file.

⊕Hint: The URL for this page is already saved in nc_races$race_link[1].

• race_page: Read the page for the first district and save the result as race_page.

• candidate_info: Scrape the candidate info and save as a character vector called candidate_info. It will look something like the following.

  "G K Butterfield (D)\n • Incumbent" "Roger Allison (R)" 

⊕Hint: You’ll need to deal with the dollar signs ($) and commas (,) in the values.

• raised, spent, cash_on_hand: Scrape the raised, spent, and cash on hand funds amounts and save them as numeric values.

• last_report: Scrape the date of last_report. This variable should have a Date class, formatted as "yyyy-mm-dd", e.g. "2018-06-30". You can use the mdy() function from the lubridate package. This function takes in a character vector that is a date, and turns it into a vector with proper Date class.

mdy("06/30/2018")

## [1] "2018-06-30"

• race: Scrape the name of the race from the header of the page. Note that this text will say "North Carolina District 01 2018 Race". Remove " 2018 Race" from the character string so that it reads ""North Carolina District 01".

• candidates: Combine this information in a data frame called candidates with the variables candidate_info, raised, spent, cash_on_hand, last_report, and race.

Lastly, we need to clean up the information in the candidate_info data frame

⊕Hint: You’ll need to escape the parantheses with backslashes.

• Create a new variable called party which has the following properties:
  • If candidate_info contains the character string (R), label Republican
  • If candidate_info contains the character string (D), label Democrat
  • Otherwise label Third party

• Create a new variable called status which has the value Incumbent if candidate_info contains the character string Incumbent, otherwise label Challenger

• Create a new variable called candidate_name which has only the candidate name from the candidate_info variable. This means deleting everything starting with the first open paranthesis (() and trimming the character string. This is going to require getting fancy with regular expressions. To match line breaks, indicated with \n, use "\\n". And to match everything starting with the first (, use "\\((.*)". For example,

⊕At this point take a break to this short but detailed article on regular expressions. Learning a bit more on regular experessions will give you a lot of facilty for working with text data.

candidate_info <- "  G K Butterfield (D)\n • Incumbent\n    "

# remove all line breaks
candidate_info %>%
  str_remove_all("\\n")

## [1] "  G K Butterfield (D) • Incumbent    "

# remove all line breaks + remove everything starting with (
candidate_info %>%
  str_remove_all("\\n") %>%
  str_remove("\\((.*)")

## [1] "  G K Butterfield "

# remove all line breaks + remove everything starting with ( + trim white spaces
candidate_info %>%
  str_remove_all("\\n") %>%
  str_remove("\\((.*)") %>%
  str_trim()

## [1] "G K Butterfield"

⊕Hint: select(race, everything()) would reorder the columns to move race up to be the first variable, and all the other variables would remain in the same order as before.

• Finally reorganize the data frame so that it only contains the variables candidate_name, race, raised, spent, cash_on_hand, last_report, party, and status, in this order. Note that candidate_info is dropped. You will need to use the select() function to reorder the columns. As a requirement for this exercise, you should also use the everything() function, which will allow you to not have to type out all the variable names. See the help for this function to find out more about its usage.

• Save the resulting candidates data frame as nc_candidates_dist01.csv in your data folder.

4. Load nc_candidates_dist01.csv in your R Markdown document, save it as nc_candidates_dist01, and print it. Confirm that the data you have matches the one on the webpage.

At this point make sure you’ve committed and pushed your work so far.

Scrape candidates from District 2

In your scrape-nc.R file, copy and paste the code you developed in the previous section for scraping District 1 data to the appropriate section for District 2. Change the input URL to the link for the second district, nc_races$race_link[2].

The goal is to scrape the data for District 2 just like you scraped the data for District 1 in the previous section. You might find that you need to make some adjustments to your code. This is ok, go ahead and make them. But make sure to implement the same changes in your code for scraping District 1 candidates as well. You should get to a point where the same exact code works for both districts.

Write out the resulting data frame as nc_candidates_dist02.csv in your data/ folder.

5. Load nc_candidates_dist02.csv in your R Markdown document, save it as nc_candidates_dist02, and print it. Confirm that the data you have matches the one on the webpage.

Once again, make sure you’ve committed and pushed your work so far.

Writing a function

Now that you have confirmed your code works on two different pages, we’ll work on turning your code into a function.

In the R script titled scrape-race.R create a function called scrape_race that has one input, url, and that outputs a tibble with number of rows corresponding to the number of candidates running for each race and six variables.

6. Source this function in your R Markdown file with the source() function as shown below. This will make the function available for use in the rest of your R Markdown document.

source("scrape-race.R)

7. Apply this function to the URL for the North Carolina’s first district race, as shown below, and display the result. Confirm that your result matches your answer in Exercise 4.

scrape_race(nc_races$race_link[1])

8. Now apply this function to the URL for the North Carolina’s first district race, as shown below, and display the result. Confirm that your result matche your answer in Exercise 5.

scrape_race(nc_races$race_link[2])

Mapping the function

9. Now that you have confirmed that your function works for the first two districts when applied one-by-one, map it over the links for these two NC races and confirm that the result you get is a combination of the results from Exercises 7 and 8.

map_df(nc_races$race_link[1:2], scrape_race)

10. And now that you have confirmed that mapping works for the first two districts, map the function over the links for all NC races and save the result as nc_candidates as shown below. Then, glimpse() at the resulting data frame.

nc_candidates <- map_df(nc_races$race_link, scrape_race)

If you’re happy with your answer for Exercise 10, I recommend caching the results of the code chunk so that your code for scraping data from all NC races does not have to rerun every time you knit your document. To do this you need to is to add the following option to your code chunk: cache = TRUE, e.g. if the label you used for your code chunk is map-all, your code chunk definition will look like this:

{r map-all, cache = TRUE}

Then, knit your document one more time. Your code will run again, but this time a new folder will be created in your project where cached results will be saved. As with everything that changes in your repo, you will need to commit and push this folder as well. Going forward your code will only rerun if the code in the cached code chunk changes.

⊕Remember that you can use everything() for “all the other columns”.

11. Join the nc_candidates and nc_races data frames by the race variable using an inner_join(). Then, reorganize the columns of of the resulting data frame to be in the following order: candidate_name, race_type, race, all the other columns. Save the result as nc_candidates, and glimpse() at the resulting data frame.

12. Answer the following questions based on the nc_candidates data frame you created in Exercise 11:
    a. What is the median amount of funds raised by all candidates in NC?
    b. What are the median amounts of funds raised by Republican, Democratic, and third party candidates in NC?
    c. Calculate the percentage of cash_on_hand compared to raised for all candidates in NC. Save the result as a new variable in the data frame called perc_cash_on_hand.
    d. Among all NC candidates, which candidate has the highest percentage of cash on hand?
    e. On average, have incumbents or challengers raised more money? Does this vary by party affiliation?