# Lab 02 - Data wrangling and visualization

### Due: Friday, Jan 31 at 11:59pm

If you are curious about how raw data from the ACS were cleaned and prepared, see the code that the FiveThirtyEight authors used (be warned: it’s a bit outside of the scope of this course!).

Often times, the first step in using statistics to turn information into knowledge is to summarize and describe the raw information - the data. In this lab we explore data on college majors and earnings, specifically the data behind the FiveThirtyEight story The Economic Guide To Picking A College Major. These data originally come from the American Community Survey (ACS) 2010-2012 Public Use Microdata Series.

There are many considerations that go into picking a major; two of them are earnings potential and employment prospects. They are important, but they don’t tell the whole story. Keep this in mind as you analyze the data.

# Getting Started

## Clone Assignment Repo

• First, go to the link for the assignment in order to create your personal private repository corresponding to the assignment. This link may be found at https://classroom.github.com/a/j30XyFAc. Accepting the assignment creates a repository similar to lab02-[github_name].

• Next, navigate to the course GitHub website, and click on the repository corresponding to your account. For example, mine would be lab02-shawnsanto. Or, simply go to the repository you just created (there is a link provided when you created the repository).

• Click on the green Clone or download button in that repository, and copy the git URL (it should end in .git)

• Go to RStudio Cloud and into the STA 199 course workspace. Create a New Project from Git Repo and copy the git URL from your personal workspace. Make sure “Add packages from the base project” is checked.

• Click OK, and you should see the contents from your GitHub repo in the Files pane in RStudio.

## Configure git

Remember that we first have to configure RStudio Cloud to talk to GitHub. An easy way to do this is to use the use_git_config() function from the usethis package.

Type the following lines of code in the Console in RStudio filling in your name and email address.

library(usethis)
use_git_config(user.name="Your name", user.email="your-git-hub-email@email.edu")

## Project name:

Currently your project is called Untitled Project. Update the name of your project to be “Lab 02 - Data wrangling and visualization”.

## Packages

In this lab we will work with the tidyverse and fivethirtyeight packages. Load the packages into the Console (they have already been installed for you).

library(tidyverse)
library(fivethirtyeight)

# Warm up

Before we introduce the data, let’s do a few things, which hopefully should be review.

## YAML:

Open the R Markdown (Rmd) file in your project, change the author name to your name, and knit the document.

## Stage, commit, and push changes:

• Go to the Git pane in RStudio.
• View the Diff and confirm that you are happy with the changes.
• Stage the files you want to commit.
• Add commit message “Updated name” in the Commit message box and hit Commit.
• Click on Push. This will prompt a dialog box where you first need to enter your GitHub user name and password.

For the remainder of this lab, we won’t tell you when to commit – it is up to you to commit at appropriate intervals with meaningful commit messages. Be sure to commit at least three times during this lab.

The data frame we will be working with today is called college_recent_grads and it’s in the fivethirtyeight package.

To find out more about the dataset, type the following in your Console: ?college_recent_grads. A question mark before the name of an object will bring up its help file. This command must be run in the Console.

college_recent_grads is a tidy data frame, with each row representing an observation and each column representing a variable.

Take a quick peek at your data frame and view its dimensions with the glimpse() function.

glimpse(college_recent_grads)

The description of the variables, i.e. the codebook, is given below.

rank Rank by median earnings
major_code Major code, FO1DP in ACS PUMS
major Major description
major_category Category of major from Carnevale et al
total Total number of people with major
sample_size Sample size (unweighted) of full-time, year-round ONLY (used for earnings)
men Male graduates
women Female graduates
sharewomen Women as share of total
employed Number employed (ESR == 1 or 2)
employed_full_time Employed 35 hours or more
employed_part_time Employed less than 35 hours
employed_full_time_yearround Employed at least 50 weeks (WKW == 1) and at least 35 hours (WKHP >= 35)
unemployed Number unemployed (ESR == 3)
unemployment_rate Unemployed / (Unemployed + Employed)
median Median earnings of full-time, year-round workers
p25th 25th percentile of earnings
p75th 75th percentile of earnings
college_jobs Number with job requiring a college degree
non_college_jobs Number with job not requiring a college degree
low_wage_jobs Number in low-wage service jobs

The college_recent_grads data frame is a trove of information. Let’s think about some questions we might want to answer with these data.

• Which major has the lowest unemployment rate?
• Which major has the highest percentage of women?
• How do the distributions of median income compare across major categories?
• Do women tend to choose majors with lower or higher earnings?

In the next section we will answer these questions.

# Data wrangling and visualization

## Which major has the lowest unemployment rate?

In order to answer this question all we need to do is sort the data. We can use the arrange() function to do this, and sort it by the unemployment_rate variable. By default, arrange() sorts in ascending order, which is what we want here – we’re interested in the major with the lowest unemployment rate.

college_recent_grads %>%
arrange(unemployment_rate)

This gives us what we wanted, but not in an ideal form. First, the name of the major barely fits on the page. Second, some of the variables are not that useful (e.g. major_code, major_category) and some we might want front and center are not easily viewed (e.g. unemployment_rate).

We can use the select() function to choose which variables to display and in which order:

Note how easily we expanded our code with adding another step to our pipeline, with the pipe operator: %>%.

college_recent_grads %>%
arrange(unemployment_rate) %>%
select(rank, major, unemployment_rate)

This is looking better, but do we really need all those decimal places in the unemployment variable? Not really!

There are two ways we can address this problem. One would be to round the unemployment_rate variable in the dataset or we can change the number of digits displayed, without touching the input data.

Below are instructions for how you would do both of these:

• Round the variable itself: We create a new variable with the mutate() function. In this case, we’re overwriting the existing unemployment_rate variable, by rounding it to 4 decimal places.
college_recent_grads %>%
arrange(unemployment_rate) %>%
select(rank, major, unemployment_rate) %>%
mutate(unemployment_rate = round(unemployment_rate, digits = 4))
• Change the displayed number of digits without touching data: We can add an option to our R Markdown document to change the displayed number of digits in the output. To do so, add a new chunk, and set:
options(digits = 2)

Note that the digits in options is scientific digits, and in round they are decimal places. If you’re thinking “Wouldn’t it be nice if they were consistent?”…so do we.

You don’t need to do both of these, that would be redundant. The next exercise asks you to choose one.

1. Which of these options, changing the input data or altering the number of digits displayed without touching the input data, is the better option? Explain your reasoning. Implement the option you chose.

## Which major has the highest percentage of women?

To answer such a question we need to arrange the data in descending order. For example, if earlier we were interested in the major with the highest unemployment rate, we would use the following:

The desc() function specifies that we want unemployment_rate in descending order.

college_recent_grads %>%
arrange(desc(unemployment_rate)) %>%
select(rank, major, unemployment_rate)
1. Using what you’ve learned so far, arrange the data in descending order with respect to proportion of women in a major, and display only the major, the total number of people with that major, and proportion of women. Show only the top 3 majors by adding slice(1:3) at the end of the pipeline.

## How do the distributions of median income compare across major categories?

A percentile is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall. For example, the 20th percentile is the value below which 20% of the observations may be found. (Source: Wikipedia

There are three types of incomes reported in this data frame: p25th, median, and p75th. These correspond to the 25th, 50th, and 75th percentiles of the income distribution of sampled individuals for a given major.

1. Why do we often choose the median, rather than the mean, to describe the typical income of a group of people?

In order to answer our question about the distributions of median income across major categories, we need to group the data by major_category. Then, we need a way to summarize the distributions of median income within these groups. This decision will depend on the shapes of these distributions. Let’s create a visualization to see the shapes of these distributions.

Function ggplot() will allow us to create our plots. The first argument is the data frame, and the next argument gives the mapping of the variables of the data to the aesthetic elements of the plot.

Let’s start simple and take a look at the distribution of all median incomes, without considering major categories.

ggplot(data = college_recent_grads, mapping = aes(x = median)) +
geom_histogram()

Along with the plot, we get a message:

stat_bin() using bins = 30. Pick better value with binwidth.

This is telling us that we might want to reconsider the binwidth we chose for our histogram – or more accurately, the binwidth we didn’t specify. It’s good practice to always think in the context of the data and try out a few binwidths before settling on your final choice. You might ask yourself: “What would be a meaningful difference in median incomes?” $1 is obviously too little,$10000 might be too high.

1. Plot two histograms of median incomes, one with a bin width of $1000 and one with a bin width of$5000. Select one and explain your reasoning for your choice. Argument binwidth is for the geom_histogram() function. So to specify a binwidth of $1000, you would use geom_histogram(binwidth = 1000). We can also calculate summary statistics for this distribution using the summarise() function: college_recent_grads %>% summarise(min = min(median), max = max(median), mean = mean(median), med = median(median), sd = sd(median), q1 = quantile(median, probs = 0.25), q3 = quantile(median, probs = 0.75)) 1. Based on the shape of the histogram you created in the previous exercise, determine which of these summary statistics is useful for describing the distribution. Write up your description (remember shape, center, spread, any unusual observations) and include the summary statistic output as well. Next, we facet the plot by major category. ggplot(data = college_recent_grads, mapping = aes(x = median)) + geom_histogram() + facet_wrap( ~ major_category, ncol = 4) 1. Plot the distribution of median income using a histogram, faceted by major_category. Use the binwidth you chose in the earlier exercise. Now that we’ve seen the shapes of the distributions of median incomes for each major category, we should have a better idea for which summary statistic to use to quantify the typical median income. 1. Which major category has the highest typical (you’ll need to decide what this means) median income? Use the partial code below, filling it in with the appropriate statistic and function. Also note that we are looking for the highest statistic, so make sure to arrange your resulting data frame in the correct order. college_recent_grads %>% group_by(major_category) %>% summarise(___ = ___(median)) %>% arrange(___) 1. Which major category is the least popular in this sample? To answer this question we use a new function called count(), which first groups the data and then counts the number of observations in each category (see below). Arrange the results so that the major with the lowest observations is in row 1 of the resulting data frame. college_recent_grads %>% count(major_category) ## All STEM fields aren’t the same One section of the FiveThirtyEight story is titled “All STEM fields aren’t the same”. Let’s see if this is true. First, let’s create a new vector called stem_categories that lists the major categories that are considered STEM fields. Ask your TA what function c() does if you are not sure. Also, check the help with ?c. stem_categories <- c("Biology & Life Science", "Computers & Mathematics", "Engineering", "Physical Sciences") We can use this to create a new variable in our data frame indicating whether a major is STEM or not. college_recent_grads <- college_recent_grads %>% mutate(major_type = ifelse(major_category %in% stem_categories, "stem", "not stem")) Let’s unpack this: with mutate() we create a new variable called major_type, which is defined as "stem" if the major_category is in the vector called stem_categories we created earlier, and as "not stem" otherwise. %in% is a logical operator. Other logical operators that are commonly used are Operator Operation x < y less than x > y greater than x <= y less than or equal to x >= y greater than or equal to x != y not equal to x == y equal to x %in% y group membership x | y or x & y and !x not Here is a small example showing how operator %in% works. x <- c("red", "white", "blue") y <- c("green", "red", "red", "orange", "white", "black", "purple", "blue") y %in% x #> [1] FALSE TRUE TRUE FALSE TRUE FALSE FALSE TRUE We can use logical operators to filter() our data. Here will filter based on the conditions that the major type is STEM and the median salary is less than$36,000.

college_recent_grads %>%
filter(
major_type == "stem",
median < 36000
)
1. Which STEM majors have median salaries equal to or less than the median for all majors’ median earnings? Your output should only be a table that shows the major name, median, 25th percentile, and 75th percentile earning for that major, and should be sorted such that the major with the highest median earning is on top. Do not include narrative; the table itself will suffice.

## What types of majors do women tend to choose?

1. Create a scatterplot of median income vs. proportion of women in that major. Set the point color as to whether the major is in a STEM field or not. Describe the association between these three variables.

# Submission

Knit to PDF to create a PDF document. Stage and commit all remaining changes, and push your work to GitHub. Make sure all files are updated on your GitHub repo.