Illegal_Immigration.Rmd

---
title: "U.S. Illegal Immigration Arrests from 2000-2016"
author: 'Contributor: Bryan Daetz'
date: "March 6, 2017"
output: html_document
runtime: shiny
subtitle: Data Visualization Using R
resource_files:
- arrestLocations.csv
- arrests.csv
- arrests.csv
- arrestLocations.csv
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

## Table of Contents
* Project Goal
* Packages Used
* Cleaning the Data
* Overview of Yearly Arrest Totals
* Arrests at Each Border
* Sectors with the Most Arrests
* Final Thoughts

## Project Goal
Illegal immigration has always been a controversial issue.  This is especially true now with our country's current politcal situation.  Like everyone else, I have my own opinions on the topic, but I will do my best to remain unbiased in my analysis and report only facts rather than opinions.  My goal is to explore trends in US illegal immigration arrests from 2000-2016 as objectively as possible.

## Packages Used
```{r message = FALSE}
library(tidyr)    #tidyr and stringr used for cleaning data
library(stringr)
library(plyr)     #plyr and dplyr used for aggregating data
library(dplyr)
library(ggplot2)  #ggplot2 and plotly used for data visualition
library(plotly)
library(ggmap)   #ggmap used for geocode function to get latitude and longitude info for cities
```

## Cleaning the Data
The orignial illegal immigration arrests [data set](https://www.kaggle.com/cbp/illegal-immigrants) from Kaggle is extremely messy so before any real analysis can be done I'm going to have to do some cleaning.

```{r warning = FALSE}
#Importing Illegal Immigration data set from Kaggle
arrests <- read.csv("arrests.csv")

#attempting to clean the untidy dataframe
arrests <- gather(arrests, Description, Number_Arrested, -Border, -Sector, -State.Territory)
arrests <- separate(arrests, Description, c("Year", "Demographic"))

#removing the X's from the Year column
arrests$Year <- gsub(pattern = "X", replacement = "", x = arrests$Year)

#the Year column is currently a character vector and we need it to be a numerical vector to be able to create
#meaningful graphs
arrests$Year <- as.integer(arrests$Year)

#changing "All" in the Demographic column to "All Immigrants" to make it more clear
arrests$Demographic <- str_replace(arrests$Demographic, "All", "All Immigrants")
```


For anyone who is interested, I have uploaded the clean data set [here](https://github.com/bryandaetz/US_Illegal_Immigration_Arrests/blob/master/immigration_arrests.csv) on my github page.



## Overview of Yearly Arrest Totals

First, I want to take a look at the yearly illegal immigration arrest totals in the United States for both demographics: Mexicans and All Immigrants.  

```{r warning = FALSE, fig.height = 6, fig.width = 10}
#creating a new dataframe with yearly arrest totals
#it appears the original dataframe already included totals as observations where Border == United States
totals <- arrests %>%
              group_by(Year, Demographic) %>%
              filter(Border == "United States") %>%
              arrange(Demographic)

#creating an area plot comparing yearly arrest totals of all immigrants and of only mexican immigrants 
tot <- ggplot(totals, aes(x = Year, y = Number_Arrested, fill = Demographic)) +
          geom_area(alpha = 0.65, position = "dodge") +
          scale_fill_manual(values = c("skyblue1", "skyblue4")) +
          xlab("Year") +
          ylab("Total Arrests") +
          ggtitle("Total Illegal Immigration Arrests") +
          theme_minimal()
ggplotly(tot)
```

Looking at the graph, it is clear the overall trend is decreasing.  The arrest totals in 2016 are significantly lower than they were in 2000, and aside from a spike in 2004 and 2005, they have been decreasing fairly consistently every year. Additionally, the percentage of these arrests accounted for by Mexican immigrants has been decreasing as well. On the table below, you can see that the percentage has decreased by over 50% from 2000 to 2016.


```{r eval = FALSE}
#creating separate dataframes with just "Mexicans" arrests and just "All Immigrants" arrests to find the percentage
# of arrests accounted for by Mexican immigrants each year
mexican_arrests <-  filter(arrests, Border == "United States", Demographic == "Mexicans")
all_arrests <- filter(arrests, Border == "United States", Demographic == "All Immigrants")

#creating a new dataframe with these percentages (rounded to 2 decimal places) as well as the number of Mexican
# immigrants arrested and the total number of arrests for each year
percentages <- data.frame(all_arrests$Year, 
                          mexican_arrests$Number_Arrested,
                          all_arrests$Number_Arrested,
                          round(mexican_arrests$Number_Arrested / all_arrests$Number_Arrested * 100, digits = 2))
names(percentages) <- c("Year","Mexicans_Arrested", "Total_Arrests", "Percentage")

percentages
```



| Year | Mexicans Arrested   | Total Arrests   | Percentage |
|------|---------------------|-----------------|------------|
| 2000 |         1,636,883   |    1,676,438    |   97.64    |
| 2001 |         1,224,047   |    1,266,214    |   96.67    |
| 2002 |           917,993   |      955,310    |   96.09    |
| 2003 |           882,012   |      931,557    |   94.68    |
| 2004 |          1,085,006  |     1,160,395   |   93.50    |
| 2005 |          1,023,905  |     1,189,075   |   86.11    |
| 2006 |           981,066   |     1,089,092   |   90.08    |
| 2007 |           808,688   |      876,704    |   92.24    |
| 2008 |           661,766   |      723,825    |   91.43    |
| 2009 |           503,386   |      556,041    |   90.53    |
| 2010 |           404,365   |      463,382    |   87.26    |
| 2011 |           286,154   |      340,252    |   84.10    |
| 2012 |           265,755   |      364,768    |   72.86    |
| 2013 |           267,734   |      420,789    |   63.63    |
| 2014 |           229,178   |      486,651    |   47.09    |
| 2015 |           188,122   |      337,117    |   55.80    |
| 2016 |           192,969   |      415,816    |   46.41    |

## Arrests at Each Border

Next, I would like to take a look at the yearly arrest totals at each of the three borders in the data set: Coast, North and Southwest.  At first glance, it is already clear that the number of arrests is significantly lower for the Coast and North borders.  The vast majority of the arrests have occured along the Southwest border.

```{r warning = FALSE, fig.height = 6, fig.width = 10}

#creating a new dataframe with yearly arrest totals by border
#again the original dataframe already included this totals as observations where Sector == All
by_border <- arrests %>%
                  group_by(Year, Demographic) %>%
                  filter(Sector == "All") %>%
                  arrange(Demographic)

#Since the arrest totals are so much higher for the southwest than for the other two borders it may make more 
#sense to create individual graphs for each border instead of facet wrapping.
#To avoid rewriting the code for the graph for each border I chose to write a function that will create a graph
#for a given border.
border <- function(x, label) {
t <- ggplot(filter(by_border, Border == x), 
                aes(x = Year, y = Number_Arrested, fill = Demographic)) +
              geom_area(alpha = 0.65, position = "dodge") +
              scale_fill_manual(values = c("skyblue1", "skyblue4")) +
              xlab("Year") +
              ylab("Total Arrests") +
              ggtitle(label) +
              theme_minimal()
ggplotly(t)
}

border("Coast", "Illegal Immigration Arrests Along the Coast")
border("North", "Illegal Immigration Arrests at Northern Border")
border("Southwest", "Illegal Immigration Arrests at Southwest Border")

```

Unsurpringly, the percentage of arrests accounted for by Mexican immigrants is clearly lower for the Coast and North borders than it is for the Southwest border.  It is surpising, however, to see that there were still a decent amount of Mexican immigrants that were arrested along the United States/Canadian border (North).  The data set didn't specify the nature of any of the arrests so I can only speculate about how this could have occured. It is possible that these people had illegally immigrated from Mexico earlier and were living in the northern United States when they were later arrested by the border patrol there, but it is impossible to confirm this theory with the data I have currently.

## Sectors with the Most Arrests

The last element of the data set that I'd like to explore is the sectors with the most arrests each year.  The original data set was somewhat unclear on what they meant by the word sector.  The sectors listed are all cities in the United States, but it isn't clear whether they are intended to be just the cities, the surrounding counties, or some other area around the cities.

In any case, the first step is to create bar plots showing the sectors with the most arrests each year.  In earlier iterations of the project, I considered using a facet wrap to show the plots for each year side by side, but since the data spans 17 years, this approach proved to be too cluttered and difficult to read.  I also considered having the bar plots show the arrest totals for all 21 sectors, but again this made them extremely cluttered.  My solution was to write a function that finds the sectors with the 8 highest arrest totals for a given year and displays them on a bar plot.

``` {r}
#creating a function to show a bar graph of the 8 sectors with the highest arrest totals for a given year
yearPlot <- function(yr, title = paste("Sectors with the Highest Arrest Totals in", as.character(yr))) {
    temp <- filter(arrests, Sector != "", Sector != "All", Year == yr) #filtering out rows that don't apply to
                                                                    ## to a specific Sector
                                                                      #filtering by the provided Year value
    
    top8 <- temp %>%
              filter(Demographic == "All Immigrants") %>% 
      #finding the Sectors with the 8 highest arrest totals for that year
              arrange(desc(Number_Arrested))            
    top8 <- top8[1:8,]
    
    temp <- filter(temp, Sector %in% top8$Sector)  #filtering by the Sectors w/ the 8 highest arrest totals
    temp$Demographic <- ordered(temp$Demographic, levels = c("Mexicans", "All Immigrants"))
                                                              #setting the level order for the Demographic
                                                              # attribute so the smaller values on the bar plot
                                                              # aren't hidden
    
    
    #creating a bar plot comparing the arrest totals for all illegal immigrants and for only mexican immigrants
    plot <- ggplot(temp, aes(x = Sector, y = Number_Arrested, fill = Demographic)) +
                geom_bar(stat = "identity", position = "identity", alpha = .65) +
                scale_fill_manual(values = c("skyblue4", "skyblue1")) +
                xlab("Sector") +
                ylab("Total Arrests") +
                ggtitle(title) +
                theme_minimal() +
                theme(axis.text.x = element_text(size = 8))
    ggplotly(plot)    #making the plot interactive
}    
```

Since the bar plots will only show the arrest totals for 8 of the 21 sectors each year, I would also like to plot all the sectors on a US map.  The map will be interactive and will display the arrest totals for Mexicans and All Immigrants when you hover over a sector.  Unfortunately, this requires latitude and longitude information for the sectors, which the original data set did not include.  Luckily, the ggmap package includes a geocode function that can get this information for me.  However, this function takes a while to run so I decided to create a separate csv file of the data set with the latitude and longitude information to avoid running it more than once.  That file is also available [here](https://github.com/bryandaetz/US_Illegal_Immigration_Arrests/blob/master/arrestLocations.csv) on my github page for those who are interested.

```{r eval = FALSE}
#creating a vector with all the different sectors
Sector <- levels(arrests$Sector)
#taking out the sectors "All" and "" which are used for totals and aren't actual individual sectors
Sector <- Sector[3:length(Sector)]

#using the geocode command to get the latitude and longitude for each sector
locations <- data.frame(Sector, geocode(Sector, output = "latlon"))

#subset of arrests dataframe with only the rows for individual sectors, no totals
arrests2 <- filter(arrests, Sector != "", Sector != "All")

#merging the arrests2 and locations dataframes to get the latitude and longitude information for each Sector
arrests_loc <- join(x = arrests2, y = locations,
                    by = "Sector", type = "left")
write.csv(arrests_loc, file = "arrestLocations.csv")
```

```{r}
#importing csv file with locations to avoid having to run the geocode command more than once
arrests_loc <- read.csv("arrestLocations.csv")
```

I found the code that I am using to import a blank U.S. map on the plotly documentation website.  After importing the map, the next step is to write a function, similar to what I did for the bar plots, that will plot all the sectors (with information on the arrest totals) on the map for a given year.

```{r}
# getting a map of the United States to show the areas with the most arrests
# this was the example US map from the plotly documentation page
g <- list(
  scope = "usa",
  projection = list(type = "albers usa"),
  showland = TRUE,
  landcolor = toRGB("gray95"),
  subunitcolor = toRGB("gray85"),
  countrycolor = toRGB("gray85"),
  countrywidth = 0.5,
  subunitwidth = 0.5
)

#creating a function to create a map of the areas with the most arrests for a given year
mapPlot <- function(yr, title = paste("Illegal Immigration Arrests in", as.character(yr))) {
        tmp <- filter(arrests_loc, Year == yr)
        p <- plot_geo(tmp, lat = ~lat, lon = ~lon, 
                      color = ~Demographic, 
                      colors = c("skyblue1", "skyblue4"), 
                      size = ~Number_Arrested,
                      sizes = c(10, 300),
                      alpha = 0.65,
                      text = ~paste('Demographic: ', Demographic, 
                        '</br> Sector: ', Sector,
                        '</br> Arrests: ', Number_Arrested)) %>%
          add_markers() %>%
          layout(title = title, geo = g)
        print(p)
}
```

```{r eval = FALSE}
  selectInput("yr", label = "Year",
              choices = c(2000:2016), selected = 2016)
  renderPlotly({
              yearPlot(input$yr)
  })
  renderPlotly({
              mapPlot(input$yr)
  })
```

```{r echo = FALSE, fig.height = 10, fig.width = 15}
  selectInput("yr", label = "Year",
              choices = c(2000:2016), selected = 2016)
  renderPlotly({
              yearPlot(input$yr)
  })
  renderPlotly({
              mapPlot(input$yr)
  })
```


By selecting a year on the input year, you can now look at both the bar plot and U.S. map showing the sectors with the highest arrest totals for that year.  The circles on the U.S. map scale to the number of arrests in that sector (i.e. bigger circle = more arrests).  Interestingly, Tucson, Arizona had the most arrests from 2000-2012 and Rio Grande Valley, Texas had the most arrests from 2013-2016.

## Final Thoughts

While the data set did have its limitations, it is undeniable that the number of U.S. illegal immigration arrests has been decreasing over the past 17 years.  It is also clear that most of the arrests did occur along the Southwest border each year, but despite that, the percentage of arrests accounted for by Mexicans has been decreasing significantly.  Last year, for instance, only 46.41% of the illegal immigrants arrested were Mexican, down from 97.64% in 2000.  

Additionally, in recent years, the sector with the most arrests has been Rio Grande Valley, Texas.  In 2016 for example, it was responsible for 186,830 arrests, over 100,000 more than Tucson, Arizona, the sector with the next highest arrest total.  Interestingly though, only 46,334 of the 186,830 arrests in Rio Grande Valley were Mexican immigrants.

Unfortunately, since the data set didn't provide information about the national origin of the other illegal immigrants in the All Immigrants demographic, there isn't much else I can conclude from the data.  It also would have been nice to be able to determine the reason that there were a decent amount of Mexican immigrants arrested along the northern border.  If I can find more data on those topics in the future I'll have to add that to my current findings.