ExploreBulkData.Rmd

---
title: "Explore Bill Status bulk data"
output: 
  rmdformats::robobook:
    code_folding: hide
    toc_depth: 3
editor_options: 
  markdown: 
    wrap: sentence
---

Enacting laws is one of the most important powers of a government's legislative branch, yet the process of designing, proposing, and passing laws is relatively opaque.
An effective legislature is characterized by a strong understanding of problems impacting society (both at present and in the future), group decision-making processes which improve the quality of proposed legislation, and efficient use of time and resources to prioritize legislation in the most pressing areas.

The US government maintains a [repository](https://www.govinfo.gov/bulkdata/BILLSTATUS) of congressional bill data which offers us a view of how bills pass through congress and how congress members act in the process.

The goal of exploring legislative data is to provide a clearer view of who our representatives are and what they are doing.
An ideal solution combines Datamade's [beautifully simple representative finder](https://myreps.datamade.us/) and Project Vote Smart's [vast collection of data on legislator activities](https://justfacts.votesmart.org/).

Other questions worth investigating:

-   how long does it take for a bill to become law?
-   which legislative processes are the most time-intensive?
-   which decision-makers are acting in each stage?
-   what legislative paths can a bill take? map the legislative process for bills and get a sense for how legislator behaviour affects a bill's likelihood of being considered, its content, and the eventual outcome
-   Make a sankey diagram depicting how bills flow through congress

It might be worth reaching out to: <https://github.com/unitedstates/congress>

```{r setup, include=F}
library(extrafont)
library(tictoc)
library(furrr)
library(here)
library(reactable)
plan(multisession)
source("R/parsing_functions.R")

knit_print.data.frame = function(x, ...){
  knitr::knit_print(
    mokeR::reactable(x,
                     compact = T, 
                     resizable = T),
    ...
    )
}

action_codes = read_csv(here("data", "action_codes.csv"), col_types = "cc")
```

# Collecting the data

In this post, we'll look at the 117th congress' legislative actions using [govinfo's BILLSTATUS data](https://www.govinfo.gov/bulkdata/BILLSTATUS/117).
You can download the zip files for each bill type ([hr](https://www.govinfo.gov/bulkdata/BILLSTATUS/117/hr), [sjres](https://www.govinfo.gov/bulkdata/BILLSTATUS/117/sjres), [etc.](https://www.govinfo.gov/bulkdata/BILLSTATUS/117)) and unzip the XML files within.
Below is my directory structure which mimics that of govinfo, but you could also just dump them all into a single folder.

```{r dir-tree}
fs::dir_tree(here("data", "BILLSTATUS", "117"), recurse = F)
```

First we'll parse bills which were introduced in the house, or the bill types beginning with "h".
With the directory structure, I create a list of files by bill type.

House-originated bills (H.R.) are by far the most common, while simple resolutions (H. Res.) are the second most common.
Joint resolutions (H.J. Res.) and concurrent resolutions are introduced much less frequently.

```{r list-bill-files}
bill_types = list.files(here("data", "BILLSTATUS","117"))

(bill_folders = here("data", "BILLSTATUS", "117", 
                          bill_types) %>% 
    str_remove_all("\\/OneDrive"))

bill_files = map(bill_folders, list.files, full.names = T) %>% 
  set_names(bill_types)

```

How many bill XML files are in each folder?

```{r count-bills}
bill_files  %>% 
  map(length)
```

# Examine a single bill

To start off, let's have a look at a single bill.
What do we know about each piece of legislation?

```{r sample-bill}
# Sample a bill
sample_bill_file = sample(bill_files$hr, 1)

# Extract the bill status data
tic("Extract single bill")
(sample_bill_df = extract_bill_status(sample_bill_file, 
                                     # get_votes = F,
                                     log_types = "console"))
toc()

# View the bill's contents
glimpse(sample_bill_df)

```

Let's put this into a a nicer table (Note I have a preference for the date format `dd/mm/yyyy` so that is the format used here)

```{r sample-bill-rt}
sample_col_group = if("latest_action_action_date" %in% colnames(sample_bill_df)){
  list(
      colGroup(name = "Latest Action",
               columns = c("latest_action_action_date", "latest_action_text"))
    )
  
} else {
  NULL
}

select(sample_bill_df, 
       bill_id, title, where(negate(is_list)), introduced_date, create_date,
       -any_of("constitutionalAuthorityStatementText")) %>% 
  reactable(
    columns = list(
      originChamber = colDef(name = "Chamber of Origin", width = 100),
      bill_id = colDef(name = "ID", width = 80),
      congress = colDef(show = F, width = 60),
      bill_type = colDef(show = F,name = "Bill Type", width = 50),
      bill_number = colDef(show = F,name = "Bill #", width = 40),
      title = colDef(name = "Bill Title", maxWidth = 480),
      create_date = colDef(show = F,name = "Created", width = 100,
                          format = colFormat(date = T, locales = "en-GB")),
      latest_action_action_date = colDef(name = "Date", width = 100,
                                       format = colFormat(date = T, locales = "en-GB")),
      latest_action_text = colDef(name = "Action", maxWidth = 360),
      version = colDef(show=F),
      update_date = colDef(show = F,name = "Updated", width = 100,
                          format = colFormat(date = T, locales = "en-GB")),
      introduced_date = colDef(name = "Introduced", width = 104, 
                              format = colFormat(date = T, locales = "en-GB"))
    ),
    columnGroups = sample_col_group
  )
```

## Actions

What actions are taken on a bill?
After a bill is introduced, how do members of congress interact with it?

To answer questions like these, we need to `unnest()` the `actions` column.

```{r sample-actions}
(sample_actions = sample_bill_df %>% 
  select(bill_type, bill_number, bill_id, title, create_date, update_date, title,
         -where(is_list), actions) %>%
  unnest(actions) %>% 
   select(bill_id, 
          action_type, action_code, action_text, 
          any_of(c("action_date", "action_time")), 
         action_source_name, action_source_code,
         action_committee_name, action_committee_system_code,
         everything()) %>% 
   # Order bills and actions
  arrange(bill_type, bill_number, action_date))
```

Let's visualize what this timeline might look like, counting the number of each type of action taken.

```{r sample-action-timeline}
sample_actions %>% 
  # Sort cols and rows
  select(bill_id, bill_type, bill_number, action_type, action_date, action_text, action_committee_name, action_source_name) %>%
  mutate(action_type = fct_explicit_na(action_type, na_level = "(Null Action Type)")) %>% 
  arrange(bill_type, bill_number, action_date, action_type) %>% 
  # Plot
  ggplot(., aes(y = action_type, x = action_date))+
  # Line connecting points
  geom_line(aes(group = bill_id), size = .8, alpha = .9, colour = my_col_pal[2])+
  # Points using the count of each action type on a given date
  geom_jitter(aes(colour = action_type), 
              size = 3, alpha = .85, 
              show.legend = F,
              height = 0.1, width = .05)+
  labs(title = "Sample bill timeline",
       subtitle = str_c("Timeline of actions for bill ", sample_bill_df$bill_id, ", by action type"))+
  scale_x_date(name = "Action Date", breaks = scales::breaks_pretty(n = 6), labels = scales::label_date_short())+
  # scale_y_continuous(name = "# of actions", 
  #                    breaks = scales::breaks_width(1), limits = c(0, NA))+
  scale_colour_manual(values = viz_colours[1:length(unique(sample_actions$action_type))])+
  scale_size(guide = "none", range = c(2,7))+
  theme_moke(plots_pane = T)+
  theme(legend.position = "top", legend.justification = "left",
        axis.title.y.left = element_blank(), axis.title.x = element_blank(),
        axis.ticks.x = element_line())
```

However, it looks like we will need to clean the actions because there appears to be a lot of duplicate `IntroReferral` actions.
Below is a table of these actions in our sample bill: `r sample_bill_df$bill_id`

```{r sample-intro-actions}
sample_actions %>% 
  filter(action_type == "IntroReferral") %>% 
  select(bill_id, title, action_type, action_code, action_date, action_text, action_source_name, action_committee_name)
```

# Bills dataset

Now this process is a bit time intensive so I save this object as an `.Rds` file for faster future loading.
Here it is being saved/loaded

```{r save-load-rds}
# Save all_bills object
# saveRDS(all_bills, here("data", "cleaned", "BILLSTATUS_117.Rds"))

# Load all_bills object
tic("Read bills rds")
all_bills = readRDS(here("data", "cleaned", "BILLSTATUS_117.Rds"))
toc()
```

```{r}
glimpse(all_bills)
```

What bill types are in the data?

```{r}
count(all_bills, origin_chamber, bill_type, sort = T)
```

How about bill types by their origin?

```{r}
count(all_bills, origin_chamber, sort = T)
```

How many bills become law?

```{r}
all_bills %>% 
  mutate(became_law = map_chr(actions, ~if_else("BecameLaw" %in% .$action_type, "Passed", "Not Passed"))) %>% 
  count(bill_type, became_law, wt = n_distinct(bill_id)) %>% 
  pivot_wider(id_cols = bill_type, names_from = became_law, values_from = n,
              values_fill = 0)
```

As it would happen, CONRES and RES bills cannot become law, so we can focus on eligible bills for now.

```{r}
eligible_bills = all_bills %>% 
  filter(bill_type %in% c("HR","HJRES", "S", "SJRES")) %>% 
  mutate(became_law = map_chr(actions, ~if_else("BecameLaw" %in% .$action_type, "Passed", "Not Passed"))) 

eligible_bills_summary = eligible_bills %>% 
  group_by(became_law, bill_type) %>% 
  summarise(actions = map_dbl(actions, nrow), 
            .groups = "drop")
```

What policy areas does legislation focus on?

```{r fig.width = 10, fig.height = 7}
eligible_bills_policy_areas = eligible_bills %>% 
  unnest(policy_areas) %>% 
  add_count(policy_areas, wt = n_distinct(bill_id), name = "policy_n") %>% 
  count(policy_areas, became_law, policy_n, sort = T) %>% 
  group_by(policy_areas) %>% 
  mutate(passed_n = sum((became_law=="Passed")*n)) %>% 
  ungroup() %>% 
  arrange(desc(passed_n), policy_areas) %>%
  mutate(policy_areas = fct_reorder(policy_areas, passed_n))

eligible_bills_policy_areas %>% 
  ggplot(., aes(y = policy_areas, x = n, fill = became_law))+
  geom_col(position = position_identity())+
  geom_text(aes(label = scales::comma(n), x = if_else(n != 0, n+10L, NA_integer_)), 
            hjust = 0, size= 3,
            family = "Noto Sans")+
  scale_x_continuous(labels = scales::comma, 
                     expand = expansion(add = c(0, 100)), 
                     breaks = scales::breaks_pretty())+
  labs(x = "# of Bills", y = NULL,
       title = "Policy areas focus of US Congressional legislation",
       subtitle = "Bills introduced in the 117th US Congress, by legislative outcome")+
  theme_moke(plots_pane = T)+
  theme(legend.position = c(.9, .98), panel.grid.major.y = element_blank(),
        panel.grid.major.x = element_line(),
        legend.title = element_blank())

```

```{r fig.width = 10, fig.height = 7}
eligible_bills_legislative_subjects = eligible_bills %>% 
  unnest(legislative_subjects) %>% 
  add_count(legislative_subjects, wt = n_distinct(bill_id), name = "subject_n") %>% 
  count(legislative_subjects, became_law, subject_n, sort = T) %>% 
  group_by(legislative_subjects) %>% 
  mutate(passed_n = sum((became_law=="Passed")*n)) %>% 
  ungroup() %>% 
  arrange(desc(passed_n), legislative_subjects) %>%
  mutate(legislative_subjects = fct_reorder(legislative_subjects, passed_n))

eligible_bills_legislative_subjects %>% 
  ggplot(., aes(y = legislative_subjects, x = n, fill = became_law))+
  geom_col(position = position_identity())+
  geom_text(aes(label = scales::comma(n), x = if_else(n != 0, n+10L, NA_integer_)), 
            hjust = 0, size= 3,
            family = "Noto Sans")+
  scale_x_continuous(labels = scales::comma, 
                     expand = expansion(add = c(0, 100)), 
                     breaks = scales::breaks_pretty())+
  labs(x = "# of Bills", y = NULL,
       title = "Legislative subjects of US Congressional legislation",
       subtitle = "Bills introduced in the 117th US Congress, by legislative outcome")+
  theme_moke(plots_pane = T)+
  theme(legend.position = c(.9, .98), panel.grid.major.y = element_blank(),
        panel.grid.major.x = element_line(),
        legend.title = element_blank())
```

```{r}

```

hen were the bills passed?

```{r}
eligible_bills %>% 
  arrange(latest_action_action_date) %>% 
  group_by(origin_chamber, week = week(latest_action_action_date)) %>% 
  mutate(week_date = max(latest_action_action_date)) %>% 
  ungroup() %>% 
  group_by(origin_chamber, week_date) %>% 
  summarise(bills_considered = n(),
            bills_passed = sum(became_law == "Passed")) %>% 
  mutate(cum_bills_considered = cumsum(bills_considered),
         cum_bills_passed = cumsum(bills_passed)) %>% 
  ungroup() %>% 
  pivot_longer(bills_considered:cum_bills_passed,
               names_to = c(".value", "type"), 
               names_sep = "(?<=bills)_", 
               values_to = "bills") %>% 
  filter(type == "passed") %>% 
  ggplot(., aes(x = week_date, y = cum_bills, colour = origin_chamber))+
  geom_line()+
  labs(title = "Bills passed throughout the course of the 117th Congress",
       subtitle = "Cumulative legislation passed, by house of Congress",
       y = NULL,
       x = NULL)+
  scale_y_continuous(expand = expansion())+
  scale_x_datetime(breaks = scales::breaks_pretty(n=10), labels = scales::label_date_short())+
  # facet_wrap(~bill_type)+
  theme_moke(plots_pane = T)+
  theme(legend.position = "bottom", legend.title = element_blank())
```

```{r}
eligible_bills_by_week = eligible_bills %>% 
  mutate(scope = factor("all", levels = c("all", "passed only"), labels = c("", "Passed"))) %>% 
  group_by(week = week(latest_action_action_date)) %>% 
  mutate(week_date = max(latest_action_action_date)) %>% 
  ungroup()

eligible_bills_by_week_passed = filter(eligible_bills_by_week, became_law == "Passed") %>% 
  mutate(scope = factor("passed only", levels = c("all", "passed only"), labels = c("", "Passed")))

eligible_bills_by_week %>% 
  # bind_rows(eligible_bills_by_week_passed) %>% 
  #count(week_date, became_law, wt = n_distinct(bill_id))
  ggplot(., aes(x = week_date, fill = became_law))+
  # geom_histogram(binwidth = 60*60*24*7, show.legend = F)+
  # geom_bar(width = 60*60*24*5, show.legend = F)+
  stat_count(data = filter(eligible_bills_by_week, became_law == "Passed"),
             geom = "text", aes(label = ..count.., y = ..count..+10), 
             position = "stack", size = 3)+
  # geom_dots(position = "dodge", orientation = "horizontal")+
  scale_y_continuous(expand = expansion())+
  scale_x_datetime(labels = scales::label_date_short(), breaks = scales::breaks_pretty(n = 9))+
  labs(x = NULL, y = NULL,
       title = " in 117th Congress")+
  facet_wrap(~scope, ncol = 1, scales = "free_y")+
  theme_moke(plots_pane = T)
```

```{r}
range(eligible_bills$latest_action_action_date)
eligible_bills %>% 
  ggplot(., aes(x = latest_action_action_date, fill = became_law, colour = became_law, group = NA))+
  geom_dots(#binwidth = 60*60*24*c(5,10), scale = .5,
            orientation = "vertical")+
  scale_y_continuous(expand = expansion(add = c(0, 1)))+
  theme_moke(plots_pane = T)
```

\

```{r}
actions_unnested %>% count(action_type)
```

How many actions occur on the different bill types?

```{r}
library(ggdist)
bill_actions_count = all_bills %>% 
  group_by(origin_chamber, bill_type) %>% 
  summarise(actions = map_dbl(actions, nrow), 
            .groups = "drop")

bill_actions_count %>% 
  group_by(bill_type) %>% 
  summarise(across(actions, list(min = min, mean = mean, median = median, max = max)))
```

TODO: Visualize the

```{r}
bill_actions_count%>% 
  ggplot(., aes(x = bill_type, y = actions, fill = origin_chamber, colour = origin_chamber))+
  stat_dots(show.legend = F)+
  facet_wrap(~origin_chamber, nrow = 2, scales = "free_x")+
  # geom_density(bins = 50, aes(fill = origin_chamber), show.legend = F)+
  # scale_y_continuous(labels = scales::percent, expand = expansion())+
  # scale_x_continuous(limits = c(0, NA), breaks = scales::pretty_breaks())+
  # scale_fill_manual(values = viz_colours[1:2])+
  # facet_wrap(origin_chamber~bill_type, scales = "free_x", nrow = 2)+
  theme_moke()
```

When are these bills created/updated?

```{r bill-dates-summary}
all_bills %>% 
  # Calculate summary stats for each date var
  summarise(
    across(c(create_date, update_date),
           .fns = list(min = min, mean = mean, max = max, median = median), 
           na.rm=T, .names = "{.fn}_{.col}")
  ) %>% 
  # pivot for table
  pivot_longer(
    everything(),
    names_to = c("type", ".value"),
    names_sep = "_",
    values_to = "stat"
    )
```

What does this look like on a histogram?

```{r bills-dates-histogram}
all_bills %>% 
  select(bill_id, bill_type, ends_with("date")) %>% 
  pivot_longer(
    cols = ends_with("ate"), 
    names_to = "type", 
    values_to = "time", 
    names_transform = list(type = ~str_replace_all(., c("_date" = "", "(?=_).+" = "")))) %>% 
  mutate(date = as.Date(time)) %>% 
  ggplot(., aes(x = date, fill = type))+
  geom_histogram(alpha = .8,
                 binwidth = 14, position = position_identity())+
  facet_grid(bill_type~type, scales = "free_y")+
  theme_moke(plots_pane = T, title_family = NULL, subtitle_family = NULL, text_family = NULL)
```

So now that we have the dataset in our hands, what can we do with it?

## Parse Actions

There are a number of actions that legislators can take on a bill in congress.
For a start the bill needs to be introduced, then it can be sent to a committee for consideration, debated on the floor, and sent to the president to become law for simple or joint resolutions.

To get an accurate timeline of the actions taken on each bill, we will want to look at the `action_date`, `actionTime` (for Floor actions), `action_type`, `action_code`, and `action_text`.

The actions tibbles are not necessarily structurally consistent.
What fields are present and when?

```{r action-col-count}
all_bills %>% 
  select(bill_id, actions) %>% 
  # Create a list of field names for each actions dataframe
  mutate(actions_cols = map(actions, colnames)) %>% 
  unnest(actions_cols) %>%
  # Count the number of bills the field is available for
  count(actions_cols, wt = n_distinct(bill_id), sort = T)
```

Committee information is missing for a very small number of cases and timing information is missing for the vast majority of actions.
When is time information available?

```{r check-action-time-avail}
all_bills %>% 
  select(bill_id, actions) %>%
  unnest(actions) %>% 
  # Filter to rows with a time
  filter(!is.na(actionTime)) %>% 
  count(action_type, sort = T)
```

Now, let's `unnest()` the actions data and see what action-level information looks like.

```{r unnest-actions}
(actions_unnested = select(all_bills, where(negate(is_list)), starts_with("action")) %>% 
   # Unnest actions
  unnest(actions) %>% 
   # Create action timestamp
  mutate(action_ts = make_datetime(year = year(action_date), 
                                   month = month(action_date), day = day(action_date), 
                                   hour = coalesce(hour(actionTime), 0), 
                                   min = coalesce(minute(actionTime), 0), 
                                   sec = coalesce(second(actionTime), 0), 
                                   tz = "US/Eastern"),
         action_type = fct_explicit_na(
           factor(action_type, 
                              levels = c("IntroReferral", "Committee", "Floor", 
                                         "Discharge", "President", "BecameLaw"), 
                              ordered = T),
           na_level = "(Missing Action Type)"
         ),
         action_source_name = factor(action_source_name,
                                     levels = c("Library of Congress", "House floor actions", 
                                                "House committee actions", "Senate"),
                                     ordered = T)) %>% 
   
   # Join in action codes
   left_join(action_codes, by = c("action_code" = "Code")) %>% 
   # Order bills and actions
   arrange(bill_id, action_ts, action_type, action_source_name) %>% 
   # Number actions
  group_by(bill_type, bill_number) %>% 
   # Number actions
  mutate(action_number = row_number(),
         became_law = ("BecameLaw" %in% action_type), .after = actionTime) %>% 
  ungroup())
```

What action types are most common?

```{r count-action-type}
actions_unnested %>%
  count(bill_type, action_type) %>% 
  arrange(bill_type, desc(n)) %>% 
  pivot_wider(names_from = action_type, values_from = n, values_fill = 0)
```

Or in a graph:

```{r action-type-prop, fig.width = 10, fig.height = 6}
actions_unnested %>%
  count(bill_type, action_type) %>% 
  arrange(bill_type, n) %>% 
  # Calculate % of bill's actions
  group_by(bill_type) %>% 
  mutate(pct = n/sum(n),
         pct_lab = if_else(pct > .08, scales::percent(pct, accuracy = .1), NA_character_)) %>% 
  ungroup() %>%
  # Plot
  ggplot(., aes(y = bill_type, fill = fct_rev(action_type)))+
  geom_col(aes(x = n), 
           colour = my_col_pal[1], width = .6, 
           position = position_fill())+
  geom_text(aes(x = n, label = pct_lab), 
           position = position_fill(vjust = .5),
           family = fonts$subtitle, fontface = "bold",
           colour = my_col_pal[1],
           size = 5)+
  scale_x_continuous(name = NULL, 
                     labels = scales::percent, 
                     breaks = scales::breaks_pretty(),
                     expand = expansion(add = .01))+
  scale_fill_manual(name = "Action\nType", 
                    values = viz_colours[length(unique(actions_unnested$action_type)):1],
                    guide = guide_legend(title.position = "left", 
                                         title.hjust = .5,
                                         label.position = "top", 
                                         keywidth = unit(2.5, "cm"),
                                         #direction = "horizontal", 
                                         nrow = 1, reverse = T))+
  labs(title = "Most common congressional actions taken on bills",
       subtitle = "Actions as proportion of total bill activity, by bill type",
       y = NULL)+
  # facet_wrap(~bill_type, scales = "free_y")+
  theme_moke(plots_pane = T)+
  theme(legend.position = "top",
        axis.line.y = element_blank(),
        legend.title = element_text(margin = margin(l = 0, r = 20)),
        legend.box.just = "left",
        panel.grid.major.y = element_blank(), 
        panel.grid.major.x = element_line())
```

Can we get an idea of the order of actions based on their average position in the data?

```{r action-order-summary}
(action_order_summary = actions_unnested %>% 
  group_by(bill_type, action_type) %>% 
  summarise(across(action_number, list(min = min, 
                                       y25 = ~quantile(., .25), 
                                       mean = mean, median = median, 
                                       y75 = ~quantile(., .75),
                                       max = max), na.rm=T)))
```

```{r action-order-boxplot}
action_order_summary %>%
  pivot_longer(
    cols = starts_with("action_n"), 
    names_to = "measure", values_to = "value",
    names_transform = list(
      measure = ~str_remove_all(., "^action_number_")
      )
    )

action_order_summary %>% 
  ggplot(., aes(x = action_type))+
  geom_boxplot(
    aes(ymin = action_number_min, 
        lower = action_number_y25, 
        middle = action_number_median, 
        upper = action_number_y75,
        ymax = action_number_max),
    stat = "identity"
  )+
  facet_wrap(~bill_type)+
  theme_moke(plots_pane = T)+
  theme(axis.text.x = element_text(angle = 45, vjust = 1, hjust = 1))
```

```{r action-order-boxplot2}
actions_unnested %>% 
  group_by(action_type) %>% 
  mutate(mean_num = mean(action_number)) %>% 
  ungroup() %>% 
  arrange(mean_num) %>% 
  mutate(action_type = fct_inorder(factor(action_type))) %>% 
  ggplot(., aes(x = action_type, y = action_number))+
  geom_boxplot()+
  labs(title = "In what order do actions typically occur?", x = NULL)+
  facet_wrap(~bill_type)+
  theme_moke(plots_pane = T)

```

```{r sample-actions-plot}
actions_unnested %>% 
  filter(bill_id == sample(bill_id, 1)) -> sample_bill_actions

sample_bill_actions %>% 
  # Plot
  ggplot(., aes(x = action_date))+
  # Line connecting points
  geom_line(aes(y = ..count.., group = bill_id), stat = "count", size = .8, alpha = .8)+
  # Points using the count of each action type on a given date
  geom_point(aes(colour = action_type, size = ..count..), stat = "count")+
  labs(title = "Bill actions over time", 
       x = "Action Date", y = "# of actions")+
  scale_size(guide = "none", range = c(2,7))+
  theme_moke(plots_pane = T)
```

## Cleaning Action Codes

The actions data is messy.
There are a number of cases where actions are duplicated in the data which need to reconcile in order to make sense of the legislative timeline.
Are we able to identify duplicate actions using action codes?

```{r action-codes}
actions_unnested %>% 
  select(bill_id, action_date, actionTime, action_type, action_code, Action) %>% 
  count(action_code, Action) %>% 
  arrange(action_code, Action)
```

How many of the actions are actually coded?

```{r action-code-hist}
actions_unnested %>% 
  group_by(bill_id) %>%
  summarise(
    actions = n(),
    actions_w_code = sum(!is.na(Action)),
    actions_coded = actions_w_code/actions
  ) %>% 
  ggplot(., aes(x = actions_coded))+
  geom_histogram(fill = my_col_pal[5], colour = my_col_pal[1])+
  scale_x_continuous(labels = scales::percent)+
  scale_y_continuous(labels = scales::comma, expand = expansion())+
  labs(title = "Presence of action codes in House bills",
       subtitle = "Proportion of coded actions in house congressional data",
       y = "# of bills", x = "% of actions coded")+
  theme_moke(plots_pane = T)
```

Below are a list of the cases I have found:

-   Action code `E30000` and `36000` with action type `President` and `BecameLaw` respectively appear to both represent the president signing a bill into law, while action codes `E40000` and `36000` with the same respective action types represent a bill becoming a public law. In this case, action type `President` and `BecameLaw` are interchangeable
-   Action code `Intro-H` appears to be a duplicate of action code `1000`, both of which are `IntroReferral` action types

Which action types have action codes?

```{r action-codes-summary}
actions_unnested %>% 
  group_by(action_type) %>% 
  summarise(
    n = n(),
    n_coded = sum(!is.na(action_code)),
    pct_coded = n_coded/n
  )
```

### Intro/Referral

```{r intro-action-codes}
actions_unnested %>% 
  filter(action_type == "IntroReferral") %>% 
  count(action_source_name, action_code, sort = T)
```

```{r intro-actions}
actions_unnested %>% 
  filter(action_type == "IntroReferral") %>% 
  count(action_code, Action, action_text) %>% 
  arrange(action_code, desc(n))# %>% 
  # View("Intro/Referral Action Codes")
```

Below is a table of the action codes and their meaning

| Action Code | Action                                                   | Given/Inferred |
|-------------|----------------------------------------------------------|----------------|
| `1000`      | Introduced in House                                      | Given          |
| `B00100`    | Sponsor introductory remarks on measure                  | Inferred       |
| `H11100`    | Referred to Committee                                    | Inferred       |
| `H11210`    | Committee granted an extension for further consideration | Inferred       |
| `Intro-H`   | Introduced in House                                      | Inferred       |

: Intro/Referral Action Codes

Is there a difference between the `action_code` **1000** and **Intro-H**?

```{r compare-intro}
actions_to_compare = c("1000", "Intro-H")

actions_unnested %>% 
  filter(action_code %in% actions_to_compare) %>% 
  add_count(bill_id) %>% 
  filter(n>1) %>% 
  select(bill_id, bill_type, action_code, action_source_name, action_ts, 
         Action, action_type, action_text) %>% 
  arrange(bill_id, action_code)
```

Do these two actions occur for every bill?

```{r intro-code-comparison}
actions_unnested %>% 
  filter(action_code %in% actions_to_compare) %>% 
  count(bill_id, action_code) %>% 
  pivot_wider(id_cols = bill_id, names_from = action_code, values_from = n, values_fill = 0) %>% 
  count(`1000`, `Intro-H`, name = "bills")
```

No, `1000` occurs for every bill, but `Intro-H` does not.

So we can remove `Intro-H` actions without losing any information.
This reduces the size of the actions data by \~8,000 rows.

```{r intro-code-dedupe}
actions_unnested %>% 
  select(bill_id, title, introduced_date, 
         action_ts, action_type, action_code, action_text,
         starts_with("action_source"), starts_with("action_committee")) %>% 
  # Remove Intro-H action codes (keep NAs)
  filter(replace_na(action_code != "Intro-H", T)) %>% 
  arrange(bill_id, action_ts)
```

### Committee

```{r committee-action-codes}
actions_unnested %>% 
  filter(action_type == "Committee") %>% 
  count(action_source_name, action_code, sort = T)
```

```{r committee-actions}
actions_unnested %>% 
  filter(action_type == "Committee") %>% 
  count(action_code, Action, action_text) %>% 
  arrange(action_code, desc(n))# %>% 
  # View("Committee Action Codes")

```

| Action Code | Action                                    | Given/Inferred |
|-------------|-------------------------------------------|----------------|
| `14000`     | Reported to Senate                        | Given          |
| `14500`     | Senate committee discharged               | Given          |
| `5000`      | Reported to House                         | Given          |
| `5500`      | House committee discharged                | Given          |
| `H12100`    | House committee reported original measure | Inferred       |
| `H12200`    | Reported or amended by committee          | Inferred       |
| `H12210`    | Supplemental report filed                 | Inferred       |

: Committee Action Codes

Committee actions which do not have an action code include the following types of actions:

-   Referred to Subcommittee

-   Committee /Subcommittee Hearings Held

-   Ordered to be Reported

    -   in the Nature of a Substitute

    -   by Voice Vote/Unanimous Consent/the Yeas and Nays

-   Subcommittee Consideration and Mark-up Session Held

-   Subcommittee discharged

-   Forwarded by Subcommittee to Full Committee by Voice Vote

-   Executive Comment Requested/Received

### Floor

```{r floor-action-codes}
actions_unnested %>% 
  filter(action_type == "Floor") %>% 
  count(action_source_name, action_code, sort = T)
```

```{r floor-actions}
actions_unnested %>% 
  filter(action_type == "Floor") %>% 
  add_count(action_code, name = "action_code_count") %>% 
  # Select distinct action codes and their descriptions
  distinct(action_code, action_code_count,
           Action, action_text_trunc = str_trunc(action_text, width = 60),
           action_text) %>%
  # sort by frequency
  arrange(desc(action_code_count), action_code)# %>% 
  # View("Floor Action Codes")
```

+-------------+----------------------------------------------------------------------------------------------+----------------+
| Action Code | Action                                                                                       | Given/Inferred |
+=============+==============================================================================================+================+
| `17000`     | Passed/agreed to in Senate                                                                   | Given          |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `8000`      | Passed/agreed to in House                                                                    | Given          |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `9000`      | Failed of passage/not agreed to in House                                                     | Given          |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `E20000`    | Presented to President                                                                       | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H12700`    | Provision of consideration and/or debate                                                     | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H1B000`    | Pursuant to provisions of H. Res. 41/85/330/380/403/473/486/504/ 535/601/912, resolution is: | Inferred       |
|             |                                                                                              |                |
|             | -   considered passed House                                                                  |                |
|             |                                                                                              |                |
|             | -   considered vacated                                                                       |                |
|             |                                                                                              |                |
|             | -   laid on the table                                                                        |                |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H1L210`    | Rules committee resolution reported to house                                                 | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H1L220`    | Rule passed House                                                                            | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H30000`    | Considered:                                                                                  | Inferred       |
|             |                                                                                              |                |
|             | -   as privileged matter                                                                     |                |
|             |                                                                                              |                |
|             | -   as unfinished business                                                                   |                |
|             |                                                                                              |                |
|             | -   by unanimous consent                                                                     |                |
|             |                                                                                              |                |
|             | -   under suspension of the rules                                                            |                |
|             |                                                                                              |                |
|             | -   under provisions of rule H. Res. XXX                                                     |                |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H30200`    | -   Request for unanimous consent to consider bill                                           | Inferred       |
|             |                                                                                              |                |
|             | -   Chair lays bill before the House w/o objection                                           |                |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H30300`    | Move to suspend rules and pass the bill/agree to the resolution                              | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H30800`    | Consideration initiated                                                                      | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H35000`    | Previous question ordered:                                                                   | Inferred       |
|             |                                                                                              |                |
|             | -   pursuant to the rule                                                                     |                |
|             |                                                                                              |                |
|             | -   pursuant to a previous order of the House                                                |                |
|             |                                                                                              |                |
|             | -   agreed to by vote                                                                        |                |
|             |                                                                                              |                |
|             | -   without objection                                                                        |                |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H36100`    | Move to commit w/instructions to a Select committee                                          | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H36110`    | Failed move to commit w/instructions                                                         | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H36200`    | Move to recommit to committee                                                                | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H36210`    | Failed move to recommit to committee                                                         | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H36500`    | Move to table the measure                                                                    | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H36510`    | Passed move to table the measure                                                             | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H36600`    | Move to:                                                                                     | Inferred       |
|             |                                                                                              |                |
|             | -   reconsider                                                                               |                |
|             |                                                                                              |                |
|             | -   table motion to reconsider                                                               |                |
|             |                                                                                              |                |
|             | -   agree                                                                                    |                |
|             |                                                                                              |                |
|             | -   postpone consideration                                                                   |                |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H36610`    | Passed move to:                                                                              | Inferred       |
|             |                                                                                              |                |
|             | -   reconsider                                                                               |                |
|             |                                                                                              |                |
|             | -   table motion to reconsider                                                               |                |
|             |                                                                                              |                |
|             | -   agree                                                                                    |                |
|             |                                                                                              |                |
|             | -   postpone consideration                                                                   |                |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H36700`    | Move to refer to a select committee                                                          | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H37100`    | Passage of/agreeing to resolution                                                            | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H37220`    | Vote demanded at conclusion of debate                                                        | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H37300`    | Vote on motion to suspend the rules and pass the bill /                                      | Inferred       |
|             |                                                                                              |                |
|             | Passage of bill under suspension of rules                                                    |                |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H38310`    | Motion to reconsider laid on the table                                                       | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H38400`    | Move to reconsider vote                                                                      | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H38410`    | Vote on motion to reconsider                                                                 | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H38800`    | Measure title amended                                                                        | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H38900`    | Clerk authorized to make technical and formatting corrections                                | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H8A000`    | Previous question on the motion was ordered                                                  | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+
| `H8D000`    | Debate                                                                                       | Inferred       |
+-------------+----------------------------------------------------------------------------------------------+----------------+

: Floor Action Codes

```{r floor-missing-codes}
actions_unnested %>% 
  filter(action_type == "Floor", is.na(action_code)) %>% 
  count(action_text, sort = T)# %>% 
  # View("Floor Actions missing codes")
```

Floor actions which do not have an action code appear to be Senate actions, but include the following types of actions:

-   Message on Senate action sent to the house

-   Passed Senate w/o amendment by Unanimous Consent/Voice Vote

-   Passed Senate with amendment

-   Cloture motion

    -   on the measure

    -   on the motion to proceed to the measure

    -   invoked/not invoked

-   Motion to proceed to consideration of measure in Senate

-   Motion to proceed to consideration of measure *agreed to* in Senate

-   Received, considered, and agreed to in the senate

-   Considered by Senate

-   Measure laid before Senate (by motion/unanimous consent)

-   Motion to commit to Senate Committee

-   Motion to appeal the ruling of the chair

-   Motion to reconsider cloture vote

### Discharge

```{r discharge-action-codes}
actions_unnested %>% 
  filter(action_type == "Discharge") %>% 
  count(action_source_name, action_code, sort = T)
```

```{r discharge-actions}
actions_unnested %>% 
  filter(action_type == "Discharge") %>% 
  add_count(action_code, name = "action_code_count") %>% 
  # Select distinct action codes and their descriptions
  distinct(action_code, action_code_count,
           Action, action_text_trunc = str_trunc(action_text, width = 60),
           action_text) %>%
  # sort by frequency
  arrange(desc(action_code_count), action_code)# %>% 
  # View("Discharge Action Codes")

```

| Action Code | Action                         | Given/Inferred |
|-------------|--------------------------------|----------------|
| `H12300`    | Committee discharged           | Inferred       |
| `H17000`    | Motion to discharge committee  | Inferred       |
| `H12490`    | Discharged from House Calendar | Inferred       |

: Discharge Action Codes

### President

```{r president-action-codes}
actions_unnested %>% 
  filter(action_type == "President") %>% 
  count(action_source_name, action_code, sort = T)
```

```{r president-actions}
actions_unnested %>% 
  filter(action_type == "President") %>% 
  add_count(action_code, name = "action_code_count") %>% 
  # Select distinct action codes and their descriptions
  distinct(action_code, action_code_count,
           Action, action_text_trunc = str_trunc(action_text, width = 60),
           action_text) %>%
  # sort by frequency
  arrange(desc(action_code_count), action_code)# %>% 
  # View("President Action Codes")
```

| Action Code | Action                 | Given/Inferred |
|-------------|------------------------|----------------|
| `28000`     | Presented to President | Given          |
| `E30000`    | Signed by President    | Inferred       |
| `E40000`    | Became Public Law      | Inferred       |

: President Action Codes

### Became Law

```{r becamelaw-action-codes}
actions_unnested %>% 
  filter(action_type == "BecameLaw") %>% 
  count(action_source_name, action_code, sort = T)

```

```{r becamelaw-actions}
actions_unnested %>% 
  filter(action_type == "BecameLaw") %>% 
  add_count(action_code, name = "action_code_count") %>% 
  # Select distinct action codes and their descriptions
  distinct(action_code, action_code_count,
           Action, action_text_trunc = str_trunc(action_text, width = 60),
           action_text) %>%
  # sort by frequency
  arrange(desc(action_code_count), action_code)# %>% 
  # View("Became Law Action Codes")
```

| Action Code | Action            | Given/Inferred |
|-------------|-------------------|----------------|
| `36000`     | Became Public Law | Given          |

: Became Law Action Codes

### Missing Action Type

```{r missing-actions}
actions_unnested %>% 
  filter(action_type == "(Missing Action Type)") %>% 
  count(action_source_name, action_code) %>% 
  arrange(action_code, desc(n))

```

```{r missing-action-codes}
actions_unnested %>% 
  filter(action_type == "(Missing Action Type)") %>% 
  add_count(action_code, name = "action_code_count") %>% 
  # Select distinct action codes and their descriptions
  distinct(action_source_name, action_code, action_code_count,
           Action, action_text_trunc = str_trunc(action_text, width = 60),
           action_text) %>%
  # sort by frequency
  arrange(desc(action_code_count), action_code)# %>% 
  # View("Missing Type Action Codes")
```

| Action Code | Action                                                                            | Given/Inferred |
|-------------|-----------------------------------------------------------------------------------|----------------|
| `H12410`    | Placed on the Union Calendar                                                      | Inferred       |
| `H12420`    | Placed on the House Calendar                                                      | Inferred       |
| `H41931`    | Result of motion to reconsider                                                    | Inferred       |
| `19500`     | Resolving differences                                                             | Inferred       |
| `H41610`    | Result of motion that the House agree to Senate amendment                         | Inferred       |
| `H82000`    | Move to table motion to reconsider/postpone consideration/refer                   | Inferred       |
| `H12440`    | Motion to place bill on consensus calendar                                        | Inferred       |
| `H41400`    | Previous question ordered pursuant to rule/previous House order                   | Inferred       |
| `H12430`    | Placed on the Private Calendar                                                    | Inferred       |
| `H40110`    | Request unanimous consent that the House to agree to Senate amendment(s)          | Inferred       |
| `H40140`    | Move to suspend rules and agree to Senate amendment(s)                            | Inferred       |
| `H40150`    | Move that the House agree to Senate amendment(s)                                  | Inferred       |
| `80000`     | Amendment failed by the House                                                     | Inferred       |
| `H30230`    | An objection was heard to the unanimous consent request to consider the measure.  | Inferred       |
| `H40130`    | House moved to agree to the Senate amendment, pursuant to previous special order. | Inferred       |
| `H41930`    | Move to reconsider the vote                                                       | Inferred       |

: Missing Action Type Action Codes

## Policy areas

What policy areas are present:

```{r policy-areas-unnest}
(policy_areas = all_bills %>% 
  unnest(policy_areas, keep_empty = T)) %>% 
  count(policy_areas, sort = T)
```

What about the different legislative subjects?

```{r leg-subj-unnest}
(legislative_subjects = policy_areas %>% 
  unnest(legislative_subjects, keep_empty = T)) %>% 
  count(legislative_subjects, sort = T)
```

Does the presence of legislative subjects differ by bill type?

```{r leg-subj-bill-type}
legislative_subjects %>% 
  # Count number of bills of each type
  add_count(bill_type, name = "total_bills", wt = n_distinct(bill_id)) %>% 
  # Remove obs missing legislative subjects
  filter(!is.na(legislative_subjects)) %>% 
  # Count non-missing legislative subject rows
  count(bill_type, total_bills, wt = n_distinct(bill_id)) %>% 
  # Calculate prop of total with legislative subjects
  mutate(have_leg_subject = n/total_bills)
```

```{r policy-subj-counts}
legislative_subjects %>% 
  # Count bills by policy area
  add_count(policy_areas, name = "policy_area_count", wt = n_distinct(bill_id)) %>%
  # Count legislative subject/policy area combinations
  count(policy_areas, legislative_subjects, policy_area_count) %>% 
  arrange(-policy_area_count, -n)
```

Look at Government Operations and Politics legislation

```{r gov-ops-subset}
(gov_ops_bills = legislative_subjects %>% 
  # filter("Government Operations and Politics" %in% policy_areas)) # if single row per bill
   filter(policy_areas == "Government Operations and Politics"))
```

```{r timeline-dist, eval=F}
library(ggdist)
gov_ops_bills %>% 
  select(bill_id, ends_with("Date"), legislative_subjects) %>% 
  add_count(legislative_subjects, wt = n_distinct(bill_id)) %>% 
  filter(n > 100) %>% 
  mutate(active = interval(start = introduced_date, end = latest_action_action_date)) %>% 
  # pivot_longer(
  #   cols = ends_with("Date"), 
  #   names_to = "type", 
  #   values_to = "time", 
  #   names_transform = list(type = ~str_replace_all(., c("Date" = "", "(?=_).+" = "")))) %>% 
  # mutate(date = as.Date(time),
  #        week = floor_date(date, unit = "week")) %>% 
  # count(legislative_subjects, week)
  ggplot(., aes(y = bill_id))+
  # Line for each bill, point for each date
  # geom_line()+
  # geom_point(aes(), alpha = .8)+
  geom_linerange(aes(xmin = int_start(active), 
                     xmax = int_end(active), 
                     group = bill_id))+
  # stat_halfeye(
  # 
  # )+
  # Axes
  # scale_x_date(name = NULL, labels = scales::label_date_short())+
  scale_y_discrete(name = NULL, position = "right")+
  # scale_colour_manual(name = "Date", values = viz_colours[1:4])+
  # Facet by legislative subject
  facet_wrap(~legislative_subjects, scales = "free_y", ncol = 1,
             strip.position = "left")+
  # Theming
  theme_moke(plots_pane = T, text_family = NULL, title_family = NULL, subtitle_family = NULL)+
  theme(panel.grid.major.y = element_blank(),
        panel.grid.major.x = element_line(colour = my_col_pal[4]),
        axis.line = element_blank(),
        axis.text.y = element_blank(),
        legend.position = "top")
```

Let's look at the Administrative law and regulatory procedures bills

```{r admin-laws-subset}
(admin_law_bills = gov_ops_bills %>% 
  filter(legislative_subjects == "Administrative law and regulatory procedures") %>% 
  select(1:2, title, latest_action_text, latest_action_action_date, everything()))
```

```{r admin-law-actions}
admin_law_bills %>% 
  select(bill_id, title, actions) %>% 
  unnest(actions) %>% 
  # count(action_type)
  select(bill_id, title, 
         action_text, action_type, action_code, action_date, actionTime, 
         starts_with("action_source"), starts_with("action_committee")) %>% 
  # Remove intro actions with a missing action code
  filter(action_code != "Intro-H"|is.na(action_code)) %>%
  # Create action timestamp to incorporate both date and time
  mutate(action_ts = make_datetime(year = year(action_date), 
                                   month = month(action_date), 
                                   day = day(action_date), 
                                   hour = coalesce(hour(actionTime), 0), 
                                   min = coalesce(minute(actionTime), 0), 
                                   sec = coalesce(second(actionTime), 0), 
                                   tz = "US/Eastern"), 
         .before = action_date) %>% 
   arrange(bill_id, action_ts, action_type, action_source_name) %>% 
   # Number actions
  group_by(bill_id) %>% 
  mutate(action_number = row_number(),
         became_law = ("BecameLaw" %in% action_type), .after = actionTime) %>% 
  ungroup() %>% 
  select(-action_ts, -became_law) %>% 
  reactable(resizable = T)
```

View number of actions taken on a bill by whether or not they became law

```{r cum-bill-actions, fig.height = 7, fig.width = 8}
actions_unnested %>% 
  group_by(bill_id) %>% 
  mutate(became_law = ("BecameLaw" %in% action_type)) %>% 
  ggplot(., aes(x = action_ts, y = action_number))+
  geom_line(aes(group = bill_id, colour = became_law, alpha = became_law, size = became_law))+
  scale_x_datetime(name = NULL, expand = expansion())+
  scale_y_continuous(expand = expansion())+
  scale_alpha_manual(values = c(.5, .9))+
  scale_size_manual(values = c(.4, 1))+
  scale_colour_manual(values = my_col_pal[c(3,6)])+
  labs(title = "Cumulative bill actions, by legislation type and whether it became law")+
  facet_wrap(~bill_type, scales = "free_y")+
  theme_moke(plots_pane = T)
```