W3_network.Rmd

---
title: "Network Analaysis"
author: "Meg Wu"
date: "2019/10/28"
output:
  html_document:
    theme: spacelab
    highlight: zenburn
    toc: yes
    toc_float:
      collapsed: no
    df_print: paged
editor_options:
  chunk_output_type: inline
---

<style>
    .tocify-header{color:darkred; font-size:16px;}
    .tocify-subheader{color:tomato; font-size:14px;}
    h1{color:darkred; font-size:40px;}
    h2{color:tomato; font-size:24px;}
    strong {background-color:yellow;}
</style>


# The method: Network Analysis
- Like，就是按讚
- Follow，追蹤
- Retweet，轉推（RT@）。此教學案例所定義的「互動」以轉推為主
- Reply，回推
- Mention，提到某人
- 例子: https://twitter.com/lazysundaytalk/status/1174210332014915584



# 00 Preparing data

## 0.3 Loading packages

```{r message=FALSE, warning=FALSE}
library(tidyverse) # coding with tidy style 
library(lubridate) # processing time
library(igraph) # plotting network
options(stringsAsFactors = F) # avoid converting character to factor silently
```



## 0.2 Loading data

```{r}
users_1 <- read_csv("data/china_082019_1_users_csv_hashed.csv")
tweets_1 <- read_csv("data/china_082019_1_tweets_csv_hashed.csv")

# users_2 <- read_csv("data/china_082019_2_users_csv_hashed.csv")
# tweets_2 <- read_csv("data/china_082019_2_tweets_csv_hashed.csv")
```

## 0.3 Previewing data
```{r}
tweets_1 %>% glimpse()
```


# 01 Preparing network data

## 1.1 Extracting RT(Retweet) relations
- RT relationship：user_screen_name -> RT @someone said something
- 意即「誰（user_screen_name, 在此為被刪除的帳號們）」RT了@someone（某人，不見得只有被刪除的帳號）
- 在網絡分析上，通常以from-to來表達一個有向關係（A RT B為有向關係）
- 將推文時間當作第三個變項`tweet_time`

```{r}

# Filtering RT relation (366,253 out of 1,906.831)
rt_1 <- tweets_1 %>%
    select(user_screen_name, tweet_text, tweet_time) %>%
    filter(str_detect(tweet_text, "RT @"))

# In each tweet, detecting retweet targets when number-of-RT  ==  1.
rt_edge_1 <- rt_1 %>%
    mutate(rt_count = str_count(tweet_text, "RT @\\w+")) %>%
    filter(rt_count == 1) %>%
    mutate(to = str_extract(tweet_text, "RT @\\w+")) %>%
    mutate(to = str_sub(to, 5, nchar(to))) %>%
    select(from = user_screen_name, to, tweet_time)

# In each tweet, detecting retweet targets when number-of-RT  >  1.
rt_edge <- rt_1 %>%
    mutate(rt_count = str_count(tweet_text, "RT @\\w+")) %>%
    filter(rt_count > 1) %>%
    mutate(to = str_extract_all(tweet_text, "RT @\\w+")) %>%
    unnest(to) %>%
    mutate(to = str_sub(to, 5, nchar(to))) %>%
    select(from = user_screen_name, to, tweet_time) %>%
    bind_rows(rt_edge_1)
```

## 1.2 Extracting innert group RT relations

```{r}
rt_edge_inner <- rt_edge %>%
    filter(to %in% users_1$user_screen_name)
```



# 02 Network data analysis

## 2.1 data frame to network

```{r}
library(igraph)

g <- rt_edge_inner %>% 
    mutate(year = year(tweet_time)) %>%
    # filter(year >= 2019) %>%
    count(from, to, year) %>%
    select(from, to, year, weight = n) %>%
    graph.data.frame(directed = T)
```


## 2.2 Inspecting vertices(V) and edges(E)

```{r}
E(g)$weight
E(g)$year %>% head

# V(g)
# V(g)$name
```


## 2.3 Detecting communities

```{r}

# Detecting community by information flow
V(g)$comm <- membership(infomap.community(g))

# V(g)$comm
table(V(g)$comm)
```


## 2.4 Computing vertices properties
- for detecting prominent actors, users, or nodes in a graph

```{r}

# centrality degree
V(g)$degree <- degree(g, mode="all")

# in-degree: interaction from out to in.
V(g)$indegree <- degree(g, mode="in")

#out-degree: Interaction from in to out
V(g)$outdegree <- degree(g, mode="out")

# closeness centrality: average of the shortest path length to other nodes
V(g)$closeness <- centralization.closeness(g)$res

# betweenness centrality: bridging degree of vertices
V(g)$betweenness <- centralization.betweenness(g)$res

```


## 2.5 Tabulating node properties

```{r}
nodes_all <- data.frame(name = V(g)$name,
                        degree = V(g)$degree,
                        indegree = V(g)$indegree,
                        outdegree = V(g)$outdegree,
                        closeness = V(g)$closeness,
                        community = V(g)$comm,
                        betweeness =V(g)$betweenness)

# nodes_all %>% View
```



# 03 Network Visualization

## 3.1 Basic plotting

```{r}
plot(g)
plot(g, edge.arrow.size = .4, vertex.label = NA)
```


## 3.2 (Option) Plotting vertices degree distribution

```{r}
plot(degree.distribution(g, cumulative=T), pch=20, xlab="degree", ylab="cumulative frequency")
```


## 3.3 Better network plotting

```{r}
l <- layout.fruchterman.reingold(g)
# l <- layout_with_kk(g)

plot(g, 
     layout = l,
     # vertex.color = rgb(1, 1, 0, 0.2),
     vertex.color = factor(V(g)$comm),
     vertex.size  = sqrt(V(g)$degree)*3,
     vertex.frame.color= rgb(0, 0, 0, 0.5),
     vertex.label = str_sub(V(g)$name, 1, 10),
     vertex.label.cex = 0.6,
     vertex.label.color = rgb(0, 0, 0, 0.5),
     edge.curved = 0.1,
     edge.arrow.size = 0.1, 
     edge.width = sqrt(E(g)$weight+1),
     edge.color = E(g)$year,
     edge.label = E(g)$year,
     edge.label.cex = 0.4,
     edge.label.color = E(g)$year
     )
```



# 04 Plotting network only for hk-anti-extraction

## 4.1 Extracting accounts and tweets related to hk anti-extract
- Tweets ate after 2019-01-01
- Copy from W2_dplyr.rmd

```{r}
# Keywords to detect
detects <- "港警|逃犯條例|反修例|遊行|修例|反送中|anti-extradition|hongkong|hkpolicebrutality|soshk|hongkongprotesters|HongKongPolice|hkpoliceforce|freedomHK|antiELAB|HongKongProtests|antiextraditionlaw|HongKongProtest|七一|游行|民阵|HongKong|逃犯条例|民陣|撐警|香港眾志|HongKongProterst|林鄭|警队|力撑"

hk_tweets <- tweets_1 %>%
    filter(tweet_time > as.Date("2019-01-01")) %>%
    mutate(hits = str_extract_all(tweet_text, detects)) %>%
    drop_na(hits)
```


## 4.2 Segmenting RT actions before and after 2019-06-05
- before: 2019-01-01 ~ 2019-06-04
- after: 2019-06-05 ~

```{r}
hk_rt_compare <- hk_tweets %>%
    mutate(date0605 = if_else(tweet_time > as.Date("2019-06-05"), 
                                  "after", "before")) %>%
    filter(str_detect(tweet_text,"RT @")) %>%
    select(user_screen_name, tweet_text, date0605)
    
hk_rt_compare  %>% count(date0605)
```


## 4.3 Segmenting data before and after 2019-06-05

```{r}

# Building rt edgelist of hk related tweets
hk_rt_edgelist <- hk_rt_compare %>%
    mutate(rt = str_extract_all(tweet_text, "RT @\\w+")) %>%
    unnest(rt) %>%
    mutate(to = str_sub(rt, 5, nchar(rt))) %>%
    # filter(to %in% users_1$user_screen_name) %>%
    select(from = user_screen_name, to, date0605)
```


## 4.4 Building network

```{r}
# Building graph: converting from edgelist
g_hk <- hk_rt_edgelist %>% 
    count(from, to, date0605) %>%
    select(from, to, date0605, weight = n) %>%
    graph.data.frame(directed = T)

# Examing network properties: communities
V(g_hk)$comm <- membership(infomap.community(g_hk))
table(V(g_hk)$comm)

# Examing ego-network properties
V(g_hk)$degree <- degree(g_hk, mode="all")
V(g_hk)$indegree <- degree(g_hk, mode="in")
V(g_hk)$outdegree <- degree(g_hk, mode="out")
V(g_hk)$closeness <- centralization.closeness(g_hk)$res
V(g_hk)$betweenness <- centralization.betweenness(g_hk)$res
V(g_hk)$deleted <- if_else(V(g_hk)$name %in% users_1$user_screen_name, 
                           rgb(1, 0.25, 0, 0.3), rgb(0, 0, 0, 0.1))

# Saving network properties into a table
nodes_hk_rt <- data_frame(name = V(g_hk)$name,
                        degree = V(g_hk)$degree,
                        indegree = V(g_hk)$indegree,
                        outdegree = V(g_hk)$outdegree,
                        closeness = V(g_hk)$closeness,
                        community = V(g_hk)$comm,
                        betweeness =V(g_hk)$betweenness,
                        deleted = V(g_hk)$deleted)
```


## 4.5 Plotting network

```
# Plotting network
plot(g_hk)
plot(g_hk, edge.arrow.size=.4,vertex.label=NA)
plot(degree.distribution(g_hk, cumulative=T), pch=20,xlab="degree", ylab="cumulative frequency")
```


## 4.6 Better plotting

```{r}
# Better network plotting
l <- layout.fruchterman.reingold(g_hk)

plot(g_hk, 
     layout = l,
     vertex.color = V(g_hk)$deleted,
     vertex.size  = sqrt(V(g_hk)$degree)*2,
     vertex.frame.color= rgb(0, 0, 0, 0.5),
     vertex.label = str_sub(V(g_hk)$name, 1, 10),
     vertex.label.cex = 0.6,
     vertex.label.color = rgb(0, 0, 0, 0.5),
     edge.curved = .1,
     edge.arrow.size = .1, 
     edge.width = E(g_hk)$weight/2,
     edge.color = factor(E(g_hk)$date0605)
     )
```



# (Option 1) Co-RT relationship

```{r}
library(widyr)

hk_rt_edgelist$from %>% unique() %>% length
hk_rt_edgelist$to %>% unique() %>% length

g_co_rt <- hk_rt_edgelist %>%
    filter(date0605 == "after") %>%
    pairwise_count(from, to, sort = TRUE) %>%
    filter(n > 1) %>%
    select(from = item1, to = item2, weight = n) %>%
    graph.data.frame(directed = F)

V(g_co_rt)$degree <- degree(g_co_rt, mode="all")
    
l <- layout.kamada.kawai(g_co_rt)
# l <- layout_nicely(g_co_rt)

plot(g_co_rt, 
     layout = l,
     vertex.color = rgb(1, 1, 0, 0.5),
     vertex.size  = sqrt(V(g_co_rt)$degree),
     vertex.frame.color= rgb(0, 0, 0, 0.5),
     vertex.label = str_sub(V(g_co_rt)$name, 1, 10),
     vertex.label.cex = 0.6,
     vertex.label.color = rgb(0, 0, 0, 0.5),
     edge.curved = .1,
     edge.width = E(g_co_rt)$weight/10,
     edge.label = E(g_co_rt)$weight,
     edge.label.cex = 0.4
     )
```



# (Option 2) RT-Co relationship

```{r}
library(widyr)

# hk_rt_edgelist$from %>% unique() %>% length
# hk_rt_edgelist$to %>% unique() %>% length

g_rt_co <- hk_rt_edgelist %>%
    filter(date0605 == "after") %>%
    pairwise_count(to, from, sort = TRUE) %>%
    filter(n > 1) %>%
    select(from = item1, to = item2, weight = n) %>%
    graph.data.frame(directed = F)

V(g_rt_co)$degree <- degree(g_rt_co, mode="all")
V(g_rt_co)$deleted <- if_else(V(g_rt_co)$name %in% users_1$user_screen_name, 
                           rgb(1, 0.25, 0, 0.3), rgb(0, 0, 0, 0.1))
    
l <- layout.kamada.kawai(g_rt_co)
# l <- layout_nicely(g_rt_co)

plot(g_rt_co, 
     layout = l,
     vertex.color = V(g_rt_co)$deleted,
     vertex.size  = sqrt(V(g_rt_co)$degree),
     vertex.frame.color= rgb(0, 0, 0, 0.5),
     vertex.label = str_sub(V(g_rt_co)$name, 1, 10),
     vertex.label.cex = 0.6,
     vertex.label.color = rgb(0, 0, 0, 0.5),
     edge.curved = .1,
     edge.width = E(g_rt_co)$weight/10,
     edge.label = E(g_rt_co)$weight,
     edge.label.cex = 0.4
     )
```