Exercise2CTA.Rmd

---
title: "Exercise2CTA"
output: html_document
date: "2024-02-07"
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
getwd()
```

##Packages

We load the required packages.
```{r packages}
library(academictwitteR) # for fetching Twitter data
library(tidyverse) # loads dplyr, ggplot2, and others
library(readr) # more informative and easy way to import data
library(stringr) # to handle text elements
library(tidytext) # includes set of functions useful for manipulating text
library(quanteda) # includes functions to implement Lexicoder
library(textdata)

```

## Data

We use the data from the Academic twitter package, for the period of January 1st 2020, to May 1st 2020, only accessing the tweets by given users (UK newspapers, whose handles are found manually). We access the RDS (R Data Set) object, and name it "tweets".
```{r dataset}

tweets  <- readRDS(gzcon(url("https://github.com/cjbarrie/CTA-ED/blob/main/data/sentanalysis/newstweets.rds?raw=true")))


```

Inspecting data with the "head" command we see that it is a tibble of 31 characteristics, which we can see using "colnames". To see what is important, we use this, then select relevant information, and rename some of the variables to make them easier to understand. 

```{r filter data}

head(tweets)
colnames(tweets)

tweets <- tweets %>%
  select(user_username, text, created_at, user_name,
         retweet_count, like_count, quote_count) %>%
  rename(username = user_username,
         newspaper = user_name,
         tweet = text)

```

Now we have a data set that covers the newspapers, the text of the tweets, when they were posted, and counts of retweets, quotes,and likes. 
The next important step is to tokenise the words, filtering out punctuation and stopwords. Now we have a longer tibble, tokenised by word. (with all other characteristics being the same within a tweet.)

```{r tokenise}

tidy_tweets <- tweets %>% 
  mutate(desc = tolower(tweet)) %>%
  unnest_tokens(word, desc) %>%
  filter(str_detect(word, "[a-z]"))%>%
   filter(!word %in% stop_words$word)

head(tidy_tweets)

```

Using the tidytext package, we look at 3 different sentiment dictionaries: AFINN, bing, and nrc. We can see that AFINN gives a score, and checking the values the value column takes, we see it ranges from -5 to 5. (very negative to very positive). Bing has a binary negative/positive classification (with no "neutral" option). NRC has 10 named sentiments, with each word getting a 1/0 binary value for each sentiment. 

```{r sentimentdict}
afinn<-get_sentiments("afinn")

bing<-get_sentiments("bing")

nrc<-get_sentiments("nrc")

unique(afinn$value)
unique(bing$sentiment)
unique(nrc$sentiment)

```

We can, for example, look for words that denote "fear" in the nrc lexicon and only keep the keywords that appear in tweets. We use the inner_join so none of the others are kept. While the by argument is not specified, this is ok in this instance, as they only share one named column: words. All seem reasonable in our opinion, except "mum", which is the most common "fear" related word, but likely to come up, eg. on Mother's day, which is in March. So we take it out.

```{r fear}

nrc_fear <- get_sentiments("nrc") %>% 
  filter(sentiment == "fear")

fear_tidy_tweets<-tidy_tweets %>%
  inner_join(nrc_fear) %>%
  count(word, sort = TRUE)%>%
  filter(word != "mum")

head(fear_tidy_tweets)
```

In order to adequately look at trends by date, we add a column that gives the order of tweets, added once we arrange by date. Note that we have to change the date formatting first. Also note that as.Date is quite a slow function.
```{r order}
tidy_tweets$date <- as.Date(tidy_tweets$created_at)

tidy_tweets <- tidy_tweets %>%
  arrange(date)

tidy_tweets$order <- 1:nrow(tidy_tweets)
```

We can find sentiment scores over time looking at negative vs positive sentiments. Note that for nrc, we disregard all sentiments except positive and negative, taking away some nuance. However, negative emotions will still be coded as negative here, supporting use of this feature.

```{r nrc plot}
tweets_nrc_sentiment <- tidy_tweets %>%
  inner_join(get_sentiments("nrc")) %>%
  count(date, index = order %/% 1000, sentiment) %>%
  spread(sentiment, n, fill = 0) %>%
  mutate(sentiment = positive - negative)

tweets_bing_sentiment<- tidy_tweets %>%
  inner_join(get_sentiments("bing")) %>%
  count(date, index = order %/% 1000, sentiment) %>%
  spread(sentiment, n, fill = 0) %>%
  mutate(sentiment = positive - negative)

tweets_afinn_sentiment <-tidy_tweets %>%
  inner_join(get_sentiments("afinn")) %>%
  group_by(date, index = order %/% 1000) %>% 
  summarise(sentiment = sum(value))

```

Now, we will plot them all together in ggplot for comparison.

``` {r plotsent}

ggplot()+
  geom_smooth(data=tweets_bing_sentiment,aes(date, sentiment),method= loess, alpha=0.25,colour="darkred") +
  geom_point(data=tweets_bing_sentiment,aes(date, sentiment),alpha=0.9,size=0.1,colour="darkred")+
  geom_smooth(data=tweets_afinn_sentiment,aes(date, sentiment),method= loess, alpha=0.25,colour="darkblue")+
   geom_point(data=tweets_afinn_sentiment,aes(date, sentiment),alpha=0.9,size=0.1,colour="darkblue")+
  geom_smooth(data=tweets_nrc_sentiment,aes(date, sentiment),method= loess, alpha=0.25,colour="darkgreen")+
   geom_point(data=tweets_nrc_sentiment,aes(date, sentiment),alpha=0.9,size=0.1,colour="darkgreen")+
    geom_text(data =tweets_bing_sentiment, date == "May"), aes(label = Bing, colour = "darkred", x = Inf, y = sentiment), hjust = -.1) +
  geom_text(data =tweets_afinn_sentiment, date == "May"), aes(label = Bing, colour = "darkblue", x = Inf, y = sentiment), hjust = -.1)+
  geom_text(data =tweets_nrc_sentiment, date == "May"), aes(label = Bing, colour = "darkred", x = Inf, y = sentiment), hjust = -.1) +
 labs(title="Sentiment over Time",
        x ="Date", y = "Sentiment")
  
```
 

Moving on, we can make a minimal dictionary of death related words, as follows. Here, we have no weighting, giving each a value of 1. We do not keep the gerund form: killing/dying as they can refer to other meanings: "millenials killing the DVD industry".

```{r value}
word<- c('death', 'illness', 'hospital', 'life', 'health',
             'fatality', 'morbidity', 'deadly', 'dead', 'victim')
value <- c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
mordict <- data.frame(word, value)

```

We then plot this, first as a basic bar chart of frequency. However, as this doesn't take into account the length of tweets/overall words tweeted per day. 
```{r plotmordict}
tidy_tweets %>%
  inner_join(mordict_df) %>%
  group_by(date, index = order %/% 1000) %>% 
  summarise(morwords = sum(value)) %>% 
  ggplot(aes(date, morwords)) +
  geom_bar(stat= "identity") +
  labs(main="Mentions of Mortality over time", y="Mortality Words", x="date")

```

Hence, we have to get the total tweets by day, and plot, using "full_join",as this keeps dates with no mortality words. We find the percentage of words per day that are mortality related, giving the value 0 to all other days. We go straight from mutating to plotting for efficiency. We use mordict as a vector, not a data frame here.
```{r mordictplot2}

totals <- tidy_tweets %>%
  mutate(obs=1) %>%
  group_by(date) %>%
  summarise(sum_words = sum(obs))

mordict_vect<- c('death', 'illness', 'hospital', 'life', 'health',
             'fatality', 'morbidity', 'deadly', 'dead', 'victim')

#plot
tidy_tweets %>%
  mutate(obs=1) %>%
   group_by(date) %>%
  filter(grepl(paste0(mordict_vect, collapse = "|"), word)) %>%
  summarise(sum_mwords = sum(obs)) %>%
  full_join(totals, word, by="date") %>%
  mutate(sum_mwords= ifelse(is.na(sum_mwords), 0, sum_mwords),
         pctmwords = sum_mwords/sum_words)%>%
  ggplot(aes(date, pctmwords)) +
  geom_point(alpha=0.5) +
  geom_smooth(method= loess, alpha=0.25)+
  labs(main="Relative Mentions of Mortality over time", y="% Mortality Words", x="Date")

```

###LEXICODER

Using the quanteda package, we create a corpus, making sure to fix the date storage first. We then tokenise the text and remove punctuation. Using an inbuilt data dictionary, we take away neutral words, and "look up" how the tokens will be scored.

```{r corpustok1}

tweets$date <- as.Date(tweets$created_at)

tweet_corpus <- corpus(tweets, text_field = "tweet", docvars = "date")
toks_news <- tokens(tweet_corpus, remove_punct = TRUE)


data_dictionary_LSD2015_pos_neg <- data_dictionary_LSD2015[1:2]

toks_news_lsd <- tokens_lookup(toks_news, dictionary = data_dictionary_LSD2015_pos_neg)

```
We now have a list, and can group this by date to get a document feature matrix, which we then plot. To use ggplot we need to recreate it in tidy format. 
```{r dfmplot}
dfmat_news_lsd <- dfm(toks_news_lsd) %>% 
  dfm_group(groups = date)

negative <- dfmat_news_lsd@x[1:121]
positive <- dfmat_news_lsd@x[122:242]
date <- dfmat_news_lsd@Dimnames$docs


tidy_sent <- as.data.frame(cbind(negative, positive, date))

tidy_sent$negative <- as.numeric(tidy_sent$negative)
tidy_sent$positive <- as.numeric(tidy_sent$positive)
tidy_sent$sentiment <- tidy_sent$positive - tidy_sent$negative
tidy_sent$date <- as.Date(tidy_sent$date)

tidy_sent %>%
  ggplot()+
  geom_line(aes(date, sentiment))+
  geom_line(aes(date,positive), colour="darkgreen")+
  geom_line(aes(date,negative), colour="darkred")+
  labs(main="Sentiment over Time", y="Sentiment Score",x="Date")
```

##Exercise 1
Take a subset of the tweets data by “user_name” These names describe the name of the newspaper source of the Twitter account. Do we see different sentiment dynamics if we look only at different newspaper sources?

We use the "tidy_tweets" set created earlier, keep newspaper, and merge the data sets to compare. FOR SOME REASON; ONLY TWO ARE SHOWN?

``` {r tweetsbyusername}


tweets_nrc_sentiment2 <- tidy_tweets %>%
  inner_join(get_sentiments("nrc")) %>%
  count(date, index = order %/% 1000, sentiment,newspaper) %>%
  spread(sentiment, n, fill = 0) %>%
  mutate(sentimentn = positive - negative)

tweets_bing_sentiment2<- tidy_tweets %>%
  inner_join(get_sentiments("bing")) %>%
  count(date, index = order %/% 1000, sentiment,newspaper) %>%
  spread(sentiment, n, fill = 0) %>%
  mutate(sentimentb = positive - negative)

tweets_afinn_sentiment2 <-tidy_tweets %>%
  inner_join(get_sentiments("afinn")) %>%
  group_by(date, index = order %/% 1000,newspaper) %>% 
  summarise(sentimenta = sum(value))

merged_sentiments2 <- merge(tweets_afinn_sentiment2,tweets_bing_sentiment2) %>%
              merge(tweets_nrc_sentiment2)
head(merged_sentiments2)

```
Having kept "newspaper", we try to plot it.

```{r newspaperplot}
ggplot(merged_sentiments2)+
  geom_smooth(aes(date, sentimenta),method= loess, alpha=0.25, colour="darkred") +
  geom_point(aes(date, sentimenta),alpha=0.1,size=0.1,colour="darkred")+
geom_smooth(aes(date, sentimentb),method= loess, alpha=0.25,colour="darkblue") +
  geom_point(aes(date, sentimentb),alpha=0.1,size=0.1,colour="darkblue")+
geom_smooth(aes(date, sentimentn),method= loess, alpha=0.25,colour="darkgreen") +
  geom_point(aes(date, sentimentn),alpha=0.1,size=0.1,colour="darkgreen")+
facet_wrap(~newspaper)+
  ylim(-20,10)+
 labs(title="Sentiment over Time",
        x ="Date", y = "Sentiment")


```

Based on observed data, we see very different newspaper dynamics. However the confidence intervals cast some cold water on the actual differences we can support. 


2.Build your own (minimal) dictionary-based filter technique and plot the result

We do not include "ace", as this could include card/gambling news. Similarly, the short form "pan" could refer to kitchen equipment.

``` {r lgbtdict}

word <- c('lgbt', 'lgbtq', 'queer', 'gay', 'bisexual','lesbian',
             'bi', 'transgender', 'trans', 'pansexual', 'asexual')
value <- c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1,)
lgbtdict <- data.frame(word, value)
lgbtdict

```

We think this will include plurals, because it's string matching, but this depends how it works with the tokenisation.


```{r plotlgbt}

totals <- tidy_tweets %>%
  mutate(obs=1) %>%
  group_by(date) %>%
  summarise(sum_words = sum(obs))

lgbt_vect<- c('lgbt', 'lgbtq', 'queer', 'gay', 'bisexual','lesbian',
             'bi', 'transgender', 'trans', 'pansexual', 'asexual')

#plot
tidy_tweets %>%
  mutate(obs=1) %>%
   group_by(date) %>%
  filter(grepl(paste0(lgbt_vect, collapse = "|"), word)) %>%
  summarise(sum_lwords = sum(obs)) %>%
  full_join(totals, word, by="date") %>%
  mutate(sum_lwords= ifelse(is.na(sum_lwords), 0, sum_lwords),
         pctlwords = sum_lwords/sum_words)%>%
  ggplot(aes(date, pctlwords)) +
  geom_point(alpha=0.5) +
  geom_smooth(method= loess, alpha=0.25)+
  labs(main="Relative Mentions of LGBT+ over time", y="% LGBT+ Words", x="Date")


```



#3.Apply the Lexicoder Sentiment Dictionary to the news tweets, but break down the analysis by newspaper


```{r corpustok}
tweets$date <- as.Date(tweets$created_at)
tweets <- tweets %>% mutate(id = row_number())
tweet_corpus <- corpus(tweets, text_field = "tweet", docvars ="id")
toks_news <- tokens(tweet_corpus, remove_punct = TRUE)


data_dictionary_LSD2015_pos_neg <- data_dictionary_LSD2015[1:2]

toks_news_lsd <- tokens_lookup(toks_news, dictionary = data_dictionary_LSD2015_pos_neg)

```

We now have a list, and can group this by date to get a document feature matrix, which we then plot. 

```{r dfmplot}

dfmat_news_lsd <- dfm(toks_news_lsd) %>% 
  dfm_group(groups = id)

negative <- dfmat_news_lsd@x[1:121]
positive <- dfmat_news_lsd@x[122:242]
id<- as.numeric(dfmat_news_lsd@Dimnames$docs)


# merge it with newspaper data again. (issue with dates?)

tidy_first<-cbind(negative, positive, id)

tweets_simple<-tweets[c("date","newspaper","id")]
tidy_news<-merge(tweets_simple,tidy_first, by="id")

tidy_news$negative <- as.numeric(tidy_news$negative)
tidy_news$positive <- as.numeric(tidy_news$positive)
tidy_news$sentiment <- tidy_news$positive - tidy_news$negative
tidy_news$newspaper<- as.factor(tidy_news$newspaper)
tidy_news$date<-as.factor(tidy_news$date)

levels(tidy_news$newspaper)
tidy_news$newspaper
X<-split(tidy_news,tidy_news$newspaper)


dm<-X$`Daily Mail U.K.`
dm<-dm%>%
  select(-c("newspaper","id"))%>%
  group_by(date) %>% 
summarise_at(c("sentiment","positive","negative"), sum, na.rm = TRUE)%>%
  mutate(date=as.Date(date))}

dm%>%
  ggplot()+
  geom_line(aes(date, sentiment))+
  geom_line(aes(date,positive), colour="darkgreen")+
  geom_line(aes(date,negative), colour="darkred")+
  labs(main="Sentiment over Time", y="Sentiment Score",x="Date")+
  scale_x_date(date_labels = "%b")

```

# couldn't get the for loop or lapply to work
X<-lapply(
  select(-c("newspaper","id"))%>%
  group_by(date) %>% 
summarise_at(c("sentiment","positive","negative"), sum, na.rm = TRUE)%>%
  mutate(date=as.Date(date)))