Practice getting data from the Twitter API
library(tidyverse)
library(broom)
set.seed(1234)
theme_set(theme_minimal())
Run the code below in your console to download this exercise as a set of R scripts.
usethis::use_course("uc-cfss/getting-data-from-the-web-api-access")
There are several packages for R for accessing and searching Twitter. Twitter actually has two separate APIs:
- The REST API - this allows you programmatic access to read and write Twitter data. For research purposes, this allows you to search the recent history of tweets and look up specific users.
- The Streaming API - this allows you to access the public data flowing through Twitter in real-time. It requires your R session to be running continuously, but allows you to capture a much larger sample of tweets while avoiding rate limits for the REST API.
Popular packages for the Twitter API in R include:
twitteRis the most popular package for R, but it only allows you to access the REST API. It is also deprecated (not actively updated), in lieu of a new up-and-coming package (identified below)streamRis more complicated, but allows you to query the Streaming API from R. It is ancient in computational terms (last updated in January 2014), but does what it needs to do.rtweetis a relatively recent addition to the R package universe that allows you to access both the REST and streaming APIs.
Using rtweet
Here, we are going to practice using the rtweet package to search Twitter.
install.packages("rtweet")
library(rtweet)
OAuth authentication
All you need is a Twitter account (user name and password) and you can be up in running in minutes!
Simply send a request to Twitter’s API (with a function like search_tweets(), get_timeline(), get_followers(), get_favorites(), etc.) during an interactive session of R, authorize the embedded rstats2twitter app (approve the browser popup), and your token will be created and saved/stored (for future sessions) for you!
Students using RStudio Workbench will not be able to follow this authentication process since you cannot run an interactive session of R that connects to your web browser. Instead, I will provide you with a script during class that you can run to create your token.
Searching tweets
To find 3000 recent tweets using the “rstats” hashtag:
rt <- search_tweets(
q = "#rstats",
n = 3000,
include_rts = FALSE
)
rt
## # A tibble: 3,000 × 90
## user_id status_id created_at screen_name text source
## <chr> <chr> <dttm> <chr> <chr> <chr>
## 1 4263007693 1496929184542867462 2022-02-24 19:25:03 gp_pulipaka "Top 1… Buffer
## 2 4263007693 1496711514069295107 2022-02-24 05:00:07 gp_pulipaka "6 Fre… Buffer
## 3 4263007693 1496696497039200256 2022-02-24 04:00:26 gp_pulipaka "A Lis… Buffer
## 4 4263007693 1496707222461718530 2022-02-24 04:43:03 gp_pulipaka "AI Be… Buffer
## 5 4263007693 1496695646891483145 2022-02-24 03:57:04 gp_pulipaka "#AI B… Buffer
## 6 4263007693 1496710741445025796 2022-02-24 04:57:02 gp_pulipaka "#AI B… Buffer
## 7 4263007693 1496172703681859584 2022-02-22 17:19:04 gp_pulipaka "The C… Buffer
## 8 4263007693 1496340562055778305 2022-02-23 04:26:05 gp_pulipaka "Autom… Buffer
## 9 4263007693 1496161379690106894 2022-02-22 16:34:04 gp_pulipaka "The C… Buffer
## 10 4263007693 1496303058275508228 2022-02-23 01:57:03 gp_pulipaka "#AI B… Buffer
## # … with 2,990 more rows, and 84 more variables: display_text_width <dbl>,
## # reply_to_status_id <chr>, reply_to_user_id <chr>,
## # reply_to_screen_name <chr>, is_quote <lgl>, is_retweet <lgl>,
## # favorite_count <int>, retweet_count <int>, quote_count <int>,
## # reply_count <int>, hashtags <list>, symbols <list>, urls_url <list>,
## # urls_t.co <list>, urls_expanded_url <list>, media_url <list>,
## # media_t.co <list>, media_expanded_url <list>, media_type <list>, …
q- the search queryn- maximum number of tweets to be returnedinclude_rts = FALSE- exclude retweets generated by Twitter’s built-in “retweet” function. We only want original tweets.
The resulting object is a tibble data frame with one row for each tweet. The data frame contains the full text of the tweet (text), the username of the poster (screen_name), as well as a wealth of metadata.
Note that the Twitter REST API limits all searches to the past 6-9 days. You will not retrieve any earlier results.
Searching users
Use get_timeline() or get_timelines() to retrieve tweets from one or more specified Twitter users. This only works for users with public profiles or those that have authorized your app.
countvoncount <- get_timeline(user = "countvoncount", n = 1000)
countvoncount
## # A tibble: 1,000 × 90
## user_id status_id created_at screen_name text source
## <chr> <chr> <dttm> <chr> <chr> <chr>
## 1 555129716 1468700740285874180 2021-12-08 21:55:18 CountVonCount Three… Count…
## 2 555129716 1468640341196738564 2021-12-08 17:55:18 CountVonCount Three… Count…
## 3 555129716 1468429755242455040 2021-12-08 03:58:30 CountVonCount Three… Count…
## 4 555129716 1468339156493426693 2021-12-07 21:58:29 CountVonCount Three… Count…
## 5 555129716 1468034370204221444 2021-12-07 01:47:23 CountVonCount Three… Count…
## 6 555129716 1467958871172034568 2021-12-06 20:47:22 CountVonCount Three… Count…
## 7 555129716 1467613900413992960 2021-12-05 21:56:35 CountVonCount Three… Count…
## 8 555129716 1467266070788595724 2021-12-04 22:54:26 CountVonCount Three… Count…
## 9 555129716 1466873476569092105 2021-12-03 20:54:24 CountVonCount Three… Count…
## 10 555129716 1466552417869312006 2021-12-02 23:38:38 CountVonCount Three… Count…
## # … with 990 more rows, and 84 more variables: display_text_width <dbl>,
## # reply_to_status_id <lgl>, reply_to_user_id <lgl>,
## # reply_to_screen_name <lgl>, is_quote <lgl>, is_retweet <lgl>,
## # favorite_count <int>, retweet_count <int>, quote_count <int>,
## # reply_count <int>, hashtags <list>, symbols <list>, urls_url <list>,
## # urls_t.co <list>, urls_expanded_url <list>, media_url <list>,
## # media_t.co <list>, media_expanded_url <list>, media_type <list>, …
With get_timelines(), you are not limited to only the most recent 6-9 days of tweets.
Visualizing tweets
Because the resulting objects are data frames, you can perform standard data transformation, summarization, and visualization on the underlying data.
rtweet includes the ts_plot() function which automates some common time series visualization methods. For example, we can quickly visualize the frequency of #rstats tweets:
ts_plot(rt, by = "3 hours")
The by argument allows us to aggregate over different lengths of time.
ts_plot(rt, by = "1 hours")
And because ts_plot() uses ggplot2, we can modify the graphs using familiar ggplot2 functions:
ts_plot(rt, by = "3 hours") +
theme(plot.title = element_text(face = "bold")) +
labs(
x = NULL, y = NULL,
title = "Frequency of #rstats Twitter statuses from past 9 days",
subtitle = "Twitter status (tweet) counts aggregated using three-hour intervals",
caption = "\nSource: Data collected from Twitter's REST API via rtweet"
)
Exercise: Practice using rtweet
- Find the 1000 most recent tweets by Katy Perry, Kim Kardashian West, and Rihanna.
Visualize their tweet frequency by week. Who posts most often? Who posts least often?
Click for the solution
popstars <- get_timelines( user = c("katyperry", "KimKardashian", "rihanna"), n = 1000 ) popstars %>% group_by(screen_name) %>% ts_plot(by = "week")
Acknowledgments
- This page is derived in part from “UBC STAT 545A and 547M”, licensed under the CC BY-NC 3.0 Creative Commons License.
- OAuth token storage derived from “Obtaining and using access tokens”.
Session Info
devtools::session_info()
## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.1.2 (2021-11-01)
## os macOS Monterey 12.1
## system aarch64, darwin20
## ui X11
## language (EN)
## collate en_US.UTF-8
## ctype en_US.UTF-8
## tz America/Chicago
## date 2022-02-24
## pandoc 2.17.1.1 @ /Applications/RStudio.app/Contents/MacOS/quarto/bin/ (via rmarkdown)
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date (UTC) lib source
## askpass 1.1 2019-01-13 [1] CRAN (R 4.1.0)
## assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.1.0)
## backports 1.4.1 2021-12-13 [1] CRAN (R 4.1.1)
## blogdown 1.7 2021-12-19 [1] CRAN (R 4.1.1)
## bookdown 0.24 2021-09-02 [1] CRAN (R 4.1.1)
## brio 1.1.3 2021-11-30 [1] CRAN (R 4.1.1)
## broom * 0.7.12 2022-01-28 [1] CRAN (R 4.1.1)
## bslib 0.3.1 2021-10-06 [1] CRAN (R 4.1.1)
## cachem 1.0.6 2021-08-19 [1] CRAN (R 4.1.1)
## callr 3.7.0 2021-04-20 [1] CRAN (R 4.1.0)
## cellranger 1.1.0 2016-07-27 [1] CRAN (R 4.1.0)
## cli 3.2.0 2022-02-14 [1] CRAN (R 4.1.1)
## codetools 0.2-18 2020-11-04 [1] CRAN (R 4.1.2)
## colorspace 2.0-3 2022-02-21 [1] CRAN (R 4.1.1)
## crayon 1.5.0 2022-02-14 [1] CRAN (R 4.1.1)
## curl 4.3.2 2021-06-23 [1] CRAN (R 4.1.0)
## DBI 1.1.2 2021-12-20 [1] CRAN (R 4.1.1)
## dbplyr 2.1.1 2021-04-06 [1] CRAN (R 4.1.0)
## desc 1.4.0 2021-09-28 [1] CRAN (R 4.1.1)
## devtools 2.4.3 2021-11-30 [1] CRAN (R 4.1.1)
## digest 0.6.29 2021-12-01 [1] CRAN (R 4.1.1)
## dplyr * 1.0.7 2021-06-18 [1] CRAN (R 4.1.0)
## ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.1.0)
## evaluate 0.15 2022-02-18 [1] CRAN (R 4.1.1)
## fansi 1.0.2 2022-01-14 [1] CRAN (R 4.1.1)
## farver 2.1.0 2021-02-28 [1] CRAN (R 4.1.0)
## fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.1.0)
## forcats * 0.5.1 2021-01-27 [1] CRAN (R 4.1.1)
## fs 1.5.2 2021-12-08 [1] CRAN (R 4.1.1)
## generics 0.1.2 2022-01-31 [1] CRAN (R 4.1.1)
## ggplot2 * 3.3.5 2021-06-25 [1] CRAN (R 4.1.1)
## glue 1.6.1 2022-01-22 [1] CRAN (R 4.1.1)
## gtable 0.3.0 2019-03-25 [1] CRAN (R 4.1.1)
## haven 2.4.3 2021-08-04 [1] CRAN (R 4.1.1)
## here 1.0.1 2020-12-13 [1] CRAN (R 4.1.0)
## highr 0.9 2021-04-16 [1] CRAN (R 4.1.0)
## hms 1.1.1 2021-09-26 [1] CRAN (R 4.1.1)
## htmltools 0.5.2 2021-08-25 [1] CRAN (R 4.1.1)
## httr 1.4.2 2020-07-20 [1] CRAN (R 4.1.0)
## jquerylib 0.1.4 2021-04-26 [1] CRAN (R 4.1.0)
## jsonlite 1.7.3 2022-01-17 [1] CRAN (R 4.1.1)
## knitr 1.37 2021-12-16 [1] CRAN (R 4.1.1)
## labeling 0.4.2 2020-10-20 [1] CRAN (R 4.1.0)
## lifecycle 1.0.1 2021-09-24 [1] CRAN (R 4.1.1)
## lubridate 1.8.0 2021-10-07 [1] CRAN (R 4.1.1)
## magrittr 2.0.2 2022-01-26 [1] CRAN (R 4.1.1)
## memoise 2.0.1 2021-11-26 [1] CRAN (R 4.1.1)
## modelr 0.1.8 2020-05-19 [1] CRAN (R 4.1.0)
## munsell 0.5.0 2018-06-12 [1] CRAN (R 4.1.0)
## openssl 1.4.6 2021-12-19 [1] CRAN (R 4.1.1)
## pillar 1.7.0 2022-02-01 [1] CRAN (R 4.1.1)
## pkgbuild 1.3.1 2021-12-20 [1] CRAN (R 4.1.1)
## pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.1.0)
## pkgload 1.2.4 2021-11-30 [1] CRAN (R 4.1.1)
## prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.1.0)
## processx 3.5.2 2021-04-30 [1] CRAN (R 4.1.0)
## progress 1.2.2 2019-05-16 [1] CRAN (R 4.1.0)
## ps 1.6.0 2021-02-28 [1] CRAN (R 4.1.0)
## purrr * 0.3.4 2020-04-17 [1] CRAN (R 4.1.0)
## R6 2.5.1 2021-08-19 [1] CRAN (R 4.1.1)
## Rcpp 1.0.8 2022-01-13 [1] CRAN (R 4.1.1)
## readr * 2.1.1 2021-11-30 [1] CRAN (R 4.1.1)
## readxl 1.3.1 2019-03-13 [1] CRAN (R 4.1.0)
## remotes 2.4.2 2021-11-30 [1] CRAN (R 4.1.1)
## reprex 2.0.1 2021-08-05 [1] CRAN (R 4.1.1)
## rlang 1.0.1 2022-02-03 [1] CRAN (R 4.1.1)
## rmarkdown 2.11 2021-09-14 [1] CRAN (R 4.1.1)
## rprojroot 2.0.2 2020-11-15 [1] CRAN (R 4.1.0)
## rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.1.0)
## rtweet * 0.7.0 2020-01-08 [1] CRAN (R 4.1.0)
## rvest 1.0.2 2021-10-16 [1] CRAN (R 4.1.1)
## sass 0.4.0 2021-05-12 [1] CRAN (R 4.1.0)
## scales 1.1.1 2020-05-11 [1] CRAN (R 4.1.0)
## sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.1.1)
## stringi 1.7.6 2021-11-29 [1] CRAN (R 4.1.1)
## stringr * 1.4.0 2019-02-10 [1] CRAN (R 4.1.1)
## testthat 3.1.2 2022-01-20 [1] CRAN (R 4.1.1)
## tibble * 3.1.6 2021-11-07 [1] CRAN (R 4.1.1)
## tidyr * 1.1.4 2021-09-27 [1] CRAN (R 4.1.1)
## tidyselect 1.1.1 2021-04-30 [1] CRAN (R 4.1.0)
## tidyverse * 1.3.1 2021-04-15 [1] CRAN (R 4.1.0)
## tzdb 0.2.0 2021-10-27 [1] CRAN (R 4.1.1)
## usethis 2.1.5 2021-12-09 [1] CRAN (R 4.1.1)
## utf8 1.2.2 2021-07-24 [1] CRAN (R 4.1.0)
## vctrs 0.3.8 2021-04-29 [1] CRAN (R 4.1.0)
## withr 2.4.3 2021-11-30 [1] CRAN (R 4.1.1)
## xfun 0.29 2021-12-14 [1] CRAN (R 4.1.1)
## xml2 1.3.3 2021-11-30 [1] CRAN (R 4.1.1)
## yaml 2.3.5 2022-02-21 [1] CRAN (R 4.1.1)
##
## [1] /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library
##
## ──────────────────────────────────────────────────────────────────────────────