Duplicates you caused!

Duplicates are not always the fault of the data itself. We can cause our own duplicates through incorrectly written code.

Joining – using the incorrect fields or edits to fields needed pivot_wider – including too many columns in the select pivot_longer – including too many columns in the pivot Integrating validation steps, unit testing, and code reviews into your work will reduce the number of “coder-caused” duplicates.

Fixing the duplicates

Option 1:

Correct the dupes individually with if_else or case_when. This method is best for errors in one or 2 rows as a quick and easy fix.

correct_students <- students |>
  mutate(grade = if_else(student_id == ______, CORRECT-GRADE, grade)) |>
  distinct() 

Option 2:

Systematic errors can be fixed by taking a summarize on the incorrect column. In this case, we could assume that the lower grade-level is the correct one for all duplicate enrollments. This method works better for systematic issues that you know how to correct, such as taking the higher of two homework assignemtns.

correct_students <- students |>
   group_by(student_id) |> 
   summarize(grade = min(grade))

correct_students

Option 3:

Output the duplicates and manually choose which version to keep.

students |> 
  get_dupes(student_id) |>
  write_csv("../data/dupes.csv", na = "")

# Create this file in excel manually, often by asking your coworkers for help
dupes_remove <- read_csv("../data/dupes_correct.csv") |> 
  filter(delete == 1)

students_correct <- anti_join(students, dupes_remove) 

DOCUMENT DOCUMENT DOCUMENT.

Now that you’ve fixed the duplicates, whether in the database and/or code, DOCUMENT what you did and WHY, so that when the data changes and new duplicates are found, the code still runs.

Word counts

I start with TF-IDF. The goal here is to see what I was talking about each year and how it may differ as I got older. As a reminder, I have changed the names of all my friends and family for privacy. That way you don’t know who “nameofbestfriend” is and why I stopped mentioning “nameofbestfriend” in 2006. (We had a bit of a falling out at the end of HS.)

tfidf <- lj_words_corrected %>%
  count(year, word) %>% 
  tidytext::bind_tf_idf(word, year, n) %>%
  anti_join(stop_words, by = "word") %>%
  group_by(year) %>%
  top_n(n = 10, wt = tf_idf) %>%
  ungroup() %>%
  filter(n >= 2) 

Look at 2009 - clearly my only entries were my Norse myth college class. I remember I put a few of my class papers on my Live Journal.

We can look at the differences between TF-IDF and a regular word count, while accounting for stop words.

wordcount <- lj_words_corrected %>%
  count(year, word) %>%
  anti_join(stop_words, by = "word") %>%
  group_by(year) %>%
  top_n(n = 10, wt = n) %>%
  ungroup() %>%
  filter(n >= 3) 

Sentiment

Next I look at sentiment. I remember using live journal to be super angsty. I assumed that I would largely see negative sentiment and words across the years.

df_plot <- lj_words_corrected %>%
  left_join (get_sentiments("bing"), by = "word" ) %>%
  rename(bing_sentiment = sentiment) %>%
  left_join (get_sentiments("nrc"), by = "word" ) %>%
  rename(nrc_sentiment = sentiment) %>%
  pivot_longer(cols = c(bing_sentiment, nrc_sentiment), 
               names_to = "sentiment_type", values_to = "sentiment") %>%
  count(sentiment_type, year, sentiment) %>%
  filter(! is.na(sentiment) ) %>%
  rename(count = n ) %>%
  group_by(sentiment_type, year) %>%
  mutate(total = sum(count)) %>%
  ungroup() %>%
  mutate(percent = count / total,
         year_month = ymd(str_c(year, "01", "01", sep = "-"))
         ) 

Instead, it seems my words were largely more positive then negative. (Outside of 2009 - which is either largely from my anxiety attacks that year or Norse mythology is just super depressing.) Not as ansgty as I remember!

Ah, but did I mark my “current mood” / how are you feeling” part as positive as my words are? You be the judge.

What kind of data do we work with?

The Data Analytics team focuses at the overall picture, so while we don’t work on reporting data directly back to the state (our regional teams have excellent people working on this) we do get to work with most of the data the organization collects.

The data we work with generally fits into one of these buckets…

• Assessments: Interim assessments are taken as practice tests through out the school year.
• Exams: Common Core aligned state exams, SAT, PSAT, APs, …etc
• Classroom: Assignment grades, attendance, suspensions, …etc
• Teacher: student - course - teacher linkage information
• Staff Data: HR and Recruitment

Unfortunately, given the amount of data we have and the number of sources it may be coming from, we have a ton of data challenges to overcome in every analysis.

Of course, every piece of data has its own challenges and “messy nature,” but there are patterns.

• Missing/Incomplete data
• Different data sources without matching IDs (i.e HR to Teacher to Student)
• Movement between schools and courses of students and teachers
• Alignment of data and data processes across all schools and regions
• Student IDs that change
• Human data reporting error
• Historical data quality

Some of these challenges are easy to fix and some are harder. For example, a messy excel sheet can be cleaned (by hand or by code). We’ve developed (or are developing in some cases) systems to work with most of these types of challenges.

• Messy excel sheets (historical or human entered)
• Column names that don’t apply anymore
• Lack of historical documentation
• Finding duplicate tests
• Students that take half of one test and the other half of another
• Vanishing leading zeros
• Tracking of student IDs that change
• Lack of common definitions (i.e “cohort”)
• How to refer to school years or school abbreviations
• Data audits

An example, using janitor to clean a messy excel file.

I’m often tasked with cleaning roster files, which contain entry and exit data for students. These files can be very messy due to students who moved between schools or were not exited properly from the system, causing duplicates.

To clean this data, first I read it in with read_excel and use janitor's clean_names to convert all the column names to something I can use. remove_empty() removes entire columns or rows that are NA as excel sometimes can’t tell where there is data and where there isn’t.

I choose to use col_type = "text" in my read_excel statement, because I sometimes have to deal with leading zeros, NAs that are not written as NAs, or other text fields in my numerical columns. Reading in as text and converting later allows me to find and correct problems before they become NAs. I use mutate_at to convert these columns back to numbers after examining that everything looks good.

students <- readxl::read_excel(filepath, sheet="Sheet1", col_types = "text") %>%
  janitor::clean_names() %>%
  janitor::remove_empty_cols() %>%
  janitor::remove_empty_rows() %>%
  dplyr::mutate_at(vars(entrydate, exitdate, student_id, yearsinuncommon), as.numeric) %>%
  dplyr::mutate_at(vars(entrydate, exitdate), excel_numeric_to_date) 

The next step in data cleaning is to look for duplicates. Luckily, janitor has a super helpful get_dupes() function which does just that!

students %>% 
  get_dupes(student_id)

# A tibble: 2 x 6
  student_id dupe_count grade yearsinuncommon  entrydate   exitdate
       <dbl>      <int> <dbl>           <dbl>     <date>     <date>
1    2342675          2    10               1 2017-11-11 2017-12-11
2    2342675          2    11               1 2017-11-11 2017-12-11

In this example data, I have one student with duplicate information. They’re in two different grades, ugh! Now, I have to choose a method to correct this student in my data.

mutate(students, grade = if_else(student_id == 2342675, 10, grade))

group_by(students, student_id) %>% summarize(grade = min(grade))

dupes_correct <- read_csv("dupes_correct.csv")
left_join(students, dupes_correct) %>%
  replace_na(list(keep = 0)) %>%
  assert(not_na, keep) %>%
  filter(keep = 0)

check <- students %>% 
  get_dupes(student_id) %>% 
  verify(nrow(.) == 0)

Gathering Data

The first part of this analysis is grabbing all the text from YouTube. To access the API, I use the tuber package by Gaurav Sood.

library(tidyverse)
library(tuber)

yt_oauth(app_id, app_password)

The fist step was to find the channel id to access the LBD playlist. I do a quick search for lizziebennet to find some videos that I know are part of the series.

search <- yt_search("lizziebennet")[1:5, ] 
search %>% select(title, channelId)

##                                                       title
## 1                         My Name is Lizzie Bennet  - Ep: 1
## 2 The Lizzie Bennet Diaries - Episódio 98 (LEGENDADO PT-BR)
## 3                                      Yeah I Know - Ep: 61
## 4                                 Introducing Lizzie Bennet
## 5                                  The Lizzie Trap - Ep: 78
##                  channelId
## 1 UCXfbQAimgtbk4RAUHtIAUww
## 2 UCfhdE-vIhW9GD0eGdd300ag
## 3 UCXfbQAimgtbk4RAUHtIAUww
## 4 UCGaVdbSav8xWuFWTadK6loA
## 5 UCXfbQAimgtbk4RAUHtIAUww

With the channel ID in hand, I can now access the channel’s resources to find the playlist ID, which I will use to access all the videos in that playlist. list_channel_resources for tuber creates a list of channel attributes and buried in that list in the playlist ID.

# Channel Information
a <- list_channel_resources(filter = c(channel_id="UCXfbQAimgtbk4RAUHtIAUww"), part="contentDetails")

# Uploaded playlists:
playlist_id <- a$items[[1]]$contentDetails$relatedPlaylists$uploads

playlist_id

## [1] "UUXfbQAimgtbk4RAUHtIAUww"

The YouTube API automatically pages videos so the max you get per page is 50. I know I need more than that, so I created a function that I call a few times to get all the videos. (This way works, but I would love any comments on how to make it better.)

# pass NA as next page to get first page
nextPageToken <- NA
vid_info <-{}

# Loop over every available page
repeat {
  vids <- get_playlist_items(filter= c(playlist_id=playlist_id), page_token = nextPageToken)
  vid_ids <- map(vids$items, "contentDetails") %>%
      map("videoId")  %>%
      unlist()
    
  vid_info <- vid_info %>% bind_rows(tibble(ids = vid_ids))
    
  # get the token for the next page
  nextPageToken <- ifelse(!is.null(vids$nextPageToken), vids$nextPageToken, NA)
    
  # if no more pages then done
  if(is.na(nextPageToken)){
     break
  }

}

# check that I have all 112 videos
nrow(vid_info)

## [1] 112

Now that I have a list of video IDs, I can use get_captions to access the text of the videos. I also use xmlTreeParse and xmlToList to covert the caption into into an easily accessible lines of text. I put the text, video ID, and video title in a tibble for use in tidydata.

library(XML)

getText <- function(id){
  x <- get_captions(id, lang = "en")
  title <- get_video_details(id)$title
  a <- xmlTreeParse(x)
  text <- a$doc$children$transcript
  text <- xmlToList(text, simplify = TRUE, addAttributes = FALSE) %>% 
    tibble() %>%
    mutate(id = id, title = title)
  return(text) 
}

text <- map_df(vid_ids, getText) %>% 
  set_names(c("text", "vid_id", "title"))

I don’t actually want to refer to each video by it’s full title, so I do some data munching to get each episode’s number (1-100). Notice, the Q&A videos do not get a episode number assigned to them. For the sake of this analysis, I’ve decided to only work with the main 100 episodes.

titles <- text %>%
  distinct(title) %>%
  mutate(title = ifelse(title == "Question and Answers #3 (ft. Caroline Lee)", "Questions and Answers #3 (ft. Caroline Lee)", title),
         ep_num = gsub("[- .)(+!',/]|[a-zA-Z]*:?","", title),
         ep_num = ifelse(title == "2 + 1 - Ep: 73", 73, ep_num),
         ep_num = ifelse(title == "25 Douchebags and a Gentleman - Ep:18", 18, ep_num),
         ep_num = ifelse(title == "Bing Lee and His 500 Teenage Prostitutes - Ep: 4", 4, ep_num),
         ep_num = parse_number(ep.num)
         ) %>%
  filter(!grepl("Questions and Answers", title)) %>%
  arrange(ep_num) 

One of the problems with using captions, is the messy text. I used a simple set of gsub commands to transform obvious punctuation marks into their English counterparts. I also pulled out the character SPEAKING the words from the text itself. I left this column alone in the data set, but might one day go back and focus an analysis on speaking characters.

library(tidytext)
library(stringr)

lizziebennet <- text %>%
  left_join(titles, by="title") %>%
  filter(!is.na(ep_num)) %>%
  arrange(ep_num) %>%
  mutate(linenumber = row_number()) %>%
  mutate(text = gsub("&#39;", "'", text),
         text = gsub("&quot;", '\"', text),
         text = gsub("&amp;", "and", text),
         character = str_extract(text, "^[a-zA-Z]*:"),
         text = sub("^[a-zA-Z]*:", "", text)
         ) %>%
  arrange(ep_num, linenumber)

Okay, so now the text is mostly in place. The first thing I did was look at word counts. The most common words are not surprising, it’s just a list of the characters.

lizziebennet %>%
  tidytext::unnest_tokens(word, text) %>%
  anti_join(stop_words, by="word") %>%
  count(word, sort=TRUE) %>%
  top_n(10)

## # A tibble: 10 × 2
##         word     n
##        <chr> <int>
## 1     lizzie   460
## 2       jane   301
## 3      darcy   243
## 4       bing   232
## 5    collins   220
## 6      lydia   196
## 7     bennet   194
## 8  charlotte   180
## 9       yeah   178
## 10      time   176

Not surprisingly, the most common trigrams are from the phrase that begins every episode, “My name is Lizzie Bennet and…”

lizziebennet %>%
  tidytext::unnest_tokens(word, text, token="ngrams", n=3) %>%
  count(word, sort=TRUE) %>%
  top_n(10)

## # A tibble: 10 × 2
##                 word     n
##                <chr> <int>
## 1         my name is   106
## 2   is lizzie bennet    96
## 3     name is lizzie    96
## 4  lizzie bennet and    84
## 5       i don't know    40
## 6          oh my god    36
## 7           a lot of    33
## 8        going to be    31
## 9       what are you    29
## 10     mr collins oh    28

I was also especially amused by So good to see you! and THE MOST AWKWARD DANCE EVER being in the Top 10 5-grams.

## # A tibble: 12 × 2
##                           word     n
##                          <chr> <int>
## 1     my name is lizzie bennet    95
## 2    name is lizzie bennet and    83
## 3       is lizzie bennet and i    19
## 4    is lizzie bennet and this    14
## 5    lizzie bennet and this is    11
## 6           so good to see you     9
## 7       had nothing to do with     5
## 8     is lizzie bennet and i'm     5
## 9       lizzie bennet and i am     5
## 10 the most awkward dance ever     5
## 11     what are you doing here     5

Sentiment Analysis

I’ve chosen to use the Bing lexicon (because of Bing Lee, get it?). In Tidydata, sentiment analysis is easy because you just join the lexicon against your tokenzied words.

bing <- sentiments %>%
        filter(lexicon == "bing") %>%
        select(-score)

lbwordcount <- lizziebennet %>%
  tidytext::unnest_tokens(word, text) %>%
  anti_join(stop_words) %>%
  count(title)
  
lbsentiment <- lizziebennet %>%
  tidytext::unnest_tokens(word, text) %>%
  anti_join(stop_words) %>%
  inner_join(bing) %>% 
  count(title, index=ep_num, sentiment) %>% 
  spread(sentiment, n, fill = 0) %>% 
  left_join(lbwordcount) %>%
  mutate(sentiment = positive - negative,
         sentiment = sentiment / n)  

Most positive sentiment episodes:

## # A tibble: 5 × 2
##                                       title  sentiment
##                                       <chr>      <dbl>
## 1                    Care Packages - Ep: 58 0.09623431
## 2                         The End - Ep: 100 0.09375000
## 3                   Jane Chimes In - Ep: 12 0.09132420
## 4 My Parents: Opposingly Supportive - Ep: 3 0.08415842
## 5      Wishing Something Universal - Ep: 76 0.08018868

Most negative sentiment episodes:

## # A tibble: 5 × 2
##                            title   sentiment
##                            <chr>       <dbl>
## 1   Turn About the Room - Ep: 32 -0.15217391
## 2        How About That - Ep: 91 -0.09937888
## 3          Staff Spirit - Ep: 59 -0.09745763
## 4 How to Hold a Grudge  - Ep: 74 -0.09352518
## 5      Meeting Bing Lee - Ep: 28 -0.07614213

The next step was to visualize this in a way where you can look at the sentiment over the episodes.

library(viridis)
theme_set(theme_bw()) # a theme with a white background

ggplot(lbsentiment, aes(x=index, sentiment, fill=as.factor(index))) +
        geom_bar(stat = "identity", show.legend = FALSE) +
        theme_minimal(base_size = 13) +
        geom_text(data=plot_text, aes(x=index, y=sentiment, label=index), size=3.5) + 
        labs(title = "Sentiment in Lizzie Bennet Diaries",
             y = "Sentiment") +
        scale_fill_viridis(end = 0.75, discrete=TRUE, direction = -1) +
        scale_x_discrete(expand=c(0.02,0)) +
        theme(strip.text=element_text(hjust=0)) +
        theme(strip.text=element_text(face = "italic")) +
        theme(axis.title.x=element_blank()) +
        theme(axis.ticks.x=element_blank()) +
        theme(axis.text.x=element_blank())

Sentiment by main episode of LBD.

Julia’s sentiment analysis of the original text is much more positive than my LBD analysis, with two negative portions relating to Darcy proposing to Elizabeth and Lydia running away with Wickham. I had expected a similar “Wickham” negative spike in this plot, and while the section of Wickham related episodes (Ep 84 to Ep 89) is surely negative it’s not more negative than some of the introductory episodes.

One could argue, that since most of the Lydia - Wickham story line happens off screen and in Lydia’s blogs, that would explain that lack of a clear negative spike in the Wickham episodes.

More sentiment

Continuing the analysis, I wanted to look at which words were causing the largest effect on the overall sentiment.

bing_word_counts %>%
  group_by(sentiment) %>%
  top_n(10) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(y = "Contribution to sentiment",
       x = NULL) +
  coord_flip()

Given that this is a modern adaption, it’s interesting that much like the analysis done on the original “miss” is the top contribution to negative sentiment. In the original text I would assume a higher count of “Miss Bennet’s” to the modernized version. However, Lizzie does talk about you she’ll miss Charlotte, or she misses her home… etc, so it’s not too surprising to see it have a considerable contribution here.

I did a bit of an investigation into this with bigrams.

lizziebennet %>%
  tidytext::unnest_tokens(bigram, text, token="ngrams", n=2) %>%
  separate(bigram, c("word1", "word2"), sep = " ") %>%
  filter(word1 == "miss") %>%
  mutate(miss_in_name = ifelse(word2 %in% c("bennet", "lu"), "Yes", "No")) %>%
  count(miss_in_name)

## # A tibble: 2 × 2
##   miss_in_name     n
##          <chr> <int>
## 1           No    26
## 2          Yes    27

And oddly enough, the use of the word “miss” is about half and half between “I miss [person/thing]” and “Miss Bennet” type phrases. Interesting! (Anyone want to guess who refers to Lizzie as Miss Bennet the most? Unsurprisingly, it’s Ricky Collins.)

More with Bigrams

bigrams_separated <- lizziebennet %>%
  tidytext::unnest_tokens(bigram, text, token="ngrams", n=2) %>%
  separate(bigram, c("word1", "word2"), sep = " ") %>%
  filter(!word1 %in% stop_words$word, 
         !word2 %in% stop_words$word) %>%
  count(word1, word2, sort = TRUE)

bigrams_separated %>% 
  ungroup() %>%
  top_n(10) 

## # A tibble: 11 × 3
##      word1   word2     n
##      <chr>   <chr> <int>
## 1   lizzie  bennet   132
## 2     bing     lee    43
## 3   george wickham    24
## 4      hey  lizzie    24
## 5       de  bourgh    22
## 6    ricky collins    21
## 7      los angeles    20
## 8     miss  bennet    19
## 9     tour  leader    18
## 10   video    blog    17
## 11 william   darcy    17

Not surprisingly, the common bigrams are first and last names of characters, but there’s also some fun other popular bigrams with “tour leader” and “video blog.” I guess vlog wasn’t super popular to use on it’s own yet.

Network of Words

One of my favorite part of tidytext is the example on making a bigram network. It’s just so fun!

library(igraph)
library(ggraph)

set.seed(42)

bigrams_separated %>%
  filter(n > 5) %>%
  graph_from_data_frame() %>%
  ggraph(layout = "fr") +
  geom_edge_link() +
  geom_node_point() +
  geom_node_text(aes(label = name), vjust = 1, hjust = 1) +
  theme(axis.title.x=element_blank()) +
  theme(axis.ticks.x=element_blank()) +
  theme(axis.text.x=element_blank()) +
  theme(axis.title.y=element_blank()) +
  theme(axis.ticks.y=element_blank()) +
  theme(axis.text.y=element_blank())

I especially enjoy the Bennet sister cluster in the left corner.

I leave you with this last picture.

Some of cast of Lizzie Bennet Diaries and me. Vidcon 2014

How R Helps Airbnb Make the Most of Its Data

Ricardo Bion, Airbnb

• AirBnD started in 2008 with 1 city and 1 room, now there are 3M homes in 71K cities
• 100+ data scietists using a mix of lanaguages mostly R, but lots of python
• Why to use an R Packages:
  • Passing around functions required duplication of work, where as a package can include data, test, add-ins, vignettes, R markdown and notebook templates
  • The AirBnB packages have consistent API, branded visualization, branded templates, and of course function functions
• Education:
  • made a difference in confidence of R stats
  • new hire buddy
  • intro classes at datacamp if interested, sponsored by airbnb
  • peer support with office hours, code review, R slack group
  • learning lunch, journal club, offsites
• Reproducibility:
  • scale knowedlge
  • knowledge repo
  • posts have tags with topics, date, then served as a web ui
  • uses github for peer review
  • branded template
  • incorprate best practices from academia and software

Hearing about reproducibility and R packages at Airbnb from @ricardobion at NYR #rstatsnyc pic.twitter.com/uGSZwtX1XM

— Julia Silge (@juliasilge) April 21, 2017

Fine Grained Visual Category Recognition and Perceptual Embedding

Serge Belongie, Cornell University

• Really intersting talk on using Stochastic Neighbor algorithm with crowd sourcing to get a visual similarity of images.
• Motivation of humans and computers working together
• His talk focused on detecting what type of bird was in an image
• Detecting that there is a bird in the picture is getting easy for computer, but detecting the exact name of the bird is much more difficult

.@SergeBelongie showing a stochastic neighbor algorithm with crowd sourcing to get a visual similarity of images #rstatsnyc pic.twitter.com/TRuIruTELS

— Erin Grand (@astroeringrand) April 21, 2017

An R Cloud Computing Lifeline: The Missing Manual for Running R on Amazon Cloud

Kelly O’Briant, B23

• Working with R in the cloud is different from working with Rstudio on your computer, you have to install all your favroite packages again every time you start up a new server
• bigger instances sizes, analysis while sleeping, running multiple R servers at the same time, instances themselves are disaposable and renewable resources
• able to use tools in a more powerful manor

@b23kelly created a custom R package to set up new projects/deployed servers fast without having to reconfigure anew each time #rstatsnyc

— Alec Barrett (@alecbarrett) April 21, 2017

R Makes the World Go ’Round: Data-Driven Decision Making at JetBlue

Catherine Zhou, JetBlue

One of the interesting take always throughout the conference was that you can start coding in R pretty quickly, if you start with the right ideas and tools about what R is. Most intro stats classes (mine included) treat R and other programming languages as a calculator. But R is so much more than that! Catherine Z made a point about giving people templates with tidyverse functions to produce their own analyses. In my own work, I’m helping my coworker think of tidyverse in a similar way to computing excel tasks, i.e group_by() %>% summarise() is equivalent to a pivot table, and mutate() adds a new column the same you way you might by drag-and-dropping an excel equation.

There were also several good comments on how you can learn slowly by doing something small in R (such as doing a bit of cleaning) and then porting it back out to excel or tableau to finish your analysis. Best takeaway? Not everyone needs to be fluent in R. - Catherine

.@catherinezh Easy sells: Automation and reproducibility, but not everyone needs to be fluent. pic.twitter.com/EancCU2eWb

— Erin Grand (@astroeringrand) April 21, 2017

Theoretical Statistics is the Theory of Applied Statistics: How to Think About What We Do

Andrew Gelman, Columbia

Reiterating Andrew Gelman’s point about how p-value statistical testing is actually a pretty bad framework for hypothesis testing most of the time. time to really brush up on Bayesian stats. I also liked the general point that you should have to state what you expect to find (alternate hypotheses, estimated effect sizes, whatever) before you go barging around looking for anything and act like whatever you end up with is what you were searching for all along.

The Unreasonable Effectiveness of Empathy - The killer skill needed for a successful technical career

JD Long, RenaissanceRe

• Analysis doesn’t end at result delivery - it ends at developing and proselytizing new business strategies and innovation.
• Automating Excel workflows can be a first step towards bringing R to a team.
• Agile development tells user stories as an empathy hack.

As a ______
I want ______
So I can ______ 

• Person-level stories (the near) are always more meaningful than data stories (the far). We need to balance both as Data Scientists.
• In development, we need to have an actual user in mind, rather than a theoretical user who wants everything.

“As you tell your data stories, think about the individual people in your data and your consumers” - @CMastication #rstatsnyc pic.twitter.com/f9DZGCprDO

— Emily Robinson (@robinson_es) April 21, 2017

Using Human Mobility Data to Assess Public Circulation Health

Michael Kane

• talking about cell phone tower data -> homan ciculatory data
• anytime you are connecting or handing off to a cell phone tower, the data is logged
• secruity at: IDS are all hashed version of cell phones
• 35 TB data set! big! 10 GB every day
• touching ALL of the day when doing analysis
• use R, with Hadoop
• using HMR in cluster (hapood)
• looks at inflow and outflow counts (flux)
• are providing reliable information on human mobility on storms
• want the learner that is the most REGULAR not the closest to accuracy

From Agreeing to Marching to Organizing: OSS Needs You

Mike Malecki and Neal Richardson

• Best ways to contribute to open source are to start with improving documentation
• Open source contributions: failing test with fix > failing test > bug report
• Remember to include sessionInfo()
• When releasing a package, release quickly, but also slowly - take time to fix dumb decisions
  • Bring something new to the community, but don’t reinvent the wheel
  • Tell people about your package (social media), then listen to how they’re using it
  • When thinking about a package: documentation > usability > performance > features

Corrections to common problem with academic graphics: * Log-log plots with missing or symmetric error bars can be fixed by forcing asymmetric error bars. When there are small errors, the log can show as a negative error, which often means that plot won’t do anything. In Matplotlib the default (for the y axis) is to map all negative values a very small positive one. The code for that is:

plt.yscale('log', nonposx='clip')

• Tick marks: The defaults for tick marks and labels are often too large, too small, facing the wrong direction or else wise strange. In ggplot in R this can be manipulated under theme:

For example, to make everything expect your points or lines disappear you’d use:

 theme(axis.line=element_blank(),
        axis.text.x=element_blank(),
        axis.text.y=element_blank(),
        axis.ticks=element_blank(),
        axis.title.x=element_blank(),
        axis.title.y=element_blank(),
        legend.position="none",
        panel.background=element_blank(),
        panel.border=element_blank(),
        panel.grid.major=element_blank(),
        panel.grid.minor=element_blank(),
        plot.background=element_blank()) 

Each of these could also be modified to make the text larger or smaller, change the font, rotate the labels…etc.

To change the direction and size of the tick labels you’d use something like:

theme(axis.text.x  = element_text(angle=90, vjust=0.5, size=16)

• You can reduce clutter on the graph by using fewer (labeled) tick marks.

• Always remember to label your axes! This is done in python with:

import matplotlib.pyplot as plt
plt.xlabel('X Label')
plt.ylabel('Y Label')

or in ggplot2 in R using:

  labs(title = "Plot Title") +
  ylab("X Label") +
  xlab("Y Label")

• The problem of too many overlapping lines or points can be solved in various ways depending on the data. Sometimes, changing the colors and alpha of the points might be enough. In other cases, it’s best to separate out the information into a table of plots.

For example, below is the original plot from my research showing the intensity of different molecules across velocities. The plot places all four molecules on the same graph with a key indicating which is which. In color, this graphic might make more sense, but it is still hard to make out the individual curves. Plus, the key is small and referring back to it is time consuming and annoying.

In fixing the graphic, while also including more information from my other sources, I separated out the each of the molecules and sources into a table of spectra. This un-clutters the plot and allows you to more easily visualize trends in the sources. (Notice how the plot is missing axes labels - shame on me!)

The code for this plot was done in IDL - a language mostly used only by astronomers (after looking at the code, you’ll see why no one else joined in the fun…) If you’re interested, you can check it out here.

For the future, I want to try and remake some of my research plots in R for better practice with R and ggplot2, using something along these lines:

data %>%
ggplot(aes(x=vel,y=tb)) + 
geom_line() + 
facet_wrap(~pillar)

• In attempts to not waste space, you should examine the size and scale of the axes. This often shows up as a problem when an outlier or two that expand the axes such that much of the plot is empty. In these cases, you can crop the plot to the main data and include an arrow to show where the outlier is.

Most importantly, don’t display empty plots like this infamous plot from Wittke-Thompson JK and Pluzhnikov A, Cox NJ (2005).

• Sometimes, to save room or otherwise, it’s best to display the information in a table format instead of bar graphs. For example, this plot from a science article titled “Studying Gender in Conference Talks – data from the 223rd meeting of the American Astronomical Society” shows the large difference in number questions asked by males vs number asked by females, given a male or female chair. This plot displays the most significant finding from the analysis: a strong dependence on session chair gender. Still, this information could have easily been shown in a table instead of a graph. This would be a useful plot for a presentation on the subject, but not needed in the article.

Common color problems.

• Eye piercing bight colors and/or use of rainbow colors. We’ve discussed the problems with the rainbow in class, but as a reminder: the rainbow color scheme includes colors which are hard to see, doesn’t have a universally understood order, artificially exaggerates differences in color while softening the differences between others and (importantly for print) doesn’t convert well to black and white.

Contrast is one of THE biggest problems I see in academic figures. Things like cyan or yellow on white, red on blue, navy on black… these cause major problems (and headaches) when reading text or trying to discern between lines. Your plot doesn’t have to be pretty, but it does have to be legible!

Color in astronomy maps often tags along with Color-coded image of the molecular cloud