R on TanHo.ca

Data Science Hangout

Thu, 23 Oct 2025 00:00:00 +0000

Recording
#

DSLC Slack: Debugging Netlify Deploys

Thu, 19 Dec 2024 00:00:00 +0000

I greatly enjoy the DSLC Slack Community ( dslc.io/join), and spend a fair bit of time answering questions! This post is part of a series where I curate some interesting questions and/or answers that I’ve come across there.

Question
#

Hi all! I’m having a hard time understanding why netlify checks on a Pull Request with only a empty line added in the commit are failing for a blogdown website. I created this PR as a small example, but basically any kind of PR is failing nevermind its content, and the site is published-rendered without problem if we commit directly to main.

I’m super puzzled by this, does anyone have any tips on how can I debug it? Thanks 🙏

- MC Nanton

Answer
#

Let’s look through some of your Netlify deploy logs. I found these from your GitHub repo automated checks:

editor’s note: archived the deploy logs in question to this gist just in case the deploy logs are deleted at some point

It seems like your PR deploy logs show this warning immediately before failing:

WARN 2024/12/10 21:35:19 Module "hugo-academic" is not compatible with this Hugo
version; run "hugo mod graph" for more information.

and this definitely doesn’t show up on your main deploy logs. Let’s try to figure out where your hugo versions are specified. Here’s a relevant section of the deploy logs:

35:19 PM: ❯ Current directory
35:19 PM: /opt/build/repo
35:19 PM: ​
35:19 PM: ❯ Config file
35:19 PM: /opt/build/repo/netlify.toml
35:19 PM: ​
35:19 PM: ❯ Context
35:19 PM: deploy-preview

The logs say that the file responsible for specifying hugo version is netlify.toml, so let’s have a look at what that looks like on GitHub right now:

[context.production.environment]
HUGO_VERSION = "0.64.0"

Notice that it specifies context.production here, and that the context for the failing logs is deploy-preview. If we go back into the failing logs, we can confirm that the hugo-version specified here isn’t working properly:

4:35:12 PM: Installing Hugo 0.57.2

So a solution that should work is to update the netlify.toml to specify the HUGO_VERSION for the deploy-preview context:

[context.production.environment]
HUGO_VERSION = "0.64.0"
[context.deploy-preview.environment]
HUGO_VERSION = "0.64.0"

and this should do the trick. You might need to push it to main instead of to a PR branch though, Netlify usually uses the config from the main repo branch.

(Update: this solution worked!)

The Beginner Data Science Programming Book That Doesn't Exist (Yet?)

Tue, 10 Dec 2024 00:00:00 +0000

Ideas are worth nothing unless executed, and I’m tired of adding ideas to my backlog only to never get to them. This series aims to set those ideas free - let me know if you take it on!

Hot take: the ideal beginner data science programming book doesn’t exist (yet)
#

The beloved R for Data Science is a fantastic book, and yet I struggled immensely with following along with it. I think it’s because I can only care so much about flower irises, penguins, and NYC taxis: I quit halfway through the third chapter of R4DS on my first attempt in 2018ish, … because I got bored.

Since then, I’ve gotten treatment for ADHD (which probably explains a lot, tbh), but I think what really helped the most with learning programming was that I started working on projects that I actually cared about: fantasy football and NFL.

I’d work on something for a while, get stuck on something I didn’t know, Ctrl-F the R4DS book + dplyr/tidyr reference manuals, figure out what I needed to know and how it fit into what I already knew, and then keep going. Reading R4DS cover to cover was utterly unhelpful until I knew how to fit each concept into a mental model that I knew how to use.

My love of all things R and data has since taken off (enough to have made a whole career of it), and now I often get asked what books I’d recommend for learning NFL/sports analytics, R, data science, programming etc. I supply the easy answer (“you should read R4DS”), but I’m a little uncomfortable with it because the book I wish I could recommend doesn’t quite exist.

the ideal book for learning programming and NFL analytics
#

There are a lot of introductory data science and programming books out there, and even ones that apply the context of NFL: both Eric Eager and Brad Congelio have great iterations of this (and the latter is free online)!

I’m so glad these books exist now and they’d have certainly helped a lot when I first got started, but I’m not quite sure these are the ones I’ve been wanting:

how do they stay up to date with a fast-evolving programming landscape?
how do they compare with the existing literature that is beloved by the community, like R4DS, and how should beginners proceed if the books disagree on best code approach?
what happens if I want to learn another programming language other than R?

I think the book I want to exist would be structured as a companion to books like R4DS, but not include any language-specific code whatsoever. Instead, I’d focus on a few goals:

create a series of case studies and exercises that challenge learners to write their own solutions based on domain-specific research questions
- provide starter data of appropriate complexity/cleanliness in a data repository
cover a full breadth of high-level data science / programming topics
- kind of like R4DS’s visualize, transform, import, program, communicate sections
maintain up-to-date links to readings for each case study that help learners find best-practice guidance on a given topic in a supported programming language
- each case study should provide links to “prerequisite reading” to solve it, e.g. links to relevant R4DS chapters for R
- should be easily extended to other languages by linking to equivalent material, e.g. pivoting data, group by + summarize are relational data concepts that have parallels in almost every language
introduce the essential background knowledge for the domain in question
- for NFL this should include concepts like EPA, fourth downs, completion probability, expected rushing yards etc - essentially covering the body of public research
suggest further research questions that can be tackled beyond the controlled sandbox of the book/repository

I think something like this would be massively helpful for introducing people to the relevant public research in a domain, and with putting technical topics into practical use for learners. It would be robust to existing languages evolving (could approach the same case study in e.g. tidyverse, data.table etc), easily extended to new languages (e.g. Python, Julia, Rust), and even to new technologies (probably could do the whole thing in DuckDB one day). The format also would be easy to adopt in different sports (soccer, basketball, baseball, hockey?) and to other domains altogether.

Of course, I’ve been sitting on this idea for a while: I first started a repo for this idea ~ three years ago and convinced friends like Sebastian Carl to start thinking about it, but neither of us have made much progress, which I think is more due to just a lack of time/energy/expertise in it. I still think it’s a worthwhile idea and am writing this post mostly to set it free and hope someone is inspired to pick it up!

A similar idea has come up since then: the SCORE project led by CMU’s Statistics and Data Science department aims to provide case studies across all sports for learning statistics and data science, and has a set of modules already. I’m not sure if ~ learning to program in a given language is in the scope of the project - it seems to be very statistics-focused - but it could otherwise be an interesting parallel project?

Adventures In Scaling Shiny: Lessons Learned Building DynastyProcess

Thu, 10 Oct 2024 00:00:00 +0000

Recording
#

Materials
#

Repo: https://github.com/tanho63/talk_shinyprod2024_scaling

tidyr pivot trick: names_to -> .value

Thu, 30 Nov 2023 00:00:00 +0000

I originally shared this tip for extracting data from column names as a tweet, and I’ve found myself sharing it over and over again since. Here’s a brief transcription now that I’m no longer on Twitter.

One of my favourite data-wrangling tricks is tidyr::pivot_longer’s names_to & .value sentinel - use-cases don’t come up often, but it’s so deeply ✨ satisfying ✨ when it does!

Scenario
#

Let’s say you have this starting dataframe:

# A tibble: 601 × 22
 season title caught_fred caught_daphnie caught_velma caught_shaggy
 <chr> <chr> <chr> <chr> <chr> <chr>
 1 1 What a Nig… FALSE FALSE FALSE TRUE
 2 1 A Clue for… FALSE FALSE FALSE TRUE
 3 1 Hassle in … FALSE FALSE FALSE FALSE
 4 1 Mine Your … TRUE FALSE FALSE FALSE
 5 1 Decoy for … FALSE FALSE FALSE FALSE
 6 1 What the H… TRUE FALSE FALSE FALSE
 7 1 Never Ape … TRUE FALSE FALSE FALSE
 8 1 Foul Play … FALSE FALSE FALSE FALSE
 9 1 The Backst… FALSE FALSE FALSE TRUE
10 1 Bedlam in … FALSE FALSE FALSE TRUE
# ℹ 591 more rows
# ℹ 16 more variables: caught_scooby <chr>, captured_fred <chr>,
# captured_daphnie <chr>, captured_velma <chr>, captured_shaggy <chr>,
# captured_scooby <chr>, unmask_fred <chr>, unmask_daphnie <chr>,
# unmask_velma <chr>, unmask_shaggy <chr>, unmask_scooby <chr>,
# snack_fred <chr>, snack_daphnie <chr>, snack_velma <chr>,
# snack_shaggy <chr>, snack_scooby <chr>
# ℹ Use `print(n = ...)` to see more rows

and you want this ending dataframe:

# A tibble: 3,005 × 7
 season title character caught captured unmask snack
 <chr> <chr> <chr> <chr> <chr> <chr> <chr>
 1 1 What a Night for a Knight fred FALSE FALSE FALSE TRUE
 2 1 What a Night for a Knight daphnie FALSE FALSE FALSE FALSE
 3 1 What a Night for a Knight velma FALSE FALSE FALSE FALSE
 4 1 What a Night for a Knight shaggy TRUE FALSE FALSE FALSE
 5 1 What a Night for a Knight scooby TRUE FALSE TRUE FALSE
 6 1 A Clue for Scooby Doo fred FALSE TRUE TRUE FALSE
 7 1 A Clue for Scooby Doo daphnie FALSE TRUE FALSE FALSE
 8 1 A Clue for Scooby Doo velma FALSE TRUE FALSE TRUE
 9 1 A Clue for Scooby Doo shaggy TRUE FALSE FALSE FALSE
10 1 A Clue for Scooby Doo scooby FALSE FALSE FALSE FALSE
# ℹ 2,995 more rows
# ℹ Use `print(n = ...)` to see more rows

How do you extract the characters (Fred, Velma, Daphnie, Shaggy, Scooby) from the column names into a character column?

Past me
#

Hmm, well, I can pivot_longer, then separate the name, then pivot_wider again?

three_lines <- x %>%
 pivot_longer(cols = -c("season","title")) %>%
 separate(name,into = c("action","character"), sep = "_") %>%
 pivot_wider(names_from = "action", values_from = "value")

# A tibble: 3,005 × 7
 season title character caught captured unmask snack
 <chr> <chr> <chr> <chr> <chr> <chr> <chr>
 1 1 What a Night for a Knight fred FALSE FALSE FALSE TRUE
 2 1 What a Night for a Knight daphnie FALSE FALSE FALSE FALSE
 3 1 What a Night for a Knight velma FALSE FALSE FALSE FALSE
 4 1 What a Night for a Knight shaggy TRUE FALSE FALSE FALSE
 5 1 What a Night for a Knight scooby TRUE FALSE TRUE FALSE
 6 1 A Clue for Scooby Doo fred FALSE TRUE TRUE FALSE
 7 1 A Clue for Scooby Doo daphnie FALSE TRUE FALSE FALSE
 8 1 A Clue for Scooby Doo velma FALSE TRUE FALSE TRUE
 9 1 A Clue for Scooby Doo shaggy TRUE FALSE FALSE FALSE
10 1 A Clue for Scooby Doo scooby FALSE FALSE FALSE FALSE
# ℹ 2,995 more rows
# ℹ Use `print(n = ...)` to see more rows

This seems to work okay!

But what if I told you there’s a one-liner for this exact situation?
#

pivot_longer has a “names_to” argument that takes a special “sentinel value” called ".value" - it immediately re-pivots the columns so that “character” (fred, daphne etc) stays as a column and then the “.value” part becomes colnames again (caught, captured etc)

one_line <- x %>%
 pivot_longer(
 cols = -c("season","title"),
 names_to = c(".value","character"),
 names_sep = "_"
 )

# A tibble: 3,005 × 7
 season title character caught captured unmask snack
 <chr> <chr> <chr> <chr> <chr> <chr> <chr>
 1 1 What a Night for a Knight fred FALSE FALSE FALSE TRUE
 2 1 What a Night for a Knight daphnie FALSE FALSE FALSE FALSE
 3 1 What a Night for a Knight velma FALSE FALSE FALSE FALSE
 4 1 What a Night for a Knight shaggy TRUE FALSE FALSE FALSE
 5 1 What a Night for a Knight scooby TRUE FALSE TRUE FALSE
 6 1 A Clue for Scooby Doo fred FALSE TRUE TRUE FALSE
 7 1 A Clue for Scooby Doo daphnie FALSE TRUE FALSE FALSE
 8 1 A Clue for Scooby Doo velma FALSE TRUE FALSE TRUE
 9 1 A Clue for Scooby Doo shaggy TRUE FALSE FALSE FALSE
10 1 A Clue for Scooby Doo scooby FALSE FALSE FALSE FALSE
# ℹ 2,995 more rows
# ℹ Use `print(n = ...)` to see more rows

Why use many line when one line do trick?

You can read more about this pivot_longer feature in this tidyr vignette.

Bonus: data.table equivalent
#

I learned later (from Hadley quote-tweeting my thread) that this feature was inspired by a similar feature in data.table, so I wanted to track down how it would be done there. As it turns out, kind of difficult because the feature isn’t in the CRAN version: it was in development as of October 2020, merged to main in May 2021, and still hasn’t made it onto CRAN because of some data.table governance issues (which are finally getting close to being resolved?)

In any case, with the main branch of rdatatable/data.table as of today, here’s how you could do the same pivot trick:

# requires development version of data.table and/or 1.15.0+
rlang::check_installed("data.table (>= 1.14.99)")
data.table::data.table(x) |>
 data.table::melt(
 id.vars = c("season", "title"),
 # measure() is not actually exported but is silently parsed under the hood by melt()
 # - uses `sep = "_"` to identify & process the column names (i.e. split with sep "_")
 # - new column names (i.e. character) are provided unquoted
 # - value.name is equivalent to tidyr's `.value`
 measure.vars = measure(value.name, character, sep = "_")
 )

 season title character caught captured unmask snack
 <char> <char> <char> <char> <char> <char> <char>
1 What a Night for a Knight fred FALSE FALSE FALSE TRUE
1 A Clue for Scooby Doo fred FALSE TRUE TRUE FALSE
1 Hassle in the Castle fred FALSE FALSE TRUE TRUE
1 Mine Your Own Business fred TRUE FALSE TRUE FALSE
1 Decoy for a Dognapper fred FALSE FALSE FALSE FALSE
 ---
2 The Dreaded Remake of Jekyll & Hyde! scooby FALSE FALSE FALSE FALSE
Movie Happy Halloween, Scooby-Doo! scooby FALSE FALSE FALSE FALSE
Movie Scooby-Doo! The Sword and the Scoob scooby FALSE FALSE FALSE FALSE
2 Dark Diner of Route 66! scooby TRUE FALSE FALSE FALSE
2 Total Jeopardy! scooby FALSE FALSE FALSE FALSE

Full Code
#

library(tidyverse)

scooby_data <- read.csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-07-13/scoobydoo.csv")

x <-
 scooby_data %>%
 select(season, title,
 starts_with("caught"),
 starts_with("captured"),
 starts_with("unmask"),
 starts_with("snack"),
 -contains("other"),
 -contains("not")) %>%
 filter(title!= "Wrestle Maniacs")

three_lines <- x %>%
 pivot_longer(cols = -c("season","title")) %>%
 separate(name,into = c("action","character"), sep = "_") %>%
 pivot_wider(names_from = "action", values_from = "value")

one_line <- x %>%
 pivot_longer(cols = -c("season","title"), names_to = c(".value","character"), names_sep = "_")

rlang::check_installed("data.table (>= 1.14.9)")
data.table::data.table(x) |>
 data.table::melt(
 id.vars = c("season", "title"),
 # measure() is not actually exported but is silently parsed under the hood by melt()
 # - uses `sep = "_"` to identify & process the column names (i.e. split with sep "_")
 # - new column names (i.e. character) are provided unquoted/as bare symbols
 # - value.name is equivalent to tidyr's `.value`
 measure.vars = measure(value.name, character, sep = "_")
 )

Original gist for reference.

Effective Logging Strategies

Fri, 13 Oct 2023 00:00:00 +0000

Recording
#

Materials
#

Repo: https://github.com/tanho63/talk_shinyprod2023_logging

Debugging Shiny Apps

Fri, 17 Mar 2023 00:00:00 +0000

Recording
#

Materials
#

https://github.com/tanho63/talk_shinyconf2023_debugging

blogdown, distill, and personal websites with R

Fri, 04 Feb 2022 00:00:00 +0000

I’ve recently been asked three times for my advice on whether to use blogdown or distill when getting started with your first website with R, so in true DRob fashion: here’s a blog post!

2023 note This post never actually got properly fleshed out beyond a set of bullet notes, but I figure it’s still worth sharing 😅 Have added a couple of 2023 notes as well.

TLDR
#

They’re both great, and incredibly well supported within the R community.
It doesn’t matter what you start with, you can redo your site later. You probably will want to at some point down the line regardless, it’s part of the learning process! Your content will transfer with minimal effort.
I recommend starting with blogdown and Hugo Apero in most cases, EXCEPT
…if you know you dislike CSS and styling, in which case Distill is perfect for you
Buy a domain name, serve it with Netlify or GitHub Pages, and go forth and grow your garden!

Distill
#

Distill is minimalist in that it doesn’t let you do a lot aesthetically - this means that most distill sites are straightforward and clean and incredibly easy to maintain/use going forward.
The most you’ll do is pick fonts and colours - everything else is pretty much set out for you.

2023 note RStudio/Posit folks are promoting Quarto websites for this minimal/simple aesthetic now.

Distill sites will still work (Distill simplicity at it’s finest!), but if you haven’t started your site yet you might be interested in starting from the “latest hotness”.

Another compelling reason to use to Quarto is the ability to use Python and Observable JS alongside with R chunks, so if that’s of interest I’d look into switching.

Blogdown
#

blogdown is decidedly not minimal - everything is flexible and can be laid out in any number of ways
this means you can pick out some kickass hugo themes from somewhere and plug them in so that your site doesn’t look like other people’s sites
- e.g. https://tanho.ca/ and https://ffverse.com/ are both blogdown sites, but don’t have the same feel at all
the flexibility means you’re more likely to break your site somehow, and also increases the temptation that you’ll end up nuking your site and rebuilding it from scratch.
the latest/safest hotness in blogdown is Alison Hill’s Hugo Apero theme
I spend a bunch of time doing front-end with Shiny so I’m not as scared to wade into the guts of css/html crap - this site is primarily made from a theme called Toha, and ffverse.com from a theme called Osprey

2023 note I’ve since ported tanho.ca to a new theme, Blowfish.

I found that I was overriding a lot of the Toha theme components manually and realized I wanted something that was more flexible.

Which one do I choose?
#

Whichever one appeals to you, if you’ve read all the way through to here! In most cases, I think there’s a lot of value in learning to use blogdown, and would recommend starting there with Apero…except if:

you don’t care about the aesthetics,
you know you hate CSS and style decisions, or
you want to get started quickly

in which case Distill is perfect for you!

Setting Up Your Own Shiny Server

Mon, 11 Oct 2021 00:00:00 +0000

Recording
#

Wrangling JSON and Lists

Mon, 02 Aug 2021 00:00:00 +0000

Recording
#

Web Scraping with rvest

Mon, 19 Jul 2021 00:00:00 +0000

Recording
#

Adventures in HTTP Testing for R

Mon, 08 Mar 2021 00:00:00 +0000

Embrace unit testing karma

Karma says:

Do good things and good things will happen to you.

Do them the way you know.

Do them the way you like.

Karma is flexible. Testing needs flexibility.

Karma thrives on creativity. Testing needs creativity. - Testivus

While developing ffscrapr, I’ve taken a deeper dive into HTTP testing for R packages. I’ve settled on using httptest and find it very convenient path to building and using mock data, especially since I’m using httr extensively within the package. I wish Scott Chamberlain & Maelle Salmon’s new book HTTP testing in R had been written before I started - that would have been incredibly helpful as a guiderail for my experimentations! Here are some thoughts on what I’ve discovered while testing ffscrapr, mostly trying to capture parts of my experiences that differ from what’s already in the HTTP testing book.

Why test?
#

Testing is about making sure that the package (and functions) work as intended, in spite of external factors. With API packages, sources of errors can include:

internet connectivity (both from the user and from the API)
API response changes (format changes, variable naming, etc)
dependency changes (this could include base R versions, operating systems, as well as the more obvious package dependencies)
users doing unexpected things with your package (finding corner cases)

A good test setup should test at least the first three of these (user-side corner cases being difficult to foresee at time of function definition), and ideally should test each area individually while holding the other two parts constant. For example, you want to test how the package handles internet connectivity without worrying about whether the API response changes, or whether the dependencies have changed. You also want to test the package every time the dependencies change, without worrying about internet connectivity or API responses changing. And lastly, you want to test and make sure that the real API response has not changed, because it might return unexpected results. In addition to these concerns, another really important element to consider is making sure that it is easy to maintain the testing setup.

For ffscrapr, I think the most important of these are testing the code for dependency changes and testing the code for API response changes. Internet connectivity seems fairly straightforward to write code for, and is worth doing at least once, but especially in a heavier-weight package like ffscrapr (which imports quite liberally from tidyverse packages like dplyr, purrr, tidyr etc), dependency testing is crucial to making sure the package continues to clean and process the data in the intended fashion. Also, fantasy platform APIs do change on a pretty frequent basis, adding and removing features nearly every year. It’s worth testing against the live APIs to make sure that ffscrapr continues to operate as intended and without errors.

Why not skip CRAN testing?
#

Many API packages skip all testing on CRAN - it’s a hassle to maintain, and the risk of being booted from CRAN can outweigh the effort of making sure tests don’t fail. I’m here to suggest otherwise! I think developers are missing out on one of CRAN’s most important benefits, which is being a massive dependency testing suite. Packages are tested on many different combinations of operating systems and R versions on a regular basis, and CRAN will also require your dependency packages to make sure that changes they make don’t break your package (and/or if they do, to give you advanced notice that it will break your package). This gives you time to make the appropriate changes to the package to accommodate the change!

Skipping these tests on CRAN is doing yourself a disservice, in my opinion, especially with the development of packages like httptest (which I use) and alternatives like vcr/webmockr - I think the extra work in making the package CRAN-testable (at least, to make sure that your R code processes the responses properly) is worth doing. These packages essentially create local copies of the data that the API would return from your http query, and redirect your request to read from the local files instead of letting your request go through to the API - allowing you to test while holding the API responses constant.

Storing mock data in a separate repository
#

In an ideal world, tests and files are small and can be stored alongside the test data, such that you don’t overrun CRAN’s limitations on compressed or installed package sizes (5MB, generally speaking). With ffscrapr, I was finding that responses from certain sites (cough, Fleaflicker, ESPN, cough) were enormous - they package a whole whack of player data within each call, such that their responses were coming back at 5-10MB each!

In consulting the available documentation, I came across this GitHub issue in the httptest repository, in which Neal Richardson describes some of his suggested approaches to solving this problem:

Creating a function to redact the responses on capture
Manually pruning the captured responses
Refactor larger functions so that they can be tested on smaller chunks of data
Lastly (and least-optimally, in Neal’s opinion) - creating a separate data package that holds the response cache

I took a stab at each of these approaches, but one by one I had to discard them:

API responses for fantasy football platforms are subject to change on a pretty regular basis - I’ve had to rebuild the captured responses several times since I started developing this package in August 2020, and corner cases are unlikely to appear in the first 5-10 responses of data
Manually pruning the captured responses is unfeasible for similar reasons as above, and is also infinitely more time-consuming!
I did try to refactor some of the larger/longer-running functions into smaller/more-testable units, especially since APIs frequently have options to limit and paginate. However, some of these requests were very difficult to configure in this fashion, and other functions that I wanted to test needed to aggregate large amounts of data. I wanted the tests to be easily evaluated via the naked eye.

Thus, I landed on creating a separate package of test data files as my best route forward. I briefly considered the idea of pushing the data package to CRAN - this would ensure that it would always be “accessible” to the CRAN package wherever accessed - but discarded the prospects of both getting an exemption on package size AND maintaining regular updates to the test data. Instead, I opted for a separate GitHub repository of test files, which I considered to be a relatively reliable place to store the data.

Of course, a few concerns popped up with this option, which needed handling:

I would need to check that I could download the data at the time of test/vignette-build, and then skip testing/evaluating if GitHub or the data were unavailable.
I would need to find a way to version-control and make sure that I didn’t accidentally delete test data that would cause CRAN testing to break - that would defeat the whole purpose of going through all the trouble of mocking the data!

I managed the latter concern with a GitHub release, which allowed me to version-lock the test-data to match the CRAN version. I’ll need to make sure to update the links within the package at every CRAN release, but I think this should be feasible!

2023 note I didn’t know of the existence of piggyback at the time of this writing but if I did I would have used it instead so as to not have to track another repository. It’s been buried on my project list, but I’ve wanted to make a PR to add a piggyback option to httptest for a while now.

Accessing the mock data
#

This needed some finagling, but I eventually settled on the following code in my testthat setup file:

suppressPackageStartupMessages({
 library(testthat)
 library(httptest)
 library(checkmate)
})

(Packages)

if (identical(Sys.getenv("MOCK_BYPASS"), "true")) with_mock_api <- force 

If a system environment variable named MOCK_BYPASS has value of true, override the definition of “with_mock_api” to be “force”, which stops the mocking behaviour and replaces it with normal evaluation.

download_mock <- !identical(Sys.getenv("MOCK_BYPASS"), "true") & !is.null(curl::nslookup("github.com", error = FALSE))

If not mock_bypass AND github.com is available (via a curl check), set download_mock to TRUE, otherwise set to FALSE. This checks as to whether the mock data should be downloaded at all.

skip <- FALSE

if (download_mock) {
 tryCatch(
 expr = {
 download.file("https://github.com/dynastyprocess/ffscrapr-tests/archive/1.3.0.zip", "f.zip")
 unzip("f.zip", exdir = ".")

 httptest::.mockPaths(new = "ffscrapr-tests-1.3.0")

 withr::defer(
 unlink(c("ffscrapr-tests-1.3.0", "f.zip"), recursive = TRUE, force = TRUE),
 testthat::teardown_env()
 )
 },
 warning = function(e) skip <<- TRUE,
 error = function(e) skip <<- TRUE
 )
}

With “skip” as default FALSE, try to download the data from the archive link (version controlled) and then unzip it. Then, set the .mockPaths to read this unzipped directory. When done, delete the zipped file and the test data. If anything generates a warning or error, set skip to TRUE.

skippy <- function() NULL
if (skip) skippy <- function() testthat::skip(message = "Unable to download test data")

With skippy defined by default as a function that does nothing, if skip is TRUE (because of the previous step), change the definition of skippy to skip all tests.

I think this addresses my concerns - if running in a non-live-situation AND mock data can be downloaded, do so, otherwise skip all tests! A similar setup and teardown happens in all the vignette RMDs as well.

Capturing the mock data
#

Capturing the mock data turned out to be a fairly straightforward process. I made sure to have the test-data folder in the same top-level directory as the package itself, and then capturing the mock data was comfortably done with:

library(httptest)
start_capturing("../ffscrapr-tests")
{run the tests}
stop_capturing()
{commit and push the test data}

I’ve had some problems with my PC’s locale settings encoding the JSON oddly in the past, but I usually get around that by jumping onto an available RStudio server instance on a Linux server and re-recording the tests there when necessary. (This is probably overkill - if you’re reading this and have better suggestions, let me know!)

Testing the real APIs with GitHub Actions
#

CRAN obviously cannot test the real APIs, because it cannot handle failures gracefully. Instead, I have a GitHub Action that does this by running the package tests with the MOCK_BYPASS environment variable set to “true”. Paired with a cron schedule of once a week, this successfully catches API changes and issues before users can find them, which helps me anticipate and start patching bugs more effectively.

DSLC Slack: Changing Window Titles in Shiny

Tue, 20 Oct 2020 00:00:00 +0000

`Dhiraj` asks:
#

I have a shiny application which uses the navbarPage() layout. The app has multiple tabPanels with their own titles. Is it possible to have the windowTitle of the navbarPage dynamic to show the titles of the tabPanels ?

Answer
#

Here’s a minimal example of the Shiny and JS code you’ll need:

library(shiny)
library(tidyverse)

custom_js <- 'Shiny.addCustomMessageHandler("handler_windowtitle", switch_window_title );
function switch_window_title(message){
 document.title = message;
}'

ui <- navbarPage(
 title = 'navbarpage',
 id = 'displayed_tab',
 tags$head(tags$script(custom_js)),
 tabPanel(title = "StrangePlanet"),
 tabPanel(title = "DarthVader")
)

server <- function(input, output, session) {
 observeEvent(
 input$displayed_tab,
 session$sendCustomMessage("handler_windowtitle",input$displayed_tab)
 )
}

shinyApp(ui, server)

If you give navbarPage an ID, it’ll create an input variable that can be accessed as input${id}, returning the value argument of the tabPanel. In this case, tabPanel returns the title argument as the default value - you can specify something else if you prefer.

In a production app, I’d also move the JS into a separate JS file (stored in www) and use includeJS() instead of using the tags$script method.

DSLC Slack: Explaining Shiny to Your IT Team

Thu, 15 Oct 2020 00:00:00 +0000

`Piranha` asks:
#

Hello, This might be bit of a stretch…. I could use a bit of help in prepping for some upcoming conversations with the head of my company’s IT/Security.

I am trying to get IT support for Shiny (currently IT supports Tableau).

Question: Is there any kind of high level summary of this Admin Guide document? https://docs.rstudio.com/shiny-server/

More specifically, I want to summarize the official document into about a 3-5 slide presentation. Topics that I want to include:

What is Shiny? (3-5 bullet points, in language that a “traditional” IT person will understand)

How to think about security and authentication for both the Shiny Server free and Pro.

How to think about Dockerizing an application.

I attempted to summarize the official docs into a 3-5 slide PowerPoint (ugh, yes, PowerPoint!), but am really struggling. This is a massive document and I am not an IT person!
Any pointers would be really helpful.
Thanks!

Answer (by me)
#

I don’t know if I have a document to point you at, but I would suggest a mental model where you separate Shiny the app (the app.R file and all of the things included in it) from the ways you deploy Shiny. Writing out some of my personal thoughts on it:

Shiny (the app):

is an R package that allows you to write UI (client-side) and server-side logic in R, which can be implemented quickly (from a code-writing perspective) and powerfully (leverages all of the modelling/data-wrangling packages and abilities of R).
has an architecture consisting of a client (the browser) and a server (an R process). The two are connected by a websocket that receives state changes from the client, such as new values for input controls, and distributes state changes from the server, such as new output values.

Shiny is not self-contained because it relies on R and other libraries (Shiny, as well as whatever else you use in the app) that are installed onto the server. Docker is one of the best ways to handle this, because it creates a container that holds an installation of R and all of the relevant packages, and can be version-locked to ensure stability. Once a Shiny app has been dockerized, it can be turned over to a devops team to deploy, and the devops team does not have to know R or Shiny in order to use the app.

This allows for a division of expertise:

the data scientist makes the app, and makes sure the docker container builds and works as intended
the devops engineer can take care of the details with deploying and securing the apps, by controlling/restricting access to the container itself rather than forcing the data scientist to worry about it inside the app.
Both the data scientist and the devops engineer should responsible for performance:
- data scientist by improving the code for efficiency
- engineer by supplying the containers with the right amount of processing power/memory/load-balancing
A dedicated devops person can pretty easily deploy a docker container into an existing app/microservices system, if such already exists.

At the other end of the spectrum, shinyapps.io and RS Connect are solutions that RStudio provides to help data teams who want this sort of devops process but don’t want to have the work done in house. They’ll manage the servers for you and make it very easy for R users to just upload all of the app data and configs, and they’ll manage deployment, performance and authentication for you.

I’m a data generalist for a small team, and manage the full stack from data import/wrangle/model to app dev to app deploy. I wouldn’t necessarily recommend that path if your team is large enough - let devops be devops and data scientists be data scientists where possible!

Some followup questions
#

In a discussion earlier this week, a person from IT asked me “I dont know R, can you tell me how Shiny connects to the user’s browser?” I didn’t know much about WebSocket technology at the time, so my response was: “Shiny comes with its own webserver, but a common practice is to use nginx webserver in front of the shiny app”.

Now that I know a bit more…would it be accurate to some something along these lines: “Shiny uses WebSocket technology to create a 2-way communication between the server and the browser. This allows the data scientist to leverage the full data-processing and analytical capabilities of R. In our current process, we need to do processing/analysis in one tool (e.g. SAS, R) and visualization/sharing in Tableau/Excel - creating lots of inefficiencies. Using Shiny, we can fine-tune and scale our analysis to exactly what is needed by the client and serve that via a Shiny dashboard.”

Furthermore, “From the data science team’s perspective, this is our ask. We need the ability to deliver interactive analysis to our clients at scale. Can you help us do that in a secure way? We’ll build the Shiny app and dockerize it. We need help from IT to provide secure access by clients”.

and response:

Shiny uses WebSocket technology to create a 2-way communication between the server and the browser

I cribbed that chunk from here: https://community.rstudio.com/t/is-there-any-equivalent-of-web-sockets-in-shiny-world/17908. If you’re hoping to communicate how Shiny works to an IT person, I would use the wording as described rather than paraphrasing it too much - receives state changes from client and distributes state changes from server is high-level enough for most IT people to understand - and you lose something by abstracting beyond that.

Using Shiny, we can fine-tune and scale our analysis to exactly what is needed by the client and serve that via a Shiny dashboard

I’d change this to say that it allows you to use the same R code that you use to process and generate the data right in the app, which allows you to package the “business logic” right into the app for immediate interactive analyses. I’m not familiar with your team and the ask - but identifying the docker container as the handoff point where IT helps deploy the app and serve it securely and effectively to clients is a good start. That lets IT do the security bit.

DSLC Slack: Things I Learned From Advanced R Book Club

Tue, 22 Sep 2020 00:00:00 +0000

I set out reading this book hoping to understand the metaprogramming chapter, and ended up learning a bunch of other stuff along the way!

Here’s a non-exhaustive list of things I learned from each of the sections of Advanced R:

I - Foundations:
#

I notice that I use the switch function a ton now, where previously I used if-else nests.
I use condition-handling to pass errors back to the user of my Shiny apps now.
I now know what an Environment is and where and why R would look for something. (This helps both in function-writing and debugging!)
I write a heck of a lot of functions (this ties into II: Functional Programming, of course)

II - Functional Programming
#

I map the crap out of everything now (if I didn’t before, it’s way worse now!), and have learned to use functions in a ton of different ways.
I now understand what a function factory is, and what to do if a function somehow returns me a function instead (as opposed to panicking about it!)
I now know about a series of modifiers to help adjust function behaviours (memoise, quietly, safely, possibly) - although I use tryCatch as much as I do these.

III - Object Oriented Programming
#

I wrote a package that uses S3! It’s cool to minimize the user interface for multiple platforms and the mental load accompanying.
I still lean to using the explicit “switch” function a lot instead of writing out a useMethod call. May be a habit/pattern worth breaking in the future.
I’m now comfortable USING an R6 package - I don’t know that I’d reach for an R6 system now but the whole “function$method” thing doesn’t scare me anymore
S4? wot that

IV Metaprogramming
#

I now know how to pass expressions to tidyverse stuff!
I now know where to look in the Metaprogramming and how to intepret the text.

V Techniques
#

Debugging: I now know how to use browser(), and love it! Also debug()!
Performance measure/improve: still don’t do a ton of this, even though I’m a shiny dev. I feel like I should care more but I mostly work with smaller data (sub 1,000,000 rows) right now.
I … still don’t know C++ or Rcpp. ¯\(ツ)/¯

Toy Code
#

library(dplyr)
library(rlang)

grouping_variables <- switch(
 params$group_type,
 "S" = exprs(CraftCode, starts_with("Product"), ModelCode, Elevation),
 "B" = exprs(starts_with("Rate")),
 "P" = exprs(starts_with("Part"))
)
 
null_bids <- bids %>%
 group_by(SupplierID, ProjectID, !!!grouping_variables) %>%
 slice_max(LastModDate) %>%
 ungroup() %>%
 select(!!!grouping_variables, ProjectID, SupplierAID, SupplierName, UnitPrice) %>%
 left_join(lowest_bids, by = as.character(grouping_variables))

Metaprogramming - how to pass expressions into select statements. Also example of switch.

R on TanHo.ca

Data Science Hangout

Recording #

DSLC Slack: Debugging Netlify Deploys

Question #

Answer #

The Beginner Data Science Programming Book That Doesn't Exist (Yet?)

Hot take: the ideal beginner data science programming book doesn’t exist (yet) #

the ideal book for learning programming and NFL analytics #

Adventures In Scaling Shiny: Lessons Learned Building DynastyProcess

Recording #

Materials #

tidyr pivot trick: names_to -> .value

Scenario #

Past me #

But what if I told you there’s a one-liner for this exact situation? #

Bonus: data.table equivalent #

Full Code #

Effective Logging Strategies

Recording #

Materials #

Debugging Shiny Apps

Recording #

Materials #

blogdown, distill, and personal websites with R

TLDR #

Distill #

Blogdown #

Which one do I choose? #

Setting Up Your Own Shiny Server

Recording #

Wrangling JSON and Lists

Recording #

Web Scraping with rvest

Recording #

Adventures in HTTP Testing for R

Why test? #

Why not skip CRAN testing? #

Storing mock data in a separate repository #

Accessing the mock data #

Capturing the mock data #

Testing the real APIs with GitHub Actions #

DSLC Slack: Changing Window Titles in Shiny

Dhiraj asks: #

Answer #

DSLC Slack: Explaining Shiny to Your IT Team

Piranha asks: #

Answer (by me) #

Some followup questions #

DSLC Slack: Things I Learned From Advanced R Book Club

I - Foundations: #

II - Functional Programming #

III - Object Oriented Programming #

IV Metaprogramming #

V Techniques #

Toy Code #

Recording
#

Question
#

Answer
#

Hot take: the ideal beginner data science programming book doesn’t exist (yet)
#

the ideal book for learning programming and NFL analytics
#

Recording
#

Materials
#

Scenario
#

Past me
#

But what if I told you there’s a one-liner for this exact situation?
#

Bonus: data.table equivalent
#

Full Code
#

Recording
#

Materials
#

Recording
#

Materials
#

TLDR
#

Distill
#

Blogdown
#

Which one do I choose?
#

Recording
#

Recording
#

Recording
#

Why test?
#

Why not skip CRAN testing?
#

Storing mock data in a separate repository
#

Accessing the mock data
#

Capturing the mock data
#

Testing the real APIs with GitHub Actions
#

`Dhiraj` asks:
#

Answer
#

`Piranha` asks:
#

Answer (by me)
#

Some followup questions
#

I - Foundations:
#

II - Functional Programming
#

III - Object Oriented Programming
#

IV Metaprogramming
#

V Techniques
#

Toy Code
#