An introduction to {shinyValidator}

2022-11-14

David Granjon, Novartis

Welcome

Workshop Material

Slides: https://rinpharma2022.rinterface.com/

Clone this repository with the RStudio IDE or via the command line.

git clone https://github.com/RinteRface/rinpharma2022.git
cd rinpharma2022

Then run renv::restore() to install the dependencies.

During the workshop day, a live sandbox platform is accessible at rstd.io/class. The id will be given in the zoom chat.

Pre-requisites

If you want to run {shinyValidator} locally (not on CI/CD), you must have:

shinycannon installed for the load-test part. See here.
A chrome browser installed like chromium.
git installed and a GitHub account.
A recent R version, if possible R>= 4.1.0.

This workshop is recorded. If it does not work for you, it is better to listen and stay on track. Fear not! You can retry later on your side with the recording.

Setup {shinyValidator}

{golem}

We add some useful files, basic test and link to git:

path <- file.path("<FOLDER>", "<PKG>") 
golem::create_golem(path)
usethis::use_mit_license() # or whatever license
usethis::use_testthat()
usethis::use_test("dummy")
usethis::use_git()

Put some real server code

Copy this into app_server.R:

output$distPlot <- renderPlot({
  hist(rnorm(input$obs))
})

Copy this into app_ui.R:

fluidPage(
  sliderInput(
    "obs",
    "Number of observations:",
    min = 0,
    max = 1000,
    value = 500
  ),
  plotOutput("distPlot")
)

{renv}

system("echo 'RENV_PATHS_LIBRARY_ROOT = ~/.renv/library' >> .Renviron")

# SCAN the project and look for dependencies
renv::init()
# install missing packages
renv::install("<PACKAGE>")
# Capture new dependencies after package installation
renv::snapshot()system("echo 'RENV_PATHS_LIBRARY_ROOT = ~/.renv/library' >> .Renviron")

# SCAN the project and look for dependencies
renv::init()
# install missing packages
renv::install("<PACKAGE>")
# Capture new dependencies after package installation
renv::snapshot()system("echo 'RENV_PATHS_LIBRARY_ROOT = ~/.renv/library' >> .Renviron")

# SCAN the project and look for dependencies
renv::init()
# install missing packages
renv::install("<PACKAGE>")
# Capture new dependencies after package installation
renv::snapshot()system("echo 'RENV_PATHS_LIBRARY_ROOT = ~/.renv/library' >> .Renviron")

# SCAN the project and look for dependencies
renv::init()
# install missing packages
renv::install("<PACKAGE>")
# Capture new dependencies after package installation
renv::snapshot()

Install {shinyValidator}

devtools::install_github("Novartis/shinyValidator")
library(shinyValidator)
# At the root of your R package
use_validator("github")
devtools::document() # update help
renv::snapshot()

Review the file structure

{shinyValidator}: step by step

Overall concept

%%{init: {'theme':'dark'}}%%
flowchart TD
  subgraph CICD
    direction TB
    subgraph DMC 
      direction LR
      E[Lint] --> F[Quality]
      F --> G[Performance]
    end
    subgraph POC 
      direction LR
      H[Lint] --> I[Quality]
    end
  end
  A(Shiny Project) --> B(DMC App)
  A --> C(Poof of concept App POC)
  B --> |strict| D[Expectations]
  C --> |low| D
  D --> CICD 
  CICD --> |create| J(Global HTML report)
  J --> |deploy| K(Deployment server)
  click A callback "Tooltip for a callback"
  click B callback "DMC: data monitoring committee"
  click D callback "Apps have different expectations"
  click E callback "Lint code: check code formatting, style, ..."
  click F callback "Run R CMD check + headless crash test (shinytest2)"
  click G callback "Optional tests: profiling, load test, ..."
  click J callback "HTML reports with multiple tabs"
  click K callback "RStudio Connect, GitLab/GitHub pages, ..."

Audit app

audit_app() is the main function ¹:

audit_app <- function(
  headless_actions = NULL,
  timeout = NULL,
  scope = c("manual", "DMC", "POC"),
  output_validation = FALSE,
  coverage = TRUE,
  load_testing = TRUE,
  profile_code = TRUE,
  check_reactivity = TRUE,
  ...
) {
  ###
}

headless actions: pass shinytest2 instructions.
timeout: wait app to start.
…: parameters to pass to run_app() such as database logins, …
scope: predefined set of parameters (see examples).

Audit app: example

audit_app(profile_code = FALSE, ...)

%%{init: {'theme':'dark'}}%%
graph TD
  A(Check) --> B(Crashtest)
  B --> C(Loadtest)
  C --> D(Coverage)
  D --> E(Reactivity)
  click A callback "devtools::check"
  click B callback "{shinytest2}"
  click C callback "{shinyloadtest}"
  click D callback "{covr}"
  click E callback "{reactlog}"

Audit app: headless manipulation (1/2)

# app refers to the headless app instance.
audit_app({
  app$set_inputs(obs = 30)
  app$get_screenshot("plop.png")
  # ... pass any other commands from shinytest2 API
})

shinytest2 hex logo.

This code is run during crash test, profiling and reactivity check.

Headless manipulation: your turn 🎮 (2/2)

Run the following code step by step¹:

# Start the app 
library(shinytest2)
headless_app <- AppDriver$new("./app.R")
# View the app for debugging (does not work from Workbench!)
headless_app$view()
headless_app$set_inputs(obs = 1)
headless_app$get_value(input = "obs")
# You can also run JS code!
headless_app$run_js(
  "$('#obs')
    .data('shiny-input-binding')
    .setValue(
      $('#obs'), 
      100
    );
  "
)
# Now you can call any function
# Close the connection before leaving 
headless_app$stop()# Start the app 
library(shinytest2)
headless_app <- AppDriver$new("./app.R")
# View the app for debugging (does not work from Workbench!)
headless_app$view()
headless_app$set_inputs(obs = 1)
headless_app$get_value(input = "obs")
# You can also run JS code!
headless_app$run_js(
  "$('#obs')
    .data('shiny-input-binding')
    .setValue(
      $('#obs'), 
      100
    );
  "
)
# Now you can call any function
# Close the connection before leaving 
headless_app$stop()# Start the app 
library(shinytest2)
headless_app <- AppDriver$new("./app.R")
# View the app for debugging (does not work from Workbench!)
headless_app$view()
headless_app$set_inputs(obs = 1)
headless_app$get_value(input = "obs")
# You can also run JS code!
headless_app$run_js(
  "$('#obs')
    .data('shiny-input-binding')
    .setValue(
      $('#obs'), 
      100
    );
  "
)
# Now you can call any function
# Close the connection before leaving 
headless_app$stop()# Start the app 
library(shinytest2)
headless_app <- AppDriver$new("./app.R")
# View the app for debugging (does not work from Workbench!)
headless_app$view()
headless_app$set_inputs(obs = 1)
headless_app$get_value(input = "obs")
# You can also run JS code!
headless_app$run_js(
  "$('#obs')
    .data('shiny-input-binding')
    .setValue(
      $('#obs'), 
      100
    );
  "
)
# Now you can call any function
# Close the connection before leaving 
headless_app$stop()# Start the app 
library(shinytest2)
headless_app <- AppDriver$new("./app.R")
# View the app for debugging (does not work from Workbench!)
headless_app$view()
headless_app$set_inputs(obs = 1)
headless_app$get_value(input = "obs")
# You can also run JS code!
headless_app$run_js(
  "$('#obs')
    .data('shiny-input-binding')
    .setValue(
      $('#obs'), 
      100
    );
  "
)
# Now you can call any function
# Close the connection before leaving 
headless_app$stop()# Start the app 
library(shinytest2)
headless_app <- AppDriver$new("./app.R")
# View the app for debugging (does not work from Workbench!)
headless_app$view()
headless_app$set_inputs(obs = 1)
headless_app$get_value(input = "obs")
# You can also run JS code!
headless_app$run_js(
  "$('#obs')
    .data('shiny-input-binding')
    .setValue(
      $('#obs'), 
      100
    );
  "
)
# Now you can call any function
# Close the connection before leaving 
headless_app$stop()

Headless test debugging tools.

About monkey testing (1/2)

run_crash_test() runs a gremlins.js test if no headless action are passed:

Video Player is loading.

Current Time 0:00

Duration 0:29

Loaded: 63.07%

Stream Type LIVE

Remaining Time 0:29

Improve

Your turn 🎮

Disable other checks

Modify shinyValidator::audit_app parameters:

shinyValidator::audit_app(
  load_testing = FALSE, 
  profile_code = FALSE, 
  check_reactivity = FALSE
)

For learning purposes we disable load test, profiling, ... at the moment ...

Add server testing

usethis::use_test("app-server-test")
# Inside app-server-test
testServer(app_server, {
  session$setInputs(obs = 0)
  # There should be an error
  expect_error(output$distPlot)
  session$setInputs(obs = 100)
  str(output$distPlot)
})
# Test it
devtools::test()usethis::use_test("app-server-test")
# Inside app-server-test
testServer(app_server, {
  session$setInputs(obs = 0)
  # There should be an error
  expect_error(output$distPlot)
  session$setInputs(obs = 100)
  str(output$distPlot)
})
# Test it
devtools::test()usethis::use_test("app-server-test")
# Inside app-server-test
testServer(app_server, {
  session$setInputs(obs = 0)
  # There should be an error
  expect_error(output$distPlot)
  session$setInputs(obs = 100)
  str(output$distPlot)
})
# Test it
devtools::test()usethis::use_test("app-server-test")
# Inside app-server-test
testServer(app_server, {
  session$setInputs(obs = 0)
  # There should be an error
  expect_error(output$distPlot)
  session$setInputs(obs = 100)
  str(output$distPlot)
})
# Test it
devtools::test()usethis::use_test("app-server-test")
# Inside app-server-test
testServer(app_server, {
  session$setInputs(obs = 0)
  # There should be an error
  expect_error(output$distPlot)
  session$setInputs(obs = 100)
  str(output$distPlot)
})
# Test it
devtools::test()usethis::use_test("app-server-test")
# Inside app-server-test
testServer(app_server, {
  session$setInputs(obs = 0)
  # There should be an error
  expect_error(output$distPlot)
  session$setInputs(obs = 100)
  str(output$distPlot)
})
# Test it
devtools::test()usethis::use_test("app-server-test")
# Inside app-server-test
testServer(app_server, {
  session$setInputs(obs = 0)
  # There should be an error
  expect_error(output$distPlot)
  session$setInputs(obs = 100)
  str(output$distPlot)
})
# Test it
devtools::test()

Server tests are run without the UI, inputs have to be changed manually with session.

Run shinyValidator::audit_app and have a look at the coverage tab.

Customize Crash test

Goal: UI feature testing. Is the app starting? Is the background dark? ...

Leverage shinytest2 power¹, app being the Shiny app to audit.

shinyValidator::audit_app(
  {
    app$set_inputs(obs = 1000)
    app$get_screenshot("plop.png")
  },
  load_testing = FALSE, 
  profile_code = FALSE, 
  check_reactivity = FALSE
)

Run the above code and have a look at the screenshots.

Output checks (1/3)

Goal: track if an output has changed after a commit...

Create this function in helpers.R:

make_hist <- function(val) {
  hist(rnorm(val))
}

Add it to app_server.R:

output$distPlot <- renderPlot({
  make_hist(input$obs)
})

Enable output check in shinyValidator::audit_app:

shinyValidator::audit_app(
  load_testing = FALSE, 
  profile_code = FALSE, 
  check_reactivity = FALSE, 
  output_validation = TRUE
)

Output checks (2/3)

usethis::use_test("test-base-plot")
renv::install("vdiffr")
# Inside test-base-plot
test_that("Base plot OK", {
  set.seed(42) # to avoid the test from failing due to randomness :)
  vdiffr::expect_doppelganger("Base graphics histogram", make_hist(500))
})
# Test it
devtools::test()usethis::use_test("test-base-plot")
renv::install("vdiffr")
# Inside test-base-plot
test_that("Base plot OK", {
  set.seed(42) # to avoid the test from failing due to randomness :)
  vdiffr::expect_doppelganger("Base graphics histogram", make_hist(500))
})
# Test it
devtools::test()usethis::use_test("test-base-plot")
renv::install("vdiffr")
# Inside test-base-plot
test_that("Base plot OK", {
  set.seed(42) # to avoid the test from failing due to randomness :)
  vdiffr::expect_doppelganger("Base graphics histogram", make_hist(500))
})
# Test it
devtools::test()usethis::use_test("test-base-plot")
renv::install("vdiffr")
# Inside test-base-plot
test_that("Base plot OK", {
  set.seed(42) # to avoid the test from failing due to randomness :)
  vdiffr::expect_doppelganger("Base graphics histogram", make_hist(500))
})
# Test it
devtools::test()usethis::use_test("test-base-plot")
renv::install("vdiffr")
# Inside test-base-plot
test_that("Base plot OK", {
  set.seed(42) # to avoid the test from failing due to randomness :)
  vdiffr::expect_doppelganger("Base graphics histogram", make_hist(500))
})
# Test it
devtools::test()

A svg snapshot is created during first run. If you change the plot, snapshots are compared.

Output checks (3/3)

We slightly modify make_hist():

make_hist <- function(val) {
  hist(rnorm(val * 2))
}

Run the test again:

devtools::test()

Failure may be reviewed with:

testthat::snapshot_review('basic-plot')

Run shinyValidator::audit_app and look at the outputs tab.

Performance: Code profiling (1/2)

Goal: track any performance bottleneck.

Add this¹ to helpers.R:

slow_func <- function(n) {
  vec <- NULL # Or vec = c()
  for (i in seq_len(n))
    vec <- c(vec, i)
  vec
}

Call it in app_server.R:

app_server <- function(input, output, session) {
  output$distPlot <- renderPlot({
    slow_func(5*10^4) # you may reduce if needed
    make_hist(input$obs)
  })
}

Performance: Code profiling (2/3)

Modify the custom headless script by adding a timeout:

shinyValidator::audit_app(
  {
    app$set_inputs(obs = 1000, timeout_ = 15 * 1000)
    app$get_screenshot("plop.png")
  },
  timeout = 15,
  load_testing = FALSE, 
  profile_code = TRUE, 
  check_reactivity = FALSE
)

Run shinyValidator::audit_app and have a look at the profiling tab.

Performance: Load testing (1/2)

Goal: Check if the app can support concurrent user sessions.

Load testing might not work well at the moment...

shinyValidator::audit_app(
  {
    app$set_inputs(obs = 1000, timeout_ = 15 * 1000)
    app$get_screenshot("plop.png")
  },
  timeout = 15,
  load_testing = TRUE, 
  profile_code = TRUE, 
  check_reactivity = FALSE
)

Let’s add CI/CD

{shinyValidator} CI/CD file

In case you need to control branches triggering {shinyValidator}:

on:
  push:
    branches: [main, master, <CUSTOM_BRANCH>]
  pull_request:
    branches: [main, master, <CUSTOM_BRANCH>]

name: shinyValidator

%%{init: {'theme':'dark'}}%%
gitGraph
  commit
  commit
  branch develop
  checkout develop
  commit
  commit
  checkout main
  merge develop
  commit
  commit id: "Normal" tag: "v1.0.0"

If you have to change the R version, os, …:

strategy:
  fail-fast: false
  matrix:
    config:
      - {os: ubuntu-latest,   r: 'devel', http-user-agent: 'release'}
      - {os: ubuntu-latest,   r: 'release'}
      - {os: ubuntu-latest,   r: 'oldrel-1'}

- name: Lint code
  shell: Rscript {0}
  run: shinyValidator::lint_code()

- name: Audit app 🏥
  shell: Rscript {0}
  run: shinyValidator::audit_app()

- name: Deploy to GitHub pages 🚀
  if: github.event_name != 'pull_request'
  uses: JamesIves/github-pages-deploy-action@4.1.4
  with:
    clean: false
    branch: gh-pages
    folder: public

Example: disable other checks

Modify GitHub actions yaml file:

- name: Audit app 🏥
  shell: Rscript {0}
  run: shinyValidator::audit_app(load_testing = FALSE, profile_code = FALSE, check_reactivity = FALSE)

An introduction to {shinyValidator} 2022-11-14 David Granjon, Novartis

An introduction to {shinyValidator}
Welcome
Hi, I am David Granjon
Program
Workshop Material
Pre-requisites
Introduction
Clothes don’t make the man
From prototype to production
Available tools
Automate: CI/CD
Stop … I am lost …
Welcome {shinyValidator}
Setup {shinyValidator}
{golem}
{golem}
Put some real server code
Create empty GitHub repo
Add remote repo to local
{renv}
{renv}
Install {shinyValidator}
{shinyValidator}: step by step
Overall concept
Audit app
Audit app: example
Audit app: using scope parameter
Audit app: headless manipulation (1/2)
Headless manipulation: your turn 🎮 (2/2)
About monkey testing (1/2)
About monkey testing (2/2)
Report example
Your turn 🎮
Pro tip
Improve
Disable other checks
Add server testing
Add server testing
Customize Crash test
Output checks (1/3)
Output checks (2/3)
Output checks (2/3)
Output checks (3/3)
Performance: Code profiling (1/2)
Performance: Code profiling (2/3)
Performance: Code profiling (3/3)
Performance: Load testing (1/2)
Performance: Load testing (2/2)
Let’s add CI/CD
Pipeline output
{shinyValidator} CI/CD file
Example: disable other checks
Run our first pipeline
What’s next?
Test with your own app
Be patient
Recommended reading
Mastering Shiny UI
Thank you!
Disclaimer