Quality criteria (QC) computations

Computation of common quality criteria for evaluation of model predictions performance: standard qc (average fold error, maximal error), bias (mean predictions error), precision (root mean square error), Student's t-Test, correlation test and linear regression.

quality_criteria(run, predictions = "PRED", log_data = FALSE,
  alpha = 0.05, drop_empty_splits = FALSE)

Arguments

run	pmxploit NONMEM run object.
predictions	character. Name of the predictions column in the result tables. Default is "PRED" for population predictions.
alpha	numeric. Alpha risk. Used for bias confidence interval computation and T-test comparing observations and predictions. Default is 5%.
drop_empty_splits	logical. Drop empty split groups from QC results.

Value

A list containing the quality criteria analysis results.

Details

For quality criteria computations, residuals are computed based on the formula:

\(pred_err_i = pred_i - obs_i\)

• Standard QC

  • Maximal Error: $$ME=max(|obs-pred|)$$

  • Absolute Average Fold Error:

    $$AAFE=10^(mean(log10(pred/obs)))$$ For reference, see https://www.ncbi.nlm.nih.gov/pubmed/26696327

• Bias: Mean Prediction Error (MPE)

  • Absolute: mean(pred_err)

  • Confidence interval for a given alpha

  • Relative: mean(pred_err/obs)

• Precision: Root Mean Square Error (RMSE)

  • Absolute: Student's t-Test estimate of t.test((obs - pred)^2)

  • Confidence interval for a given alpha

  • Relative: rmse/mean(obs)

• Student's t-Test: observations vs predictions (paired, two-sided)

  Returns estimate, statistic, p-value, degrees of freedom (parameter) and confidence interval given alpha.

• Correlation test between observations and predictions

  Returns estimate, statistic, p-value, degrees of freedom (parameter) and confidence interval given alpha.

• Linear regression: \(pred = intercept + slope * obs\)

  Returns intercept and slope estimates, standard errors, statistics and p-values given alpha.

Examples

EXAMPLERUN %>% quality_criteria()
#> # A tibble: 1 x 9
#>   n_observations standard bias  precision t_test_obs t_test_res correlation_test
#>            <int> <list>   <lis> <list>    <list>     <list>     <list>          
#> 1           9391 <tibble… <tib… <tibble … <S3: htes… <tibble [… <S3: htest>     
#> # … with 2 more variables: linear_regression <list>, data <list>

EXAMPLERUN %>% quality_criteria(alpha = 0.01)
#> # A tibble: 1 x 9
#>   n_observations standard bias  precision t_test_obs t_test_res correlation_test
#>            <int> <list>   <lis> <list>    <list>     <list>     <list>          
#> 1           9391 <tibble… <tib… <tibble … <S3: htes… <tibble [… <S3: htest>     
#> # … with 2 more variables: linear_regression <list>, data <list>

EXAMPLERUN %>% group_by(SEX) %>% quality_criteria()
#> # A tibble: 2 x 10
#>   Gender n_observations standard bias  precision t_test_obs t_test_res
#>   <chr>           <int> <list>   <lis> <list>    <list>     <list>    
#> 1 Male             5178 <tibble… <tib… <tibble … <S3: htes… <tibble […
#> 2 Female           4213 <tibble… <tib… <tibble … <S3: htes… <tibble […
#> # … with 3 more variables: correlation_test <list>, linear_regression <list>,
#> #   data <list>

EXAMPLERUN %>% filter(STUD == 2) %>% quality_criteria()
#> # A tibble: 1 x 9
#>   n_observations standard bias  precision t_test_obs t_test_res correlation_test
#>            <int> <list>   <lis> <list>    <list>     <list>     <list>          
#> 1           1688 <tibble… <tib… <tibble … <S3: htes… <tibble [… <S3: htest>     
#> # … with 2 more variables: linear_regression <list>, data <list>

EXAMPLERUN %>% filter(STUD != 2) %>% quality_criteria()
#> # A tibble: 1 x 9
#>   n_observations standard bias  precision t_test_obs t_test_res correlation_test
#>            <int> <list>   <lis> <list>    <list>     <list>     <list>          
#> 1           7703 <tibble… <tib… <tibble … <S3: htes… <tibble [… <S3: htest>     
#> # … with 2 more variables: linear_regression <list>, data <list>

EXAMPLERUN %>% filter(AGE <= 24) %>% quality_criteria()
#> # A tibble: 1 x 9
#>   n_observations standard bias  precision t_test_obs t_test_res correlation_test
#>            <int> <list>   <lis> <list>    <list>     <list>     <list>          
#> 1            728 <tibble… <tib… <tibble … <S3: htes… <tibble [… <S3: htest>     
#> # … with 2 more variables: linear_regression <list>, data <list>