[florence-bockting]

Fixes #281
Fixes #223, #213, #201, #135, #220

Summary

This PR adds a predictive performance API. It computes density scores, scoring rules, and point-prediction metrics under four evaluation sources: in-sample, PSIS-LOO, k-fold, and holdout (test data).

• New entry points: insample_pred_measure(), loo_pred_measure(), kfold_pred_measure(), test_pred_measure(), and pred_measure().
• New built-in measures: measure_*(). supported_measures_list() lists them.
• measure_rps() and measure_srps() use one ypred draw matrix and a PWM estimator ((loo_)(s)crps could ask for only one argument with predictions #223).
• loo_pred_measure() accepts a pre-computed psis_object (loo_predictive_metric and loo_crps could accept psis objects to reduce computation #213).
• A user function or a named list of functions defines a custom measure (Add R2 #201, user defined loss/utility functions #135).
• model_compare() compares models on all measures (loo_compare for crps and loo_crps #220). It comes from Introduce model_compare() for model comparison with support of new predictive measures #380, which is merged into this branch.

elpd(), crps(), scrps(), loo_crps(), loo_scrps(), loo_predictive_metric(), and loo_compare() are deprecated. They still work with their current APIs.

The comment below lists all new features and functions. It marks the parts that come from #380.

How to navigate this PR

• notes/developer-notes.md: status, design decisions, numerical comparisons
• notes/design-discussions/model_compare.md: review decisions for model_compare() (Introduce model_compare() for model comparison with support of new predictive measures #380)
• vignettes/articles-online-only/overview-measures.Rmd: supported measures and their definitions
• vignettes/articles-online-only/pred-measure-workflow.Rmd: end-to-end workflow
• vignettes/articles-online-only/model-comparison.Rmd: Model comparison: Explanation of model_compare() (Introduce model_compare() for model comparison with support of new predictive measures #380)
• vignettes/migration-guide.Rmd: deprecated functions and their replacements

Deprecations (implementations retained)

Deprecated	Replacement
elpd()	measure_elpd() / *_pred_measure()
crps(), scrps() (x, x2)	measure_rps(), measure_srps() (ypred)
loo_crps(), loo_scrps()	loo_pred_measure(..., measure = "rps" | "srps")
loo_predictive_metric()	loo_pred_measure()
loo_compare() (#380)	model_compare()

Each deprecated function warns once per session.

Dependencies

Vignettes, test-data generation, and pkgdown CI install brms from GitHub master as k-fold support for categorical models (brms#1890) is not yet on CRAN. Remove the pin after the next brms CRAN release.

Known limitations

• group_ids grouping is not implemented.
• The formula derivations article is not finished.
• The brms GitHub pin remains.

Acknowledgements

This work builds on a prototype by @VisruthSK (VisruthSK/loo-sandbox) under guidance from @jgabry.

[github-actions]

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 4597a83 is merged into loo-v3.0.0:

Status	Name	Current	Incoming	Interval
✔️	loo_matrix	1.75s	1.75s	[-0.3%, +0.24%]
✔️	loo_function	1.65s	1.66s	[-0.31%, +0.91%]

Further explanation regarding interpretation and methodology can be found in the documentation.

[florence-bockting]

There is an open issue requesting a subset() method for psis objects #343 and an (unrelated) open PR #110. Might be worth considering whether we can address this as well in this refactoring PR. If yes, we should

• close PR Subsetting psis objects #110 with link to this PR and also
• [ ] mark issue Feature Request: public subset() method for psis objects #343 as fixed in the description text of this PR.

Moved this to PR #379

[florence-bockting]

Overview of changes

Items marked (#380) come from the merged integrate-loo_compare PR. #380 has the full details for those items in its overview comment.

User-facing functions

Function	Status	PR
insample_pred_measure(), loo_pred_measure(), kfold_pred_measure(), test_pred_measure()	new	#363
pred_measure()	new	#363
measure_elpd(), measure_mlpd(), measure_ic()	new	#363
measure_acc(), measure_bacc(), measure_brier()	new	#363
measure_mae(), measure_mse(), measure_rmse(), measure_r2()	new	#363
measure_rps(), measure_srps()	new	#363
ptw_log_pred_density()	new	#363
supported_measures_list()	new	#363
print.pred_measure(), print.loo_pred_measure(), print.measure(), dim.pred_measure()	new S3 methods	#363
elpd(), crps(), scrps(), loo_crps(), loo_scrps(), loo_predictive_metric()	deprecated	#363
model_compare() (methods default, psis_loo_ss_list)	new	#380
custom_measure()	new	#380
print.compare.loo()	new arguments measures, simplify	#380
loo_compare()	deprecated	#380
print.compare.loo_ss()	removed	#380

Internal functions

New in #363

• R/pred_measure.R (new file): S3 entry points only
• R/pred_measure-builtin.R (new file): .create_measure_structure()
• R/pred_measure-compute.R (new file): do_pred_measure(), .get_psis_object(),
  .measure_estimate_se(), .elpd_pointwise(), .compute_measure(),
  .effective_param(), .compute_effective_param(), .measure_result_name(),
  .merge_matrix(), .build_pred_measure(), .add_attributes()
• R/pred_measure-helpers.R (new file): .normalize_measure(), .measure_entry_custom(),
  .check_measure_entry_names(), .prepare_measures(), .any_needs_elpd(),
  .measure_n_obs(), .validate_measure_result(), .validate_numeric_matrix(),
  .validate_numeric_vector(), .normalize_and_validate_log_weights(),
  .inform_ignored_inputs(), .validate_probs(), .lppd_from_inputs(),
  .acc_pointwise(), .point_error_from_inputs(), .copy_attrs(),
  .normalize_log_weights(), .exx_pwm(), .validate_control(), subset_measures()
• R/print.R: .fr(), .pred_measure_source_label()
• R/helpers.R: .deprecate_once()
• R/elpd.R: .elpd_matrix_impl()

From #380

• R/model_compare.R, R/model_compare-pred_measure.R, R/model_compare-print.R,
  R/model_compare.psis_loo_ss_list.R: all functions (see the Introduce model_compare() for model comparison with support of new predictive measures #380 comment)
• R/pred_measure-builtin.R: .se_diff_rmse(), .se_diff_r2(), .se_diff_bacc(),
  .se_r2_delta(), .measure_info()
• R/pred_measure-compute.R: .detect_posthoc(), .warn_posthoc()
• R/print.R: .measure_digits(), .se_digits(), .resolve_digits(), .format_estimates()

[florence-bockting]

@jgabry just a quick checkin, whether you received the review notification.
But don't be pressured, I just wanted to ensure that this PR is not forgotten.

[jgabry]

Thanks for the reminder! I missed the first notification. It looks like there are a couple of merge conflicts, although they're files that don't affect my ability to review it.