Model validation

Model validation#

How well does a TEXAS calibration actually fit, and how do we know the generalized-logistic Bayesian model improves on existing TEX₈₆ calibrations? This page summarises the validation strategy used in the manuscript; the full analysis (with figures) lives in notebooks/manuscripts/SI_code2_TEXAS_analysis.ipynb.

Validation rests on three complementary checks:

  1. Forward calibration performance — how accurately the curve reproduces the observed proxy (RMSE / R²), benchmarked against existing calibrations.

  2. Residual diagnostics — whether any structure remains in the residuals (spatially, or against temperature) after fitting.

  3. Variance partitioning — how much of the proxy variance is thermal vs non-thermal (ecology, nutrients) vs irreducible noise.

1. Forward calibration performance#

The forward model is evaluated on the coretop training set (N ≈ 1513) by predicting Scaled RI from in-situ SST and comparing to the observed proxy. RMSE is reported in Scaled RI units (the proxy is normalised to ~[0, 1]), so it is directly comparable across calibrations.

Calibration

Non-thermal terms

RMSE (Scaled RI)

Linear

—

0.082

TEX₈₆ᴴ (Kim et al. 2010)

—

0.085

BAYSPAR (Tierney & Tingley 2014)

—

0.069

TEXAS

thermal only

0.066

TEXAS

+ GDGT-2/3 (β_{G₂/₃})

0.060

TEXAS

+ GDGT-2/3 + NO₃

0.059

Values as reported in the manuscript / AGU25 PP33D-1102. Reproduce them from SI_code2 (Fig. 7, Fig. 8, Fig. S11).

The thermal-only TEXAS model already beats the linear and TEX₈₆ᴴ calibrations, and adding the non-thermal corrections lowers RMSE further — without the regression dilution that affects linear T ~ TEX₈₆ fits.

You can reproduce a prediction-vs-observation comparison for any cached posterior with the public API:

import numpy as np
from TEXAS import predict_proxy_from_T

pred = predict_proxy_from_T(
    temperatures=coretop_df["SST"].values,
    posterior="gen_logi_fixed_hier_crtp_univ_priorApprox_SST_scaledRI_cren3",
)
rmse = np.sqrt(np.mean((coretop_df["scaledRI_cren3"] - pred["p50"]) ** 2))

2. Residual diagnostics#

Lower RMSE is necessary but not sufficient — a good calibration should also leave no systematic structure in its residuals. The manuscript examines residuals two ways:

  • Against temperature — residual-vs-SST plots (Fig. 7–8). Linear and TEX₈₆ᴴ show curvature at the temperature extremes (the proxy saturates); the generalized-logistic TEXAS curve removes that structure.

  • Spatially — kriged residual maps (Fig. 9, Fig. S12–S13), rendered with TEXAS.plotting.plot_residual_maps. Regional residual patterns (e.g. the Mediterranean and Red Sea) shrink once the GDGT-2/3 and NO₃ corrections are applied, indicating the non-thermal terms capture a real ecological signal rather than overfitting noise.

3. Variance partitioning#

To quantify what the proxy variance is made of, the coretop proxy variance is decomposed (Venn-style, after Peres-Neto et al. 2006) into thermal, non-thermal (GDGT-2/3 and NO₃), and residual components, using the posterior-mean process noise σ²_proxyObs from nested fits:

Component

Share of total proxy variance

Thermal (SST)

74.5 %

GDGT-2/3 (unique)

2.8 %

NO₃ (unique)

2.2 %

GDGT-2/3 ∩ NO₃ (shared)

0.4 %

Residual / irreducible noise

20.0 %

scaledRI_cren3, SST, N ≈ 1513 coretops. See SI_code2.

Temperature dominates (≈ 75 %), confirming Scaled RI is primarily a thermometer, while the non-thermal predictors each explain a small but robustly non-zero slice. The corresponding regression coefficients are resolved away from zero with essentially full posterior probability:

Coefficient

Posterior mean ± SD

P(β < 0)

β_{G₂/₃}

−0.0058 ± 0.0004

≈ 1.0

β_{NO₃}

−0.0329 ± 0.0025

≈ 1.0

Because the model is Bayesian, significance is reported as the posterior probability that each coefficient has the expected sign — not a frequentist p-value.

Note. Variance partitioning applies to the forward calibration only. An inverse (invT) reconstruction’s posterior width mixes the temperature prior, the likelihood, and propagated calibration uncertainty, and cannot be cleanly partitioned — use the invT σ uncertainty maps for that instead.

4. Inverse reconstruction performance#

For the inverse direction (proxy → temperature), TEXAS reconstructs coretop SST with an RMSE of ≈ 3.9 °C and R² ≈ 0.86, using a diffuse temperature prior (prior_mu_t = WOA23 SST, prior_sigma_t = 10 °C) so the reconstruction is driven by the data, not the prior (≈ 87 % likelihood weight). This makes the reported R² a meaningful measure of skill rather than an artefact of an informative prior.

See SI_code2 (T-residual plots, Fig. 9) and the paleo applications in SI_code3_paleo_showcases.ipynb for downcore reconstructions with full posterior uncertainty.

References#

  • Peres-Neto, P. R., Legendre, P., Dray, S., & Borcard, D. (2006). Variation partitioning of species data matrices. Ecology, 87(10), 2614–2625.

  • Kim, J.-H., et al. (2010). New indices for calibrating the relationship of archaeal lipids with sea surface temperature. GCA, 74, 4639–4654.

  • Tierney, J. E., & Tingley, M. P. (2014). A Bayesian, spatially-varying calibration model for the TEX₈₆ proxy. GCA, 127, 83–106.