Variational Expectation Maximization (VEM)

Authors

Mohamed Tarek

Vijay Ivaturi

WarningExperimental Feature

VEM is an experimental estimation method introduced in Pumas 2.8. Its API may change in future releases without deprecation.

In this tutorial, we introduce Variational Expectation Maximization (VEM), a new estimation method available in Pumas.Experimental. We will cover the conceptual background, fit a warfarin PK model with VEM, compare the results to FOCE, and provide a comprehensive reference of VEM options.

using Pumas
using Pumas.Experimental: VEM
using PumasUtilities
using PharmaDatasets
using DataFramesMeta
using CairoMakie
using AlgebraOfGraphics
using SummaryTables
using Random: default_rng, seed!
seed!(123)
Random.TaskLocalRNG()

1 Conceptual Background

1.1 The Marginal Likelihood Challenge

In nonlinear mixed-effects (NLME) modeling, we want to estimate population parameters while accounting for between-subject variability through random effects. The marginal likelihood requires integrating over all possible random effect values for every subject, and these integrals are generally intractable.

Different estimation methods handle this challenge in different ways:

  • FOCE/LaplaceI approximate the integral using a second-order Taylor expansion around the mode of the random effects.
  • SAEM uses Markov Chain Monte Carlo (MCMC) sampling within an Expectation-Maximization framework.
  • VEM maximizes a lower bound on the log-likelihood using variational inference.

1.2 The Evidence Lower Bound (ELBO)

VEM works by maximizing the Evidence Lower Bound (ELBO), which is a lower bound on the log marginal likelihood. The ELBO equals the log-likelihood minus a KL divergence term that measures how far the approximate posterior is from the true posterior. A higher ELBO typically means a better fit.

Intuitively, VEM simultaneously finds:

  1. Population parameters (fixed effects, variance components, residual error), and
  2. An approximation to each subject’s random effect posterior distribution.

This joint optimization is performed in a single level, unlike FOCE/LaplaceI which alternates between inner (random effects) and outer (population parameters) optimization loops.

One key advantage of VEM is that it can handle any response data distribution, including discrete, censored, truncated, time-to-event and even custom distributions. FOCE by comparison is limited to a restricted set of distributions, and while LaplaceI can handle any response distribution, it can be very slow for complex models.

1.3 Presampling vs. Resampling

VEM has two optimization variants controlled by the presample keyword:

Presampling (presample=true, default) Resampling (presample=false)
Optimizer L-BFGS (deterministic) Adam (stochastic)
Phases 1 (default) 10 (default)
Samples Pre-drawn at start of phase, reused across iterations Re-drawn at every iteration

In the resampling variant, the number of samples increases (by default, unless overridden) and the learning rate of Adam decreases from one phase to the next.

1.4 Diagonal vs. Full Posterior Covariance

VEM approximates each subject’s random effect posterior with a multivariate normal distribution. The diagonal keyword controls the structure of this approximation:

  • diagonal=true (default): assumes independent random effects in the posterior. Faster, fewer parameters to optimize.
  • diagonal=false: captures correlations between random effects in the posterior. Slower but may be more accurate when random effects are correlated.

2 Warfarin PKPD Model with VEM

We will fit an oral PKPD model to the warfarin dataset, first using VEM and then comparing with FOCE.

2.1 Model Definition

warfarin_model = @model begin
    @param begin
        # PK
        pop_CL ∈ RealDomain(lower = 0.0)
        pop_V ∈ RealDomain(lower = 0.0)
        pop_tabs ∈ RealDomain(lower = 0.0)
        pop_lag ∈ RealDomain(lower = 0.0, upper = 1.0)
        pk_Ω ∈ PSDDomain(3)
        σ_prop ∈ RealDomain(lower = 0.0)
        σ_add ∈ RealDomain(lower = 0.0)
        # PD
        pop_e0 ∈ RealDomain(lower = 0.0)
        pop_emax ∈ RealDomain()
        pop_c50 ∈ RealDomain(lower = 0.0)
        pop_tover ∈ RealDomain(lower = 0.0)
        pd_Ω ∈ PSDDomain(3)
        σ_fx ∈ RealDomain(lower = 0.0)
    end

    @random begin
        pk_η ~ MvNormal(pk_Ω)
        pd_η ~ MvNormal(pd_Ω)
    end

    @covariates begin
        FSZV
        FSZCL
    end

    @pre begin
        # PK
        CL = FSZCL * pop_CL * exp(pk_η[1])
        Vc = FSZV * pop_V * exp(pk_η[2])
        tabs = pop_tabs * exp(pk_η[3])
        Ka = log(2) / tabs
        # PD
        e0 = pop_e0 * exp(pd_η[1])
        emax = pop_emax
        c50 = pop_c50 * exp(pd_η[2])
        tover = pop_tover * exp(pd_η[3])
        kout = log(2) / tover
        rin = e0 * kout
    end

    @dosecontrol begin
        lags = (Depot = pop_lag,)
    end

    @init begin
        Turnover = e0
    end

    @vars begin
        cp := Central / Vc
        ratein := Ka * Depot
        pd := 1 + emax * cp / (c50 + cp)
    end

    @dynamics begin
        Depot' = -ratein
        Central' = ratein - CL * cp
        Turnover' = rin * pd - kout * Turnover
    end

    @derived begin
        conc ~ @. CombinedNormal(cp, σ_add, σ_prop)
        pca ~ @. Normal(Turnover, σ_fx)
    end
end
PumasModel
  Parameters: pop_CL, pop_V, pop_tabs, pop_lag, pk_Ω, σ_prop, σ_add, pop_e0, pop_emax, pop_c50, pop_tover, pd_Ω, σ_fx
  Random effects: pk_η, pd_η
  Covariates: FSZV, FSZCL
  Dynamical system variables: Depot, Central, Turnover
  Dynamical system type: Nonlinear ODE
  Derived: conc, pca
  Observed: conc, pca
TipSame Model for VEM and FOCE

The @model definition is identical regardless of whether you fit with VEM or FOCE/LaplaceI. VEM works with standard @model definitions – no special model macro is required.

NoteRandom effect on emax

Note that no random effect is used on emax. We tried adding a random effect on emax and estimating the model parameters. However, that resulted in the row and column of pd_Ω corresponding to that random effect to be all close to 0 which caused numerical issues for the asymptotic esimtator of the confidence interval. So no random effect is used on emax here for simplicity.

2.2 Starting Values

The model definition above gives each parameter’s domain but no values. Those are kept separately and passed to fit as the third argument.

warfarin_init = (;
    # PK
    pop_CL = 0.15,
    pop_V = 10.0,
    pop_tabs = 0.5,
    pop_lag = 0.5,
    pk_Ω = Diagonal([0.1, 0.1, 0.1]),
    σ_prop = 0.1,
    σ_add = 0.5,
    # PD
    pop_e0 = 100.0,
    pop_emax = -1.0,
    pop_c50 = 1.0,
    pop_tover = 14.0,
    pd_Ω = Diagonal([0.01, 0.01, 0.01]),
    σ_fx = 0.01,
)
(pop_CL = 0.15, pop_V = 10.0, pop_tabs = 0.5, pop_lag = 0.5, pk_Ω = Diagonal([0.1, 0.1, 0.1]), σ_prop = 0.1, σ_add = 0.5, pop_e0 = 100.0, pop_emax = -1.0, pop_c50 = 1.0, pop_tover = 14.0, pd_Ω = Diagonal([0.01, 0.01, 0.01]), σ_fx = 0.01)

These are starting values, not estimates. They are chosen to be plausible for warfarin and of the right order of magnitude, without being read off a previous analysis of these data. Published analyses of the warfarin data of (O’Reilly et al. 1963; O’Reilly and Aggeler 1968) report a clearance near 0.134 L/h/70kg and a central volume near 8.11 L/70kg (Holford 2010), which is the neighbourhood the fits below should reach.

2.3 Data Loading

warfarin_data = @chain dataset("pumas/warfarin_pumas") begin
    @rtransform begin
        :FSZV = :wtbl / 70
        :FSZCL = (:wtbl / 70)^0.75
    end
    # Concentrations of exactly zero are below the limit of quantification
    @rtransform @passmissing :conc = iszero(:conc) ? missing : :conc
end
first(warfarin_data, 5)
5×12 DataFrame
Row id time evid amt cmt conc pca wtbl age sex FSZV FSZCL
Int64 Float64 Int64 Float64? Int64? Float64? Float64? Float64 Int64 String1 Float64 Float64
1 1 0.0 1 100.0 1 missing missing 66.7 50 M 0.952857 0.96443
2 1 0.5 0 missing missing missing missing 66.7 50 M 0.952857 0.96443
3 1 1.0 0 missing missing 1.9 missing 66.7 50 M 0.952857 0.96443
4 1 2.0 0 missing missing 3.3 missing 66.7 50 M 0.952857 0.96443
5 1 3.0 0 missing missing 6.6 missing 66.7 50 M 0.952857 0.96443
pop = read_pumas(
    warfarin_data;
    id = :id,
    time = :time,
    amt = :amt,
    cmt = :cmt,
    evid = :evid,
    covariates = [:FSZV, :FSZCL],
    observations = [:conc, :pca],
)
Population
  Subjects: 32
  Covariates: FSZV, FSZCL
  Observations: conc, pca

2.4 Fitting with Default VEM

Fitting with VEM follows the same fit interface as all other estimation methods. Simply pass VEM() as the estimation algorithm:

@time fpm_vem =
    fit(warfarin_model, pop, warfarin_init, VEM(); optim_options = (; show_every = 20))
[ Info: Checking the initial parameter values.
[ Info: The initial negative log-likelihood and its gradient are finite. Check passed.
Iter     Function value   Gradient norm 
     0     1.370837e+08     2.741671e+08
 * time: 1.7881393432617188e-5
    20     1.564026e+03     6.122502e+02
 * time: 12.843366861343384
    40     1.089103e+03     1.694481e+02
 * time: 25.48479199409485
    60     9.628931e+02     8.742054e+01
 * time: 37.98119783401489
    80     9.229682e+02     3.339154e+01
 * time: 51.08288097381592
   100     8.854265e+02     2.973702e+01
 * time: 65.68621802330017
   120     8.543415e+02     3.583795e+01
 * time: 80.04285788536072
   140     7.954205e+02     1.365657e+01
 * time: 96.64965200424194
   160     7.506251e+02     1.367600e+01
 * time: 112.97022795677185
   180     7.305785e+02     7.946752e+00
 * time: 128.41657781600952
   200     7.227177e+02     9.669863e+00
 * time: 144.1189248561859
   220     7.192623e+02     1.050782e+01
 * time: 160.97357082366943
   240     7.179981e+02     3.724778e+00
 * time: 177.12614798545837
   260     7.175690e+02     2.301297e+00
 * time: 192.53336191177368
202.409365 seconds (216.35 M allocations: 49.708 GiB, 8.24% gc time, 1.65% compilation time)
FittedPumasModel

Dynamical system type:               Nonlinear ODE
Solver(s): (OrdinaryDiffEqVerner.Vern7,OrdinaryDiffEqRosenbrock.Rodas5P)

Number of subjects:                             32

Observation records:         Active        Missing
    conc:                       247             51
    pca:                        232             66
    Total:                      479            117

Number of parameters:      Constant      Optimized
                                  0             23

Estimation method:              VEM (EXPERIMENTAL)

-----------------------
            Estimate
-----------------------
pop_CL       0.13453
pop_V        7.9978
pop_tabs     0.39644
pop_lag      0.88715
pk_Ω₁,₁      0.066617
pk_Ω₂,₁      0.0087813
pk_Ω₃,₁      0.03865
pk_Ω₂,₂      0.020336
pk_Ω₃,₂     -0.026128
pk_Ω₃,₃      0.98922
σ_prop       0.086936
σ_add        0.40221
pop_e0      96.334
pop_emax    -1.07
pop_c50      1.5399
pop_tover   14.123
pd_Ω₁,₁      0.0030506
pd_Ω₂,₁     -0.0035378
pd_Ω₃,₁      0.0012884
pd_Ω₂,₂      0.15121
pd_Ω₃,₂     -0.033438
pd_Ω₃,₃      0.013114
σ_fx         3.8386
-----------------------

During fitting, VEM displays a convergence trace showing the negative ELBO (the objective being minimized). The objective value is the negative of the lower bound on the log marginal likelihood, so lower values indicate a better fit.

2.5 Extracting Results

Population parameter estimates can be extracted with coef:

simple_table(coeftable(fpm_vem))
parameter description constant estimate
pop_CL pop_CL false 0.135
pop_V pop_V false 8
pop_tabs pop_tabs false 0.396
pop_lag pop_lag false 0.887
pk_Ω₁,₁ pk_Ω₁,₁ false 0.0666
pk_Ω₂,₁ pk_Ω₂,₁ false 0.00878
pk_Ω₃,₁ pk_Ω₃,₁ false 0.0387
pk_Ω₂,₂ pk_Ω₂,₂ false 0.0203
pk_Ω₃,₂ pk_Ω₃,₂ false -0.0261
pk_Ω₃,₃ pk_Ω₃,₃ false 0.989
σ_prop σ_prop false 0.0869
σ_add σ_add false 0.402
pop_e0 pop_e0 false 96.3
pop_emax pop_emax false -1.07
pop_c50 pop_c50 false 1.54
pop_tover pop_tover false 14.1
pd_Ω₁,₁ pd_Ω₁,₁ false 0.00305
pd_Ω₂,₁ pd_Ω₂,₁ false -0.00354
pd_Ω₃,₁ pd_Ω₃,₁ false 0.00129
pd_Ω₂,₂ pd_Ω₂,₂ false 0.151
pd_Ω₃,₂ pd_Ω₃,₂ false -0.0334
pd_Ω₃,₃ pd_Ω₃,₃ false 0.0131
σ_fx σ_fx false 3.84

The posterior approximation modes of the random effects for each subject are available via empirical_bayes:

ebes = empirical_bayes(fpm_vem)
first(ebes, 3)
3-element Vector{@NamedTuple{pk_η::Vector{Float64}, pd_η::Vector{Float64}}}:
 (pk_η = [0.7633979507167724, 0.06262002422414326, 1.5556607656380148], pd_η = [-0.043885040897370386, 0.024512735415743975, 0.047430651899558286])
 (pk_η = [0.028963819307562175, 0.05556058347354534, 0.49741869058327526], pd_η = [0.041023290535329354, 0.3523518822722693, -0.09258365993614884])
 (pk_η = [-0.2969995728624792, -0.03351332562511915, -0.38595164249093505], pd_η = [0.012830929683162167, -0.6692621174224556, 0.08296529831994724])

Note that these generally may not correspond to modes of the true posterior.

2.6 Log-Likelihood Computation

VEM does not compute the log-likelihood during fitting. To evaluate the log-likelihood after fitting, you must specify an approximation method:

ll_foce = loglikelihood(fpm_vem, FOCE())
-1062.7904574118725
WarningLog-Likelihood Approximation Starting Points

When using FOCE() (or LaplaceI()) for log-likelihood evaluation, the empirical Bayes estimates (EBEs) are computed for each subject starting from the mode of the prior (i.e., zero random effects in the Gaussian case) rather than the VEM posterior approximation’s mode.

For well-behaved models, this is unlikely to cause a significant difference in the log-likelihood approximation. However, this behavior will likely change in a future release to use the VEM posterior approximation’s mode as the starting point for the Laplace approximation.

2.7 Diagnostics

Model diagnostics are available through inspect. For models with Gaussian error models (like our proportional normal error), you must explicitly specify how to compute weighted residuals using wres_approx:

res = inspect(fpm_vem; wres_approx = FOCE())
[ Info: Calculating predictions.
[ Info: Calculating weighted residuals.
[ Info: Calculating empirical bayes.
[ Info: Evaluating dose control parameters.
[ Info: Evaluating individual parameters.
[ Info: Done.
FittedPumasModelInspection

Likelihood approximation used for weighted residuals: FOCE
NoteWeighted Residuals Linearization Point

When computing the weights of the weighted residuals, the linearization of the mean and variance are performed at the mode of the approximate posterior for each subject, learned by VEM, rather than the mode of the true posterior.

The predictions used for the residuals are also computed using the mode of the VEM posterior approximation, which may differ from the EBEs or the modes of the true posterior.

Goodness-of-fit plots work as other estimation methods in Pumas:

goodness_of_fit(res, observations = [:conc])

goodness_of_fit(res, observations = [:pca])

Warningwres_approx Required for Gaussian Errors

When using VEM with Gaussian-like error models (Normal, ProportionalNormal, etc.), you must pass wres_approx = FOCE() or wres_approx = LaplaceI() to inspect. Omitting it will raise an error. For other error models (Beta, Gamma, Poisson, Bernoulli, TimeToEvent), this is not needed.

2.8 Predictions and VPCs

Predictions and visual predictive checks work as other estimation methods in Pumas:

preds = predict(fpm_vem)
first(preds, 3)
Prediction
  Subjects: 3
  Predictions: conc, pca
  Covariates: FSZV, FSZCL
NoteVEM Posterior Approximation’s Mode in Predictions

The individual predictions are computed using the mode of the VEM posterior approximation for each subject, which may differ from the EBEs or the modes of the true posterior.

conc_vpc_res = vpc(fpm_vem, observations = [:conc]);
vpc_plot(conc_vpc_res)

pca_vpc_res = vpc(fpm_vem, observations = [:pca]);
vpc_plot(pca_vpc_res)

2.9 Inference Workaround

VEM does not currently support infer for computing standard errors and confidence intervals:

infer(fpm_vem)
ArgumentError: Asymptotic inference is not supported for the estimation method VEM (EXPERIMENTAL).
Stacktrace:
 [1] infer(fpm::Pumas.FittedPumasModel{Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Vector{Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}}, Pumas.Experimental.VEMResult{Pumas.Experimental.ThreadedVEMObj{true, false, true, true, Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Vector{Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}}, @NamedTuple{pop_CL::Float64, pop_V::Float64, pop_tabs::Float64, pop_lag::Float64, pk_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_prop::Float64, σ_add::Float64, pop_e0::Float64, pop_emax::Float64, pop_c50::Float64, pop_tover::Float64, pd_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_fx::Float64}, Random.TaskLocalRNG, Nothing, Nothing, @NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, Vector{Array{Float64, 3}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::TransformVariables.Constant{Nothing}, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::TransformVariables.Constant{Nothing}, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, DifferentiationInterfaceForwardDiffExt.ForwardDiffGradientPrep{Tuple{Pumas._L_param{false, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}}, ADTypes.AutoForwardDiff{nothing, Nothing}, SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, Tuple{}}, ForwardDiff.GradientConfig{ForwardDiff.Tag{Pumas._L_param{false, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}}, Float64}, Float64, 12, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Pumas._L_param{false, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}}, Float64}, Float64, 12}}}, Tuple{}}, Vector{DifferentiationInterfaceReverseDiffExt.ReverseDiffGradientPrep{Tuple{typeof(Pumas.Experimental.G1_prior), ADTypes.AutoReverseDiff{false}, Vector{Float64}, Tuple{DifferentiationInterface.Constant{Pumas.Experimental._L_rfx2{true, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::TransformVariables.Constant{Nothing}, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::TransformVariables.Constant{Nothing}, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}, @NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, Matrix{Float64}, @NamedTuple{pop_CL::Float64, pop_V::Float64, pop_tabs::Float64, pop_lag::Float64, pk_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_prop::Float64, σ_add::Float64, pop_e0::Float64, pop_emax::Float64, pop_c50::Float64, pop_tover::Float64, pd_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_fx::Float64}}}}}, ReverseDiff.GradientConfig{ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}, Nothing}}, Vector{DifferentiationInterfaceForwardDiffExt.ForwardDiffGradientPrep{Tuple{typeof(Pumas.Experimental.G1_likelihood), ADTypes.AutoForwardDiff{nothing, Nothing}, Vector{Float64}, Tuple{DifferentiationInterface.Constant{Pumas.Experimental._L_rfx2{true, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::TransformVariables.Constant{Nothing}, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::TransformVariables.Constant{Nothing}, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}, @NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, Matrix{Float64}, @NamedTuple{pop_CL::Float64, pop_V::Float64, pop_tabs::Float64, pop_lag::Float64, pk_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_prop::Float64, σ_add::Float64, pop_e0::Float64, pop_emax::Float64, pop_c50::Float64, pop_tover::Float64, pd_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_fx::Float64}}}}}, ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Pumas.Experimental.G1_likelihood), Float64}, Float64, 9, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Pumas.Experimental.G1_likelihood), Float64}, Float64, 9}}}, Tuple{Nothing}}}, Vector{DifferentiationInterfaceReverseDiffExt.ReverseDiffTwoArgJacobianPrep{Tuple{typeof(Pumas.Experimental.get_randeffs!), SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, ADTypes.AutoReverseDiff{false}, Vector{Float64}, Tuple{DifferentiationInterface.Constant{Pumas.Experimental._L_rfx2{true, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::TransformVariables.Constant{Nothing}, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::TransformVariables.Constant{Nothing}, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}, @NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, Matrix{Float64}, @NamedTuple{pop_CL::Float64, pop_V::Float64, pop_tabs::Float64, pop_lag::Float64, pk_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_prop::Float64, σ_add::Float64, pop_e0::Float64, pop_emax::Float64, pop_c50::Float64, pop_tover::Float64, pd_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_fx::Float64}}}}}, ReverseDiff.JacobianConfig{ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, Vector{ReverseDiff.TrackedReal{Float64, Float64, Nothing}}}, Nothing}}, ADTypes.AutoForwardDiff{nothing, Nothing}, ADTypes.AutoReverseDiff{false}, ADTypes.AutoForwardDiff{nothing, Nothing}, Vector{Matrix{Float64}}}, Optim.MultivariateOptimizationResults{Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.BackTracking{Float64, Int64}, Returns{Nothing}}, Vector{Float64}, Float64, Float64, Vector{Optim.OptimizationState{Float64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.BackTracking{Float64, Int64}, Returns{Nothing}}}}, @NamedTuple{x_converged::Bool, f_converged::Bool, g_converged::Bool, f_limit_reached::Bool, g_limit_reached::Bool, h_limit_reached::Bool, time_limit::Bool, callback::Bool, f_increased::Bool, ls_failed::Bool, iterations::Bool, small_trustregion_radius::Bool}}, Vector{Float64}}, Pumas.Experimental.VEM{ADTypes.AutoForwardDiff{nothing, Nothing}}, Vector{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}}, @NamedTuple{diffeq_options::@NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, ensemblealg::SciMLBase.EnsembleThreads, constantcoef::Tuple{}}, Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.OptimState{NLSolversBase.OnceDifferentiable{Float64, Vector{Float64}, Float64, Vector{Float64}}, Optim.LBFGSState{Vector{Float64}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Float64, Vector{Float64}}, Optim.Options{Float64, Nothing}}}; level::Float64, rethrow_error::Bool, ensemblealg::SciMLBase.EnsembleThreads, sandwich_estimator::Bool)
   @ Pumas.Experimental ~/run/_work/PumasTutorials.jl/PumasTutorials.jl/custom_julia_depot/packages/Pumas/btqOJ/src/estimation/vem.jl:2588
 [2] infer(fpm::Pumas.FittedPumasModel{Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Vector{Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}}, Pumas.Experimental.VEMResult{Pumas.Experimental.ThreadedVEMObj{true, false, true, true, Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Vector{Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}}, @NamedTuple{pop_CL::Float64, pop_V::Float64, pop_tabs::Float64, pop_lag::Float64, pk_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_prop::Float64, σ_add::Float64, pop_e0::Float64, pop_emax::Float64, pop_c50::Float64, pop_tover::Float64, pd_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_fx::Float64}, Random.TaskLocalRNG, Nothing, Nothing, @NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, Vector{Array{Float64, 3}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::TransformVariables.Constant{Nothing}, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::TransformVariables.Constant{Nothing}, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, DifferentiationInterfaceForwardDiffExt.ForwardDiffGradientPrep{Tuple{Pumas._L_param{false, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}}, ADTypes.AutoForwardDiff{nothing, Nothing}, SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, Tuple{}}, ForwardDiff.GradientConfig{ForwardDiff.Tag{Pumas._L_param{false, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}}, Float64}, Float64, 12, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Pumas._L_param{false, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}}, Float64}, Float64, 12}}}, Tuple{}}, Vector{DifferentiationInterfaceReverseDiffExt.ReverseDiffGradientPrep{Tuple{typeof(Pumas.Experimental.G1_prior), ADTypes.AutoReverseDiff{false}, Vector{Float64}, Tuple{DifferentiationInterface.Constant{Pumas.Experimental._L_rfx2{true, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::TransformVariables.Constant{Nothing}, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::TransformVariables.Constant{Nothing}, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}, @NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, Matrix{Float64}, @NamedTuple{pop_CL::Float64, pop_V::Float64, pop_tabs::Float64, pop_lag::Float64, pk_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_prop::Float64, σ_add::Float64, pop_e0::Float64, pop_emax::Float64, pop_c50::Float64, pop_tover::Float64, pd_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_fx::Float64}}}}}, ReverseDiff.GradientConfig{ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}}, Nothing}}, Vector{DifferentiationInterfaceForwardDiffExt.ForwardDiffGradientPrep{Tuple{typeof(Pumas.Experimental.G1_likelihood), ADTypes.AutoForwardDiff{nothing, Nothing}, Vector{Float64}, Tuple{DifferentiationInterface.Constant{Pumas.Experimental._L_rfx2{true, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::TransformVariables.Constant{Nothing}, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::TransformVariables.Constant{Nothing}, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}, @NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, Matrix{Float64}, @NamedTuple{pop_CL::Float64, pop_V::Float64, pop_tabs::Float64, pop_lag::Float64, pk_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_prop::Float64, σ_add::Float64, pop_e0::Float64, pop_emax::Float64, pop_c50::Float64, pop_tover::Float64, pd_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_fx::Float64}}}}}, ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(Pumas.Experimental.G1_likelihood), Float64}, Float64, 9, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(Pumas.Experimental.G1_likelihood), Float64}, Float64, 9}}}, Tuple{Nothing}}}, Vector{DifferentiationInterfaceReverseDiffExt.ReverseDiffTwoArgJacobianPrep{Tuple{typeof(Pumas.Experimental.get_randeffs!), SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, ADTypes.AutoReverseDiff{false}, Vector{Float64}, Tuple{DifferentiationInterface.Constant{Pumas.Experimental._L_rfx2{true, (:pop_CL, :pop_V, :pop_tabs, :pop_lag, :pk_Ω, :σ_prop, :σ_add, :pop_e0, :pop_emax, :pop_c50, :pop_tover, :pd_Ω, :σ_fx), TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::Pumas.PSDTransform, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::Pumas.PSDTransform, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, TransformVariables.TransformTuple{@NamedTuple{pop_CL::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_V::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tabs::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_lag::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVScale{Float64}, TransformVariables.TVLogistic}}, pk_Ω::TransformVariables.Constant{Nothing}, σ_prop::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, σ_add::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_e0::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_emax::TransformVariables.Identity, pop_c50::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pop_tover::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}, pd_Ω::TransformVariables.Constant{Nothing}, σ_fx::TransformVariables.CompositeScalarTransform{Tuple{TransformVariables.TVShift{Float64}, TransformVariables.TVExp}}}}, Pumas.PumasModel{(pop_CL = 1, pop_V = 1, pop_tabs = 1, pop_lag = 1, pk_Ω = 6, σ_prop = 1, σ_add = 1, pop_e0 = 1, pop_emax = 1, pop_c50 = 1, pop_tover = 1, pd_Ω = 6, σ_fx = 1), 6, (:Depot, :Central, :Turnover), (:CL, :Vc, :tabs, :Ka, :e0, :emax, :c50, :tover, :kout, :rin), Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.ParamSet{@NamedTuple{pk_η::Pumas.RandomEffect{false, Main.Notebook.var"#13#14"}, pd_η::Pumas.RandomEffect{false, Main.Notebook.var"#15#16"}}}, Pumas.PreObj{Pumas.TimeDispatcher{Main.Notebook.var"#17#18", Main.Notebook.var"#19#20"}}, Pumas.DCPObj{Main.Notebook.var"#25#26"}, Main.Notebook.var"#27#28", SciMLBase.ODEProblem{Nothing, Tuple{Nothing, Nothing}, false, Nothing, SciMLBase.ODEFunction{false, SciMLBase.FullSpecialize, ModelingToolkitBase.GeneratedFunctionWrapper{(2, 3, false), Main.Notebook.var"#29#30", Main.Notebook.var"#31#32"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ModelingToolkitBase.ObservedFunctionCache{ModelingToolkitBase.System, Nothing}, Nothing, ModelingToolkitBase.System, Nothing, Nothing}, @Kwargs{}, SciMLBase.StandardODEProblem}, Pumas.DerivedObj{(:conc, :pca), (:Ka, :emax, :Vc, :c50), (:Central, :Depot, :Turnover), (), Main.Notebook.var"#33#34"}, Main.Notebook.var"#35#36", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Pumas.Subject{@NamedTuple{conc::Vector{Union{Missing, Float64}}, pca::Vector{Union{Missing, Float64}}}, Pumas.ConstantCovar{@NamedTuple{FSZV::Float64, FSZCL::Float64}}, Pumas.DosageRegimen{Float64, Union{Missing, Int64}, Float64, Float64, Float64, Float64}, Vector{Float64}}, @NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, Matrix{Float64}, @NamedTuple{pop_CL::Float64, pop_V::Float64, pop_tabs::Float64, pop_lag::Float64, pk_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_prop::Float64, σ_add::Float64, pop_e0::Float64, pop_emax::Float64, pop_c50::Float64, pop_tover::Float64, pd_Ω::PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}, σ_fx::Float64}}}}}, ReverseDiff.JacobianConfig{ReverseDiff.TrackedArray{Float64, Float64, 1, Vector{Float64}, Vector{Float64}}, Vector{ReverseDiff.TrackedReal{Float64, Float64, Nothing}}}, Nothing}}, ADTypes.AutoForwardDiff{nothing, Nothing}, ADTypes.AutoReverseDiff{false}, ADTypes.AutoForwardDiff{nothing, Nothing}, Vector{Matrix{Float64}}}, Optim.MultivariateOptimizationResults{Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.BackTracking{Float64, Int64}, Returns{Nothing}}, Vector{Float64}, Float64, Float64, Vector{Optim.OptimizationState{Float64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.BackTracking{Float64, Int64}, Returns{Nothing}}}}, @NamedTuple{x_converged::Bool, f_converged::Bool, g_converged::Bool, f_limit_reached::Bool, g_limit_reached::Bool, h_limit_reached::Bool, time_limit::Bool, callback::Bool, f_increased::Bool, ls_failed::Bool, iterations::Bool, small_trustregion_radius::Bool}}, Vector{Float64}}, Pumas.Experimental.VEM{ADTypes.AutoForwardDiff{nothing, Nothing}}, Vector{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}}, @NamedTuple{diffeq_options::@NamedTuple{ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::OrdinaryDiffEqCore.CompositeAlgorithm{0, Tuple{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{OrdinaryDiffEqVerner.Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, OrdinaryDiffEqRosenbrock.Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{LinearAlgebra.RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, ensemblealg::SciMLBase.EnsembleThreads, constantcoef::Tuple{}}, Pumas.ParamSet{@NamedTuple{pop_CL::Pumas.RealDomain{Float64, Nothing, Float64}, pop_V::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tabs::Pumas.RealDomain{Float64, Nothing, Float64}, pop_lag::Pumas.RealDomain{Float64, Float64, Float64}, pk_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_prop::Pumas.RealDomain{Float64, Nothing, Float64}, σ_add::Pumas.RealDomain{Float64, Nothing, Float64}, pop_e0::Pumas.RealDomain{Float64, Nothing, Float64}, pop_emax::Pumas.RealDomain{Nothing, Nothing, Float64}, pop_c50::Pumas.RealDomain{Float64, Nothing, Float64}, pop_tover::Pumas.RealDomain{Float64, Nothing, Float64}, pd_Ω::Pumas.PSDDomain{PDMats.PDMat{Float64, LinearAlgebra.Symmetric{Float64, Matrix{Float64}}, LinearAlgebra.Cholesky{Float64, Matrix{Float64}}}}, σ_fx::Pumas.RealDomain{Float64, Nothing, Float64}}}, Pumas.OptimState{NLSolversBase.OnceDifferentiable{Float64, Vector{Float64}, Float64, Vector{Float64}}, Optim.LBFGSState{Vector{Float64}, Vector{Vector{Float64}}, Vector{Vector{Float64}}, Float64, Vector{Float64}}, Optim.Options{Float64, Nothing}}})
   @ Pumas.Experimental ~/run/_work/PumasTutorials.jl/PumasTutorials.jl/custom_julia_depot/packages/Pumas/btqOJ/src/estimation/vem.jl:2581
 [3] top-level scope
   @ ~/run/_work/PumasTutorials.jl/PumasTutorials.jl/tutorials/estimation/01-vem-introduction.qmd:436

The recommended workaround is to use the VEM estimates as a starting point for a FOCE/LaplaceI fit, then run inference on the FOCE/LaplaceI result. This typically converges very quickly since VEM provides excellent initial values:

@time fpm_foce_from_vem = fit(
    warfarin_model,
    pop,
    coef(fpm_vem),
    FOCE();
    init_randeffs = empirical_bayes(fpm_vem),
    optim_options = (; show_every = 20),
)
Iter     Function value   Gradient norm 
     0     1.062790e+03     1.792981e+01
 * time: 9.107589721679688e-5
    20     1.060478e+03     5.650826e-02
 * time: 35.07386016845703
 41.588715 seconds (18.25 M allocations: 6.916 GiB, 4.93% gc time, 2.01% compilation time)
FittedPumasModel

Dynamical system type:               Nonlinear ODE
Solver(s): (OrdinaryDiffEqVerner.Vern7,OrdinaryDiffEqRosenbrock.Rodas5P)

Number of subjects:                             32

Observation records:         Active        Missing
    conc:                       247             51
    pca:                        232             66
    Total:                      479            117

Number of parameters:      Constant      Optimized
                                  0             23

Likelihood approximation:                     FOCE
Likelihood optimizer:                         BFGS

Termination Reason:                   GradientNorm
Log-likelihood value:                   -1060.4784

-----------------------
            Estimate
-----------------------
pop_CL       0.13502
pop_V        8.0302
pop_tabs     0.53613
pop_lag      0.87911
pk_Ω₁,₁      0.068383
pk_Ω₂,₁      0.0078911
pk_Ω₃,₁      0.032845
pk_Ω₂,₂      0.021011
pk_Ω₃,₂     -0.0037261
pk_Ω₃,₃      0.84876
σ_prop       0.086852
σ_add        0.43139
pop_e0      96.362
pop_emax    -1.0874
pop_c50      1.629
pop_tover   14.421
pd_Ω₁,₁      0.0028558
pd_Ω₂,₁      0.0017457
pd_Ω₃,₁      0.0005316
pd_Ω₂,₂      0.14308
pd_Ω₃,₂     -0.03598
pd_Ω₃,₃      0.017667
σ_fx         3.5643
-----------------------
loglikelihood(fpm_foce_from_vem)
-1060.4784281340214
infer(fpm_foce_from_vem)
[ Info: Calculating: variance-covariance matrix.
[ Info: Done.
Asymptotic inference results using sandwich estimator

Dynamical system type:               Nonlinear ODE
Solver(s): (OrdinaryDiffEqVerner.Vern7,OrdinaryDiffEqRosenbrock.Rodas5P)

Number of subjects:                             32

Observation records:         Active        Missing
    conc:                       247             51
    pca:                        232             66
    Total:                      479            117

Number of parameters:      Constant      Optimized
                                  0             23

Likelihood approximation:                     FOCE
Likelihood optimizer:                         BFGS

Termination Reason:                   GradientNorm
Log-likelihood value:                   -1060.4784

----------------------------------------------------------------
            Estimate     SE          95.0% C.I.
----------------------------------------------------------------
pop_CL       0.13502     0.0066431   [  0.122     ;  0.14804  ]
pop_V        8.0302      0.21531     [  7.6082    ;  8.4522   ]
pop_tabs     0.53613     0.17456     [  0.194     ;  0.87826  ]
pop_lag      0.87911     0.053534    [  0.77419   ;  0.98404  ]
pk_Ω₁,₁      0.068383    0.024782    [  0.01981   ;  0.11696  ]
pk_Ω₂,₁      0.0078911   0.0072278   [ -0.0062751 ;  0.022057 ]
pk_Ω₃,₁      0.032845    0.055674    [ -0.076274  ;  0.14196  ]
pk_Ω₂,₂      0.021011    0.0063455   [  0.008574  ;  0.033448 ]
pk_Ω₃,₂     -0.0037261   0.059504    [ -0.12035   ;  0.1129   ]
pk_Ω₃,₃      0.84876     0.3884      [  0.087499  ;  1.61     ]
σ_prop       0.086852    0.013681    [  0.060037  ;  0.11367  ]
σ_add        0.43139     0.072076    [  0.29013   ;  0.57266  ]
pop_e0      96.362       1.1767      [ 94.055     ; 98.668    ]
pop_emax    -1.0874      0.045932    [ -1.1775    ; -0.9974   ]
pop_c50      1.629       0.32997     [  0.98228   ;  2.2757   ]
pop_tover   14.421       0.76654     [ 12.918     ; 15.923    ]
pd_Ω₁,₁      0.0028558   0.0010026   [  0.00089064;  0.0048209]
pd_Ω₂,₁      0.0017457   0.0037382   [ -0.005581  ;  0.0090724]
pd_Ω₃,₁      0.0005316   0.0022352   [ -0.0038494 ;  0.0049126]
pd_Ω₂,₂      0.14308     0.043295    [  0.05822   ;  0.22793  ]
pd_Ω₃,₂     -0.03598     0.014257    [ -0.063923  ; -0.0080377]
pd_Ω₃,₃      0.017667    0.0092047   [ -0.00037388;  0.035708 ]
σ_fx         3.5643      0.30609     [  2.9644    ;  4.1643   ]
----------------------------------------------------------------
NoteTrue Posterior Mode vs Posterior Approximation Mode

Initializing a FOCE fit with VEM estimates and running for 0 or more iterations is also a way to get a fitted Pumas model with standard EBEs (true posterior mode, instead of the VEM posterior approximation’s mode) that can be used for downstream functions, such as predict, inspect and loglikelihood.

TipVEM as an Initializer

Using VEM to initialize FOCE is a powerful workflow. VEM provides good population parameter estimates and per-subject random effect posteriors, which together allow FOCE to converge quickly and reliably.

3 Comparing VEM and FOCE

3.1 Direct FOCE Fit

Let us also fit the model directly with FOCE from the starting values above for comparison:

@time fpm_foce_direct =
    fit(warfarin_model, pop, warfarin_init, FOCE(); optim_options = (; show_every = 20))
Iter     Function value   Gradient norm 
     0     5.000726e+06     9.991374e+06
 * time: 0.00011396408081054688
    20     1.712871e+03     2.628079e+02
 * time: 29.26493000984192
    40     1.226181e+03     1.772134e+02
 * time: 54.97340703010559
    60     1.189379e+03     8.230921e+01
 * time: 80.97688698768616
    80     1.162811e+03     8.317623e+01
 * time: 106.51585507392883
   100     1.144415e+03     3.683253e+01
 * time: 133.15930199623108
   120     1.134098e+03     2.054584e+01
 * time: 157.86574697494507
   140     1.125687e+03     5.096888e+01
 * time: 183.51366090774536
   160     1.118357e+03     3.044155e+01
 * time: 208.34813690185547
   180     1.111687e+03     3.780514e+01
 * time: 232.02583193778992
   200     1.109201e+03     1.197017e+01
 * time: 257.68543696403503
   220     1.108590e+03     1.388418e+00
 * time: 286.6743609905243
   240     1.108491e+03     1.280312e+00
 * time: 311.20018100738525
   260     1.108438e+03     7.018229e-01
 * time: 334.74742698669434
   280     1.108414e+03     2.499001e-01
 * time: 357.8505289554596
   300     1.108407e+03     5.740849e-02
 * time: 379.0789279937744
   320     1.108407e+03     2.046286e-02
 * time: 400.0554609298706
   340     1.108407e+03     7.750814e-03
 * time: 420.53369402885437
441.745753 seconds (132.26 M allocations: 65.558 GiB, 5.60% gc time, 0.04% compilation time)
FittedPumasModel

Dynamical system type:               Nonlinear ODE
Solver(s): (OrdinaryDiffEqVerner.Vern7,OrdinaryDiffEqRosenbrock.Rodas5P)

Number of subjects:                             32

Observation records:         Active        Missing
    conc:                       247             51
    pca:                        232             66
    Total:                      479            117

Number of parameters:      Constant      Optimized
                                  0             23

Likelihood approximation:                     FOCE
Likelihood optimizer:                         BFGS

Termination Reason:                   GradientNorm
Log-likelihood value:                    -1108.407

------------------------
            Estimate
------------------------
pop_CL       0.13506
pop_V        7.1456
pop_tabs     0.61907
pop_lag      0.87233
pk_Ω₁,₁      0.067995
pk_Ω₂,₁      0.028048
pk_Ω₃,₁      0.010207
pk_Ω₂,₂      0.46439
pk_Ω₃,₂     -0.52773
pk_Ω₃,₃      1.4127
σ_prop       0.088233
σ_add        0.44546
pop_e0      96.345
pop_emax    -1.0865
pop_c50      1.6247
pop_tover   14.406
pd_Ω₁,₁      0.0028373
pd_Ω₂,₁      0.0016626
pd_Ω₃,₁      0.00055343
pd_Ω₂,₂      0.13759
pd_Ω₃,₂     -0.033347
pd_Ω₃,₃      0.016697
σ_fx         3.5228
------------------------
loglikelihood(fpm_foce_direct)
-1108.4070366184428

3.2 Compare Estimates

compare_estimates(; VEM = fpm_vem, VEM_FOCE = fpm_foce_from_vem, FOCE = fpm_foce_direct)
23×4 DataFrame
Row parameter VEM VEM_FOCE FOCE
String Float64? Float64? Float64?
1 pop_CL 0.13453 0.135019 0.135055
2 pop_V 7.99776 8.03022 7.14558
3 pop_tabs 0.396436 0.536134 0.619069
4 pop_lag 0.88715 0.879113 0.872328
5 pk_Ω₁,₁ 0.0666168 0.0683831 0.0679948
6 pk_Ω₂,₁ 0.0087813 0.00789109 0.0280481
7 pk_Ω₃,₁ 0.0386503 0.032845 0.0102071
8 pk_Ω₂,₂ 0.0203362 0.0210109 0.464393
9 pk_Ω₃,₂ -0.0261285 -0.00372611 -0.527727
10 pk_Ω₃,₃ 0.98922 0.848756 1.41271
11 σ_prop 0.0869357 0.0868518 0.0882328
12 σ_add 0.40221 0.431393 0.445463
13 pop_e0 96.334 96.3617 96.3449
14 pop_emax -1.07 -1.08743 -1.08655
15 pop_c50 1.53989 1.62901 1.62474
16 pop_tover 14.1233 14.4208 14.406
17 pd_Ω₁,₁ 0.00305062 0.00285578 0.0028373
18 pd_Ω₂,₁ -0.00353783 0.0017457 0.00166261
19 pd_Ω₃,₁ 0.00128838 0.000531597 0.000553428
20 pd_Ω₂,₂ 0.151206 0.143078 0.137592
21 pd_Ω₃,₂ -0.0334376 -0.0359802 -0.0333467
22 pd_Ω₃,₃ 0.0131138 0.017667 0.0166972
23 σ_fx 3.83855 3.56433 3.52277

3.3 Compare Log-Likelihoods

DataFrame(
    Method = [
        "Deterministic VEM",
        "FOCE initialized from deterministic VEM",
        "FOCE direct",
    ],
    LogLikelihood = [
        loglikelihood(fpm_vem, FOCE()),
        loglikelihood(fpm_foce_from_vem),
        loglikelihood(fpm_foce_direct),
    ],
) |> simple_table
Method LogLikelihood
Deterministic VEM -1063
FOCE initialized from deterministic VEM -1060
FOCE direct -1108

4 VEM Options Reference

This section provides a comprehensive reference for all VEM keyword arguments. The options that control the algorithm itself are keyword arguments to VEM(). The optimizer (optim_alg, optim_options), the random number generator (rng), the parallelism (ensemblealg), the ODE solver options (diffeq_options), and the parameters held constant (constantcoef) are keyword arguments to fit, as they are for the other estimation methods. Each section below states which of the two it documents.

4.1 Presampling vs. Resampling (presample)

The presample keyword controls which VEM variant is used. Many other defaults change depending on this choice:

Option Presampling (presample=true, default) Resampling (presample=false)
Optimizer optim_alg (to fit) L-BFGS (deterministic) Adam (stochastic)
Phases nphases 1 10
Initial samples init_nsamples 15 3
Sample increment nsamples_increment half the initial samples (7 by default) the initial samples (3 by default)
Max samples max_nsamples Unlimited 25

Presampling draws samples once per phase and reuses them across iterations, which allows a deterministic optimizer (L-BFGS) to be used. Resampling draws fresh samples at every iteration, which requires a stochastic optimizer (Adam) but can be more appropriate for DeepPumas models with neural networks. However, generally the stochastic optimizer needs many more iterations to converge than the deterministic one, so presampling is a good default for most models.

Note that all of the above options can be overridden regardless of the presample setting.

# Resampling VEM
@time fpm_vem_resample = fit(
    warfarin_model,
    pop,
    warfarin_init,
    VEM(; presample = false, nphases = 5);
    optim_options = (; iterations = 1000, show_every = 20),
)
[ Info:           phase             iter       obj      ∇obj     s∇obj  
[ Info:           1 / 5         1 / 1000   1.4e+08   3.6e+08   1.0e+00
[ Info:           1 / 5        20 / 1000   2.7e+06   1.4e+07   4.0e-02
[ Info:           1 / 5        40 / 1000   8.9e+05   3.7e+06   1.0e-02
[ Info:           1 / 5        50 / 1000   6.9e+05   5.7e+06   1.6e-02
[ Info: Terminating phase due to sgtol.
[ Info:           2 / 5         1 / 1000   6.5e+05   4.2e+06   1.0e+00
[ Info:           2 / 5        20 / 1000   2.6e+05   1.6e+06   3.9e-01
[ Info:           2 / 5        40 / 1000   1.4e+05   4.0e+05   9.7e-02
[ Info:           2 / 5        60 / 1000   8.8e+04   2.0e+05   4.8e-02
[ Info:           2 / 5        68 / 1000   7.5e+04   1.5e+05   3.6e-02
[ Info: Terminating phase due to sgtol.
[ Info:           3 / 5         1 / 1000   7.6e+04   1.6e+05   1.0e+00
[ Info:           3 / 5        20 / 1000   6.4e+04   1.7e+05   1.1e+00
[ Info:           3 / 5        40 / 1000   5.4e+04   1.4e+05   8.8e-01
[ Info:           3 / 5        60 / 1000   4.6e+04   9.5e+04   6.1e-01
[ Info:           3 / 5        80 / 1000   4.1e+04   8.2e+04   5.3e-01
[ Info:           3 / 5       100 / 1000   3.6e+04   7.9e+04   5.1e-01
[ Info:           3 / 5       120 / 1000   3.2e+04   6.8e+04   4.3e-01
[ Info:           3 / 5       140 / 1000   2.9e+04   6.0e+04   3.8e-01
[ Info:           3 / 5       160 / 1000   2.7e+04   5.7e+04   3.7e-01
[ Info:           3 / 5       180 / 1000   2.4e+04   4.9e+04   3.1e-01
[ Info:           3 / 5       200 / 1000   2.2e+04   5.8e+04   3.7e-01
[ Info:           3 / 5       220 / 1000   2.0e+04   4.3e+04   2.7e-01
[ Info:           3 / 5       240 / 1000   1.9e+04   4.3e+04   2.8e-01
[ Info:           3 / 5       260 / 1000   1.8e+04   3.6e+04   2.3e-01
[ Info:           3 / 5       280 / 1000   1.7e+04   4.1e+04   2.6e-01
[ Info:           3 / 5       300 / 1000   1.6e+04   3.2e+04   2.0e-01
[ Info:           3 / 5       320 / 1000   1.5e+04   2.9e+04   1.9e-01
[ Info:           3 / 5       340 / 1000   1.4e+04   2.8e+04   1.8e-01
[ Info:           3 / 5       360 / 1000   1.3e+04   3.8e+04   2.4e-01
[ Info:           3 / 5       380 / 1000   1.2e+04   4.1e+04   2.6e-01
[ Info:           3 / 5       400 / 1000   1.2e+04   2.4e+04   1.6e-01
[ Info:           3 / 5       420 / 1000   1.1e+04   2.3e+04   1.5e-01
[ Info:           3 / 5       440 / 1000   1.1e+04   2.0e+04   1.3e-01
[ Info:           3 / 5       460 / 1000   1.0e+04   1.9e+04   1.2e-01
[ Info:           3 / 5       480 / 1000   9.9e+03   1.9e+04   1.2e-01
[ Info:           3 / 5       500 / 1000   9.3e+03   1.7e+04   1.1e-01
[ Info:           3 / 5       520 / 1000   9.0e+03   1.7e+04   1.1e-01
[ Info:           3 / 5       540 / 1000   8.6e+03   1.6e+04   1.0e-01
[ Info:           3 / 5       560 / 1000   8.2e+03   1.6e+04   1.1e-01
[ Info:           3 / 5       580 / 1000   7.9e+03   1.9e+04   1.2e-01
[ Info:           3 / 5       600 / 1000   7.6e+03   1.4e+04   9.1e-02
[ Info:           3 / 5       620 / 1000   7.4e+03   1.3e+04   8.5e-02
[ Info:           3 / 5       640 / 1000   7.1e+03   1.4e+04   8.7e-02
[ Info:           3 / 5       660 / 1000   7.0e+03   1.4e+04   9.1e-02
[ Info:           3 / 5       680 / 1000   6.7e+03   1.2e+04   7.5e-02
[ Info:           3 / 5       700 / 1000   6.5e+03   1.2e+04   7.8e-02
[ Info:           3 / 5       720 / 1000   6.3e+03   1.3e+04   8.3e-02
[ Info:           3 / 5       740 / 1000   6.1e+03   1.1e+04   6.9e-02
[ Info:           3 / 5       760 / 1000   5.9e+03   1.2e+04   7.8e-02
[ Info:           3 / 5       780 / 1000   5.8e+03   9.8e+03   6.3e-02
[ Info:           3 / 5       800 / 1000   5.6e+03   1.1e+04   6.9e-02
[ Info:           3 / 5       820 / 1000   5.4e+03   9.7e+03   6.2e-02
[ Info:           3 / 5       840 / 1000   5.3e+03   8.8e+03   5.6e-02
[ Info:           3 / 5       860 / 1000   5.2e+03   1.1e+04   7.2e-02
[ Info:           3 / 5       880 / 1000   5.0e+03   8.6e+03   5.5e-02
[ Info:           3 / 5       900 / 1000   4.9e+03   8.2e+03   5.2e-02
[ Info:           3 / 5       920 / 1000   4.8e+03   1.1e+04   7.0e-02
[ Info:           3 / 5       940 / 1000   4.7e+03   7.4e+03   4.8e-02
[ Info:           3 / 5       960 / 1000   4.6e+03   7.5e+03   4.8e-02
[ Info:           3 / 5       980 / 1000   4.4e+03   8.2e+03   5.3e-02
[ Info:           3 / 5      1000 / 1000   4.4e+03   7.5e+03   4.8e-02
[ Info:           4 / 5         1 / 1000   4.4e+03   7.9e+03   1.0e+00
[ Info:           4 / 5        20 / 1000   4.3e+03   7.7e+03   9.7e-01
[ Info:           4 / 5        40 / 1000   4.2e+03   7.1e+03   9.0e-01
[ Info:           4 / 5        60 / 1000   4.1e+03   6.8e+03   8.7e-01
[ Info:           4 / 5        80 / 1000   4.0e+03   6.2e+03   7.9e-01
[ Info:           4 / 5       100 / 1000   4.0e+03   6.2e+03   7.9e-01
[ Info:           4 / 5       120 / 1000   3.9e+03   5.8e+03   7.4e-01
[ Info:           4 / 5       140 / 1000   3.8e+03   5.8e+03   7.4e-01
[ Info:           4 / 5       160 / 1000   3.8e+03   7.3e+03   9.3e-01
[ Info:           4 / 5       180 / 1000   3.7e+03   5.5e+03   7.0e-01
[ Info:           4 / 5       200 / 1000   3.6e+03   5.5e+03   7.0e-01
[ Info:           4 / 5       220 / 1000   3.6e+03   5.4e+03   6.9e-01
[ Info:           4 / 5       240 / 1000   3.5e+03   5.2e+03   6.6e-01
[ Info:           4 / 5       260 / 1000   3.5e+03   6.0e+03   7.6e-01
[ Info:           4 / 5       280 / 1000   3.4e+03   5.0e+03   6.4e-01
[ Info:           4 / 5       300 / 1000   3.4e+03   6.3e+03   8.0e-01
[ Info:           4 / 5       320 / 1000   3.3e+03   5.5e+03   7.0e-01
[ Info:           4 / 5       340 / 1000   3.3e+03   5.5e+03   6.9e-01
[ Info:           4 / 5       360 / 1000   3.2e+03   4.9e+03   6.2e-01
[ Info:           4 / 5       380 / 1000   3.2e+03   4.8e+03   6.1e-01
[ Info:           4 / 5       400 / 1000   3.1e+03   5.2e+03   6.6e-01
[ Info:           4 / 5       420 / 1000   3.1e+03   4.5e+03   5.7e-01
[ Info:           4 / 5       440 / 1000   3.1e+03   4.1e+03   5.3e-01
[ Info:           4 / 5       460 / 1000   3.0e+03   4.1e+03   5.2e-01
[ Info:           4 / 5       480 / 1000   3.0e+03   4.1e+03   5.2e-01
[ Info:           4 / 5       500 / 1000   2.9e+03   4.0e+03   5.0e-01
[ Info:           4 / 5       520 / 1000   2.9e+03   3.8e+03   4.8e-01
[ Info:           4 / 5       540 / 1000   2.9e+03   3.7e+03   4.7e-01
[ Info:           4 / 5       560 / 1000   2.8e+03   3.9e+03   4.9e-01
[ Info:           4 / 5       580 / 1000   2.8e+03   3.7e+03   4.7e-01
[ Info:           4 / 5       600 / 1000   2.8e+03   4.2e+03   5.3e-01
[ Info:           4 / 5       620 / 1000   2.7e+03   3.5e+03   4.5e-01
[ Info:           4 / 5       640 / 1000   2.7e+03   3.3e+03   4.3e-01
[ Info:           4 / 5       660 / 1000   2.7e+03   3.4e+03   4.3e-01
[ Info:           4 / 5       680 / 1000   2.6e+03   3.4e+03   4.3e-01
[ Info:           4 / 5       700 / 1000   2.6e+03   3.4e+03   4.3e-01
[ Info:           4 / 5       720 / 1000   2.6e+03   3.3e+03   4.2e-01
[ Info:           4 / 5       740 / 1000   2.6e+03   3.1e+03   3.9e-01
[ Info:           4 / 5       760 / 1000   2.5e+03   3.7e+03   4.7e-01
[ Info:           4 / 5       780 / 1000   2.5e+03   3.2e+03   4.0e-01
[ Info:           4 / 5       800 / 1000   2.5e+03   3.2e+03   4.0e-01
[ Info:           4 / 5       820 / 1000   2.5e+03   3.3e+03   4.2e-01
[ Info:           4 / 5       840 / 1000   2.4e+03   4.1e+03   5.2e-01
[ Info:           4 / 5       860 / 1000   2.4e+03   2.9e+03   3.6e-01
[ Info:           4 / 5       880 / 1000   2.4e+03   2.9e+03   3.7e-01
[ Info:           4 / 5       900 / 1000   2.4e+03   2.7e+03   3.4e-01
[ Info:           4 / 5       920 / 1000   2.3e+03   2.8e+03   3.5e-01
[ Info:           4 / 5       940 / 1000   2.3e+03   3.0e+03   3.7e-01
[ Info:           4 / 5       960 / 1000   2.3e+03   2.5e+03   3.2e-01
[ Info:           4 / 5       980 / 1000   2.3e+03   2.5e+03   3.2e-01
[ Info:           4 / 5      1000 / 1000   2.3e+03   2.6e+03   3.3e-01
[ Info:           5 / 5         1 / 1000   2.3e+03   2.7e+03   1.0e+00
[ Info:           5 / 5        20 / 1000   2.2e+03   2.5e+03   9.5e-01
[ Info:           5 / 5        40 / 1000   2.2e+03   2.4e+03   9.2e-01
[ Info:           5 / 5        60 / 1000   2.2e+03   2.5e+03   9.2e-01
[ Info:           5 / 5        80 / 1000   2.2e+03   2.7e+03   1.0e+00
[ Info:           5 / 5       100 / 1000   2.2e+03   2.6e+03   9.7e-01
[ Info:           5 / 5       120 / 1000   2.2e+03   2.4e+03   9.1e-01
[ Info:           5 / 5       140 / 1000   2.2e+03   2.4e+03   9.0e-01
[ Info:           5 / 5       160 / 1000   2.2e+03   2.4e+03   9.0e-01
[ Info:           5 / 5       180 / 1000   2.2e+03   2.5e+03   9.5e-01
[ Info:           5 / 5       200 / 1000   2.2e+03   2.3e+03   8.8e-01
[ Info:           5 / 5       220 / 1000   2.2e+03   2.3e+03   8.9e-01
[ Info:           5 / 5       240 / 1000   2.2e+03   2.4e+03   9.0e-01
[ Info:           5 / 5       260 / 1000   2.2e+03   2.4e+03   9.0e-01
[ Info:           5 / 5       280 / 1000   2.2e+03   2.3e+03   8.6e-01
[ Info:           5 / 5       300 / 1000   2.2e+03   2.4e+03   8.9e-01
[ Info:           5 / 5       320 / 1000   2.2e+03   2.3e+03   8.7e-01
[ Info:           5 / 5       340 / 1000   2.2e+03   2.3e+03   8.5e-01
[ Info:           5 / 5       360 / 1000   2.2e+03   2.3e+03   8.5e-01
[ Info:           5 / 5       380 / 1000   2.1e+03   2.2e+03   8.4e-01
[ Info:           5 / 5       400 / 1000   2.1e+03   2.2e+03   8.5e-01
[ Info:           5 / 5       420 / 1000   2.1e+03   2.3e+03   8.5e-01
[ Info:           5 / 5       440 / 1000   2.1e+03   2.5e+03   9.3e-01
[ Info:           5 / 5       460 / 1000   2.1e+03   2.2e+03   8.4e-01
[ Info:           5 / 5       480 / 1000   2.1e+03   2.5e+03   9.4e-01
[ Info:           5 / 5       500 / 1000   2.1e+03   2.2e+03   8.4e-01
[ Info:           5 / 5       520 / 1000   2.1e+03   2.2e+03   8.4e-01
[ Info:           5 / 5       540 / 1000   2.1e+03   2.4e+03   9.2e-01
[ Info:           5 / 5       560 / 1000   2.1e+03   2.5e+03   9.5e-01
[ Info:           5 / 5       580 / 1000   2.1e+03   2.1e+03   8.1e-01
[ Info:           5 / 5       600 / 1000   2.1e+03   2.3e+03   8.5e-01
[ Info:           5 / 5       620 / 1000   2.1e+03   2.1e+03   8.0e-01
[ Info:           5 / 5       640 / 1000   2.1e+03   2.2e+03   8.4e-01
[ Info:           5 / 5       660 / 1000   2.1e+03   2.2e+03   8.4e-01
[ Info:           5 / 5       680 / 1000   2.1e+03   2.1e+03   8.0e-01
[ Info:           5 / 5       700 / 1000   2.1e+03   2.1e+03   7.9e-01
[ Info:           5 / 5       720 / 1000   2.1e+03   2.3e+03   8.8e-01
[ Info:           5 / 5       740 / 1000   2.1e+03   2.1e+03   7.8e-01
[ Info:           5 / 5       760 / 1000   2.1e+03   2.1e+03   7.8e-01
[ Info:           5 / 5       780 / 1000   2.1e+03   2.1e+03   7.9e-01
[ Info:           5 / 5       800 / 1000   2.0e+03   2.4e+03   8.9e-01
[ Info:           5 / 5       820 / 1000   2.0e+03   2.3e+03   8.5e-01
[ Info:           5 / 5       840 / 1000   2.0e+03   2.1e+03   7.7e-01
[ Info:           5 / 5       860 / 1000   2.0e+03   2.5e+03   9.3e-01
[ Info:           5 / 5       880 / 1000   2.0e+03   2.0e+03   7.6e-01
[ Info:           5 / 5       900 / 1000   2.0e+03   2.0e+03   7.6e-01
[ Info:           5 / 5       920 / 1000   2.0e+03   2.1e+03   7.9e-01
[ Info:           5 / 5       940 / 1000   2.0e+03   2.0e+03   7.6e-01
[ Info:           5 / 5       960 / 1000   2.0e+03   2.0e+03   7.5e-01
[ Info:           5 / 5       980 / 1000   2.0e+03   2.2e+03   8.4e-01
[ Info:           5 / 5      1000 / 1000   2.0e+03   2.1e+03   7.9e-01
1458.739189 seconds (1.61 G allocations: 308.094 GiB, 7.12% gc time, 0.39% compilation time)
FittedPumasModel

Dynamical system type:               Nonlinear ODE
Solver(s): (OrdinaryDiffEqVerner.Vern7,OrdinaryDiffEqRosenbrock.Rodas5P)

Number of subjects:                             32

Observation records:         Active        Missing
    conc:                       247             51
    pca:                        232             66
    Total:                      479            117

Number of parameters:      Constant      Optimized
                                  0             23

Estimation method:              VEM (EXPERIMENTAL)

-----------------------
            Estimate
-----------------------
pop_CL       0.16289
pop_V        9.0283
pop_tabs     1.4096
pop_lag      0.89858
pk_Ω₁,₁      0.14117
pk_Ω₂,₁      0.074728
pk_Ω₃,₁      0.33276
pk_Ω₂,₂      0.20382
pk_Ω₃,₂      0.011606
pk_Ω₃,₃      1.6695
σ_prop       0.033996
σ_add        2.8624
pop_e0      95.31
pop_emax    -1.14
pop_c50      1.8619
pop_tover   13.417
pd_Ω₁,₁      0.0089951
pd_Ω₂,₁      0.0034843
pd_Ω₃,₁     -0.0049231
pd_Ω₂,₂      0.024728
pd_Ω₃,₂     -0.016465
pd_Ω₃,₃      0.032257
σ_fx         0.70753
-----------------------
loglikelihood(fpm_vem_resample, FOCE())
-2091.5530177086052

The number of iterations and phases have been restricted in the above example for demonstration purposes.

NoteReading the VEM Convergence Trace

When presampling is enabled, the trace is deterministic and should show steady convergence. The trace printed during fitting has three columns:

Column Meaning
Iter Current iteration within the phase.
Function value The negative ELBO — the objective being minimized. Lower values indicate a better fit.
Gradient norm The gradient norm — how steep the objective landscape is. Decreasing values indicate the optimizer is converging.

When resampling is enabled, the trace is stochastic and may show more fluctuation. In the stochastic case, the trace printed during fitting has five columns:

Column Meaning
phase Current phase / total phases. Each phase increases the sample count.
iter Current iteration / max iterations within this phase.
obj The negative ELBO — the objective being minimized. Lower values indicate a better fit.
∇obj The gradient norm — how steep the objective landscape is. Decreasing values indicate the optimizer is converging.
s∇obj The scaled gradient norm — gradient norm relative to the first iteration of the current phase (normalized to 1.0). Shows relative convergence progress within each phase.

A phase may terminate early with “Terminating phase due to sgtol” when the scaled gradient norm drops below the stopping tolerance (0.05), triggering the next phase, typically with an increase in the sample size and a decrease in the learning rate. For stochastic optimizers like Adam (resampling VEM), some fluctuation in ∇obj is normal because the gradients are estimated from random samples.

compare_estimates(;
    VEM_presample = fpm_vem,
    VEM_resample = fpm_vem_resample,
    VEM_FOCE = fpm_foce_from_vem,
    FOCE = fpm_foce_direct,
)
23×5 DataFrame
Row parameter VEM_presample VEM_resample VEM_FOCE FOCE
String Float64? Float64? Float64? Float64?
1 pop_CL 0.13453 0.16289 0.135019 0.135055
2 pop_V 7.99776 9.02833 8.03022 7.14558
3 pop_tabs 0.396436 1.40958 0.536134 0.619069
4 pop_lag 0.88715 0.898585 0.879113 0.872328
5 pk_Ω₁,₁ 0.0666168 0.141171 0.0683831 0.0679948
6 pk_Ω₂,₁ 0.0087813 0.0747281 0.00789109 0.0280481
7 pk_Ω₃,₁ 0.0386503 0.33276 0.032845 0.0102071
8 pk_Ω₂,₂ 0.0203362 0.203824 0.0210109 0.464393
9 pk_Ω₃,₂ -0.0261285 0.0116061 -0.00372611 -0.527727
10 pk_Ω₃,₃ 0.98922 1.6695 0.848756 1.41271
11 σ_prop 0.0869357 0.0339964 0.0868518 0.0882328
12 σ_add 0.40221 2.86235 0.431393 0.445463
13 pop_e0 96.334 95.3103 96.3617 96.3449
14 pop_emax -1.07 -1.13996 -1.08743 -1.08655
15 pop_c50 1.53989 1.8619 1.62901 1.62474
16 pop_tover 14.1233 13.4172 14.4208 14.406
17 pd_Ω₁,₁ 0.00305062 0.00899515 0.00285578 0.0028373
18 pd_Ω₂,₁ -0.00353783 0.00348428 0.0017457 0.00166261
19 pd_Ω₃,₁ 0.00128838 -0.00492307 0.000531597 0.000553428
20 pd_Ω₂,₂ 0.151206 0.0247284 0.143078 0.137592
21 pd_Ω₃,₂ -0.0334376 -0.0164645 -0.0359802 -0.0333467
22 pd_Ω₃,₃ 0.0131138 0.0322566 0.017667 0.0166972
23 σ_fx 3.83855 0.707526 3.56433 3.52277
DataFrame(
    Method = [
        "Deterministic VEM",
        "Stochastic VEM",
        "FOCE initialized from deterministic VEM",
        "FOCE direct",
    ],
    LogLikelihood = [
        loglikelihood(fpm_vem, FOCE()),
        loglikelihood(fpm_vem_resample, FOCE()),
        loglikelihood(fpm_foce_from_vem),
        loglikelihood(fpm_foce_direct),
    ],
) |> simple_table
Method LogLikelihood
Deterministic VEM -1063
Stochastic VEM -2092
FOCE initialized from deterministic VEM -1060
FOCE direct -1108

Note that stochastic VEM took much longer to converge and it only got close to the best parameters. Deterministic VEM will generally converge faster. Stochastic VEM is mostly useful for models with neural networks and large populations where subject_ratio < 1 (to be demonstrated below) can be used to get quick estimates by running the algorithm for a few iterations only.

4.2 Sample Size Control

VEM draws samples from a base distribution to estimate the ELBO gradient. More samples improve accuracy but increase computation time.

When nsamples is set to a nonzero value, it overrides init_nsamples and sets nsamples_increment to 0, so the sample count stays fixed across all phases:

# Fixed 30 samples per subject across all phases
@time fpm_vem_nsamples = fit(
    warfarin_model,
    pop,
    warfarin_init,
    VEM(; nsamples = 30);
    optim_options = (; show_every = 20),
)
[ Info: Checking the initial parameter values.
[ Info: The initial negative log-likelihood and its gradient are finite. Check passed.
Iter     Function value   Gradient norm 
     0     1.374727e+08     2.749450e+08
 * time: 2.09808349609375e-5
    20     1.550251e+03     5.815802e+02
 * time: 31.568580865859985
    40     1.082107e+03     1.660056e+02
 * time: 64.77731084823608
    60     9.610787e+02     2.306563e+01
 * time: 92.59014391899109
    80     9.228660e+02     2.706740e+01
 * time: 123.87429785728455
   100     8.818840e+02     1.738420e+01
 * time: 161.69694185256958
   120     8.493450e+02     4.231016e+01
 * time: 201.65474390983582
   140     8.043672e+02     1.933396e+01
 * time: 239.0825219154358
   160     7.603885e+02     1.282275e+01
 * time: 273.10702896118164
   180     7.324729e+02     1.321452e+01
 * time: 306.0147759914398
   200     7.226272e+02     3.851044e+00
 * time: 353.58604097366333
   220     7.178002e+02     1.130184e+00
 * time: 401.90573501586914
   240     7.168652e+02     7.567175e+00
 * time: 445.86555886268616
458.667157 seconds (393.31 M allocations: 89.216 GiB, 7.22% gc time)
FittedPumasModel

Dynamical system type:               Nonlinear ODE
Solver(s): (OrdinaryDiffEqVerner.Vern7,OrdinaryDiffEqRosenbrock.Rodas5P)

Number of subjects:                             32

Observation records:         Active        Missing
    conc:                       247             51
    pca:                        232             66
    Total:                      479            117

Number of parameters:      Constant      Optimized
                                  0             23

Estimation method:              VEM (EXPERIMENTAL)

-----------------------
            Estimate
-----------------------
pop_CL       0.13432
pop_V        7.9897
pop_tabs     0.39726
pop_lag      0.8834
pk_Ω₁,₁      0.064757
pk_Ω₂,₁      0.0083885
pk_Ω₃,₁      0.030556
pk_Ω₂,₂      0.020286
pk_Ω₃,₂     -0.027913
pk_Ω₃,₃      0.91077
σ_prop       0.086739
σ_add        0.40521
pop_e0      96.306
pop_emax    -1.0702
pop_c50      1.5436
pop_tover   14.117
pd_Ω₁,₁      0.0032088
pd_Ω₂,₁     -0.0037275
pd_Ω₃,₁      0.0010459
pd_Ω₂,₂      0.14785
pd_Ω₃,₂     -0.033741
pd_Ω₃,₃      0.014401
σ_fx         3.7984
-----------------------
loglikelihood(fpm_vem_nsamples, FOCE())
-1062.68021696948

Notice that increasing the number of samples improved the quality of the ELBO however that does not guarantee an improvement in the FOCE-approximated log likelihood of the solution found. In this case, it improved it. Heuristically, setting nsamples to 15-30 seems to be a good starting point for most models, but the best value can vary based on the number of random effects in the model.

For multi-phase runs, the number of samples in phase p is: min(init_nsamples + (p - 1) * nsamples_increment, max_nsamples). More samples in later phases refine the estimate as the parameters approach their final values.

# Resampling with more phases and custom sample progression
fit(
    model,
    pop,
    params,
    VEM(;
        presample = false,
        nphases = 15,
        init_nsamples = 5,
        nsamples_increment = 5,
        max_nsamples = 50,
    ),
)

4.3 Optimizer Options (optim_options)

Fine-grained control over the optimizer is passed through the optim_options named tuple. This is a keyword to fit, not to VEM(). The available options depend on which variant is used.

For presampling (L-BFGS):

Option Default Description
iterations 3000 Maximum iterations per phase
show_trace true Print convergence information
show_every 5 Print every N iterations
f_reltol 1e-5 Relative tolerance on the objective
g_tol 1e-3 Gradient norm tolerance
fit(model, pop, params, VEM(); optim_options = (; iterations = 5000, show_every = 10))

For resampling (Adam):

Option Default Description
iterations 20000 Maximum iterations per phase
show_trace true Print convergence information
show_every 10 Print every N iterations
subject_ratio 1.0 Fraction of subjects per iteration (mini-batching)

Setting subject_ratio < 1.0 enables mini-batching, where only a random subset of subjects is used per iteration. This can speed up each iteration for large datasets at the cost of noisier gradients:

fit(
    model,
    pop,
    params,
    VEM(; presample = false);
    optim_options = (; iterations = 30000, subject_ratio = 0.5),
)

4.4 Prior on Population Parameters (prior)

By default (prior=true), VEM adds the log prior on the population parameters, if defined in the @param block, to the ELBO. This allows for maximum a-posteriori (MAP) estimation of the population parameters, after marginalizing the random effects. Setting prior=false ignores the prior contribution, which means the ELBO depends only on the likelihood:

fit(model, pop, params, VEM(; prior = false))

If the @param block has no prior distributions and only includes domains, this option is not used.

4.5 Base Distribution Standard Deviation (base_dist_std)

VEM’s variational family uses a reparameterization trick: samples are drawn from a base distribution and transformed to approximate the posterior. The base_dist_std parameter controls the standard deviation of this base distribution (default: 0.1):

fit(model, pop, params, VEM(; base_dist_std = 0.05))

Smaller values start the optimization with a tighter initial approximation to the posterior, which can help stability for sensitive models. Larger values allow broader initial exploration. The default of 0.1 works well for most models.

4.6 Posterior Covariance Structure (diagonal)

VEM approximates each subject’s random effect posterior with a multivariate normal distribution. The diagonal keyword controls the covariance structure:

  • diagonal=true (default): assumes independent random effects in the posterior. Faster and requires fewer variational parameters.
  • diagonal=false: captures correlations between random effects in the posterior. Slower but may be more accurate when random effects are correlated.
fpm_full_cov = fit(
    warfarin_model,
    pop,
    warfarin_init,
    VEM(; diagonal = false);
    optim_options = (; show_every = 20),
)
[ Info: Checking the initial parameter values.
[ Info: The initial negative log-likelihood and its gradient are finite. Check passed.
Iter     Function value   Gradient norm 
     0     1.368292e+08     2.736580e+08
 * time: 2.6941299438476562e-5
    20     1.525485e+03     5.300085e+02
 * time: 16.681507110595703
    40     1.103091e+03     2.490318e+02
 * time: 31.316335916519165
    60     9.616820e+02     4.585779e+01
 * time: 43.586803913116455
    80     8.954688e+02     4.166251e+01
 * time: 57.02774500846863
   100     8.118614e+02     7.086030e+01
 * time: 72.26464104652405
   120     7.623553e+02     9.704948e+01
 * time: 89.80058193206787
   140     7.333012e+02     1.749287e+01
 * time: 107.1308491230011
   160     7.201056e+02     1.222130e+01
 * time: 124.47755908966064
   180     7.143766e+02     2.752501e+00
 * time: 142.4066789150238
   200     7.108468e+02     3.909115e+00
 * time: 166.86034893989563
   220     7.075539e+02     6.408487e+00
 * time: 199.35051107406616
   240     7.053211e+02     1.071369e+01
 * time: 229.38029193878174
   260     7.041626e+02     1.499914e+01
 * time: 254.64632105827332
   280     7.031582e+02     1.646967e+01
 * time: 272.8636751174927
   300     7.024212e+02     2.161227e+00
 * time: 291.43470311164856
   320     7.017475e+02     2.988461e+01
 * time: 309.0804340839386
   340     7.009747e+02     4.088582e+00
 * time: 328.33753514289856
   360     6.996141e+02     4.386131e+00
 * time: 348.9258201122284
   380     6.983768e+02     2.137981e+00
 * time: 367.09574604034424
   400     6.972393e+02     5.685682e+00
 * time: 384.0184030532837
   420     6.953632e+02     4.324135e+00
 * time: 400.6962969303131
   440     6.936934e+02     6.003071e+00
 * time: 418.0892460346222
   460     6.927456e+02     1.074300e+01
 * time: 435.73579001426697
   480     6.918063e+02     3.361402e+00
 * time: 456.6340711116791
   500     6.908235e+02     1.331224e+00
 * time: 480.790687084198
   520     6.901069e+02     3.909953e+00
 * time: 501.35266399383545
FittedPumasModel

Dynamical system type:               Nonlinear ODE
Solver(s): (OrdinaryDiffEqVerner.Vern7,OrdinaryDiffEqRosenbrock.Rodas5P)

Number of subjects:                             32

Observation records:         Active        Missing
    conc:                       247             51
    pca:                        232             66
    Total:                      479            117

Number of parameters:      Constant      Optimized
                                  0             23

Estimation method:              VEM (EXPERIMENTAL)

-----------------------
            Estimate
-----------------------
pop_CL       0.13422
pop_V        8.0044
pop_tabs     0.42091
pop_lag      0.88938
pk_Ω₁,₁      0.067359
pk_Ω₂,₁      0.0081471
pk_Ω₃,₁      0.055683
pk_Ω₂,₂      0.02143
pk_Ω₃,₂     -0.0063547
pk_Ω₃,₃      1.0164
σ_prop       0.086551
σ_add        0.43006
pop_e0      96.402
pop_emax    -1.0866
pop_c50      1.6153
pop_tover   14.37
pd_Ω₁,₁      0.0020935
pd_Ω₂,₁      0.0050701
pd_Ω₃,₁      0.0010105
pd_Ω₂,₂      0.13637
pd_Ω₃,₂     -0.035903
pd_Ω₃,₃      0.012341
σ_fx         3.894
-----------------------
loglikelihood(fpm_full_cov, FOCE())
-1064.1164257202418

Note that setting diagonal = false does not always lead to a better fit. It is generally recommended to start with diagonal = true and only switch to diagonal = false if there is reason to believe that the random effects are highly correlated.

4.7 Holding Parameters Constant (constantcoef)

You can hold specific parameters constant during estimation using the constantcoef keyword. This is a keyword to fit, not to VEM():

fit(model, pop, params, VEM(); constantcoef = (:σ_prop,))
Note

constantcoef is a keyword argument to fit, not to VEM(). This is consistent with how it works for FOCE and other algorithms.

4.8 Reproducibility (rng)

VEM uses random sampling internally, so results vary across runs by default. Set a specific random number generator for reproducible results. This is a keyword to fit, not to VEM():

rng = default_rng()
seed!(rng, 123)
fit(model, pop, params, VEM(); rng)
Note

Even with presampling, different seeds produce different results because the pre-drawn samples differ. For exact reproducibility across runs, always set the rng.

4.9 Threading (ensemblealg)

Control parallelism across subjects with this keyword to fit:

  • EnsembleThreads() (default): uses multiple threads for gradient computation.
  • EnsembleSerial(): single-threaded, useful for debugging.
fit(model, pop, params, VEM(); ensemblealg = EnsembleSerial())

4.10 ODE Solver Options (diffeq_options)

Pass differential equation solver options through diffeq_options, another keyword to fit:

fit(model, pop, params, VEM(); diffeq_options = (; abstol = 1e-8, reltol = 1e-8))

This is useful when the default ODE tolerances are insufficient for a stiff or sensitive model.

5 When to Use VEM vs FOCE/LaplaceI

Choose VEM when:

  • You have a response model that FOCE does not support and LaplaceI is too slow, e.g. truncated, censored or time-to-event data.
  • The model has many random effects and FOCE is too slow or struggles to converge, and you want a faster alternative, e.g. to initialize FOCE or as a standalone method.
  • You have a very large dataset and want a fast initial fit to use as a starting point for FOCE/LaplaceI, then stochastic VEM with mini-batching can help.

Choose FOCE when:

  • You have a model without too many random effects and the response model is supported by FOCE.
  • You need a well-established (non-experimental) fitting method, diagnostics and inference tools.

Choose LaplaceI when:

  • Your response model is not supported by FOCE.
  • You need a well-established (non-experimental) fitting method, diagnostics and inference tools.

6 Conclusion

In this tutorial, we introduced VEM as a flexible estimation method in Pumas. We covered its conceptual foundations (ELBO maximization, presampling vs. resampling, and posterior covariance structure), demonstrated its use with a warfarin PK model, showed how to compare results with FOCE, and provided a reference for the main VEM options.

7 References

Holford, N H G. 2010. PKPD Workshop. University of Auckland. https://holford.fmhs.auckland.ac.nz/docs/pkpd-workshop-nonmem7.pdf.
O’Reilly, R A, and P M Aggeler. 1968. “Studies on Coumarin Anticoagulant Drugs. Initiation of Warfarin Therapy Without a Loading Dose.” Circulation 38 (1): 169–77. https://doi.org/10.1161/01.cir.38.1.169.
O’Reilly, R A, P M Aggeler, and L S Leong. 1963. “Studies on the Coumarin Anticoagulant Drugs: The Pharmacodynamics of Warfarin in Man.” Journal of Clinical Investigation 42 (10): 1542–51. https://doi.org/10.1172/JCI104839.