Fitting Non-Identifiable and Poorly Identifiable Models using Bayesian Inference

Author

Mohamed Tarek

In this tutorial, we will see how to use Bayesian inference in Pumas to fit a non-identifiable (or a poorly identifiable) model by sampling from the full posterior. Unlike maximum likelihood estimation methods which only find point estimates for the model parameters, Bayesian methods can be used to sample from the full posterior of the model parameters making it more robust to identifiability issues.

1 Model Identifiability

One of the goals of statistical learning is to identify the underlying parameter values in a parametric model that best fit the observed data. In many practical scenarios, some parameters in a model may not be identifiable, for one of the following problems:

  1. The model is over-parameterized with redundant parameters, where a continuum of parameter values, e.g \(0 \leq \theta \leq 1\), would all give identical model predictions. This tends to happen when a dynamical model has many compartments and associated parameters but only one or a few compartments are observed. Another case where this happens is if there is a typo in the model where a parameter is not used everywhere it should.
  2. The model has symmetries such that a discrete set of parameter values, e.g. \(\theta = -1\) or \(\theta = 1\), give identical model predictions. This tends to happen with inappropriate use of the “absolute value” function (abs) if used on a function that can be either negative or positive potentially creating 2 separate modes. This problem can sometimes be fixed by using appropriate parameter bounds, e.g. \(\theta \geq 0\), in the model.
  3. The data is insufficient to learn some parameters’ values but more data would have been sufficient.

The first 2 problems are problems in the model describing structural non-identifiability and the third problem is a problem in the data or data-model mismatch describing practical non-identifiability (sometimes called non-estimability).

Type of Identifiability Description
Globally Structurally Identifiable Every set of parameter values \(\theta\) makes a unique model prediction \(\mu(\theta)\).
Globally Practically Identifiable Globally structurally identifiable and there is enough data to estimate the data-generating parameter values.
Locally Structurally Identifiable Only dis-connected parameters values \(\{\theta_1, \theta_2, \dots, \theta_m\}\) can result in the same model prediction \(\mu\), i.e. \(\mu(\theta_1) = \mu(\theta_2) = \dots = \mu(\theta_m)\). However, in the neighborhood \(N(\theta)\) of each set of parameters \(\theta\), each \(\theta' \in N(\theta)\) results in a unique model prediction \(\mu(\theta')\).
Locally Practically Identifiable Locally structurally identifiable and there is enough data to estimate the (potentially non-unique but dis-connected) data-generating parameter values.
NoteKey Points
  • Local (or global) practical identifiablity is what we usually want in analyses.
  • Practical identifiability (estimability) implies structural identifiability.
NotePoor Identifiability

When a model may be identifiable in exact arithmetic but its identifiability is sensitive to numerical errors in computations, we will call it poorly identifiable. When the term “poorly identifiable models” is used in the rest of this tutorial, this also includes truly non-identifiable models.

2 Example

Let’s consider an example of fitting a poorly identifiable model.

2.1 Loading Pumas

First, let’s load Pumas:

using Pumas

2.2 Two Compartment Model

Now let’s define a single-subject, 2-compartment model with a depot, central and peripheral compartments and a proportional error model.

model = @model begin
    @param begin
        θ ∈ VectorDomain(lower = zeros(5))
        σ ∈ RealDomain(lower = 0.0)
    end
    @pre begin
        CL = θ[1]
        Vc = θ[2]
        Ka = θ[3]
        Vp = θ[4]
        Q = θ[5]
    end
    @dynamics begin
        Depot' = -Ka * Depot
        Central' = Ka * Depot - (CL + Q) / Vc * Central + Q / Vp * Peripheral
        Peripheral' = Q / Vc * Central - Q / Vp * Peripheral
    end
    @derived begin
        cp := @. Central / Vc
        dv ~ @. CombinedNormal(cp, 1e-6, σ)
    end
end
PumasModel
  Parameters: θ, σ
  Random effects:
  Covariates:
  Dynamical system variables: Depot, Central, Peripheral
  Dynamical system type: Matrix exponential
  Derived: dv
  Observed: dv
NoteSubject Model vs Population Model

Note that studying the identifiability of subject models is still relevant when fitting a population version of the model with typical values and random effects. The reason is that when evaluating the log likelihood using the Laplace method or first-order conditional estimation (FOCE), one of the steps involved is finding the empirical Bayes estimates (EBE) which is essentially fitting a subject’s version of the model, fixing population parameters and estimating random effects. Non-identifiability in the EBE estimation can often cause the EBE estimation to fail or to be so numerically unstable that the population parameters’ fit itself fails because the marginal likelihood (and its gradient) computed with Laplace/FOCE relied on incorrect or numerically unstable EBEs.

2.3 Parameter Values

Let’s define some parameter values to use for simulation.

params = (θ = [35, 100, 0.5, 210, 30], σ = 0.1)
(θ = [35.0, 100.0, 0.5, 210.0, 30.0], σ = 0.1)

2.4 Subject Definition

Next we define a subject skeleton with a single bolus dose and no observations.

skeleton = Subject(
    id = 1,
    time = 0.0:0.5:30.0,
    events = DosageRegimen(3000, time = 0.0, cmt = 1),
    observations = (; dv = fill(missing, 61)),
)
Subject
  ID: 1
  Events: 1
  Observations: dv: (n=0)

2.5 Fisher Information Matrix

The expected Fisher Information Matrix (FIM) is an important diagnostic which can be used to detect local practical identifiability (LPI) given the model and the experiment design [1]. The first order approximation of the expected FIM [2,3] has also been used successfully to analyze the LPI of pharmacometric nonlinear mixed effect (NLME) models [4,5,6].

The positive definiteness (non-singularity) of the expected FIM \(F(\theta)\) at parameters \(\theta\) is a sufficient condition for LPI at \(\theta\). Under more strict assumptions which are more difficult to verify, the positive definiteness of \(F(\theta)\) is even a necessary condition for LPI [1].

To compute a first-order approximation of the expected FIM, we will use the OptimalDesign package:

using OptimalDesign

times = OptimalDesign.ObsTimes(skeleton.time)
F = OptimalDesign.fim(model, [skeleton], params, [times])
6×6 Symmetric{Float64, Matrix{Float64}}:
  10.3418      -0.518982     101.492    …   -2.3499       -46.085
  -0.518982     0.0483945    -10.4502        0.14191        1.83455
 101.492      -10.4502      2494.59        -30.8857      -390.729
  -0.0215956   -0.00137419     1.96229      -0.0218341     -0.261046
  -2.3499       0.14191      -30.8857        0.810094       9.47803
 -46.085        1.83455     -390.729    …    9.47803    12000.0

2.6 Procedure for Detecting Practical Non-Identifiability

Next, let’s do an eigenvalue decomposition of F to find the smallest eigenvalue. The smallest eigenvalue is always the first one because they are sorted.

E = eigen(F)
E.values[1]
0.00034486766130990094

It is close to 0! This implies that the matrix is very close to being singular. While this is not strictly a proof of local non-identifiability, it is one step towards detecting non-identifiability.

Tip

The non-singularity of the expected FIM is generally only a sufficient (not necessary) condition for local identifiability. So there are some locally identifiable models that have a singular or undefined expected FIM. However, there is a subclass of models satisfying strict assumptions for which the non-singularity of the expected FIM is a necessary condition for local identifiability [1]. In practice, these assumptions are difficult to verify for a general model but if our model happens to satisfy these assumptions, then a singular expected FIM would imply local non-identifiability. In that case, the eigenvector(s) corresponding to the 0 eigenvalue would be useful diagnostics as they point in the directions along which changes in the parameters will have no effect on the log likelihood.

To summarize:

  • Being “Non-Singular” Isn’t Always a Must: A non-singular expected FIM always signals that a model is “locally identifiable” (meaning you can find a locally unique solution for its parameters). However, some locally identifiable models can still have singular or undefined expected FIM.
  • Sometimes, It’s Absolutely Necessary: There’s a special group of models where a non-singular expected FIM is required to be locally identifiable. It’s hard to know if your model falls into this group.
  • Practical Use: Even if your expected FIM is singular, it may be a helpful diagnostic. The eigenvectors of the FIM with 0 eigenvalues show you directions where changing the model’s parameters may not affect the model’s predictions – this may help you pinpoint where the model is fuzzy.

To prove if a model is practically non-identifiable, it suffices to find a single set \(\Theta\) such that for all parameters \(\theta \in \Theta\), the log likelihood is unchanged. To find \(\Theta\) numerically, one can

  1. Assume parameter values \(\theta_0\),
  2. Define a criteria for changing the parameters \(\theta\) from their current values \(\theta_0\), and then create a candidate set of parameters \(\Theta_c\),
  3. Simulate synthetic data using the parameters \(\theta_0\),
  4. Evaluate the log likelihood \(L(\theta)\) for all \(\theta \in \Theta_c\),
  5. Evaluate the sensitivity of the log likelihood to local changes within \(\Theta_c\).

One way to construct a candidate \(\Theta_c\) is as the set \(\{\theta_0 + \alpha \cdot d : \alpha \in [-\epsilon, \epsilon] \}\) for a small \(\epsilon > 0\), where \(d\) is an eigenvector corresponding to the smallest eigenvalue of the expected FIM, \(F\). The sensitivity of the log likelihood to changes within \(\Theta_c\) can then be quantified as the average value of:

\[ \Bigg( \frac{L(\theta_0 + \alpha \cdot d) - L(\theta_0)}{\alpha} \Bigg)^2 \]

for all \(\alpha \in [-\epsilon, \epsilon]\), \(\alpha \neq 0\). Let’s follow this procedure for the above model assuming \(\theta_0\) is params.

2.7 Simulating Data

Here we simulate a synthetic subject using the skeleton subject we have. We fix the seed of the pseudo-random number generator for reproducibility.

using Random

Random.seed!(12345)
pop = [Subject(simobs(model, skeleton, params))]
Population
  Subjects: 1
  Observations: dv

To evaluate the log likelihood of params given pop, we can use the loglikelihood function:

ll0 = loglikelihood(model, pop, params, NaivePooled())
32.82509838337166

Since there are no random effects in this model, we use the NaivePooled() algorithm in Pumas.

2.8 Local Sensitivity Analysis of Log Likelihood

The eigenvectors corresponding to the smallest eigenvalue of F give us the directions that are likely to have the largest standard error in the maximum likelihood estimates. These are the most promising directions to test when constructing candidate sets \(\Theta_c\) for identifiability analysis. To get the eigenvector corresponding to the smallest eigenvalue, you can run:

d = E.vectors[:, 1]
6-element Vector{Float64}:
 -0.002380469921277704
 -0.8754610365061731
 -0.0037426498922106567
  0.4829767082714525
  0.016786862766484546
  8.218940732069106e-8

The order of parameters is the same as the order of definition in the model: θ and then σ.

Note that the potential non-identifiability seems to be mostly in the second and fourth parameter, Vc and Vp. More precisely, the above eigenvector implies that simultaneously increasing Vc and decreasing Vp (or vice versa) by the ratios given in d may have little to no effect on the log likelihood.

Let’s try to add α * d[1:end-1] to params.θ (i.e. move params.θ in the direction d[1:end-1]) and evaluate the log likelihood for different step sizes α. To do that, we will first define a function that moves params.θ and call it on different step values α. These choices of α values correspond to a discrete candidate set \(\Theta_c\).

function moveθ(α)
    # unpacking the fields of params to variables with the same names
    (; θ, σ) = params
    # move θ by step * d[1:end-1]
    return (; θ = θ + α * d[1:end-1], σ)
end
# move by both negative and positive steps (excluding α = 0)
αs = vcat(-1e-3 .* (1:10), 1e-3 .* (1:10))
20-element Vector{Float64}:
 -0.001
 -0.002
 -0.003
 -0.004
 -0.005
 -0.006
 -0.007
 -0.008
 -0.009
 -0.01
  0.001
  0.002
  0.003
  0.004
  0.005
  0.006
  0.007
  0.008
  0.009
  0.01
newparams = moveθ.(αs)
20-element Vector{@NamedTuple{θ::Vector{Float64}, σ::Float64}}:
 (θ = [35.00000238046992, 100.0008754610365, 0.5000037426498922, 209.99951702329173, 29.999983213137234], σ = 0.1)
 (θ = [35.000004760939845, 100.00175092207301, 0.5000074852997844, 209.99903404658346, 29.99996642627447], σ = 0.1)
 (θ = [35.00000714140976, 100.00262638310952, 0.5000112279496767, 209.9985510698752, 29.9999496394117], σ = 0.1)
 (θ = [35.00000952187968, 100.00350184414603, 0.5000149705995689, 209.99806809316692, 29.999932852548934], σ = 0.1)
 (θ = [35.000011902349605, 100.00437730518253, 0.500018713249461, 209.99758511645865, 29.99991606568617], σ = 0.1)
 (θ = [35.00001428281953, 100.00525276621903, 0.5000224558993532, 209.99710213975038, 29.999899278823403], σ = 0.1)
 (θ = [35.00001666328945, 100.00612822725554, 0.5000261985492455, 209.9966191630421, 29.999882491960634], σ = 0.1)
 (θ = [35.00001904375937, 100.00700368829204, 0.5000299411991377, 209.99613618633384, 29.999865705097868], σ = 0.1)
 (θ = [35.00002142422929, 100.00787914932856, 0.5000336838490299, 209.99565320962554, 29.999848918235102], σ = 0.1)
 (θ = [35.00002380469921, 100.00875461036506, 0.5000374264989221, 209.99517023291727, 29.999832131372337], σ = 0.1)
 (θ = [34.99999761953008, 99.9991245389635, 0.4999962573501078, 210.00048297670827, 30.000016786862766], σ = 0.1)
 (θ = [34.999995239060155, 99.99824907792699, 0.49999251470021555, 210.00096595341654, 30.00003357372553], σ = 0.1)
 (θ = [34.99999285859024, 99.99737361689048, 0.4999887720503234, 210.0014489301248, 30.0000503605883], σ = 0.1)
 (θ = [34.99999047812032, 99.99649815585397, 0.49998502940043116, 210.00193190683308, 30.000067147451066], σ = 0.1)
 (θ = [34.999988097650395, 99.99562269481747, 0.49998128675053893, 210.00241488354135, 30.00008393431383], σ = 0.1)
 (θ = [34.99998571718047, 99.99474723378097, 0.4999775441006467, 210.00289786024962, 30.000100721176597], σ = 0.1)
 (θ = [34.99998333671055, 99.99387177274446, 0.49997380145075454, 210.0033808369579, 30.000117508039366], σ = 0.1)
 (θ = [34.99998095624063, 99.99299631170796, 0.4999700588008623, 210.00386381366616, 30.000134294902132], σ = 0.1)
 (θ = [34.99997857577071, 99.99212085067144, 0.4999663161509701, 210.00434679037446, 30.000151081764898], σ = 0.1)
 (θ = [34.99997619530079, 99.99124538963494, 0.49996257350107787, 210.00482976708273, 30.000167868627663], σ = 0.1)

Now let’s evaluate the log likelihoods of all these parameter sets.

lls = map(newparams) do p
    loglikelihood(model, pop, p, NaivePooled())
end
20-element Vector{Float64}:
 32.82509438583191
 32.825090388377994
 32.82508639226626
 32.825082395873956
 32.82507840138675
 32.82507440629848
 32.82507041276352
 32.82506641963859
 32.8250624272587
 32.825058435243456
 32.82510238182574
 32.825106381262806
 32.82511038109052
 32.82511438157983
 32.825118382829395
 32.82512238386422
 32.82512638675639
 32.82513038898445
 32.82513439365729
 32.82513839764922

To compute the average sensitivity within \(\Theta_c\), we then call:

sens = abs2.((lls .- ll0) ./ αs)
mean(sens)
1.5985066566509463e-5

Not very sensitive! Therefore, we can conclude that the log likelihood is almost the same inside \(\Theta_c\). This is a strong sign of non-identifiability, or at least poor identifiability.

Let’s contrast this with a random direction d2:

d2 = normalize(rand(6))
function moveθ2(α)
    # unpacking the fields of params to variables with the same names
    (; θ, σ) = params
    # move θ by step * d2[1:end-1]
    return (; θ = θ + α * d2[1:end-1], σ)
end
newparams2 = moveθ2.(αs)
lls2 = map(newparams2) do p
    loglikelihood(model, pop, p, NaivePooled())
end
sens2 = abs2.((lls2 .- ll0) ./ αs)
mean(sens2)
648.2254070922876

Notice the difference in sensitivity compared to a random direction.

Note

In practice, it can be difficult to be definitive about practical non-identifiability with numerical tests due to the nature of computation in floating point numbers where numerical errors can accumulate and either:

  1. Mask a truly singular matrix by reporting its smallest eigenvalue as very close to 0 but not exactly 0 in floating point numbers, or
  2. Make a truly non-singular matrix appear singular because its smallest eigenvalue was close to 0 in exact arithmetic but was computed as exactly 0 in floating point arithmetic.

A small enough average local sensitivity can therefore be taken as numerically equivalent to 0, i.e. local practical non-identifiability. To be more general, we will sometimes use the term poor identifiability to refer to the case when the model is approximately non-identifiable.

3 Fitting a Poorly Identifiable Model

3.1 Maximum Likelihood Estimation

When using maximum likelihood (ML) estimation to fit a poorly identifiable model, the parameter values you get can be dependent on arbitrary factors such as:

  • The initial parameter estimates. Optimization algorithms will typically converge to values close to the initial value which is arbitrary.
  • Level of noise in the data. Different levels of noise in the data can cause the optimization algorithm to take different trajectories reaching different optimal parameter values at the end.
  • The implementation details of the optimization algorithm. For example, some optimization algorithms implicitly favor parameter values with the smallest norm.

Most of these factors have no statistical significance and can be considered arbitrary in any analysis. Therefore, any insights drawn from poorly identifiable parameter values fitted with ML estimation may be flawed. Luckily, the standard error estimation, if done right, will often reveal signs of poor identifiability. However, common techniques for estimating standard errors can often break down when the model is poorly identifiable. For instance,

  1. Asymptotic estimates of standard errors require the model to be locally identifiable,
  2. Bootstrap relies on the same arbitrary optimization algorithm for fitting the data resamples so its estimates are as unreliable as the ML estimates. For instance, since all the re-fits in bootstrapping are typically initialized from the ML estimates, there is a high probability that the optimization algorithm will only converge to nearby values, significantly under-estimating the variance in the ML estimates.
  3. Sampling importance resampling (SIR) may be able to handle non-identifiable models but it requires a good proposal and its results are sensitive to the proposal used.

Example: Sensitivity to Initial Estimates

Let’s fit the model to the same subject but with 2 different sets of initial estimates:

params1 = (θ = [35, 100, 0.5, 210, 30], σ = 0.1)
fpm1 = fit(model, pop, params1, NaivePooled())
[ Info: Checking the initial parameter values.
[ Info: The initial negative log-likelihood and its gradient are finite. Check passed.
Iter     Function value   Gradient norm 
     0    -3.282510e+01     5.896826e+01
 * time: 0.07213592529296875
     1    -3.312144e+01     4.134793e+01
 * time: 5.923446893692017
     2    -3.410517e+01     2.822025e+01
 * time: 5.924860000610352
     3    -3.506276e+01     2.569455e+01
 * time: 5.9260969161987305
     4    -3.577916e+01     2.022773e+01
 * time: 5.927320957183838
     5    -3.600599e+01     1.161060e+01
 * time: 5.928526878356934
     6    -3.604984e+01     1.270240e+01
 * time: 5.929525852203369
     7    -3.607833e+01     1.436656e+00
 * time: 5.930523872375488
     8    -3.607861e+01     2.182965e-01
 * time: 5.9315290451049805
     9    -3.607863e+01     4.628923e-02
 * time: 5.932524919509888
    10    -3.607864e+01     2.722381e-02
 * time: 5.9335198402404785
    11    -3.607869e+01     1.854695e-01
 * time: 5.934515953063965
    12    -3.607871e+01     1.841239e-01
 * time: 5.935518026351929
    13    -3.607872e+01     6.087452e-02
 * time: 5.9365270137786865
    14    -3.607872e+01     2.801100e-03
 * time: 5.9375128746032715
    15    -3.607872e+01     2.888554e-04
 * time: 5.938511848449707
FittedPumasModel

Dynamical system type:          Matrix exponential

Number of subjects:                              1

Observation records:         Active        Missing
    dv:                          61              0
    Total:                       61              0

Number of parameters:      Constant      Optimized
                                  0              6

Likelihood approximation:              NaivePooled
Likelihood optimizer:                         BFGS

Termination Reason:                   GradientNorm
Log-likelihood value:                    36.078723

----------------
     Estimate
----------------
θ₁    34.397
θ₂   103.17
θ₃     0.46566
θ₄   204.44
θ₅    26.234
σ      0.092273
----------------
params2 = (θ = [10.0, 300, 1.0, 10.0, 5], σ = 0.2)
fpm2 = fit(model, pop, params2, NaivePooled())
[ Info: Checking the initial parameter values.
[ Info: The initial negative log-likelihood and its gradient are finite. Check passed.
Iter     Function value   Gradient norm 
     0     4.245141e+02     6.802189e+02
 * time: 4.410743713378906e-5
     1     1.499851e+02     4.520553e+01
 * time: 0.000759124755859375
     2     1.436202e+02     4.485778e+01
 * time: 0.0012869834899902344
     3     1.108788e+02     4.698772e+01
 * time: 0.0018260478973388672
     4     9.146761e+01     4.889582e+01
 * time: 0.0023491382598876953
     5     8.325568e+01     5.003716e+01
 * time: 0.0030870437622070312
     6     7.881725e+01     4.601078e+01
 * time: 0.003590106964111328
     7     6.471152e+01     3.571234e+01
 * time: 0.004119157791137695
     8     5.748754e+01     4.136350e+01
 * time: 0.00463414192199707
     9     5.643470e+01     4.672399e+01
 * time: 0.005256175994873047
    10     5.237584e+01     4.395656e+01
 * time: 0.005746126174926758
    11     4.391362e+01     4.297657e+01
 * time: 0.006359100341796875
    12     2.277857e+01     9.735685e+01
 * time: 0.007371187210083008
    13     2.127919e+01     1.418506e+01
 * time: 0.008001089096069336
    14     2.121702e+01     1.601608e+01
 * time: 0.008440017700195312
    15     2.061525e+01     1.765934e+01
 * time: 0.008888959884643555
    16     1.978391e+01     1.427740e+01
 * time: 0.009342193603515625
    17     1.890461e+01     2.054031e+01
 * time: 0.009797096252441406
    18     1.805151e+01     1.411743e+01
 * time: 0.01026010513305664
    19     1.765438e+01     4.276387e+00
 * time: 0.010717153549194336
    20     1.749679e+01     4.669042e+00
 * time: 0.011185169219970703
    21     1.726469e+01     5.241416e+00
 * time: 0.011641979217529297
    22     1.523866e+01     6.437334e+01
 * time: 0.01224207878112793
    23     1.469960e+01     1.955791e+01
 * time: 0.012830018997192383
    24     1.397284e+01     1.424796e+01
 * time: 0.013323068618774414
    25     1.382833e+01     8.680282e+00
 * time: 0.013880014419555664
    26     1.374761e+01     5.561786e+00
 * time: 0.014428138732910156
    27     1.355864e+01     7.085081e+00
 * time: 0.014999151229858398
    28     1.314458e+01     2.063312e+01
 * time: 0.015542984008789062
    29     1.205917e+01     4.811438e+01
 * time: 0.016129016876220703
    30     1.034827e+01     1.045833e+02
 * time: 0.016703128814697266
    31     4.222913e+00     6.238413e+01
 * time: 0.017435073852539062
    32     3.194372e+00     5.267902e+01
 * time: 0.018465042114257812
    33     7.800478e-01     3.803726e+01
 * time: 0.019214153289794922
    34    -4.236905e+00     1.611463e+02
 * time: 0.020219087600708008
    35    -8.976253e+00     7.865685e+01
 * time: 0.02089405059814453
    36    -9.751292e+00     1.095163e+01
 * time: 0.021548032760620117
    37    -1.028967e+01     1.947071e+01
 * time: 0.022114038467407227
    38    -1.047514e+01     1.177636e+01
 * time: 0.02266716957092285
    39    -1.051073e+01     1.202394e+01
 * time: 0.02321600914001465
    40    -1.052710e+01     1.036226e+01
 * time: 0.023756980895996094
    41    -1.063664e+01     3.401984e+00
 * time: 0.024346113204956055
    42    -1.074045e+01     2.892187e+01
 * time: 0.024918079376220703
    43    -1.085738e+01     1.051008e+01
 * time: 0.025474071502685547
    44    -1.100875e+01     4.221376e+00
 * time: 0.026014089584350586
    45    -1.101963e+01     2.837064e+00
 * time: 0.026551008224487305
    46    -1.102003e+01     9.801380e-01
 * time: 0.027080059051513672
    47    -1.102048e+01     1.533013e-01
 * time: 0.02760910987854004
    48    -1.102050e+01     6.436902e-02
 * time: 0.028164148330688477
    49    -1.102050e+01     4.090068e-03
 * time: 0.028695106506347656
    50    -1.102050e+01     2.193585e-03
 * time: 0.029229164123535156
    51    -1.102050e+01     3.446624e-04
 * time: 0.02975916862487793
FittedPumasModel

Dynamical system type:          Matrix exponential

Number of subjects:                              1

Observation records:         Active        Missing
    dv:                          61              0
    Total:                       61              0

Number of parameters:      Constant      Optimized
                                  0              6

Likelihood approximation:              NaivePooled
Likelihood optimizer:                         BFGS

Termination Reason:                   GradientNorm
Log-likelihood value:                    11.020497

---------------
     Estimate
---------------
θ₁    30.943
θ₂   214.07
θ₃    73.489
θ₄   169.26
θ₅    12.056
σ      0.13904
---------------

Now let’s display the estimates of θ side by side:

hcat(coef(fpm1).θ, coef(fpm2).θ)
5×2 Matrix{Float64}:
  34.3968     30.9432
 103.168     214.072
   0.465658   73.4888
 204.442     169.263
  26.2337     12.0563

Notice the significant difference in the final estimates when changing nothing but the initial estimates. Also note that the 2 sets of coefficients are not in the same neighbourhood and don’t have similar log likelihoods. This is indicative of the existence of multiple dis-connected local optima.

Example: Sensitivity to Noise Level

To demonstrate the sensitivity to noise level, we will re-simulate the synthetic subject with the same seed but using a higher σ.

Random.seed!(12345)
newparams = (; θ = params.θ, σ = 0.2)
newpop = [Subject(simobs(model, skeleton, newparams))]
Population
  Subjects: 1
  Observations: dv

Now let’s do the fit once with pop and once with newpop:

fpm1 = fit(model, pop, params, NaivePooled())
[ Info: Checking the initial parameter values.
[ Info: The initial negative log-likelihood and its gradient are finite. Check passed.
Iter     Function value   Gradient norm 
     0    -3.282510e+01     5.896826e+01
 * time: 1.8835067749023438e-5
     1    -3.312144e+01     4.134793e+01
 * time: 0.0007789134979248047
     2    -3.410517e+01     2.822025e+01
 * time: 0.0013589859008789062
     3    -3.506276e+01     2.569455e+01
 * time: 0.001968860626220703
     4    -3.577916e+01     2.022773e+01
 * time: 0.0025680065155029297
     5    -3.600599e+01     1.161060e+01
 * time: 0.0031599998474121094
     6    -3.604984e+01     1.270240e+01
 * time: 0.0036590099334716797
     7    -3.607833e+01     1.436656e+00
 * time: 0.0041239261627197266
     8    -3.607861e+01     2.182965e-01
 * time: 0.004602909088134766
     9    -3.607863e+01     4.628923e-02
 * time: 0.00509190559387207
    10    -3.607864e+01     2.722381e-02
 * time: 0.005581855773925781
    11    -3.607869e+01     1.854695e-01
 * time: 0.006101846694946289
    12    -3.607871e+01     1.841239e-01
 * time: 0.006599903106689453
    13    -3.607872e+01     6.087452e-02
 * time: 0.0070989131927490234
    14    -3.607872e+01     2.801100e-03
 * time: 0.0075588226318359375
    15    -3.607872e+01     2.888554e-04
 * time: 0.008031845092773438
FittedPumasModel

Dynamical system type:          Matrix exponential

Number of subjects:                              1

Observation records:         Active        Missing
    dv:                          61              0
    Total:                       61              0

Number of parameters:      Constant      Optimized
                                  0              6

Likelihood approximation:              NaivePooled
Likelihood optimizer:                         BFGS

Termination Reason:                   GradientNorm
Log-likelihood value:                    36.078723

----------------
     Estimate
----------------
θ₁    34.397
θ₂   103.17
θ₃     0.46566
θ₄   204.44
θ₅    26.234
σ      0.092273
----------------
fpm2 = fit(model, newpop, newparams, NaivePooled())
[ Info: Checking the initial parameter values.
[ Info: The initial negative log-likelihood and its gradient are finite. Check passed.
Iter     Function value   Gradient norm 
     0     8.763732e+00     2.735810e+01
 * time: 1.5974044799804688e-5
     1     8.412271e+00     2.872564e+01
 * time: 0.0007219314575195312
     2     7.854724e+00     2.512209e+01
 * time: 0.0013110637664794922
     3     6.568123e+00     2.332059e+01
 * time: 0.0019049644470214844
     4     5.591452e+00     1.244045e+01
 * time: 0.002502918243408203
     5     4.983159e+00     5.673971e+00
 * time: 0.003097057342529297
     6     4.869556e+00     1.443635e+00
 * time: 0.003570079803466797
     7     4.855068e+00     2.391983e+00
 * time: 0.004036903381347656
     8     4.854320e+00     3.036248e-01
 * time: 0.004498958587646484
     9     4.854166e+00     8.437796e-01
 * time: 0.004962921142578125
    10     4.853745e+00     3.065731e-01
 * time: 0.0054340362548828125
    11     4.853406e+00     3.478444e-01
 * time: 0.0059010982513427734
    12     4.851733e+00     9.959541e-01
 * time: 0.006365060806274414
    13     4.850649e+00     1.410163e+00
 * time: 0.006858110427856445
    14     4.849757e+00     6.215710e-01
 * time: 0.007339000701904297
    15     4.849444e+00     3.065089e-01
 * time: 0.0078029632568359375
    16     4.849275e+00     1.952213e-01
 * time: 0.008278131484985352
    17     4.849185e+00     1.946687e-01
 * time: 0.008749008178710938
    18     4.849164e+00     4.150091e-02
 * time: 0.009340047836303711
    19     4.849152e+00     1.148130e-01
 * time: 0.009824037551879883
    20     4.849150e+00     2.840234e-02
 * time: 0.010308027267456055
    21     4.849150e+00     5.940655e-03
 * time: 0.010783910751342773
    22     4.849149e+00     9.732258e-04
 * time: 0.011276006698608398
FittedPumasModel

Dynamical system type:          Matrix exponential

Number of subjects:                              1

Observation records:         Active        Missing
    dv:                          61              0
    Total:                       61              0

Number of parameters:      Constant      Optimized
                                  0              6

Likelihood approximation:              NaivePooled
Likelihood optimizer:                         BFGS

Termination Reason:                   GradientNorm
Log-likelihood value:                   -4.8491495

---------------
     Estimate
---------------
θ₁    33.85
θ₂   125.51
θ₃     0.5001
θ₄   192.8
θ₅    22.25
σ      0.18103
---------------

Now let’s display the estimates of θ side by side:

hcat(coef(fpm1).θ, coef(fpm2).θ)
5×2 Matrix{Float64}:
  34.3968     33.8498
 103.168     125.511
   0.465658    0.5001
 204.442     192.802
  26.2337     22.25

Given that we have many observations per subject, this level of fluctuation due to a higher noise is a symptom of non-identifiability. Also note the big difference compared to the true data-generating parameter values!

params.θ
5-element Vector{Float64}:
  35.0
 100.0
   0.5
 210.0
  30.0
NoteSummary

In this section, we have demonstrated beyond reasonable doubt that ML estimation workflows can be unreliable when fitting poorly identifiable models, potentially leading to erroneous conclusions in a study. So if we cannot rely on ML estimation for fitting poorly identifiable models, what can we use? The answer is Bayesian inference.

3.2 Bayesian Inference

Using Bayesian methods to sample from the posterior of the parameter estimates is a mathematically sound way to fit non-identifiable models because even non-identifiable models have a well-defined posterior distribution, when their parameters are assigned prior distributions. If multiple parameter values all fit the data well, then all such values will be plausible samples from the posterior distribution, assuming reasonable priors were used.

3.2.1 Model Definition

Let’s see how to minimally change our model above to make it Bayesian using a weakly informative priors “roughly in the ballpark”:

bayes_model = @model begin
    @param begin
        θ ~ MvLogNormal([log(50), log(150), log(1.0), log(100.0), log(10.0)], I(5))
        σ ~ Uniform(0.0, 1.0)
    end
    @pre begin
        CL = θ[1]
        Vc = θ[2]
        Ka = θ[3]
        Vp = θ[4]
        Q = θ[5]
    end
    @dynamics begin
        Depot' = -Ka * Depot
        Central' = Ka * Depot - (CL + Q) / Vc * Central + Q / Vp * Peripheral
        Peripheral' = Q / Vc * Central - Q / Vp * Peripheral
    end
    @derived begin
        cp := @. Central / Vc
        dv ~ @. CombinedNormal(cp, 1e-6, σ)
    end
end
PumasModel
  Parameters: θ, σ
  Random effects:
  Covariates:
  Dynamical system variables: Depot, Central, Peripheral
  Dynamical system type: Matrix exponential
  Derived: dv
  Observed: dv

3.2.2 Sampling from the Posterior

Now to fit it, we need to pass an instance of BayesMCMC as the algorithm in fit. In this case, we used 4 chains for sampling with 3000 samples per chain out of which the first 1500 samples will be used to adapt the mass matrix and step size of the No-U-Turn sampler (NUTS) used in Pumas. All the chains are also parallelized using multi-threading.

# Setting the pseudo-random number generator's seed for reproducibility
Random.seed!(54321)
bayes_alg = BayesMCMC(; nsamples = 3_000, nadapts = 1_500)
bayes_fpm = fit(bayes_model, pop, params, bayes_alg)
[ Info: Checking the initial parameter values.
[ Info: The initial log probability and its gradient are finite. Check passed.
[ Info: Checking the initial parameter values.
[ Info: The initial log probability and its gradient are finite. Check passed.
[ Info: Checking the initial parameter values.
[ Info: The initial log probability and its gradient are finite. Check passed.
[ Info: Checking the initial parameter values.
[ Info: The initial log probability and its gradient are finite. Check passed.
Chains MCMC chain (3000×6×4 Array{Float64, 3}):

Iterations        = 1:1:3000
Number of chains  = 4
Samples per chain = 3000
Wall duration     = 62.27 seconds
Compute duration  = 236.14 seconds
parameters        = θ₁, θ₂, θ₃, θ₄, θ₅, σ

Use `describe(chains)` for summary statistics and quantiles.

Now let’s discard the NUTS warmup samples as burn-in:

bayes_fpm_samples = discard(bayes_fpm; burnin = 1500)
Chains MCMC chain (1500×6×4 Array{Float64, 3}):

Iterations        = 1:1:1500
Number of chains  = 4
Samples per chain = 1500
Wall duration     = 62.27 seconds
Compute duration  = 236.14 seconds
parameters        = θ₁, θ₂, θ₃, θ₄, θ₅, σ

Use `describe(chains)` for summary statistics and quantiles.
describe(bayes_fpm_samples)
Chains MCMC chain (1500×6×4 Array{Float64, 3}):

Iterations        = 1:1:1500
Number of chains  = 4
Samples per chain = 1500
Wall duration     = 62.27 seconds
Compute duration  = 236.14 seconds
parameters        = θ₁, θ₂, θ₃, θ₄, θ₅, σ

Summary Statistics

  parameters       mean       std      mcse    ess_bulk    ess_tail      rhat  ⋯
      Symbol    Float64   Float64   Float64     Float64     Float64   Float64  ⋯

          θ₁    34.2575    0.6505    0.0128   2626.4786   2839.8792    1.0008  ⋯
          θ₂   139.1009   34.6161    1.0403   1100.1141   2523.5763    1.0019  ⋯
          θ₃     0.6520    0.1826    0.0054   1113.4952   2474.7101    1.0018  ⋯
          θ₄   189.9844   18.6254    0.4592   1686.6339   2907.6474    1.0010  ⋯
          θ₅    23.2338    2.9658    0.0790   1439.0114   2139.5452    1.0013  ⋯
           σ     0.0983    0.0099    0.0002   3100.5374   3237.8020    1.0006  ⋯

                                                                1 column omitted

Quantiles

  parameters       2.5%      25.0%      50.0%      75.0%      97.5% 
      Symbol    Float64    Float64    Float64    Float64    Float64 

          θ₁    32.9460    33.8310    34.2608    34.6875    35.5165
          θ₂    71.1514   113.0744   143.1076   166.0241   196.8979
          θ₃     0.3492     0.5067     0.6486     0.7813     1.0120
          θ₄   158.9525   176.6386   187.5311   201.3261   232.9174
          θ₅    17.3257    21.0883    23.3721    25.4154    28.6471
           σ     0.0813     0.0912     0.0977     0.1042     0.1199

Notice how Bayesian inference was able to quantify the uncertainty in the non-identifiable parameters (mostly θ₂ which is Vc and θ₄ which is Vp) reflected in the large standard deviation of the marginal posterior of some parameters, std, relative to its mean value, mean. This is consistent with the eigenvector d we used earlier to prove that the model is poorly identifiable which also showed the poor identifiability was mostly prevalent in these 2 parameters.

d
6-element Vector{Float64}:
 -0.002380469921277704
 -0.8754610365061731
 -0.0037426498922106567
  0.4829767082714525
  0.016786862766484546
  8.218940732069106e-8

3.2.3 Convergence Diagnostics

By default, we print summary statistics and a few convergence diagnostics: effective samples size (ess_bulk) and \(\hat{R}\) (rhat). In this case, the diagnostics look reasonable. The rhat is close to 1 and the minimum ess_bulk is around 1000.

Next we show the trace plot of all the parameters

using PumasPlots

trace_plot(bayes_fpm_samples)

It can be seen that the chains are mostly well mixed. Occasional jumps seem to happen which may indicate the presence of another mode in the posterior with a relatively small probability mass.

Now let’s look at the auto-correlation plot:

autocor_plot(bayes_fpm_samples)

Some auto-correlation seems to persist in some chains so let’s try some thinning, by keeping only one out of every 5 samples.

thin_bayes_fpm_samples = discard(bayes_fpm; burnin = 1500, ratio = 0.2)

autocor_plot(thin_bayes_fpm_samples)

Looks better!

NoteTo Thin Or Not To Thin

In general, thinning is not recommended unless there are extremely high levels of auto-correlation in the samples. This is because the thinned samples will always have less information than the full set of samples before thinning. However, we perform thinning in this tutorial for demonstration purposes.

3.2.4 Posterior Predictive Check

Now let’s do a posterior predictive check by first simulating 1000 scenarios from the posterior distribution of the response including residual noise.

obstimes = 0.0:0.5:30.0
sims = simobs(
    bayes_model,
    pop,
    sample(bayes_fpm_samples, 1000)...;
    simulate_error = true,
    obstimes,
)
1000-element Vector{Vector{Pumas.SimulatedObservations{PumasModel{(θ = 5, σ = 1), 0, (:Depot, :Central, :Peripheral), (Symbol("##A##"), Symbol("##b##")), ParamSet{@NamedTuple{θ::MvLogNormal{Float64, PDMats.PDiagMat{Float64, Vector{Float64}}, Vector{Float64}}, σ::Uniform{Float64}}}, ParamSet{@NamedTuple{}}, Pumas.PreObj{var"#23#24"}, Pumas.DCPObj{Returns{Returns{@NamedTuple{}}}}, var"#25#26", Pumas.LinearODE, Pumas.DerivedObj{(:dv,), (:Vc,), (:Central,), (), var"#27#28"}, var"#29#30", ModelingToolkitBase.System, Pumas.PumasModelOptions{Pumas.AllSteadyStateStates}}, Subject{@NamedTuple{dv::Vector{Float64}}, Pumas.ConstantCovar{@NamedTuple{}}, DosageRegimen{Float64, Symbol, Float64, Float64, Float64, Float64}, Vector{Float64}}, Float64, @NamedTuple{θ::Vector{Float64}, σ::Float64}, @NamedTuple{}, @NamedTuple{}, @NamedTuple{saveat::Vector{Float64}, ss_abstol::Float64, ss_reltol::Float64, ss_maxiters::Int64, abstol::Float64, reltol::Float64, alg::CompositeAlgorithm{0, Tuple{Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}}, OrdinaryDiffEqCore.AutoSwitch{Vern7{typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!), Static.False, Val{true}}, Rodas5P{1, ADTypes.AutoForwardDiff{1, Nothing}, LinearSolve.LUFactorization{RowMaximum}, typeof(OrdinaryDiffEqCore.DEFAULT_PRECS), Val{:forward}(), true, nothing, typeof(OrdinaryDiffEqCore.trivial_limiter!), typeof(OrdinaryDiffEqCore.trivial_limiter!)}, Rational{Int64}, Int64}}, continuity::Symbol}, @NamedTuple{Depot::Vector{Float64}, Central::Vector{Float64}, Peripheral::Vector{Float64}}, Pumas.PKPDAnalyticalSolution{StaticArraysCore.SVector{3, Float64}, 2, Vector{StaticArraysCore.SVector{3, Float64}}, Vector{Float64}, Pumas.SplitInterp{@NamedTuple{var"##A##"::StaticArraysCore.SMatrix{3, 3, Float64, 9}, var"##b##"::StaticArraysCore.SVector{3, Int64}}, Returns{@NamedTuple{}}, Pumas.ConstantAssembler}, Pumas.AnalyticalPKPDProblem{StaticArraysCore.SVector{3, Float64}, Float64, false, Pumas.LinearODE, Vector{Pumas.Event{Float64, Float64, Float64}}, Vector{Float64}, Pumas.SplitInterp{@NamedTuple{var"##A##"::StaticArraysCore.SMatrix{3, 3, Float64, 9}, var"##b##"::StaticArraysCore.SVector{3, Int64}}, Returns{@NamedTuple{}}, Pumas.ConstantAssembler}, Pumas.AllSteadyStateStates}}, @NamedTuple{}, @NamedTuple{CL::Vector{Float64}, Vc::Vector{Float64}, Ka::Vector{Float64}, Vp::Vector{Float64}, Q::Vector{Float64}, var"##A##"::Vector{StaticArraysCore.SMatrix{3, 3, Float64, 9}}, var"##b##"::Vector{StaticArraysCore.SVector{3, Int64}}}, @NamedTuple{dv::Vector{Float64}}, @NamedTuple{}}}}:
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 ⋮
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv
 Simulated population (Vector{<:Subject}), n = 1, variables: dv

then we can do a visual predictive check (VPC) plot using the simulations

vpc_res = vpc(sims; observations = [:dv], ensemblealg = EnsembleThreads())

vpc_plt = vpc_plot(
    vpc_res;
    simquantile_medians = true,
    observations = true,
    axis = (xlabel = "Time (h)", ylabel = "Concentration", xticks = 0:2:30),
)
[ Info: Detected 1000 scenarios and 1 subjects in the input simulations. Running VPC.
[ Info: Continuous VPC

NoteSummary

With very few changes in the model and a few lines of code, we were able to obtain samples from the full posterior of the parameters of our poorly identifiable model.

4 Uncertainty Propagation, Queries and Decision Making

Just because a model is non-identifiable does not mean that the model is useless or less correct. In fact, more correct models that incorporate more biological processes tend to be non-identifiable because we can only observe/measure very few variables in the model, while simplified models are more likely to be identifiable. Given samples from the posterior of a non-identifiable model, one can do the following:

  • Propagate the uncertainty forward to the predictions to get samples from the posterior predictive distribution, instead of relying on a single prediction using the ML estimates.
NoteValue of Uncertainty Quantification

Parameter uncertainty due to structural non-identifiability will by definition have no effect on the model’s predictions when predicting the observed response. However, uncertainty due to practical non-identifiability, or otherwise insufficient data, can have an impact on the model’s predictions. In this case, basing decisions on the full posterior predictive distribution instead of a single prediction from the ML estimates will make the decisions more robust to parameter uncertainty due to insufficient observations and model misspecification.

  • Ask probabilistic questions given your data. For example, what’s the probability that the drug effect is \(> 0\)? Or what’s the probability that the new drug A is better than the control drug B after only 3 months of data? Or what’s the probability of satisfying a therapeutic criteria for efficacy and safety given the current dose?
  • What-if analysis (aka counter-factual simulation) and dose optimization. For example, you can make predictions assuming the subject is a pediatric using the model parameters’ posterior inferred from an adult’s data. Or you can test different dose levels to select the best dose according to some therapeutic criteria. This can also be done in the non-Bayesian setting.

4.1 Probabilistic Questions

In this section, we show how to

  1. Estimate the probability that a parameter is more than a specific value, and
  2. Estimate the probability that the subject satisfies a desired therapeutic criteria. From there, one can simulate multiple doses and choose the dose that maximizes this probability.

To estimate the probability that CL > 35, we can run:

mean(bayes_fpm_samples) do p
    p.θ[1] > 35
end
0.1245

To estimate the probability that a subject satisfies a desired therapeutic criteria, we first simulate from the posterior predictive distribution (without residual error):

_ipreds = simobs(
    bayes_model,
    pop,
    sample(bayes_fpm_samples, 1000)...;
    simulate_error = false,
    obstimes,
)
ipreds = reduce(vcat, _ipreds)
Simulated population (Vector{<:Subject})
  Simulated subjects: 1000
  Simulated variables: dv

Next, we can estimate the area-under-curve (auc) and maximum drug concentration (cmax) given the different posterior samples. NCA.jl works on tables of profiles, so we put every posterior sample into one table as a profile of its own and compute the endpoints for all of them in a single call.

using NCA

function auc_and_cmax(ipreds, dose)
    profiles = [
        DataFrame(; id = i, time = obstimes, concentration = ipred.observations.dv) for
        (i, ipred) in enumerate(ipreds)
    ]
    nca_df = reduce(vcat, profiles)
    insertcols!(nca_df, :dose => dose, :route => "ev")
    run_nca(read_nca(nca_df), [:aucinf_obs, :cmax])
end

nca_params = auc_and_cmax(ipreds, 3000)
1000×9 DataFrame
975 rows omitted
Row id profile dose route duration tau assumed_tau aucinf_obs cmax
Int64 Int64 Float64 String Missing Missing Missing Float64 Float64
1 1 1 3000.0 ev missing missing missing 90.2852 10.1255
2 2 2 3000.0 ev missing missing missing 88.4059 10.4894
3 3 3 3000.0 ev missing missing missing 85.1699 9.07835
4 4 4 3000.0 ev missing missing missing 88.5137 9.71473
5 5 5 3000.0 ev missing missing missing 85.8443 9.10036
6 6 6 3000.0 ev missing missing missing 87.8149 9.87364
7 7 7 3000.0 ev missing missing missing 88.4382 10.1677
8 8 8 3000.0 ev missing missing missing 86.5894 9.61153
9 9 9 3000.0 ev missing missing missing 93.0069 10.4824
10 10 10 3000.0 ev missing missing missing 85.786 10.0023
11 11 11 3000.0 ev missing missing missing 88.5882 9.61268
12 12 12 3000.0 ev missing missing missing 89.5232 9.40283
13 13 13 3000.0 ev missing missing missing 86.6169 9.59954
⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
989 989 989 3000.0 ev missing missing missing 85.1625 9.73993
990 990 990 3000.0 ev missing missing missing 84.7075 9.4949
991 991 991 3000.0 ev missing missing missing 86.6139 9.68615
992 992 992 3000.0 ev missing missing missing 88.4034 9.96856
993 993 993 3000.0 ev missing missing missing 87.292 9.62792
994 994 994 3000.0 ev missing missing missing 87.4482 9.70066
995 995 995 3000.0 ev missing missing missing 90.532 9.73284
996 996 996 3000.0 ev missing missing missing 87.0965 9.59861
997 997 997 3000.0 ev missing missing missing 86.6931 9.88242
998 998 998 3000.0 ev missing missing missing 85.9897 9.66493
999 999 999 3000.0 ev missing missing missing 86.8469 9.17577
1000 1000 1000 3000.0 ev missing missing missing 89.3014 10.4734

The result is a DataFrame with one row per posterior sample, so the probability of satisfying a therapeutic criteria is the fraction of its rows that satisfy the criteria.

prob = mean((nca_params.aucinf_obs .> 90) .& (nca_params.cmax .< 15.0))
0.039

To compute the probability of efficacy and safety separately, one can instead run:

prob1 = mean(nca_params.aucinf_obs .> 90)
prob2 = mean(nca_params.cmax .< 15.0)
prob1, prob2
(0.039, 1.0)

4.2 Counter-factual Analysis and Dose Optimization

After developing a model, one may be interested in simulating scenarios, e.g. different covariates or doses, that have not been observed in the data while reusing the same posterior distribution of the parameters learnt from the data. This can be used to select a new dose that maximizes the probabilities of efficacy and safety simultaneously given the previously observed data.

In Pumas, you can do this by defining a new subject that includes the new covariates or dose and then passing that to simobs. First, let’s define a new skeleton subject that represents our counter-factual scenario where a dose of 3200 was administered instead of the 3000 used in the observed case:

cf_skeleton = Subject(pop[1]; events = DosageRegimen(3200; time = 0.0, cmt = 1))
Subject
  ID: 1
  Events: 1
  Observations: dv: (n=61)

To simulate from the posterior predictive distribution of this new subject using the posterior of the parameters, you can run:

cf_ipreds = simobs(
    bayes_model,
    cf_skeleton,
    sample(thin_bayes_fpm_samples, 1000; subject = 1)...;
    simulate_error = false,
    obstimes,
)
Simulated population (Vector{<:Subject})
  Simulated subjects: 1000
  Simulated variables: dv

We can then re-evaluate the probability of satisfying a therapeutic criteria:

cf_nca_params = auc_and_cmax(cf_ipreds, 3200)
cf_prob = mean((cf_nca_params.aucinf_obs .> 90) .& (cf_nca_params.cmax .< 15.0))
0.958

We can see that the probability increased. To understand why, let’s look at the probabilities of the auc and cmax criteria separately:

cf_prob1 = mean(cf_nca_params.aucinf_obs .> 90)
cf_prob2 = mean(cf_nca_params.cmax .< 15.0)
cf_prob1, cf_prob2
(0.958, 1.0)

Contrast this to the old dose’s probabilities:

prob1, prob2
(0.039, 1.0)

Note that this example was not particularly interesting because of the dense sampling which despite of it, the uncertainty in the parameters was still high. This implies that the uncertainty was largely due to structural identifiability issues in the model. Since uncertainty in parameters due to structural non-identifiability does not affect the model predictions, the posterior predictive distribution was much more concentrated than the parameters’ posterior.

To estimate the mean and standard deviation of the predictions at each point in time we can run:

μs = mean.(postprocess((gen, _) -> gen.dv, ipreds))
σs = std.(postprocess((gen, _) -> gen.dv, ipreds))
61-element Vector{Float64}:
 0.0
 0.3259164494841519
 0.38989187317240526
 0.36671138635867945
 0.3306589582111061
 0.3023647961504641
 0.27992143364492933
 0.2580605856773273
 0.23457981523675434
 0.20992484210368284
 ⋮
 0.014000613299634783
 0.014313792598618764
 0.014680738236213698
 0.015086003633322612
 0.015516139156839363
 0.015959775792931872
 0.016407525261302153
 0.016851783396820946
 0.017286496387976663

To get the average relative standard deviations (ignoring the first prediction which is 0), we run:

mean(σs[2:end] ./ μs[2:end])
0.025554161276741743

So the predictions are not very sensitive to the parameter uncertainty.

5 Summary

In this tutorial, we have seen how to test for model non-identifiability using the Fisher information matrix and sensitivity analysis. We have shown that maximum likelihood estimation is unreliable when fitting poorly identifiable models. And we have seen how one can use Bayesian inference to: 1) fit non-identifiable or poorly identifiable models to data, 2) ask probabilistic questions of the model, and 3) simulate counter-factual scenarios.

6 References

  1. Thomas J. Rothenberg. Identification in parametric models. Econometrica, 1971.
  2. F. Mentre, A Mallet, and D. Baccar. Optimal design in random-effects regression models. Biometrika, 1997.
  3. S. Retout and F Mentre. Further development of the fisher information matrix in nonlinear mixed-effects models with evaluation in population pharmacokinetics. Journal of biopharmaceutical statistics, 2003.
  4. V. Shivva, K. Korell, I. Tucker, and S. Duffull. An approach for identifiability of population pharmacokinetic-pharmacodynamic models. CPT Pharmacometrics & Systems Pharmacology, 2013.
  5. Stephen Dufful, A workflow for resolving model internal consistency in use-reuse settings (aka repairing unstable models). PAGANZ, 2024.
  6. Dan Wright. The identifiability of a turnover model for allopurinol urate-lowering effect. PAGANZ, 2024.