A Quick Run-through of Pumas Bioequivalence Outputs
Authors
Yoni Nazarathy
Andreas Noack
This tutorial presents a quick overview of the types of bioequivalence outputs you can expect when using Pumas products. See other Pumas bioequivalence tutorials for an in-depth discussion of bioequivalence topics.
1 Overview
The Bioequivalence package in Pumas provides out of the box, simple to use, bioequivalence analysis. In general, you need a dataset arising from a bioequivalence trial in the form of a Julia DataFrame, and you can then apply the easy to use pumas_be function on that dataset. For example, here is an example dataset appearing in Park, Kim, and Bae (2020).
This dataset consists of Cmax measurements over 6 subjects with a \(2 \times 2\) crossover design. Observe that in this 2 period design, subject number 3 and subject number 6 both have missing measurements. For these subjects there is data for only a single period, while the other subjects have data for both periods.
To carry out an average bioequivalence analysis on this dataset all that is needed is to apply pumas_be as follows.
Here the first argument to pumas_be is the DataFrame. The keyword argument endpoint specifies which column in the dataframe is the endpoint. Note that with standard column names such as :id, :sequence, and :period, no further details are needed. Yet, if the dataframe would have had a different structure, then the specific columns to use could be specified. Use ? pumas_be to see the options.
As you can see the output of pumas_be first contains a small formatted table that indicates how many subjects are in each sequence and each period. Then a formatted table with the statistical results appears. In this tutorial we briefly survey the contents of such formatted output.
In general, you can also assign the result of pumas_be to a Julia variable or object, for example,
Here be_result can then be queried for additional information, yet this is rarely needed. Note that in the above code we suppressed the output with ;.
When you use a terminal, the output will not appear exactly as above, but will rather be formatted textually and appear as follows (you may need to scroll left and right on the text below to see all of it):
sequence ╲ period │ 1 2
──────────────────┼─────
RT │ 3 2
TR │ 3 2
-----------------------------------------------------------------------------------------------------------------
Paradigm: Non replicated crossover design
Model: Linear model
Criteria: Standard ABE
Endpoint: cmax
Formulations: Reference(R), Test(T)
-----------------------------------------------------------------------------------------------------------------
Results(cmax) Assessment Criteria
-----------------------------------------------------------------------------------------------------------------
R Geometric Marginal Mean 160.5
Geometric Naive Mean 165.2
-----------------------------------------------------------------------------------------------------------------
T Geometric Marginal Mean 139.8
Geometric Naive Mean 147.9
Geometric Mean T/R Ratio (%) 87.08
Degrees of Freedom 2
90% Confidence Interval (%) [55.16, 137.5] Fail CI ⊆ [80, 125]
-----------------------------------------------------------------------------------------------------------------
Variability CV (%) | σ̂ 22.39 | 0.2212
-----------------------------------------------------------------------------------------------------------------
ANOVA Formulation (p-value) 0.4698
Sequence (p-value) 0.2088
Period (p-value) 0.4684
-----------------------------------------------------------------------------------------------------------------
This is a text/plain output format in contrast to a text/html format. The former appears when you use a Julia REPL (Read Evaluate Print Loop), as when using Pumas via JuliaHub or other means. The latter, a more stylized SummaryTables format, appears when you use PumasCP. It will also appear if you use a Pluto notebook, a Jupyter notebook, or Quarto.
Throughout this tutorial we present the outputs in both formats in a dynamic tabs such as this. The default tab, labeled HTML is what you will see in PumasCP, Pluto, Quarto, or Jupyter outputs. The Terminal tab uses code such as show(stdout, MIME"text/plain"()... to emulate terminal output in Quarto (as this tutorial is created in Quarto).
sequence ╲ period │ 1 2
──────────────────┼─────
RT │ 3 2
TR │ 3 2
-----------------------------------------------------------------------------------------------------------------
Paradigm: Non replicated crossover design
Model: Linear model
Criteria: Standard ABE
Endpoint: cmax
Formulations: Reference(R), Test(T)
-----------------------------------------------------------------------------------------------------------------
Results(cmax) Assessment Criteria
-----------------------------------------------------------------------------------------------------------------
R Geometric Marginal Mean 160.5
Geometric Naive Mean 165.2
-----------------------------------------------------------------------------------------------------------------
T Geometric Marginal Mean 139.8
Geometric Naive Mean 147.9
Geometric Mean T/R Ratio (%) 87.08
Degrees of Freedom 2
90% Confidence Interval (%) [55.16, 137.5] Fail CI ⊆ [80, 125]
-----------------------------------------------------------------------------------------------------------------
Variability CV (%) | σ̂ 22.39 | 0.2212
-----------------------------------------------------------------------------------------------------------------
ANOVA Formulation (p-value) 0.4698
Sequence (p-value) 0.2088
Period (p-value) 0.4684
-----------------------------------------------------------------------------------------------------------------
Note that in a terminal you can still create a SummaryTables format by using the result_summary_table function.
1.1 Getting data
In general you can read in dataframes using the CSV package, yet in this tutorial we use Pumas’ PharmaDatasets package to get example dataframes of bioequivalanece studies.
Here is one way to access the example bioequivalence datasets supplied via that package (we use the datasets function to list all datasets and then filter to only keep the bioequivalence ones). As you can see there are 65 such datasets.
Finally once you have chosen which dataset you want, such as for example bioequivalence/RTTR_TRRT/PJ2017_4_3, you can retrieve it via the dataset function. This is equivalent to reading a CSV file with Julia’s CSV package (see for example this Pumas Tutorial).
data =dataset("bioequivalence/RTTR_TRRT/PJ2017_4_3");
Note that it is also sometimes common to use joinpath("bioequivalence", "RTRT_TRTR", "PJ2017_4_4") in place of the path string "bioequivalence/RTTR_TRRT/PJ2017_4_3".
Here were show how many rows are in the dataset and present the first few rows:
println("Number of rows: ", nrow(data))simple_table(first(data, 10))
Bioequivalence endpoint data is typically assumed to be log-normally distributed. Hence to carry out analysis using the typical statistical machinery, a log transformation is needed.
Note that pumas_be carries out this log transformation internally by default. Hence you do not need to carry out this log transformation yourself.
If your data already happens to be log transformed, then to let pumas_be know not to carry out the transformation, specify logtransformed = true as follows:
Observe that in this case, the data is not log transformed and hence we did not need to specify true, but should have rather stayed with the default false setting. With this erroneous specification of logtransformed = true, Pumas does not log transform the data. As a result, we are presented with nonsensical numerical output involving very large numbers.
2 Outputs for various paradigms
Let us now get a basic understanding of bioequivalence output in the various bioequivalence paradigms that are supported by Pumas. In particular, different types of bioequivalence trials have different experimental designs, and sometimes use different types of statistical tests.
Pumas supports all of the popular paradigms, and these are:
Non replicated crossover design - This is perhaps the most common type of bioequivalence trial and is analyzed with “average bioequivalence” (ABE) using a linear model.
Multiple treatments - Similarly to the previous paradigm, here multiple test formulations are considered. Pumas supports 2, or 3 test formulations (hence 3 or 4 total formulations).
Parallel design - This is a single period design which involves the simplest use of a TOST (Two One Sided t-test) procedure.
Two period parallel design - This is a two period design, repeating the treatment over both periods. It is a form of parallel design that allows to estimate the within subject variability both for test and reference.
Nonparametric - When the endpoint is Tmax, the time point at which concentration is maximal, there are often observations from a small discrete set. In this case a nonparametric test is considered, and we call this the nonparametric paradigm.
Fully-replicated design reference scaling for highly variable drugs - When highly variable drug products are considered (and the trial was specified as such a-priori), we can use reference scaling analysis. Here Pumas supports guidance specifications from multiple agencies including FDA and EMA. In this tutorial we focus only on FDA. The analysis adheres to the draft Guidance on Progesterone, FDA (2011).
Fully-replicated design reference scaling for NTID - Fully replicated trials are also useful for drugs that are classified as NTID (Narrow Therapeutic Index Drugs). Here Pumas also provides a reference scaling analysis as specified by the FDA, including the specific criteria for acceptance/not-acceptance of such trials. The analysis adheres to the draft guidance on Warfarin Sodium, FDA (2012).
We now show examples of each of these paradigms and comment on some basic attributes of the output. In most cases, pumas_be automatically detects the design and infers the paradigm that should be used. In some specific cases, we may specify other options, as we describe below.
2.1 Non replicated crossover design
data =dataset(joinpath("bioequivalence", "RT_TR", "PJ2017_3_1.csv"))be_result =pumas_be(data);
Here is a simple RT|TR, \(2 \times 2\) crossover design. Observe that when the endpoint name in the data is AUC there is no need to specify the endpoint as a keyword argument in pumas_be. Here is the output.
sequence ╲ period │ 1 2
──────────────────┼───────
RT │ 17 17
TR │ 15 15
-----------------------------------------------------------------------------------------------------------------
Paradigm: Non replicated crossover design
Model: Linear model
Criteria: Standard ABE
Endpoint: AUC
Formulations: Reference(R), Test(T)
-----------------------------------------------------------------------------------------------------------------
Results(AUC) Assessment Criteria
-----------------------------------------------------------------------------------------------------------------
R Geometric Marginal Mean 1879
Geometric Naive Mean 1887
-----------------------------------------------------------------------------------------------------------------
T Geometric Marginal Mean 1848
Geometric Naive Mean 1845
Geometric Mean T/R Ratio (%) 98.36
Degrees of Freedom 30
90% Confidence Interval (%) [94.07, 102.8] Pass CI ⊆ [80, 125]
-----------------------------------------------------------------------------------------------------------------
Variability CV (%) | σ̂ 10.52 | 0.1049
-----------------------------------------------------------------------------------------------------------------
ANOVA Formulation (p-value) 0.5334
Sequence (p-value) 0.1818
Period (p-value) 0.00235
-----------------------------------------------------------------------------------------------------------------
As you can see the output starts with a small table counting the number of observations within each sequence type and period. We then see a 5 line title, it is split into several sections. The title indicates the paradigm (Non replicated crossover design in this case), the type of statistical model (Linear model in this case), the criteria which may be reference scaling or other criteria (Standard ABE in this case), the endpoint (AUC), and finally the coding of the formulations (R for reference and T for test in this case).
Let us now specify the main output appearing below the title. Yet first let us highlight the most critical bit of output. The 90% confidence interval for the geometric mean ratio GMR is [94.07, 102.8] in this case. More on it below.
The first section, R is for the reference formulation. Here we see the Geometric Marginal Mean, of the reference formulation. This uses an approach similar to SAS’s LS-Means approach to estimate the mean of the endpoint for the reference formulation. See for example Searle, Speed, and Milliken (1980). In this section we also present the Geometric Naive Mean, which is simply based on averaging of the observations without using the fit linear model. The marginal mean and the naive mean would be the same if there are no missing observations or imbalanced data.
The second section, T is for the test formulation. This section presents the most important results. First it has the marginal mean and the naive mean just like in the R section. Importantly, the estimated Geometric Mean T/R Ratio is presented as a percentage. Being close to 100% implies having little difference between the reference and the test. The degrees of freedom used in the statistical test are presented, and most importantly, the 90% confidence interval for the geometric mean is presented. In this case [94.07, 102.8].
As you can see, this confidence interval is used in the Criteria which requires it to be a subset (maths notation \(\subseteq\)) of [80,125]. Since this is the case with this data, the Assessment column contains a Pass, indicating that this data would pass a bioequivalence trial under such criteria.
The next section, Variability, is all about estimation of the within subject variability. In the case of this model, we use the estimated residual standard deviation from the model, denoted \(\hat{\sigma}\) of the log transformed data. This is 0.1049 in for this dataset. When converted to a coefficient of variation and presented as a percentage, this value is 10.52.
Finally, we have the section ANOVA. It presents type III p-values for the various effects in the model.
2.2 Multiple treatments
When considering a dataset with multiple treatments (or formulations), we can encode the treatments in the dataset via any letters (or characters) of our choosing. For example, in this dataset the characters A through D are used.
If there are multiple reference formulations then guidances suggest testing each the test formulation each time with a single reference. So in that case pumas_be does not handle multiple formulations but rather leaves it to the user to filter out the data prior. However, the case of multiple test formulations is handled by pumas_be.
The pumas_be function always treats the lowest letter in alphabetical order as the reference treatment, and the others are test treatments. In fact, even in the two treatment case you can deviate from R and T if needed and the lower letter will be used as a reference. In any case, as you see in the output below, there is a description of what is reference (A) and what is test (B, C and D).
This dataset is from Example 4.6 of Patterson and Jones (2017). Note that in that example it is designed to compare two separate pairs of dosages. Here we treat it as a reference compared against three test formulations.
data =dataset(joinpath("bioequivalence", "ADBC_BACD_CBDA_DCAB", "PJ2017_4_6"))be_result =pumas_be(data);
sequence ╲ period │ 1 2 3 4
──────────────────┼───────────
ADBC │ 7 7 7 7
BACD │ 7 7 7 7
CBDA │ 7 7 7 7
DCAB │ 7 7 7 7
--------------------------------------------------------------------------------------------------------------------------
Paradigm: 4 formulations
Model: Linear model
Criteria: Standard ABE
Endpoint: AUC
Formulations: Reference(A), Test(B, C and D)
--------------------------------------------------------------------------------------------------------------------------
Results(AUC) Assessment Criteria
--------------------------------------------------------------------------------------------------------------------------
A Geometric Marginal Mean 367.9
Geometric Naive Mean 367.9
--------------------------------------------------------------------------------------------------------------------------
B Geometric Marginal Mean 369.7
Geometric Naive Mean 369.7
Geometric Mean B/A Ratio (%) 100.5
Degrees of Freedom 78
90% Confidence Interval (%) [94.7, 106.6] Pass CI ⊆ [80, 125]
--------------------------------------------------------------------------------------------------------------------------
C Geometric Marginal Mean 2985
Geometric Naive Mean 2985
Geometric Mean C/A Ratio (%) 811.4
Degrees of Freedom 78
90% Confidence Interval (%) [764.8, 860.8] Fail CI ⊆ [80, 125]
--------------------------------------------------------------------------------------------------------------------------
D Geometric Marginal Mean 2879
Geometric Naive Mean 2879
Geometric Mean D/A Ratio (%) 782.6
Degrees of Freedom 78
90% Confidence Interval (%) [737.6, 830.2] Fail CI ⊆ [80, 125]
--------------------------------------------------------------------------------------------------------------------------
Variability CV (%) | σ̂ 13.35 | 0.1329
--------------------------------------------------------------------------------------------------------------------------
If you compare the output above to the output appearing in Section 2.1, you’ll see that a similar pattern is used, yet there is now a section for each treatment, and there is no ANOVA section.
Since A is the reference, the output is similar in its organization to what we had in Section 2.1. In this case, as the are no missing observations we must have that the marginal means and naive means are the same.
As for the B, C, and D sections, one for each test formulation, we get a geometric mean ratio in each case, and just like in the two treatment case we can consider an average bioequivalence analysis in each case.
Observe that there is still a single linear model fit in this case, and the estimated variability is a function of that model, based on all formulations.
Now you may observe that in this example, formulations C and D fail with estimates and confidence intervals for the GMR way out of the desired range. In this dataset this is because they are with different dosage amounts which are in fact 8 times as high. For this we transform the data by dividing the AUC by 8 every time the formulation is C or D. This is done with the following code using @rtransform from the DataFramesMeta package.
As you can see, now the in this case, all formulations pass. Such a transformation obviously only needs to be done when the experimental setup merits it.
sequence ╲ period │ 1
──────────────────┼───
R │ 20
T │ 20
------------------------------------------------------------------------------------------------------------------
Paradigm: Parallel design
Model: T test (equal variance)
Criteria: Standard ABE
Endpoint: AUC
Formulations: Reference(R), Test(T)
------------------------------------------------------------------------------------------------------------------
Results(AUC) Assessment Criteria
------------------------------------------------------------------------------------------------------------------
R Geometric Naive Mean 1.378
------------------------------------------------------------------------------------------------------------------
T Geometric Naive Mean 0.9916
Geometric Mean T/R Ratio (%) 71.97
Degrees of Freedom 38
90% Confidence Interval (%) [38.59, 134.3] Fail CI ⊆ [80, 125]
------------------------------------------------------------------------------------------------------------------
Variability CV (%) | σ̂ 170.03 | 1.1656
------------------------------------------------------------------------------------------------------------------
Observe that in this case there are no marginal means estimated, only naive means. Further in this example the estimated confidence interval is not a subset of [80,125] and hence we fail to establish bioequivalence.
One option we may use is homogeneity = false (the default for parallel designs is true). When setting false we use the Welch–Satterthwaite approach to approximate the degrees of freedom in the t-test. As you can see below with this option, the point estimates did not change, but the degrees of freedom changed.
sequence ╲ period │ 1
──────────────────┼───
R │ 20
T │ 20
------------------------------------------------------------------------------------------------------------------
Paradigm: Parallel design
Model: T test (unequal variance)
Criteria: Standard ABE
Endpoint: AUC
Formulations: Reference(R), Test(T)
------------------------------------------------------------------------------------------------------------------
Results(AUC) Assessment Criteria
------------------------------------------------------------------------------------------------------------------
R Geometric Naive Mean 1.378
------------------------------------------------------------------------------------------------------------------
T Geometric Naive Mean 0.9916
Geometric Mean T/R Ratio (%) 71.97
Degrees of Freedom 19.99
90% Confidence Interval (%) [38.04, 136.2] Fail CI ⊆ [80, 125]
------------------------------------------------------------------------------------------------------------------
Variability CV (%) | σ̂ 170.03 | 1.1656
------------------------------------------------------------------------------------------------------------------
2.4 Two Period Parallel design
Consider now a two period parallel design. The name of the endpoint column here is just a generic :PK and we thus specify this for endpoint.
data =dataset("bioequivalence/RR_TT/From_PJ2017_4_2")be_result =pumas_be(data; endpoint =:PK);
This carries out a Two One Sided t-test (TOST) as it does for the parallel design but it also estimates the within subject variability for the test and reference separably.
sequence ╲ period │ 1 2
──────────────────┼───────
RR │ 34 35
TT │ 38 38
-----------------------------------------------------------------------------------------------------------------
Paradigm: Parallel design
Model: T test (unequal variance)
Criteria: Standard ABE
Endpoint: PK
Formulations: Reference(R), Test(T)
-----------------------------------------------------------------------------------------------------------------
Results(PK) Assessment Criteria
-----------------------------------------------------------------------------------------------------------------
R Geometric Naive Mean 3.72
-----------------------------------------------------------------------------------------------------------------
T Geometric Naive Mean 3.891
Geometric Mean T/R Ratio (%) 104.6
Degrees of Freedom 141.4
90% Confidence Interval (%) [95.34, 114.8] Pass CI ⊆ [80, 125]
-----------------------------------------------------------------------------------------------------------------
Variability CVᵣ (%) | σ̂ᵣ 13.29 | 0.1323
CVₜ (%) | σ̂ₜ 8.14 | 0.0813
-----------------------------------------------------------------------------------------------------------------
2.5 Nonparameteric
data =dataset(joinpath("bioequivalence", "RT_TR", "PJ2017_3_16.csv"))be_result =pumas_be(data; endpoint =:Tmax);
When the endpoint is named :Tmax (as it is in this dataset) then pumas_be detects this is a nonparameteric paradigm. This can also be done with the nonparametric keyword argument to pumas_be. See ? pumas_be for help.
sequence ╲ period │ 1 2
──────────────────┼───────
RT │ 17 17
TR │ 15 15
----------------------------------------------------------------------------------------------------------------
Paradigm: Nonparametric analysis
Model: Signed rank test
Criteria: Standard ABE
Endpoint: Tmax
Formulations: Reference(R), Test(T)
----------------------------------------------------------------------------------------------------------------
Results(Tmax) Assessment Criteria
----------------------------------------------------------------------------------------------------------------
T 90% Confidence Interval (%) [81.64, 141.5] Fail CI ⊆ [80, 125]
----------------------------------------------------------------------------------------------------------------
In this case, a 90% confidence interval for the ratio of centrality between reference and test is presented. Since the interval is not contained within the 80 – 125 range, we see that the assessment in this case is Fail.
2.6 Fully-replicated design reference scaling for highly variable drugs
data =dataset(joinpath("bioequivalence", "RTTR_TRRT", "SLTGSF2020_DS16"))be_result =pumas_be(data, FDA_HighlyVariable; endpoint =:PK);
We now consider a fully replicate 4 period RTTR|TRRT design. Importantly, you’ll see a second positional argument in the pumas_be call. This is FDA_HighlyVariable. It is a type (constant) specified in the Bioequivalence package that indicates to pumas_be to carry out reference scaling as specified in FDA (2011).
This FDA_HighlyVariable argument replaces a default second positional argument, StandardBioequivalenceCriterion which is not visible in the calls made to pumas_be above. With FDA_HighlyVariable we now have a richer analysis and model taking place. It is meant for studies that were approved for reference scaling prior to the study.
Paradigm: Replicated crossover that supports reference scaling
Model: Mixed model (unequal variance)
Criteria: FDA RSABE for HV
Endpoint: PK
Formulations: Reference(R), Test(T)
Results(PK)
Assessment
Criteria
R
Geometric Marginal Mean
0.5916
Geometric Naive Mean
0.5957
T
Geometric Marginal Mean
0.4663
Geometric Naive Mean
0.4688
Geometric Mean T/R Ratio (%)
78.83
Fail
GMR ∈ [80, 125]
Degrees of Freedom
86.56
90% Confidence Interval (%)
[69.31, 89.66]
Variability
CVᵣ (%) | σ̂ᵣ
49.72 | 0.47
OK for RS
CVᵣ ≥ Minimal CVᵣ
CVₜ (%) | σ̂ₜ
51.41 | 0.4843
Variability Ratio (%)
103.1
ANOVA
Formulation (p-value)
0.00283
Sequence (p-value)
0.5266
Period (p-value)
0.03876
Reference Scaling Params
Reference Scaling Constant
0.7967
Minimal CVᵣ for Reference Scaling (%)
30.0 | 0.294
Reference Scaling Analysis
Geometric Mean T/R Ratio (%)
78.83
Standard Error (Log Scale)
0.0528
90% Confidence Interval (%)
[72.12, 86.18]
Degrees of Freedom
36
Howe's Approx RSABE Stat (95%)
-0.04805
Pass
≤ 0
show(stdout, MIME"text/plain"(), be_result)
sequence ╲ period │ 1 2 3 4
──────────────────┼───────────────
RTTR │ 20 20 20 20
TRRT │ 18 18 18 18
-------------------------------------------------------------------------------------------------------------------------------
Paradigm: Replicated crossover that supports reference scaling
Model: Mixed model (unequal variance)
Criteria: FDA RSABE for HV
Endpoint: PK
Formulations: Reference(R), Test(T)
-------------------------------------------------------------------------------------------------------------------------------
Results(PK) Assessment Criteria
-------------------------------------------------------------------------------------------------------------------------------
R Geometric Marginal Mean 0.5916
Geometric Naive Mean 0.5957
-------------------------------------------------------------------------------------------------------------------------------
T Geometric Marginal Mean 0.4663
Geometric Naive Mean 0.4688
Geometric Mean T/R Ratio (%) 78.83 Fail GMR ∈ [80, 125]
Degrees of Freedom 86.56
90% Confidence Interval (%) [69.31, 89.66]
-------------------------------------------------------------------------------------------------------------------------------
Variability CVᵣ (%) | σ̂ᵣ 49.72 | 0.47 OK for RS CVᵣ ≥ Minimal CVᵣ
CVₜ (%) | σ̂ₜ 51.41 | 0.4843
Variability Ratio (%) 103.1
-------------------------------------------------------------------------------------------------------------------------------
ANOVA Formulation (p-value) 0.00283
Sequence (p-value) 0.5266
Period (p-value) 0.03876
-------------------------------------------------------------------------------------------------------------------------------
Reference Scaling Params Reference Scaling Constant 0.7967
Minimal CVᵣ for Reference Scaling (%) 30.0 | 0.294
-------------------------------------------------------------------------------------------------------------------------------
Reference Scaling Analysis Geometric Mean T/R Ratio (%) 78.83
Standard Error (Log Scale) 0.0528
90% Confidence Interval (%) [72.12, 86.18]
Degrees of Freedom 36
Howe's Approx RSABE Stat (95%) -0.04805 Pass ≤ 0
-------------------------------------------------------------------------------------------------------------------------------
With reference scaling analysis there are a several conditions that need to be checked. First is the analysis of the within subject variability of the reference product, CVᵣ. If this quantity exceeds the minimal threshold (%30), then reference scaling analysis is carried out. Otherwise, a standard analysis is carried out. In this example you see a case where the reference scaling passes this threshold and hence you see OK for RS in the Variability section.
For example, it occurs (due to chance) with this data that if we take only the first 28 rows of this dataset then CVᵣ drops to a much lower value of 12.93 percent, in which case we see Not OK for RS:
Paradigm: Replicated crossover that supports reference scaling
Model: Mixed model (unequal variance)
Criteria: FDA RSABE for HV
Endpoint: PK
Formulations: Reference(R), Test(T)
Results(PK)
Assessment
Criteria
R
Geometric Marginal Mean
0.583
Geometric Naive Mean
0.6551
T
Geometric Marginal Mean
0.4835
Geometric Naive Mean
0.5166
Geometric Mean T/R Ratio (%)
82.93
Degrees of Freedom
9.875
90% Confidence Interval (%)
[58.23, 118.2]
Fail
CI ⊆ [80, 125]
Variability
CVᵣ (%) | σ̂ᵣ
12.93 | 0.1288
Not OK for RS
CVᵣ ≥ Minimal CVᵣ
CVₜ (%) | σ̂ₜ
57.49 | 0.5344
Variability Ratio (%)
414.9
ANOVA
Formulation (p-value)
0.3597
Sequence (p-value)
0.5878
Period (p-value)
0.00234
Reference Scaling Params
Reference Scaling Constant
0.7967
Minimal CVᵣ for Reference Scaling (%)
30.0 | 0.294
Reference Scaling Analysis
Geometric Mean T/R Ratio (%)
82.93
Standard Error (Log Scale)
0.0918
90% Confidence Interval (%)
[68.93, 99.78]
Degrees of Freedom
5
Howe's Approx RSABE Stat (95%)
0.1255
Importantly, contrasting the above two examples, in the first one (OK for RS) you will see that the relevant criteria are:
GMR ∈ [80, 125] (point estimate). This is with the mathematical \(\in\) (in) symbol.
≤ 0 (Howe’s approximation RSABE Stat being non-positive).
Where on the other hand, in the second example (Not OK for RS), the relevant criteria is only the standard ABE:
CI ⊆ [80, 125]. This is with the mathematical ⊆ (subset) symbol.
Let’s now analyze the sections of the output, one by one, considering the first example (OK for RS). The first section for R is the same as in the standard average bioequivalence (Section 2.1). In the section for T we already see a difference. In this case, as we carry out FDA’s reference scaling criteria, there is a requirement for the point estimate of the geometric mean ratio (GMR) to lie within the standard bounds.
Further, when we consider the Variability section, now the model can estimate within subject variability separately for the reference and the test. We see these estimates, again both in terms of the log transformed standard deviation estimates, and in terms of coefficients of variation. Now since reference scaling can only be carried out if CVᵣ is above the minimal value (set at 30 in this case), we see an assessment of OK for RS in this example. Note that if CVᵣ would have not reached the minimal value, standard average bioequivalence analysis would have been performed. We also see a variability ratio between test and reference as a percentage. This is not used in this case, but is used for NTIDs. See (Section 2.8).
Now the additional section Reference Scaling Params presents constants used in the computation. These can be adjusted if needed via other arguments to pumas_be. Importantly in this case, the 0.7967 constant is specified by the FDA.
Finally the section Reference Scaling Analysis presents estimates from the Auxiliary model which are used to compute Howe’s Approximation RSABE statistic as required in FDA (2011). The critical important value is Howe's Approx RSABE Stat, which needs to be negative. See more details in our specific tutorial for reference scaling.
2.7 Partial-replicated design reference scaling for highly variable drugs
data =dataset(joinpath("bioequivalence", "RRT_RTR_TRR", "SLTGSF2020_DS07"))be_result =pumas_be(data, FDA_HighlyVariable; endpoint =:PK);
This case is similar to Section 2.6, yet Pumas only estimates the within subject variability for the reference product. Compare the Variability section in this case with the Variability of the previous case where within subject variability is estimated for both products. Following the FDA (2011), we are still able to carry out a reference scaling analysis as long as CVᵣ ≥ Minimal CVᵣ. This is indeed the case for this dataset, and further in this dataset both criteria are fulfilled:
The point estimate GMR ∈ [80, 125] since the estimated GMR is 89.58.
The Howe's Approx RSABE Stat is ≤ 0 since it is at a value of -0.06284.
Paradigm: Replicated crossover that supports reference scaling
Model: Mixed model (unequal variance)
Criteria: FDA RSABE for HV
Endpoint: PK
Formulations: Reference(R), Test(T)
Results(PK)
Assessment
Criteria
R
Geometric Marginal Mean
99.41
Geometric Naive Mean
99.41
T
Geometric Marginal Mean
89.05
Geometric Naive Mean
89.05
Geometric Mean T/R Ratio (%)
89.58
Pass
GMR ∈ [80, 125]
Degrees of Freedom
357.7
90% Confidence Interval (%)
[86.44, 92.83]
Variability
CVᵣ (%) | σ̂ᵣ
34.23 | 0.3329
OK for RS
CVᵣ ≥ Minimal CVᵣ
ANOVA
Formulation (p-value)
0
Sequence (p-value)
0.08425
Period (p-value)
0.7295
Reference Scaling Params
Reference Scaling Constant
0.7967
Minimal CVᵣ for Reference Scaling (%)
30.0 | 0.294
Reference Scaling Analysis
Geometric Mean T/R Ratio (%)
89.58
Standard Error (Log Scale)
0.0216
90% Confidence Interval (%)
[86.44, 92.83]
Degrees of Freedom
357
Howe's Approx RSABE Stat (95%)
-0.06284
Pass
≤ 0
show(stdout, MIME"text/plain"(), be_result)
sequence ╲ period │ 1 2 3
──────────────────┼──────────────
RRT │ 120 120 120
RTR │ 120 120 120
TRR │ 120 120 120
-------------------------------------------------------------------------------------------------------------------------------
Paradigm: Replicated crossover that supports reference scaling
Model: Mixed model (unequal variance)
Criteria: FDA RSABE for HV
Endpoint: PK
Formulations: Reference(R), Test(T)
-------------------------------------------------------------------------------------------------------------------------------
Results(PK) Assessment Criteria
-------------------------------------------------------------------------------------------------------------------------------
R Geometric Marginal Mean 99.41
Geometric Naive Mean 99.41
-------------------------------------------------------------------------------------------------------------------------------
T Geometric Marginal Mean 89.05
Geometric Naive Mean 89.05
Geometric Mean T/R Ratio (%) 89.58 Pass GMR ∈ [80, 125]
Degrees of Freedom 357.7
90% Confidence Interval (%) [86.44, 92.83]
-------------------------------------------------------------------------------------------------------------------------------
Variability CVᵣ (%) | σ̂ᵣ 34.23 | 0.3329 OK for RS CVᵣ ≥ Minimal CVᵣ
-------------------------------------------------------------------------------------------------------------------------------
ANOVA Formulation (p-value) 0
Sequence (p-value) 0.08425
Period (p-value) 0.7295
-------------------------------------------------------------------------------------------------------------------------------
Reference Scaling Params Reference Scaling Constant 0.7967
Minimal CVᵣ for Reference Scaling (%) 30.0 | 0.294
-------------------------------------------------------------------------------------------------------------------------------
Reference Scaling Analysis Geometric Mean T/R Ratio (%) 89.58
Standard Error (Log Scale) 0.0216
90% Confidence Interval (%) [86.44, 92.83]
Degrees of Freedom 357
Howe's Approx RSABE Stat (95%) -0.06284 Pass ≤ 0
-------------------------------------------------------------------------------------------------------------------------------
2.8 Fully-replicated design reference scaling for NTID
data =dataset(joinpath("bioequivalence", "RTTR_TRRT", "PJ2017_4_3"))be_result =pumas_be(data, FDA_NarrowTherapeuticIndex);
When dealing with drugs that are specified as narrow therapeutic index (NTI) we use the FDA (2012) guidance. This is specfieid by specifying FDA_NarrowTherapeuticIndex as the second argument to pumas_be. Such analysis can only be carried out on four period designs, and is somewhat similar to the analysis appearing in Section 2.6. Here the criteria are:
CI ⊆ [80, 125] as in average bioequivalence. In the example below this is a pass since the confidence interval is [99.31, 108.2].
Similar to reference scaling for highly variable drugs, Howe's Approx RSABE Stat is ≤ 0. In this example this is a failure since the statistic is at 0.0001105.
Finally the ratio of the reference and test variabilities needs to satisfy a quantile condition. The Variability ratio is at 135 in the example below and the associated qunatile is at 2.118. The condition of the qunatile being ≤ 2.5 is satisfied.
Paradigm: Replicated crossover that supports reference scaling
Model: Mixed model (unequal variance)
Criteria: FDA RSABE for NTI
Endpoint: AUC
Formulations: Reference(R), Test(T)
Results(AUC)
Assessment
Criteria
R
Geometric Marginal Mean
7152
Geometric Naive Mean
7131
T
Geometric Marginal Mean
7412
Geometric Naive Mean
7454
Geometric Mean T/R Ratio (%)
103.6
Degrees of Freedom
15.24
90% Confidence Interval (%)
[99.31, 108.2]
Pass
CI ⊆ [80, 125]
Variability
CVᵣ (%) | σ̂ᵣ
8.02 | 0.0801
CVₜ (%) | σ̂ₜ
10.84 | 0.1081
Variability Ratio (%)
135
ANOVA
Formulation (p-value)
0.1624
Sequence (p-value)
0.3184
Period (p-value)
0.665
Reference Scaling Params
Reference Scaling Constant
1.11
Reference Scaling Analysis
Geometric Mean T/R Ratio (%)
103.7
Standard Error (Log Scale)
0.0253
90% Confidence Interval (%)
[99.2, 108.4]
Degrees of Freedom
15
Howe's Approx RSABE Stat (95%)
0.0001105
Fail
≤ 0
Variability Ratio Quantile (95%)
2.118
Pass
≤ 2.5
show(stdout, MIME"text/plain"(), be_result)
sequence ╲ period │ 1 2 3 4
──────────────────┼───────────
RTTR │ 8 8 8 8
TRRT │ 9 9 9 8
-----------------------------------------------------------------------------------------------------------------------
Paradigm: Replicated crossover that supports reference scaling
Model: Mixed model (unequal variance)
Criteria: FDA RSABE for NTI
Endpoint: AUC
Formulations: Reference(R), Test(T)
-----------------------------------------------------------------------------------------------------------------------
Results(AUC) Assessment Criteria
-----------------------------------------------------------------------------------------------------------------------
R Geometric Marginal Mean 7152
Geometric Naive Mean 7131
-----------------------------------------------------------------------------------------------------------------------
T Geometric Marginal Mean 7412
Geometric Naive Mean 7454
Geometric Mean T/R Ratio (%) 103.6
Degrees of Freedom 15.24
90% Confidence Interval (%) [99.31, 108.2] Pass CI ⊆ [80, 125]
-----------------------------------------------------------------------------------------------------------------------
Variability CVᵣ (%) | σ̂ᵣ 8.02 | 0.0801
CVₜ (%) | σ̂ₜ 10.84 | 0.1081
Variability Ratio (%) 135
-----------------------------------------------------------------------------------------------------------------------
ANOVA Formulation (p-value) 0.1624
Sequence (p-value) 0.3184
Period (p-value) 0.665
-----------------------------------------------------------------------------------------------------------------------
Reference Scaling Params Reference Scaling Constant 1.11
-----------------------------------------------------------------------------------------------------------------------
Reference Scaling Analysis Geometric Mean T/R Ratio (%) 103.7
Standard Error (Log Scale) 0.0253
90% Confidence Interval (%) [99.2, 108.4]
Degrees of Freedom 15
Howe's Approx RSABE Stat (95%) 0.0001105 Fail ≤ 0
Variability Ratio Quantile (95%) 2.118 Pass ≤ 2.5
-----------------------------------------------------------------------------------------------------------------------
2.9 Partial-replicated design without reference scaling
data =dataset(joinpath("bioequivalence", "RTT_TRR", "PJ2017_4_2"))be_result =pumas_be(data);
This final case deal with partial replicate designs not specified in the FDA (2011) guidance. In this case a reference scaling analysis is not carried out. Notice that if we were to specify FDA_HighlyVariable as a second argument, an error would have been thrown.
Paradigm: Replicated crossover that does not support reference scaling
Model: Mixed model (unequal variance)
Criteria: Standard ABE
Endpoint: AUC
Formulations: Reference(R), Test(T)
Results(AUC)
Assessment
Criteria
R
Geometric Marginal Mean
58.38
Geometric Naive Mean
56.58
T
Geometric Marginal Mean
49.07
Geometric Naive Mean
55.29
Geometric Mean T/R Ratio (%)
84.05
Degrees of Freedom
66.8
90% Confidence Interval (%)
[76.39, 92.49]
Fail
CI ⊆ [80, 125]
Variability
CVᵣ (%) | σ̂ᵣ
43.93 | 0.4201
CVₜ (%) | σ̂ₜ
26.72 | 0.2626
ANOVA
Formulation (p-value)
0.00346
Sequence (p-value)
0.1655
Period (p-value)
0.02729
show(stdout, MIME"text/plain"(), be_result)
sequence ╲ period │ 1 2 3
──────────────────┼───────────
RTT │ 37 38 38
TRR │ 33 34 35
-------------------------------------------------------------------------------------------------------------------
Paradigm: Replicated crossover that does not support reference scaling
Model: Mixed model (unequal variance)
Criteria: Standard ABE
Endpoint: AUC
Formulations: Reference(R), Test(T)
-------------------------------------------------------------------------------------------------------------------
Results(AUC) Assessment Criteria
-------------------------------------------------------------------------------------------------------------------
R Geometric Marginal Mean 58.38
Geometric Naive Mean 56.58
-------------------------------------------------------------------------------------------------------------------
T Geometric Marginal Mean 49.07
Geometric Naive Mean 55.29
Geometric Mean T/R Ratio (%) 84.05
Degrees of Freedom 66.8
90% Confidence Interval (%) [76.39, 92.49] Fail CI ⊆ [80, 125]
-------------------------------------------------------------------------------------------------------------------
Variability CVᵣ (%) | σ̂ᵣ 43.93 | 0.4201
CVₜ (%) | σ̂ₜ 26.72 | 0.2626
-------------------------------------------------------------------------------------------------------------------
ANOVA Formulation (p-value) 0.00346
Sequence (p-value) 0.1655
Period (p-value) 0.02729
-------------------------------------------------------------------------------------------------------------------
3 Conclusion
This tutorial presented the major outputs of the Bioequivalence Pumas package. These outputs can be seen both with the Pumas platform or with PumasCP. Other Bioequivlance tutorials cover the statistical and guidance aspects of these cases.
Searle, Shayle R, F Michael Speed, and George A Milliken. 1980. “Population Marginal Means in the Linear Model: An Alternative to Least Squares Means.”The American Statistician 34 (4): 216–21.
