Hi, my Name is Florian Müller-Sallanz and I'm a statistician at Stegmann Systems. In today's Blog i want to talk about the different combination methods available in PLA. About their similarities, their differences and especially about when you should use which method. Since recapping all the details of the relevant chapters would not fit within the scope of a single blog, a certain familiarity with EP 5.3.6, USP<111> and USP<1034> is assumed in the following.

Heterogeneous or homogeneous, that is the question

But let's start with a little quiz: How many different methods for performing a Combination of assay results are available in the CoA-Document of Bioassay 26? I'll give you a hint: It's one more than in Bioassay 25. 

The answer (obviously) is six:

  • unweighted
  • homogenously weighted (USP1034/EP) 
  • homogenously weighted (USP111)
  • heterogenously weighted (USP111)
  • heterogenously weighted (USP1034)
  • heterogenously weighted (EP)

So why does PLA offer you two different picklists with a total of 12 possible combinations for choosing between 6 methods? Because all three of the relevant chapters, USP<111>, USP<1034> and EP 5.3.6, agree that combination should be performed differently depending on whether or not the combined assays are homogeneous or heterogeneous. But what does that mean?

Generally speaking, the assays, that is: their potencies, are homogeneous if they can be assumed to be samples of the same (normal) distribution. If that is the case, the statistic

is approximately -distributed. This equation uses the notation from USP<111>, that is:  is the potency of the i-th assay,  is the (homogeneous) weight of the i-th assay and  is the weighted mean potency. If the statistic is too large, you should assume that the assays are heterogeneous. In this case, if you configure your CoA to choose the weighting by homogeneity, the homogeneous weights will be discarded and the heterogeneous weights will be used instead.

As a rule of thumb, if your combination contains samples for which the confidence intervals have no overlap, you should assume heterogeneity.

In this example, the confidence intervals for the first and last sample do not overlap. The -statistic is approximately 12.56 and the assays fail the test on homogeneity (95% significance).

Six Methods, all alike in dignity?

All this homogeneity and heterogeneity and weighting this and  that seems rather complicated, so why not use an unweighted combination?

While the unweighted combination is pretty straightforward and simple, all the information contained in the confidence intervals, i.e. the information about your intra-assay variation, about your uncertainty, is discarded. The confidence interval for the combined potency has to be constructed from the full inter-assay variability. Usually this results in a larger confidence interval, that is, in more uncertainty about the combined potency. The following example illustrates this point. As before, the assays from group 2 are heterogeneous while the assays from group 1 are homogeneous. While looking through the examples (all of which are available as an .edpdp document export for the biological assay package 26) keep in mind that for constructing confidence intervals, you are pursuing two different goals: Firstly, a 95% confidence interval should cover the unknown "true" potency at least 95 % of the time. Secondly, a tighter interval represents a better grip on the possible locations of the unknown "true" potency that is estimated by the combination. Optimally, the interval should be as wide as necessary, but as tight as possible.

Unweighted Group 1 Group 2
Potency 1.02 1.00
Potency Range (%) 4.34 8.85

 

homogenously weighted (USP1034/EP) Group 1 Group 2
Potency 1.02 1.00
Potency Range (%) 4.34 8.85

 

homogenously weighted (USP111) Group 1 Group 2
Potency 1.02 1.01
Potency Range (%) 4.32 4.51

 

heterogenously weighted (USP111) Group 1 Group 2
Potency 1.02 1.00
Potency Range (%) 3.87 6.09

 

heterogenously weighted (USP1034) Group 1 Group 2
Potency 1.03 1.00
Potency Range (%) 2.83 6.09

 

heterogenously weighted (EP) Group 1 Group 2
Potency 1.02 1.01
Potency Range (%) 4.13 5.02

 

Before we examine the confidence intervals, a quick aside about the combined potency: As you can see, the estimations of combined potencies are nearly identical across all combination methods, and the methods mainly differ in the width of the confidence intervals. Even if this is only a small sample containing only two combinations of unknown "true" relative potencies, this already suggests that you need a considerable number of assay run replicates with known potencies (i.e. reference vs reference) and with values over the entire range of expected combined potencies to determine whether any combination methods yields much greater accuracy for determining the combined potency.  

But let's return to the confidence intervals. For the homogeneous group 1, most of the percentage ranges lie between 3.8 and 4.2. Using heterogeneous weights according to USP<1034> results in a much smaller confidence interval. So that sounds great, right? Not so fast! Looking at the report shows that for these assays and this method, the estimated inter-assay variation is smaller than 0!

Therefore, the width of the confidence interval is artificially deflated. For the heterogeneous USP<1034> method, negative inter-assay variation might occur if variance between potencies is small and individual confidence intervals are large. For these types of assays, I would recommend to avoid this method.

For the examplary assays from group 1, all other methods yield comparable confidence intervals. If you suspect your confidence intervals to be very noisy due to low degrees of freedom, or you are not able to compute confidence intervals with some regularity (e.g. due to vanishing denominators in Fieller's Formula), you should use unweighted combination. 

For the heterogeneous examples from group 2, unweighted combinations result the largest confidence interval. Since the unweighted method does not take into account that some of the assays in this group are more precise than others and does not reduce the weight the very imprecise assays, the resulting interval is probably too large, Now let's look at the two methods using homogeneous weighting although the assays are heterogeneous. They yield the two tightest confidence intervals, but comparing the homogeneous weights with the heterogeneous weights.

Assay\Weight homogenous Weight (normalized Weight) heterogenous Weight (USP) heterogenous Weight (EP)
Assay 1 2640.6583 1016.2334 1581.3713
Assay 2 2300.8672 961.5835 1452.8805
Assay 3 673.9596 478.6748 575.5603
Assay 4 1646.8100 824.6966 1161.5701
Assay 5 2434.5729 984.1724 1505.0749

shows that the homogeneous weights are too large and are resulting in a confidence interval that is too tight. Since the inter-assay variation is non-negative, there is no difference between the USP<111> and USP<1034> methods. As expected, all three heterogeneous methods yield comparable results. 

Methods to the Madness

So which method for combining potencies should you choose? As always it depends on your assays, but if you need an educated guess for jump-starting your analysis, you might find the following recommendations useful:

1) I'd recommend against using the heterogeneously weighted USP<1034> if you suspect your inter-assay variation might become negative, the confidence interval might be too tight.

2) I'd recommend against using the unweighted method for heterogeneous assays, the confidence interval might be to wide.

3) I'd recommend against using any of the homogeneously weighted methods for heterogeneous assays, the confidence interval might be too tight.

4) If you frequently can't compute confidence intervals for individual assays, you should use unweighted combination (or redesign your assay).   

5) If you assays are usually really heterogeneous, i would recommend the heterogeneous (EP) method,

6) If your unsure whether a majority of your assays are heterogeneous,  but some assay runs might pass the test on homogeneity, i would recommend the heterogeneous USP<111> method. 

7) If you assays are homogeneous, i would recommend using the homogeneous USP<1034>/EP method,

I hope you enjoyed today's blog about differnt methods for combination of assay results avilable in PLA. As always feel free to comment or to contact us directly at support@biossay.com if you have any questions or remarks concerning this topic.