Sample Sizes for Performance Testing of Combination Products and Drug Packaging

When developing combination products or evaluating primary packaging for drug products one must evaluate the performance of the devices and the packaging to ensure that they are fit-for-purpose. Numerous international standardsⁱ and pharmacopeial chaptersⁱⁱ focus on functional performance testing and they all recommend conducting tests that simulate real-world conditions based on the intended use of the device or packaging. This simulation includes matching the physical properties of the drug when using placebos, duplicating the number of punctures and the expected needle size that would be encountered in actual practice and testing samples that had been conditioned with the appropriate sterilization treatments, transportation simulations, storage conditions and thermal exposure to approximate actual expected sample environmental exposure which might affect performance.

After determining the appropriate design verification tests that will be needed to establish that the combination product meets the design input requirements or that the drug packaging adequately protects the drug product, the next consideration is justifying the number of test samples required to demonstrate acceptable performance, a design verification requirementⁱⁱⁱ. Addressing this starts with deciding on the required reliability of the combination product or drug packaging failure mode being evaluated, which is integral to the assessment of risk.

A high-risk failure mode generally necessitates 99% reliability to ensure that the failure occurs no more than 1% of the time. Meanwhile, a medium-risk failure mode typically requires 97.5% reliability, whereas a low-risk failure mode is usually acceptable with 95% reliability (sometimes those reliability values are chosen to be 99%, 95% and 90%, respectively). For whatever failure mode the test is evaluating, consider the consequence of failing to meet the acceptance criteria and the severity of the resulting harm. Critical harms typically involve patient death and are high risk. Major harms typically result in permanent patient impairment or the loss of the device’s primary function (i.e., drug delivery to the appropriate location) and are medium risk. Minor harms usually result in patient discomfort, where the harm is reversible, and are low risk.

To determine the appropriate sample size for variable tests (those tests which generate a numerical value for a result) in design verification, a small sample size of about 10-20 samples could first be tested to determine how the product performs (alternatively, existing data from previous studies may be utilized). Statistical techniques can be used to determine the sample size for testing based on the confidence level (typically 95%), the desired reliability level, the average result of the above initial sample set, the standard deviation of the results from the above initial sample set and the acceptance criteria (or specification limits) of the test method. If the results for each member of the sample set are very similar (displaying a small standard deviation) and are far from the specification limit(s), then a small sample set would be sufficient in design verification to assure that the product would meet the specifications. However, if the results show a wide range of values and/or are close to the specification limit, it is necessary to use a larger sample size in design verification to assure that outliers do not exceed the specification. Alternatively, ISO 11608-1:2022 Annex F^iv simply recommends using 20 samples for 95% reliability and 30 samples for 97.5% reliability, for test result data that is normally distributed.

For tests that produce pass/fail results, known as attribute test methods, one will find tables, such as the one shown in ISO 11608-1:2022 Annex F and reproduced as Table 1 below, which show that for 99% reliability and 95% confidence, 299 samples (n) must be tested; for 95% reliability and 95% confidence, 59 samples (n) must be tested; while for 90% reliability and 95% confidence, 29 samples (n) must be tested. These values come from the calculation of an attribute sampling plan using the Bayes Success-Run Theorem,^v which states that:

$n=\frac{\ln (1-\mathrm{Confidence})}{\ln (\mathrm{Reliability})}$

Table 1 – Recommended attribute sample size, per ISO 11608-1:2024 Annex F

Notice that this table shows that successfully testing a single sample only produces assurance of 5% reliability while successfully testing 5 samples only produces assurance of 50% reliability. This is not to say that 95% and 50% of samples, respectively, will fail but that one can only be assured that at least 5% and 50% of samples, respectively, will pass.

For variable test methods, results can be close to the acceptance criteria (just passing) or far away (comfortably passing). For attribute tests, the results, by definition, either pass or fail with no indication of the margin of passing or failing and therefore a larger sample size is required. If one needs to establish 99% reliability at 95% confidence, attaining 299 passing test results might prove to be challenging because, for one reason or another, one may have a failure, which is acceptable because one is looking for 99% reliability and not 100% reliability. Using the binomial distribution function, one can calculate the number of samples that must be tested to establish reliability while observing zero, one or even two failures at the approximately 95% confidence level. For example, if one wants to show 99% reliability and tests 299 samples but achieves two failures, one could test additional samples, up to a total of 628, and as long as there are only the two failures, then one has still demonstrated 99% reliability. Of course, one should not “test into compliance” and, if anticipating a chance of failure, it is best to plan to test 628 samples from the start. These permissible failures to still achieve the desired level of reliability, at a minimum of a 95% confidence level, are shown in Table 2.

Table 2 – Permissible failures to still achieve the desired level of reliability, at a minimum of a 95% confidence level, based on the binomial distribution function.

Consider the testing in USP <382> for fragmentation, which is based on ISO 8871-5:2016 Annex B. The test consists of puncturing 12 vials, not-less-than four times each (as anticipated in the intended application), gathering all the generated elastomer or coating film fragments, and counting those ≥ 150 µm in their longest linear dimension. The acceptance criterion for this test is that not more than five fragments are formed from the total of the not-less-than 48 punctures. That test result is a single data point for an attribute test (the test does not worry about how many fragments are observed, as long as it is not more than five – a pass/fail criterion). If one were to evaluate the risk of having excessive particles in an injectable and if, hypothetically, one requires 90% reliability, one would need 29 data points of passing results. That means that one would need to evaluate 12 stoppers/test x 29 tests = 348 stoppers to demonstrate 90% reliability.

It is crucial to ensure that once test methods are selected for product design verification, the results are statistically meaningful. This can be accomplished by using an adequately large sample size, which is determined by the type of test method (variable or attribute) and the desired levels of reliability and confidence.

West Analytical Services has a laboratory in Exton, PA (US-FDA inspected) and one in Waterford, Ireland (an Irish-HPRA^vi accredited facility located in an EU Member State) ready to help design and perform Design Verification, stability and release testing^vii for combination products. Contact West to learn more about how we can help you with your testing needs.

References