One of the simplest tests of a radiation portal monitor (RPM) is a series of n repeats (a vehicle drive-through) in which the ith repeat records a total number of counts Xi and alarms if Xi ≥ T where T is an alarm threshold. The RPM performance tests we consider use n repeats to estimate the probability p = P(Xi ≥ T). This paper addresses criterion A for testing RPMs, where criterion A is: for specified source strength, we must be at least 95% confident that p ≥ 0.5. To assess criterion A, we consider a distribution-free test and a test relying on assuming the counts Xi have approximately a Poisson distribution. Both test options require tolerance interval construction.