Their abstract doesn't match their actual paper contents. That's unfortunate. Their summary indicates rates in terms of test cases (repeating your quote a bit):
> 75% of test cases built correctly, 57% passed reliably [implying test cases by context], and 25% increased coverage [same implication]
The actual report talks about test classes, where each class has one or more test cases.
> (1) 75% of test classes had at least one new test case that builds correctly.
> (2) 57% of test classes had at least one test case that builds cor- rectly and passes reliably.
> (3) 25% of test classes had at least one test case that builds cor- rectly, passes and increases line coverage compared to all other test classes that share the same build target.
Those are two very different statements. They even have a footnote acknowledging this:
> For a given attempt to extend a test class, there can be many attempts to generate a test case, so the success rate per test case is typically considerably lower than that per test class.
But then in their conclusion they misrepresent their findings again, like the abstract:
> When we use TestGen-LLM in its experimental mode (free from the confounding factors inherent in deployment), we found that the success rate per test case was 25% (See Section 3.3). However, line coverage is a stringent requirement for success. Were we to relax the requirement to require only that test cases build and pass, then the success rate rises to 57%.
> 75% of test cases built correctly, 57% passed reliably [implying test cases by context], and 25% increased coverage [same implication]
The actual report talks about test classes, where each class has one or more test cases.
> (1) 75% of test classes had at least one new test case that builds correctly.
> (2) 57% of test classes had at least one test case that builds cor- rectly and passes reliably.
> (3) 25% of test classes had at least one test case that builds cor- rectly, passes and increases line coverage compared to all other test classes that share the same build target.
Those are two very different statements. They even have a footnote acknowledging this:
> For a given attempt to extend a test class, there can be many attempts to generate a test case, so the success rate per test case is typically considerably lower than that per test class.
But then in their conclusion they misrepresent their findings again, like the abstract:
> When we use TestGen-LLM in its experimental mode (free from the confounding factors inherent in deployment), we found that the success rate per test case was 25% (See Section 3.3). However, line coverage is a stringent requirement for success. Were we to relax the requirement to require only that test cases build and pass, then the success rate rises to 57%.