I am performance testing a function using XCTestCase measure(_:).
If the percentage increase in time (from the set baseline) is 62% worse, the test still passes: 
However, if the percentage increase in time is 65% worse, the test fails: 
and the failure message says "(max allowed: 10%)"
My questions:
- Does the "max allowed: 10%" message imply the test should have failed in the first case when the time increased by 62%?
- Is the max allowed increase officially documented anywhere because I can't find it in any official documentation.
- Is there a way to customize the max allowed percent increase? I see there is a way to adjust the max STDEV, but that's not what I mean.
Note: In all my testing it appears that if (%increase - STDEV) >= 60% the test will fail, otherwise the test will pass.
from Xcode performance testing failure percentage
No comments:
Post a Comment