example generation regression between 6.47.0 -> 6.103.1
#4014
Comments
|
Thanks for the report! It would be great if you could bisect the versions here and narrow it down to a smaller version range (ideally, a single version). We've made quite a few core engine changes starting at around Looking through #2308 I see this comment about |
|
@tybug Can you explain more on what this tracking of IRTree does since I'm not very familiar with the codebase? and if it is possible to not having this |
import sys
import os
import time
import hypothesis.strategies as st
from hypothesis import Phase, given, settings
MAX_EXAMPLES = int(os.getenv("MAX_EXAMPLES", "60"))
def gen_empty_fingerprint():
return {i: {} for i in range(8)}
@settings(max_examples=MAX_EXAMPLES, deadline=None,
phases=(Phase.reuse, Phase.generate, Phase.shrink))
@given(data=st.data())
def get_fuzzy_car_interfaces(data):
fingerprint_strategy = st.fixed_dictionaries({key: st.dictionaries(st.integers(min_value=0, max_value=0x800),
st.integers(min_value=0, max_value=64)) for key in
gen_empty_fingerprint()})
# TODO: Add another st.fixed_dictionaries()
data.draw(fingerprint_strategy)
if __name__ == "__main__":
# first bad commit: 5de1fe84252051594fdc6879d4920c357a6d1368 - more likely to generate boundary values
# from 3.5 - 3.8s: 6e2f394a253761677cdcc0990a32df54a62f079a
# - better after revert: 6e2f394a253761677cdcc0990a32df54a62f079a
# from 4s - >5s: 1e76ce2e52e450d54470ed09b9c65fb1b598fb5c - trackIRTree in ConjectureData
lower = float(sys.argv[1])
higher = float(sys.argv[2])
num_iterations = 20
start = time.monotonic()
for _ in range(num_iterations):
get_fuzzy_car_interfaces()
print(f"{num_iterations} iterations takes {time.monotonic() - start}s")
time_taken = time.monotonic() - start
sys.exit(int(lower <= time_taken <= higher))I have a random script here for testing. 6.47.0 is > 2x faster than 6.103.1 |
|
Still looking into this, but some initial diagnosis: it seems that |
|
@bongbui321 could you give |
|
Great! I'll try it out sometime this week, and try to identify some more speed regressions post |
|
yeah, I think that's #4066 as that commit changed |
|
@tybug. I see an improvement with the new I wonder if using pytest-xdist doesn't take effect of the recent improvement you made |
|
@bongbui321 thanks for checking. The graph in #4076 indicates we have ~recent systemic performance problems, so this may be a 'wait and see'. Sorry I couldn't be more helpful for the moment. |
Hello, we're interested in using the latest version of hypothesis but noticed a significant increase in example generation between versions:
making an issue since I've seen in the past that some regression occurred here and was the result of some necessary code fixes that sped up performance #2308
pinning to 6.47.0:
on latest 6.103.1:
The text was updated successfully, but these errors were encountered: