Add Debug scripts #1839

JulianGCalderon · 2024-09-17T20:49:30Z

Add debug scripts

This PR was initially inside of #1824, but I moved it to reduce the size of that PR.

Description

It adds some scripts (and documents them) for debugging differences in memory, trace, public and private air inputs.

Checklist

Linked to Github Issue
Unit tests added
Integration tests added.
This change requires new documentation.
- Documentation has been added/updated.
- CHANGELOG has been updated.

github-actions · 2024-09-17T20:59:01Z

**Hyper Thereading Benchmark results**




hyperfine -r 2 -n "hyper_threading_main threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_main' -n "hyper_threading_pr threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 1
  Time (mean ± σ):     30.108 s ±  0.065 s    [User: 29.330 s, System: 0.776 s]
  Range (min … max):   30.062 s … 30.155 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 1
  Time (mean ± σ):     30.469 s ±  0.130 s    [User: 29.685 s, System: 0.782 s]
  Range (min … max):   30.377 s … 30.561 s    2 runs
 
Summary
  'hyper_threading_main threads: 1' ran
    1.01 ± 0.00 times faster than 'hyper_threading_pr threads: 1'




hyperfine -r 2 -n "hyper_threading_main threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_main' -n "hyper_threading_pr threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 2
  Time (mean ± σ):     16.780 s ±  0.023 s    [User: 29.555 s, System: 0.776 s]
  Range (min … max):   16.763 s … 16.796 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 2
  Time (mean ± σ):     16.912 s ±  0.075 s    [User: 29.881 s, System: 0.763 s]
  Range (min … max):   16.859 s … 16.966 s    2 runs
 
Summary
  'hyper_threading_main threads: 2' ran
    1.01 ± 0.00 times faster than 'hyper_threading_pr threads: 2'




hyperfine -r 2 -n "hyper_threading_main threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_main' -n "hyper_threading_pr threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 4
  Time (mean ± σ):     11.444 s ±  0.184 s    [User: 41.568 s, System: 0.977 s]
  Range (min … max):   11.313 s … 11.574 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 4
  Time (mean ± σ):     11.502 s ±  0.688 s    [User: 41.806 s, System: 0.968 s]
  Range (min … max):   11.015 s … 11.988 s    2 runs
 
Summary
  'hyper_threading_main threads: 4' ran
    1.01 ± 0.06 times faster than 'hyper_threading_pr threads: 4'




hyperfine -r 2 -n "hyper_threading_main threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_main' -n "hyper_threading_pr threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 6
  Time (mean ± σ):     11.369 s ±  0.026 s    [User: 41.853 s, System: 0.951 s]
  Range (min … max):   11.351 s … 11.387 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 6
  Time (mean ± σ):     11.513 s ±  0.475 s    [User: 41.796 s, System: 0.999 s]
  Range (min … max):   11.176 s … 11.849 s    2 runs
 
Summary
  'hyper_threading_main threads: 6' ran
    1.01 ± 0.04 times faster than 'hyper_threading_pr threads: 6'




hyperfine -r 2 -n "hyper_threading_main threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_main' -n "hyper_threading_pr threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 8
  Time (mean ± σ):     11.237 s ±  0.159 s    [User: 41.822 s, System: 0.979 s]
  Range (min … max):   11.125 s … 11.349 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 8
  Time (mean ± σ):     11.364 s ±  0.032 s    [User: 41.818 s, System: 1.002 s]
  Range (min … max):   11.341 s … 11.387 s    2 runs
 
Summary
  'hyper_threading_main threads: 8' ran
    1.01 ± 0.01 times faster than 'hyper_threading_pr threads: 8'




hyperfine -r 2 -n "hyper_threading_main threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_main' -n "hyper_threading_pr threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 16
  Time (mean ± σ):     11.173 s ±  0.198 s    [User: 42.185 s, System: 1.092 s]
  Range (min … max):   11.033 s … 11.313 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 16
  Time (mean ± σ):     11.411 s ±  0.043 s    [User: 42.031 s, System: 1.083 s]
  Range (min … max):   11.381 s … 11.442 s    2 runs
 
Summary
  'hyper_threading_main threads: 16' ran
    1.02 ± 0.02 times faster than 'hyper_threading_pr threads: 16'

github-actions · 2024-09-17T21:11:45Z

Benchmark Results for unmodified programs 🚀

Command	Mean [s]	Min [s]	Max [s]	Relative
`base big_factorial`	2.392 ± 0.014	2.376	2.416	1.00
`head big_factorial`	2.401 ± 0.012	2.382	2.429	1.00 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
`base big_fibonacci`	2.361 ± 0.062	2.313	2.515	1.01 ± 0.03
`head big_fibonacci`	2.332 ± 0.021	2.310	2.384	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base blake2s_integration_benchmark`	8.626 ± 0.190	8.436	8.934	1.00
`head blake2s_integration_benchmark`	8.810 ± 0.322	8.482	9.491	1.02 ± 0.04

Command	Mean [s]	Min [s]	Max [s]	Relative
`base compare_arrays_200000`	2.525 ± 0.063	2.473	2.669	1.00 ± 0.04
`head compare_arrays_200000`	2.524 ± 0.066	2.472	2.686	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base dict_integration_benchmark`	1.652 ± 0.006	1.644	1.661	1.01 ± 0.01
`head dict_integration_benchmark`	1.634 ± 0.008	1.628	1.653	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base field_arithmetic_get_square_benchmark`	1.396 ± 0.036	1.366	1.469	1.02 ± 0.03
`head field_arithmetic_get_square_benchmark`	1.368 ± 0.010	1.353	1.388	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base integration_builtins`	8.932 ± 0.501	8.548	10.289	1.01 ± 0.07
`head integration_builtins`	8.831 ± 0.287	8.522	9.515	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base keccak_integration_benchmark`	9.065 ± 0.206	8.810	9.470	1.00
`head keccak_integration_benchmark`	9.126 ± 0.381	8.771	10.131	1.01 ± 0.05

Command	Mean [s]	Min [s]	Max [s]	Relative
`base linear_search`	2.500 ± 0.041	2.465	2.590	1.00
`head linear_search`	2.527 ± 0.077	2.463	2.704	1.01 ± 0.04

Command	Mean [s]	Min [s]	Max [s]	Relative
`base math_cmp_and_pow_integration_benchmark`	1.713 ± 0.020	1.696	1.767	1.00
`head math_cmp_and_pow_integration_benchmark`	1.730 ± 0.061	1.697	1.866	1.01 ± 0.04

Command	Mean [s]	Min [s]	Max [s]	Relative
`base math_integration_benchmark`	1.671 ± 0.019	1.657	1.725	1.01 ± 0.01
`head math_integration_benchmark`	1.661 ± 0.015	1.647	1.689	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base memory_integration_benchmark`	1.400 ± 0.014	1.389	1.436	1.00
`head memory_integration_benchmark`	1.402 ± 0.013	1.388	1.429	1.00 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
`base operations_with_data_structures_benchmarks`	1.780 ± 0.030	1.760	1.860	1.01 ± 0.02
`head operations_with_data_structures_benchmarks`	1.769 ± 0.012	1.748	1.793	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`base pedersen`	571.3 ± 2.6	566.9	576.1	1.00 ± 0.00
`head pedersen`	570.8 ± 1.2	569.3	573.0	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`base poseidon_integration_benchmark`	674.9 ± 6.7	665.4	685.9	1.00 ± 0.01
`head poseidon_integration_benchmark`	673.6 ± 5.9	665.2	684.6	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base secp_integration_benchmark`	1.984 ± 0.014	1.968	2.018	1.00
`head secp_integration_benchmark`	1.999 ± 0.036	1.973	2.095	1.01 ± 0.02

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`base set_integration_benchmark`	683.6 ± 5.8	677.8	693.2	1.00 ± 0.02
`head set_integration_benchmark`	683.1 ± 11.6	676.2	714.6	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
`base uint256_integration_benchmark`	4.808 ± 0.112	4.698	5.076	1.00
`head uint256_integration_benchmark`	4.818 ± 0.108	4.716	5.081	1.00 ± 0.03

codecov · 2024-09-17T21:12:22Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.83%. Comparing base (5f8a011) to head (6f292f3).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1839   +/-   ##
=======================================
  Coverage   94.83%   94.83%           
=======================================
  Files         101      101           
  Lines       39579    39579           
=======================================
  Hits        37536    37536           
  Misses       2043     2043

Flag	Coverage Δ
	`94.83% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pefontana · 2024-09-20T21:20:27Z

CHANGELOG.md

@@ -2,6 +2,8 @@

 #### Upcoming Changes

+* docs: Add debugging scripts and documentation [#1839](https://github.com/lambdaclass/cairo-vm/pull/1839)


Documentation and this kind of scripts that don't modify the VM doesn't need a line in the changelog
Remove it

Done. I added "documentation" label to ignore the changelog check.

pefontana · 2024-09-20T21:23:35Z

docs/debugging.md

+
+To compare the public inputs, run:
+```bash
+scripts/air_public_inputs_differ.bash file1 file2


Can you be a little clearer here.
Instead of file do <AIR-PUBLIC-INPUT-1> <AIR-PUBLIC-INPUT-2>

Try to do the same to the other commands

JulianGCalderon added 3 commits September 17, 2024 17:44

Add debug scripts

6f98945

Add debugging document

48c3438

Add changelog

cdcefc9

JulianGCalderon marked this pull request as ready for review September 19, 2024 21:04

JulianGCalderon requested review from igaray, Oppen, fmoletta, juanbono and pefontana as code owners September 19, 2024 21:04

pefontana reviewed Sep 20, 2024

View reviewed changes

JulianGCalderon added 2 commits September 20, 2024 18:52

Revert changelog

0bb3f07

Improve script usage code blocks

ef97fc1

JulianGCalderon added the documentation Improvements or additions to documentation label Sep 20, 2024

Remove extra space

6f292f3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Debug scripts #1839

Add Debug scripts #1839

JulianGCalderon commented Sep 17, 2024 •

edited

Loading

github-actions bot commented Sep 17, 2024 •

edited

Loading

github-actions bot commented Sep 17, 2024 •

edited

Loading

codecov bot commented Sep 17, 2024 •

edited

Loading

pefontana Sep 20, 2024

JulianGCalderon Sep 20, 2024

pefontana Sep 20, 2024

JulianGCalderon Sep 20, 2024

		@@ -2,6 +2,8 @@

		#### Upcoming Changes

		* docs: Add debugging scripts and documentation [#1839](https://github.com/lambdaclass/cairo-vm/pull/1839)

Add Debug scripts #1839

Are you sure you want to change the base?

Add Debug scripts #1839

Conversation

JulianGCalderon commented Sep 17, 2024 • edited Loading

Add debug scripts

Description

Checklist

github-actions bot commented Sep 17, 2024 • edited Loading

github-actions bot commented Sep 17, 2024 • edited Loading

codecov bot commented Sep 17, 2024 • edited Loading

Codecov Report

pefontana Sep 20, 2024

Choose a reason for hiding this comment

JulianGCalderon Sep 20, 2024

Choose a reason for hiding this comment

pefontana Sep 20, 2024

Choose a reason for hiding this comment

JulianGCalderon Sep 20, 2024

Choose a reason for hiding this comment

JulianGCalderon commented Sep 17, 2024 •

edited

Loading

github-actions bot commented Sep 17, 2024 •

edited

Loading

github-actions bot commented Sep 17, 2024 •

edited

Loading

codecov bot commented Sep 17, 2024 •

edited

Loading