test: e2e replay #3993
Open
test: e2e replay #3993
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Krystof Woldrich <[email protected]>
Co-authored-by: Krystof Woldrich <[email protected]>
Co-authored-by: Krystof Woldrich <[email protected]>
Co-authored-by: Krystof Woldrich <[email protected]>
Android (legacy) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 27ef4ee | 317.40 ms | 321.70 ms | 4.30 ms |
| 6e8584e | 447.10 ms | 474.71 ms | 27.61 ms |
| 148f924 | 492.65 ms | 500.28 ms | 7.63 ms |
| 34aba08 | 328.10 ms | 342.84 ms | 14.74 ms |
| 70e6261 | 482.65 ms | 495.70 ms | 13.05 ms |
| 575f9da | 415.26 ms | 422.98 ms | 7.72 ms |
| 1c65324 | 426.37 ms | 460.36 ms | 33.99 ms |
| 15c80ab+dirty | 336.27 ms | 350.58 ms | 14.31 ms |
| 1d86dd6 | 405.14 ms | 411.06 ms | 5.92 ms |
| 31fcca2 | 391.22 ms | 414.78 ms | 23.56 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 27ef4ee | 17.73 MiB | 19.82 MiB | 2.08 MiB |
| 6e8584e | 17.73 MiB | 19.86 MiB | 2.12 MiB |
| 148f924 | 17.73 MiB | 19.94 MiB | 2.21 MiB |
| 34aba08 | 17.73 MiB | 19.80 MiB | 2.07 MiB |
| 70e6261 | 17.73 MiB | 19.94 MiB | 2.21 MiB |
| 575f9da | 17.73 MiB | 19.83 MiB | 2.10 MiB |
| 1c65324 | 17.73 MiB | 19.95 MiB | 2.21 MiB |
| 15c80ab+dirty | 17.73 MiB | 20.04 MiB | 2.31 MiB |
| 1d86dd6 | 17.73 MiB | 19.86 MiB | 2.12 MiB |
| 31fcca2 | 17.73 MiB | 19.90 MiB | 2.17 MiB |
iOS (legacy) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| dadc233+dirty | 1223.20 ms | 1236.88 ms | 13.68 ms |
| 5571a20+dirty | 1203.57 ms | 1204.57 ms | 1.00 ms |
| 70e6261+dirty | 1220.09 ms | 1230.04 ms | 9.95 ms |
| 62a750b+dirty | 1216.60 ms | 1229.14 ms | 12.54 ms |
| acadc0f+dirty | 1264.38 ms | 1290.06 ms | 25.68 ms |
| 6e8584e+dirty | 1274.50 ms | 1296.82 ms | 22.32 ms |
| c398f67+dirty | 1219.67 ms | 1225.66 ms | 5.99 ms |
| 86d6d2c+dirty | 1267.55 ms | 1286.21 ms | 18.66 ms |
| e5c9b8b+dirty | 1258.57 ms | 1267.32 ms | 8.75 ms |
| 31fcca2+dirty | 1209.17 ms | 1216.21 ms | 7.04 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| dadc233+dirty | 2.36 MiB | 2.84 MiB | 486.85 KiB |
| 5571a20+dirty | 2.36 MiB | 2.92 MiB | 569.93 KiB |
| 70e6261+dirty | 2.36 MiB | 3.03 MiB | 680.42 KiB |
| 62a750b+dirty | 2.36 MiB | 2.92 MiB | 570.00 KiB |
| acadc0f+dirty | 2.36 MiB | 2.83 MiB | 480.37 KiB |
| 6e8584e+dirty | 2.36 MiB | 2.88 MiB | 533.17 KiB |
| c398f67+dirty | 2.36 MiB | 3.04 MiB | 696.27 KiB |
| 86d6d2c+dirty | 2.36 MiB | 2.82 MiB | 462.82 KiB |
| e5c9b8b+dirty | 2.36 MiB | 2.87 MiB | 520.43 KiB |
| 31fcca2+dirty | 2.36 MiB | 2.90 MiB | 552.95 KiB |
iOS (new) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| dadc233+dirty | 1266.52 ms | 1282.55 ms | 16.03 ms |
| 5571a20+dirty | 1228.09 ms | 1233.45 ms | 5.36 ms |
| 70e6261+dirty | 1224.90 ms | 1231.02 ms | 6.12 ms |
| 62a750b+dirty | 1228.12 ms | 1230.53 ms | 2.41 ms |
| acadc0f+dirty | 1271.12 ms | 1272.28 ms | 1.16 ms |
| 6e8584e+dirty | 1271.71 ms | 1281.26 ms | 9.55 ms |
| c398f67+dirty | 1227.31 ms | 1230.00 ms | 2.69 ms |
| 86d6d2c+dirty | 1291.62 ms | 1296.80 ms | 5.18 ms |
| e5c9b8b+dirty | 1276.90 ms | 1280.92 ms | 4.02 ms |
| 31fcca2+dirty | 1222.04 ms | 1226.51 ms | 4.47 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| dadc233+dirty | 2.92 MiB | 3.40 MiB | 492.53 KiB |
| 5571a20+dirty | 2.92 MiB | 3.48 MiB | 575.54 KiB |
| 70e6261+dirty | 2.92 MiB | 3.59 MiB | 686.11 KiB |
| 62a750b+dirty | 2.92 MiB | 3.48 MiB | 575.59 KiB |
| acadc0f+dirty | 2.92 MiB | 3.39 MiB | 487.34 KiB |
| 6e8584e+dirty | 2.92 MiB | 3.44 MiB | 536.52 KiB |
| c398f67+dirty | 2.92 MiB | 3.60 MiB | 701.89 KiB |
| 86d6d2c+dirty | 2.92 MiB | 3.37 MiB | 464.31 KiB |
| e5c9b8b+dirty | 2.92 MiB | 3.43 MiB | 524.50 KiB |
| 31fcca2+dirty | 2.92 MiB | 3.46 MiB | 557.31 KiB |
Android (new) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 52a8031+dirty | 330.72 ms | 358.76 ms | 28.03 ms |
| 86d6d2c+dirty | 267.21 ms | 325.24 ms | 58.04 ms |
| 2534337+dirty | 597.14 ms | 665.04 ms | 67.90 ms |
| 22e31b6+dirty | 295.75 ms | 346.73 ms | 50.98 ms |
| dadc233+dirty | 363.19 ms | 370.37 ms | 7.18 ms |
| 148f924+dirty | 347.36 ms | 389.13 ms | 41.77 ms |
| c398f67+dirty | 315.08 ms | 345.60 ms | 30.52 ms |
| 5bb8d5f+dirty | 356.71 ms | 389.65 ms | 32.94 ms |
| d197b5c+dirty | 258.75 ms | 313.61 ms | 54.86 ms |
| 15c80ab+dirty | 276.38 ms | 327.54 ms | 51.17 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 52a8031+dirty | 7.15 MiB | 8.09 MiB | 965.95 KiB |
| 86d6d2c+dirty | 7.15 MiB | 8.09 MiB | 962.69 KiB |
| 2534337+dirty | 7.15 MiB | 8.11 MiB | 988.68 KiB |
| 22e31b6+dirty | 7.15 MiB | 8.10 MiB | 981.29 KiB |
| dadc233+dirty | 7.15 MiB | 8.04 MiB | 910.84 KiB |
| 148f924+dirty | 7.15 MiB | 8.21 MiB | 1.07 MiB |
| c398f67+dirty | 7.15 MiB | 8.21 MiB | 1.07 MiB |
| 5bb8d5f+dirty | 7.15 MiB | 8.21 MiB | 1.06 MiB |
| d197b5c+dirty | 7.15 MiB | 8.09 MiB | 962.72 KiB |
| 15c80ab+dirty | 7.15 MiB | 8.09 MiB | 966.13 KiB |
test/e2e/test/e2e.test.ts
Outdated
| expect(replay).toBeDefined(); | ||
| expect(replay.replay_id.length).toBe(32); | ||
|
|
||
| const replayInfo = await fetchReplay(replay.replay_id); |
There was a problem hiding this comment.
I thought possibly the Android replays are uploaded due to a short time limit, but if I understand the code correctly, this fetch will wait up to 10 minutes, checking the endpoint every 1s.
Is that right? @vaind
There was a problem hiding this comment.
yeah, nothing to do with timeouts. Replay capture doesn't work at all on android emulator in GHA at the moment
There was a problem hiding this comment.
Also no screenshots are captured by appium, if you look into an artifact for any of the Android jobs:
From what I can tell it's an old/recurring issue of the emulator on GHA, see actions/runner-images#6814
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📢 Type of change
📜 Description
Tests that a replay is part of the error event when session replay options are enabled.
💡 Motivation and Context
💚 How did you test it?
📝 Checklist
sendDefaultPIIis enabled🔮 Next steps