Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize the peak memory usage when loading dataset. #514

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Light-V
Copy link

@Light-V Light-V commented Apr 19, 2024

I've made a small optimization in data loading process that reduces peak memory usage when handling large datasets.
Previously, we were using np.array() to convert large datasets from h5py objects into NumPy arrays. This operation, while straightforward, was causing a spike in memory usage due to the creation of a new array copy, which could lead to OOM errors when working with particularly large datasets.
To mitigate this issue, I have replaced np.array() with np.asarray() in the relevant sections of the code. The np.asarray() function, unlike np.array(), will attempt to pass through the input data without creating a new array copy if the input is already a NumPy array. This behavior helps to reduce unnecessary memory allocation and can be particularly effective in scenarios where memory is a constraint.

@maumueller
Copy link
Collaborator

Thanks @Light-V. Did you observe a change in index building time/search performance with this change? I have made very bad experiences without this cast, but this was a long time ago.

@Light-V
Copy link
Author

Light-V commented Apr 19, 2024

@maumueller Thank you for your suggestion. I will give it a try and post the performance comparison later.

@Light-V
Copy link
Author

Light-V commented Apr 23, 2024

Hi, @maumueller

I have run ann-benchmark on qsg-ngt with these two different ways to load dataset. And here's the search result:

diff result.1 result.2
3c3
< 0: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 3.000) 0.994 9433.409

0: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 3.000) 0.994 9616.273
5c5
< 1: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 0.000) 0.461 40888.565


1: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 0.000) 0.462 41178.560
7c7
< 2: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.500) 0.922 19599.222


2: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.500) 0.923 19792.332
9c9
< 3: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 2.000) 0.730 24640.807


3: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 2.000) 0.732 24812.362
11c11
< 4: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 2.000) 0.980 13130.141


4: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 2.000) 0.981 13396.878
13c13
< 5: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.500) 0.626 28672.532


5: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.500) 0.628 28876.686
15c15
< 6: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 5.000) 0.994 7805.384


6: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 5.000) 0.994 7890.528
17c17
< 7: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.500) 0.954 16309.917


7: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.500) 0.956 16625.485
19c19
< 8: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 3.000) 0.955 14523.425


8: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 3.000) 0.955 14687.466
21c21
< 9: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 0.000) 0.813 27601.332


9: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 0.000) 0.814 27835.597
23c23
< 10: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 8.000) 1.000 2951.642


10: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 8.000) 1.000 3008.100
25c25
< 11: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 3.000) 0.998 7070.455


11: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 3.000) 0.999 7199.444
27c27
< 12: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.200) 0.696 28557.780


12: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.200) 0.698 28513.283
29c29
< 13: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.200) 0.875 22116.188


13: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.200) 0.876 22458.581
31c31
< 14: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 3.000) 0.980 11920.713


14: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 3.000) 0.981 12182.230
33c33
< 15: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 5.000) 1.000 4511.494


15: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 5.000) 1.000 4587.789
35c35
< 16: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 2.000) 0.991 10027.577


16: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 2.000) 0.992 10132.040
37c37
< 17: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 2.000) 0.845 21594.956


17: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 2.000) 0.844 21941.602
39c39
< 18: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.200) 0.910 18668.998


18: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.200) 0.912 19092.833
41c41
< 19: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.200) 0.538 31914.098


19: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.200) 0.536 32022.208
43c43
< 20: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 0.000) 0.625 36063.251


20: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 0.000) 0.628 36227.625
45c45
< 21: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.200) 0.803 25712.051


21: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.200) 0.804 25995.405
47c47
< 22: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 10.000) 0.999 4170.250


22: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 10.000) 0.999 4243.505
49c49
< 23: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 2.000) 0.911 19146.093


23: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 2.000) 0.911 19330.184
51c51
< 24: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 0.000) 0.740 32339.626


24: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 0.000) 0.740 32329.880
53c53
< 25: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 2.000) 0.955 16193.192


25: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 2.000) 0.956 16318.249
55c55
< 26: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 3.000) 0.841 19298.415


26: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 3.000) 0.840 19437.719
57c57
< 27: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 20.000) 1.000 2212.972


27: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 20.000) 1.000 2235.706
59c59
< 28: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.500) 0.771 25701.573


28: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.500) 0.772 25698.865
61c61
< 29: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.500) 0.860 22676.155


29: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.500) 0.861 23154.836
63c63
< 30: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 3.000) 0.916 16764.549


30: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 3.000) 0.913 16869.750

And the build time is 3791.798057794571s and 3794.1561596393585s. I think there is no significant different between them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants