Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python] Memory leak when using engine="pyarrow" while reading csv files via pd.read_csv() #44147

Open
AtanasKostov2 opened this issue Sep 17, 2024 · 0 comments
Open

[Python] Memory leak when using engine="pyarrow" while reading csv files via pd.read_csv() #44147

AtanasKostov2 opened this issue Sep 17, 2024 · 0 comments
Labels
Component: Other Component: Python Type: bug

Comments

@AtanasKostov2
Copy link

Describe the bug, including details regarding any error messages, version, and platform.

I have a 3.5GB csv file and I'm using pyarrow as the engine to speed up the read time. Unfortunately it causes memory to be allocated and not freed afterwards.

image_2024-09-17_134817340
image

image

OS: windows 10
CPU: i9-12900H

Component(s)

Python, Other
@raulcd raulcd changed the title Memory leak when using engine="pyarrow" while reading csv files via pd.read_csv() [Python] Memory leak when using engine="pyarrow" while reading csv files via pd.read_csv() Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Other Component: Python Type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@AtanasKostov2