You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Bug / Clarification needed: Impact of mmap on memory_usage #345
Disclaimer: I know that "memory usage" is a very fuzzy term once you actually look at the details. Nevertheless, I was really confused by my observations described below and think it might be a bug or deserve an explicit explanation in the docs.
Short story: I wanted to demonstrate reduced memory usage by using mmap instead of data = file.read() and wanted to back it up with measurements from memory_profiler.memory_usage.
When mmaping a 4GB file I found that memory_profiler.memory_usage reports a peak usage of around 4GB, while the memory usage indicators on my system monitoring tools (htop and KSysGuard) don't even flinch.
Long story:
Function I use memory_usage on. File: find_keys.py:
I am really surprised by that output from memory_usage. Especially since my memory monitoring tools do not show any such peak. (KSysGuard screenshot)
The usage displayed in the screenshot makes more sense to me, given my understanding of mmap: It lets the OS "efficiently" swap pages between disk and memory, giving me fast access and minimal memory footprint. But I by no means understand the details.
The commented-out sleep(10) in my function was my attempt at giving htop / KSysGuard more time to "pick up" a peak in the memory usage. I was suspecting that they "poll" the memory usage and everything just happened "too fast" for them. But it didn't change the output of neither memory_usage nor KSysGuard.
Reacted by Oriane Nédey, Lucas Cimon and Igor Rodionov
Activity