Repository navigation
Closed
Description
If the following addon:
my-object.h
#include <napi.h>
class MyObject : public Napi::ObjectWrap<MyObject> {
public:
static Napi::Object Init(Napi::Env env, Napi::Object exports);
MyObject(const Napi::CallbackInfo& info);
private:
static Napi::FunctionReference constructor;
void Open(const Napi::CallbackInfo& info);
};my-object.cc
#include "my-object.h"
class OpenAsyncWorker : public Napi::AsyncWorker {
public:
OpenAsyncWorker(Napi::Function& callback)
: Napi::AsyncWorker(callback) {
}
void Execute() {
}
};
Napi::FunctionReference MyObject::constructor;
Napi::Object MyObject::Init(Napi::Env env, Napi::Object exports) {
Napi::HandleScope scope(env);
Napi::Function func = DefineClass(env, "MyObject", {
InstanceMethod("open", &MyObject::Open)
});
constructor = Napi::Persistent(func);
constructor.SuppressDestruct();
exports.Set("MyObject", func);
return exports;
}
MyObject::MyObject(const Napi::CallbackInfo& info)
: Napi::ObjectWrap<MyObject>(info) {
}
void MyObject::Open(const Napi::CallbackInfo& info) {
Napi::Function cb = info[0].As<Napi::Function>();
OpenAsyncWorker* worker = new OpenAsyncWorker(cb);
worker->Queue();
}addon.cc
#include <napi.h>
#include "my-object.h"
Napi::Object InitAll(Napi::Env env, Napi::Object exports) {
return MyObject::Init(env, exports);
}
NODE_API_MODULE(addon, InitAll)is tested with the following JavaScript:
'use strict';
const addon = require('bindings')('addon.node');
const myObject = new addon.MyObject();
let count = 0;
function open() {
myObject.open(() => {
count += 1;
if (count % 1000000 === 0) {
console.log(count);
}
open();
});
}
open();then the memory usage of the node process keeps increasing and increasing and the process ends up crashing after the async open method has been called about 18 million times.
Here's the output of the program:
pi@raspberrypi:~/node-addon-api-leak $ node test/leak.js
(node:1304) Warning: N-API is an experimental feature and could change at any time.
1000000
2000000
3000000
4000000
5000000
6000000
7000000
8000000
9000000
10000000
11000000
12000000
13000000
14000000
15000000
16000000
17000000
18000000
Killed
The complete addon can be seen here.
Metadata
Metadata
Assignees
Labels
No labels
Activity
mhdawson commentedon Mar 14, 2018
My first guess is that you may just be missing some required cleanup after your async event runs or you create so many objects so quickly that they just don't get to run fast enough and you get a large backlog an eventually run out of memory due to that backlog
I'm travelling today but will to take a closer look later in the week.
fivdi commentedon Mar 14, 2018
This would be good. Unfortunately I can't see anything that's missing although I've looked at everything many times.
Only one instance of MyObject is created from JavaScript and it's async
openmethods is called approx. 18000000 times. This means that approx. 18000000 instances ofOpenAsyncWorkerwill be created. However, each call to the asyncopenmethod (except the first call) occurs in the completion callback of the previous call toopen. If I understand the code in AsyncWorker::OnWorkComplete correctly then it deletes the current instance ofOpenAsyncWorkerimmediately after calling the completion callback. If I'm not mistaken this means that there will never be more than two instances ofOpenAsyncWorkerat any time.Excellent, no rush.
I don't want to send you on a wild goose chase here but this issue reminds me a bit of this one. However, I have no idea if the issues are in any way related.
fivdi commentedon Apr 3, 2018
@mhdawson Were you able to verify whether or not there's a memory leak here?
mhdawson commentedon Apr 4, 2018
I've not had time to investigate yet. Have a lot of things on my plate, sorry.
mhdawson commentedon Apr 10, 2018
Some investigation using the existing asyncworker
Running under valgrind with 5000 and 1000 interations showed:
5000
LEAK SUMMARY:
==16094== definitely lost: 5,008 bytes in 294 blocks
==16094== indirectly lost: 80 bytes in 1 blocks
==16094== possibly lost: 304 bytes in 1 blocks
==16094== still reachable: 74,395 bytes in 1,153 blocks
==16094== suppressed: 0 bytes in 0 blocks
==16094== Reachable blocks (those to which a pointer was found) are not shown.
==16094== To see them, rerun with: --leak-check=full --show-leak-kinds=all
10000
==16109== LEAK SUMMARY:
==16109== definitely lost: 4,944 bytes in 292 blocks
==16109== indirectly lost: 80 bytes in 1 blocks
==16109== possibly lost: 304 bytes in 1 blocks
==16109== still reachable: 74,395 bytes in 1,153 blocks
==16109== suppressed: 0 bytes in 0 blocks
==16109== Reachable blocks (those to which a pointer was found) are not shown.
==16109== To see them, rerun with: --leak-check=full --show-leak-kinds=all
So I don't see an obvious leak on the native side.
The heap does seem to grow but running with
node --trace-gc --max_old_space_size=128 asyncworker.jsseems to grow to the max heap size and then continue to run and the virtual and resident sizes stay consistent.
From this don't think its something on the asyncworker side, so willl need to integrate the use of ObjectWrap to see if we get unbounded memory growth with that.
mhdawson commentedon Apr 10, 2018
Existing asyncworker test case with use of object wrap integrated:
asyncworker.cc
asyncworker.js
mhdawson commentedon Apr 10, 2018
using the following to limit the number of loops in the open() function:
Valgrind on 10000
Valgrind on 20000
So I don't see any native memory leak based on number of times open is called.
mhdawson commentedon Apr 10, 2018
Running with
node --trace-gc --max_old_space_size=128 asyncworker.jsand watching memory with top, memory increases and then stabilizes at (shown at about 14 mins into the run) ~
gc activity also stabilizes at frequent gc's freeing about 6MB each time
mhdawson commentedon Apr 10, 2018
These were done with the lastest node master and current node-addon-api repo.
mhdawson commentedon Apr 10, 2018
@fivdi can you take a look at the code I used above, I think I integrated your example properly but it does not look like there is memory leaking to me.
I see you are on a raspberry pi which might not have as much memory as I do. Can you try with --max_old_space_size=128 to see if the process still gets terminated?
fivdi commentedon Apr 11, 2018
@mhdawson
It looks like the examples was integrated correctly into
asyncworker.ccbut it still looks like there is a memory leak to me (see below.)Yes, it's a Raspberry Pi 3 with 1GB of RAM. I tried --max_old_space_size=128 with my example from the original post, here's the output:
leak.jswas modified to displaycountevery 1e5 iterations rather than every 1e6 iterations. As can be seen there is fatal error as the JavaScript heap runs out of memory.The program was running for several hours before the error occurred. The displaying of
countto the console got slower and slower as time went on. Everything looked stable withtop.fivdi commentedon Apr 11, 2018
@mhdawson I did the last test with v10.0.0-nightly2018040653aaa55a3a and
npm i fivdi/node-addon-api-leak.fivdi commentedon Apr 12, 2018
@mhdawson When the example you posted above is integrated into asyncworker.cc and asyncworker.js is modified to:
The output is:
After
2300000 2018-04-11T20:39:14.743Zwas logged to the console the program ran for several hours before the fatal error "JavaScript heap out of memory" occurred.sierrars commentedon May 10, 2018
I found this issue when trying to pinpoint my own memory leak, which I believe is caused by the same problem. While I am fairly new with native add-on things, I'd like to add one observation hoping it is helpful.
The same symptom can be observed on both my application and @fivdi 's https://github.com/fivdi/node-addon-api-leak repo. So I am using node-addon-api-leak as the example. By running
node --inspect test/leak.jsand then attaching to it with the chrome dev tools' allocation timeline tool, it can be seen that sometimes when an AsyncWorker instance is constructed(fivdi's test case shows this very consistently), there is anObject(56 bytes in size and retained by GC root) left in the heap. Over time they add up and I believe this also causes my own app's memory leak(I could be wrong of course). See the screenshot attached. With my limited experience, I couldn't figure out where these objects are from. While the AsyncWorker constructor creates a new object as the receiver(if not given), that doesn't seem to be the object in question.BTW, all this is on macOS and I haven't tried it on any other os. Node-addon-api version is 1.3.0.
gabrielschulhof commentedon May 10, 2018
I ran node-addon-api-leak on my laptop, and it seems to top off at 7.8% memory usage, and it stops printing after 26000000. It continues running at 100%.
4 remaining items
gabrielschulhof commentedon May 11, 2018
@mhdawson and I backed the JS part of the node-addon-api-leak test with the following native code, lacking an AsyncResource, and it did not leak:
sierrars commentedon May 12, 2018
Thanks @mhdawson for the quick turnaround!
async_wrap: fix memory leak in AsyncResource
async_wrap: fix memory leak in AsyncResource
async_wrap: fix memory leak in AsyncResource
async_wrap: fix memory leak in AsyncResource
async_wrap: fix memory leak in AsyncResource