You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@dennismartensson: My hope is that we can reproduce the memory leak on our end – let us start from that. It might be useful if you can give us (privately, if you prefer) some sample log entries that you sending. We'll try those in our load-test.
We track requests with a child logger that is attached to the req with a id: req.log = log.child({ req_id: req.getId() })
and we use the restifyplugins.requestLogger to create request Logs.
Out side of that most logs are very standard texted logs like req.log.info('created user')
Hope that helps and if you like more info please let me know.
With the above fix in place, I no longer see the JS heap or node external memory (i.e. memory bound to JS objects) growing. The RSS does grow fairly rapidly. This leaves native modules, and gRPC is the only native module present.
I have opened grpc/grpc#10445 to see if this is possibly a leak in gRPC.
With the gRPC issue fixed, I believe #2198 would address the remaining leak – the problem was that the log messages were being written faster than the writable stream (which only implemented _write) could drain them. Adding support for _writev fixes this issue.
Activity
ofrobots commentedon Mar 24, 2017
@dennismartensson: My hope is that we can reproduce the memory leak on our end – let us start from that. It might be useful if you can give us (privately, if you prefer) some sample log entries that you sending. We'll try those in our load-test.
dennismartensson commentedon Mar 25, 2017
I will share some here and then if you need more we can talk privately.
We use:
We track requests with a child logger that is attached to the req with a id:
req.log = log.child({ req_id: req.getId() })and we use the
restifyplugins.requestLoggerto create request Logs.Out side of that most logs are very standard texted logs like
req.log.info('created user')Hope that helps and if you like more info please let me know.
ofrobots commentedon Mar 31, 2017
I can reproduce this leak.
dennismartensson commentedon Mar 31, 2017
Hi @ofrobots
Good let me know if we can help out with some thing. We are more then happy to send some traffic to it when you have a fix out.
Thanks for digging in to this.
ofrobots commentedon Mar 31, 2017
I believe this should be fixed by d053b20. Need to confirm today.
ofrobots commentedon Apr 1, 2017
Upon a deeper investigation, that commit doesn't seem to fix it 😞 .
I am still investigating.
[-]logging-bunyan: memory leak on GKE app[/-][+]logging: memory leak[/+]ofrobots commentedon Apr 3, 2017
With the above fix in place, I no longer see the JS heap or node external memory (i.e. memory bound to JS objects) growing. The RSS does grow fairly rapidly. This leaves native modules, and gRPC is the only native module present.
I have opened grpc/grpc#10445 to see if this is possibly a leak in gRPC.
murgatroid99 commentedon Apr 7, 2017
I have published grpc@1.2.3, which includes a fix for the leak.
logging-bunyan: implement _writev as well
logging-bunyan: implement _writev as well
ofrobots commentedon Apr 10, 2017
With the gRPC issue fixed, I believe #2198 would address the remaining leak – the problem was that the log messages were being written faster than the writable stream (which only implemented
_write) could drain them. Adding support for_writevfixes this issue.ofrobots commentedon Apr 11, 2017
@dennismartensson It would be great if you can try with @google-cloud/logging-bunyan@0.4.1 and report back if you have any issues. Thanks!
dennismartensson commentedon Apr 11, 2017
@ofrobots Thanks for all the work. We will try it out soon!
gaofanmichael commentedon Oct 26, 2017
This issue was moved to googleapis/nodejs-logging-bunyan#3
chore(main): release 6.9.1 (#2135)
feat(spanner): Add support for Cloud Spanner Default Backup Schedules (…
chore(main): release 6.9.1 (googleapis#2135)