option to limit total threads

[Grinnz]

When not using a pool or when extra_threads is set, NDN spawns additional one-off threads for requests beyond what the pool can handle. This can lead to resource exhaustion in extreme cases. A max_threads or similar option could limit how many threads can be spawned at once before queueing.

[olegwtf]

Can u make pull request?

[olegwtf]

Ok. I started to work on it.

[olegwtf]

Looks implemented in 7076f00. Can you test it?

new(threads_limit => 10); # do not count timed out threads in limit
new(threads_strict_limit => 10); # count all threads including timed out

[olegwtf]

No comments. Closing.

[Grinnz]

This can be tested in the mojo use case by patching Mojo::IOLoop::Client to add a threads_limit or threads_strict_limit on this line: https://metacpan.org/release/Mojolicious/source/lib/Mojo/IOLoop/Client.pm#L48

[Grinnz]

@daoswald can you test this?

[daoswald]

If I pull in the Net::DNS::Native repository from GitHub and install it, and then modify the Net::DNS::Native->new call on line 48 of Mojo::IOLoop::Client to contain (pool => 5, extra_thread => 1, threads_strict_limit => 150) .... and then pound an application with 1000 requests, I'll get: socketpair(): Too many open files at /home/doswald/perl5/perlbrew/perls/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/Net/DNS/Native.pm line 49.

…

On Tue, Nov 5, 2019 at 10:49 AM Dan Book ***@***.***> wrote: @daoswald <https://github.com/daoswald> can you test this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12?email_source=notifications&email_token=AAG376L54W6KDKXVVWCLBPLQSGW2ZA5CNFSM4JCLKHHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDDVX5Q#issuecomment-549936118>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG376KF6FP3XHLG4E54KUDQSGW2ZANCNFSM4JCLKHHA> .

-- David Oswald [email protected]

[daoswald]

I should follow-up with additional information: I verified that I'm hitting the new code by intentionally misspelling threads_strict_limit to something else, and when I do that I get the 'unsupported option' message. So I know that I'm hitting the correct code, because when I spell the option correctly there's no such warning.

…

On Tue, Nov 5, 2019 at 1:36 PM David Oswald ***@***.***> wrote: If I pull in the Net::DNS::Native repository from GitHub and install it, and then modify the Net::DNS::Native->new call on line 48 of Mojo::IOLoop::Client to contain (pool => 5, extra_thread => 1, threads_strict_limit => 150) .... and then pound an application with 1000 requests, I'll get: socketpair(): Too many open files at /home/doswald/perl5/perlbrew/perls/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/Net/DNS/Native.pm line 49. On Tue, Nov 5, 2019 at 10:49 AM Dan Book ***@***.***> wrote: > @daoswald <https://github.com/daoswald> can you test this? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#12?email_source=notifications&email_token=AAG376L54W6KDKXVVWCLBPLQSGW2ZA5CNFSM4JCLKHHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDDVX5Q#issuecomment-549936118>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAG376KF6FP3XHLG4E54KUDQSGW2ZANCNFSM4JCLKHHA> > . > -- David Oswald ***@***.***

-- David Oswald [email protected]

[olegwtf]

You need to increase number of allowed open files per process (ulimit -n). Each getaddrinfo() call creates 2 open handles independently of the way it will be processed: immediately or queued. I see no way to decrease number of open filehandles. Suggestions are welcome.