* Re: Support for Rack 3 headers formatted as arrays
@ 2023-01-11 11:40 6% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2023-01-11 11:40 UTC (permalink / raw)
To: Jean Boussier; +Cc: Martin Posthumus, unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
> Interestingly, this patch would also help with Ruby 3.2 compatibility.
>
> Ruby 3.2 removed `Kernel#=~`, so when misbehaving app return header values
> in e.g. Integer
> unicorn now choke on it:
Thanks for that note. 2 things:
1. Ugh. That sort disregard for compatibility at the expense of making
things "better" is why I and most people I've known no longer do new
projects in Ruby :< Heck, I started rewriting many of unicorn's tests
in Perl5 a few weeks ago since I want to be able to trust them decades
from now.
2. I suppose unicorn will continue letting misbehaving apps have
their way. It would be nice if developers consistently
tested with Rack::Lint, of course
> Whereas on 3.1 and older, `42 =~ /\n/` would simply not match and cast the
> header as a String.
Now I wonder if case/when === behavior will break someday...
^ permalink raw reply [relevance 6%]
* Re: [PATCH] Make the gem usable as a git gem
@ 2022-07-08 12:12 5% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2022-07-08 12:12 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@gmail.com> wrote:
> Ok, so I’m sorry, but I’ve now tried for hours and
> I can’t for the life of me figure out how to send a patch
> cleanly with the tools I have available.
Ugh, I guess your employer doesn't want you fixing Linux kernel bugs,
either :<
> So you can download the patch here:
> https://github.com/byroot/unicorn/commit/e7f49852875de54cce974c7cbdd5540ddc5e4eeb.patch
OK, I suggest setting up mirrors on repo.or.cz or somewhere
else, as well, for redundancy and accessibility behind
blockades/firewalls.
> ——
>
> Bundler allow to install arbitrary revision
> of a gem via its git repository, e.g.
>
> ```ruby
> gem "unicorn", git: "https://yhbt.net/unicorn.git"
> ```
>
> But this currently doesn't work with unicorn because
> some files are not committed, and they are only generated
> by the makefile.
>
> This change would make it much easier to test
> unreleased versions.
Understood; but I'm not in favor of having generated files in
version control due to noise from needless diffs/dirty-states.
And I just had problems in another project because of generated
hunks the other day.
I suggest pointing the gemfile towards your own repo and
rebasing this patch on top of upstream, instead.
> NB: I didn't commit ext/unicorn_http/unicorn_http.c
> because I think you might as well generate it and commit
> it yourself so that you don't have to review it.
Minor changes to the .rl files (or Ragel itself) can lead noisy
comment/whitespace changes; and a constant dirty index if two
contributors are using different revisions of Ragel[1].
I encounter constant dirty index with autotools and gnulib-using
projects across different systems, too; and it's a PITA to deal
with when trying to fix platform-specific bugs.
> --- /dev/null
> +++ b/lib/unicorn/version.rb
> @@ -0,0 +1 @@
> +Unicorn::Const::UNICORN_VERSION = '6.1.0.dirty'
That's a regression, a version based on `git describe'
can be quite informative, especially when somebody isn't
sure if they've loaded the correct version or not.
[1] on a side note, I started getting rid of the Ragel dependency
last year (replacing with vendored picohttpparser), but got
sidetracked with Real-Life crap which I'm still dealing with.
The problem is Ragel 7 (experimental) won't be compatible with
6, and FreeBSD only packages 7 while Debian-based is staying
with 6 for now. There were nasty incompatibilities going from
Ragel 5->6 back in the Mongrel days, too; so I'd rather just go
with a standalone C parser I've been abusing elsewhere.
^ permalink raw reply [relevance 5%]
* Re: [PATCH] Master promotion with SIGURG (CoW optimization)
@ 2022-07-07 10:23 5% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2022-07-07 10:23 UTC (permalink / raw)
To: Jean Boussier; +Cc: unicorn-public
Jean Boussier <jean.boussier@shopify.com> wrote:
> > OK, any numbers from Puma users which can be used to project
> > improvements for unicorn (and other servers)?
>
> There are some user reports here: https://github.com/puma/puma/issues/2258
> but they are mixed in reports for two other new featurs.
>
> Some reports are up to 20-30% savings, but I'd expect unicorn to benefit
> even more from it, given that typical puma users spawn less processes
> than with unicorn.
OK, those numbers sound very good (but I can't view
graphics/screenshots from w3m)
> > > They also include automatic reforking after a predetermined amount
> > > of requests (1k by default).
> >
> > 1k seems like a lot of requests (more on this below)
>
> Agreed. Shared pages start being invalidated much faster than that.
>
> > > - master: on `SIGURG`
> > > - Forward `SIGURG` to a chosen worker.
> >
> > OK, I guess SIGURG is safe to use since nobody else relies on it (right?).
>
> That's my understanding, an alternative could be to re-use USR2 and have a
> config flag to define wether it is a rexec or refork.
SIGURG is fine, having SIGUSR2 mean two things seems fragile and error-prone
> > Right. PID file races and corner cases were painful to deal
> > with in the earliest days of this project, and I don't look
> > forward to having to deal with them ever again...
> >
> > It looked like the world was moving towards socket-activation
> > with dedicated process managers and away from fragile PID files;
> > so being self-daemonization dependent seems like a step back.
>
> Agreed. We're currently running unicorn as PID 1 inside containers
> and I'm not exactly looking forward to have to minitor a PID file.
>
> Another avenue I explored was to keep the existing master and
> refork from one of the worker like puma, but re-assign the parent
> to the orignal master using PR_SET_CHILD_SUBREAPER.
>
> Here's my notes of how it works:
>
> ### Requirements:
>
> - Need `PR_SET_CHILD_SUBREAPER`, so Linux 3.4+ only (May 2012)
I'm fine with requiring Linux for this feature, existing
features need to continue working on *BSD...
> - Need `File.mkfifo`, so Ruby 2.3 (Dec 2015), but can be shimed for older
> rubies.
I don't think it's necessary to use FIFOs at all (see below)
> ### Flow:
>
> - master: set `PR_SET_CHILD_SUBREAPER` during boot.
> - master: create a temporary directory (`$TMP`).
> - master: spawn initial workers the old way.
>
> - master: on `SIGCHLD` (a child died):
> - Fake signal oldest worker (`SIGURG`).
> - Write the new worker number in the fake signal pipe (at the same time).
>
> - worker: on `SIGURG` (should spawn a sibling):
> - Note: worker fake signals are processed after the current request is
> completed.
> - Run `before_fork` callback (MAYBE: need a special `before_refork`
> callback?)
> - create a pipe.
> - daemonize (fork twice so that the new process is attributed to the
> nearest `CHILD_SUBREAPER`, aka the original master).
> - wait for the first child to exit.
> - write into the pipe to signal the parent is dead.
> - Run `after_fork` callback (MAYBE: need a special `after_refork` callback?)
*_fork callbacks already work for your current patch, though, right?;
so hopefully *_refork won't be needed...
> - new sibling after fork:
> - wait for the parent to exit (though the temporary pipe).
> - create a named pipe (`File.mkfifo`) at `$TMP/$WORKER_NUM.pipe`.
> - create a pidfile at `$TMP/WORKER_NUM.pid`.
> - Open the named pipe in `IO::RDONLY | IO::NONBLOCK` (otherwise it would
> block until the master open in write mode).
> - NB: Need to convert it to a `Kgio::Pipe` with
> `Kgio::Pipe.for_fd(f.to_i)`, and keep `f` need `autoclose = false`.
sidenote: no need for kgio, I've been meaning to get rid of it
(Ruby 2.3+ has everything we'd need, I think...)
> - Create the `Unicorn::Worker` instance with that pipe and worker number.
> - Send `SIGURG` to the parent process (should be the master).
> - Wait for `SIGCONT` in the named pipe.
> - enter the worker loop.
>
> - master: on `SIGURG`:
> - for each outstanding refork order:
> - Look for `$TMP/$WORKER_NUM.pid` and `$TMP/$WORKER_NUM.pipe`
> - Read the pidfile
> - Open the pipe with `IO::RDONLY | IO::NONBLOCK`
> - NB: Need to convert it to a `Kgio::Pipe` with
> `Kgio::Pipe.for_fd(f.to_i)`, and keep `f` need `autoclose = false`.
> - Delete pidfile and named pipe.
> - Register that new worker normally.
> - Write `SIGCONT` into the pipe.
>
> I can share the patch if you are interested, but the extra complexity
> and Linux only features made me prefer the master-promotion approach.
I would prefer this Linux-only approach if it gets rid of PID
files and doesn't introduce new temporary files/FIFOs.
It seems socketpair(..., SOCK_SEQPACKET) can be used for
packetized bidirectional IPC, perhaps with send_io + recv_io iff
necessary. There shouldn't be any need for new, fragile FS
interactions.
Another idea, have you considered letting master work on some requests?
Signal handling would be delayed, and EINTR handling bugs in
gems could be exposed, but it'd be completely portable...
> > > However it work well enough for demonstration.
> > >
> > > So I figured I'd ask wether the feature is desired upstream
> > > before investing more effort into it.
> >
> > It really depends on:
> >
> > 1) what memory savings and/or speedups can be achieved
> >
> > 2) what support can we expect from you (and other users) for
> > this feature and potential regressions going forward?
> >
> > I don't have the free time nor mental capacity to deal with
> > fallout from major new features such as this, myself.
>
> Yeah, this is just a RFC, I wouldn't submit a final patch without
> first deploying it on our infra with good results. I'm just on the
> fence between trying to get this upstream vs maintaining our own
> server that solely work like this, hence somewhat simpler and that
> can make more assumptions.
Of course you also realize unicorn remains culturally-incompatible
with 99.9% of the open source world since I refuse to rely on
graphics drivers to work on a daemon, proprietary software,
terms-of-service, etc...
If anything goes wrong, I'd likely CC you anyways.
> > The hook interface would be yet another documentation+support
> > burden I'm worried about (and also more unicorn-specific stuff
> > to lock people in :<).
>
> Understandable. I suppose it can be done with a monitoring process.
> Check `smaps_rollup` and send `SIGURG` when deemed necessary.
>
> More moving pieces but keep unicorn simpler.
*shrug* I've also been favoring more vertical integration in
other places to keep the overall stack simpler.
It depends on whether the monitoring process/library can work
with other Rack servers, probably; and signal compatibility.
> > A completely different idea which should achieve the same goal,
> > completely independent of unicorn:
> >
> > App startup/loading can be made to trigger warmup paths which
> > hit common endpoints. IOW, app loading could run a small
> > warmup suite which simulates common requests to heat up caches
> > and trigger JIT.
>
> Yeah, I don't really believe in this for a few reasons:
>
> - That would slow boot time.
Yes, and startup time is already a nasty thing, anyways...
> - On large apps there's just too many codepath for this to be realistic.
I was thinking a lazy solution could be to have some middleware
measure coverage and log requests to support auto-replay, somehow.
> - Many endpoints require database and other network access you probably
> don't want in the master.
Wouldn't the *_fork hooks handle that, anyways?
> - Endpoints may have side effects.
Yeah, this would optimize the GET endpoints, at least.
Not sure what percentage of POST needs to be optimized...
> > Ultimately (long-term for ruby-core), it would be better to make
> > JIT (and perhaps even VM cache) results persistent and shareable
> > across different processes, somehow... That would help all Ruby
> > tools, even short-lived CLI tools. Portable locking would be a
> > pain, I suppose, but proprietary OSes are always a pain :P
>
> Based on my limited understanding of both JIT and VM caches, I don't think
> that's really possible.
Ugh, I still think it can be made possible, somehow. But I no longer
use Ruby for new projects...
> The VM itself could definitely do better CoW wise, and I have some proposals
> on that front (https://bugs.ruby-lang.org/issues/18885) but that will take
> time and will never be perfect.
>
> That's why I think periodically promoting a new master has potential.
>
> > There's some line-wrapping issues caused by your MUA;
>
> Urk. Ok, trying another client now, and I'll resend the patch.
Nope, still wrapped. According to the git-format-patch(1) manpage,
Thunderbird can work w/o extensions, actually.
IME attachments might work mostly well, nowadays (test sending
to yourself and using "git am" to apply, first, of course).
Most on vger will disagree with me on attachments, though...
"git send-email" and "git imap-send" remain the recommended
options, but mutt works well, too.
> > Perhaps documenting this as experimental and subject to removal
> > at any time would make the addition of a major new feature an
> > easier pill to swallow; but I still think this is better outside
> > of app servers.
>
> Up to you, if you don't feel like maintaining such a feature I would
> perfectly understand.
I'm fine with "maintaining it" if it just means Cc-ing you on
bugs related to this :>
^ permalink raw reply [relevance 5%]
* [PATCH 6/6] use EPOLLEXCLUSIVE on Linux 4.5+
@ 2021-10-01 3:09 3% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2021-10-01 3:09 UTC (permalink / raw)
To: unicorn-public
While the capabilities of epoll cannot be fully exploited given
our primitive design; avoiding thundering herd wakeups on larger
SMP machines while below 100% utilization is possible with
Linux 4.5+.
With this change, only one worker wakes up per-connect(2)
(instead of all of them via select(2)), avoiding the thundering
herd effect when the system is mostly idle.
Saturated instances should not notice the difference if they
rarely had multiple workers sleeping in select(2). This change
benefits non-saturated instances.
With 2 parallel clients and 8 workers on a nominally (:P)
8-core CPU (AMD FX-8320), the uconnect.perl test script
invocation showed a reduction from ~3.4s to ~2.5s when
reading an 11-byte response body:
echo worker_processes 8 >u.conf.rb
bs=11 ruby -I lib -I test/ruby-2.5.5/ext/unicorn_http/ bin/unicorn \
test/benchmark/dd.ru -E none -l /tmp/u.sock -c u.conf.rb
time perl -I lib -w test/benchmark/uconnect.perl \
-n 100000 -c 2 /tmp/u.sock
Times improve less as "-c" increases for uconnect.perl (system
noise and timings are inconsistent). The benefit of this change
should be more noticeable on systems with more workers (and
more cores).
I wanted to use EPOLLET (Edge-Triggered) to further reduce
syscalls, here, (similar to the old select()-avoidance bet) but
that would've either added too much complexity to deduplicate
wakeup sources, or run into the same starvation problem we
solved in April 2020[1].
Since the kernel already has the complexity and deduplication
built-in for Level-Triggered epoll support, we'll just let the
kernel deal with it.
Note: do NOT take this as an example of how epoll should be used
in a sophisticated server. unicorn is primitive by design and
cannot use threads nor handle multiple clients at once, thus it
it only uses epoll in this extremely limited manner.
Linux 4.5+ users will notice a regression of one extra epoll FD
per-worker and at least two epoll watches, so
/proc/sys/fs/epoll/max_user_watches may need to be changed along
with RLIMIT_NOFILE.
This change has also been tested on Linux 3.10.x (CentOS 7.x)
and FreeBSD 11.x to ensure compatibility with systems without
EPOLLEXCLUSIVE.
Various EPOLLEXCLUSIVE discussions over the years:
https://yhbt.net/lore/lkml/?q=s:EPOLLEXCLUSIVE+d:..20211001&x=t&o=-1
[1] https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
---
ext/unicorn_http/epollexclusive.h | 125 ++++++++++++++++++++++++++++++
ext/unicorn_http/extconf.rb | 1 +
ext/unicorn_http/unicorn_http.rl | 3 +
lib/unicorn/http_server.rb | 17 +++-
lib/unicorn/select_waiter.rb | 6 ++
t/test-lib.sh | 3 +-
test/unit/test_waiter.rb | 34 ++++++++
7 files changed, 184 insertions(+), 5 deletions(-)
create mode 100644 ext/unicorn_http/epollexclusive.h
create mode 100644 lib/unicorn/select_waiter.rb
create mode 100644 test/unit/test_waiter.rb
diff --git a/ext/unicorn_http/epollexclusive.h b/ext/unicorn_http/epollexclusive.h
new file mode 100644
index 0000000..2d2a589
--- /dev/null
+++ b/ext/unicorn_http/epollexclusive.h
@@ -0,0 +1,125 @@
+/*
+ * This is only intended for use inside a unicorn worker, nowhere else.
+ * EPOLLEXCLUSIVE somewhat mitigates the thundering herd problem for
+ * mostly idle processes since we can't use blocking accept4.
+ * This is NOT intended for use with multi-threaded servers, nor
+ * single-threaded multi-client ("C10K") servers or anything advanced
+ * like that. This use of epoll is only appropriate for a primitive,
+ * single-client, single-threaded servers like unicorn that need to
+ * support SIGKILL timeouts and parent death detection.
+ */
+#if defined(HAVE_EPOLL_CREATE1)
+# include <sys/epoll.h>
+# include <errno.h>
+# include <ruby/io.h>
+# include <ruby/thread.h>
+#endif /* __linux__ */
+
+#if defined(EPOLLEXCLUSIVE) && defined(HAVE_EPOLL_CREATE1)
+# define USE_EPOLL (1)
+#else
+# define USE_EPOLL (0)
+#endif
+
+#if USE_EPOLL
+/*
+ * :nodoc:
+ * returns IO object if EPOLLEXCLUSIVE works and arms readers
+ */
+static VALUE prep_readers(VALUE cls, VALUE readers)
+{
+ long i;
+ int epfd = epoll_create1(EPOLL_CLOEXEC);
+ VALUE epio;
+
+ if (epfd < 0) rb_sys_fail("epoll_create1");
+
+ epio = rb_funcall(cls, rb_intern("for_fd"), 1, INT2NUM(epfd));
+
+ Check_Type(readers, T_ARRAY);
+ for (i = 0; i < RARRAY_LEN(readers); i++) {
+ int rc;
+ struct epoll_event e;
+ rb_io_t *fptr;
+ VALUE io = rb_ary_entry(readers, i);
+
+ e.data.u64 = i; /* the reason readers shouldn't change */
+
+ /*
+ * I wanted to use EPOLLET here, but maintaining our own
+ * equivalent of ep->rdllist in Ruby-space doesn't fit
+ * our design at all (and the kernel already has it's own
+ * code path for doing it). So let the kernel spend
+ * cycles on maintaining level-triggering.
+ */
+ e.events = EPOLLEXCLUSIVE | EPOLLIN;
+ io = rb_io_get_io(io);
+ GetOpenFile(io, fptr);
+ rc = epoll_ctl(epfd, EPOLL_CTL_ADD, fptr->fd, &e);
+ if (rc < 0) rb_sys_fail("epoll_ctl");
+ }
+ return epio;
+}
+#endif /* USE_EPOLL */
+
+#if USE_EPOLL
+struct ep_wait {
+ struct epoll_event *events;
+ rb_io_t *fptr;
+ int maxevents;
+ int timeout_msec;
+};
+
+static void *do_wait(void *ptr) /* runs w/o GVL */
+{
+ struct ep_wait *epw = ptr;
+
+ return (void *)(long)epoll_wait(epw->fptr->fd, epw->events,
+ epw->maxevents, epw->timeout_msec);
+}
+
+/* :nodoc: */
+/* readers must not change between prepare_readers and get_readers */
+static VALUE
+get_readers(VALUE epio, VALUE ready, VALUE readers, VALUE timeout_msec)
+{
+ struct ep_wait epw;
+ long i, n;
+ VALUE buf;
+
+ Check_Type(ready, T_ARRAY);
+ Check_Type(readers, T_ARRAY);
+ epw.maxevents = RARRAY_LENINT(readers);
+ buf = rb_str_buf_new(sizeof(struct epoll_event) * epw.maxevents);
+ epw.events = (struct epoll_event *)RSTRING_PTR(buf);
+ epio = rb_io_get_io(epio);
+ GetOpenFile(epio, epw.fptr);
+
+ epw.timeout_msec = NUM2INT(timeout_msec);
+ n = (long)rb_thread_call_without_gvl(do_wait, &epw, RUBY_UBF_IO, NULL);
+ if (n < 0) {
+ if (errno != EINTR) rb_sys_fail("epoll_wait");
+ n = 0;
+ }
+ /* Linux delivers events in order received */
+ for (i = 0; i < n; i++) {
+ struct epoll_event *ev = &epw.events[i];
+ VALUE obj = rb_ary_entry(readers, ev->data.u64);
+
+ if (RTEST(obj))
+ rb_ary_push(ready, obj);
+ }
+ rb_str_resize(buf, 0);
+ rb_gc_force_recycle(buf);
+ return Qfalse;
+}
+#endif /* USE_EPOLL */
+
+static void init_epollexclusive(VALUE mUnicorn)
+{
+#if USE_EPOLL
+ VALUE cWaiter = rb_define_class_under(mUnicorn, "Waiter", rb_cIO);
+ rb_define_singleton_method(cWaiter, "prep_readers", prep_readers, 1);
+ rb_define_method(cWaiter, "get_readers", get_readers, 3);
+#endif
+}
diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb
index 46070a7..13dec41 100644
--- a/ext/unicorn_http/extconf.rb
+++ b/ext/unicorn_http/extconf.rb
@@ -38,4 +38,5 @@
message("no, needs Ruby 2.6+\n")
end
+have_func('epoll_create1', %w(sys/epoll.h))
create_makefile("unicorn_http")
diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index e934a32..605b23f 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -12,6 +12,7 @@
#include "common_field_optimization.h"
#include "global_variables.h"
#include "c_util.h"
+#include "epollexclusive.h"
void init_unicorn_httpdate(void);
@@ -1017,5 +1018,7 @@ void Init_unicorn_http(void)
id_clear = rb_intern("clear");
#endif
id_is_chunked_p = rb_intern("is_chunked?");
+
+ init_epollexclusive(mUnicorn);
}
#undef SET_GLOBAL
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 7f33f98..21f2a05 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -685,7 +685,6 @@ def init_worker_process(worker)
LISTENERS.each { |sock| sock.close_on_exec = true }
worker.user(*user) if user.kind_of?(Array) && ! worker.switched
- self.timeout /= 2.0 # halve it for select()
@config = nil
build_app! unless preload_app
@after_fork = @listener_opts = @orig_app = nil
@@ -705,11 +704,22 @@ def reopen_worker_logs(worker_nr)
exit!(77) # EX_NOPERM in sysexits.h
end
+ def prep_readers(readers)
+ wtr = Unicorn::Waiter.prep_readers(readers)
+ @timeout *= 500 # to milliseconds for epoll, but halved
+ wtr
+ rescue
+ require_relative 'select_waiter'
+ @timeout /= 2.0 # halved for IO.select
+ Unicorn::SelectWaiter.new
+ end
+
# runs inside each forked worker, this sits around and waits
# for connections and doesn't die until the parent dies (or is
# given a INT, QUIT, or TERM signal)
def worker_loop(worker)
readers = init_worker_process(worker)
+ waiter = prep_readers(readers)
reopen = false
# this only works immediately if the master sent us the signal
@@ -722,8 +732,7 @@ def worker_loop(worker)
begin
reopen = reopen_worker_logs(worker.nr) if reopen
worker.tick = time_now.to_i
- tmp = ready.dup
- while sock = tmp.shift
+ while sock = ready.shift
# Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
# but that will return false
if client = sock.kgio_tryaccept
@@ -735,7 +744,7 @@ def worker_loop(worker)
# timeout so we can .tick and keep parent from SIGKILL-ing us
worker.tick = time_now.to_i
- ret = IO.select(readers, nil, nil, @timeout) and ready = ret[0]
+ waiter.get_readers(ready, readers, @timeout)
rescue => e
redo if reopen && readers[0]
Unicorn.log_error(@logger, "listen loop error", e) if readers[0]
diff --git a/lib/unicorn/select_waiter.rb b/lib/unicorn/select_waiter.rb
new file mode 100644
index 0000000..cb84aab
--- /dev/null
+++ b/lib/unicorn/select_waiter.rb
@@ -0,0 +1,6 @@
+# fallback for non-Linux and Linux <4.5 systems w/o EPOLLEXCLUSIVE
+class Unicorn::SelectWaiter # :nodoc:
+ def get_readers(ready, readers, timeout) # :nodoc:
+ ret = IO.select(readers, nil, nil, timeout) and ready.replace(ret[0])
+ end
+end
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 7f97958..e70d0c6 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -94,7 +94,8 @@ check_stderr () {
set +u
_r_err=${1-${r_err}}
set -u
- if grep -v $T $_r_err | grep -i Error
+ if grep -v $T $_r_err | grep -i Error | \
+ grep -v NameError.*Unicorn::Waiter
then
die "Errors found in $_r_err"
elif grep SIGKILL $_r_err
diff --git a/test/unit/test_waiter.rb b/test/unit/test_waiter.rb
new file mode 100644
index 0000000..0995de2
--- /dev/null
+++ b/test/unit/test_waiter.rb
@@ -0,0 +1,34 @@
+require 'test/unit'
+require 'unicorn'
+require 'unicorn/select_waiter'
+class TestSelectWaiter < Test::Unit::TestCase
+
+ def test_select_timeout # n.b. this is level-triggered
+ sw = Unicorn::SelectWaiter.new
+ IO.pipe do |r,w|
+ sw.get_readers(ready = [], [r], 0)
+ assert_equal [], ready
+ w.syswrite '.'
+ sw.get_readers(ready, [r], 1000)
+ assert_equal [r], ready
+ sw.get_readers(ready, [r], 0)
+ assert_equal [r], ready
+ end
+ end
+
+ def test_linux # ugh, also level-triggered, unlikely to change
+ IO.pipe do |r,w|
+ wtr = Unicorn::Waiter.prep_readers([r])
+ wtr.get_readers(ready = [], [r], 0)
+ assert_equal [], ready
+ w.syswrite '.'
+ wtr.get_readers(ready = [], [r], 1000)
+ assert_equal [r], ready
+ wtr.get_readers(ready = [], [r], 1000)
+ assert_equal [r], ready, 'still ready (level-triggered :<)'
+ assert_nil wtr.close
+ end
+ rescue SystemCallError => e
+ warn "#{e.message} (#{e.class})"
+ end if Unicorn.const_defined?(:Waiter)
+end
^ permalink raw reply related [relevance 3%]
* [PATCH] prevent single listener from monopolizing a worker
@ 2020-04-16 9:24 5% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2020-04-16 9:24 UTC (permalink / raw)
To: Stan Hu; +Cc: unicorn-public
Stan Hu <stanhu@gmail.com> wrote:
> That seems to work, thanks!
Thanks for confirming. I'll push the patch below out.
(ugh, dealing with crazy packet loss all around)
Expect a v5.6.0 release within a few days or week at most.
(hopefully no regressions).
And... I wonder, are most deployments nowadays single listener?
I don't think I've used multiple listeners for this aside from
experiments in the early days.
---------8<----------
Subject: [PATCH] prevent single listener from monopolizing a worker
In setups with multiple listeners, it's possible for our greedy
select(2)-avoidance optimization to get pinned on a single, busy
listener and starve the other listener(s).
Prevent starvation by retrying the select(2)-avoidance
optimization if and only if all listeners were active. This
should have no effect on the majority of deployments with only a
single listener.
Thanks for Stan Hu for reporting and testing.
Reported-by: Stan Hu <stanhu@gmail.com>
Tested-by: Stan Hu <stanhu@gmail.com>
Link: https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
---
lib/unicorn/http_server.rb | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index a52931a..45a2e97 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -686,6 +686,7 @@ def worker_loop(worker)
trap(:USR1) { nr = -65536 }
ready = readers.dup
+ nr_listeners = readers.size
@after_worker_ready.call(self, worker)
begin
@@ -708,7 +709,7 @@ def worker_loop(worker)
# we're probably reasonably busy, so avoid calling select()
# and do a speculative non-blocking accept() on ready listeners
# before we sleep again in select().
- unless nr == 0
+ if nr == nr_listeners
tmp = ready.dup
redo
end
^ permalink raw reply related [relevance 5%]
* Re: [PATCH] check_client_connection: use tcp state on linux
2017-03-08 12:06 7% ` Simon Eskildsen
@ 2017-03-13 20:16 0% ` Simon Eskildsen
0 siblings, 0 replies; 17+ results
From: Simon Eskildsen @ 2017-03-13 20:16 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
I've put ccc-tcp-v3 in production today--it appears to be working as
expected, still rejecting at the exact same rate as before (100-200
per second for us during steady-state).
On Wed, Mar 8, 2017 at 7:06 AM, Simon Eskildsen
<simon.eskildsen@shopify.com> wrote:
> Patch-set looks great Eric, thanks!
>
> I'm hoping to test this in production later this week or next, but
> we're a year behind on Unicorn (ugh), so will need to carefully roll
> that out.
>
> On Tue, Mar 7, 2017 at 7:26 PM, Eric Wong <e@80x24.org> wrote:
>> Eric Wong <e@80x24.org> wrote:
>>> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
>>> > @@ -28,6 +29,7 @@ class Unicorn::HttpParser
>>> > # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
>>> > # 2.2+ optimizes hash assignments when used with literal string keys
>>> > HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
>>> > + EMPTY_ARRAY = [].freeze
>>>
>>> (minor) this was made before commit e06b699683738f22
>>> ("http_request: freeze constant strings passed IO#write")
>>> but I've trivially fixed it up, locally.
>>
>> And I actually screwed it up, pushed out ccc-tcp-v2 branch
>> with that fix squashed in :x
>>
>> Writing tests, now...
^ permalink raw reply [relevance 0%]
* Re: [PATCH] check_client_connection: use tcp state on linux
@ 2017-03-08 12:06 7% ` Simon Eskildsen
2017-03-13 20:16 0% ` Simon Eskildsen
0 siblings, 1 reply; 17+ results
From: Simon Eskildsen @ 2017-03-08 12:06 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn-public
Patch-set looks great Eric, thanks!
I'm hoping to test this in production later this week or next, but
we're a year behind on Unicorn (ugh), so will need to carefully roll
that out.
On Tue, Mar 7, 2017 at 7:26 PM, Eric Wong <e@80x24.org> wrote:
> Eric Wong <e@80x24.org> wrote:
>> Simon Eskildsen <simon.eskildsen@shopify.com> wrote:
>> > @@ -28,6 +29,7 @@ class Unicorn::HttpParser
>> > # Drop these frozen strings when Ruby 2.2 becomes more prevalent,
>> > # 2.2+ optimizes hash assignments when used with literal string keys
>> > HTTP_RESPONSE_START = [ 'HTTP', '/1.1 ']
>> > + EMPTY_ARRAY = [].freeze
>>
>> (minor) this was made before commit e06b699683738f22
>> ("http_request: freeze constant strings passed IO#write")
>> but I've trivially fixed it up, locally.
>
> And I actually screwed it up, pushed out ccc-tcp-v2 branch
> with that fix squashed in :x
>
> Writing tests, now...
^ permalink raw reply [relevance 7%]
* subscription to new ML probably working...
@ 2014-05-08 8:43 6% Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2014-05-08 8:43 UTC (permalink / raw)
To: mongrel-unicorn; +Cc: unicorn-public
I'll migrate everybody else over around 20140514T20:00Z, otherwise you
might get duplicate messages from being subscribed to both lists.
If you're willing to deal with dupes for a few days, you can try
subscribe/unsubscribe/help commands:
unicorn-public+subscribe@bogomips.org
unicorn-public+unsubscribe@bogomips.org
unicorn-public+help@bogomips.org
(help will tell you the rest)
Stop reading here unless you're interested in implementation details :>
I only have mlmmj configured to handle the +commands, outgoing SMTP,
and bounces. It does not receive normal list traffic like a standard
mlmmj install. Instead, I have it configured to replay data from ssoma
every minute, so I used (as the mlmmj user):
NAME=unicorn-public
URL=git://bogomips.org/unicorn-public.git
ssoma add $NAME $URL "command:ssoma-replay -L $HOME/spool/$NAME"
Where ssoma-replay is a tiny shell script which reads from stdin:
------------------------------8<--------------------------
#!/bin/sh
MSG=$(mktemp -t ssoma-replay.orig.$USER.XXXXXX || exit 1)
cat > "$MSG"
exec /usr/bin/mlmmj-send "$@" -m "$MSG"
------------------------------8<--------------------------
This means it should be easy to have outgoing SMTP on a different server
and allow other folks to run the same service.
Other notes: mlmmj has defaults I like out-of-the-box (unlike mailman,
ugh). However, archive retrieval with mlmmj is rather painful,
but we have public-inbox and ssoma :)
[1] http://mlmmj.org/
^ permalink raw reply [relevance 6%]
* Re: Issues with PID file renaming
2013-11-26 1:40 0% ` Michael Fischer
@ 2013-11-26 2:07 0% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2013-11-26 2:07 UTC (permalink / raw)
To: unicorn list
Michael Fischer <mfischer@zendesk.com> wrote:
> On Mon, Nov 25, 2013 at 5:20 PM, Eric Wong <normalperson@yhbt.net> wrote:
>
> >> What I notice is the following: when upgrading with 4.6.2 the file
> >> unicorn.pid is copied to unicorn.pid.oldbin and the unicorn.pid file
> >> is updated (the (new) master process). After a while the worker pid
> >> files are updated and the unicorn.pid.oldbin file disappears, and all
> >> is fine.
> >
> > Ugh. This is what I feared... Slow startup time of most Ruby web apps
> > doesn't help.
>
> Why doesn't the new master write its pid file immediately upon
> forking? It shouldn't have to wait for everything else to load,
> should it?
It used to do this in 4.6, but I wanted to support dropping the PID
later in case of config failures. I'll revert that part of the change.
> > How about having the old process create a hard link to .oldbin,
> > and having the new one override the pid if Process.ppid == pid file?
> > The check is still racy, but that's what pid files are :<
>
> If you decide to implement this, please make it optional. The current
> behavior is correct IMO, and it will work much better if the above
> issue is addressed.
I've decided against that :)
Just pushed this out to git://bogomips.org/unicorn
as 795c3527337ff4f03ae6db08c5df01141565ed96
---------------------------- 8< -----------------------------
Subject: [PATCH] always write PID file early for compatibility
This reduces the window for a non-existent PID for folks who monitor
PIDs (not a great idea anyways). Unfortunately, this change also brings
us back to the case where having a PID later (for other process monitors)
is beneficial but more unicorn releases exist where we write the PID
early.
Thanks to Jimmy Soho for reporting this issue.
ref: <CAHStS5gFYcPBDxkVizAHrOeDKAkjT69kruFdgaY0CbB+vLbK8Q@mail.gmail.com>
This partially reverts 7d6ac0c17eb29a00a5b74099dbb3d4d015999f27
Folks: please monitor your app with HTTP requests rather than checking
processes, a stuck/wedged Ruby VM is still a running one.
---
lib/unicorn/http_server.rb | 13 ++++---------
1 file changed, 4 insertions(+), 9 deletions(-)
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 402f133..f15c8a7 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -136,20 +136,15 @@ class Unicorn::HttpServer
trap(:CHLD) { awaken_master }
# write pid early for Mongrel compatibility if we're not inheriting sockets
- # This was needed for compatibility with some health checker a long time
- # ago. This unfortunately has the side effect of clobbering valid PID
- # files.
- self.pid = config[:pid] unless ENV["UNICORN_FD"]
+ # This is needed for compatibility some Monit setups at least.
+ # This unfortunately has the side effect of clobbering valid PID if
+ # we upgrade and the upgrade breaks during preload_app==true && build_app!
+ self.pid = config[:pid]
self.master_pid = $$
build_app! if preload_app
bind_new_listeners!
- # Assuming preload_app==false, we drop the pid file after the app is ready
- # to process requests. If binding or build_app! fails with
- # preload_app==true, we'll never get here and the parent will recover
- self.pid = config[:pid] if ENV["UNICORN_FD"]
-
spawn_missing_workers
self
end
--
EW
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [relevance 0%]
* Re: Issues with PID file renaming
2013-11-26 1:20 6% ` Eric Wong
@ 2013-11-26 1:40 0% ` Michael Fischer
2013-11-26 2:07 0% ` Eric Wong
0 siblings, 1 reply; 17+ results
From: Michael Fischer @ 2013-11-26 1:40 UTC (permalink / raw)
To: unicorn list
On Mon, Nov 25, 2013 at 5:20 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> What I notice is the following: when upgrading with 4.6.2 the file
>> unicorn.pid is copied to unicorn.pid.oldbin and the unicorn.pid file
>> is updated (the (new) master process). After a while the worker pid
>> files are updated and the unicorn.pid.oldbin file disappears, and all
>> is fine.
>
> Ugh. This is what I feared... Slow startup time of most Ruby web apps
> doesn't help.
Why doesn't the new master write its pid file immediately upon
forking? It shouldn't have to wait for everything else to load,
should it?
> How about having the old process create a hard link to .oldbin,
> and having the new one override the pid if Process.ppid == pid file?
> The check is still racy, but that's what pid files are :<
If you decide to implement this, please make it optional. The current
behavior is correct IMO, and it will work much better if the above
issue is addressed.
--Michael
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: Issues with PID file renaming
@ 2013-11-26 1:20 6% ` Eric Wong
2013-11-26 1:40 0% ` Michael Fischer
0 siblings, 1 reply; 17+ results
From: Eric Wong @ 2013-11-26 1:20 UTC (permalink / raw)
To: unicorn list
Jimmy Soho <jimmy.soho@gmail.com> wrote:
> Hi Eric,
>
> Since we upgraded from 4.6.2 to 4.7.0 we're regularly having issues
> where one or more unicorn master processes would not upgrade
> correctly, resulting in an (old) master process.
>
> What I notice is the following: when upgrading with 4.6.2 the file
> unicorn.pid is copied to unicorn.pid.oldbin and the unicorn.pid file
> is updated (the (new) master process). After a while the worker pid
> files are updated and the unicorn.pid.oldbin file disappears, and all
> is fine.
Ugh. This is what I feared... Slow startup time of most Ruby web apps
doesn't help.
> When upgrading with 4.7.0 though the file unicorn.pid.oldbin is
> created, but there is no unicorn.pid file. After a while (when the new
> master process has finished initialising, and is ready to fork the
> workers) the worker pid files are updated and a unicorn.pid file
> appears, and then the unicorn.pid.oldbin file disappears.
>
> So there is a relatively long period where there is no unicorn.pid file.
>
> I think the problem for us is caused by monit, our process monitor,
> which monitors the unicorn.pid file:
>
> check process unicorn with pidfile
> /srv/app.itrp-staging.com/shared/pids/unicorn.pid
> start program = "/etc/init.d/unicorn start"
> stop program = "/etc/init.d/unicorn stop"
> ...
Is there an alternate way to monitor unicorn with monit?
But I'd like to keep your case working if possible.
> Because the unicorn.pid file is absent for a relatively long period
> monit will try to start unicorn, while an upgrade is in progress.
> (which might be another issue: the start action should recognise a
> start or upgrade process is already underway; sadly this check too
> relies on the existence of the unicorn.pid file)
>
> I think the way 4.6.2 worked is better. There should be a pid file for
> the new master process the moment it's created.
> What do you think?
How about having the old process create a hard link to .oldbin,
and having the new one override the pid if Process.ppid == pid file?
The check is still racy, but that's what pid files are :<
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
* Re: HEAD responses contain body
@ 2013-06-13 19:21 6% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2013-06-13 19:21 UTC (permalink / raw)
To: Jonathan Rudenberg; +Cc: unicorn list
Jonathan Rudenberg <jonathan@titanous.com> wrote:
> On Jun 13, 2013, at 2:22 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Jonathan Rudenberg <jonathan@titanous.com> wrote:
> >> RFC 2616 section 9.4[1] states:
> >>
> >>> The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response.
> >>
> >> A HEAD request against this simple Rack app running on unicorn-4.6.2:
> >>
> >> require 'rack'
> >>
> >
> > + use Rack::Head
> >
> >> run lambda { |env| [200, {}, []] }
> >
> > The Rack::Head middleware should be used to correctly strip HEAD
> > responses of their bodies (frameworks such as Rails/Sinatra should
> > already add Rack::Head to the middleware stack for you)
>
> This does not change the result, as the Rack::Head implementation looks like this:
>
> def call(env)
> status, headers, body = @app.call(env)
>
> if env["REQUEST_METHOD"] == "HEAD"
> body.close if body.respond_to? :close
> [status, headers, []]
> else
> [status, headers, body]
> end
> end
OK, I think you were hitting another problem because you were lacking
Rack::ContentType
Try the following:
-----------------------8<---------------------
require 'rack'
use Rack::ContentLength # less ambiguous than Rack::Chunked adding '0'
use Rack::Head
use Rack::ContentType
run lambda { |env| [200, {}, []] }
-----------------------8<---------------------
I added the Rack::ContentLength (it's already in the default middleware
stack) since I believe Rack::Chunked adding the '0' is a violation of
rfc2616... I'll need to read more closely to be sure.
> >> HTTP/1.1 500 Internal Server Error
> >
> >> As you can see, not only is there a zero-length chunked encoding body,
> >> but for some unknown reason there is a 500 response with no body as
> >> well.
> >
> > Try using "-d" on the command-line to enable debugging to see what the
> > error is (and check the logs/stderr output).
>
> Exception `Errno::ENOTCONN' at /Users/titanous/.gem/ruby/1.9.3/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:565 - Socket is not connected
Ugh, that's an unfortunate side effect of the client closing the
connection, first :/
> > Also, what RACK_ENV (or -E/--env) are you using? It could be the
> > incorrect HEAD response tripping Rack::Lint under development mode.
>
> None, specified, I'm booting unicorn with no configuration or flags specified.
That defaults the RACK_ENV to to "development", so you got
Rack::ContentLength, Rack::Chunked, Rack::CommonLogger,
Rack::ShowExceptions and Rack::Lint
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
* Re: Address already in use
@ 2012-06-25 23:57 6% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2012-06-25 23:57 UTC (permalink / raw)
To: unicorn list
Manuel Palenciano Guerrero <mpalenciano@gmail.com> wrote:
> Hi,
>
> First, thanks Eric, Jérémy and Aaron for replying. I really appreciate it.
>
> Yes Eric, I can see the line... "inherited addr=/tmp/unicorn.app.sock fd=..."
>
> here is the full log
>
> -------------------------------------------------
> I, [2012-06-21T11:40:44.282224 #29212] INFO -- : inherited addr=/tmp/unicorn.sublimma_staging.sock fd=3
> I, [2012-06-21T11:40:44.282480 #29212] INFO -- : Refreshing Gem list
> master process ready
> worker=0 ready
> worker=1 ready
> reaped #<Process::Status: pid=28683,exited(0)> worker=0
> reaped #<Process::Status: pid=28684,exited(0)> worker=1
> master complete
Ugh, lack of formatting caused by Rails mucking with Logger is annoying.
Can you add the following to your unicorn config?
Configurator::DEFAULTS[:logger].formatter = Logger::Formatter.new
(that's from unicorn.bogomips.org/FAQ.html)
> E, [2012-06-21T11:40:46.386486 #29401] ERROR -- : adding listener failed addr=/tmp/unicorn.sublimma_staging.sock (in use)
I'm curious if PID=29401 is a worker of pid 29212. Or you're starting
another master somewhere...
> /var/www/sublimma/staging/shared/bundle/ruby/1.8/gems/unicorn-4.3.0/lib/unicorn/socket_helper.rb:140:in `initialize': Address already in use - /tmp/unicorn.sublimma_staging.sock (Errno::EADDRINUSE)
> -------------------------------------------------
>
> my unicorn.rb: https://gist.github.com/2991110
Did you share the correct config? Your config has:
listen "/tmp/unicorn.app_production.sock"
But your config has: /tmp/unicorn.sublimma_staging.sock
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
* Re: Running unicorn gracefully on Heroku
@ 2012-03-07 20:22 7% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2012-03-07 20:22 UTC (permalink / raw)
To: unicorn list; +Cc: Seamus Abshere
Seamus Abshere <seamus@abshere.net> wrote:
> I just posted a StackOverflow question about running unicorn
> gracefully on Heroku...
>
> http://stackoverflow.com/questions/9605703/how-can-i-tell-unicorn-to-understand-herokus-signals
Ugh, can you just send/copy the question over here next time so we don't
have to follow a link and parse out? Thanks.
> Would this group be able to provide any wisdom?
It's best to talk to the Heroku folks about this since it's their
service. Maybe some of them hang out on this ML...
> PS. Please CC: me on any responses!
Done, thanks for the heads up.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 7%]
* Re: unicorn worker partially restarted when using USR2
@ 2011-05-07 17:54 6% ` Eric Wong
0 siblings, 0 replies; 17+ results
From: Eric Wong @ 2011-05-07 17:54 UTC (permalink / raw)
To: unicorn list
Stone <stones.gao@gmail.com> wrote:
> In the deployment I did kill -s USR2 `cat unicorn_pid` to gracefully
> restart unicorn. But I noticed that unicorn workers were partially
> restarted when using USR2.
You need to send a SIGQUIT to the original master process, otherwise
both masters and all workers continue running. This allows you to test
the new deploy (combined with SIGWINCH).
The last section of http://unicorn.bogomips.org/SIGNALS.html
describes this process in detail.
> The following is how I did test:
>
> start unicorn in production environment:
>
> the action has the following code:
Ugh, don't you have something like "ps axf" that shows you a process
tree? It's much easier just to *see* the process tree.
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
* Re: Unicorn 0.97.0 old master is never dying
2010-03-19 8:23 6% ` Eric Wong
@ 2010-03-19 8:55 0% ` ghazel
0 siblings, 0 replies; 17+ results
From: ghazel @ 2010-03-19 8:55 UTC (permalink / raw)
To: Eric Wong; +Cc: unicorn list
>> Here is the entire config file: http://codepad.org/v6lUsuzD
>
> Ugh. I can't even get to that file right now. I actually prefer
> to have config files inline so:
>
> a) I can comment easily on them
> b) I don't have to be online when reading them
> (I sync my mail to a local machine and lose connectivity often)
We're still working on that internet that works when you're not
online. In the meantime:
# Sample configuration file for Unicorn (not Rack)
#
# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
# documentation.
require '/data/myapp/current/config/unicorn_constants'
# Use at least one worker per core if you're on a dedicated server,
# more will usually help for _short_ waits on databases/caches.
worker_processes UnicornConstants::NUM_WORKERS
# Help ensure your application will always spawn in the symlinked
# "current" directory that Capistrano sets up.
working_directory "/data/myapp/current" # available in 0.94.0+
# listen on both a Unix domain socket and a TCP port,
# we use a shorter backlog for quicker failover when busy
listen "/tmp/unicorn.sock", :backlog => 64
listen 8080, :tcp_nopush => true, :tries => -1
# nuke workers after X seconds instead of (60 seconds default)
timeout 300
# feel free to point this anywhere accessible on the filesystem
pid "/data/myapp/shared/pids/unicorn.pid"
# some applications/frameworks log to stderr or stdout, so prevent
# them from going to /dev/null when daemonized here:
stderr_path "/data/myapp/shared/log/unicorn.stderr.log"
stdout_path "/data/myapp/shared/log/unicorn.stdout.log"
# combine REE with "preload_app true" for memory savings
# http://rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
GC.copy_on_write_friendly = true
before_fork do |server, worker|
# the following is highly recomended for Rails + "preload_app true"
# as there's no need for the master process to hold a connection
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
# The following is only recommended for memory/DB-constrained
# installations. It is not needed if your system can house
# twice as many worker_processes as you have configured.
#
# This allows a new master process to incrementally
# phase out the old master process with SIGTTOU to avoid a
# thundering herd (especially in the "preload_app false" case)
# when doing a transparent upgrade. The last worker spawned
# will then kill off the old master process with a SIGQUIT.
old_pid = "#{server.config[:pid]}.oldbin"
if File.exists?(old_pid) && server.pid != old_pid
begin
Process.kill("QUIT", File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
# someone else did our job for us
end
end
end
after_fork do |server, worker|
# per-process listener ports for debugging/admin/migrations
addr = "127.0.0.1:#{9000 + worker.nr}"
server.listen(addr, :tries => -1, :delay => 5, :tcp_nopush => true)
# the following is *required* for Rails + "preload_app true",
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
# if preload_app is true, then you may also want to check and
# restart any other shared sockets/descriptors such as Memcached,
# and Redis. TokyoCabinet file handles are safe to reuse
# between any number of forked children (assuming your kernel
# correctly implements pread()/pwrite() system calls)
end
-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 0%]
* Re: Unicorn 0.97.0 old master is never dying
@ 2010-03-19 8:23 6% ` Eric Wong
2010-03-19 8:55 0% ` ghazel
0 siblings, 1 reply; 17+ results
From: Eric Wong @ 2010-03-19 8:23 UTC (permalink / raw)
To: ghazel; +Cc: unicorn list
ghazel@gmail.com wrote:
> On Thu, Mar 18, 2010 at 1:40 AM, Eric Wong <normalperson@yhbt.net> wrote:
> > ghazel@gmail.com wrote:
> >> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but
> >> sending a USR2 to the 0.97.0 master launches a new master and all the
> >> workers, but never kills the old master. The new workers retry binding
> >> their admin TCP ports forever. Killing all the Unicorns and
> >> downgrading to returns peace to the forest.
> >>
> >> Here are the before_fork and after_fork from my config/unicorn.rb:
> >> http://codepad.org/5LyVtyq7
> >>
> >> Did something change with the Unicorn master in 0.97.0 that would
> >> require a change in my config, or is this a bug?
> >>
> >> -Greg
> >
> > Hi Greg,
> >
> > Odd, are you passing the pid file path via the command-line?
> >
> > I don't see the pid directive in the config file you posted there, so
> > there was nothing to check with File.exists?. Are you using `unicorn'
> > or `unicorn_rails'?
> >
> > There were some startup changes in 0.97.0, but I couldn't reproduce what
> > you're seeing with 'pid'. That said, I'm not sure if that method of
> > killing old masters in the after_fork is very common.
> >
> > --
> > Eric Wong
> >
>
> Eric,
>
> I am using "unicorn_rails -D -E $ENV -c config/unicorn.rb". Here is
> the entire config file: http://codepad.org/v6lUsuzD
Ugh. I can't even get to that file right now. I actually prefer
to have config files inline so:
a) I can comment easily on them
b) I don't have to be online when reading them
(I sync my mail to a local machine and lose connectivity often)
On a side note, maybe an strace/truss can help figure out what's
going on. James seems to be having a similar problem in the other
thread...
> What is the preferred way to do an nginx-style restart? My system does
> not really have the memory to start twice the number of workers.
If you're doing it during low traffic periods, I would SIGTTOU away
most (maybe all but one) of your workers and then SIGHUP the master
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [relevance 6%]
Results 1-17 of 17 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2010-03-17 9:51 Unicorn 0.97.0 old master is never dying ghazel
2010-03-18 8:40 ` Eric Wong
2010-03-18 16:35 ` ghazel
2010-03-19 8:23 6% ` Eric Wong
2010-03-19 8:55 0% ` ghazel
2011-05-07 11:54 unicorn worker partially restarted when using USR2 Stone
2011-05-07 17:54 6% ` Eric Wong
2012-03-07 16:49 Running unicorn gracefully on Heroku Seamus Abshere
2012-03-07 20:22 7% ` Eric Wong
2012-06-25 13:02 Address already in use Manuel Palenciano Guerrero
2012-06-25 20:28 ` Eric Wong
2012-06-25 21:03 ` Manuel Palenciano Guerrero
2012-06-25 23:57 6% ` Eric Wong
2013-06-13 16:09 HEAD responses contain body Jonathan Rudenberg
2013-06-13 18:22 ` Eric Wong
2013-06-13 18:42 ` Jonathan Rudenberg
2013-06-13 19:21 6% ` Eric Wong
2013-11-26 1:00 Issues with PID file renaming Jimmy Soho
2013-11-26 1:20 6% ` Eric Wong
2013-11-26 1:40 0% ` Michael Fischer
2013-11-26 2:07 0% ` Eric Wong
2014-05-08 8:43 6% subscription to new ML probably working Eric Wong
2017-02-25 14:03 [PATCH] check_client_connection: use tcp state on linux Simon Eskildsen
2017-02-25 16:19 ` Simon Eskildsen
2017-02-25 23:12 ` Eric Wong
2017-02-27 11:44 ` Simon Eskildsen
2017-02-28 21:12 ` Eric Wong
2017-03-01 3:18 ` Eric Wong
2017-03-06 21:32 ` Simon Eskildsen
2017-03-07 22:50 ` Eric Wong
2017-03-08 0:26 ` Eric Wong
2017-03-08 12:06 7% ` Simon Eskildsen
2017-03-13 20:16 0% ` Simon Eskildsen
2020-04-15 5:06 Sustained queuing on one listener can block requests from other listeners Stan Hu
2020-04-15 5:26 ` Eric Wong
2020-04-16 5:46 ` Stan Hu
2020-04-16 6:59 ` Eric Wong
2020-04-16 7:24 ` Stan Hu
2020-04-16 9:24 5% ` [PATCH] prevent single listener from monopolizing a worker Eric Wong
2021-10-01 3:09 [PATCH 0/6] reduce thundering herds on Linux 4.5+ Eric Wong
2021-10-01 3:09 3% ` [PATCH 6/6] use EPOLLEXCLUSIVE " Eric Wong
2022-07-05 20:05 [PATCH] Master promotion with SIGURG (CoW optimization) Jean Boussier
2022-07-06 2:33 ` Eric Wong
2022-07-06 7:40 ` Jean Boussier
2022-07-07 10:23 5% ` Eric Wong
2022-07-08 7:46 [PATCH] Make the gem usable as a git gem Jean Boussier
2022-07-08 12:12 5% ` Eric Wong
2022-12-09 21:52 Support for Rack 3 headers formatted as arrays Martin Posthumus
2022-12-10 2:42 ` Eric Wong
2022-12-12 15:52 ` Martin Posthumus
2022-12-25 22:56 ` Eric Wong
2023-01-11 11:12 ` Jean Boussier
2023-01-11 11:40 6% ` Eric Wong
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).