From normalperson at yhbt.net Wed Jan 5 18:57:02 2011 From: normalperson at yhbt.net (Eric Wong) Date: Wed, 5 Jan 2011 23:57:02 +0000 Subject: [ANN] unicorn 3.3.0 - minor optimizations Message-ID: <20110105235702.GA15982@dcvr.yhbt.net> Certain applications that already serve hundreds/thousands of requests a second (per-worker) should experience performance improvements due to Time.now.httpdate usage being removed and reimplemented in C. There are also minor internal changes and cleanups for Rainbows! * http://unicorn.bogomips.org/ * mongrel-unicorn at rubyforge.org * git://git.bogomips.org/unicorn.git -- Eric Wong From normalperson at yhbt.net Thu Jan 6 02:20:01 2011 From: normalperson at yhbt.net (Eric Wong) Date: Wed, 5 Jan 2011 23:20:01 -0800 Subject: [PATCH] close client socket after closing response body Message-ID: <20110106072001.GA28721@dcvr.yhbt.net> I am wondering if there are any apps affected by this bug (and perhaps keeping people from switching Unicorn). It's a fairly esoteric case, so I probably won't make another release until tomorrow (sleepy now, will probably screw something else up or realize something else is broken :) Anyways it's pushed out to master and 1.1.x-stable in case people want^Wneed it *right* *now*: >From b72a86f66c722d56a6d77ed1d2779ace6ad103ed Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Wed, 5 Jan 2011 22:39:03 -0800 Subject: [PATCH] close client socket after closing response body Response bodies may capture the block passed to each and save it for body.close, so don't close the socket before we have a chance to call body.close --- lib/unicorn/http_response.rb | 1 - lib/unicorn/http_server.rb | 1 + t/t0018-write-on-close.sh | 23 +++++++++++++++++++++++ t/write-on-close.ru | 11 +++++++++++ (Unnecessary unit test case omitted for email) test/unit/test_response.rb | 18 +++++++++--------- 5 files changed, 44 insertions(+), 10 deletions(-) create mode 100755 t/t0018-write-on-close.sh create mode 100644 t/write-on-close.ru diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb index 3a03cd6..62b3ee9 100644 --- a/lib/unicorn/http_response.rb +++ b/lib/unicorn/http_response.rb @@ -38,7 +38,6 @@ module Unicorn::HttpResponse end body.each { |chunk| socket.write(chunk) } - socket.close # flushes and uncorks the socket immediately ensure body.respond_to?(:close) and body.close end diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index e2a4db7..3a6e51e 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -538,6 +538,7 @@ class Unicorn::HttpServer end @request.headers? or headers = nil http_response_write(client, status, headers, body) + client.close # flush and uncork socket immediately, no keepalive rescue => e handle_error(client, e) end diff --git a/t/t0018-write-on-close.sh b/t/t0018-write-on-close.sh new file mode 100755 index 0000000..3afefea --- /dev/null +++ b/t/t0018-write-on-close.sh @@ -0,0 +1,23 @@ +#!/bin/sh +. ./test-lib.sh +t_plan 4 "write-on-close tests for funky response-bodies" + +t_begin "setup and start" && { + unicorn_setup + unicorn -D -c $unicorn_config write-on-close.ru + unicorn_wait_start +} + +t_begin "write-on-close response body succeeds" && { + test xGoodbye = x"$(curl -sSf http://$listen/)" +} + +t_begin "killing succeeds" && { + kill $unicorn_pid +} + +t_begin "check stderr" && { + check_stderr +} + +t_done diff --git a/t/write-on-close.ru b/t/write-on-close.ru new file mode 100644 index 0000000..54a2f2e --- /dev/null +++ b/t/write-on-close.ru @@ -0,0 +1,11 @@ +class WriteOnClose + def each(&block) + @callback = block + end + + def close + @callback.call "7\r\nGoodbye\r\n0\r\n\r\n" + end +end +use Rack::ContentType, "text/plain" +run(lambda { |_| [ 200, [%w(Transfer-Encoding chunked)], WriteOnClose.new ] }) -- Eric Wong From normalperson at yhbt.net Thu Jan 6 19:16:11 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 7 Jan 2011 00:16:11 +0000 Subject: [ANN] unicorn 3.3.1 and 1.1.6 - one minor, esoteric bugfix In-Reply-To: <20110106072001.GA28721@dcvr.yhbt.net> References: <20110106072001.GA28721@dcvr.yhbt.net> Message-ID: <20110107001610.GB7639@dcvr.yhbt.net> Eric Wong wrote: > I am wondering if there are any apps affected by this bug (and > perhaps keeping people from switching Unicorn). > > It's a fairly esoteric case, so I probably won't make another release > until tomorrow (sleepy now, will probably screw something else up > or realize something else is broken :) > > Anyways it's pushed out to master and 1.1.x-stable in case people > want^Wneed it *right* *now*: I've just released v1.1.6 and v3.3.1 with the fix from last night, maybe somebody in a dark corner of the web can finally migrate to Unicorn because of this bugfix :) -- Eric Wong From jimmy.soho at gmail.com Fri Jan 7 21:00:51 2011 From: jimmy.soho at gmail.com (Jimmy Soho) Date: Sat, 8 Jan 2011 13:00:51 +1100 Subject: Thread.current Message-ID: Hi All, Having an akward noob moment... I have unicorn_rails (1.1.5) running with 2 workers, with rails 2.3.10 in development mode. In environment.rb at the bottom I have this line of code: puts "#{Time.current} #{Thread.current.object_id}: #{Thread.current.keys.inspect}" In a simple controller I have this: def index puts "#{Time.current} #{Thread.current.object_id}: #{Thread.current.keys.inspect}" sleep 5 puts "#{Time.current} #{Thread.current.object_id}: #{Thread.current.keys.inspect}" render :text => "foo" end In window 1 I tail log/unicorn.log. In windows 2 and 3 I start at about the same time: curl http://localhost:3000 The output is this: 2011-01-08 01:53:56 UTC 2148444460: [:__inspect_key__, :i18n_config, :__recursive_key__] worker=1 ready 2011-01-08 01:53:56 UTC 2148444460: [:__inspect_key__, :i18n_config, :__recursive_key__] worker=0 ready 2011-01-08 01:53:57 UTC 2148444460: [:__inspect_key__, :i18n_config, :__recursive_key__] 2011-01-08 01:53:59 UTC 2148444460: [:__inspect_key__, :i18n_config, :__recursive_key__] 2011-01-08 01:54:02 UTC 2148444460: [:__inspect_key__, :i18n_config, :__recursive_key__] 2011-01-08 01:54:04 UTC 2148444460: [:__inspect_key__, :i18n_config, :__recursive_key__] Looking at the timings the 2 requests seem to have been handled in parallel, as expected. However, the Thread.current value within those parallel requests is always the same. Are two separate requests not handled by different threads? How does that work?? Cheers, Jim From normalperson at yhbt.net Fri Jan 7 21:57:20 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 7 Jan 2011 18:57:20 -0800 Subject: Thread.current In-Reply-To: References: Message-ID: <20110108025720.GA16042@dcvr.yhbt.net> Jimmy Soho wrote: > Hi All, > > Having an akward noob moment... I have unicorn_rails (1.1.5) running > with 2 workers, with rails 2.3.10 in development mode. In > environment.rb at the bottom I have this line of code: > > puts "#{Time.current} #{Thread.current.object_id}: #{Thread.current.keys.inspect}" Add the PID ($$), too: puts "#{Time.current} #$$ #{Thread.current.object_id}: #{Thread.current.keys.inspect}" > In a simple controller I have this: > > def index > puts "#{Time.current} #{Thread.current.object_id}: #{Thread.current.keys.inspect}" > sleep 5 > puts "#{Time.current} #{Thread.current.object_id}: #{Thread.current.keys.inspect}" > render :text => "foo" > end > > In window 1 I tail log/unicorn.log. > In windows 2 and 3 I start at about the same time: curl http://localhost:3000 > > The output is this: > > 2011-01-08 01:53:56 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > worker=1 ready > 2011-01-08 01:53:56 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > worker=0 ready > 2011-01-08 01:53:57 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > 2011-01-08 01:53:59 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > 2011-01-08 01:54:02 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > 2011-01-08 01:54:04 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > > Looking at the timings the 2 requests seem to have been handled in > parallel, as expected. Yes, that's because you have two worker *processes* > However, the Thread.current value within those parallel requests is > always the same. Just a lucky coincidence :) > Are two separate requests not handled by different threads? How does > that work?? Threads (and any other object) object_ids are unique to each process. They are not unique within all the processes of a machine. When dealing with native threads under Linux, gettid() is a non-portable Linux syscall to get the unique identifier of a thread throughout the entire system. That's the only way I know of if you want a single unique identifier (and of course prepending the PID to it). -- Eric Wong From curtis.schofield at gmail.com Fri Jan 7 22:09:08 2011 From: curtis.schofield at gmail.com (Curtis j Schofield) Date: Fri, 7 Jan 2011 19:09:08 -0800 Subject: Thread.current In-Reply-To: References: Message-ID: On Fri, Jan 7, 2011 at 6:00 PM, Jimmy Soho wrote: > Hi All, > > Having an akward noob moment... I have unicorn_rails (1.1.5) running > with 2 workers, with rails 2.3.10 in development mode. In > environment.rb at the bottom I have this line of code: > > ? ?puts "#{Time.current} #{Thread.current.object_id}: > #{Thread.current.keys.inspect}" > > In a simple controller I have this: > > ? def index > ? ? puts "#{Time.current} #{Thread.current.object_id}: > #{Thread.current.keys.inspect}" > ? ? sleep 5 > ? ? puts "#{Time.current} #{Thread.current.object_id}: > #{Thread.current.keys.inspect}" > ? ? render :text => "foo" > ? end > > In window 1 I tail log/unicorn.log. > In windows 2 and 3 I start at about the same time: ?curl http://localhost:3000 > > The output is this: > > 2011-01-08 01:53:56 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > worker=1 ready > 2011-01-08 01:53:56 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > worker=0 ready > 2011-01-08 01:53:57 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > 2011-01-08 01:53:59 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > 2011-01-08 01:54:02 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > 2011-01-08 01:54:04 UTC 2148444460: [:__inspect_key__, :i18n_config, > :__recursive_key__] > > Looking at the timings the 2 requests seem to have been handled in > parallel, as expected. > > However, the Thread.current value within those parallel requests is > always the same. > > Are two separate requests not handled by different threads? How does that work?? > Unicorn is a multi-process model with Inter-process communication - more akin to a unix service - threads are not considered a wise investment in the ruby community. Worker1 / Worker0 are entirely separate processes - as is the master unicorn process. Review Fork for more details. http://en.wikipedia.org/wiki/Fork_(operating_system) From jimmy.soho at gmail.com Sat Jan 8 00:54:37 2011 From: jimmy.soho at gmail.com (Jimmy Soho) Date: Sat, 8 Jan 2011 16:54:37 +1100 Subject: Thread.current In-Reply-To: References:

Message-ID: >>> However, the Thread.current value within those parallel requests is >>> always the same. EW> Just a lucky coincidence :) I guess. Got confused there for a moment because of it. :) Using $$ made things more clear. Thanks Jimmy From jimmy.soho at gmail.com Tue Jan 11 17:52:12 2011 From: jimmy.soho at gmail.com (Jimmy Soho) Date: Wed, 12 Jan 2011 09:52:12 +1100 Subject: Thread.current In-Reply-To: References:

Message-ID: Hi, Some more questions still: It seems a worker uses the exact same thread to handle each request. Is that guaranteed to happen for the lifetime of a worker? Or are there cases where a unicorn worker might spin a new thread to handle the next requests? If the same thread is always used, isn't that a potential issue when programmers use thread local variables, which are not reset at the next request? (I know, the usage of thread local variables is not recommended, but take a random rails project, go into their $GEM_HOME and do grep -r Thread.current . , see what I mean..) Cheers, Jimmy On Sat, Jan 8, 2011 at 4:54 PM, Jimmy Soho wrote: >>>> However, the Thread.current value within those parallel requests is >>>> always the same. > > EW> Just a lucky coincidence :) > > I guess. Got confused there for a moment because of it. :) ?Using $$ > made things more clear. > > > > Thanks > > Jimmy > From normalperson at yhbt.net Tue Jan 11 18:12:27 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 11 Jan 2011 15:12:27 -0800 Subject: Thread.current In-Reply-To: References:

Message-ID: <20110111231227.GA27058@dcvr.yhbt.net> Jimmy Soho wrote: > Hi, > > Some more questions still: > > It seems a worker uses the exact same thread to handle each request. Correct. > Is that guaranteed to happen for the lifetime of a worker? Or are > there cases where a unicorn worker might spin a new thread to handle > the next requests? Unicorn itself is always single-threaded and never spawns new threads. > If the same thread is always used, isn't that a potential issue when > programmers use thread local variables, which are not reset at the > next request? (I know, the usage of thread local variables is not > recommended, but take a random rails project, go into their $GEM_HOME > and do grep -r Thread.current . , see what I mean..) Thats the problem of those libraries/apps, not Unicorn. They can try Rainbows! using the :ThreadSpawn option which behaves much like Mongrel 1, but uses Rainbows! in production that I know of. The Rack +env+ hash is the safe/universal way to store request-local variables across different web servers. -- Eric Wong From jpr5 at darkridge.com Tue Jan 11 18:12:51 2011 From: jpr5 at darkridge.com (Jordan Ritter) Date: Tue, 11 Jan 2011 15:12:51 -0800 Subject: Thread.current In-Reply-To: References:

Message-ID: Unicorn is purely about employing a multi-process model, not a multi-thread model; it specifically avoids spawning threads to handle inbound requests. In fact, I'll bet that inside each request, Thread.current == Thread.main. Separate from Unicorn, when running a rack-compatilbe app in multithreaded mode (the default when the app is invoked directly via rackup + config.ru), there's no guarantee about which thread will service a given request. This fact may not matter to you, depending on what you're trying to do. That said, you *could* use Thread local storage for per-request storage in either unicorn or multithreaded situations, so long as you wiped your storage at the beginning/end of each request -- but that's a crappy idiom, even if it might be "common" (don't know what you're referring to offhand). Can't suggest a more appropriate pattern without knowing more about what you're actually trying to do. cheers, --jordan On Jan 11, 2011, at 2:52 PM, Jimmy Soho wrote: > Hi, > > Some more questions still: > > It seems a worker uses the exact same thread to handle each request. > > Is that guaranteed to happen for the lifetime of a worker? Or are > there cases where a unicorn worker might spin a new thread to handle > the next requests? > > If the same thread is always used, isn't that a potential issue when > programmers use thread local variables, which are not reset at the > next request? (I know, the usage of thread local variables is not > recommended, but take a random rails project, go into their $GEM_HOME > and do grep -r Thread.current . , see what I mean..) > From jimmy.soho at gmail.com Tue Jan 11 22:07:38 2011 From: jimmy.soho at gmail.com (Jimmy Soho) Date: Wed, 12 Jan 2011 14:07:38 +1100 Subject: Thread.current In-Reply-To: References:

Message-ID: Some component don't always have to work within a webserver context, and therefor assume no access to a Rack +env+ hash. Widely used examples are the i18n and the active_support gem. In our case we have for example queued jobs that are executed with the full rails environment loaded, which does not have a rack context. This is not an issue, but does explain (to some extent) why some components use thread local storage instead of the rack +env+ context. I'm not trying to use Thread local storage myself, it is forced upon us. ;-) I'm trying to determine if the components we must use due to dependencies, and which do use Thread local storage, if they are leaking data from one request into the next request if you are within a unicorn context. There are cases where we want this, and there are cases where we don't want this leakage. Take for example activesupport's usage of Time.zone. Under water this is set in a thread local var. If you set Time.zone in one request, but not in the next request, using unicorn the next request will use the time zone of the previous request. Using rack or mongrel (in multithreaded mode) you don't have this issue perse (though they have other issues then). Same for the i18n gem and it's usage of the I18n.locale value, which is also set in a thread local var. So yeah, unfortunately I have to take into account this "crappy idiom" and need to know exactly which thread local vars are set by all the components we use, and determine which of those must be reset before each request. Cheers, Jimmy On Wed, Jan 12, 2011 at 10:12 AM, Jordan Ritter wrote: > Unicorn is purely about employing a multi-process model, not a multi-thread model; it specifically avoids spawning threads to handle inbound requests. ? In fact, I'll bet that inside each request, Thread.current == Thread.main. > > Separate from Unicorn, when running a rack-compatilbe app in multithreaded mode (the default when the app is invoked directly via rackup + config.ru), there's no guarantee about which thread will service a given request. ?This fact may not matter to you, depending on what you're trying to do. > > That said, you *could* use Thread local storage for per-request storage in either unicorn or multithreaded situations, so long as you wiped your storage at the beginning/end of each request -- but that's a crappy idiom, even if it might be "common" (don't know what you're referring to offhand). ?Can't suggest a more appropriate pattern without knowing more about what you're actually trying to do. > > cheers, > --jordan > > On Jan 11, 2011, at 2:52 PM, Jimmy Soho wrote: > >> Hi, >> >> Some more questions still: >> >> It seems a worker uses the exact same thread to handle each request. >> >> Is that guaranteed to happen for the lifetime of a worker? Or are >> there cases where a unicorn worker might spin a new thread to handle >> the next requests? >> >> If the same thread is always used, isn't that a potential issue when >> programmers use thread local variables, which are not reset at the >> next request? ?(I know, the usage of thread local variables is not >> recommended, but take a random rails project, go into their $GEM_HOME >> and do grep -r Thread.current . , see what I mean..) >> > > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying > From normalperson at yhbt.net Wed Jan 12 23:26:36 2011 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 13 Jan 2011 04:26:36 +0000 Subject: Thread.current In-Reply-To: References:

Message-ID: <20110113042636.GA8264@dcvr.yhbt.net> Jimmy Soho wrote: > Take for example activesupport's usage of Time.zone. Under water this > is set in a thread local var. If you set Time.zone in one request, but > not in the next request, using unicorn the next request will use the > time zone of the previous request. Using rack or mongrel (in > multithreaded mode) you don't have this issue perse (though they have > other issues then). > > Same for the i18n gem and it's usage of the I18n.locale value, which > is also set in a thread local var. > > So yeah, unfortunately I have to take into account this "crappy idiom" > and need to know exactly which thread local vars are set by all the > components we use, and determine which of those must be reset before > each request. You can probably just write a trivial middleware to clear all keys in Thread.current before every request. Or play around with Rainbows! with a single-threaded ThreadSpawn: cat >> unicorn.conf.rb < References:

<20110113042636.GA8264@dcvr.yhbt.net> Message-ID: <28E5C7A1-2B77-4B90-AEE3-C2812A327366@darkridge.com> For the record, the "clear the Thread.current storage before/after each request" is what I think is a crappy idiom. YMMV I guess. cheers, --jordan On Jan 12, 2011, at 8:26 PM, Eric Wong wrote: > Jimmy Soho wrote: >> Take for example activesupport's usage of Time.zone. Under water this >> is set in a thread local var. If you set Time.zone in one request, but >> not in the next request, using unicorn the next request will use the >> time zone of the previous request. Using rack or mongrel (in >> multithreaded mode) you don't have this issue perse (though they have >> other issues then). >> >> Same for the i18n gem and it's usage of the I18n.locale value, which >> is also set in a thread local var. >> >> So yeah, unfortunately I have to take into account this "crappy idiom" >> and need to know exactly which thread local vars are set by all the >> components we use, and determine which of those must be reset before >> each request. > > You can probably just write a trivial middleware to clear all > keys in Thread.current before every request. Or play around with > Rainbows! with a single-threaded ThreadSpawn: > > cat >> unicorn.conf.rb < Rainbows! do > use :ThreadSpawn > worker_connections 1 > keepalive_timeout 0 > end > EOF > > And then just run "rainbows" instead of "unicorn". > > -- > Eric Wong > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying From ghazel at gmail.com Thu Jan 13 16:20:51 2011 From: ghazel at gmail.com (ghazel at gmail.com) Date: Thu, 13 Jan 2011 13:20:51 -0800 Subject: Unicorn 3.3.1 "Too many open files" error in kgio_tryaccept Message-ID: Strange error which took down my server today: Unhandled listen loop exception #. /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_server.rb:606:in `kgio_tryaccept' /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_server.rb:606:in `worker_loop' /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_server.rb:605:in `each' /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_server.rb:605:in `worker_loop' ... My unicorn.stderr.log is 7.6GB, and I rotate it every day... Before all that started I got: Read error: # /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_response.rb:37:in `write' /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_response.rb:37:in `http_response_write' /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_server.rb:540:in `process_client' /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_server.rb:607:in `worker_loop' /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_server.rb:605:in `each' /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_server.rb:605:in `worker_loop' But the servers seemed to be basically operating at that point. -Greg From normalperson at yhbt.net Thu Jan 13 18:06:09 2011 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 13 Jan 2011 23:06:09 +0000 Subject: Unicorn 3.3.1 "Too many open files" error in kgio_tryaccept In-Reply-To: References: Message-ID: <20110113230609.GA1519@dcvr.yhbt.net> ghazel at gmail.com wrote: > Strange error which took down my server today: > > Unhandled listen loop exception #. > ... > > My unicorn.stderr.log is 7.6GB, and I rotate it every day... > > Before all that started I got: > > Read error: # > /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_response.rb:37:in > > But the servers seemed to be basically operating at that point. Weird. I have many questions: Are you processing uploads at all? Is nginx in front of Unicorn? Do you open a lot of files in your application and never close them?[1] How does lsof output look for your workers? Can you check any exception capturing/logging middleware you have for any other errors that may be swallowed? (especially while reading env["rack.input"]) I changed the socket close ordering for 3.3.1 to workaround an esoteric "bug", but I don't see how this could happen right now... Does this happen with 3.3.0? Or 1.1.6? hiswhich has the same "bugfix" as 3.3.1. Also, is anybody else having this problem? [1] - GC should be able to cleanup open files on MRI/REE anyways -- Eric Wong From ghazel at gmail.com Thu Jan 13 21:17:41 2011 From: ghazel at gmail.com (ghazel at gmail.com) Date: Thu, 13 Jan 2011 18:17:41 -0800 Subject: Unicorn 3.3.1 "Too many open files" error in kgio_tryaccept In-Reply-To: <20110113230609.GA1519@dcvr.yhbt.net> References: <20110113230609.GA1519@dcvr.yhbt.net> Message-ID: On Thu, Jan 13, 2011 at 3:06 PM, Eric Wong wrote: > ghazel at gmail.com wrote: >> Strange error which took down my server today: >> >> Unhandled listen loop exception #. >> ... >> >> My unicorn.stderr.log is 7.6GB, and I rotate it every day... >> >> Before all that started I got: >> >> Read error: # >> /usr/local/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/unicorn-3.3.1/lib/unicorn/http_response.rb:37:in >> >> But the servers seemed to be basically operating at that point. > > Weird. ?I have many questions: > > Are you processing uploads at all? Yes, but I have been processing uploads on Unicorn since 0.99.something > Is nginx in front of Unicorn? Yes. > Do you open a lot of files in your application and never close them?[1] I don't believe so, and certain nothing has changed there in quite awhile. > How does lsof output look for your workers? Hm. The workers seem to be at 80-90 file descriptors each. I did catch one at 787 (!) with mostly handles to a geoip database from the geoip gem, but they got collected pretty quickly. Perhaps that's the cause! > Can you check any exception capturing/logging middleware you have > for any other errors that may be swallowed? (especially > while reading env["rack.input"]) I don't have any non-standard Rails 2.3.10 middleware which reads rack.input. > I changed the socket close ordering for 3.3.1 to workaround an esoteric > "bug", but I don't see how this could happen right now... > > Does this happen with 3.3.0? I've only seen it once. If I manage to reproduce it I'll try with 3.3.0. -Greg From normalperson at yhbt.net Thu Jan 13 21:47:53 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 14 Jan 2011 02:47:53 +0000 Subject: Unicorn 3.3.1 "Too many open files" error in kgio_tryaccept In-Reply-To: References: <20110113230609.GA1519@dcvr.yhbt.net> Message-ID: <20110114024753.GA2453@dcvr.yhbt.net> ghazel at gmail.com wrote: > On Thu, Jan 13, 2011 at 3:06 PM, Eric Wong wrote: > > ghazel at gmail.com wrote: > > How does lsof output look for your workers? > > Hm. The workers seem to be at 80-90 file descriptors each. I did catch > one at 787 (!) with mostly handles to a geoip database from the geoip > gem, but they got collected pretty quickly. Perhaps that's the cause! OK, that's a fairly likely cause of EMFILE. A tip for geoip users: Install the io-extra gem to get IO.pread. This allows you to reuse the same file descriptor with geoip automatically between any number of threads/processes without reopening it. > > Does this happen with 3.3.0? > > I've only seen it once. If I manage to reproduce it I'll try with 3.3.0. The 3.3.x series generates less garbage than previous versions, so it could be causing the GC to run less frequently and increase the chance of the GC failing to collect descriptors. -- Eric Wong From ghazel at gmail.com Thu Jan 13 23:20:00 2011 From: ghazel at gmail.com (ghazel at gmail.com) Date: Thu, 13 Jan 2011 20:20:00 -0800 Subject: Unicorn 3.3.1 "Too many open files" error in kgio_tryaccept In-Reply-To: <20110114024753.GA2453@dcvr.yhbt.net> References: <20110113230609.GA1519@dcvr.yhbt.net> <20110114024753.GA2453@dcvr.yhbt.net> Message-ID: On Thu, Jan 13, 2011 at 6:47 PM, Eric Wong wrote: > ghazel at gmail.com wrote: >> On Thu, Jan 13, 2011 at 3:06 PM, Eric Wong wrote: >> > ghazel at gmail.com wrote: >> > How does lsof output look for your workers? >> >> Hm. The workers seem to be at 80-90 file descriptors each. I did catch >> one at 787 (!) with mostly handles to a geoip database from the geoip >> gem, but they got collected pretty quickly. Perhaps that's the cause! > > OK, that's a fairly likely cause of EMFILE. > > A tip for geoip users: > > ?Install the io-extra gem to get IO.pread. ?This allows you to reuse > ?the same file descriptor with geoip automatically between any number > ?of threads/processes without reopening it. Fascinating. Google tells be you've been down this road before. My code previously was doing: def get_city(ip) GeoIP.new("/path/to/geo.dat").city(ip) end Which seems to create one fd per call (and leave it for the GC to cleanup). What's the new proposed interface if io-extra is installed? Keep a global GeoIP object somewhere? My fix was to stuff it in to Thread.current, but obviously that has one fd per thread. -Greg From normalperson at yhbt.net Fri Jan 14 04:17:47 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 14 Jan 2011 09:17:47 +0000 Subject: Unicorn 3.3.1 "Too many open files" error in kgio_tryaccept In-Reply-To: References: <20110113230609.GA1519@dcvr.yhbt.net> <20110114024753.GA2453@dcvr.yhbt.net> Message-ID: <20110114091747.GC30950@dcvr.yhbt.net> ghazel at gmail.com wrote: > On Thu, Jan 13, 2011 at 6:47 PM, Eric Wong wrote: > > ghazel at gmail.com wrote: > >> On Thu, Jan 13, 2011 at 3:06 PM, Eric Wong wrote: > >> > ghazel at gmail.com wrote: > >> > How does lsof output look for your workers? > >> > >> Hm. The workers seem to be at 80-90 file descriptors each. I did catch > >> one at 787 (!) with mostly handles to a geoip database from the geoip > >> gem, but they got collected pretty quickly. Perhaps that's the cause! > > > > OK, that's a fairly likely cause of EMFILE. > > > > A tip for geoip users: > > > > ?Install the io-extra gem to get IO.pread. ?This allows you to reuse > > ?the same file descriptor with geoip automatically between any number > > ?of threads/processes without reopening it. > > Fascinating. Google tells be you've been down this road before. My > code previously was doing: > def get_city(ip) > GeoIP.new("/path/to/geo.dat").city(ip) > end > > Which seems to create one fd per call (and leave it for the GC to > cleanup). What's the new proposed interface if io-extra is installed? > Keep a global GeoIP object somewhere? My fix was to stuff it in to > Thread.current, but obviously that has one fd per thread. Yes, with io-extra, just define a constant somewhere and use it in any thread/process at any time. No need to deal with locks or after_fork hooks in Unicorn at all, either, pread() is just that awesome :) GEO_DB = GeoIP.new("/path/to/geo.dat") def get_city(ip) GEO_DB.city(ip) end -- Eric Wong From normalperson at yhbt.net Fri Jan 21 15:37:12 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 21 Jan 2011 12:37:12 -0800 Subject: [PATCH] git.bogomips.org => bogomips.org Message-ID: <20110121203712.GA7986@dcvr.yhbt.net> If people are watching the cgit page, do not be alarmed, I'm just trimming the URL length to save precious bytes :D >From d770d09dfd9e5d7148379c58cdf9a020cbdc63b6 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Fri, 21 Jan 2011 12:28:39 -0800 Subject: [PATCH] git.bogomips.org => bogomips.org bogomips.org is slimming down and losing URL weight :) --- .wrongdoc.yml | 4 ++-- README | 4 ++-- Rakefile | 5 +++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/.wrongdoc.yml b/.wrongdoc.yml index 9121575..10f10b5 100644 --- a/.wrongdoc.yml +++ b/.wrongdoc.yml @@ -1,6 +1,6 @@ --- -cgit_url: http://git.bogomips.org/cgit/unicorn.git -git_url: git://git.bogomips.org/unicorn.git +cgit_url: http://bogomips.org/unicorn.git +git_url: git://bogomips.org/unicorn.git rdoc_url: http://unicorn.bogomips.org/ changelog_start: v1.1.5 merge_html: diff --git a/README b/README index b4bbae2..9dda04e 100644 --- a/README +++ b/README @@ -85,13 +85,13 @@ You may also install it via RubyGems on Gemcutter: You can get the latest source via git from the following locations (these versions may not be stable): - git://git.bogomips.org/unicorn.git + git://bogomips.org/unicorn.git git://repo.or.cz/unicorn.git (mirror) You may browse the code from the web and download the latest snapshot tarballs here: -* http://git.bogomips.org/cgit/unicorn.git (cgit) +* http://bogomips.org/unicorn.git (cgit) * http://repo.or.cz/w/unicorn.git (gitweb) See the HACKING guide on how to contribute and build prerelease gems diff --git a/Rakefile b/Rakefile index 598cf07..ffdf982 100644 --- a/Rakefile +++ b/Rakefile @@ -1,8 +1,9 @@ # -*- encoding: binary -*- autoload :Gem, 'rubygems' +require 'wrongdoc' -cgit_url = "http://git.bogomips.org/cgit/unicorn.git" -git_url = ENV['GIT_URL'] || 'git://git.bogomips.org/unicorn.git' +cgit_url = Wrongdoc.config[:cgit_url] +git_url = Wrongdoc.config[:git_url] desc "post to RAA" task :raa_update do -- Eric Wong From alexey.bondar at gmail.com Sun Jan 23 16:28:51 2011 From: alexey.bondar at gmail.com (Alexey Bondar) Date: Mon, 24 Jan 2011 00:28:51 +0300 Subject: "Read error: #" and uploads Message-ID: <061729D0-F2FD-4ABE-9AE7-418C81C2F467@gmail.com> Hi. Unicorn 3.3.1 fails with "Read error: #" on request with attachments bigger than ~150kb. Ruby: 1.9.2-p0 Rails: 3.0.1 Full stack trace: http://pastie.org/private/w1vwhbhdm9xtwwnpqog6wa Configuration: http://pastie.org/private/hdpiqs0d3xj3qmsjwg I use nginx 0.8.x as frontend server. Nginx client_body_buffer_size is 128mb. Any ides why this happens? From normalperson at yhbt.net Sun Jan 23 19:59:09 2011 From: normalperson at yhbt.net (Eric Wong) Date: Sun, 23 Jan 2011 16:59:09 -0800 Subject: "Read error: #" and uploads In-Reply-To: <061729D0-F2FD-4ABE-9AE7-418C81C2F467@gmail.com> References: <061729D0-F2FD-4ABE-9AE7-418C81C2F467@gmail.com> Message-ID: <20110124005909.GA24994@dcvr.yhbt.net> Alexey Bondar wrote: > Hi. > > Unicorn 3.3.1 fails with "Read error: #" on request with attachments bigger than ~150kb. > > Ruby: 1.9.2-p0 > Rails: 3.0.1 > > Full stack trace: http://pastie.org/private/w1vwhbhdm9xtwwnpqog6wa > Configuration: http://pastie.org/private/hdpiqs0d3xj3qmsjwg > > I use nginx 0.8.x as frontend server. Nginx client_body_buffer_size is 128mb. Anything in the nginx error log? Did this happen with Unicorn 1.1.x? The input code changed in the 3.x series drastically to support non-rewindable input, so some bugs could've crept in. -- Eric Wong From otokorasii at gmail.com Sun Jan 23 20:09:24 2011 From: otokorasii at gmail.com (Sean Grove) Date: Sun, 23 Jan 2011 17:09:24 -0800 Subject: Unicorn renaming unicorn.pid.oldbin back to unicorn.pid (or, is unicorn afraid to die?) Message-ID: <353A7398-DEBE-426B-9BC1-AB43334CFB4E@gmail.com> Whenever I send a USR2 signal to unicorn, it correctly renames the pid file from unicorn.pid to unicorn.pid.oldbin, and a new instance starts launching. However, almost immediately (~2 seconds), it renames it back to unicorn.pid. The new instance of unicorn finished launching to find that unicorn.pid is already running, and refuses to start up. I'm not sure if I've misconfigured something, but I'd love to get to the bottom of this! Thanks for the help! All the best, Sean Grove From normalperson at yhbt.net Mon Jan 24 14:09:39 2011 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 24 Jan 2011 19:09:39 +0000 Subject: Unicorn renaming unicorn.pid.oldbin back to unicorn.pid (or, is unicorn afraid to die?) In-Reply-To: <353A7398-DEBE-426B-9BC1-AB43334CFB4E@gmail.com> References: <353A7398-DEBE-426B-9BC1-AB43334CFB4E@gmail.com> Message-ID: <20110124190939.GA13545@dcvr.yhbt.net> Sean Grove wrote: > Whenever I send a USR2 signal to unicorn, it correctly renames the pid > file from unicorn.pid to unicorn.pid.oldbin, and a new instance starts > launching. However, almost immediately (~2 seconds), it renames it > back to unicorn.pid. The new instance of unicorn finished launching to > find that unicorn.pid is already running, and refuses to start up. > > I'm not sure if I've misconfigured something, but I'd love to get to > the bottom of this! Are you somehow clearing the UNICORN_FD environment variable in your master process (perhaps when you load your app). Which version of Ruby are you using? What does your stderr log say? -- Eric Wong From normalperson at yhbt.net Mon Jan 24 19:51:52 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 25 Jan 2011 00:51:52 +0000 Subject: "Read error: #" and uploads In-Reply-To: References: <061729D0-F2FD-4ABE-9AE7-418C81C2F467@gmail.com> <20110124005909.GA24994@dcvr.yhbt.net> <20110124181919.GA24783@dcvr.yhbt.net> Message-ID: <20110125005152.GB1921@dcvr.yhbt.net> Alexey Bondar wrote: (back to mailing list) > On Jan 24, 2011, at 21:19 , Eric Wong wrote: > > Eric Wong wrote: > >> Alexey Bondar wrote: > >>> Hi. > >>> > >>> Unicorn 3.3.1 fails with "Read error: #" on request with attachments bigger than ~150kb. > >>> > >>> Ruby: 1.9.2-p0 > >>> Rails: 3.0.1 > >>> > >>> Full stack trace: http://pastie.org/private/w1vwhbhdm9xtwwnpqog6wa > >>> Configuration: http://pastie.org/private/hdpiqs0d3xj3qmsjwg > >>> > >>> I use nginx 0.8.x as frontend server. Nginx client_body_buffer_size is 128mb. > >> > >> Anything in the nginx error log? Did this happen with Unicorn 1.1.x? > >> The input code changed in the 3.x series drastically to support > >> non-rewindable input, so some bugs could've crept in. > > In nginx log only two messages: body buffered to disk and backend timeout. Wait, nginx buffered the body to disk when client_body_buffer_size is 128mb and you have trouble with a 150kb request? > Same issue with 1.1.x OK. Which timeout exactly are you hitting in nginx? How slow is your client? Is your request queue very huge and you don't have enough Unicorn workers to handle the load? -- Eric Wong From normalperson at yhbt.net Mon Jan 24 22:17:57 2011 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 24 Jan 2011 19:17:57 -0800 Subject: "Read error: #" and uploads In-Reply-To: References: <061729D0-F2FD-4ABE-9AE7-418C81C2F467@gmail.com> <20110124005909.GA24994@dcvr.yhbt.net> <20110124181919.GA24783@dcvr.yhbt.net> <20110125005152.GB1921@dcvr.yhbt.net> Message-ID: <20110125031757.GA6871@dcvr.yhbt.net> Alexey Bondar wrote: > On Jan 25, 2011, at 03:51 , Eric Wong wrote: > > Wait, nginx buffered the body to disk when client_body_buffer_size is > > 128mb and you have trouble with a 150kb request? > > Oops, my fault: not body_buffer_size, but client_max_body_size. client_body_buffer_size is 128k. > > As I understand, in this case nginx should buffer whole request body, before sending to backend? Yes, nginx always buffers the whole request body > >> Same issue with 1.1.x > > > > OK. Which timeout exactly are you hitting in nginx? > > Logs: > 2011/01/23 21:09:05 [warn] 79095#0: *261 a client request body is buffered to a temporary file /var/spool/nginx-client-body/3/00/0000000003 > 2011/01/23 21:09:35 [error] 79095#0: *261 upstream timed out (60: Operation timed out) while sending request to upstream Very strange. Do you process env["rack.input"] directly from your app or do you let Rails/Rack handle everything? This probably won't help, but can you try increasing the proxy_send_timeout in nginx? What's your Unicorn timeout? From normalperson at yhbt.net Mon Jan 24 22:53:34 2011 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 24 Jan 2011 19:53:34 -0800 Subject: "Read error: #" and uploads In-Reply-To: <25C1DE1D-59D2-4E80-92E6-FDC0897814BA@gmail.com> References: <061729D0-F2FD-4ABE-9AE7-418C81C2F467@gmail.com> <20110124005909.GA24994@dcvr.yhbt.net> <20110124181919.GA24783@dcvr.yhbt.net> <20110125005152.GB1921@dcvr.yhbt.net> <20110125031757.GA6871@dcvr.yhbt.net> <25C1DE1D-59D2-4E80-92E6-FDC0897814BA@gmail.com> Message-ID: <20110125035334.GB8124@dcvr.yhbt.net> Alexey Bondar wrote: > Unicorn timeout is 75s. I just tried with 75s proxy_send_timeout: same > issue. What client are you using to send? I wonder if Content-Length is somehow screwed up because a client is trying to compress the data. What happens when you try hitting Unicorn directly without nginx? -- Eric Wong From normalperson at yhbt.net Mon Jan 24 23:10:44 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 25 Jan 2011 04:10:44 +0000 Subject: "Read error: #" and uploads In-Reply-To: <41BA1BA1-3DB7-45C2-8503-17026080D04A@gmail.com> References: <061729D0-F2FD-4ABE-9AE7-418C81C2F467@gmail.com> <20110124005909.GA24994@dcvr.yhbt.net> <20110124181919.GA24783@dcvr.yhbt.net> <20110125005152.GB1921@dcvr.yhbt.net> <20110125031757.GA6871@dcvr.yhbt.net> <25C1DE1D-59D2-4E80-92E6-FDC0897814BA@gmail.com> <20110125035334.GB8124@dcvr.yhbt.net> <41BA1BA1-3DB7-45C2-8503-17026080D04A@gmail.com> Message-ID: <20110125041044.GA10853@dcvr.yhbt.net> Alexey Bondar wrote: > On Jan 25, 2011, at 06:53 , Eric Wong wrote: > > What client are you using to send? I wonder if Content-Length is > > somehow screwed up because a client is trying to compress the data. > > Different: Webkit based browsers, FF, Opera. OK. I'm fairly certain those clients don't make compressed requests, but I'm out of ideas as to what could be wrong... > > What happens when you try hitting Unicorn directly without nginx? > > Hm... How I can directly hit unicorn via unix socket? Just have it listen on a TCP port, too (it can handle multiple listen statements in the config). Maybe we can narrow it down to something in the nginx config... -- Eric Wong From normalperson at yhbt.net Mon Jan 24 23:54:51 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 25 Jan 2011 04:54:51 +0000 Subject: "Read error: #" and uploads In-Reply-To: References: <20110124181919.GA24783@dcvr.yhbt.net> <20110125005152.GB1921@dcvr.yhbt.net> <20110125031757.GA6871@dcvr.yhbt.net> <25C1DE1D-59D2-4E80-92E6-FDC0897814BA@gmail.com> <20110125035334.GB8124@dcvr.yhbt.net> <41BA1BA1-3DB7-45C2-8503-17026080D04A@gmail.com> <20110125041044.GA10853@dcvr.yhbt.net> Message-ID: <20110125045451.GB10853@dcvr.yhbt.net> Alexey Bondar wrote: > On Jan 25, 2011, at 07:10 , Eric Wong wrote: > > OK. I'm fairly certain those clients don't make compressed requests, > > but I'm out of ideas as to what could be wrong... > > > > Just have it listen on a TCP port, too (it can handle multiple > > listen statements in the config). Maybe we can narrow it down > > to something in the nginx config... > > Hm. Yeah, direct access works well. > Uh, what can be wrong with nginx config in this case? :( I would trim your config down the the bare minimum and use the defaults wherever you can. Then start adding things back in and see what broke it. -- Eric Wong From normalperson at yhbt.net Tue Jan 25 16:33:27 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 25 Jan 2011 13:33:27 -0800 Subject: [PATCH] KNOWN_ISSUES: FreeBSD 8 and sendfile can be buggy In-Reply-To: <5F765353-9348-4264-84FE-B639FB198868@gmail.com> References: <20110125005152.GB1921@dcvr.yhbt.net> <20110125031757.GA6871@dcvr.yhbt.net> <25C1DE1D-59D2-4E80-92E6-FDC0897814BA@gmail.com> <20110125035334.GB8124@dcvr.yhbt.net> <41BA1BA1-3DB7-45C2-8503-17026080D04A@gmail.com> <20110125041044.GA10853@dcvr.yhbt.net> <20110125045451.GB10853@dcvr.yhbt.net> <5F765353-9348-4264-84FE-B639FB198868@gmail.com> Message-ID: <20110125213327.GA9272@dcvr.yhbt.net> Alexey Bondar wrote: > On Jan 25, 2011, at 07:54 , Eric Wong wrote: > > Alexey Bondar wrote: > >> Hm. Yeah, direct access works well. > >> Uh, what can be wrong with nginx config in this case? :( > > > > I would trim your config down the the bare minimum and use the defaults > > wherever you can. Then start adding things back in and see what broke > > it. > > Maxim Dounin from nginx mail list says that this issue caused by some issues with sendfile in > FreeBSD 8. > > After setting sendfile to off uploads works well. > > You can add this info to known 3rd party issues. :) Thanks Alexey! >From 8ac0ae45a04f5f121f323c182403ef6eb0d8aa18 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Tue, 25 Jan 2011 13:30:21 -0800 Subject: [PATCH] KNOWN_ISSUES: FreeBSD 8 and sendfile can be buggy Reported by Alexey Bondar. --- KNOWN_ISSUES | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES index bc3dac5..95f4e43 100644 --- a/KNOWN_ISSUES +++ b/KNOWN_ISSUES @@ -6,6 +6,10 @@ acceptable solution. Those issues are documented here. * For notes on sandboxing tools such as Bundler or Isolate, see the {Sandbox}[link:Sandbox.html] page. +* nginx with "sendfile on" under FreeBSD 8 is broken when + uploads are buffered to disk. Disabling sendfile is required to + work around this bug which should be fixed in newer versions of FreeBSD. + * Under Ruby 1.9.1, methods like Array#shuffle and Array#sample will segfault if called after forking. This is fixed in trunk (r26936) and should be backported to the next 1.9.1 stable release (after p378). -- Eric Wong From normalperson at yhbt.net Tue Jan 25 17:09:50 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 25 Jan 2011 14:09:50 -0800 Subject: [PATCH] examples/nginx.conf: use try_files directive Message-ID: <20110125220950.GA11505@dcvr.yhbt.net> I'm lazy but I finally updated from using nginx 0.6.x myself for http://unicorn.bogomips.org/ and http://bogomips.org/unicorn.git to 0.8.54... >From 09afcf2ce9fc89d77b6b282bbf00a78c73741a4b Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Tue, 25 Jan 2011 13:58:29 -0800 Subject: [PATCH] examples/nginx.conf: use try_files directive This feature is in nginx 0.7.x and 0.8.x and optimized better than the "if" directive in nginx.conf ref: http://wiki.nginx.org/Pitfalls ref: http://wiki.nginx.org/IfIsEvil --- examples/nginx.conf | 18 +++++++++++------- 1 files changed, 11 insertions(+), 7 deletions(-) diff --git a/examples/nginx.conf b/examples/nginx.conf index 70d1851..52ec245 100644 --- a/examples/nginx.conf +++ b/examples/nginx.conf @@ -98,7 +98,16 @@ http { # path for static files root /path/to/app/current/public; - location / { + # Prefer to serve static files directly from nginx to avoid unnecessary + # data copies from the application server. + # + # try_files directive appeared in in nginx 0.7.27 and has stabilized + # over time. Older versions of nginx (e.g. 0.6.x) requires + # "if (!-f $request_filename)" which was less efficient: + # http://bogomips.org/unicorn.git/tree/examples/nginx.conf?id=v3.3.1#n127 + try_files $uri/index.html $uri.html $uri @app; + + location @app { # an HTTP header important enough to have its own Wikipedia entry: # http://en.wikipedia.org/wiki/X-Forwarded-For proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; @@ -122,12 +131,7 @@ http { # clients, really. # proxy_buffering off; - # Try to serve static files from nginx, no point in making an - # *application* server like Unicorn/Rainbows! serve static files. - if (!-f $request_filename) { - proxy_pass http://app_server; - break; - } + proxy_pass http://app_server; } # Rails error pages -- Eric Wong From ghazel at gmail.com Fri Jan 28 04:57:33 2011 From: ghazel at gmail.com (ghazel at gmail.com) Date: Fri, 28 Jan 2011 01:57:33 -0800 Subject: Kernel.srand after_fork Message-ID: Hi, We recently noticed random numbers from the Unicorn workers were turning up lots of duplicates across requests. Is it possible that the random seed generated when the ruby interpreter starts is duplicated when the workers are forked, and not re-seeded after fork? If so, should I call Kernel.srand in after_fork, or is this something which should be added to Unicorn? -Greg From normalperson at yhbt.net Fri Jan 28 05:26:49 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 28 Jan 2011 10:26:49 +0000 Subject: Kernel.srand after_fork In-Reply-To: References: Message-ID: <20110128102649.GB25922@dcvr.yhbt.net> ghazel at gmail.com wrote: > Hi, > > We recently noticed random numbers from the Unicorn workers were > turning up lots of duplicates across requests. Is it possible that the > random seed generated when the ruby interpreter starts is duplicated > when the workers are forked, and not re-seeded after fork? > > If so, should I call Kernel.srand in after_fork, or is this something > which should be added to Unicorn? It looks like a 1.8 bug, which version of Ruby are you running? Ruby 1.9.1 actually had a bug where it was reseeded improperly and led to segfaults (see KNOWN_ISSUES). 1.9.2 is fine afaik. -- Eric Wong From ghazel at gmail.com Fri Jan 28 05:57:23 2011 From: ghazel at gmail.com (ghazel at gmail.com) Date: Fri, 28 Jan 2011 02:57:23 -0800 Subject: Kernel.srand after_fork In-Reply-To: <20110128102649.GB25922@dcvr.yhbt.net> References: <20110128102649.GB25922@dcvr.yhbt.net> Message-ID: On Fri, Jan 28, 2011 at 2:26 AM, Eric Wong wrote: > ghazel at gmail.com wrote: >> Hi, >> >> We recently noticed random numbers from the Unicorn workers were >> turning up lots of duplicates across requests. Is it possible that the >> random seed generated when the ruby interpreter starts is duplicated >> when the workers are forked, and not re-seeded after fork? >> >> If so, should I call Kernel.srand in after_fork, or is this something >> which should be added to Unicorn? > > It looks like a 1.8 bug, which version of Ruby are you running? ?Ruby > 1.9.1 actually had a bug where it was reseeded improperly and led to > segfaults (see KNOWN_ISSUES). ?1.9.2 is fine afaik. REE 1.8.7 So, calling Kernel.srand in after_fork will fix the 1.8 bug and the 1.9.1 bug and work fine on 1.9.2? -Greg From normalperson at yhbt.net Fri Jan 28 12:51:11 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 28 Jan 2011 17:51:11 +0000 Subject: Kernel.srand after_fork In-Reply-To: References: <20110128102649.GB25922@dcvr.yhbt.net> Message-ID: <20110128175111.GA18352@dcvr.yhbt.net> ghazel at gmail.com wrote: > So, calling Kernel.srand in after_fork will fix the 1.8 bug and the > 1.9.1 bug and work fine on 1.9.2? Yes. The 1.9.1 bug is fixed in p429 AFAIK, but I seem to recall p429 being completely broken with gems. Just use 1.9.2 if you're on 1.9 -- Eric Wong From normalperson at yhbt.net Fri Jan 28 13:14:20 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 28 Jan 2011 18:14:20 +0000 Subject: Kernel.srand after_fork In-Reply-To: <20110128175111.GA18352@dcvr.yhbt.net> References: <20110128102649.GB25922@dcvr.yhbt.net> <20110128175111.GA18352@dcvr.yhbt.net> Message-ID: <20110128181420.GA29648@dcvr.yhbt.net> Eric Wong wrote: > ghazel at gmail.com wrote: > > So, calling Kernel.srand in after_fork will fix the 1.8 bug and the > > 1.9.1 bug and work fine on 1.9.2? Just reported the bug to ruby-core and pushed this out: >From f4caf6b6bdea902abaadd3c04b2af94f056c4ff1 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Fri, 28 Jan 2011 18:11:26 +0000 Subject: [PATCH] KNOWN_ISSUES: document broken RNG+fork in newer Ruby 1.8 Reported by: ghazel at gmail.com ref: --- KNOWN_ISSUES | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES index f218b83..259681d 100644 --- a/KNOWN_ISSUES +++ b/KNOWN_ISSUES @@ -3,6 +3,11 @@ Occasionally odd {issues}[link:ISSUES.html] arise without a transparent or acceptable solution. Those issues are documented here. +* Under some versions of Ruby 1.8, it is necessary to call +srand+ in an + after_fork hook to get correct random number generation. + + See http://redmine.ruby-lang.org/issues/show/4338 + * For notes on sandboxing tools such as Bundler or Isolate, see the {Sandbox}[link:Sandbox.html] page. -- Eric Wong From tils at tils.net Mon Jan 31 04:47:02 2011 From: tils at tils.net (Tilmann Singer) Date: Mon, 31 Jan 2011 10:47:02 +0100 Subject: Multiple master processes Message-ID: <87ei7te76x.wl%tils@tils.net> Hi, I'm seeing the following output of ps after starting unicorn, which as I understand it means that there are multiple master processes running, each with its own pid. Is that normal behaviour or is there something wrong with my setup? $ ps xf|grep 4002 8022 ? S 0:00 unicorn master -c config/unicorn.rb -E production -p 4002 -D 8024 ? S 0:00 \_ unicorn master -c config/unicorn.rb -E production -p 4002 -D 26743 ? S 0:00 | \_ unicorn master -c config/unicorn.rb -E production -p 4002 -D 26734 ? S 0:22 \_ unicorn worker[0] -c config/unicorn.rb -E production -p 4002 -D 26736 ? S 0:00 | \_ unicorn worker[0] -c config/unicorn.rb -E production -p 4002 -D 26739 ? S 0:00 | \_ unicorn worker[0] -c config/unicorn.rb -E production -p 4002 -D 26737 ? S 0:22 \_ unicorn worker[2] -c config/unicorn.rb -E production -p 4002 -D 26740 ? S 0:00 | \_ unicorn worker[2] -c config/unicorn.rb -E production -p 4002 -D 26741 ? S 0:00 | \_ unicorn worker[2] -c config/unicorn.rb -E production -p 4002 -D 26742 ? S 0:21 \_ unicorn worker[1] -c config/unicorn.rb -E production -p 4002 -D 26744 ? S 0:00 \_ unicorn worker[1] -c config/unicorn.rb -E production -p 4002 -D 26745 ? S 0:00 \_ unicorn worker[1] -c config/unicorn.rb -E production -p 4002 -D $ cat tmp/pids/unicorn.pid 8022 This is on Linux 2.4.27, ruby 1.9.2, unicorn 3.3.1, rails 3.0.3 in an rvm gemset. # config/unicorn.rb worker_processes 3 pid "tmp/pids/unicorn.pid" stderr_path "log/unicorn.stderr.log" stdout_path "log/unicorn.stdout.log" I start unicorn like this: unicorn -c config/unicorn.rb -E production -p 4002 -D Using "bundle exec unicorn" or unicorn_rails instead doesn't seem to make a difference. Starting the same project on my desktop with I believe is mostly the same setup except that it is Linux 2.6.34, the process list looks like this, which looks normal to me: $ ps xf|grep 4002 2692 ? Sl 0:00 unicorn master -c config/unicorn.rb -E production -p 4002 -D 2695 ? Rl 0:05 \_ unicorn worker[0] -c config/unicorn.rb -E production -p 4002 -D 2698 ? Rl 0:05 \_ unicorn worker[1] -c config/unicorn.rb -E production -p 4002 -D 2700 ? Rl 0:05 \_ unicorn worker[2] -c config/unicorn.rb -E production -p 4002 -D Til From normalperson at yhbt.net Mon Jan 31 13:08:15 2011 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 31 Jan 2011 18:08:15 +0000 Subject: Multiple master processes In-Reply-To: <87ei7te76x.wl%tils@tils.net> References: <87ei7te76x.wl%tils@tils.net> Message-ID: <20110131180815.GA17643@dcvr.yhbt.net> Tilmann Singer wrote: > Hi, > > I'm seeing the following output of ps after starting unicorn, which as > I understand it means that there are multiple master processes > running, each with its own pid. Is that normal behaviour or is there > something wrong with my setup? It's normal behavior for Linux 2.4.x + Ruby 1.9, don't worry about it. > $ ps xf|grep 4002 > 8022 ? S 0:00 unicorn master -c config/unicorn.rb -E production -p 4002 -D > 8024 ? S 0:00 \_ unicorn master -c config/unicorn.rb -E production -p 4002 -D > 26743 ? S 0:00 | \_ unicorn master -c config/unicorn.rb -E production -p 4002 -D > 26734 ? S 0:22 \_ unicorn worker[0] -c config/unicorn.rb -E production -p 4002 -D > 26736 ? S 0:00 | \_ unicorn worker[0] -c config/unicorn.rb -E production -p 4002 -D > 26739 ? S 0:00 | \_ unicorn worker[0] -c config/unicorn.rb -E production -p 4002 -D > 26737 ? S 0:22 \_ unicorn worker[2] -c config/unicorn.rb -E production -p 4002 -D > 26740 ? S 0:00 | \_ unicorn worker[2] -c config/unicorn.rb -E production -p 4002 -D > 26741 ? S 0:00 | \_ unicorn worker[2] -c config/unicorn.rb -E production -p 4002 -D > 26742 ? S 0:21 \_ unicorn worker[1] -c config/unicorn.rb -E production -p 4002 -D > 26744 ? S 0:00 \_ unicorn worker[1] -c config/unicorn.rb -E production -p 4002 -D > 26745 ? S 0:00 \_ unicorn worker[1] -c config/unicorn.rb -E production -p 4002 -D > $ cat tmp/pids/unicorn.pid > 8022 > > This is on Linux 2.4.27, ruby 1.9.2 With LinuxThreads in 2.4, the native threads used by Ruby 1.9 each had a unique PID and /proc entry whereas with NPTL in 2.6 hides the top-level entries of tasks that weren't processes (still visible in /proc/$pid/task/*) "tasks" are just the in-kernel representation of either processes and native threads in all Linux versions. They're created via the same system call, clone(2), so pthread_create() and fork() are just wrappers around clone() with different arguments. It's just 2.4 had fewer options for clone() that made it weirder and less POSIX-conformant. The pthreads(7) manpage has more details on the differences between NPTL and LinuxThreads. Wow, I didn't realize people ran the latest Rubies on 2.4.x! -- Eric Wong From jamie at jamiedubs.com Mon Jan 31 14:24:43 2011 From: jamie at jamiedubs.com (Jamie Wilkinson) Date: Mon, 31 Jan 2011 11:24:43 -0800 Subject: Multiple master processes In-Reply-To: <87ei7te76x.wl%tils@tils.net> References: <87ei7te76x.wl%tils@tils.net> Message-ID: <16D30BB4-B596-4653-B213-329D527BA7AB@jamiedubs.com> On Jan 31, 2011, at 1:47 AM, Tilmann Singer wrote: > Hi, > > I'm seeing the following output of ps after starting unicorn, which as > I understand it means that there are multiple master processes > running, each with its own pid. Is that normal behaviour or is there > something wrong with my setup? > > # config/unicorn.rb > worker_processes 3 > pid "tmp/pids/unicorn.pid" > stderr_path "log/unicorn.stderr.log" > stdout_path "log/unicorn.stdout.log" Unicorn doesn't kill the old master on it's own, you need an after_fork() like so (from the default/suggested unicorn conf) http://unicorn.bogomips.org/examples/unicorn.conf.rb