From jano at dimaion.com Fri Dec 2 14:39:49 2011 From: jano at dimaion.com (Damian Janowski) Date: Fri, 2 Dec 2011 16:39:49 -0300 Subject: Weird random Zbatery exception Message-ID: Hello everyone, I'm giving Zbatery a try, currently testing with ThreadPool. By the way, this instance is sitting behind Varnish. I can't figure out precise steps to trigger this exception, but it happens after a few GET/POST requests: /usr/local/lib/ruby/gems/1.9.1/gems/unicorn-4.1.1/lib/unicorn/http_server.rb:361:in `awaken_master': undefined method `kgio_trywrite' for nil:NilClass (NoMethodError) from /usr/local/lib/ruby/gems/1.9.1/gems/unicorn-4.1.1/lib/unicorn/http_server.rb:130:in `block in start' from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:32:in `call' from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:32:in `join' from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:32:in `block in worker_loop' from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:30:in `each' from /usr/local/lib/ruby/gems/1.9.1/gems/rainbows-4.3.1/lib/rainbows/thread_pool.rb:30:in `worker_loop' from /usr/local/lib/ruby/gems/1.9.1/gems/zbatery-4.1.1/lib/zbatery.rb:100:in `join' from /usr/local/lib/ruby/gems/1.9.1/gems/zbatery-4.1.1/bin/zbatery:121:in `' from /usr/local/bin/zbatery:19:in `load' from /usr/local/bin/zbatery:19:in `
' (Needless to say, I googled already without much luck). Any ideas? D. From normalperson at yhbt.net Fri Dec 2 17:47:11 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 2 Dec 2011 14:47:11 -0800 Subject: Weird random Zbatery exception In-Reply-To: References: Message-ID: <20111202224711.GA23422@dcvr.yhbt.net> Damian Janowski wrote: > I'm giving Zbatery a try, currently testing with ThreadPool. Thanks for giving it a try! :) > I can't figure out precise steps to trigger this exception, but it > happens after a few GET/POST requests: > /usr/local/lib/ruby/gems/1.9.1/gems/unicorn-4.1.1/lib/unicorn/http_server.rb:361:in > `awaken_master': undefined method `kgio_trywrite' for nil:NilClass > (NoMethodError) Does your application fork/exec other processes? (`` or "system()", popen all included). > (Needless to say, I googled already without much luck). I think you found a new bug, does the following patch fix it? diff --git a/lib/zbatery.rb b/lib/zbatery.rb index 735b1e6..17d7c06 100644 --- a/lib/zbatery.rb +++ b/lib/zbatery.rb @@ -82,6 +82,7 @@ module Rainbows trap(:USR1) { Thread.new { reopen_logs } } trap(:USR2) { Thread.new { reexec } } trap(:HUP) { Thread.new { reexec; stop } } + trap(:CHLD, "IGNORE") # technically feasible in some cases, just not sanely supportable: %w(TTIN TTOU WINCH).each do |sig| From normalperson at yhbt.net Fri Dec 2 18:40:03 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 2 Dec 2011 15:40:03 -0800 Subject: Weird random Zbatery exception In-Reply-To: <20111202224711.GA23422@dcvr.yhbt.net> References: <20111202224711.GA23422@dcvr.yhbt.net> Message-ID: <20111202234003.GA27418@dcvr.yhbt.net> Eric Wong wrote: > diff --git a/lib/zbatery.rb b/lib/zbatery.rb > index 735b1e6..17d7c06 100644 > --- a/lib/zbatery.rb > +++ b/lib/zbatery.rb > @@ -82,6 +82,7 @@ module Rainbows > trap(:USR1) { Thread.new { reopen_logs } } > trap(:USR2) { Thread.new { reexec } } > trap(:HUP) { Thread.new { reexec; stop } } > + trap(:CHLD, "IGNORE") On second thought, "IGNORE" will probably break existing code. trap(:CHLD, "DEFAULT") should be better. Btw, this issue shouldn't affect Rainbows!, only Zbatery. Can you confirm? From normalperson at yhbt.net Fri Dec 2 20:46:32 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 2 Dec 2011 17:46:32 -0800 Subject: Weird random Zbatery exception In-Reply-To: <20111202234003.GA27418@dcvr.yhbt.net> References: <20111202224711.GA23422@dcvr.yhbt.net> <20111202234003.GA27418@dcvr.yhbt.net> Message-ID: <20111203014632.GA3639@dcvr.yhbt.net> Eric Wong wrote: > Eric Wong wrote: > > + trap(:CHLD, "IGNORE") > > On second thought, "IGNORE" will probably break existing code. > trap(:CHLD, "DEFAULT") should be better. I just pushed out Zbatery 4.1.1.2.g3693 (prerelease) to RubyGems.org with trap(:CHLD,"DEFAULT") + test cases. You can install it with: gem install --pre zbatery Let me know if it fixes your problem since I've been meaning to release for 90d1d769b9af83d6d1 anyways (unlink oldbin pid file during graceful shutdown). (I also found https://redmine.ruby-lang.org/issues/5700 while working on this :o) From jano at dimaion.com Mon Dec 5 13:02:10 2011 From: jano at dimaion.com (Damian Janowski) Date: Mon, 5 Dec 2011 15:02:10 -0300 Subject: Weird random Zbatery exception In-Reply-To: <20111202224711.GA23422@dcvr.yhbt.net> References: <20111202224711.GA23422@dcvr.yhbt.net> Message-ID: On Fri, Dec 2, 2011 at 7:47 PM, Eric Wong wrote: > Thanks for giving it a try! :) Thank you, Eric, for your prompt reply! > Does your application fork/exec other processes? ?(`` or "system()", > popen all included). Not in my application code, and I'm pretty sure not in any of the few libraries I'm using. From jano at dimaion.com Mon Dec 5 13:02:44 2011 From: jano at dimaion.com (Damian Janowski) Date: Mon, 5 Dec 2011 15:02:44 -0300 Subject: Weird random Zbatery exception In-Reply-To: <20111202234003.GA27418@dcvr.yhbt.net> References: <20111202224711.GA23422@dcvr.yhbt.net> <20111202234003.GA27418@dcvr.yhbt.net> Message-ID: On Fri, Dec 2, 2011 at 8:40 PM, Eric Wong wrote: > Btw, this issue shouldn't affect Rainbows!, only Zbatery. > Can you confirm? I can confirm Rainbows has been stable for a few days (whereas Zbatery crashed after a few requests). I'll test the pre release now and let you know, thank you again. From jano at dimaion.com Mon Dec 5 13:29:24 2011 From: jano at dimaion.com (Damian Janowski) Date: Mon, 5 Dec 2011 15:29:24 -0300 Subject: Weird random Zbatery exception In-Reply-To: <20111203014632.GA3639@dcvr.yhbt.net> References: <20111202224711.GA23422@dcvr.yhbt.net> <20111202234003.GA27418@dcvr.yhbt.net> <20111203014632.GA3639@dcvr.yhbt.net> Message-ID: On Fri, Dec 2, 2011 at 10:46 PM, Eric Wong wrote: > I just pushed out Zbatery 4.1.1.2.g3693 (prerelease) to RubyGems.org > with trap(:CHLD,"DEFAULT") + test cases. > > You can install it with: gem install --pre zbatery > > Let me know if it fixes your problem since I've been meaning to release > for 90d1d769b9af83d6d1 anyways (unlink oldbin pid file during graceful > shutdown). It does seem to fix the problem. Thank you very much! From normalperson at yhbt.net Mon Dec 5 13:51:30 2011 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 5 Dec 2011 18:51:30 +0000 Subject: [ANN] Zbatery 4.1.2 - we don't fork, but our apps may! Message-ID: <20111205185130.GA23161@dcvr.yhbt.net> * http://zbatery.bogomip.org/ * rainbows-talk at rubyforge.org * git://bogomips.org/zbatery.git * http://zbatery.bogomip.org/NEWS.atom.xml Changes: There are two bugfixes in this release. Rack applications that use fork() internally should now behave as-expected when receiving SIGCHLD. The pid file is also unlinked during a graceful shutdown. Thanks to Damian Janowski for the bug report and confirming the fix! From danny.tran at digisynd.com Mon Dec 5 18:42:57 2011 From: danny.tran at digisynd.com (Danny Tran) Date: Mon, 5 Dec 2011 17:42:57 -0600 Subject: Rainbows! + Nginx Message-ID: I'm trying to run a Rails app that utilizes HTTP streaming with Rainbows! behind Nginx. Here is my nginx.conf:https://gist.github.com/a749361acd09b6ae80e2 Despite setting proxy_buffering off; it appears that nginx is still buffering the output because I do not get a streaming response. ?When I curl the app directly (port 3000) I get a proper streaming response, but when I curl nginx (which proxies to the Rails app) I do not get a streaming response. Any thoughts? Thanks,Danny From normalperson at yhbt.net Mon Dec 5 19:51:04 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 6 Dec 2011 00:51:04 +0000 Subject: Rainbows! + Nginx In-Reply-To: References: Message-ID: <20111206005104.GA20671@dcvr.yhbt.net> Danny Tran wrote: > I'm trying to run a Rails app that utilizes HTTP streaming with > Rainbows! behind Nginx. > Here is my nginx.conf:https://gist.github.com/a749361acd09b6ae80e2 > Despite setting proxy_buffering off; it appears that nginx is still > buffering the output because I do not get a streaming response. ?When > I curl the app directly (port 3000) I get a proper streaming response, > but when I curl nginx (which proxies to the Rails app) I do not get a > streaming response. > Any thoughts? Does your Rack app actually set "X-Accel-Buffering: no" in a response header? Your config mentions it: # The Rack application may also set "X-Accel-Buffering (yes|no)" # in the response headers do disable/enable buffering on a # per-response basis. proxy_buffering off; proxy_pass_header X-Accel-Buffering; I've mostly forgotten about Rails 3.1 streaming myself :x I know current nginx versions only send HTTP/1.0 requests to the backends, which means you need to force the response to stream via "X-Accel-Buffering: no" since HTTP/1.0 doesn't have chunked responses, You should also try using "curl --http1.0" to better simulate nginx behavior with HTTP/1.0 for testing. From admail-3 at you.saikyou.biz Mon Dec 12 03:26:59 2011 From: admail-3 at you.saikyou.biz (=?ISO-2022-JP?B?GyRCNWE/TT5wSnMkTiFKM3QhSyVTJTglTSU5JUglaSU5JUglNyU5JUYlYCU6GyhC?=) Date: 12 Dec 2011 17:26:59 +0900 Subject: =?ISO-2022-JP?B?GyRCOl9CcCVvITwlKyE8NV5KZ0IuSnMbKEIhGyRCIVolRyE8JT9Gfk5PNkhMM0JnTkw8dUNtJE4kPyRhIVslYSVrJV4lLBsoQlNPSE8bJEJAODNoJUslZSE8JTkbKEI=?= Message-ID: <20111212082659.8018.qmail@www9.bizmail.jp> ??????SOHO??????14,494??2011?12?? ??????:87,799 ?????????????????????? ???????????????????????????????????????????? ???????? Please e-mail the delivery stop here.? kyo at biz-career.rulez.jp ?1? ????SOHO???????????????????????????????? ?PR???????????????????????????????? ?????????????????! SOHO???????????????! ????????????????????!!???????????????????! ????????????????????????????????!? ??? !! ?????????? ? http://www.soho-comu.rash.jp/ ?PR???????????????????????????????? ???????12??????????????? ?????????????????????????? ??SOHO??????????3??????????????? ?????????????????????????? ?1????????????????! ????????????! ????????????????????????????? ?2?????????????????????????! ??????! ?SOHO????????????????! ?3??????????????????????!?????! ????????????????????! ? ? ????????? ? ???????! ??????????!! ? ? ????? http://www.soho-comu.rash.jp/ ???????????????? ????????????????? ?????????????????????????? ????SOHO??????!! ?????????????? 1.???? ? 18,500 ? 5,550? 2.?????????????? ? 10,550 ? 45,900? 3.????????? ? 1,000 ? 31,531? 4.????????? 6 ? 42,000? 5.?????????? 4 ? 13,500? 6.???????????????? ? 90 ? 45,000? ????????183,481????? ???????????????????(1?2?4????)???????!! ? http://www.soho-comu.rash.jp/ ? ?(??????????) ????????????????????????? ???????????????! ????????????!! ??????????????????????????????????? ?????????????????????????SOHO?????????? ?????????????????????????????????????????? ??????????????????????????????????? ??????????????? ??????????????????????????????????????? ?????????????????????? ?????????????????????????????????? ??????????????????????????? ???????????????????????????????????????????? ???????? Please e-mail the delivery stop here.? kyo at biz-career.rulez.jp ??????????????????????????????????? ??????????????)??????????????Web?????????? ??????????????????????????????????? Copyright (c) 1999 ???????????????????????All rights reserved.??????? From godfat at godfat.org Thu Dec 29 19:28:25 2011 From: godfat at godfat.org (Lin Jen-Shin (godfat)) Date: Fri, 30 Dec 2011 08:28:25 +0800 Subject: Weird app error: uninitialized fiber (FiberError) Message-ID: Hello Rainbows, I was trying to `use :CoolioFiberSpawn` with `Rainbows.sleep` to simulate asynchrony HTTP requests. However, I keep seeing weird "app error: uninitialized fiber (FiberError)". I spent a few hours (ouch) tracking down to a simple case. Suppose I have a Rack app which has `Rainbows.sleep` inside a each block: https://gist.github.com/1536857 (I am running Ruby 1.9.3) class Weird def call env=nil [1].each{ # if you comment out the each block, then it works Rainbows.sleep 2 } [200, {}, ["ok\n"]] end end Then `run Weird.new` would always cause app error: uninitialized fiber (FiberError) But if I took out the each block inside Weird#call, such like: class Weird def call env=nil Rainbows.sleep 2 [200, {}, ["ok\n"]] end end Then it works fine. Moreover, if I run the app like this: run lambda{ |env| Weird.new.call } Then both with or without the each block would work. Could you please explain this for me? Or is it a bug somewhere? Thanks!! From normalperson at yhbt.net Thu Dec 29 19:44:13 2011 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 29 Dec 2011 16:44:13 -0800 Subject: Weird app error: uninitialized fiber (FiberError) In-Reply-To: References: Message-ID: <20111230004413.GA12870@dcvr.yhbt.net> "Lin Jen-Shin (godfat)" wrote: > I was trying to `use :CoolioFiberSpawn` with `Rainbows.sleep` > to simulate asynchrony HTTP requests. However, I keep seeing > weird "app error: uninitialized fiber (FiberError)". > Could you please explain this for me? Or is it a bug somewhere? This looks like a Ruby bug, possibly related to combining Fibers with fork()... Can you reproduce this with zbatery? (zbatery is Rainbows!, without fork()), just s/rainbows/zbatery/ in the command-line). This may be related to https://bugs.ruby-lang.org/issues/5700 From godfat at gmail.com Thu Dec 29 19:55:11 2011 From: godfat at gmail.com (Lin Jen-Shin (godfat)) Date: Fri, 30 Dec 2011 08:55:11 +0800 Subject: Weird app error: uninitialized fiber (FiberError) In-Reply-To: <20111230004413.GA12870@dcvr.yhbt.net> References: <20111230004413.GA12870@dcvr.yhbt.net> Message-ID: On Fri, Dec 30, 2011 at 8:44 AM, Eric Wong wrote: > This looks like a Ruby bug, possibly related to combining Fibers with > fork()... > > Can you reproduce this with zbatery? ?(zbatery is Rainbows!, without fork()), > just s/rainbows/zbatery/ in the command-line). > > This may be related to https://bugs.ruby-lang.org/issues/5700 Thanks for the quick response. I can reproduce this under zbatery as well, so I guess this is not related to fork()? My ruby and gem versions: ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin10.8.0] zbatery (4.1.2) rainbows (4.3.1) unicorn (4.1.1) cool.io (1.1.0) I am not sure if I should be happy if this is really a ruby's bug. I thought I just couldn't figure out how do fibers work, looking through a lot of sources and doing a lot of experiments. From normalperson at yhbt.net Thu Dec 29 20:35:47 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 30 Dec 2011 01:35:47 +0000 Subject: Weird app error: uninitialized fiber (FiberError) In-Reply-To: References: <20111230004413.GA12870@dcvr.yhbt.net> Message-ID: <20111230013547.GA30808@dcvr.yhbt.net> "Lin Jen-Shin (godfat)" wrote: > I can reproduce this under zbatery as well, so I guess this is > not related to fork()? My ruby and gem versions: > > ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin10.8.0] It may be an issue with LLVM/GCC on your platform, I've seen several issues on the ruby-core list related to that setup. Maybe recompiling Ruby (and all extensions) with a non-LLVM version of GCC will solve this. > I am not sure if I should be happy if this is really a ruby's bug. > I thought I just couldn't figure out how do fibers work, looking > through a lot of sources and doing a lot of experiments. Can you reproduce the issue on a GNU/Linux system? I cannot support non-Free platforms, anyways. From godfat at godfat.org Thu Dec 29 21:20:19 2011 From: godfat at godfat.org (Lin Jen-Shin (godfat)) Date: Fri, 30 Dec 2011 10:20:19 +0800 Subject: Weird app error: uninitialized fiber (FiberError) In-Reply-To: <20111230013547.GA30808@dcvr.yhbt.net> References: <20111230004413.GA12870@dcvr.yhbt.net> <20111230013547.GA30808@dcvr.yhbt.net> Message-ID: On Fri, Dec 30, 2011 at 9:35 AM, Eric Wong wrote: > Maybe recompiling Ruby (and all extensions) with a non-LLVM > version of GCC will solve this. I'll try to recompile Ruby (and all extensions) tomorrow. (I really need to sleep now) It seems now Apple does not provide a non-LLVM GCC now. My /usr/bin/gcc is a link to llvm-gcc-4.2. > Can you reproduce the issue on a GNU/Linux system? > I cannot support non-Free platforms, anyways. I should have thought this. Just tried on my Arch Linux: ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux] I can't reproduce that on the machine. That's a good news. We're deploying to Linux anyway. Thanks! (I am not sure how to narrow this down further to file a bug on Ruby's issue tracker. Maybe I'll just show them what I have now) From godfat at godfat.org Fri Dec 30 12:03:59 2011 From: godfat at godfat.org (Lin Jen-Shin (godfat)) Date: Sat, 31 Dec 2011 01:03:59 +0800 Subject: Weird app error: uninitialized fiber (FiberError) In-Reply-To: <20111230013547.GA30808@dcvr.yhbt.net> References: <20111230004413.GA12870@dcvr.yhbt.net> <20111230013547.GA30808@dcvr.yhbt.net> Message-ID: On Fri, Dec 30, 2011 at 9:35 AM, Eric Wong wrote: > It may be an issue with LLVM/GCC on your platform, I've seen > several issues on the ruby-core list related to that setup. > > Maybe recompiling Ruby (and all extensions) with a non-LLVM > version of GCC will solve this. Just want to let you know, after trying GCC 4.6 without LLVM involved and reinstalling all binary extensions, I can no longer reproduce this. What's amazing is, when I switch back to the old Ruby which is compiled with Apple's llvm-gcc, I can't reproduce that either! O_o Maybe it's restarting the computer would really solve everything, or I just did something wrong somewhere... Then I guess I shouldn't file a bug for this. Thanks for your help!