From augusto at jadedpixel.com  Wed Jun  2 14:01:59 2010
From: augusto at jadedpixel.com (Augusto Becciu)
Date: Wed, 2 Jun 2010 15:01:59 -0300
Subject: Read error: #<TypeError: can't modify frozen string> raised from 
	HttpParser
Message-ID: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com>

Hey guys,

Started running unicorn in a production server like two weeks ago.
It's been running smoothly, but looking at the logs found 44
exceptions like this:

E, [2010-06-02T16:17:15.117071 #22680] ERROR -- : Read error:
#<TypeError: can't modify frozen string>
E, [2010-06-02T16:17:15.117270 #22680] ERROR -- :
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn/http_request.rb:59:in
`headers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn/http_request.rb:59:in
`read'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:643:in
`process_client'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:716:in `worker_loop'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:714:in `each'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:714:in `worker_loop'
/u/apps/xxxx/releases/20100602004926/vendor/plugins/newrelic_rpm/lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb:7:in
`call'
/u/apps/xxxx/releases/20100602004926/vendor/plugins/newrelic_rpm/lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb:7:in
`worker_loop'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:605:in
`spawn_missing_workers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:602:in `fork'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:602:in
`spawn_missing_workers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:598:in `each'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:598:in
`spawn_missing_workers'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:612:in
`maintain_worker_count'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:276:in `start'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn.rb:28:in `run'
/usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/bin/unicorn_rails:203
/usr/bin/unicorn_rails:26:in `load'
/usr/bin/unicorn_rails:26

Ruby version: ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux],
MBARI 0x8770, Ruby Enterprise Edition 2010.01


Any ideas? Maybe a bug in the http parser?

Thanks,
Augusto

From normalperson at yhbt.net  Wed Jun  2 16:25:18 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 2 Jun 2010 20:25:18 +0000
Subject: Read error: #<TypeError: can't modify frozen string> raised
	from HttpParser
In-Reply-To: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com>
References: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com>
Message-ID: <20100602202518.GA8617@dcvr.yhbt.net>

Augusto Becciu <augusto at jadedpixel.com> wrote:
> Hey guys,
> 
> Started running unicorn in a production server like two weeks ago.
> It's been running smoothly, but looking at the logs found 44
> exceptions like this:
> 
> E, [2010-06-02T16:17:15.117071 #22680] ERROR -- : Read error:
> #<TypeError: can't modify frozen string>
> E, [2010-06-02T16:17:15.117270 #22680] ERROR -- :
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn/http_request.rb:59:in
> `headers'

<snip>

> Ruby version: ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux],
> MBARI 0x8770, Ruby Enterprise Edition 2010.01
> 
> Any ideas? Maybe a bug in the http parser?

Hi Augusto,

Somehow the reusable Unicorn::HttpRequest::BUF string constant is
getting frozen when it shouldn't be.   Do you have any code that could
be freezing that string?  That string object should never be returned to
the application via any code paths in Unicorn (env or tee_input).

-- 
Eric Wong

From augusto at jadedpixel.com  Wed Jun  2 16:47:44 2010
From: augusto at jadedpixel.com (Augusto Becciu)
Date: Wed, 2 Jun 2010 17:47:44 -0300
Subject: Read error: #<TypeError: can't modify frozen string> raised from 
	HttpParser
In-Reply-To: <20100602202518.GA8617@dcvr.yhbt.net>
References: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com> 
	<20100602202518.GA8617@dcvr.yhbt.net>
Message-ID: <AANLkTikCjTCVorHoIwsOtfSzt1Z-Mv4ZE8nf0Mqtgnqq@mail.gmail.com>

Hi Eric,

There's no way our application could be freezing that constant, at
least not intentionally.

We're using New Replic's RPM plugin, but checked it out and couldn't
find anything that could do that.

http://github.com/newrelic/rpm/blob/master/lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb

Let me know if I can help in any way.

Thanks,
Augusto

On Wed, Jun 2, 2010 at 5:25 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Augusto Becciu <augusto at jadedpixel.com> wrote:
>> Hey guys,
>>
>> Started running unicorn in a production server like two weeks ago.
>> It's been running smoothly, but looking at the logs found 44
>> exceptions like this:
>>
>> E, [2010-06-02T16:17:15.117071 #22680] ERROR -- : Read error:
>> #<TypeError: can't modify frozen string>
>> E, [2010-06-02T16:17:15.117270 #22680] ERROR -- :
>> /usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn/http_request.rb:59:in
>> `headers'
>
> <snip>
>
>> Ruby version: ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux],
>> MBARI 0x8770, Ruby Enterprise Edition 2010.01
>>
>> Any ideas? Maybe a bug in the http parser?
>
> Hi Augusto,
>
> Somehow the reusable Unicorn::HttpRequest::BUF string constant is
> getting frozen when it shouldn't be. ? Do you have any code that could
> be freezing that string? ?That string object should never be returned to
> the application via any code paths in Unicorn (env or tee_input).
>
> --
> Eric Wong
>

From normalperson at yhbt.net  Wed Jun  2 17:38:15 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 2 Jun 2010 21:38:15 +0000
Subject: Read error: #<TypeError: can't modify frozen string> raised
	from HttpParser
In-Reply-To: <AANLkTikCjTCVorHoIwsOtfSzt1Z-Mv4ZE8nf0Mqtgnqq@mail.gmail.com>
References: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com>
	<20100602202518.GA8617@dcvr.yhbt.net>
	<AANLkTikCjTCVorHoIwsOtfSzt1Z-Mv4ZE8nf0Mqtgnqq@mail.gmail.com>
Message-ID: <20100602213815.GA23255@dcvr.yhbt.net>

Augusto Becciu <augusto at jadedpixel.com> wrote:
> On Wed, Jun 2, 2010 at 5:25 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > Augusto Becciu <augusto at jadedpixel.com> wrote:
> >> Hey guys,
> >>
> >> Started running unicorn in a production server like two weeks ago.
> >> It's been running smoothly, but looking at the logs found 44
> >> exceptions like this:
> >>
> >> E, [2010-06-02T16:17:15.117071 #22680] ERROR -- : Read error:
> >> #<TypeError: can't modify frozen string>
> >> E, [2010-06-02T16:17:15.117270 #22680] ERROR -- :
> >> /usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn/http_request.rb:59:in
> >> `headers'
> >
> > <snip>
> >
> >> Ruby version: ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux],
> >> MBARI 0x8770, Ruby Enterprise Edition 2010.01
> >>
> >> Any ideas? Maybe a bug in the http parser?
> >
> > Hi Augusto,
> >
> > Somehow the reusable Unicorn::HttpRequest::BUF string constant is
> > getting frozen when it shouldn't be. ? Do you have any code that could
> > be freezing that string? ?That string object should never be returned to
> > the application via any code paths in Unicorn (env or tee_input).

Please don't top post, thanks.

> Hi Eric,
> 
> There's no way our application could be freezing that constant, at
> least not intentionally.
> 
> We're using New Replic's RPM plugin, but checked it out and couldn't
> find anything that could do that.
> 
> http://github.com/newrelic/rpm/blob/master/lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb
> 
> Let me know if I can help in any way.

Are you able to reproduce the problem without the RPM plugin?  I've
never used RPM myself, but we've heard of (and proposed some fixes)
with it over the recent months.

Some of those problems could be segfaults (on x86_64), but memory
corruption could also cause an unintentional freeze, as well...

In particular, could you try disabling compression when sending things
upstream?

(totally untested, I don't even have a New Relic account[1]).

diff --git a/lib/new_relic/agent/agent.rb b/lib/new_relic/agent/agent.rb
index 928c6d7..5e60520 100644
--- a/lib/new_relic/agent/agent.rb
+++ b/lib/new_relic/agent/agent.rb
@@ -554,7 +554,7 @@ module NewRelic
       dump_size = dump.size
       
       # small payloads don't need compression      
-      return [dump, 'identity'] if dump_size < 2000
+      return [dump, 'identity']
       
       # medium payloads get fast compression, to save CPU
       # big payloads get all the compression possible, to stay under
---

There's also 1787b22eb2d8ab8b4046ae14be349aa487abc7b5 in the
v2.12.2_beta tag of git://github.com/newrelic/rpm which raises
the compression threshold, too...

-- 
Eric Wong

[1] - yes I'm allergic to signing up for commercial things

From normalperson at yhbt.net  Thu Jun  3 13:37:49 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 3 Jun 2010 17:37:49 +0000
Subject: Fwd: Support for Soft Timeout in Unicorn
Message-ID: <20100603173749.GA19649@dcvr.yhbt.net>

Hi,

HTML attachments are wasteful and thus rejected from the mailing list.
On the other hand, it actually helps to include the patch itself
(inline) so it's readable without a (human) context switch :)

----- Forwarded message from Pierre Baillet <oct at fotonauts.com> -----

Subject: Fwd: Support for Soft Timeout in Unicorn
From: Pierre Baillet <oct at fotonauts.com>
To: unicorn at bogomips.org

Hi,

Just tried to send that through the ml, but it seems something went wrong...

Cheers,
-- 
Pierre.

---------- Forwarded message ----------
From: <mongrel-unicorn-owner at rubyforge.org>
Date: Thu, Jun 3, 2010 at 2:40 PM
Subject: Support for Soft Timeout in Unicorn
To: oct at fotonauts.com


The message's content type was not explicitly allowed



---------- Forwarded message ----------
From: Pierre Baillet <oct at fotonauts.com>
To: mongrel-unicorn at rubyforge.org
Date: Thu, 3 Jun 2010 14:16:01 +0200
Subject: Support for Soft Timeout in Unicorn
Hi,

We use Unicorn at fotopedia since yesterday in production. We switched from
Passenger due to an issue in the way Passenger was handling some error in
our main application. Things run very well on Unicorn.

We have also modified Unicorn to handle a soft timeout for its workers. The
Unicorn timeout was killing the workers without any chance for us to catch
the Rails stack trace and identify the issue. I've forked and branched
Unicorn from github and added a soft_timeout configuration value that is
used for long running workers.

The workers now handle SIGABRT and will raise an exception. This will crash
the Rails application if it can be crashed and force the framework to dump
the stack trace in the logs. Let me know if this might be useful for other
people and, why not, integrate that in the main Unicorn code !

http://github.com/octplane/unicorn/tree/SOFT_TIMEOUT_SUPPORT

Cheers,
-- 
Pierre Baillet <oct at fotonauts.com>
http://www.fotopedia.com/




-- 
Pierre Baillet <oct at fotonauts.com>
http://www.fotopedia.com/

----- End forwarded message -----

From oct at fotonauts.com  Thu Jun  3 14:06:02 2010
From: oct at fotonauts.com (Pierre Baillet)
Date: Thu, 3 Jun 2010 20:06:02 +0200
Subject: Support for Soft Timeout in Unicorn
In-Reply-To: <20100603173749.GA19649@dcvr.yhbt.net>
References: <20100603173749.GA19649@dcvr.yhbt.net>
Message-ID: <AANLkTik1EWy4op-7CTasMODjRnZAjUBpUgn-bPAPOKlx@mail.gmail.com>

Hello,

On Thu, Jun 3, 2010 at 7:37 PM, Eric Wong?<normalperson at yhbt.net>?wrote:
>
> Hi,
>
> HTML attachments are wasteful and thus rejected from the mailing list.
> On the other hand, it actually helps to include the patch itself
> (inline) so it's readable without a (human) context switch :)

Indeed,
sorry for the HTML attachment, I have no idea where it comes from. As
for the patch, here you are. This is really just a way to handle
SIGABRT in a specific way in the worker and allow proper termination
of the application. Note the FIXME comment I added in the
murder_lazy_workers method. If any worker blocks while all the others
are idle for a _timeout_ period of time, they will all be killed
anyway. The consequence of that is that Unicorn will restart all its
workers if traffic is very low on the server.

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index a363014..855f26a 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -84,7 +84,7 @@ module Unicorn
   # Listener sockets are started in the master process and shared with
   # forked worker children.

-  class HttpServer < Struct.new(:app, :timeout, :worker_processes,
+  class HttpServer < Struct.new(:app, :soft_timeout, :timeout,
:worker_processes,
                                 :before_fork, :after_fork, :before_exec,
                                 :logger, :pid, :listener_opts, :preload_app,
                                 :reexec_pid, :orig_app, :init_listeners,
@@ -393,7 +393,7 @@ module Unicorn
           when nil
             # avoid murdering workers after our master process (or the
             # machine) comes out of suspend/hibernation
-            if (last_check + timeout) >= (last_check = Time.now)
+            if (last_check + soft_timeout) >= (last_check = Time.now)
               murder_lazy_workers
             else
               # wait for workers to wakeup on suspend
@@ -581,10 +581,20 @@ module Unicorn
         stat = worker.tmp.stat
         # skip workers that disable fchmod or have never fchmod-ed
         stat.mode == 0100600 and next
-        (diff = (Time.now - stat.ctime)) <= timeout and next
-        logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
+        # FIXME: if the worker has not been working for soft_timeout,
it will be
+        # killed even if it is not blocking
+        (diff = (Time.now - stat.ctime)) <= soft_timeout and
+          diff <= timeout and next
+        # lazy since less than timeout, attempt soft kill
+        if diff < timeout
+          logger.error "worker=#{worker.nr} PID:#{wpid} soft timeout " \
+                     "(#{diff}s > #{soft_timeout}s), killing softly"
+          kill_worker(:ABRT, wpid)
+        else
+         logger.error "worker=#{worker.nr} PID:#{wpid} hard timeout " \
:
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index a363014..855f26a 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -84,7 +84,7 @@ module Unicorn
   # Listener sockets are started in the master process and shared with
   # forked worker children.

-  class HttpServer < Struct.new(:app, :timeout, :worker_processes,
+  class HttpServer < Struct.new(:app, :soft_timeout, :timeout,
:worker_processes,
                                 :before_fork, :after_fork, :before_exec,
                                 :logger, :pid, :listener_opts, :preload_app,
                                 :reexec_pid, :orig_app, :init_listeners,
@@ -393,7 +393,7 @@ module Unicorn
           when nil
             # avoid murdering workers after our master process (or the
             # machine) comes out of suspend/hibernation
-            if (last_check + timeout) >= (last_check = Time.now)
+            if (last_check + soft_timeout) >= (last_check = Time.now)
               murder_lazy_workers
             else
               # wait for workers to wakeup on suspend
@@ -581,10 +581,20 @@ module Unicorn
         stat = worker.tmp.stat
         # skip workers that disable fchmod or have never fchmod-ed
         stat.mode == 0100600 and next
-        (diff = (Time.now - stat.ctime)) <= timeout and next
-        logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
+        # FIXME: if the worker has not been working for soft_timeout,
it will be
+        # killed even if it is not blocking
+        (diff = (Time.now - stat.ctime)) <= soft_timeout and
+          diff <= timeout and next
+        # lazy since less than timeout, attempt soft kill
+        if diff < timeout
+          logger.error "worker=#{worker.nr} PID:#{wpid} soft timeout " \
+                     "(#{diff}s > #{soft_timeout}s), killing softly"
+          kill_worker(:ABRT, wpid)
+        else
+         logger.error "worker=#{worker.nr} PID:#{wpid} hard timeout " \
                      "(#{diff}s > #{timeout}s), killing"
-        kill_worker(:KILL, wpid) # take no prisoners for timeout violations
+          kill_worker(:KILL, wpid) # take no prisoners for timeout violations
+        end
       end
     end

@@ -657,6 +667,12 @@ module Unicorn
       proc_name "worker[#{worker.nr}]"
       START_CTX.clear
       init_self_pipe!
+
+      # try to handle SIGABRT correctly
+      trap('ABRT') do
+        raise SignalException, "SIGABRT"
+      end
+
       WORKERS.values.each { |other| other.tmp.close rescue nil }
       WORKERS.clear
       LISTENERS.each { |sock| sock.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 64a25e3..6efb0c5 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -14,6 +14,8 @@ module Unicorn

     # Default settings for Unicorn
     DEFAULTS = {
+      # Backward compatibility soft timeout (disabled in default configuration)
+      :soft_timeout => 60,
       :timeout => 60,
       :logger => Logger.new($stderr),
       :worker_processes => 1,
@@ -129,6 +131,23 @@ module Unicorn

     # sets the timeout of worker processes to +seconds+.  Workers
     # handling the request/app.call/response cycle taking longer than
+    # this time period will be softly killed (via SIGABRT).  This
+    # timeout is enforced by the master process itself and not subject
+    # to the scheduling limitations by the worker process.  Due the
+    # low-complexity, low-overhead implementation, timeouts of less
+    # than 3.0 seconds can be considered inaccurate and unsafe.
+    # ABORT is handled by the worker and raise an exception, offering a
+    # way to log the stack trace in your rails application.
+
+    def soft_timeout(seconds)
+      Numeric === seconds or raise ArgumentError,
+                                  "not numeric: timeout=#{seconds.inspect}"
+      seconds >= 3 or raise ArgumentError,
+                                  "too low: timeout=#{seconds.inspect}"
+      set[:soft_timeout] = seconds
+    end
+    # sets the timeout of worker processes to +seconds+.  Workers
+    # handling the request/app.call/response cycle taking longer than
     # this time period will be forcibly killed (via SIGKILL).  This
     # timeout is enforced by the master process itself and not subject
     # to the scheduling limitations by the worker process.  Due the
@@ -159,6 +178,7 @@ module Unicorn
       set[:timeout] = seconds
     end

+
     # sets the current number of worker_processes to +nr+.  Each worker
     # process will serve exactly one client at a time.  You can
     # increment or decrement this value at runtime by sending SIGTTIN


Cheers,
--
Pierre Baillet <oct at fotonauts.com>
http://www.fotopedia.com/

From normalperson at yhbt.net  Thu Jun  3 14:22:46 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 3 Jun 2010 18:22:46 +0000
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <20100603173749.GA19649@dcvr.yhbt.net>
References: <20100603173749.GA19649@dcvr.yhbt.net>
Message-ID: <20100603182246.GB19649@dcvr.yhbt.net>

Pierre Baillet wrote:
> We use Unicorn at fotopedia since yesterday in production. We switched from
> Passenger due to an issue in the way Passenger was handling some error in
> our main application. Things run very well on Unicorn.
> 
> We have also modified Unicorn to handle a soft timeout for its workers. The
> Unicorn timeout was killing the workers without any chance for us to catch
> the Rails stack trace and identify the issue. I've forked and branched
> Unicorn from github and added a soft_timeout configuration value that is
> used for long running workers.
> 
> The workers now handle SIGABRT and will raise an exception. This will crash
> the Rails application if it can be crashed and force the framework to dump
> the stack trace in the logs. Let me know if this might be useful for other
> people and, why not, integrate that in the main Unicorn code !

Hi Pierre,

I'm thinking there's a better way to do this without involving the
master process.

The current timeout implementation[1] is really the last resort,
point-of-no-return situations when the workers are completely
stuck/blocked and cannot respond to other signals[2].  If the worker can
respond to SIGABRT (especially going through the interpreter and raising
an exception), then that means it could technically respond to
Thread#raise, too...

Unfortunately, the current core Ruby timeout implementation is extremely
naive and inefficient.  It shouldn't be hard to write a better timeout
implementation that reuses a single timer Thread which can be rearmed
with every request.

This would be doable as middleware, too, and if done carefully, even
safely reusable in multi-threaded web servers.  This would be a good
addition to rack-contrib, even.  I might consider doing it myself if I
had time.

[1] - I'm not completely happy with "needing" the current timeout
      implementation in the first place.  I will at least redo it
      at some point (after Unicorn 1.x) to gain some
      scalability/performance (and perhaps lose some portability).

[2] - SIGKILL and SIGSTOP are special, userspace has no mechanism
      to block/catch/ignore those signals, so we rely on the master
      process to deliver them.

-- 
Eric Wong

From oct at fotonauts.com  Thu Jun  3 14:32:27 2010
From: oct at fotonauts.com (Pierre Baillet)
Date: Thu, 3 Jun 2010 20:32:27 +0200
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <20100603182246.GB19649@dcvr.yhbt.net>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net>
Message-ID: <AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>

Eric,

On Thu, Jun 3, 2010 at 8:22 PM, Eric Wong <normalperson at yhbt.net> wrote:
> This would be doable as middleware, too, and if done carefully, even
> safely reusable in multi-threaded web servers. ?This would be a good
> addition to rack-contrib, even. ?I might consider doing it myself if I
> had time.

I also think that doing this as a Rack middleware is probably the
right way to do that kind of worker management. However the
application we use at fotopedia is still Rails 2.1 based (hence not
Rack compatible AFAIK). This probably means we'll have keep on
maintaining this slightly hacked version on our own then.

Thank you for Unicorn ! :)
-- 
Pierre Baillet <oct at fotonauts.com>
http://www.fotopedia.com/

From normalperson at yhbt.net  Thu Jun  3 14:47:30 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 3 Jun 2010 11:47:30 -0700
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net>
	<AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>
Message-ID: <20100603184730.GA2421@dcvr.yhbt.net>

Pierre Baillet <oct at fotonauts.com> wrote:
> Eric,
> 
> On Thu, Jun 3, 2010 at 8:22 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > This would be doable as middleware, too, and if done carefully, even
> > safely reusable in multi-threaded web servers. ?This would be a good
> > addition to rack-contrib, even. ?I might consider doing it myself if I
> > had time.
> 
> I also think that doing this as a Rack middleware is probably the
> right way to do that kind of worker management. However the
> application we use at fotopedia is still Rails 2.1 based (hence not
> Rack compatible AFAIK). This probably means we'll have keep on
> maintaining this slightly hacked version on our own then.

Actually, internally, Unicorn only knows about Rack and wraps older
CGI-based Rails using the Unicorn::App::OldRails application (via
Unicorn::CGIWrapper).

"unicorn_rails" basically wraps up the following config for you,
but you could achieve the same effect using "unicorn" and an
explicitly written config.ru:

------------ config.ru -------------
ENV["RAILS_ENV"] ||= ENV["RACK_ENV"]
require 'config/boot'
require 'config/environment'
require 'unicorn/app/old_rails'
require 'unwritten_timeout_middleware'
use UnwrittenTimeoutMiddleware # :)
use Unicorn::App::OldRails::Static # optional
run Unicorn::App::OldRails.new
------------------------------------

> Thank you for Unicorn ! :)

No problem :)

-- 
Eric Wong

From chris at ozmm.org  Thu Jun  3 15:38:19 2010
From: chris at ozmm.org (Chris Wanstrath)
Date: Thu, 3 Jun 2010 12:38:19 -0700
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <20100603184730.GA2421@dcvr.yhbt.net>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net> 
	<AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com> 
	<20100603184730.GA2421@dcvr.yhbt.net>
Message-ID: <AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>

On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong <normalperson at yhbt.net> wrote:

> Actually, internally, Unicorn only knows about Rack and wraps older
> CGI-based Rails using the Unicorn::App::OldRails application (via
> Unicorn::CGIWrapper).
>
> "unicorn_rails" basically wraps up the following config for you,
> but you could achieve the same effect using "unicorn" and an
> explicitly written config.ru:

That's what we do at GitHub. We're running Rails 2.2.2 and have a
custom config.ru, thanks to Unicorn:

http://gist.github.com/424352

From oct at fotonauts.com  Thu Jun  3 15:40:19 2010
From: oct at fotonauts.com (Pierre Baillet)
Date: Thu, 3 Jun 2010 21:40:19 +0200
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net>
	<AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>
	<20100603184730.GA2421@dcvr.yhbt.net>
	<AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>
Message-ID: <AANLkTimV05UjKFPJW7xFW7OTALIrQH9Qr9I2u4LBYTwe@mail.gmail.com>

Ohai,

On Thu, Jun 3, 2010 at 9:38 PM, Chris Wanstrath <chris at ozmm.org> wrote:
> On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong <normalperson at yhbt.net> wrote:
>
>> Actually, internally, Unicorn only knows about Rack and wraps older
>> CGI-based Rails using the Unicorn::App::OldRails application (via
>> Unicorn::CGIWrapper).
>>
>> "unicorn_rails" basically wraps up the following config for you,
>> but you could achieve the same effect using "unicorn" and an
>> explicitly written config.ru:
>
> That's what we do at GitHub. We're running Rails 2.2.2 and have a
> custom config.ru, thanks to Unicorn:
>

That's great, thanks for the Idea, I'll look into rack middleware
then. I'll let you know if I manage to have something clean enough.

Cheers,
-- 
Pierre Baillet <oct at fotonauts.com>
http://www.fotopedia.com/

From augusto at jadedpixel.com  Thu Jun  3 19:05:21 2010
From: augusto at jadedpixel.com (Augusto Becciu)
Date: Thu, 3 Jun 2010 20:05:21 -0300
Subject: Read error: #<TypeError: can't modify frozen string> raised from 
	HttpParser
In-Reply-To: <20100602213815.GA23255@dcvr.yhbt.net>
References: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com> 
	<20100602202518.GA8617@dcvr.yhbt.net>
	<AANLkTikCjTCVorHoIwsOtfSzt1Z-Mv4ZE8nf0Mqtgnqq@mail.gmail.com> 
	<20100602213815.GA23255@dcvr.yhbt.net>
Message-ID: <AANLkTimy0cLsh36A4pBoYMzZJY5wBKrxCp7rgCxyiZ53@mail.gmail.com>

On Wed, Jun 2, 2010 at 6:38 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Augusto Becciu <augusto at jadedpixel.com> wrote:
>> On Wed, Jun 2, 2010 at 5:25 PM, Eric Wong <normalperson at yhbt.net> wrote:
>> > Augusto Becciu <augusto at jadedpixel.com> wrote:
>> >> Hey guys,
>> >>
>> >> Started running unicorn in a production server like two weeks ago.
>> >> It's been running smoothly, but looking at the logs found 44
>> >> exceptions like this:
>> >>
>> >> E, [2010-06-02T16:17:15.117071 #22680] ERROR -- : Read error:
>> >> #<TypeError: can't modify frozen string>
>> >> E, [2010-06-02T16:17:15.117270 #22680] ERROR -- :
>> >> /usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn/http_request.rb:59:in
>> >> `headers'
>> >
>> > <snip>
>> >
>> >> Ruby version: ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux],
>> >> MBARI 0x8770, Ruby Enterprise Edition 2010.01
>> >>
>> >> Any ideas? Maybe a bug in the http parser?
>> >
>> > Hi Augusto,
>> >
>> > Somehow the reusable Unicorn::HttpRequest::BUF string constant is
>> > getting frozen when it shouldn't be. ? Do you have any code that could
>> > be freezing that string? ?That string object should never be returned to
>> > the application via any code paths in Unicorn (env or tee_input).
>
> Please don't top post, thanks.
>
>> Hi Eric,
>>
>> There's no way our application could be freezing that constant, at
>> least not intentionally.
>>
>> We're using New Replic's RPM plugin, but checked it out and couldn't
>> find anything that could do that.
>>
>> http://github.com/newrelic/rpm/blob/master/lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb
>>
>> Let me know if I can help in any way.
>
> Are you able to reproduce the problem without the RPM plugin? ?I've
> never used RPM myself, but we've heard of (and proposed some fixes)
> with it over the recent months.
>
> Some of those problems could be segfaults (on x86_64), but memory
> corruption could also cause an unintentional freeze, as well...
>
> In particular, could you try disabling compression when sending things
> upstream?
>
> (totally untested, I don't even have a New Relic account[1]).
>
> diff --git a/lib/new_relic/agent/agent.rb b/lib/new_relic/agent/agent.rb
> index 928c6d7..5e60520 100644
> --- a/lib/new_relic/agent/agent.rb
> +++ b/lib/new_relic/agent/agent.rb
> @@ -554,7 +554,7 @@ module NewRelic
> ? ? ? dump_size = dump.size
>
> ? ? ? # small payloads don't need compression
> - ? ? ?return [dump, 'identity'] if dump_size < 2000
> + ? ? ?return [dump, 'identity']
>
> ? ? ? # medium payloads get fast compression, to save CPU
> ? ? ? # big payloads get all the compression possible, to stay under
> ---
>
> There's also 1787b22eb2d8ab8b4046ae14be349aa487abc7b5 in the
> v2.12.2_beta tag of git://github.com/newrelic/rpm which raises
> the compression threshold, too...
>
> --
> Eric Wong
>
> [1] - yes I'm allergic to signing up for commercial things
>

Thanks Eric! Unfortunately after completely disabling RPM, we keep
getting that error. :(

What else could it be?

From normalperson at yhbt.net  Thu Jun  3 19:40:30 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 3 Jun 2010 16:40:30 -0700
Subject: Read error: #<TypeError: can't modify frozen string> raised
	from HttpParser
In-Reply-To: <AANLkTimy0cLsh36A4pBoYMzZJY5wBKrxCp7rgCxyiZ53@mail.gmail.com>
References: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com>
	<20100602202518.GA8617@dcvr.yhbt.net>
	<AANLkTikCjTCVorHoIwsOtfSzt1Z-Mv4ZE8nf0Mqtgnqq@mail.gmail.com>
	<20100602213815.GA23255@dcvr.yhbt.net>
	<AANLkTimy0cLsh36A4pBoYMzZJY5wBKrxCp7rgCxyiZ53@mail.gmail.com>
Message-ID: <20100603234030.GA18377@dcvr.yhbt.net>

Augusto Becciu <augusto at jadedpixel.com> wrote:
> On Wed, Jun 2, 2010 at 6:38 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > Augusto Becciu <augusto at jadedpixel.com> wrote:
> >> On Wed, Jun 2, 2010 at 5:25 PM, Eric Wong <normalperson at yhbt.net> wrote:
> >> > Augusto Becciu <augusto at jadedpixel.com> wrote:
> >> >> Hey guys,
> >> >>
> >> >> Started running unicorn in a production server like two weeks ago.
> >> >> It's been running smoothly, but looking at the logs found 44
> >> >> exceptions like this:
> >> >>
> >> >> E, [2010-06-02T16:17:15.117071 #22680] ERROR -- : Read error:
> >> >> #<TypeError: can't modify frozen string>
> >> >> E, [2010-06-02T16:17:15.117270 #22680] ERROR -- :
> >> >> /usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn/http_request.rb:59:in
> >> >> `headers'
> >> >
> >> > <snip>
> >> >
> >> >> Ruby version: ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux],
> >> >> MBARI 0x8770, Ruby Enterprise Edition 2010.01

<snip>

> >> There's no way our application could be freezing that constant, at
> >> least not intentionally.

<snip>

> Thanks Eric! Unfortunately after completely disabling RPM, we keep
> getting that error. :(
> 
> What else could it be?

Any details about your application you're allowed to share can help us,
especially a list of library/gem dependencies and versions.

I would grep through your gems and vendor directories to ensure there's
nothing freezing objects it shouldn't freeze.  It could also be a buggy
C extension corrupting memory somewhere, too...

This isn't happening for all your worker processes, is it?

If your application logs show which PID handled each request, you can
join the PIDs from the application logs with PIDs in the error logs
generated by Unicorn.  That lets you narrow down the possible requests
that could trigger the errors and help you reproduce the problem sooner.

This is one of my favorite features of one-request-per-process model
is that you can easily narrow down which request is triggering a
particular bug without worrying about race conditions from other
requests.

This is certainly the first time I've heard of this problem, so I think
it's specific to something in your application/environment.

-- 
Eric Wong

From normalperson at yhbt.net  Thu Jun  3 21:58:38 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 4 Jun 2010 01:58:38 +0000
Subject: unicorn_rails cleanup (possible fix for Rails3) pushed
Message-ID: <20100604015838.GA21464@dcvr.yhbt.net>

Hi all,

I've pushed the following patch out go git://git.bogomips.org/unicorn
along with a few other Rails-related test updates.  This is more of a
shotgun fix (but less code is better :) since I haven't been able to
reproduce the brokeness people have been seeing with "unicorn_rails"
and Rails 3 betas.

Even though "unicorn" works perfectly well for Rails3, some folks will
inevitably run "unicorn_rails" because of the "rails" in its name.

If I were to do it all over again, I would've just made everybody write
config.ru files for Rails.  But yes, programming is like sex, make one
mistake and support it for life :)

You can grab a git gem here, too:

  http://unicorn.bogomips.org/files/unicorn-0.99.0.16.g59a625.gem

>From 4b44e21957e4cb8ec6ace5604fbe096dfd8959d2 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Thu, 3 Jun 2010 22:52:11 +0000
Subject: [PATCH] unicorn_rails: avoid duplicating config.ru logic

This should allow "unicorn_rails" to be used seamlessly
with Rails 3 projects which package config.ru for you.
---
 bin/unicorn_rails |   50 ++++++++++++++------------------------------------
 1 files changed, 14 insertions(+), 36 deletions(-)

diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 72ab288..45a9b11 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -109,53 +109,30 @@ end
 
 ru = ARGV[0] || (File.exist?('config.ru') ? 'config.ru' : nil)
 
-if ru && ru =~ /\.ru\z/
-  # parse embedded command-line options in config.ru comments
-  /^#\\(.*)/ =~ File.read(ru) and opts.parse!($1.split(/\s+/))
-end
-
-def rails_builder(ru, daemonize)
+def rails_builder(daemonize)
   # this lambda won't run until after forking if preload_app is false
   lambda do ||
     # Load Rails and (possibly) the private version of Rack it bundles.
     begin
       require 'config/boot'
+      require 'config/environment'
     rescue LoadError => err
       abort "#$0 must be run inside RAILS_ROOT: #{err.inspect}"
     end
 
-    inner_app = case ru
-    when nil
-      require 'config/environment'
-
-      defined?(::Rails::VERSION::STRING) or
-        abort "Rails::VERSION::STRING not defined by config/{boot,environment}"
-      # it seems Rails >=2.2 support Rack, but only >=2.3 requires it
-      old_rails = case ::Rails::VERSION::MAJOR
-      when 0, 1 then true
-      when 2 then Rails::VERSION::MINOR < 3 ? true : false
-      else
-        false
-      end
-
-      if old_rails
-        require 'unicorn/app/old_rails'
-        Unicorn::App::OldRails.new
-      else
-        ActionController::Dispatcher.new
-      end
-    when /\.ru$/
-      raw = File.read(ru)
-      raw.sub!(/^__END__\n.*/, '')
-      eval("Rack::Builder.new {(#{raw}\n)}.to_app", TOPLEVEL_BINDING, ru)
+    defined?(::Rails::VERSION::STRING) or
+      abort "Rails::VERSION::STRING not defined by config/{boot,environment}"
+    # it seems Rails >=2.2 support Rack, but only >=2.3 requires it
+    old_rails = case ::Rails::VERSION::MAJOR
+    when 0, 1 then true
+    when 2 then Rails::VERSION::MINOR < 3 ? true : false
     else
-      require ru
-      Object.const_get(File.basename(ru, '.rb').capitalize)
+      false
     end
 
     Rack::Builder.new do
       map_path = ENV['RAILS_RELATIVE_URL_ROOT'] || '/'
-      if inner_app.class.to_s == "Unicorn::App::OldRails"
+      if old_rails
         if map_path != '/'
           # patches + tests welcome, but I really cbf to deal with this
           # since all apps I've ever dealt with just use "/" ...
@@ -163,23 +140,24 @@ def rails_builder(ru, daemonize)
         end
         $stderr.puts "LogTailer not available for Rails < 2.3" unless daemonize
         $stderr.puts "Debugger not available" if $DEBUG
+        require 'unicorn/app/old_rails'
         map(map_path) do
           use Unicorn::App::OldRails::Static
-          run inner_app
+          run Unicorn::App::OldRails.new
         end
       else
         use Rails::Rack::LogTailer unless daemonize
         use Rails::Rack::Debugger if $DEBUG
         map(map_path) do
           use Rails::Rack::Static
-          run inner_app
+          run ActionController::Dispatcher.new
         end
       end
     end.to_app
   end
 end
 
-app = rails_builder(ru, daemonize)
+app = ru ? Unicorn.builder(ru, opts) : rails_builder(daemonize)
 options[:listeners] << "#{host}:#{port}" if set_listener
 
 if $DEBUG
-- 
Eric Wong

From mguterl at gmail.com  Thu Jun  3 22:13:24 2010
From: mguterl at gmail.com (Michael Guterl)
Date: Thu, 3 Jun 2010 22:13:24 -0400
Subject: unicorn_rails cleanup (possible fix for Rails3) pushed
In-Reply-To: <20100604015838.GA21464@dcvr.yhbt.net>
References: <20100604015838.GA21464@dcvr.yhbt.net>
Message-ID: <AANLkTimsm6LphrYezwvCkiFEVc041ka0wZ9rhJwJDk48@mail.gmail.com>

On Thu, Jun 3, 2010 at 9:58 PM, Eric Wong <normalperson at yhbt.net> wrote:
<snip>
> If I were to do it all over again, I would've just made everybody write
> config.ru files for Rails. ?But yes, programming is like sex, make one
> mistake and support it for life :)

That is probably one of the funniest things I've read in awhile.
Thanks for the unexpected source of humor! :)

Best,
Michael Guterl

From normalperson at yhbt.net  Thu Jun  3 22:48:00 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 3 Jun 2010 19:48:00 -0700
Subject: unicorn_rails cleanup (possible fix for Rails3) pushed
In-Reply-To: <AANLkTimsm6LphrYezwvCkiFEVc041ka0wZ9rhJwJDk48@mail.gmail.com>
References: <20100604015838.GA21464@dcvr.yhbt.net>
	<AANLkTimsm6LphrYezwvCkiFEVc041ka0wZ9rhJwJDk48@mail.gmail.com>
Message-ID: <20100604024800.GA24464@dcvr.yhbt.net>

Michael Guterl <mguterl at gmail.com> wrote:
> On Thu, Jun 3, 2010 at 9:58 PM, Eric Wong <normalperson at yhbt.net> wrote:
> <snip>
> > If I were to do it all over again, I would've just made everybody write
> > config.ru files for Rails. ?But yes, programming is like sex, make one
> > mistake and support it for life :)
> 
> That is probably one of the funniest things I've read in awhile.
> Thanks for the unexpected source of humor! :)

You're welcome :>  It's not my line, but I heard it many years ago.

-- 
Eric Wong

From tom at mail.ipe.ac.th  Fri Jun  4 04:58:02 2010
From: tom at mail.ipe.ac.th (Heath Vinson)
Date: Fri,  4 Jun 2010 15:58:02 +0700 (ICT)
Subject: Manufacturers Database - 1,057,119 records with 476,509 emails
Message-ID: <20100604085802.7B9A0B5D36D@ipemail.ipe.ac.th>

Here's some of the healthcare lists we have:

Alternative Medicine - 1,141,602 total records with 36,320 emails and 38.935 fax numbers
Dentists - 164k records, 45k emails, 77k fax numbers
Psychologists - 272,188 records and 9,874 emails


Theres many more too, just send me an email here for additional info/samples: Burt.Gomez at listnirvana.net

  


Send email to disappear at listnirvana.net to ensure no further communication

From peter.kieltyka at nulayer.com  Fri Jun  4 15:06:17 2010
From: peter.kieltyka at nulayer.com (Peter Kieltyka)
Date: Fri, 4 Jun 2010 15:06:17 -0400
Subject: Logging errors from Unicorn
Message-ID: <977C003B-E2CA-48B9-A3E6-C126DB231EB5@nulayer.com>

Hey guys,

I'm running a Rails 3 beta3 application using Ruby 1.9.2-preview3 and it has been running fairly well, however every now and then (usually when under high load), I see error messages like:

2010/06/04 14:34:29 [error] 1884#0: *8844 upstream prematurely closed connection while reading response header from upstream, client: 119.235.237.93, server: portal.crowdreel.com, request: "GET /post/18Vmy HTTP/1.1", upstream: "http://unix:/home/app/crowdreel-portal/tmp/sockets/unicorn.sock:/post/18Vmy", host: "crowdreel.com"

... a lot of these messages.. this is a pretty heavy duty server with lots of ram.

I have a setup similar to what I read up on github's blog as in: nginx -> unicorns via a socket upstream.

My unicorn.rb file looks like:

--------

# 16 workers and 1 master
worker_processes 16

# Load the app into the master before forking workers for super-fast worker spawn times
preload_app false

# Restart any workers that haven't responded in 30 seconds
timeout 45

# Listen on a Unix data socket
listen File.expand_path(File.dirname(__FILE__)+'/../tmp/sockets/unicorn.sock'), :backlog => 2048

----------

I don't preload the app because I start running into annoying issues where my workers use the same db connection, so instead I just have them load their own instance, its very fast to start up anyways.

I've read everything I could on this issue and exhausted Google, which leaves me to think the only thing left is some incompatibilities with 1.9.2. So I was wondering, how can I get logs from unicorn's workers/master to know what is happening to these workers, perhaps something is segfaulting?

Thx!

Regards,

Peter

From normalperson at yhbt.net  Fri Jun  4 16:06:48 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 4 Jun 2010 20:06:48 +0000
Subject: Logging errors from Unicorn
In-Reply-To: <977C003B-E2CA-48B9-A3E6-C126DB231EB5@nulayer.com>
References: <977C003B-E2CA-48B9-A3E6-C126DB231EB5@nulayer.com>
Message-ID: <20100604200648.GA6767@dcvr.yhbt.net>

Peter Kieltyka <peter.kieltyka at nulayer.com> wrote:
> Hey guys,
> 
> I'm running a Rails 3 beta3 application using Ruby 1.9.2-preview3 and
> it has been running fairly well, however every now and then (usually
> when under high load), I see error messages like:
> 
> 2010/06/04 14:34:29 [error] 1884#0: *8844 upstream prematurely closed
> connection while reading response header from upstream, client:
> 119.235.237.93, server: portal.crowdreel.com, request: "GET
> /post/18Vmy HTTP/1.1", upstream:
> "http://unix:/home/app/crowdreel-portal/tmp/sockets/unicorn.sock:/post/18Vmy",
> host: "crowdreel.com"
> 
> ... a lot of these messages.. this is a pretty heavy duty server with
> lots of ram.
> 
> I have a setup similar to what I read up on github's blog as in: nginx
> -> unicorns via a socket upstream.
> 
> My unicorn.rb file looks like:
> 
> --------
> 
> # 16 workers and 1 master
> worker_processes 16
> 
> # Load the app into the master before forking workers for super-fast worker spawn times
> preload_app false
> 
> # Restart any workers that haven't responded in 30 seconds
> timeout 45
> 
> # Listen on a Unix data socket
> listen File.expand_path(File.dirname(__FILE__)+'/../tmp/sockets/unicorn.sock'), :backlog => 2048
> 
> ----------

> I don't preload the app because I start running into annoying issues
> where my workers use the same db connection, so instead I just have
> them load their own instance, its very fast to start up anyways.
> 
> I've read everything I could on this issue and exhausted Google, which
> leaves me to think the only thing left is some incompatibilities with
> 1.9.2. So I was wondering, how can I get logs from unicorn's
> workers/master to know what is happening to these workers, perhaps
> something is segfaulting?

Hi Peter,

Set stderr_path in your unicorn config file so Unicorn can log
its errors to a file.  Otherwise Unicorn will send all errors to
/dev/null when daemonized.

I shall emphasize the importance of stderr_path if you're using the
default logger configuration in the documentation.

Any chance the "/post/18Vmy" request is hitting your timeout?  The only
way you can know is if Unicorn logged its errors.  You might also want
to check your Rails log for exceptions, too, as some uncaught errors
may show up there.

-- 
Eric Wong

From normalperson at yhbt.net  Fri Jun  4 16:47:36 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 4 Jun 2010 13:47:36 -0700
Subject: [PATCH] doc: emphasize the importance of stderr_path
In-Reply-To: <20100604200648.GA6767@dcvr.yhbt.net>
References: <977C003B-E2CA-48B9-A3E6-C126DB231EB5@nulayer.com>
	<20100604200648.GA6767@dcvr.yhbt.net>
Message-ID: <20100604204736.GA4647@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> I shall emphasize the importance of stderr_path if you're using the
> default logger configuration in the documentation.

I've just pushed the following out to unicorn.git and website.

Comments/grammar/speling korections welcome as always.

>From 2d5a4b075801493a85c6e8d2dbdf0c95002e046d Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Fri, 4 Jun 2010 13:42:34 -0700
Subject: [PATCH] doc: emphasize the importance of stderr_path

While second nature to myself, stderr_path may be an
overlooked configuration parameter for some users.  Also,
add a minimal sample configuration file that is shorter
and hopefully less intimidating to new users.
---
 examples/unicorn.conf.minimal.rb |   13 +++++++++++++
 examples/unicorn.conf.rb         |   12 +++++++++---
 lib/unicorn/configurator.rb      |   22 ++++++++++++++++++----
 3 files changed, 40 insertions(+), 7 deletions(-)
 create mode 100644 examples/unicorn.conf.minimal.rb

diff --git a/examples/unicorn.conf.minimal.rb b/examples/unicorn.conf.minimal.rb
new file mode 100644
index 0000000..2a47910
--- /dev/null
+++ b/examples/unicorn.conf.minimal.rb
@@ -0,0 +1,13 @@
+# Minimal sample configuration file for Unicorn (not Rack) when used
+# with daemonization (unicorn -D) started in your working directory.
+#
+# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
+# documentation.
+# See also http://unicorn.bogomips.org/examples/unicorn.conf.rb for
+# a more verbose configuration using more features.
+
+listen 2007 # by default Unicorn listens on port 8080
+worker_processes 2 # this should be >= nr_cpus
+pid "/path/to/app/shared/pids/unicorn.pid"
+stderr_path "/path/to/app/shared/log/unicorn.log"
+stdout_path "/path/to/app/shared/log/unicorn.log"
diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb
index e209894..37c3e81 100644
--- a/examples/unicorn.conf.rb
+++ b/examples/unicorn.conf.rb
@@ -1,4 +1,9 @@
-# Sample configuration file for Unicorn (not Rack)
+# Sample verbose configuration file for Unicorn (not Rack)
+#
+# This configuration file documents many features of Unicorn
+# that may not be needed for some applications. See
+# http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb
+# for a much simpler configuration file.
 #
 # See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
 # documentation.
@@ -22,8 +27,9 @@ timeout 30
 # feel free to point this anywhere accessible on the filesystem
 pid "/path/to/app/shared/pids/unicorn.pid"
 
-# some applications/frameworks log to stderr or stdout, so prevent
-# them from going to /dev/null when daemonized here:
+# By default, the Unicorn logger will write to stderr.
+# Additionally, ome applications/frameworks log to stderr or stdout,
+# so prevent them from going to /dev/null when daemonized here:
 stderr_path "/path/to/app/shared/log/unicorn.stderr.log"
 stdout_path "/path/to/app/shared/log/unicorn.stdout.log"
 
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 64a25e3..4fa745d 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -7,9 +7,11 @@ module Unicorn
 
   # Implements a simple DSL for configuring a Unicorn server.
   #
-  # See http://unicorn.bogomips.org/examples/unicorn.conf.rb for an
-  # example config file.  An example config file for use with nginx is
-  # also available at http://unicorn.bogomips.org/examples/nginx.conf
+  # See http://unicorn.bogomips.org/examples/unicorn.conf.rb and
+  # http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb
+  # example configuration files.  An example config file for use with
+  # nginx is also available at
+  # http://unicorn.bogomips.org/examples/nginx.conf
   class Configurator < Struct.new(:set, :config_file, :after_reload)
 
     # Default settings for Unicorn
@@ -78,6 +80,10 @@ module Unicorn
     # sets object to the +new+ Logger-like object.  The new logger-like
     # object must respond to the following methods:
     #  +debug+, +info+, +warn+, +error+, +fatal+, +close+
+    # The default Logger will log its output to the path specified
+    # by +stderr_path+.  If you're running Unicorn daemonized, then
+    # you must specify a path to prevent error messages from going
+    # to /dev/null.
     def logger(new)
       %w(debug info warn error fatal close).each do |m|
         new.respond_to?(m) and next
@@ -310,11 +316,19 @@ module Unicorn
     # file will be opened with the File::APPEND flag and writes
     # synchronized to the kernel (but not necessarily to _disk_) so
     # multiple processes can safely append to it.
+    #
+    # If you are daemonizing and using the default +logger+, it is important
+    # to specify this as errors will otherwise be lost to /dev/null.
+    # Some applications/libraries may also triggering warnings that go to
+    # stderr, and they will end up here.
     def stderr_path(path)
       set_path(:stderr_path, path)
     end
 
-    # Same as stderr_path, except for $stdout
+    # Same as stderr_path, except for $stdout.  Not many Rack applications
+    # write to $stdout, but any that do will have their output written here.
+    # It is safe to point this to the same location a stderr_path.
+    # Like stderr_path, this defaults to /dev/null when daemonized.
     def stdout_path(path)
       set_path(:stdout_path, path)
     end
-- 
Eric Wong

From normalperson at yhbt.net  Fri Jun  4 16:59:25 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 4 Jun 2010 20:59:25 +0000
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net>
	<AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>
	<20100603184730.GA2421@dcvr.yhbt.net>
	<AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>
Message-ID: <20100604205925.GA7361@dcvr.yhbt.net>

Chris Wanstrath <chris at ozmm.org> wrote:
> That's what we do at GitHub. We're running Rails 2.2.2 and have a
> custom config.ru, thanks to Unicorn:
> 
> http://gist.github.com/424352

By the way, how's the OobGC middleware working for you guys?

Luke: did you also get a chance to try this in place of my original
monkey patch?

Thanks in advance for any info you can share.

-- 
Eric Wong

From stephan.gmane at ka.ag  Mon Jun  7 09:37:53 2010
From: stephan.gmane at ka.ag (Stephan Kaag)
Date: Mon, 7 Jun 2010 13:37:53 +0000 (UTC)
Subject: unicorn failing to start
References: <4BFF84F9.2000906@gmail.com>
Message-ID: <loom.20100607T153618-985@post.gmane.org>

Same issue overhere. Did someone investigate this?

Stefan Maier <stefanmaier <at> gmail.com> writes:

> 
> Hi,
> 
> i'm trying to start up unicorn_rails with a rails 3 beta3 project. All I
> can get out of it is this:
> I, [2010-05-28T08:54:45.770957 #17852]  INFO -- : reaped
> #<Process::Status: pid 17854 exit 1> worker=0
> I, [2010-05-28T08:54:45.771200 #17852]  INFO -- : worker=0 spawning...
> I, [2010-05-28T08:54:45.774049 #17858]  INFO -- : worker=0 spawned pid=17858
> I, [2010-05-28T08:54:45.774350 #17858]  INFO -- : Refreshing Gem list
> 
> /usr/local/lib/ruby/gems/1.9.1/gems/activesupport-
3.0.0.beta3/lib/active_support/deprecation/proxy_wrappers.rb:17:in
> `new': wrong number of arguments (1 for 2) (ArgumentError)
> 	from
> /usr/local/lib/ruby/gems/1.9.1/gems/activesupport-
3.0.0.beta3/lib/active_support/deprecation/proxy_wrappers.rb:17:in
> `method_missing'



From peter.kieltyka at nulayer.com  Mon Jun  7 12:07:17 2010
From: peter.kieltyka at nulayer.com (Peter)
Date: Mon, 7 Jun 2010 16:07:17 +0000 (UTC)
Subject: Logging errors from Unicorn
References: <977C003B-E2CA-48B9-A3E6-C126DB231EB5@nulayer.com>
	<20100604200648.GA6767@dcvr.yhbt.net>
Message-ID: <loom.20100607T171713-535@post.gmane.org>


Thanks Eric!

It really helps knowing the stderr_path configuration option :) doh. I haven't
yet run into the issue since, but I will have my unicorn logs ready when it
does.

By the way, have you ever thought about moving the unicorn mailing list to
a Google group? It's much easier to interact with, search, communicate, etc.
Just saying...

Even issues like trying to post this reply here, and getting "The following
errors were found: You have  lines longer than 80 characters. Fix that."
... hehe, thats pretty funny.

Cheers.

Peter



From normalperson at yhbt.net  Mon Jun  7 12:41:07 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 7 Jun 2010 16:41:07 +0000
Subject: unicorn failing to start
In-Reply-To: <loom.20100607T153618-985@post.gmane.org>
References: <4BFF84F9.2000906@gmail.com>
	<loom.20100607T153618-985@post.gmane.org>
Message-ID: <20100607164107.GA18935@dcvr.yhbt.net>

Stephan Kaag <stephan.gmane at ka.ag> wrote:
> Stefan Maier <stefanmaier <at> gmail.com> writes:
> > i'm trying to start up unicorn_rails with a rails 3 beta3 project. All I
> > can get out of it is this:
> > I, [2010-05-28T08:54:45.770957 #17852]  INFO -- : reaped
> > #<Process::Status: pid 17854 exit 1> worker=0
> > I, [2010-05-28T08:54:45.771200 #17852]  INFO -- : worker=0 spawning...
> > I, [2010-05-28T08:54:45.774049 #17858]  INFO -- : worker=0 spawned pid=17858
> > I, [2010-05-28T08:54:45.774350 #17858]  INFO -- : Refreshing Gem list
> > 
> > /usr/local/lib/ruby/gems/1.9.1/gems/activesupport-
> 3.0.0.beta3/lib/active_support/deprecation/proxy_wrappers.rb:17:in
> > `new': wrong number of arguments (1 for 2) (ArgumentError)
> > 	from
> > /usr/local/lib/ruby/gems/1.9.1/gems/activesupport-
> 3.0.0.beta3/lib/active_support/deprecation/proxy_wrappers.rb:17:in
> > `method_missing'
>
> Same issue overhere. Did someone investigate this?

(top posting corrected)

Hi Stephan, looks like Stefan worked around the problem
by using "unicorn".

I've made some changes in unicorn.git which _may_ fix the issues
and pushed out a pre-release gem here:

  http://unicorn.bogomips.org/files/unicorn-0.99.0.16.g59a625.gem

You can read about the actual changes to the above gem here:

  http://mid.gmane.org/20100604015838.GA21464 at dcvr.yhbt.net

-- 
Eric Wong

From normalperson at yhbt.net  Mon Jun  7 12:54:31 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 7 Jun 2010 16:54:31 +0000
Subject: Logging errors from Unicorn
In-Reply-To: <loom.20100607T171713-535@post.gmane.org>
References: <977C003B-E2CA-48B9-A3E6-C126DB231EB5@nulayer.com>
	<20100604200648.GA6767@dcvr.yhbt.net>
	<loom.20100607T171713-535@post.gmane.org>
Message-ID: <20100607165431.GB18935@dcvr.yhbt.net>

Peter <peter.kieltyka at nulayer.com> wrote:
> By the way, have you ever thought about moving the unicorn mailing list to
> a Google group? It's much easier to interact with, search, communicate, etc.
> Just saying...

Google Groups get more spam, and last I checked there's no way to
download mail archives easily other than scraping the site.

The Rubyforge at least has that, but the searchability sucks (so we
mirror to gmane/mail-archive).

Librelist is nice, but didn't exist when I started this project and at
this point the pain of migration is more than the benefit.  I'll
consider Librelist if there's strong support on the mailing list here.

> Even issues like trying to post this reply here, and getting "The following
> errors were found: You have  lines longer than 80 characters. Fix that."
> ... hehe, thats pretty funny.

Where does it say that?  I usually wrap my replies at 72 characters, but
there should be nothing in the mailman config that prevents longer lines
from being posted (and some patches do go longer than 80 due to the +/-
prefixes).  Of course HTML email is hugely wasteful and that's not
allowed here.

-- 
Eric Wong

From augusto at jadedpixel.com  Mon Jun  7 22:02:43 2010
From: augusto at jadedpixel.com (Augusto Becciu)
Date: Mon, 7 Jun 2010 23:02:43 -0300
Subject: Read error: #<TypeError: can't modify frozen string> raised from 
	HttpParser
In-Reply-To: <20100603234030.GA18377@dcvr.yhbt.net>
References: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com> 
	<20100602202518.GA8617@dcvr.yhbt.net>
	<AANLkTikCjTCVorHoIwsOtfSzt1Z-Mv4ZE8nf0Mqtgnqq@mail.gmail.com> 
	<20100602213815.GA23255@dcvr.yhbt.net>
	<AANLkTimy0cLsh36A4pBoYMzZJY5wBKrxCp7rgCxyiZ53@mail.gmail.com> 
	<20100603234030.GA18377@dcvr.yhbt.net>
Message-ID: <AANLkTimuGgcwNAMcVZdViFWdF-UcW_RGyZAue7phUXps@mail.gmail.com>

On Thu, Jun 3, 2010 at 8:40 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Augusto Becciu <augusto at jadedpixel.com> wrote:
>> On Wed, Jun 2, 2010 at 6:38 PM, Eric Wong <normalperson at yhbt.net> wrote:
>> > Augusto Becciu <augusto at jadedpixel.com> wrote:
>> >> On Wed, Jun 2, 2010 at 5:25 PM, Eric Wong <normalperson at yhbt.net> wrote:
>> >> > Augusto Becciu <augusto at jadedpixel.com> wrote:
>> >> >> Hey guys,
>> >> >>
>> >> >> Started running unicorn in a production server like two weeks ago.
>> >> >> It's been running smoothly, but looking at the logs found 44
>> >> >> exceptions like this:
>> >> >>
>> >> >> E, [2010-06-02T16:17:15.117071 #22680] ERROR -- : Read error:
>> >> >> #<TypeError: can't modify frozen string>
>> >> >> E, [2010-06-02T16:17:15.117270 #22680] ERROR -- :
>> >> >> /usr/lib/ruby/gems/1.8/gems/unicorn-0.99.0/lib/unicorn/http_request.rb:59:in
>> >> >> `headers'
>> >> >
>> >> > <snip>
>> >> >
>> >> >> Ruby version: ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux],
>> >> >> MBARI 0x8770, Ruby Enterprise Edition 2010.01
>
> <snip>
>
>> >> There's no way our application could be freezing that constant, at
>> >> least not intentionally.
>
> <snip>
>
>> Thanks Eric! Unfortunately after completely disabling RPM, we keep
>> getting that error. :(
>>
>> What else could it be?
>
> Any details about your application you're allowed to share can help us,
> especially a list of library/gem dependencies and versions.
>
> I would grep through your gems and vendor directories to ensure there's
> nothing freezing objects it shouldn't freeze. ?It could also be a buggy
> C extension corrupting memory somewhere, too...
>
> This isn't happening for all your worker processes, is it?
>
> If your application logs show which PID handled each request, you can
> join the PIDs from the application logs with PIDs in the error logs
> generated by Unicorn. ?That lets you narrow down the possible requests
> that could trigger the errors and help you reproduce the problem sooner.
>
> This is one of my favorite features of one-request-per-process model
> is that you can easily narrow down which request is triggering a
> particular bug without worrying about race conditions from other
> requests.
>
> This is certainly the first time I've heard of this problem, so I think
> it's specific to something in your application/environment.
>
> --
> Eric Wong
>

Hi Eric,

Thanks for the ideas.

Ended up adding some debugging code to unicorn that lead me to the
culprit of the problem. It turned out being a bug in the HttpParser.

When Unicorn receives a request with a "Version" header, the
HttpParser transforms it into "HTTP_VERSION". After that tries to add
it to the request hash which already contains a "HTTP_VERSION" key
with the actual http version of the request. So it tries to append the
new value separated by a comma. But since the http version is a
freezed constant, the TypeError exception is raised.

According to the HTTP RFC
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.1) a
"Version" header is valid. However, it's not supported in rack, since
rack has a HTTP_VERSION env variable for the http version. So I think
the easiest way to deal with this problem is to just ignore the header
since it is extremely unusual. We were getting it from a crappy bot.

Below is a proposed patch.

Thanks a lot for your help!
Augusto


diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 264db68..aa23024 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -197,6 +197,15 @@ static void write_value(VALUE req, struct http_parser *hp,
     assert_frozen(f);
   }

+  /*
+   * ignore "Version" headers since they conflict with the HTTP_VERSION
+   * rack env variable.
+   */
+  if (rb_str_cmp(f, g_http_version) == 0) {
+    hp->cont = Qnil;
+    return;
+  }
+
   e = rb_hash_aref(req, f);
   if (NIL_P(e)) {
     hp->cont = rb_hash_aset(req, f, v);
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index a067ac8..cb30f32 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -440,4 +440,24 @@ class HttpParserNgTest < Test::Unit::TestCase
     end
   end

+  def test_ignore_version_header
+    http = "GET / HTTP/1.1\r\nVersion: hello\r\n\r\n"
+    req = {}
+    assert_equal req, @parser.headers(req, http)
+    assert_equal '', http
+    expect = {
+      "SERVER_NAME" => "localhost",
+      "rack.url_scheme" => "http",
+      "REQUEST_PATH" => "/",
+      "SERVER_PROTOCOL" => "HTTP/1.1",
+      "PATH_INFO" => "/",
+      "HTTP_VERSION" => "HTTP/1.1",
+      "REQUEST_URI" => "/",
+      "SERVER_PORT" => "80",
+      "REQUEST_METHOD" => "GET",
+      "QUERY_STRING" => ""
+    }
+    assert_equal expect, req
+  end
+
 end

From normalperson at yhbt.net  Mon Jun  7 22:45:28 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 8 Jun 2010 02:45:28 +0000
Subject: Read error: #<TypeError: can't modify frozen string> raised
	from HttpParser
In-Reply-To: <AANLkTimuGgcwNAMcVZdViFWdF-UcW_RGyZAue7phUXps@mail.gmail.com>
References: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com>
	<20100602202518.GA8617@dcvr.yhbt.net>
	<AANLkTikCjTCVorHoIwsOtfSzt1Z-Mv4ZE8nf0Mqtgnqq@mail.gmail.com>
	<20100602213815.GA23255@dcvr.yhbt.net>
	<AANLkTimy0cLsh36A4pBoYMzZJY5wBKrxCp7rgCxyiZ53@mail.gmail.com>
	<20100603234030.GA18377@dcvr.yhbt.net>
	<AANLkTimuGgcwNAMcVZdViFWdF-UcW_RGyZAue7phUXps@mail.gmail.com>
Message-ID: <20100608024528.GA16631@dcvr.yhbt.net>

Augusto Becciu <augusto at jadedpixel.com> wrote:
> Hi Eric,
> 
> Thanks for the ideas.
> 
> Ended up adding some debugging code to unicorn that lead me to the
> culprit of the problem. It turned out being a bug in the HttpParser.

Thank you Augusto,

Your explanation was good so I'll just use it as a commit message
and credit you as the author.  See the patch below and let me know
if everything looks alright before I push it out.

I have a few small things to go over, but I'll be releasing 0.999.0
tonight (which I hope will be the last before we finally celebrate
have a 1.0.0 release).

>From 7d2295fa774d1c98dfbde2b09d93d58712253d24 Mon Sep 17 00:00:00 2001
From: Augusto Becciu <augusto at jadedpixel.com>
Date: Mon, 7 Jun 2010 23:02:43 -0300
Subject: [PATCH] http: ignore Version: header if explicitly set by client

When Unicorn receives a request with a "Version" header, the
HttpParser transforms it into "HTTP_VERSION". After that tries to add
it to the request hash which already contains a "HTTP_VERSION" key
with the actual http version of the request. So it tries to append the
new value separated by a comma. But since the http version is a
freezed constant, the TypeError exception is raised.

According to the HTTP RFC
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.1) a
"Version" header is valid. However, it's not supported in rack, since
rack has a HTTP_VERSION env variable for the http version. So I think
the easiest way to deal with this problem is to just ignore the header
since it is extremely unusual. We were getting it from a crappy bot.

ref: http://mid.gmane.org/AANLkTimuGgcwNAMcVZdViFWdF-UcW_RGyZAue7phUXps at mail.gmail.com

Acked-by: Eric Wong <normalperson at yhbt.net>
---
 ext/unicorn_http/unicorn_http.rl |    9 +++++++++
 test/unit/test_http_parser_ng.rb |   20 ++++++++++++++++++++
 2 files changed, 29 insertions(+), 0 deletions(-)

diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 264db68..aa23024 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -197,6 +197,15 @@ static void write_value(VALUE req, struct http_parser *hp,
     assert_frozen(f);
   }
 
+  /*
+   * ignore "Version" headers since they conflict with the HTTP_VERSION
+   * rack env variable.
+   */
+  if (rb_str_cmp(f, g_http_version) == 0) {
+    hp->cont = Qnil;
+    return;
+  }
+
   e = rb_hash_aref(req, f);
   if (NIL_P(e)) {
     hp->cont = rb_hash_aset(req, f, v);
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index a067ac8..cb30f32 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -440,4 +440,24 @@ class HttpParserNgTest < Test::Unit::TestCase
     end
   end
 
+  def test_ignore_version_header
+    http = "GET / HTTP/1.1\r\nVersion: hello\r\n\r\n"
+    req = {}
+    assert_equal req, @parser.headers(req, http)
+    assert_equal '', http
+    expect = {
+      "SERVER_NAME" => "localhost",
+      "rack.url_scheme" => "http",
+      "REQUEST_PATH" => "/",
+      "SERVER_PROTOCOL" => "HTTP/1.1",
+      "PATH_INFO" => "/",
+      "HTTP_VERSION" => "HTTP/1.1",
+      "REQUEST_URI" => "/",
+      "SERVER_PORT" => "80",
+      "REQUEST_METHOD" => "GET",
+      "QUERY_STRING" => ""
+    }
+    assert_equal expect, req
+  end
+
 end
-- 
Eric Wong

From normalperson at yhbt.net  Mon Jun  7 22:49:32 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 8 Jun 2010 02:49:32 +0000
Subject: unicorn_rails cleanup (possible fix for Rails3) pushed
In-Reply-To: <20100604015838.GA21464@dcvr.yhbt.net>
References: <20100604015838.GA21464@dcvr.yhbt.net>
Message-ID: <20100608024932.GA16608@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> Hi all,
> 
> I've pushed the following patch out go git://git.bogomips.org/unicorn
> along with a few other Rails-related test updates.  This is more of a
> shotgun fix (but less code is better :) since I haven't been able to
> reproduce the brokeness people have been seeing with "unicorn_rails"
> and Rails 3 betas.

<snip>

> You can grab a git gem here, too:
> 
>   http://unicorn.bogomips.org/files/unicorn-0.99.0.16.g59a625.gem

Anybody get a chance to try this?

-- 
Eric Wong

From augusto at jadedpixel.com  Mon Jun  7 23:01:50 2010
From: augusto at jadedpixel.com (Augusto Becciu)
Date: Tue, 8 Jun 2010 00:01:50 -0300
Subject: Read error: #<TypeError: can't modify frozen string> raised from 
	HttpParser
In-Reply-To: <20100608024528.GA16631@dcvr.yhbt.net>
References: <AANLkTim5Zd_NUWewbSLOy2cEZ1bsJSetq120DvmI79mK@mail.gmail.com> 
	<20100602202518.GA8617@dcvr.yhbt.net>
	<AANLkTikCjTCVorHoIwsOtfSzt1Z-Mv4ZE8nf0Mqtgnqq@mail.gmail.com> 
	<20100602213815.GA23255@dcvr.yhbt.net>
	<AANLkTimy0cLsh36A4pBoYMzZJY5wBKrxCp7rgCxyiZ53@mail.gmail.com> 
	<20100603234030.GA18377@dcvr.yhbt.net>
	<AANLkTimuGgcwNAMcVZdViFWdF-UcW_RGyZAue7phUXps@mail.gmail.com> 
	<20100608024528.GA16631@dcvr.yhbt.net>
Message-ID: <AANLkTim-2ZFejrdeHPRa7ESeovgQh3WBo0euogC7eFWF@mail.gmail.com>

That looks perfect. Thanks Eric!

On Mon, Jun 7, 2010 at 11:45 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Augusto Becciu <augusto at jadedpixel.com> wrote:
>> Hi Eric,
>>
>> Thanks for the ideas.
>>
>> Ended up adding some debugging code to unicorn that lead me to the
>> culprit of the problem. It turned out being a bug in the HttpParser.
>
> Thank you Augusto,
>
> Your explanation was good so I'll just use it as a commit message
> and credit you as the author. ?See the patch below and let me know
> if everything looks alright before I push it out.
>
> I have a few small things to go over, but I'll be releasing 0.999.0
> tonight (which I hope will be the last before we finally celebrate
> have a 1.0.0 release).
>
> From 7d2295fa774d1c98dfbde2b09d93d58712253d24 Mon Sep 17 00:00:00 2001
> From: Augusto Becciu <augusto at jadedpixel.com>
> Date: Mon, 7 Jun 2010 23:02:43 -0300
> Subject: [PATCH] http: ignore Version: header if explicitly set by client
>
> When Unicorn receives a request with a "Version" header, the
> HttpParser transforms it into "HTTP_VERSION". After that tries to add
> it to the request hash which already contains a "HTTP_VERSION" key
> with the actual http version of the request. So it tries to append the
> new value separated by a comma. But since the http version is a
> freezed constant, the TypeError exception is raised.
>
> According to the HTTP RFC
> (http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.1) a
> "Version" header is valid. However, it's not supported in rack, since
> rack has a HTTP_VERSION env variable for the http version. So I think
> the easiest way to deal with this problem is to just ignore the header
> since it is extremely unusual. We were getting it from a crappy bot.
>
> ref: http://mid.gmane.org/AANLkTimuGgcwNAMcVZdViFWdF-UcW_RGyZAue7phUXps at mail.gmail.com
>
> Acked-by: Eric Wong <normalperson at yhbt.net>
> ---
> ?ext/unicorn_http/unicorn_http.rl | ? ?9 +++++++++
> ?test/unit/test_http_parser_ng.rb | ? 20 ++++++++++++++++++++
> ?2 files changed, 29 insertions(+), 0 deletions(-)
>
> diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
> index 264db68..aa23024 100644
> --- a/ext/unicorn_http/unicorn_http.rl
> +++ b/ext/unicorn_http/unicorn_http.rl
> @@ -197,6 +197,15 @@ static void write_value(VALUE req, struct http_parser *hp,
> ? ? assert_frozen(f);
> ? }
>
> + ?/*
> + ? * ignore "Version" headers since they conflict with the HTTP_VERSION
> + ? * rack env variable.
> + ? */
> + ?if (rb_str_cmp(f, g_http_version) == 0) {
> + ? ?hp->cont = Qnil;
> + ? ?return;
> + ?}
> +
> ? e = rb_hash_aref(req, f);
> ? if (NIL_P(e)) {
> ? ? hp->cont = rb_hash_aset(req, f, v);
> diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
> index a067ac8..cb30f32 100644
> --- a/test/unit/test_http_parser_ng.rb
> +++ b/test/unit/test_http_parser_ng.rb
> @@ -440,4 +440,24 @@ class HttpParserNgTest < Test::Unit::TestCase
> ? ? end
> ? end
>
> + ?def test_ignore_version_header
> + ? ?http = "GET / HTTP/1.1\r\nVersion: hello\r\n\r\n"
> + ? ?req = {}
> + ? ?assert_equal req, @parser.headers(req, http)
> + ? ?assert_equal '', http
> + ? ?expect = {
> + ? ? ?"SERVER_NAME" => "localhost",
> + ? ? ?"rack.url_scheme" => "http",
> + ? ? ?"REQUEST_PATH" => "/",
> + ? ? ?"SERVER_PROTOCOL" => "HTTP/1.1",
> + ? ? ?"PATH_INFO" => "/",
> + ? ? ?"HTTP_VERSION" => "HTTP/1.1",
> + ? ? ?"REQUEST_URI" => "/",
> + ? ? ?"SERVER_PORT" => "80",
> + ? ? ?"REQUEST_METHOD" => "GET",
> + ? ? ?"QUERY_STRING" => ""
> + ? ?}
> + ? ?assert_equal expect, req
> + ?end
> +
> ?end
> --
> Eric Wong
>

From normalperson at yhbt.net  Tue Jun  8 05:51:40 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 8 Jun 2010 09:51:40 +0000
Subject: [ANN] unicorn 0.990.0 - inching towards 1.0
Message-ID: <20100608095140.GB30419@dcvr.yhbt.net>

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

Thanks to Augusto Becciu for finding a bug in the HTTP parser
that caused a TypeError (and 500) when a rare client set the
"Version:" header which conflicts with the HTTP_VERSION header
we parse in the first line of the request[1].

Horizontal tabs are now allowed as leading whitespace in header
values as according to RFC 2616 as pointed out by
I?aki Baz Castillo[2].

Taking a hint from Rack 1.1, the "logger" configuration
parameter no longer requires a "close" method.  This means some
more Logger replacements may be used.

There's a new, optional, Unicorn (and maybe Passenger)-only
middleware, Unicorn::OobGC[2] that runs GC outside of the normal
request/response cycle to help out memory-hungry applications.
Thanks to Luke Melia for being brave enough to test and report
back on my big_app_gc.rb monkey patch[3] which lead up to this.

Rails 3 (beta) support:

  Using "unicorn" is still recommended as Rails 3 comes with
  a config.ru, but "unicorn_rails" is cleaned up a bit and
  *should* work as well as "unicorn" out-of-the-box.  Feedback
  is much appreciated.

Rubinius updates:

  USR2 binary upgrades are broken due to
  {TCPServer,UNIXServer}.for_fd[5][6] being broken
  (differently).

  Repeatedly hitting the server with signals in a tight
  loop is unusual and not recommended[7].

  There are some workarounds and general code cleanups for other
  issues[8], as well but things should generally work unless you
  need USR2 upgrades.  Feedback and reports would be greatly
  appreciated as usual.

MRI support:

  All tests (except old Rails) run and pass under 1.9.2-preview3.
  1.8.7 and 1.9.1 work well as usual and will continue to be
  supported indefinitely.

Lets hope this is the last release before 1.0.  Please report
any issues on the mailing list[9] or email us privately[a].
Don't send HTML mail.

[1] - http://mid.gmane.org/AANLkTimuGgcwNAMcVZdViFWdF-UcW_RGyZAue7phUXps at mail.gmail.com
[2] - http://mid.gmane.org/i2xcc1f582e1005070651u294bd83oc73d1e0adf72373a at mail.gmail.com
[3] - http://unicorn.bogomips.org/Unicorn/OobGC.html
[4] - http://unicorn.bogomips.org/examples/big_app_gc.rb
[5] - http://github.com/evanphx/rubinius/issues/354
[6] - http://github.com/evanphx/rubinius/issues/355
[7] - http://github.com/evanphx/rubinius/issues/356
[8] - http://github.com/evanphx/rubinius/issues/347
[9] - mailto:mongrel-unicorn at rubyforge.org
[a] - mailto:unicorn at bogomips.org

-- 
Eric Wong

From hongli at phusion.nl  Tue Jun  8 08:24:52 2010
From: hongli at phusion.nl (Hongli Lai)
Date: Tue, 8 Jun 2010 14:24:52 +0200
Subject: unicorn_rails cleanup (possible fix for Rails3) pushed
Message-ID: <AANLkTiksBxIo_PFWoiPTWi1entXZRb7D2uE-Rl7H3lbw@mail.gmail.com>

Hi Eric.

It would appear that recent Rails 3 changes have broken unicorn_rails,
just like they broke Phusion Passenger. Here's a patch which fixes the
problem.
http://gist.github.com/429944

With kind regards,
Hongli Lai

-- 
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: info at phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

From normalperson at yhbt.net  Tue Jun  8 15:20:34 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 8 Jun 2010 19:20:34 +0000
Subject: unicorn_rails cleanup (possible fix for Rails3) pushed
In-Reply-To: <AANLkTiksBxIo_PFWoiPTWi1entXZRb7D2uE-Rl7H3lbw@mail.gmail.com>
References: <AANLkTiksBxIo_PFWoiPTWi1entXZRb7D2uE-Rl7H3lbw@mail.gmail.com>
Message-ID: <20100608192034.GA2940@dcvr.yhbt.net>

Hongli Lai <hongli at phusion.nl> wrote:
> Hi Eric.
> 
> It would appear that recent Rails 3 changes have broken unicorn_rails,
> just like they broke Phusion Passenger. Here's a patch which fixes the
> problem.
> http://gist.github.com/429944

Thanks Hongli, so this only affects people who remove the
config.ru that Rails 3 creates for them?  Yikes...

A few comments:

+  if ::Rails::VERSION::MAJOR >= 3 && ::File.exist?('config/application.rb')
+    ::File.read('config/application.rb') =~ /^module (.+)$/

Maybe a more flexible regexp like this: /^module\s+([\w:]+)\s*$/ (or
maybe even starting with: "^\s*") would be more reliable for folks who
leave extra spaces around.

Unfortunately, Ruby is not Python :)

@@ -148,9 +167,9 @@ def rails_builder(daemonize)
       else
         use Rails::Rack::LogTailer unless daemonize
         use Rails::Rack::Debugger if $DEBUG
+        use Rails::Rack::Static, map_path
         map(map_path) do
-          use Rails::Rack::Static
-          run ActionController::Dispatcher.new
+          run rails_dispatcher

Changing the call to use Rails::Rack::Static there is wrong.  map_path
is the URI prefix (RAILS_RELATIVE_URL_ROOT) and not the static file
path we serve from.  It appears the deprecation in Rails 3 broke some
things and ActionDispatch::Static is configured slightly differently.

Let me know if the edited patch below looks alright to you.

I'll also push out a few Rails 3 tests that exercise the missing
config.ru cases.

>From 222ae0a353eda446a480e5c4b473a218304f9594 Mon Sep 17 00:00:00 2001
From: Hongli Lai (Phusion) <hongli at phusion.nl>
Date: Tue, 8 Jun 2010 13:22:25 +0200
Subject: [PATCH] Fix unicorn_rails compatibility with the latest Rails 3 code

This allows us to properly detect Rails 3 installations
in cases where config.ru is not present.

[ew: expanded commit message
 fixed static file serving,
 more flexible regexp for matching module ]

ref: mid.gmane.org/AANLkTiksBxIo_PFWoiPTWi1entXZRb7D2uE-Rl7H3lbw at mail.gmail.com
Acked-by: Eric Wong <normalperson at yhbt.net>
---
 bin/unicorn_rails |   26 ++++++++++++++++++++++++--
 1 files changed, 24 insertions(+), 2 deletions(-)

diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 45a9b11..ed235ba 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -109,6 +109,24 @@ end
 
 ru = ARGV[0] || (File.exist?('config.ru') ? 'config.ru' : nil)
 
+def rails_dispatcher
+  if ::Rails::VERSION::MAJOR >= 3 && ::File.exist?('config/application.rb')
+    if ::File.read('config/application.rb') =~ /^module\s+([\w:]+)\s*$/
+      app_module = Object.const_get($1)
+      begin
+        result = app_module::Application
+      rescue NameError
+      end
+    end
+  end
+
+  if result.nil? && defined?(ActionController::Dispatcher)
+    result = ActionController::Dispatcher.new
+  end
+
+  result || abort("Unable to locate the application dispatcher class")
+end
+
 def rails_builder(daemonize)
   # this lambda won't run until after forking if preload_app is false
   lambda do ||
@@ -149,8 +167,12 @@ def rails_builder(daemonize)
         use Rails::Rack::LogTailer unless daemonize
         use Rails::Rack::Debugger if $DEBUG
         map(map_path) do
-          use Rails::Rack::Static
-          run ActionController::Dispatcher.new
+          if defined?(ActionDispatch::Static)
+            use ActionDispatch::Static, "#{Rails.root}/public"
+          else
+            use Rails::Rack::Static
+          end
+          run rails_dispatcher
         end
       end
     end.to_app
-- 
Eric Wong

From hongli at phusion.nl  Tue Jun  8 15:25:38 2010
From: hongli at phusion.nl (Hongli Lai)
Date: Tue, 8 Jun 2010 21:25:38 +0200
Subject: unicorn_rails cleanup (possible fix for Rails3) pushed
In-Reply-To: <20100608192034.GA2940@dcvr.yhbt.net>
References: <AANLkTiksBxIo_PFWoiPTWi1entXZRb7D2uE-Rl7H3lbw@mail.gmail.com> 
	<20100608192034.GA2940@dcvr.yhbt.net>
Message-ID: <AANLkTimtK_mjX_8zmEccxz4j5yjl49KUu07ilK66BU-9@mail.gmail.com>

On Tue, Jun 8, 2010 at 9:20 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Thanks Hongli, so this only affects people who remove the
> config.ru that Rails 3 creates for them? ?Yikes...

No. The problem even occurs if you already have config.ru. But the
thing is, Rails 3 has deprecated ActionController::Dispatcher a few
weeks ago and replaced it with a stub. Rails::Rack::Static changed its
interface and must be constructed differently. The only way to obtain
a valid Rack endpoint for the app seems to be parsing
config/application.rb


> A few comments:
>
> + ?if ::Rails::VERSION::MAJOR >= 3 && ::File.exist?('config/application.rb')
> + ? ?::File.read('config/application.rb') =~ /^module (.+)$/
>
> Maybe a more flexible regexp like this: /^module\s+([\w:]+)\s*$/ (or
> maybe even starting with: "^\s*") would be more reliable for folks who
> leave extra spaces around.

I think it's easier to just call 'strip'. :)

> Unfortunately, Ruby is not Python :)
>
> @@ -148,9 +167,9 @@ def rails_builder(daemonize)
> ? ? ? else
> ? ? ? ? use Rails::Rack::LogTailer unless daemonize
> ? ? ? ? use Rails::Rack::Debugger if $DEBUG
> + ? ? ? ?use Rails::Rack::Static, map_path
> ? ? ? ? map(map_path) do
> - ? ? ? ? ?use Rails::Rack::Static
> - ? ? ? ? ?run ActionController::Dispatcher.new
> + ? ? ? ? ?run rails_dispatcher
>
> Changing the call to use Rails::Rack::Static there is wrong. ?map_path
> is the URI prefix (RAILS_RELATIVE_URL_ROOT) and not the static file
> path we serve from. ?It appears the deprecation in Rails 3 broke some
> things and ActionDispatch::Static is configured slightly differently.
>
> Let me know if the edited patch below looks alright to you.

Yes it looks fine. A bit overcomplicated regexp compared to using
'strip' but whatever works. :)

With kind regards,
Hongli Lai
-- 
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: info at phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

From normalperson at yhbt.net  Tue Jun  8 16:55:59 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 8 Jun 2010 13:55:59 -0700
Subject: unicorn_rails cleanup (possible fix for Rails3) pushed
In-Reply-To: <AANLkTimtK_mjX_8zmEccxz4j5yjl49KUu07ilK66BU-9@mail.gmail.com>
References: <AANLkTiksBxIo_PFWoiPTWi1entXZRb7D2uE-Rl7H3lbw@mail.gmail.com>
	<20100608192034.GA2940@dcvr.yhbt.net>
	<AANLkTimtK_mjX_8zmEccxz4j5yjl49KUu07ilK66BU-9@mail.gmail.com>
Message-ID: <20100608205559.GA29661@dcvr.yhbt.net>

Hongli Lai <hongli at phusion.nl> wrote:
> On Tue, Jun 8, 2010 at 9:20 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > Thanks Hongli, so this only affects people who remove the
> > config.ru that Rails 3 creates for them? ?Yikes...
> 
> No. The problem even occurs if you already have config.ru. But the
> thing is, Rails 3 has deprecated ActionController::Dispatcher a few
> weeks ago and replaced it with a stub. Rails::Rack::Static changed its
> interface and must be constructed differently. The only way to obtain
> a valid Rack endpoint for the app seems to be parsing
> config/application.rb

Actually, I made "unicorn_rails" completely bypass the "rails_builder"
method if there's a config.ru, so it should never hit the
ActionController::Dispatcher stuff.

> > Let me know if the edited patch below looks alright to you.
> 
> Yes it looks fine. A bit overcomplicated regexp compared to using
> 'strip' but whatever works. :)

I just kept the regexp as-is, works for me.

I just managed to push this to git://git.bogomips.org/unicorn.git before
my Internet connection died on me earlier today.  I've beefed up the
tests a bit but will probably do more later.

Eric Wong (4):
      t0300: Rails 3 test actually uses unicorn_rails
      tests: libify common rails3 setup code
      unicorn_rails: fix requires for Ruby 1.9.2
      tests: add Rails 3 test for the missing config.ru case

Hongli Lai (Phusion) (1):
      Fix unicorn_rails compatibility with the latest Rails 3 code

Thanks again!

-- 
Eric Wong

From normalperson at yhbt.net  Tue Jun  8 18:36:54 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 8 Jun 2010 22:36:54 +0000
Subject: unicorn prerelease 0.990.0.5.gbfb1
Message-ID: <20100608223654.GA19834@dcvr.yhbt.net>

Hi all,

I decided to try out the prerelease feature of Rubygems
for the first time, it seems to work...

So yes, "gem install --pre unicorn" if you're using Rails 3
(and probably like to type 'rm config.ru' :>)

Same list of changes since v0.990.0:

  Eric Wong (4):
        t0300: Rails 3 test actually uses unicorn_rails
        tests: libify common rails3 setup code
        unicorn_rails: fix requires for Ruby 1.9.2
        tests: add Rails 3 test for the missing config.ru case

  Hongli Lai (Phusion) (1):
        Fix unicorn_rails compatibility with the latest Rails 3 code

-- 
Eric Wong

From jamie at tramchase.com  Tue Jun  8 19:01:00 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Tue, 8 Jun 2010 19:01:00 -0400
Subject: funky process tree + stillborn masters
In-Reply-To: <20100428074002.GB8080@dcvr.yhbt.net>
References: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
	<20100408235548.GA11184@dcvr.yhbt.net>
	<20100409012050.GA31641@dcvr.yhbt.net>
	<20100412025200.GA29391@dcvr.yhbt.net>
	<F04EF571-4CCF-425F-82BC-C2363308C6C4@tramchase.com>
	<20100413052410.GA31794@dcvr.yhbt.net>
	<20100419182146.GA22994@dcvr.yhbt.net>
	<B26A4E0B-9155-40A1-B53C-D1F6521DA20B@tramchase.com>
	<20100428074002.GB8080@dcvr.yhbt.net>
Message-ID: <BA714A62-D910-4A6C-B8C0-AB3BEC8962F7@tramchase.com>

On Apr 28, 2010, at 12:05 AM, Jamie Wilkinson wrote:

> Update from the trenches: I've traced this down to the newrelic_rpm agent
> 
> I've also filed a bug with NewRelic:
> http://support.newrelic.com/discussions/support/2577-newrelic-agentbundler-causing-stillborn-unicorn-processes?unresolve=true

I'm happy to announce the recently released 2.12.2 newrelic_rpm gem fixed this long-running issue for me (newrelic_rpm causing some unicorns to fail to start) 
http://rubygems.org/gems/newrelic_rpm/versions/2.12.2 

Huge thanks to Bill Kayser from NewRelic & Eric Wong for his patient debugging!

-jamie
http://jamiedubs.com

From alex at simonov.me  Wed Jun  9 04:39:22 2010
From: alex at simonov.me (Alexander Simonov)
Date: Wed, 9 Jun 2010 11:39:22 +0300
Subject: [ANN] unicorn 0.990.0 - inching towards 1.0
In-Reply-To: <20100608095140.GB30419@dcvr.yhbt.net>
References: <20100608095140.GB30419@dcvr.yhbt.net>
Message-ID: <E4FB514D-0056-4D59-8DD3-2B8A33CACD56@simonov.me>


Hello!
One question: it's a normal state if i start rails app without preload and after Gem.refresh all works go down by exception from Ge.refresh and master process all time trying reup they.
And it's go to recursion. I know it's issue of people who start app, but in any case it's must be exception for stop of master process. Is i check code when starts workers and if we have preload_app false method run build_app!.
If we have preload_app true - then we check it before start worker. 
 

From oct at fotonauts.com  Wed Jun  9 08:58:15 2010
From: oct at fotonauts.com (Pierre Baillet)
Date: Wed, 9 Jun 2010 14:58:15 +0200
Subject: Working directory and config.ru
Message-ID: <AANLkTimKb7JARr_69nfVrJLvMZH3Gvs1o_KwZFLKfuxy@mail.gmail.com>

Hi,

I naively expected unicorn to honor the working_directory
configuration directive before attempting to locate the config.ru but
this does not seem to be the case. Is this behavior wanted ? From the
code, I can guess that the configuration file parsing is done much
later  than the current config.ru location attempt.

Cheers
-- 
Pierre Baillet <oct at fotonauts.com>
http://www.fotopedia.com/

From oct at fotonauts.com  Wed Jun  9 09:17:04 2010
From: oct at fotonauts.com (Pierre Baillet)
Date: Wed, 9 Jun 2010 15:17:04 +0200
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <AANLkTimV05UjKFPJW7xFW7OTALIrQH9Qr9I2u4LBYTwe@mail.gmail.com>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net>
	<AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>
	<20100603184730.GA2421@dcvr.yhbt.net>
	<AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>
	<AANLkTimV05UjKFPJW7xFW7OTALIrQH9Qr9I2u4LBYTwe@mail.gmail.com>
Message-ID: <AANLkTimubIMS1FrgGT7gCneO-7N250eRJxQjs4LACAFl@mail.gmail.com>

Hello Unicorns,

I've manage to create a simple middleware that replaces the soft
timeout feature. You can have a look at it at
http://gist.github.com/431451

Note that some weird Ruby interpreter behavior breaks at least the
first level of the generated stacktrace (it indicates the actual
method where the raise happened but the wrong line number and file).

Thanks for your assistance,
Cheers,
-- 
Pierre.


On Thu, Jun 3, 2010 at 9:40 PM, Pierre Baillet <oct at fotonauts.com> wrote:
> Ohai,
>
> On Thu, Jun 3, 2010 at 9:38 PM, Chris Wanstrath <chris at ozmm.org> wrote:
>> On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong <normalperson at yhbt.net> wrote:
>>
>>> Actually, internally, Unicorn only knows about Rack and wraps older
>>> CGI-based Rails using the Unicorn::App::OldRails application (via
>>> Unicorn::CGIWrapper).
>>>
>>> "unicorn_rails" basically wraps up the following config for you,
>>> but you could achieve the same effect using "unicorn" and an
>>> explicitly written config.ru:
>>
>> That's what we do at GitHub. We're running Rails 2.2.2 and have a
>> custom config.ru, thanks to Unicorn:
>>
>
> That's great, thanks for the Idea, I'll look into rack middleware
> then. I'll let you know if I manage to have something clean enough.
>
> Cheers,
> --
> Pierre Baillet <oct at fotonauts.com>
> http://www.fotopedia.com/
>



-- 
Pierre Baillet <oct at fotonauts.com>
http://www.fotopedia.com/

From normalperson at yhbt.net  Wed Jun  9 14:04:31 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 9 Jun 2010 11:04:31 -0700
Subject: Working directory and config.ru
In-Reply-To: <AANLkTimKb7JARr_69nfVrJLvMZH3Gvs1o_KwZFLKfuxy@mail.gmail.com>
References: <AANLkTimKb7JARr_69nfVrJLvMZH3Gvs1o_KwZFLKfuxy@mail.gmail.com>
Message-ID: <20100609180431.GA8027@dcvr.yhbt.net>

Pierre Baillet <oct at fotonauts.com> wrote:
> Hi,
> 
> I naively expected unicorn to honor the working_directory
> configuration directive before attempting to locate the config.ru but
> this does not seem to be the case. Is this behavior wanted ? From the
> code, I can guess that the configuration file parsing is done much
> later  than the current config.ru location attempt.

Hi Pierre,

Oops, definitely not wanted behavior.  Unicorn should honor
working_directory with regard to config.ru.  I'll start working
on a fix shortly, thanks for the report!

-- 
Eric Wong

From philip at ingraminternet.com  Wed Jun  9 08:51:59 2010
From: philip at ingraminternet.com (Philip Ingram)
Date: Wed, 9 Jun 2010 08:51:59 -0400
Subject: unicorn_rails in a loop of not starting...
Message-ID: <F28788A6-FC64-4BF9-8380-7AD68FB496D6@ingraminternet.com>

Hi,

Sorry for the subject, just  not sure what is going on and how to describe it.

Problem:
-------------
When running unicorn_rails in my RAILS_ROOT directory (the same one where the script/server command runs fine) i get terminal printout  saying:
==================
unicorn_rails worker[0] --path /Users/me/current_project/project -l0.0.0.0:8080 must be run inside RAILS_ROOT: #<LoadError: no such file to load -- config/boot>
I, [2010-06-09T08:42:50.154457 #22383]  INFO -- : reaped #<Process::Status: pid 22384 exit 1> worker=0
I, [2010-06-09T08:42:50.155261 #22383]  INFO -- : worker=0 spawning...
I, [2010-06-09T08:42:50.158954 #22385]  INFO -- : worker=0 spawned pid=22385
I, [2010-06-09T08:42:50.159520 #22385]  INFO -- : Refreshing Gem list
unicorn_rails worker[0] --path /Users/me/current_project/project -l0.0.0.0:8080 must be run inside RAILS_ROOT: #<LoadError: no such file to load -- config/boot>
I, [2010-06-09T08:42:51.216077 #22383]  INFO -- : reaped #<Process::Status: pid 22385 exit 1> worker=0
I, [2010-06-09T08:42:51.217098 #22383]  INFO -- : worker=0 spawning...
I, [2010-06-09T08:42:51.220027 #22386]  INFO -- : worker=0 spawned pid=22386
I, [2010-06-09T08:42:51.220538 #22386]  INFO -- : Refreshing Gem list
====================

Background:
------------------
I am running Rails 2.3.8, Ruby 1.9.2-head and have both rack (1.1.0, 1.0.1) installed, or else rails won't boot up for some silly reason.  (I.e., if i uninstall 1.1.0, rails cries that it needs 1.0.1).
I am also running unicorn (0.990.0) (fresh install) .  All of this through rvm 0.1.37.

Am i missing some installation steps that aren't described in the README/github page?
running unicorn_rails in my root directory should just work right?

Thanks.
Phililp



From normalperson at yhbt.net  Wed Jun  9 14:22:45 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 9 Jun 2010 11:22:45 -0700
Subject: [ANN] unicorn 0.990.0 - inching towards 1.0
In-Reply-To: <E4FB514D-0056-4D59-8DD3-2B8A33CACD56@simonov.me>
References: <20100608095140.GB30419@dcvr.yhbt.net>
	<E4FB514D-0056-4D59-8DD3-2B8A33CACD56@simonov.me>
Message-ID: <20100609182245.GB8027@dcvr.yhbt.net>

Alexander Simonov <alex at simonov.me> wrote:
> 
> Hello!
> 
> One question: it's a normal state if i start rails app without preload
> and after Gem.refresh all works go down by exception from Ge.refresh
> and master process all time trying reup they.  And it's go to
> recursion. I know it's issue of people who start app, but in any case
> it's must be exception for stop of master process. Is i check code
> when starts workers and if we have preload_app false method run
> build_app!.  If we have preload_app true - then we check it before
> start worker. 

Hi Alexander,

I hope I'm understanding you correctly, so you want the master process
to die if you've misdeployed or misconfigured your app?

This is one of those sharp edges of Unicorn that might be pointless
to protect against with preload_app=false...

The general rule is that you shouldn't be mucking around with Gem (or
any other library) installations on a production server unless you're
deploying.  And whenever you're deploying, you would always be checking
to see you've deployed correctly anyways, so you can detect any screwups
in the installation/deployment.

Let me know if that makes sense.

-- 
Eric Wong

From normalperson at yhbt.net  Wed Jun  9 14:43:56 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 9 Jun 2010 11:43:56 -0700
Subject: unicorn_rails in a loop of not starting...
In-Reply-To: <F28788A6-FC64-4BF9-8380-7AD68FB496D6@ingraminternet.com>
References: <F28788A6-FC64-4BF9-8380-7AD68FB496D6@ingraminternet.com>
Message-ID: <20100609184356.GA8875@dcvr.yhbt.net>

Philip Ingram <philip at ingraminternet.com> wrote:
> Hi,
> 
> Sorry for the subject, just  not sure what is going on and how to describe it.
> 
> Problem:
> -------------
> When running unicorn_rails in my RAILS_ROOT directory (the same one where the script/server command runs fine) i get terminal printout  saying:
> ==================
> unicorn_rails worker[0] --path /Users/me/current_project/project -l0.0.0.0:8080 must be run inside RAILS_ROOT: #<LoadError: no such file to load -- config/boot>

<snip>

Hi Philip,

> I am running Rails 2.3.8, Ruby 1.9.2-head and have both rack (1.1.0,
> 1.0.1) installed, or else rails won't boot up for some silly reason.
> (I.e., if i uninstall 1.1.0, rails cries that it needs 1.0.1).  I am
> also running unicorn (0.990.0) (fresh install) .  All of this through
> rvm 0.1.37.

Try the prerelease on Rubygems.org: unicorn 0.990.0.5.gbfb1 `gem install
--pre unicorn`.  Ruby 1.9.2 no longer has '.' in its default
$LOAD_PATH, so instead of "require 'config/boot'" we have to do
"require ::File.expand_path('config/boot')"

> Am i missing some installation steps that aren't described in the
> README/github page?  running unicorn_rails in my root directory should
> just work right?

I'm not sure which github page you're referring to without the full URL,
but the official docs are online at http://unicorn.bogomips.org/ (and
always distributed with the source tree).

Anyways, the prerelease should help and not introduce any regressions.
I plan on having 0.991.0 out in the next day or two.

-- 
Eric Wong

From alex at simonov.me  Thu Jun 10 01:26:14 2010
From: alex at simonov.me (Alexander Simonov)
Date: Thu, 10 Jun 2010 08:26:14 +0300
Subject: [ANN] unicorn 0.990.0 - inching towards 1.0
In-Reply-To: <20100609182245.GB8027@dcvr.yhbt.net>
References: <20100608095140.GB30419@dcvr.yhbt.net>
	<E4FB514D-0056-4D59-8DD3-2B8A33CACD56@simonov.me>
	<20100609182245.GB8027@dcvr.yhbt.net>
Message-ID: <EFCFB26E-2D97-4DB1-AD50-141D224D9B4A@simonov.me>


On Jun 9, 2010, at 9:22 PM, Eric Wong wrote:

> Alexander Simonov <alex at simonov.me> wrote:
>> 
>> Hello!
>> 
>> One question: it's a normal state if i start rails app without preload
>> and after Gem.refresh all works go down by exception from Ge.refresh
>> and master process all time trying reup they.  And it's go to
>> recursion. I know it's issue of people who start app, but in any case
>> it's must be exception for stop of master process. Is i check code
>> when starts workers and if we have preload_app false method run
>> build_app!.  If we have preload_app true - then we check it before
>> start worker. 
> 
> Hi Alexander,
> 
> I hope I'm understanding you correctly, so you want the master process
> to die if you've misdeployed or misconfigured your app?

yes, something like this.

> 
> This is one of those sharp edges of Unicorn that might be pointless
> to protect against with preload_app=false...
> 
> The general rule is that you shouldn't be mucking around with Gem (or
> any other library) installations on a production server unless you're
> deploying.  And whenever you're deploying, you would always be checking
> to see you've deployed correctly anyways, so you can detect any screwups
> in the installation/deployment.
> 
> Let me know if that makes sense.


Ok. But the main thing is the master process goes to recursion restart of workers if they return exit or exception.May be add the counter? For example, 10 times we trying to respawn working process and after stop master.
I think it makes sense. 


From normalperson at yhbt.net  Thu Jun 10 03:38:50 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 10 Jun 2010 07:38:50 +0000
Subject: [ANN] unicorn 0.990.0 - inching towards 1.0
In-Reply-To: <EFCFB26E-2D97-4DB1-AD50-141D224D9B4A@simonov.me>
References: <20100608095140.GB30419@dcvr.yhbt.net>
	<E4FB514D-0056-4D59-8DD3-2B8A33CACD56@simonov.me>
	<20100609182245.GB8027@dcvr.yhbt.net>
	<EFCFB26E-2D97-4DB1-AD50-141D224D9B4A@simonov.me>
Message-ID: <20100610073850.GA28455@dcvr.yhbt.net>

Alexander Simonov <alex at simonov.me> wrote:
> On Jun 9, 2010, at 9:22 PM, Eric Wong wrote:
> > Alexander Simonov <alex at simonov.me> wrote:
> >> 
> >> Hello!
> >> 
> >> One question: it's a normal state if i start rails app without preload
> >> and after Gem.refresh all works go down by exception from Ge.refresh
> >> and master process all time trying reup they.  And it's go to
> >> recursion. I know it's issue of people who start app, but in any case
> >> it's must be exception for stop of master process. Is i check code
> >> when starts workers and if we have preload_app false method run
> >> build_app!.  If we have preload_app true - then we check it before
> >> start worker. 
> > 
> > Hi Alexander,
> > 
> > I hope I'm understanding you correctly, so you want the master process
> > to die if you've misdeployed or misconfigured your app?
> 
> yes, something like this.
> 
> > 
> > This is one of those sharp edges of Unicorn that might be pointless
> > to protect against with preload_app=false...
> > 
> > The general rule is that you shouldn't be mucking around with Gem (or
> > any other library) installations on a production server unless you're
> > deploying.  And whenever you're deploying, you would always be checking
> > to see you've deployed correctly anyways, so you can detect any screwups
> > in the installation/deployment.
> > 
> > Let me know if that makes sense.
> 
> Ok. But the main thing is the master process goes to recursion restart
> of workers if they return exit or exception.May be add the counter?
> For example, 10 times we trying to respawn working process and after
> stop master.  I think it makes sense. 

It'd have to be a time-aware counter, because sometimes slightly broken
_will_ exit/die 10 times over the period of an hour.  But the site can
still be making enough money with 99.8% successful service and not
care about spending time/resources fixing the last 0.2% :)

But even with a proper time-aware counter, the app is still broken at
deploy time.  So I don't believe there is much point in adding more code
just to exit the process when the proper solution is for the user to
_fix_ the app.

And again (I seem to remember saying this months ago), any time you're
deploying, you should be paying extra attention to the app anyways.
That means testing with actual HTTP requests, not just seeing of the
process is up.  Just having a running process serving error pages is
equally worthless as not running the server at all.

-- 
Eric Wong

From alex at simonov.me  Thu Jun 10 04:46:57 2010
From: alex at simonov.me (Alexander Simonov)
Date: Thu, 10 Jun 2010 11:46:57 +0300
Subject: [ANN] unicorn 0.990.0 - inching towards 1.0
In-Reply-To: <20100610073850.GA28455@dcvr.yhbt.net>
References: <20100608095140.GB30419@dcvr.yhbt.net>
	<E4FB514D-0056-4D59-8DD3-2B8A33CACD56@simonov.me>
	<20100609182245.GB8027@dcvr.yhbt.net>
	<EFCFB26E-2D97-4DB1-AD50-141D224D9B4A@simonov.me>
	<20100610073850.GA28455@dcvr.yhbt.net>
Message-ID: <7225A616-7181-4DAE-A2AA-D90FEE265F91@simonov.me>


On Jun 10, 2010, at 10:38 AM, Eric Wong wrote:

> It'd have to be a time-aware counter, because sometimes slightly broken
> _will_ exit/die 10 times over the period of an hour.  But the site can
> still be making enough money with 99.8% successful service and not
> care about spending time/resources fixing the last 0.2% :)
> 
> But even with a proper time-aware counter, the app is still broken at
> deploy time.  So I don't believe there is much point in adding more code
> just to exit the process when the proper solution is for the user to
> _fix_ the app.
> 
> And again (I seem to remember saying this months ago), any time you're
> deploying, you should be paying extra attention to the app anyways.
> That means testing with actual HTTP requests, not just seeing of the
> process is up.  Just having a running process serving error pages is
> equally worthless as not running the server at all.

Hm.. It's makes sense.
But much better if it will be writing in the documentation for the future.

Thank you!


From normalperson at yhbt.net  Thu Jun 10 05:58:12 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 10 Jun 2010 09:58:12 +0000
Subject: Working directory and config.ru
In-Reply-To: <20100609180431.GA8027@dcvr.yhbt.net>
References: <AANLkTimKb7JARr_69nfVrJLvMZH3Gvs1o_KwZFLKfuxy@mail.gmail.com>
	<20100609180431.GA8027@dcvr.yhbt.net>
Message-ID: <20100610095812.GA8546@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> Pierre Baillet <oct at fotonauts.com> wrote:
> > Hi,
> > 
> > I naively expected unicorn to honor the working_directory
> > configuration directive before attempting to locate the config.ru but
> > this does not seem to be the case. Is this behavior wanted ? From the
> > code, I can guess that the configuration file parsing is done much
> > later  than the current config.ru location attempt.
> 
> Hi Pierre,
> 
> Oops, definitely not wanted behavior.  Unicorn should honor
> working_directory with regard to config.ru.  I'll start working
> on a fix shortly, thanks for the report!

I just pushed the following out with a few other cleanups.

Maintaining Rainbows!/Zbatery compatibility was a bit costly in terms of
prettiness.

Overall, my initial hesitation to support "working_directory" last year
was validated by the complexity of maintaining relative path
compatibility across the board.

I'm very glad for the integration test suite (stolen from Rainbows!)
which allows me to write tests in my "native" language :>

There'll be more tests coming when I'm more awake (and I'm actually
developing a real Rack application which will be public Real Soon
Now(TM), but more targeted at Rainbows!/Zbatery :)

>From 4accf4449390c649bce0b1c9d84314d65fd41f8e Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Thu, 10 Jun 2010 08:47:10 +0000
Subject: [PATCH] respect "working_directory" wrt config.ru

Since we added support for the "working_directory" parameter, it
often became unclear where/when certain paths would be bound.

There are some extremely nasty dependencies and ordering issues
when doing this.  It's all pretty fragile, but works for now
and we even have a full integration test to keep it working.

I plan on cleaning this up 2.x.x to be less offensive to look
at (Rainbows! and Zbatery are a bit tied to this at the moment).

Thanks to Pierre Baillet for reporting this.

ref: http://mid.gmane.org/AANLkTimKb7JARr_69nfVrJLvMZH3Gvs1o_KwZFLKfuxy at mail.gmail.com
---
 bin/unicorn                  |    5 +---
 bin/unicorn_rails            |   17 +++++++++---
 lib/unicorn.rb               |    7 ++---
 lib/unicorn/configurator.rb  |   56 ++++++++++++++++++++++++++++++++++++++++++
 lib/unicorn/launcher.rb      |    5 ++-
 t/t0003-working_directory.sh |   53 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 129 insertions(+), 14 deletions(-)
 create mode 100755 t/t0003-working_directory.sh

diff --git a/bin/unicorn b/bin/unicorn
index beece97..2cc10b1 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -107,10 +107,7 @@ opts = OptionParser.new("", 24, '  ') do |opts|
   opts.parse! ARGV
 end
 
-ru = ARGV[0] || "config.ru"
-abort "configuration file #{ru} not found" unless File.exist?(ru)
-
-app = Unicorn.builder(ru, opts)
+app = Unicorn.builder(ARGV[0] || 'config.ru', opts)
 options[:listeners] << "#{host}:#{port}" if set_listener
 
 if $DEBUG
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 2f88bc1..e9cac00 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -107,8 +107,6 @@ opts = OptionParser.new("", 24, '  ') do |opts|
   opts.parse! ARGV
 end
 
-ru = ARGV[0] || (File.exist?('config.ru') ? 'config.ru' : nil)
-
 def rails_dispatcher
   if ::Rails::VERSION::MAJOR >= 3 && ::File.exist?('config/application.rb')
     if ::File.read('config/application.rb') =~ /^module\s+([\w:]+)\s*$/
@@ -127,9 +125,20 @@ def rails_dispatcher
   result || abort("Unable to locate the application dispatcher class")
 end
 
-def rails_builder(daemonize)
+def rails_builder(ru, opts, daemonize)
+  return Unicorn.builder(ru, opts) if ru
+
+  # allow Configurator to parse cli switches embedded in the ru file
+  Unicorn::Configurator::RACKUP.update(:file => :rails, :optparse => opts)
+
   # this lambda won't run until after forking if preload_app is false
+  # this runs after config file reloading
   lambda do ||
+    # Rails 3 includes a config.ru, use it if we find it after
+    # working_directory is bound.
+    ::File.exist?('config.ru') and
+      return Unicorn.builder('config.ru', opts).call
+
     # Load Rails and (possibly) the private version of Rack it bundles.
     begin
       require ::File.expand_path('config/boot')
@@ -179,7 +188,7 @@ def rails_builder(daemonize)
   end
 end
 
-app = ru ? Unicorn.builder(ru, opts) : rails_builder(daemonize)
+app = rails_builder(ARGV[0], opts, daemonize)
 options[:listeners] << "#{host}:#{port}" if set_listener
 
 if $DEBUG
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index c026236..a7b0646 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -33,11 +33,10 @@ module Unicorn
     # Unicorn config).  The returned lambda will be called when it is
     # time to build the app.
     def builder(ru, opts)
-      if ru =~ /\.ru\z/
-        # parse embedded command-line options in config.ru comments
-        /^#\\(.*)/ =~ File.read(ru) and opts.parse!($1.split(/\s+/))
-      end
+      # allow Configurator to parse cli switches embedded in the ru file
+      Unicorn::Configurator::RACKUP.update(:file => ru, :optparse => opts)
 
+      # always called after config file parsing, may be called after forking
       lambda do ||
         inner_app = case ru
         when /\.ru$/
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index e4305c2..0716e64 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -13,6 +13,12 @@ module Unicorn
   # nginx is also available at
   # http://unicorn.bogomips.org/examples/nginx.conf
   class Configurator < Struct.new(:set, :config_file, :after_reload)
+    # :stopdoc:
+    # used to stash stuff for deferred processing of cli options in
+    # config.ru after "working_directory" is bound.  Do not rely on
+    # this being around later on...
+    RACKUP = {}
+    # :startdoc:
 
     # Default settings for Unicorn
     DEFAULTS = {
@@ -51,6 +57,8 @@ module Unicorn
     def reload #:nodoc:
       instance_eval(File.read(config_file), config_file) if config_file
 
+      parse_rackup_file
+
       # working_directory binds immediately (easier error checking that way),
       # now ensure any paths we changed are correctly set.
       [ :pid, :stderr_path, :stdout_path ].each do |var|
@@ -66,6 +74,9 @@ module Unicorn
 
     def commit!(server, options = {}) #:nodoc:
       skip = options[:skip] || []
+      if ready_pipe = RACKUP.delete(:ready_pipe)
+        server.ready_pipe = ready_pipe
+      end
       set.each do |key, value|
         value == :unset and next
         skip.include?(key) and next
@@ -411,5 +422,50 @@ module Unicorn
       set[var] = my_proc
     end
 
+    # this is called _after_ working_directory is bound.  This only
+    # parses the embedded switches in .ru files
+    # (for "rackup" compatibility)
+    def parse_rackup_file # :nodoc:
+      ru = RACKUP[:file] or return # we only return here in unit tests
+
+      # :rails means use (old) Rails autodetect
+      if ru == :rails
+        File.readable?('config.ru') or return
+        ru = 'config.ru'
+      end
+
+      File.readable?(ru) or
+        raise ArgumentError, "rackup file (#{ru}) not readable"
+
+      # it could be a .rb file, too, we don't parse those manually
+      ru =~ /\.ru\z/ or return
+
+      /^#\\(.*)/ =~ File.read(ru) or return
+      warn "ru cli opts: #{$1}"
+      RACKUP[:optparse].parse!($1.split(/\s+/))
+
+      # XXX ugly as hell, WILL FIX in 2.x (along with Rainbows!/Zbatery)
+      host, port, set_listener, options, daemonize =
+                      eval("[ host, port, set_listener, options, daemonize ]",
+                           TOPLEVEL_BINDING)
+
+      warn [ :host, :port, :set_listener, :options, :daemonize ].inspect
+      warn [ ru, host, port, set_listener, options, daemonize ].inspect
+      # XXX duplicate code from bin/unicorn{,_rails}
+      set[:listeners] << "#{host}:#{port}" if set_listener
+
+      if daemonize
+        # unicorn_rails wants a default pid path, (not plain 'unicorn')
+        if ru == :rails
+          spid = set[:pid]
+          pid('tmp/pids/unicorn.pid') if spid.nil? || spid == :unset
+        end
+        unless RACKUP[:daemonized]
+          Unicorn::Launcher.daemonize!(options)
+          RACKUP[:ready_pipe] = options.delete(:ready_pipe)
+        end
+      end
+    end
+
   end
 end
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 5ab04c7..7f3ffa6 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -56,8 +56,9 @@ class Unicorn::Launcher
       end
     end
     # $stderr/$stderr can/will be redirected separately in the Unicorn config
-    Unicorn::Configurator::DEFAULTS[:stderr_path] = "/dev/null"
-    Unicorn::Configurator::DEFAULTS[:stdout_path] = "/dev/null"
+    Unicorn::Configurator::DEFAULTS[:stderr_path] ||= "/dev/null"
+    Unicorn::Configurator::DEFAULTS[:stdout_path] ||= "/dev/null"
+    Unicorn::Configurator::RACKUP[:daemonized] = true
   end
 
 end
diff --git a/t/t0003-working_directory.sh b/t/t0003-working_directory.sh
new file mode 100755
index 0000000..3faa6c0
--- /dev/null
+++ b/t/t0003-working_directory.sh
@@ -0,0 +1,53 @@
+#!/bin/sh
+. ./test-lib.sh
+t_plan 4 "config.ru inside alt working_directory"
+
+t_begin "setup and start" && {
+	unicorn_setup
+	rtmpfiles unicorn_config_tmp
+	rm -rf $t_pfx.app
+	mkdir $t_pfx.app
+
+	port=$(expr $listen : '[^:]*:\([0-9]\+\)')
+	host=$(expr $listen : '\([^:]*\):[0-9]\+')
+
+	cat > $t_pfx.app/config.ru <<EOF
+#\--daemonize --host $host --port $port
+use Rack::ContentLength
+use Rack::ContentType, "text/plain"
+run lambda { |env| [ 200, {}, [ "#{\$master_ppid}\\n" ] ] }
+EOF
+	# we have --host/--port in config.ru instead
+	grep -v ^listen $unicorn_config > $unicorn_config_tmp
+
+	# the whole point of this exercise
+	echo "working_directory '$t_pfx.app'" >> $unicorn_config_tmp
+
+	# allows ppid to be 1 in before_fork
+	echo "preload_app true" >> $unicorn_config_tmp
+	cat >> $unicorn_config_tmp <<\EOF
+before_fork do |server,worker|
+  $master_ppid = Process.ppid # should be zero to detect daemonization
+end
+EOF
+
+	mv $unicorn_config_tmp $unicorn_config
+
+	# rely on --daemonize switch, no & or -D
+	unicorn -c $unicorn_config
+	unicorn_wait_start
+}
+
+t_begin "hit with curl" && {
+	body=$(curl -sSf http://$listen/)
+}
+
+t_begin "killing succeeds" && {
+	kill $unicorn_pid
+}
+
+t_begin "response body ppid == 1 (daemonized)" && {
+	test "$body" -eq 1
+}
+
+t_done
-- 

Eric Wong (7):
      launcher: get rid of backwards compatibility code
      isolate: don't run isolate in parallel
      Rakefile: only try rake-compiler if VERSION is defined
      Rakefile: isolate to rbx directory
      tests: set NO_PROXY when running tests
      unicorn_rails: use Kernel#warn instead of $stderr.puts
      respect "working_directory" wrt config.ru

-- 
Eric Wong

From normalperson at yhbt.net  Thu Jun 10 21:27:00 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 10 Jun 2010 18:27:00 -0700
Subject: [ANN] unicorn 0.990.0 - inching towards 1.0
In-Reply-To: <7225A616-7181-4DAE-A2AA-D90FEE265F91@simonov.me>
References: <20100608095140.GB30419@dcvr.yhbt.net>
	<E4FB514D-0056-4D59-8DD3-2B8A33CACD56@simonov.me>
	<20100609182245.GB8027@dcvr.yhbt.net>
	<EFCFB26E-2D97-4DB1-AD50-141D224D9B4A@simonov.me>
	<20100610073850.GA28455@dcvr.yhbt.net>
	<7225A616-7181-4DAE-A2AA-D90FEE265F91@simonov.me>
Message-ID: <20100611012700.GA25953@dcvr.yhbt.net>

Alexander Simonov <alex at simonov.me> wrote:
> On Jun 10, 2010, at 10:38 AM, Eric Wong wrote:
> > And again (I seem to remember saying this months ago), any time you're
> > deploying, you should be paying extra attention to the app anyways.
> > That means testing with actual HTTP requests, not just seeing of the
> > process is up.  Just having a running process serving error pages is
> > equally worthless as not running the server at all.
> 
> Hm.. It's makes sense.
> But much better if it will be writing in the documentation for the future.

Here's what I've pushed out, let me know if everything makes
sense.  I think I'll release 0.991.0 in a little bit.

>From f9baab18705218dae4c37c2c92d1ceea151bba8e Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Thu, 10 Jun 2010 17:57:08 -0700
Subject: [PATCH] docs: hopefully clarify preload_app=false behavior

While we're at it, inform people of why they might use
a symlink
---
 SIGNALS                     |    7 ++++++-
 lib/unicorn/configurator.rb |   21 ++++++++++++++++++---
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/SIGNALS b/SIGNALS
index be96892..8851775 100644
--- a/SIGNALS
+++ b/SIGNALS
@@ -14,7 +14,12 @@ Unicorn and nginx.
   will also pick up any application code changes when restarted.  If
   "preload_app" is true, then application code changes will have no
   effect; USR2 + QUIT (see below) must be used to load newer code in
-  this case.
+  this case.  When reloading the application, +Gem.refresh+ will
+  be called so updated code for your application can pick up newly
+  installed RubyGems.  It is not recommended that you uninstall
+  libraries your application depends on while Unicorn is running,
+  as respawned workers may enter a spawn loop when they fail to
+  load an uninstalled dependency.
 
 * INT/TERM - quick shutdown, kills all workers immediately
 
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 71c7a8a..533e0ed 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -311,7 +311,22 @@ module Unicorn
     #
     # In addition to reloading the unicorn-specific config settings,
     # SIGHUP will reload application code in the working
-    # directory/symlink when workers are gracefully restarted.
+    # directory/symlink when workers are gracefully restarted when
+    # preload_app=false (the default).  As reloading the application
+    # sometimes requires RubyGems updates, +Gem.refresh+ is always
+    # called before the application is loaded (for RubyGems users).
+    #
+    # During deployments, care should _always_ be taken to ensure your
+    # applications are properly deployed and running.  Using
+    # preload_app=false (the default) means you _must_ check if
+    # your application is responding properly after a deployment.
+    # Improperly deployed applications can go into a spawn loop
+    # if the application fails to load.  While your children are
+    # in a spawn loop, it is is possible to fix an application
+    # by properly deploying all required code and dependencies.
+    # Using preload_app=true means any application load error will
+    # cause the master process to exit with an error.
+
     def preload_app(bool)
       case bool
       when TrueClass, FalseClass
@@ -344,9 +359,9 @@ module Unicorn
       set_path(:stdout_path, path)
     end
 
-    # sets the working directory for Unicorn.  This ensures USR2 will
+    # sets the working directory for Unicorn.  This ensures SIGUSR2 will
     # start a new instance of Unicorn in this directory.  This may be
-    # a symlink.
+    # a symlink, a common scenario for Capistrano users.
     def working_directory(path)
       # just let chdir raise errors
       path = File.expand_path(path)
-- 
Eric Wong

From normalperson at yhbt.net  Thu Jun 10 21:56:21 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 10 Jun 2010 18:56:21 -0700
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <AANLkTimubIMS1FrgGT7gCneO-7N250eRJxQjs4LACAFl@mail.gmail.com>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net>
	<AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>
	<20100603184730.GA2421@dcvr.yhbt.net>
	<AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>
	<AANLkTimV05UjKFPJW7xFW7OTALIrQH9Qr9I2u4LBYTwe@mail.gmail.com>
	<AANLkTimubIMS1FrgGT7gCneO-7N250eRJxQjs4LACAFl@mail.gmail.com>
Message-ID: <20100611015621.GA31868@dcvr.yhbt.net>

Pierre Baillet <oct at fotonauts.com> wrote:
> Hello Unicorns,
> 
> I've manage to create a simple middleware that replaces the soft
> timeout feature. You can have a look at it at
> http://gist.github.com/431451
> 
> Note that some weird Ruby interpreter behavior breaks at least the
> first level of the generated stacktrace (it indicates the actual
> method where the raise happened but the wrong line number and file).

You should be able to avoid spawning a new thread for every request, as
that can get very expensive on some systems.  Maybe a global hash
guarded by a mutex that tells the worker Thread which thread to raise
on.  The worker thread could sleep until the next timeout registered.

But then again, take care to only spawn new threads in workers (with
preload_app=true), as threads never get carried across fork.  And
threads may leave mutexes and such in a bad state when they vanish
in the child.

-- 
Eric Wong

From normalperson at yhbt.net  Thu Jun 10 22:31:34 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 11 Jun 2010 02:31:34 +0000
Subject: [ANN] unicorn 0.991.0 - startup improvements
Message-ID: <20100611023134.GA27137@dcvr.yhbt.net>

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

The "working_directory" configuration parameter is now handled
before config.ru.  That means "unicorn" and "unicorn_rails" no
longer barfs when initially started outside of the configured
"working_directory" where a config.ru is required.  A huge
thanks to Pierre Baillet for catching this ugly UI inconsistency
before the big 1.0 release

Thanks to Hongli Lai, out-of-the-box Rails 3 (beta) support
should be improved for deployments lacking a config.ru

There are more new integration tests, cleanups and some
documentation improvements.

-- 
Eric Wong

From alex at simonov.me  Fri Jun 11 14:00:58 2010
From: alex at simonov.me (Alexander Simonov)
Date: Fri, 11 Jun 2010 21:00:58 +0300
Subject: [ANN] unicorn 0.990.0 - inching towards 1.0
In-Reply-To: <20100611012700.GA25953@dcvr.yhbt.net>
References: <20100608095140.GB30419@dcvr.yhbt.net>
	<E4FB514D-0056-4D59-8DD3-2B8A33CACD56@simonov.me>
	<20100609182245.GB8027@dcvr.yhbt.net>
	<EFCFB26E-2D97-4DB1-AD50-141D224D9B4A@simonov.me>
	<20100610073850.GA28455@dcvr.yhbt.net>
	<7225A616-7181-4DAE-A2AA-D90FEE265F91@simonov.me>
	<20100611012700.GA25953@dcvr.yhbt.net>
Message-ID: <82E61B65-6589-4134-B1B8-941DF8FAF91C@simonov.me>



On Jun 11, 2010, at 4:27 AM, Eric Wong wrote:

> Alexander Simonov <alex at simonov.me> wrote:
>> On Jun 10, 2010, at 10:38 AM, Eric Wong wrote:
>>> And again (I seem to remember saying this months ago), any time you're
>>> deploying, you should be paying extra attention to the app anyways.
>>> That means testing with actual HTTP requests, not just seeing of the
>>> process is up.  Just having a running process serving error pages is
>>> equally worthless as not running the server at all.
>> 
>> Hm.. It's makes sense.
>> But much better if it will be writing in the documentation for the future.
> 
> Here's what I've pushed out, let me know if everything makes
> sense.  I think I'll release 0.991.0 in a little bit.

Ok! Thank you!
Good. After release I will update golden_brindle too :)


> 
>> From f9baab18705218dae4c37c2c92d1ceea151bba8e Mon Sep 17 00:00:00 2001
> From: Eric Wong <normalperson at yhbt.net>
> Date: Thu, 10 Jun 2010 17:57:08 -0700
> Subject: [PATCH] docs: hopefully clarify preload_app=false behavior
> 
> While we're at it, inform people of why they might use
> a symlink


From normalperson at yhbt.net  Fri Jun 11 16:32:09 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 11 Jun 2010 20:32:09 +0000
Subject: unicorn_rails cleanup (possible fix for Rails3) pushed
In-Reply-To: <20100608192034.GA2940@dcvr.yhbt.net>
References: <AANLkTiksBxIo_PFWoiPTWi1entXZRb7D2uE-Rl7H3lbw@mail.gmail.com>
	<20100608192034.GA2940@dcvr.yhbt.net>
Message-ID: <20100611203208.GA6165@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> Hongli Lai <hongli at phusion.nl> wrote:
> > Hi Eric.
> > 
> > It would appear that recent Rails 3 changes have broken unicorn_rails,
> > just like they broke Phusion Passenger. Here's a patch which fixes the
> > problem.
> > http://gist.github.com/429944
> 
> Thanks Hongli, so this only affects people who remove the
> config.ru that Rails 3 creates for them?  Yikes...

And for the completely bizzare: Rails 3 is ultra aggressive
when searching for config.ru, it walks up the directory tree,
even going all the way up to "/" if it can't find config.ru.

I had a $HOME/config.ru leftover from a test on one of my boxes, and
usually have my working directory in $HOME/unicorn.   This was was
causing the unicorn_rails test for this (t0301) to fail because that
test relies on Rails /not/ being able to find config.ru.

So yes, don't leave random artifacts lying around and love strace :)

-- 
Eric Wong

From normalperson at yhbt.net  Fri Jun 11 16:37:07 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 11 Jun 2010 20:37:07 +0000
Subject: [ANN] unicorn 0.990.0 - inching towards 1.0
In-Reply-To: <20100608095140.GB30419@dcvr.yhbt.net>
References: <20100608095140.GB30419@dcvr.yhbt.net>
Message-ID: <20100611203707.GA7262@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> Rubinius updates:
>   but things should generally work unless you
>   need USR2 upgrades.  Feedback and reports would be greatly
>   appreciated as usual.

Log rotation for $stderr and $stdout (but not other
paths) are broken under Rubinius, too.  I only discovered this
while running the Rainbows! integration tests.

ref: http://github.com/evanphx/rubinius/issues/360

-- 
Eric Wong

From members.support at argos.co.uk  Fri Jun 11 22:16:34 2010
From: members.support at argos.co.uk (members.support at argos.co.uk)
Date: Sat, 12 Jun 2010 03:16:34 +0100
Subject: =?koi8-r?B?RyByIGUgYSB0ICBPIGYgZiBlIHI=?=
Message-ID: <l1iIQvFAldLLLinfIhyRuhTN0DjDja61EA0nU6hDRDq4@argos.co.uk>

H e l l o  D e a r  C u s t o m e r.
Argos online-store offer you the opportunity to become a cardholder of "Argos Gold Card".To find out more for additional info see below.

Copy following link (without quotes) `argos-getgoldcard.co.cc` into your browser and press `Enter`


From tachu at crowdstar.com  Sat Jun 12 05:30:54 2010
From: tachu at crowdstar.com (Jose Avila(Tachu))
Date: Sat, 12 Jun 2010 02:30:54 -0700
Subject: Issues since 0.991.0
Message-ID: <5C960A81-3C95-4FBB-B38C-A783A483CD53@crowdstar.com>

Hi Guys we currently have a farm of servers that run 0.990.0 and new ones that started running 0.991.0 Since 0.991.0 We get this error 

unicorn_rails worker[24] -c config/unicorn.rb -E production: symbol lookup error: /usr/lib64/ruby/gems/1.8/gems/unicorn-
0.991.0/lib/unicorn_http.so: undefined symbol: rb_str_set_len

Any idea as to why that is happening and how we can fix it?

Thanks

t


From normalperson at yhbt.net  Sat Jun 12 14:37:33 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Sat, 12 Jun 2010 18:37:33 +0000
Subject: Issues since 0.991.0
In-Reply-To: <5C960A81-3C95-4FBB-B38C-A783A483CD53@crowdstar.com>
References: <5C960A81-3C95-4FBB-B38C-A783A483CD53@crowdstar.com>
Message-ID: <20100612183733.GA1682@dcvr.yhbt.net>

"Jose Avila(Tachu)" <tachu at crowdstar.com> wrote:
> Hi Guys we currently have a farm of servers that run 0.990.0 and new ones that started running 0.991.0 Since 0.991.0 We get this error 
> 
> unicorn_rails worker[24] -c config/unicorn.rb -E production: symbol lookup error: /usr/lib64/ruby/gems/1.8/gems/unicorn-
> 0.991.0/lib/unicorn_http.so: undefined symbol: rb_str_set_len
> 
> Any idea as to why that is happening and how we can fix it?

Hi Jose,  I was too aggressive with a cleanup for Rubinius compatibility
that I broke things for 1.8.6 (and possibly before).

I've pushed out the following fix and also a 0.991.0.4.g0cb0 prerelease
to RubyGems.org.  You can install it with "gem install --pre unicorn"

Let me know how it works for you, thanks for the report!

>From 0cb02efd0e2e2c80667863fd404d1fad7c63bb1f Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Sat, 12 Jun 2010 18:32:12 +0000
Subject: [PATCH] http: fix rb_str_set_len() define for 1.8.6

---
 ext/unicorn_http/ext_help.h |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/ext/unicorn_http/ext_help.h b/ext/unicorn_http/ext_help.h
index cc157cb..3aa24a8 100644
--- a/ext/unicorn_http/ext_help.h
+++ b/ext/unicorn_http/ext_help.h
@@ -22,6 +22,7 @@ static void rb_18_str_set_len(VALUE str, long len)
   RSTRING(str)->len = len;
   RSTRING(str)->ptr[len] = '\0';
 }
+#  define rb_str_set_len(str,len) rb_18_str_set_len(str,len)
 #endif /* !defined(HAVE_RB_STR_SET_LEN) */
 
 /* not all Ruby implementations support frozen objects (Rubinius does not) */
-- 
Eric Wong

From normalperson at yhbt.net  Mon Jun 14 15:46:25 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 14 Jun 2010 12:46:25 -0700
Subject: Unicorn future plans
Message-ID: <20100614194625.GA4089@dcvr.yhbt.net>

Hi all,

Some of you are wondering about the future of the project, especially
since we're nearing a 1.0 release.

== 1.x - bugfixes and Rack-compatibility synchronization

The 1.x series will focus mainly on bug fixes and compatibility with
Rack as it evolves.

If Rack drops the rewindability requirement of "rack.input", Unicorn
will follow ASAP and allow TeeInput to be optional middleware with
newer Rack versions.

Rubinius should be fully-supported soon, as it's already mostly working
except for a few corner-case things Rubinius doesn't implement (issues
filed on their bug tracker).

== 2.x - the fun and crazy stuff

First off, there'll be internal API cleanups + sync with
Rainbows!/Zbatery This won't be user visible, but it'll be less ugly
inside.

=== Scalability improvements

I don't know if we'll see hundreds/thousands of CPUs in a single
application server any time soon, but 2.x will have internal changes
that'll make us more ready for that.  It could be 5-10 years before
massively multi-core machines are viable for web apps, but it'd
certainly be fun to max out Unicorn on those.

I'll be less shy about sacrificing portability for massive scalability.
Correct me if I'm wrong, but Linux is the only Free kernel that scales
to monster multicore machines right now, so I'll primarily focus on
working with features in Linux.

Currently, 8-16 cores seems to be the sweet spot (and has been for a
while), and present-day Unicorn handles that fine as-is.

=== Features (for Rainbows!, mainly)

IPv6 support and SSL will come, too.   These features will mainly be to
support Rainbows!, though some LANs could benefit from those, too.  I'll
have to review the MRI code for those, but I'm leaning towards only
these new features under 1.9.2+.

Multi-VM (MVM, Ruby 1.9 experimental branch) support will probably
happen only in Rainbows! rather than Unicorn.  A true fork() is still
safer in the event of a crash (but less memory-efficient).

== Miscellaneous

I have no plans to provide commercial support, but I'll continue
offering free support on the mailing list (or private email if you're
shy).

As some of you may have noticed; I don't endorse commercial products or
services at all.  This won't change.  However, there will probably be
more commercial support available from 3rd parties after a 1.0 release.


Thanks for reading!

-- 
Eric Wong

From hukl at berlin.ccc.de  Mon Jun 14 16:58:02 2010
From: hukl at berlin.ccc.de (John-Paul Bader)
Date: Mon, 14 Jun 2010 22:58:02 +0200
Subject: Unicorn future plans
In-Reply-To: <20100614194625.GA4089@dcvr.yhbt.net>
References: <20100614194625.GA4089@dcvr.yhbt.net>
Message-ID: <4B3E0C97-2A2C-4CC8-A802-57D01F8134A3@berlin.ccc.de>

Hey, 

your plans sound great except for ignoring FreeBSD ;)

Seriously many many Servers, including many which I administrate run on FreeBSD and they do so very well. I'm not very familiar with the internals of the FreeBSD Kernel but I can assure you that it runs perfectly on 8 and 16 core machines and I would extremely happy if I could continue to use unicorn on them.

It should be easier to maintain a FreeBSD version as they (FreeBSD developers) tend to aim for more consistency across releases than Linux. You said once that you don't run any BSD machines but I'd be happy to offer you access to one of my BSD servers for testing.

Furthermore since Mac OS X and FreeBSD share many features like kqueue and many Ruby developers are running macs it would make even more sense to at least care about BSD even if its not you primary platform. 

I don't want to start a flame war or say that one OS is better than another. Linux is certainly great but so is FreeBSD. For the SMP part I recommend the following page on freebsd.org:

http://www.freebsd.org/smp/

And maybe specifically:

http://www.freebsd.org/smp/#resources

To conclude I can only say that I'm running unicorn on several FreeBSD hosts and so far I couldn't be happier with it. As stated before, I'd be more than happy to continue using that setup.

Kindest regards and thanks for all you efforts,

John


On 14.06.2010, at 21:46, Eric Wong wrote:

> Hi all,
> 
> Some of you are wondering about the future of the project, especially
> since we're nearing a 1.0 release.
> 
> == 1.x - bugfixes and Rack-compatibility synchronization
> 
> The 1.x series will focus mainly on bug fixes and compatibility with
> Rack as it evolves.
> 
> If Rack drops the rewindability requirement of "rack.input", Unicorn
> will follow ASAP and allow TeeInput to be optional middleware with
> newer Rack versions.
> 
> Rubinius should be fully-supported soon, as it's already mostly working
> except for a few corner-case things Rubinius doesn't implement (issues
> filed on their bug tracker).
> 
> == 2.x - the fun and crazy stuff
> 
> First off, there'll be internal API cleanups + sync with
> Rainbows!/Zbatery This won't be user visible, but it'll be less ugly
> inside.
> 
> === Scalability improvements
> 
> I don't know if we'll see hundreds/thousands of CPUs in a single
> application server any time soon, but 2.x will have internal changes
> that'll make us more ready for that.  It could be 5-10 years before
> massively multi-core machines are viable for web apps, but it'd
> certainly be fun to max out Unicorn on those.
> 
> I'll be less shy about sacrificing portability for massive scalability.
> Correct me if I'm wrong, but Linux is the only Free kernel that scales
> to monster multicore machines right now, so I'll primarily focus on
> working with features in Linux.
> 
> Currently, 8-16 cores seems to be the sweet spot (and has been for a
> while), and present-day Unicorn handles that fine as-is.
> 
> === Features (for Rainbows!, mainly)
> 
> IPv6 support and SSL will come, too.   These features will mainly be to
> support Rainbows!, though some LANs could benefit from those, too.  I'll
> have to review the MRI code for those, but I'm leaning towards only
> these new features under 1.9.2+.
> 
> Multi-VM (MVM, Ruby 1.9 experimental branch) support will probably
> happen only in Rainbows! rather than Unicorn.  A true fork() is still
> safer in the event of a crash (but less memory-efficient).
> 
> == Miscellaneous
> 
> I have no plans to provide commercial support, but I'll continue
> offering free support on the mailing list (or private email if you're
> shy).
> 
> As some of you may have noticed; I don't endorse commercial products or
> services at all.  This won't change.  However, there will probably be
> more commercial support available from 3rd parties after a 1.0 release.
> 
> 
> Thanks for reading!
> 
> -- 
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
> 


From normalperson at yhbt.net  Mon Jun 14 18:10:36 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 14 Jun 2010 15:10:36 -0700
Subject: Unicorn future plans
In-Reply-To: <4B3E0C97-2A2C-4CC8-A802-57D01F8134A3@berlin.ccc.de>
References: <20100614194625.GA4089@dcvr.yhbt.net>
	<4B3E0C97-2A2C-4CC8-A802-57D01F8134A3@berlin.ccc.de>
Message-ID: <20100614221036.GB18382@dcvr.yhbt.net>

John-Paul Bader <hukl at berlin.ccc.de> wrote:
> Hey, 
> 
> your plans sound great except for ignoring FreeBSD ;)
> 
> Seriously many many Servers, including many which I administrate run
> on FreeBSD and they do so very well. I'm not very familiar with the
> internals of the FreeBSD Kernel but I can assure you that it runs
> perfectly on 8 and 16 core machines and I would extremely happy if I
> could continue to use unicorn on them.

Hi John-Paul,

Reread my post carefully, not much is changing for 8-16 core users.
FreeBSD and SMP will continue to be supported.  I wasn't referring
to SMP at all when I was talking about Linux-only bits.

> It should be easier to maintain a FreeBSD version as they (FreeBSD
> developers) tend to aim for more consistency across releases than
> Linux. You said once that you don't run any BSD machines but I'd be
> happy to offer you access to one of my BSD servers for testing.

Thanks for the offer.  I'll keep it in mind if I need it again.

> Furthermore since Mac OS X and FreeBSD share many features like kqueue
> and many Ruby developers are running macs it would make even more
> sense to at least care about BSD even if its not you primary platform. 

I know, I've fixed some things on OSX platforms via FreeBSD (OSX scares
me with its GUI-ness).  Keep in mind kqueue (and epoll) are worthless
for Unicorn itself since Unicorn only handles one client at a time.

However, Rainbows! can already use kqueue/epoll with EventMachine and
Rev.

> I don't want to start a flame war or say that one OS is better than
> another. Linux is certainly great but so is FreeBSD. For the SMP part
> I recommend the following page on freebsd.org:

Again, I wasn't talking about SMP at all.   SMP works fine on current
Unicorn and mid-sized servers.  When going to more cores, SMP itself
is a bottleneck, not the kernel.

With Unicorn 2.x, I'm targeting NUMA hardware that isn't available in
commodity servers yet.  Currently, NUMA makes _zero_ sense in web
application servers, but in case things change in a few years, we'll be
ready.

It may be a chicken-and-egg problem, too.  Hardware manufacturers are
slow to commoditize NUMA because a good chunk of software can't even
utilize SMP effectively :)

> To conclude I can only say that I'm running unicorn on several FreeBSD
> hosts and so far I couldn't be happier with it. As stated before, I'd
> be more than happy to continue using that setup.

Again, don't worry :)  You'll be able to continue using everything
as-is.

> Kindest regards and thanks for all you efforts,

No problem.  Please don't top post in the future.

-- 
Eric Wong

From hukl at berlin.ccc.de  Tue Jun 15 03:09:00 2010
From: hukl at berlin.ccc.de (John-Paul Bader)
Date: Tue, 15 Jun 2010 09:09:00 +0200
Subject: Unicorn future plans
In-Reply-To: <20100614221036.GB18382@dcvr.yhbt.net>
References: <20100614194625.GA4089@dcvr.yhbt.net>
	<4B3E0C97-2A2C-4CC8-A802-57D01F8134A3@berlin.ccc.de>
	<20100614221036.GB18382@dcvr.yhbt.net>
Message-ID: <9433B89A-0320-419A-9A06-87B3CAB83D96@berlin.ccc.de>


On 15.06.2010, at 00:10, Eric Wong wrote:

> John-Paul Bader <hukl at berlin.ccc.de> wrote:
>> Hey, 
>> 
>> your plans sound great except for ignoring FreeBSD ;)
>> 
>> Seriously many many Servers, including many which I administrate run
>> on FreeBSD and they do so very well. I'm not very familiar with the
>> internals of the FreeBSD Kernel but I can assure you that it runs
>> perfectly on 8 and 16 core machines and I would extremely happy if I
>> could continue to use unicorn on them.
> 
> Hi John-Paul,
> 
> Reread my post carefully, not much is changing for 8-16 core users.
> FreeBSD and SMP will continue to be supported.  I wasn't referring
> to SMP at all when I was talking about Linux-only bits.


Hey again,


sorry if I misinterpreted your post. My mail could've been shorter by saying that there are FreeBSD supercomputer clusters and that if we get 1000 cores or more FreeBSD will be fast to adapt so don't let it slip out of focus ;)

My offer for a FreeBSD machine account stands and I'd be more than happy to report issues as they occur to help you out if BSD issues occur.

Thanks again,

John

From nearrah2009 at gmail.com  Tue Jun 15 02:06:31 2010
From: nearrah2009 at gmail.com (NEARRA HOUPHOUET)
Date: Mon, 14 Jun 2010 23:06:31 -0700
Subject: From Miss Nearra Houphouet.
Message-ID: <mailman.0.1276586096.23073.mongrel-unicorn@rubyforge.org>

>From Miss Nearra Houphouet.

Dearest One

I am writing this letter with due respect and heart full of tears since we have not known or met ourselves previously I am asking for your assistance after I have gone through a profile that speaks well of you. I want to find out if it's possible for you to deal with individual as to investment. I came across your profile and I feel it's highly reputable that is why I pick an interest getting across to you in respect of investment at my disposal.  I will be so glad if you can allow me and lead me to the right channel towards your assistance to my situation now. I will make my proposal well known if I am given the opportunity.


I would like to use this opportunity to introduce myself to you. I am 21Years Old Girl from  (COTE D'IVOIRE WEST AFRICA) ,the only Daughter of Houphouet, My father is now late, he was a well known cocoa merchant business man in my country (cote d ivory)  he was  poisoned by his co-business partner. The main reason why I am contacting you now is to seek your assistance in the area of my future investment and also for a help hand over some huge amount of money in my possession.
This fund (US5.7 Million dollars) was deposited in a private security company outside my country in (REPUBLIC OF GHANA) a year ago by my father he made me the sole beneficiary. I am now asking you to stand on my behalf, to stand as my partner and in time of the claim and investment as well.


I have made up my mind to offer you 15%of the total money while the remaining will go into a productive investment. Pleas attach your direct and full information as you reply to me. Please email me with my private email:  nearrah2009 at gmail.com for more details.

Thanks and remain bless.
Best Regards
Nearra Houphouet.


From chewbranca at gmail.com  Tue Jun 15 13:55:54 2010
From: chewbranca at gmail.com (Russell Branca)
Date: Tue, 15 Jun 2010 10:55:54 -0700
Subject: Forking off the unicorn master process to create a background 
	worker
In-Reply-To: <20100526210542.GC24211@dcvr.yhbt.net>
References: <AANLkTinpzdLFf1bWBMRqDepIYQBKeZWb_mC0n-nlPhzD@mail.gmail.com>
	<20100526210542.GC24211@dcvr.yhbt.net>
Message-ID: <AANLkTin3HAypERjYJ9NhC-BnVVmRjEGGEHXIECs2fSfc@mail.gmail.com>

Hello Eric,


Sorry for the delayed response, with the combination of being sick and
heading out of town for a while, this project got put on the
backburner. I really appreciate your response and think its a clean
solution for what I'm trying to do. I've started back in getting the
job queue working this week, and will hopefully have a working
solution in the next day or two. A little more information about what
I'm doing, I'm trying to create a centralized resque job queue server
that each of the different applications can queue work into, so I'll
be using redis behind resque for storing jobs and what not, which
brings me an area I'm not sure of the best approach on. So when we hit
the job queue endpoint in the rack app, it spawns the new worker, and
then immediately returns the 200 ok started background job message,
which cuts off communication back to the job queue. My plan is to save
a status message of the result of the background task into redis, and
have resque check that to verify the task was successful. Is there a
better approach for returning the resulting status code with unicorn,
or is this a reasonable approach? Thanks again for your help.


-Russell

On Wed, May 26, 2010 at 2:05 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Russell Branca <chewbranca at gmail.com> wrote:
>> Hello,
>>
>> I'm trying to find an efficient way to create a new instance of a
>> rails application to perform some background tasks without having to
>> load up the entire rails stack every time, so I figured forking off
>> the master process would be a good way to go. Now I can easily just
>> increment the worker count and then send a web request in, but the new
>> worker would be part of the main worker pool, so in the time between
>> spawning a new worker and sending the request, another request could
>> have come in and snagged that worker. Is it possible to create a new
>> worker and not have it enter the main worker pool so I could access it
>> directly?
>
> Hi Russell,
>
> You could try having an endpoint in your webapp (with authentication, or
> have it reject env['REMOTE_ADDR'] != '127.0.0.1') that runs the
> background task for you. ?Since it's a background app, you should
> probably fork + Process.setsid + fork (or Process.daemon in 1.9), and
> return an HTTP response immediately so your app can serve other
> requests.
>
> The following example should be enough to get you started (totally
> untested)
>
> ------------ config.ru -------------
> require 'rack/lobster'
>
> map "/.seekrit_endpoint" do
> ?use Rack::ContentLength
> ?use Rack::ContentType, 'text/plain'
> ?run(lambda { |env|
> ? ?return [ 403, {}, [] ] if env['REMOTE_ADDR'] != '127.0.0.1'
> ? ?pid = fork
> ? ?if pid
> ? ? ?Process.waitpid(pid)
>
> ? ? ?# cheap way to avoid unintentional fd sharing with our children,
> ? ? ?# this causes the current Unicorn worker to exit after sending
> ? ? ?# the response:
> ? ? ?# Otherwise you'd have to be careful to disconnect+reconnect
> ? ? ?# databases/memcached/redis/whatever (in both the parent and
> ? ? ?# child) to avoid unintentional sharing that'll lead to headaches
> ? ? ?Process.kill(:QUIT, $$)
>
> ? ? ?[ 200, {}, [ "started background process\n" ] ]
> ? ?else
> ? ? ?# child, daemonize it so the unicorn master won't need to
> ? ? ?# reap it (that's the job of init)
> ? ? ?Process.setsid
> ? ? ?exit if fork
>
> ? ? ?begin
> ? ? ? ?# run your background code here instead of sleeping
> ? ? ? ?sleep 5
> ? ? ? ?env["rack.logger"].info "done sleeping"
> ? ? ?rescue => e
> ? ? ? ?env["rack.logger"].error(e.inspect)
> ? ? ?end
> ? ? ?# make sure we don't enter the normal response cycle back in the
> ? ? ?# worker...
> ? ? ?exit!(0)
> ? ?end
> ?})
> end
>
> map "/" do
> ?run Rack::Lobster.new
> end
>
>> I know this is not your typical use case for unicorn, and you're
>> probably thinking there is a lot better ways to do this, however, I
>> currently have a rails framework that powers a handful of standalone
>> applications on a server with limited resources, and I'm trying to
>> make a centralized queue that all the applications use, so the queue
>> needs to be able to spawn a new worker for each of the applications
>> efficiently, and incrementing/decrementing worker counts in unicorn is
>> the most efficient way I've found to spawn a new rails instance.
>
> Yeah, it's definitely an odd case and there are ways to shoot yourself
> in the foot with it (especially with unintentional fd sharing), but Ruby
> exposes all the Unix process management goodies better than most
> languages (probably better than anything else I've used).
>
>> Any help, suggestions or insight into this would be greatly appreciated.
>
> Let us know how it goes :)
>
> --
> Eric Wong
>

From normalperson at yhbt.net  Tue Jun 15 18:14:26 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 15 Jun 2010 15:14:26 -0700
Subject: Forking off the unicorn master process to create a background
	worker
In-Reply-To: <AANLkTin3HAypERjYJ9NhC-BnVVmRjEGGEHXIECs2fSfc@mail.gmail.com>
References: <AANLkTinpzdLFf1bWBMRqDepIYQBKeZWb_mC0n-nlPhzD@mail.gmail.com>
	<20100526210542.GC24211@dcvr.yhbt.net>
	<AANLkTin3HAypERjYJ9NhC-BnVVmRjEGGEHXIECs2fSfc@mail.gmail.com>
Message-ID: <20100615221426.GA24854@dcvr.yhbt.net>

Russell Branca <chewbranca at gmail.com> wrote:
> Hello Eric,
> 
> Sorry for the delayed response, with the combination of being sick and
> heading out of town for a while, this project got put on the
> backburner. I really appreciate your response and think its a clean
> solution for what I'm trying to do. I've started back in getting the
> job queue working this week, and will hopefully have a working
> solution in the next day or two. A little more information about what
> I'm doing, I'm trying to create a centralized resque job queue server
> that each of the different applications can queue work into, so I'll
> be using redis behind resque for storing jobs and what not, which
> brings me an area I'm not sure of the best approach on. So when we hit
> the job queue endpoint in the rack app, it spawns the new worker, and
> then immediately returns the 200 ok started background job message,
> which cuts off communication back to the job queue. My plan is to save
> a status message of the result of the background task into redis, and
> have resque check that to verify the task was successful. Is there a
> better approach for returning the resulting status code with unicorn,
> or is this a reasonable approach? Thanks again for your help.

Hi Russell, please don't top post, thanks.

If you already have a queue server (and presumably a standalone app
processing the queue), I would probably forgo the background Unicorn
worker entirely.

Based on my ancient (mid-2000s) knowledge of user-facing web
applications: the application should queue the job, return 200, and have
HTML meta refresh to constantly reload the page every few seconds.

Hitting the reload endpoint would check the database (Redis in this
case) for completion, and return a new HTML page to stop the meta
refresh loop.

This means you're no longer keeping a single Unicorn worker idle and
wasting it.  Nowadays you could do it with long-polling on
Rainbows!/Thin/Zbatery, too, but long-polling is less reliable for
people switching between WiFi access points.  The meta refresh method
can be a waste of power/bandwidth on the client side if the background
job takes a long time, though.

I'm familiar at all with Resque or Redis, but I suspect other folks
on this mailing list should be able to help you flesh out the details.

-- 
Eric Wong

From chewbranca at gmail.com  Tue Jun 15 18:51:48 2010
From: chewbranca at gmail.com (Russell Branca)
Date: Tue, 15 Jun 2010 15:51:48 -0700
Subject: Forking off the unicorn master process to create a background 
	worker
In-Reply-To: <20100615221426.GA24854@dcvr.yhbt.net>
References: <AANLkTinpzdLFf1bWBMRqDepIYQBKeZWb_mC0n-nlPhzD@mail.gmail.com>
	<20100526210542.GC24211@dcvr.yhbt.net>
	<AANLkTin3HAypERjYJ9NhC-BnVVmRjEGGEHXIECs2fSfc@mail.gmail.com>
	<20100615221426.GA24854@dcvr.yhbt.net>
Message-ID: <AANLkTilPqmKCkcRxACMaXJB3Uch4Es6oxipV3DpIyjkV@mail.gmail.com>

On Tue, Jun 15, 2010 at 3:14 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Russell Branca <chewbranca at gmail.com> wrote:
>> Hello Eric,
>>
>> Sorry for the delayed response, with the combination of being sick and
>> heading out of town for a while, this project got put on the
>> backburner. I really appreciate your response and think its a clean
>> solution for what I'm trying to do. I've started back in getting the
>> job queue working this week, and will hopefully have a working
>> solution in the next day or two. A little more information about what
>> I'm doing, I'm trying to create a centralized resque job queue server
>> that each of the different applications can queue work into, so I'll
>> be using redis behind resque for storing jobs and what not, which
>> brings me an area I'm not sure of the best approach on. So when we hit
>> the job queue endpoint in the rack app, it spawns the new worker, and
>> then immediately returns the 200 ok started background job message,
>> which cuts off communication back to the job queue. My plan is to save
>> a status message of the result of the background task into redis, and
>> have resque check that to verify the task was successful. Is there a
>> better approach for returning the resulting status code with unicorn,
>> or is this a reasonable approach? Thanks again for your help.
>
> Hi Russell, please don't top post, thanks.
>
> If you already have a queue server (and presumably a standalone app
> processing the queue), I would probably forgo the background Unicorn
> worker entirely.
>
> Based on my ancient (mid-2000s) knowledge of user-facing web
> applications: the application should queue the job, return 200, and have
> HTML meta refresh to constantly reload the page every few seconds.
>
> Hitting the reload endpoint would check the database (Redis in this
> case) for completion, and return a new HTML page to stop the meta
> refresh loop.
>
> This means you're no longer keeping a single Unicorn worker idle and
> wasting it. ?Nowadays you could do it with long-polling on
> Rainbows!/Thin/Zbatery, too, but long-polling is less reliable for
> people switching between WiFi access points. ?The meta refresh method
> can be a waste of power/bandwidth on the client side if the background
> job takes a long time, though.
>
> I'm familiar at all with Resque or Redis, but I suspect other folks
> on this mailing list should be able to help you flesh out the details.
>
> --
> Eric Wong
>

Hi Eric,

I have a queue server, but I don't have a standalone app processing
the jobs, because I have a large number of stand alone applications on
a single server. Right now I've got 12 separate apps running, so if I
wanted to have a standalone app for each, that would be 12 additional
applications in memory for handling background jobs. The whole reason
I want to go with the unicorn worker approach for handling background
jobs, is so I can fork off the master process as needed, avoid the
spawning time for a normal rails instance, and only use workers as
needed. This way I can have just a few workers running at any given
time, rather than 1 worker for each app. The number of apps is only
going to increase, but I want to keep the worker pool a constant. I'll
probably just update status of completion with redis, these jobs won't
be run by users, this is all background stuff like sending
notifications, data analysis, feed parsing, etc etc, so I'm planning
on just having resque initiate a request directly, and then use
unicorn to process the task in the background.

I didn't exactly follow what you meant when you were talking about a
unicorn worker being idle, from the example config.ru you responded
with earlier on, it looks like I can just spawn a new worker that will
be outside of the normal worker pool to handle the job. I'm pretty
sure this will work, I was curious about the best approach for
returning completion status, but I think just having the worker record
its status and exit is better than having long polling connections
open between the job queue and the new unicorn worker.


-Russell

From normalperson at yhbt.net  Tue Jun 15 20:06:11 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 15 Jun 2010 17:06:11 -0700
Subject: Forking off the unicorn master process to create a background
	worker
In-Reply-To: <AANLkTilPqmKCkcRxACMaXJB3Uch4Es6oxipV3DpIyjkV@mail.gmail.com>
References: <AANLkTinpzdLFf1bWBMRqDepIYQBKeZWb_mC0n-nlPhzD@mail.gmail.com>
	<20100526210542.GC24211@dcvr.yhbt.net>
	<AANLkTin3HAypERjYJ9NhC-BnVVmRjEGGEHXIECs2fSfc@mail.gmail.com>
	<20100615221426.GA24854@dcvr.yhbt.net>
	<AANLkTilPqmKCkcRxACMaXJB3Uch4Es6oxipV3DpIyjkV@mail.gmail.com>
Message-ID: <20100616000611.GA28609@dcvr.yhbt.net>

Russell Branca <chewbranca at gmail.com> wrote:
> On Tue, Jun 15, 2010 at 3:14 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > Russell Branca <chewbranca at gmail.com> wrote:
> >> Hello Eric,
> >>
> >> Sorry for the delayed response, with the combination of being sick and
> >> heading out of town for a while, this project got put on the
> >> backburner. I really appreciate your response and think its a clean
> >> solution for what I'm trying to do. I've started back in getting the
> >> job queue working this week, and will hopefully have a working
> >> solution in the next day or two. A little more information about what
> >> I'm doing, I'm trying to create a centralized resque job queue server
> >> that each of the different applications can queue work into, so I'll
> >> be using redis behind resque for storing jobs and what not, which
> >> brings me an area I'm not sure of the best approach on. So when we hit
> >> the job queue endpoint in the rack app, it spawns the new worker, and
> >> then immediately returns the 200 ok started background job message,
> >> which cuts off communication back to the job queue. My plan is to save
> >> a status message of the result of the background task into redis, and
> >> have resque check that to verify the task was successful. Is there a
> >> better approach for returning the resulting status code with unicorn,
> >> or is this a reasonable approach? Thanks again for your help.
> >
> > Hi Russell, please don't top post, thanks.
> >
> > If you already have a queue server (and presumably a standalone app
> > processing the queue), I would probably forgo the background Unicorn
> > worker entirely.
> >
> > Based on my ancient (mid-2000s) knowledge of user-facing web
> > applications: the application should queue the job, return 200, and have
> > HTML meta refresh to constantly reload the page every few seconds.
> >
> > Hitting the reload endpoint would check the database (Redis in this
> > case) for completion, and return a new HTML page to stop the meta
> > refresh loop.
> >
> > This means you're no longer keeping a single Unicorn worker idle and
> > wasting it. ?Nowadays you could do it with long-polling on
> > Rainbows!/Thin/Zbatery, too, but long-polling is less reliable for
> > people switching between WiFi access points. ?The meta refresh method
> > can be a waste of power/bandwidth on the client side if the background
> > job takes a long time, though.
> >
> > I'm familiar at all with Resque or Redis, but I suspect other folks
> > on this mailing list should be able to help you flesh out the details.
> 
> Hi Eric,
> 
> I have a queue server, but I don't have a standalone app processing
> the jobs, because I have a large number of stand alone applications on
> a single server. Right now I've got 12 separate apps running, so if I
> wanted to have a standalone app for each, that would be 12 additional
> applications in memory for handling background jobs. The whole reason
> I want to go with the unicorn worker approach for handling background
> jobs, is so I can fork off the master process as needed, avoid the
> spawning time for a normal rails instance, and only use workers as
> needed. This way I can have just a few workers running at any given
> time, rather than 1 worker for each app. The number of apps is only
> going to increase, but I want to keep the worker pool a constant. I'll
> probably just update status of completion with redis, these jobs won't
> be run by users, this is all background stuff like sending
> notifications, data analysis, feed parsing, etc etc, so I'm planning
> on just having resque initiate a request directly, and then use
> unicorn to process the task in the background.

Ah, so I guess it's a single queue server but multiple queues?  I
guess thats where I got confused with your description.

> I didn't exactly follow what you meant when you were talking about a
> unicorn worker being idle, from the example config.ru you responded
> with earlier on, it looks like I can just spawn a new worker that will
> be outside of the normal worker pool to handle the job. I'm pretty
> sure this will work, I was curious about the best approach for
> returning completion status, but I think just having the worker record
> its status and exit is better than having long polling connections
> open between the job queue and the new unicorn worker.

Yes, having the fork as I made in the example should work.  I haven't
tested it, of course :)  My instincts tell me recording the status and
exiting ASAP is better because it uses less memory.

You should test and experiment with it either way.  You know your apps,
requirements, and Redis/Resque far better than I do :)  Consider
software an evolutionary process, so whatever the "best approach" may
be, another one can usurp it eventually or be completely wrong in a
slightly different setting :)

-- 
Eric Wong

From normalperson at yhbt.net  Tue Jun 15 20:09:07 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 15 Jun 2010 17:09:07 -0700
Subject: anything left before 1.0?
Message-ID: <20100616000907.GA28484@dcvr.yhbt.net>

Hi all,

Since some folks have been running Unicorn in production for a while,
we might as well call the next release 1.0.

I have a couple of minor fixes in unicorn.git on top of 0.991.0.

Mainly, there are small compatibility fixes for the future (MRI 1.9.2
and Rubinius) and the past (MRI 1.8.6(!)).

There are also some documentation updates for Rack 1.2, but we should be
compatible with everything from Rack 0.9 (possibly earlier) to the
latest Rack 1.2.

Shortlog since 0.991.0:

Eric Wong (8):
      doc: cleanup rdoc escaping in title, hopefully
      cleanup: use modules were applicable
      t0005: Ruby 1.9.2 $LOAD_PATH fix
      http: fix rb_str_set_len() define for 1.8.6
      app/exec_cgi: rack.input may not respond to #size
      local.mk.sample: add 1.8.6-p114 to "full-test"
      tee_input: update documentation for Rack 1.2
      FAQ: help folks help themselves for processes dying in a loop

Let us know if there's anything else missing, pipe up within the next 24
hours or so...

You can always email me privately if you don't want your address on the
public mailing list.  Don't send HTML mail, though.

-- 
Eric Wong

From andrew at kongregate.com  Tue Jun 15 21:05:41 2010
From: andrew at kongregate.com (Andrew Grim)
Date: Tue, 15 Jun 2010 18:05:41 -0700
Subject: anything left before 1.0?
In-Reply-To: <20100616000907.GA28484@dcvr.yhbt.net>
References: <20100616000907.GA28484@dcvr.yhbt.net>
Message-ID: <AANLkTikXzy9FaBOcUia1Ko-aJlSHE-M8CzA4XBCmA81E@mail.gmail.com>

> Let us know if there's anything else missing, pipe up within the next 24
> hours or so...
>
> You can always email me privately if you don't want your address on the
> public mailing list. ?Don't send HTML mail, though.

Hey Eric,

I was hoping to spend some more time debugging myself, but since you
were going to release 1.0 I thought I'd get your thoughts on this.
Quick overview, I work for kongregate.com, one of the larger rails
sites, and one of the only large sites (that I know of personally)
running ruby 1.9.  We are currently running mostly mongrels, but I've
got one server testing Unicorn.  Things are mostly great, we are
seeing nearly 20ms improvement in average response time, which is
awesome.  Now to the issue.

SYMPTOM: The master process is killing the workers fairly frequently
based on the workers timing out.

CAUSE: I've added some logging to get the backtrace when I send a
SIGTERM, and it is always stuck on line 68 in http_response.rb:

      body.each { |chunk| socket.write(chunk) }

I ran some straces, and here's an example of the last few lines where
it gets killed:

06:50:23.239931 clock_gettime(CLOCK_REALTIME, {1276523423, 239967000})
= 0 <0.000172>
06:50:23.240213 write(12, "HTTP/1.1 200 OK\r\nDate: Mon, 14 J"...,
1896) = 1896 <0.000087>
06:50:23.242072 write(12, "<!DOCTYPE html PUBLIC \"-//W3C//D"...,
166842) = 128000 <0.000107>
06:50:23.242230 select(13, NULL, [12], NULL, NULL <unfinished ...>
06:51:22.167122 +++ killed by SIGKILL +++

So it's writing and then (to my understanding) waiting on the socket
to return, but you can see that for a full 60s it isn't.

My best guess off-hand is that the large size of the string being
written to the socket is causing an issue, and I have noticed that it
is happening primarily on requests that return larger payloads.  At
the same time, it isn't that much data, so I'm a little surprised it
would be an issue.  I am planning on trying to split the body up into
smaller chunks in a rack middleware or something.

Or I could be totally off.  Just wanted to see if you have any ideas,
I'm not even sure this is a Unicorn issue, definitely could be ruby
1.9 bug too.

Sorry about the long email, but I appreciate any help you can give. Thanks,
Andrew

From normalperson at yhbt.net  Tue Jun 15 22:27:27 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 15 Jun 2010 19:27:27 -0700
Subject: anything left before 1.0?
In-Reply-To: <AANLkTikXzy9FaBOcUia1Ko-aJlSHE-M8CzA4XBCmA81E@mail.gmail.com>
References: <20100616000907.GA28484@dcvr.yhbt.net>
	<AANLkTikXzy9FaBOcUia1Ko-aJlSHE-M8CzA4XBCmA81E@mail.gmail.com>
Message-ID: <20100616022727.GA29933@dcvr.yhbt.net>

Andrew Grim <andrew at kongregate.com> wrote:
> > Let us know if there's anything else missing, pipe up within the next 24
> > hours or so...
> >
> Hey Eric,
> 
> I was hoping to spend some more time debugging myself, but since you
> were going to release 1.0 I thought I'd get your thoughts on this.
> Quick overview, I work for kongregate.com, one of the larger rails
> sites, and one of the only large sites (that I know of personally)
> running ruby 1.9.  We are currently running mostly mongrels, but I've
> got one server testing Unicorn.  Things are mostly great, we are
> seeing nearly 20ms improvement in average response time, which is
> awesome.  Now to the issue.
> 
> SYMPTOM: The master process is killing the workers fairly frequently
> based on the workers timing out.
> 
> CAUSE: I've added some logging to get the backtrace when I send a
> SIGTERM, and it is always stuck on line 68 in http_response.rb:
> 
>       body.each { |chunk| socket.write(chunk) }
> 
> I ran some straces, and here's an example of the last few lines where
> it gets killed:
> 
> 06:50:23.239931 clock_gettime(CLOCK_REALTIME, {1276523423, 239967000})
> = 0 <0.000172>
> 06:50:23.240213 write(12, "HTTP/1.1 200 OK\r\nDate: Mon, 14 J"...,
> 1896) = 1896 <0.000087>
> 06:50:23.242072 write(12, "<!DOCTYPE html PUBLIC \"-//W3C//D"...,
> 166842) = 128000 <0.000107>
> 06:50:23.242230 select(13, NULL, [12], NULL, NULL <unfinished ...>
> 06:51:22.167122 +++ killed by SIGKILL +++
> 
> So it's writing and then (to my understanding) waiting on the socket
> to return, but you can see that for a full 60s it isn't.

Hi Andrew,

The timer starts when the app is initially dispatched, not when writing
starts.  You can check the log output from the master process (which
usually goes to stderr_path) and see the exact time the master process
saw before killing it.

Are you using nginx (or something else) to reverse proxy?  You should
be using nginx :)

> My best guess off-hand is that the large size of the string being
> written to the socket is causing an issue, and I have noticed that it
> is happening primarily on requests that return larger payloads.

That's unlikely.  I suspect the client you're using to hit Unicorn with
is not reading the other end of the socket, so once the kernel buffers
fill up, Unicorn blocks on the write.

Not a real solution, but you can probably hide the problem by increasing
the buffer sizes in the Linux kernel (net.core.wmem_max and
net.ipv4.tcp_wmem sysctls), but the defaults are already very generous.

> At the same time, it isn't that much data, so I'm a little surprised
> it would be an issue.  I am planning on trying to split the body up
> into smaller chunks in a rack middleware or something.

I doubt the middleware would help at all.

> Or I could be totally off.  Just wanted to see if you have any ideas,
> I'm not even sure this is a Unicorn issue, definitely could be ruby
> 1.9 bug too.

I would definitely look at your _client_ (which should be nginx).

You should isolate your client from other requests and strace that, too,
and see if it's reading off the socket at the same time.  I've used
1.9.1 pretty heavily with Rainbows! and large responses myself in
development/testing.

nginx will freeze up badly when running poorly-written Perl code with
the embedded Perl support.  Other than that it's been very solid in my
experience.

> Sorry about the long email, but I appreciate any help you can give.

No problem, let us know what you find out.

-- 
Eric Wong

From normalperson at yhbt.net  Thu Jun 17 06:08:25 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 17 Jun 2010 10:08:25 +0000
Subject: [ANN] unicorn 1.0.0 - yes, this is a real project
Message-ID: <20100617100825.GA17552@dcvr.yhbt.net>

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

There are only minor changes since 0.991.0.

For users clinging onto the past, MRI 1.8.6 support has been
restored.  Users are strongly encouraged to upgrade to the
latest 1.8.7, REE or 1.9.1.

For users looking towards the future, the core test suite and
the Rails 3 (beta) integration tests pass entirely under 1.9.2
preview3.  As of the latest rubinius.git[1], Rubinius support is
nearly complete as well.

Under Rubinius, signals may corrupt responses as they're being
written to the socket, but that should be fixable transparently
to us[4].  Support for the hardly used, hardly documented[2]
embedded command-line switches in rackup config (.ru) files is
is also broken under Rubinius.

The recently-released Rack 1.2.1 introduced no compatiblity
issues[3] in core Unicorn.  We remain compatible with all Rack
releases starting with 0.9.1 (and possibly before).

[1] tested with Rubinius upstream commit
cf4a5a759234faa3f7d8a92d68fa89d8c5048f72
[2] lets avoid the Dueling Banjos effect here :x
[3] actually, Rack 1.2.1 is broken under 1.8.6.
[4] http://github.com/evanphx/rubinius/issues/373


The Future:

* Bug/compatibility fixes as needed, of course!
* Scalability for hardware that may be common in 5-10 years
* Rainbows!  LOTS of Rainbows!

Thanks for reading!

-- 
Eric Wong

From win at wincent.com  Thu Jun 17 06:17:40 2010
From: win at wincent.com (Wincent Colaiuta)
Date: Thu, 17 Jun 2010 12:17:40 +0200
Subject: [ANN] unicorn 1.0.0 - yes, this is a real project
In-Reply-To: <20100617100825.GA17552@dcvr.yhbt.net>
References: <20100617100825.GA17552@dcvr.yhbt.net>
Message-ID: <913ADB43-85EC-4CD3-B985-1CB9CD8D8FF3@wincent.com>

Huge congratulations on hitting the 1.0 milestone. Great project and I sincerely hope that it has a long future ahead of it.

Cheers,
Wincent




From dvdplm at gmail.com  Thu Jun 17 11:31:32 2010
From: dvdplm at gmail.com (David Palm)
Date: Thu, 17 Jun 2010 08:31:32 -0700
Subject: [ANN] unicorn 1.0.0 - yes, this is a real project
In-Reply-To: <20100617100825.GA17552@dcvr.yhbt.net>
References: <20100617100825.GA17552@dcvr.yhbt.net>
Message-ID: <20100617083132603125.e6f05277@gmail.com>

> The Future:

...is going to be bright. Awesome work and fyi, I read unicorn source code to my kids before putting them to sleep. It's that good.

:) 

From jamie at tramchase.com  Thu Jun 17 18:22:33 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Thu, 17 Jun 2010 22:22:33 +0000 (UTC)
Subject: =?utf-8?b?bW9uZ3JlbF9wcm9jdGl0bGU=?= like support in Unicorn
References: <ae96cae30912021352l170af773k275b14377138202c@mail.gmail.com>
	<944a03770912021404s189a3686sb8b024a21d5a7c7a@mail.gmail.com>
Message-ID: <loom.20100618T002015-365@post.gmane.org>

Michael Guterl <mguterl <at> gmail.com> writes:

> 
> On Wed, Dec 2, 2009 at 4:52 PM, Justin Hahn <jhahn <at> rbmtechnologies.com>
wrote:
> > Before I go off ans try cutting my own horrible monkey patch to
> > unicorn, has anyone tried implementing something like
> > mongrel_proctitle for Unicorn?

> There's some rack middleware out there, RackProctitle, which provides
> this functionality.  I have not used it with Unicorn, but I don't see
> why it wouldn't work.
> 
> http://coderack.org/users/arya/entries/3-rack-proctitle

FWIW this doesn't seem to work out-of-box with Unicorn. Eric, do you know
offhand if it's possible to use middleware to call unicorn's proc_name? I'd love
to be able to watch what my unicorns are doing in realtime

Slightly related -- is it also possible to interrogate the current size of a
socket's backlog? (is this what Raindrops is for? It doesn't work out of box on
my mac, but I'd see about getting it working if it did)

Congrats on 1.0




From normalperson at yhbt.net  Thu Jun 17 19:21:50 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 17 Jun 2010 23:21:50 +0000
Subject: mongrel_proctitle like support in Unicorn
In-Reply-To: <loom.20100618T002015-365@post.gmane.org>
References: <ae96cae30912021352l170af773k275b14377138202c@mail.gmail.com>
	<944a03770912021404s189a3686sb8b024a21d5a7c7a@mail.gmail.com>
	<loom.20100618T002015-365@post.gmane.org>
Message-ID: <20100617232149.GB25862@dcvr.yhbt.net>

Jamie Wilkinson <jamie at tramchase.com> wrote:
> Michael Guterl <mguterl <at> gmail.com> writes:
> > On Wed, Dec 2, 2009 at 4:52 PM, Justin Hahn <jhahn <at> rbmtechnologies.com>
> wrote:
> > > Before I go off ans try cutting my own horrible monkey patch to
> > > unicorn, has anyone tried implementing something like
> > > mongrel_proctitle for Unicorn?
> 
> > There's some rack middleware out there, RackProctitle, which provides
> > this functionality.  I have not used it with Unicorn, but I don't see
> > why it wouldn't work.
> > 
> > http://coderack.org/users/arya/entries/3-rack-proctitle
> 
> FWIW this doesn't seem to work out-of-box with Unicorn. Eric, do you know
> offhand if it's possible to use middleware to call unicorn's proc_name? I'd love
> to be able to watch what my unicorns are doing in realtime

That coderack link no longer works for me, but I see no reason why
Rack::ProcTitle in rack-contrib wouldn't work (based on my reading of
the code):

http://github.com/rack/rack-contrib/blob/master/lib/rack/contrib/proctitle.rb

I guess it holds onto $0 after the app is done dispatching...

> Slightly related -- is it also possible to interrogate the current size of a
> socket's backlog? (is this what Raindrops is for? It doesn't work out of box on
> my mac, but I'd see about getting it working if it did)

Yes, the backlog interrogation code is in Raindrops is very
Linux-specific.  The shared memory counters should be reasonably
portable to modern POSIX-ish systems (using GCC).

I have no idea if similar backlog interrogation functionality exists in
other operating systems.  I only learned of the Linux functionality
shortly before writing Raindrops.

If somebody could provide working patches for similar functionality in
other OSes, I'd be more than happy to include them into Raindrops.

> Congrats on 1.0

Thanks :>

-- 
Eric Wong

From augusto at jadedpixel.com  Thu Jun 17 21:47:35 2010
From: augusto at jadedpixel.com (Augusto Becciu)
Date: Thu, 17 Jun 2010 22:47:35 -0300
Subject: [ANN] unicorn 1.0.0 - yes, this is a real project
In-Reply-To: <20100617100825.GA17552@dcvr.yhbt.net>
References: <20100617100825.GA17552@dcvr.yhbt.net>
Message-ID: <AANLkTil-pO4my3h6POO1XGd2nghbBwE0A1BtqClJNYUR@mail.gmail.com>

Wow, awesome. Huge congrats Eric!

On Thu, Jun 17, 2010 at 7:08 AM, Eric Wong <normalperson at yhbt.net> wrote:
> Unicorn is an HTTP server for Rack applications designed to only serve
> fast clients on low-latency, high-bandwidth connections and take
> advantage of features in Unix/Unix-like kernels. ?Slow clients should
> only be served by placing a reverse proxy capable of fully buffering
> both the the request and response in between Unicorn and slow clients.
>
> * http://unicorn.bogomips.org/
> * mongrel-unicorn at rubyforge.org
> * git://git.bogomips.org/unicorn.git
>
> Changes:
>
> There are only minor changes since 0.991.0.
>
> For users clinging onto the past, MRI 1.8.6 support has been
> restored. ?Users are strongly encouraged to upgrade to the
> latest 1.8.7, REE or 1.9.1.
>
> For users looking towards the future, the core test suite and
> the Rails 3 (beta) integration tests pass entirely under 1.9.2
> preview3. ?As of the latest rubinius.git[1], Rubinius support is
> nearly complete as well.
>
> Under Rubinius, signals may corrupt responses as they're being
> written to the socket, but that should be fixable transparently
> to us[4]. ?Support for the hardly used, hardly documented[2]
> embedded command-line switches in rackup config (.ru) files is
> is also broken under Rubinius.
>
> The recently-released Rack 1.2.1 introduced no compatiblity
> issues[3] in core Unicorn. ?We remain compatible with all Rack
> releases starting with 0.9.1 (and possibly before).
>
> [1] tested with Rubinius upstream commit
> cf4a5a759234faa3f7d8a92d68fa89d8c5048f72
> [2] lets avoid the Dueling Banjos effect here :x
> [3] actually, Rack 1.2.1 is broken under 1.8.6.
> [4] http://github.com/evanphx/rubinius/issues/373
>
>
> The Future:
>
> * Bug/compatibility fixes as needed, of course!
> * Scalability for hardware that may be common in 5-10 years
> * Rainbows! ?LOTS of Rainbows!
>
> Thanks for reading!
>
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>

From r at tomayko.com  Fri Jun 18 16:13:33 2010
From: r at tomayko.com (Ryan Tomayko)
Date: Fri, 18 Jun 2010 13:13:33 -0700
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <20100604205925.GA7361@dcvr.yhbt.net>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net>
	<AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>
	<20100603184730.GA2421@dcvr.yhbt.net>
	<AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>
	<20100604205925.GA7361@dcvr.yhbt.net>
Message-ID: <AANLkTimT-pFr0GW8fhmhQ-9a8eI98NrzXsGBsqSn5UoJ@mail.gmail.com>

On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Chris Wanstrath <chris at ozmm.org> wrote:
>> That's what we do at GitHub. We're running Rails 2.2.2 and have a
>> custom config.ru, thanks to Unicorn:
>>
>> http://gist.github.com/424352
>
> By the way, how's the OobGC middleware working for you guys?

We rolled out the OobGC middleware along with a basic RailsBench GC
config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms
(~15%) off the average response time across the site (real traffic).
The impact was significantly more for requests that allocate a lot of
objects -- as much as 50% decreases in response time for the worst
offenders. We saw no noticeable increase in CPU with OobGC set to run
every 10 requests, and a fair increase in CPU with OobGC set to run
every 5 requests.

Because I rolled this stuff out somewhat non-scientifically, I've
always wondered how much OobGC contributed to the overall savings vs.
the RailsBench GC config. Disabling the OobGC middleware but leaving
the RailsBench GC config in place, I get the following graph:

http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png

So we're spending ~1ms request time in GC with OobGC, and ~10ms
request time in GC without it.

Here's some system load graphs for the same time period just to show
that OobGC has no adverse effect when set to GC every 10 requests:

http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png

I assume the RailsBench GC patches improve the effect of OobGC
considerably by increasing the number of objects that can be allocated
between GC runs, allowing more of the GC work to be deferred to
in-between-requests time. Here's the RailsBench config we're using
today, for the record:

    RUBY_HEAP_MIN_SLOTS=800000
    RUBY_HEAP_FREE_MIN=100000
    RUBY_HEAP_SLOTS_INCREMENT=300000
    RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
    RUBY_GC_MALLOC_LIMIT=79000000

This is only barely tuned for us. I stole most of the numbers from the
Twitter and 37signals examples.

I've also experimented with tuning the GC specifically to take
advantage of OobGC:

https://gist.github.com/87d574a19372c6043c5f

# The following GC settings have been tuned for GitHub application web requests.
# Most settings are significantly higher than the example configs published by
# Twitter and 37signals. There's a couple reasons for this. First, the GitHub
# app has a memory footprint that's 3x-4x larger than the standard Rails app
# (roughly 200MB after first request compared to ~40MB-50MB). Second, because
# Unicorn is such an exceptional piece of software, we're able to schedule GC
# to run outside the context of requests so as not to effect response times.
# As such, we try to allocate enough memory to service 5 requests
without needing
# GC and then run GC manually immediately after each fifth request has been
# served but before the process starts accepting the next connection. The result
# is higher memory use (~300MB per Unicorn worker process on average) and a
# slight increase in CPU due to forced manual GC, but better response times.
# ...

Unfortunately, the bigger heap seems to cause a largish increase in
the time needed for each GC, so the unicorn workers were spending too
much time between requests. CPU and RES increases were even more than
I'd expected. It also didn't eliminate in-request GC entirely as I'd
hoped.

I eventually abandoned the idea -- even if I could get it to behave,
it's hardly worth the 1ms it would save. I mention it here because the
general approach might work in situations where the base heap size is
a bit smaller (say < 80MB) or perhaps I'm mistuning one of the
parameters.

Thanks,
Ryan

> Luke: did you also get a chance to try this in place of my original
> monkey patch?
>
> Thanks in advance for any info you can share.
>
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>

From normalperson at yhbt.net  Fri Jun 18 17:48:34 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 18 Jun 2010 21:48:34 +0000
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <AANLkTimT-pFr0GW8fhmhQ-9a8eI98NrzXsGBsqSn5UoJ@mail.gmail.com>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net>
	<AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>
	<20100603184730.GA2421@dcvr.yhbt.net>
	<AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>
	<20100604205925.GA7361@dcvr.yhbt.net>
	<AANLkTimT-pFr0GW8fhmhQ-9a8eI98NrzXsGBsqSn5UoJ@mail.gmail.com>
Message-ID: <20100618214834.GA1817@dcvr.yhbt.net>

Ryan Tomayko <r at tomayko.com> wrote:
> On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > Chris Wanstrath <chris at ozmm.org> wrote:
> >> That's what we do at GitHub. We're running Rails 2.2.2 and have a
> >> custom config.ru, thanks to Unicorn:
> >>
> >> http://gist.github.com/424352
> >
> > By the way, how's the OobGC middleware working for you guys?
> 
> We rolled out the OobGC middleware along with a basic RailsBench GC
> config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms
> (~15%) off the average response time across the site (real traffic).
> The impact was significantly more for requests that allocate a lot of
> objects -- as much as 50% decreases in response time for the worst
> offenders. We saw no noticeable increase in CPU with OobGC set to run
> every 10 requests, and a fair increase in CPU with OobGC set to run
> every 5 requests.

Cool.  Am I correct to assume the increased CPU usage at every 5
requests wasn't worth any performance gains you might have had?

> Because I rolled this stuff out somewhat non-scientifically, I've
> always wondered how much OobGC contributed to the overall savings vs.
> the RailsBench GC config. Disabling the OobGC middleware but leaving
> the RailsBench GC config in place, I get the following graph:
> 
> http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png
> 
> So we're spending ~1ms request time in GC with OobGC, and ~10ms
> request time in GC without it.

Awesome.

> Here's some system load graphs for the same time period just to show
> that OobGC has no adverse effect when set to GC every 10 requests:
> 
> http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png
> 
> I assume the RailsBench GC patches improve the effect of OobGC
> considerably by increasing the number of objects that can be allocated
> between GC runs, allowing more of the GC work to be deferred to
> in-between-requests time. Here's the RailsBench config we're using
> today, for the record:
> 
>     RUBY_HEAP_MIN_SLOTS=800000
>     RUBY_HEAP_FREE_MIN=100000
>     RUBY_HEAP_SLOTS_INCREMENT=300000
>     RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
>     RUBY_GC_MALLOC_LIMIT=79000000
> 
> This is only barely tuned for us. I stole most of the numbers from the
> Twitter and 37signals examples.
> 
> I've also experimented with tuning the GC specifically to take
> advantage of OobGC:
> 
> https://gist.github.com/87d574a19372c6043c5f
> 
> # The following GC settings have been tuned for GitHub application web requests.
> # Most settings are significantly higher than the example configs published by
> # Twitter and 37signals. There's a couple reasons for this. First, the GitHub
> # app has a memory footprint that's 3x-4x larger than the standard Rails app
> # (roughly 200MB after first request compared to ~40MB-50MB). Second, because

Yikes, 200MB after one request is a lot.  If you can easily find ways to
cut that down, it should be more of a gain than the monster heap you've
tried.

> # Unicorn is such an exceptional piece of software, we're able to schedule GC
> # to run outside the context of requests so as not to effect response times.
> # As such, we try to allocate enough memory to service 5 requests
> without needing
> # GC and then run GC manually immediately after each fifth request has been
> # served but before the process starts accepting the next connection. The result
> # is higher memory use (~300MB per Unicorn worker process on average) and a
> # slight increase in CPU due to forced manual GC, but better response times.
> # ...
> 
> Unfortunately, the bigger heap seems to cause a largish increase in
> the time needed for each GC, so the unicorn workers were spending too
> much time between requests. CPU and RES increases were even more than
> I'd expected. It also didn't eliminate in-request GC entirely as I'd
> hoped.
> 
> I eventually abandoned the idea -- even if I could get it to behave,
> it's hardly worth the 1ms it would save. I mention it here because the
> general approach might work in situations where the base heap size is
> a bit smaller (say < 80MB) or perhaps I'm mistuning one of the
> parameters.

So in conclusion, OobGC itself works, but too large of a heap isn't
worth it even for a memory hungry app.

I suppose having too big of a heap means it can fragment more badly.
Making GC run more often on a smaller heap can and give similar (or
maybe better) performance.  At best you'll get diminishing returns as
you seem to have concluded.

I have no doubt the Railsbench GC patches help.  Even with small apps,
being able being able to set a linear growth factor on long-running
servers is awesome.

Thanks for sharing this!

-- 
Eric Wong

From zargony at gmail.com  Fri Jun 18 20:19:18 2010
From: zargony at gmail.com (Andreas Neuhaus)
Date: Sat, 19 Jun 2010 02:19:18 +0200
Subject: Unicorn tempfile chown problem with Ruby 1.9
Message-ID: <AANLkTimcWiSbWMdaf58RuipDCSgN0MncQG_u-bB55Rh9@mail.gmail.com>

Hi everybody,

I ran into a problem with unicorn today. Using Ruby 1.9.2-preview3
(compiled by rvm) and a unicorn config that has a "user" statement to
let workers run under a different user, unicorn fails to start any
worker process:

/usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:189:in
`chown': No such file or directory - /tmp/0.8216141026704238
(Errno::ENOENT)
	from /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:189:in
`user'
	from /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:664:in
`init_worker_process'
	from /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:681:in
`worker_loop'
	from /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:598:in
`block (2 levels) in spawn_missing_workers'

>From what I see, Worker#user tries to chown the unlinked worker
tempfile (in unicorn.rb:189), which fails with Ruby 1.9.2, but seems
to work fine with Ruby 1.8.7. The problem happened to me on an Ubuntu
server, but I was able to reproduce it locally on OSX.

How to reproduce:
- create a simple rack app, like
class Hello; def call (env); [200, { 'Content-Type' => 'text/plain' },
['Hello!']]; end; end
run Hello.new
- create a unicorn.conf.rb with a user statement that switches to the
current user
user 'andy'
- run "unicorn -c unicorn.conf.rb" on Ruby 1.9.2
- watch workers failing to start

I'm not that deep into unicorn's internals, but since the worker
tempfile is unlinked anyway, does it actually make sense to still call
chown on it? Removing tmp.chown in Worker#user solved this for me.

Please cc replies since I'm not subscribed to the list, thanks.

regards,
Andreas Neuhaus
http://zargony.com/

From normalperson at yhbt.net  Fri Jun 18 22:44:25 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Sat, 19 Jun 2010 02:44:25 +0000
Subject: Unicorn tempfile chown problem with Ruby 1.9
In-Reply-To: <AANLkTimcWiSbWMdaf58RuipDCSgN0MncQG_u-bB55Rh9@mail.gmail.com>
References: <AANLkTimcWiSbWMdaf58RuipDCSgN0MncQG_u-bB55Rh9@mail.gmail.com>
Message-ID: <20100619024425.GA3132@dcvr.yhbt.net>

Andreas Neuhaus <zargony at gmail.com> wrote:
> Hi everybody,
> 
> I ran into a problem with unicorn today. Using Ruby 1.9.2-preview3
> (compiled by rvm) and a unicorn config that has a "user" statement to
> let workers run under a different user, unicorn fails to start any
> worker process:
> 
> /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:189:in
> `chown': No such file or directory - /tmp/0.8216141026704238
> (Errno::ENOENT)
> 	from /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:189:in
> `user'
> 	from /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:664:in
> `init_worker_process'
> 	from /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:681:in
> `worker_loop'
> 	from /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:598:in
> `block (2 levels) in spawn_missing_workers'
> 
> >From what I see, Worker#user tries to chown the unlinked worker
> tempfile (in unicorn.rb:189), which fails with Ruby 1.9.2, but seems
> to work fine with Ruby 1.8.7. The problem happened to me on an Ubuntu
> server, but I was able to reproduce it locally on OSX.
> 
> How to reproduce:
> - create a simple rack app, like
> class Hello; def call (env); [200, { 'Content-Type' => 'text/plain' },
> ['Hello!']]; end; end
> run Hello.new
> - create a unicorn.conf.rb with a user statement that switches to the
> current user
> user 'andy'
> - run "unicorn -c unicorn.conf.rb" on Ruby 1.9.2
> - watch workers failing to start
> 
> I'm not that deep into unicorn's internals, but since the worker
> tempfile is unlinked anyway, does it actually make sense to still call
> chown on it? Removing tmp.chown in Worker#user solved this for me.

Hi Andreas,

This is a bug in Ruby and I've filed a report and patch in Redmine:

 http://redmine.ruby-lang.org/issues/show/3451

Unicorn needs to fchmod(2) it as the regular user for the heartbeat
functionality to work (master process continues to run as root
if started as root).

Is Unicorn actually able to serve requests when you removed tmp.chown?
Under Linux, fchmod() doesn't work without ownership of the file.

Btw, for Unicorn 2.0, the fchmod()-based heartbeat implementation will
be a fallback only.  I'll be relying on the mmap() + atomic counters code
of Raindrops[1][2].  mmap() + atomic counters was my first choice
anyways, but I didn't want to write more C at the time.

> Please cc replies since I'm not subscribed to the list, thanks.

No problem :>


[1] - The mmap()/atomic counters parts of Raindrops should be portable
      to modern POSIX-ish systems.  I'll probably use libatomic-ops or
      similar if gcc isn't available.

[2] - http://raindrops.bogomips.org/

-- 
Eric Wong

From zargony at gmail.com  Sat Jun 19 09:38:36 2010
From: zargony at gmail.com (Andreas Neuhaus)
Date: Sat, 19 Jun 2010 15:38:36 +0200
Subject: Unicorn tempfile chown problem with Ruby 1.9
In-Reply-To: <20100619024425.GA3132@dcvr.yhbt.net>
References: <AANLkTimcWiSbWMdaf58RuipDCSgN0MncQG_u-bB55Rh9@mail.gmail.com>
	<20100619024425.GA3132@dcvr.yhbt.net>
Message-ID: <AANLkTilrAyswjJjmxcq5WWjtwLt_jWg3ABWP3eVH4iB2@mail.gmail.com>

Hi Eric,

>> /usr/local/rvm/gems/ruby-1.9.2-preview3/gems/unicorn-1.0.0/lib/unicorn.rb:189:in
>> `chown': No such file or directory - /tmp/0.8216141026704238
>> (Errno::ENOENT)

> This is a bug in Ruby and I've filed a report and patch in Redmine:
> ?http://redmine.ruby-lang.org/issues/show/3451

Thanks. I started to watch that ticket to see where it goes. In the
meantime, I tried to workaround the problem by telling unicorn to
chown the tempfile before it's unlinked (which looks like it works
fine for now). http://gist.github.com/444901

> Unicorn needs to fchmod(2) it as the regular user for the heartbeat
> functionality to work (master process continues to run as root
> if started as root).
> Is Unicorn actually able to serve requests when you removed tmp.chown?
> Under Linux, fchmod() doesn't work without ownership of the file.

You're right, it doesn't work if tmp.chown is removed. For
convenience, I always tried it on my local machine as a user and
chown'ed to my own user to trigger the problem. This way I could test
things without running it as root. When running unicorn with tmp.chown
removed as root, it throws EPERM errors as you expected.

regards,
Andreas Neuhaus

From r at tomayko.com  Mon Jun 21 15:03:37 2010
From: r at tomayko.com (Ryan Tomayko)
Date: Mon, 21 Jun 2010 12:03:37 -0700
Subject: Fwd: Support for Soft Timeout in Unicorn
In-Reply-To: <20100618214834.GA1817@dcvr.yhbt.net>
References: <20100603173749.GA19649@dcvr.yhbt.net>
	<20100603182246.GB19649@dcvr.yhbt.net>
	<AANLkTile44yxAsA3bT_nGEgLHQcCeRd5tKspdKs2rOoV@mail.gmail.com>
	<20100603184730.GA2421@dcvr.yhbt.net>
	<AANLkTilIJL2dP_nlqwn-Jh5a3KhQ0wL75L5Be_2ybQZr@mail.gmail.com>
	<20100604205925.GA7361@dcvr.yhbt.net>
	<AANLkTimT-pFr0GW8fhmhQ-9a8eI98NrzXsGBsqSn5UoJ@mail.gmail.com>
	<20100618214834.GA1817@dcvr.yhbt.net>
Message-ID: <AANLkTilUbgdyDv9W1bi-s_W6kq9sOhWfmuYkKLoKGOLj@mail.gmail.com>

On Fri, Jun 18, 2010 at 2:48 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Ryan Tomayko <r at tomayko.com> wrote:
>> On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong <normalperson at yhbt.net> wrote:
>> > Chris Wanstrath <chris at ozmm.org> wrote:
>> >> That's what we do at GitHub. We're running Rails 2.2.2 and have a
>> >> custom config.ru, thanks to Unicorn:
>> >>
>> >> http://gist.github.com/424352
>> >
>> > By the way, how's the OobGC middleware working for you guys?
>>
>> We rolled out the OobGC middleware along with a basic RailsBench GC
>> config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms
>> (~15%) off the average response time across the site (real traffic).
>> The impact was significantly more for requests that allocate a lot of
>> objects -- as much as 50% decreases in response time for the worst
>> offenders. We saw no noticeable increase in CPU with OobGC set to run
>> every 10 requests, and a fair increase in CPU with OobGC set to run
>> every 5 requests.
>
> Cool. ?Am I correct to assume the increased CPU usage at every 5
> requests wasn't worth any performance gains you might have had?

Yes. I don't have exact numbers on how much more CPU was being
utilized but it was definitely noticeable on my graphs and had very
little, if any, effect on overall response time beyond the savings at
the req/10 interval.

>> Because I rolled this stuff out somewhat non-scientifically, I've
>> always wondered how much OobGC contributed to the overall savings vs.
>> the RailsBench GC config. Disabling the OobGC middleware but leaving
>> the RailsBench GC config in place, I get the following graph:
>>
>> http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png
>>
>> So we're spending ~1ms request time in GC with OobGC, and ~10ms
>> request time in GC without it.
>
> Awesome.
>
>> Here's some system load graphs for the same time period just to show
>> that OobGC has no adverse effect when set to GC every 10 requests:
>>
>> http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png
>>
>> I assume the RailsBench GC patches improve the effect of OobGC
>> considerably by increasing the number of objects that can be allocated
>> between GC runs, allowing more of the GC work to be deferred to
>> in-between-requests time. Here's the RailsBench config we're using
>> today, for the record:
>>
>> ? ? RUBY_HEAP_MIN_SLOTS=800000
>> ? ? RUBY_HEAP_FREE_MIN=100000
>> ? ? RUBY_HEAP_SLOTS_INCREMENT=300000
>> ? ? RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
>> ? ? RUBY_GC_MALLOC_LIMIT=79000000
>>
>> This is only barely tuned for us. I stole most of the numbers from the
>> Twitter and 37signals examples.
>>
>> I've also experimented with tuning the GC specifically to take
>> advantage of OobGC:
>>
>> https://gist.github.com/87d574a19372c6043c5f
>>
>> # The following GC settings have been tuned for GitHub application web requests.
>> # Most settings are significantly higher than the example configs published by
>> # Twitter and 37signals. There's a couple reasons for this. First, the GitHub
>> # app has a memory footprint that's 3x-4x larger than the standard Rails app
>> # (roughly 200MB after first request compared to ~40MB-50MB). Second, because
>
> Yikes, 200MB after one request is a lot. ?If you can easily find ways to
> cut that down, it should be more of a gain than the monster heap you've
> tried.

Indeed. We bring in *a lot* of libraries, some of which add
embarassingly large amounts of code and data at require time (aws-s3,
googlecharts - I'm looking at you).

>> # Unicorn is such an exceptional piece of software, we're able to schedule GC
>> # to run outside the context of requests so as not to effect response times.
>> # As such, we try to allocate enough memory to service 5 requests
>> without needing
>> # GC and then run GC manually immediately after each fifth request has been
>> # served but before the process starts accepting the next connection. The result
>> # is higher memory use (~300MB per Unicorn worker process on average) and a
>> # slight increase in CPU due to forced manual GC, but better response times.
>> # ...
>>
>> Unfortunately, the bigger heap seems to cause a largish increase in
>> the time needed for each GC, so the unicorn workers were spending too
>> much time between requests. CPU and RES increases were even more than
>> I'd expected. It also didn't eliminate in-request GC entirely as I'd
>> hoped.
>>
>> I eventually abandoned the idea -- even if I could get it to behave,
>> it's hardly worth the 1ms it would save. I mention it here because the
>> general approach might work in situations where the base heap size is
>> a bit smaller (say < 80MB) or perhaps I'm mistuning one of the
>> parameters.
>
> So in conclusion, OobGC itself works, but too large of a heap isn't
> worth it even for a memory hungry app.
>
> I suppose having too big of a heap means it can fragment more badly.
> Making GC run more often on a smaller heap can and give similar (or
> maybe better) performance. ?At best you'll get diminishing returns as
> you seem to have concluded.

That sounds like an extremely plausible explanation.

> I have no doubt the Railsbench GC patches help. ?Even with small apps,
> being able being able to set a linear growth factor on long-running
> servers is awesome.

Sure is.

Thanks,
Ryan

> Thanks for sharing this!
>
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>

From normalperson at yhbt.net  Mon Jun 21 20:16:32 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 21 Jun 2010 17:16:32 -0700
Subject: [Mongrel] scaling unicorn
In-Reply-To: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
References: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
Message-ID: <20100622001632.GA10082@dcvr.yhbt.net>

snacktime <snacktime at gmail.com> wrote:
> Interested in some feeback on this (does it sound right?), or maybe
> this might be of interest to others.

Hi Chris,

I think you meant to post this to the mongrel-unicorn at rubyforge.org
list, not mongrel-users at rubyforge.org :>

> We are launching a new facebook app in a couple weeks and we did some
> load testing over the weekend on our unicorn web cluster.  The servers
> are 8 way xeon's with 24gb ram.  Our app ended up being primarily cpu
> bound.  So far the sweet spot for the number of unicorns seems to be
> around 40.  This seemed to yield the most requests per second without
> overloading the server or hitting memory bandwidth issues.  The
> backlog is at the somaxconn default of 128, I'm still not sure if we
> will bump that up or not.

The default backlog we try to specify is actually 1024 (same as
Mongrel).  But it's always a murky value anyways, as it's
kernel/sysctl-dependent.  With Unix domain sockets, some folks use
crazy values like 2048 to look better on synthetic benchmarks :)

> Increasing the number of unicorns beyond a
> certain point resulted in a noticable drop in the requests per second
> the server could handle.   I'm pretty sure the cause is the box
> running out of memory bandwidth.  The load average and resource usage
> in general (except for memory) would keep going down but so did the
> requests per second.  At 80 unicorns the requests per second dropped
> by more then half.  I'm going to disable hyperthreading and rerun some
> of the tests to see what impact that has.

That's "8 way xeon" _before_ hyperthreading, right?  Which family of
Xeons are you using, the Pentium4-based crap or the awesome new ones?

How much memory is each Unicorn worker using for your app?

40 workers for 8 physical cores sounds reasonable.  Depending on the
app, I think the reasonable range is anywhere from 2-8 workers per
physical core.  More if you're (unfortunately) limited by external
network calls, but since you claim to be CPU bound, less.

Do you have actual performance numbers you're able to share?
Mean/median request times/rates would be very useful.  If your requests
run very quickly, you may be limited by contention with the accept()
syscall on the listen socket, too.

I assume you're using nginx as the proxy, is this with Unix domain
sockets or TCP sockets?  Unix domain sockets should give a small
performance over TCP if it's all on the same box.

With TCP, you should also check to see you have enough local ports
available if you're hitting extremely high (and probably unrealistic :)
request rates.

-- 
Eric Wong

From jamie at tramchase.com  Mon Jun 21 22:34:33 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Mon, 21 Jun 2010 19:34:33 -0700
Subject: scaling unicorn
In-Reply-To: <20100622001632.GA10082@dcvr.yhbt.net>
References: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
	<20100622001632.GA10082@dcvr.yhbt.net>
Message-ID: <F596AC52-4CD1-4543-9B04-7A19762995DC@tramchase.com>


On Jun 21, 2010, at 5:16 PM, Eric Wong wrote:

>> overloading the server or hitting memory bandwidth issues.  The
>> backlog is at the somaxconn default of 128, I'm still not sure if we
>> will bump that up or not.
> 
> The default backlog we try to specify is actually 1024 (same as
> Mongrel).  But it's always a murky value anyways, as it's
> kernel/sysctl-dependent.  With Unix domain sockets, some folks use
> crazy values like 2048 to look better on synthetic benchmarks :)

Somewhat related -- I've been meaning to discuss the finer points of backlog tuning.

I've been experimenting with the multi-server socket+TCP megaunicorn configuration from your CDT:
http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000033.html

Which I think is what this sentence from TUNING is talking about?

	"Setting a very low value for the :backlog parameter in ?listen? directives can allow failover to happen more quickly if your cluster is configured for it."

Our app can catch a batch of requests which will be slow (1-3s), and these can pool on one individual server in our load-balanced EC2 cluster -- exactly the case for the multi-server failover setup. 

I've put this into production under a healthy load (5000+ RPM) and it appears to work really well!  Produces very high requests/s rates at significantly higher concurrency than without, and serves zero 502 errors (part of the goal)

I currently I have the unix socket set to a backlog of 64, then failing over to a TCP listener using backlog 1024 (so that things are queued rather than 502'd)

I can imagine there might be a case for keeping the TCP backlog low as well & serving errors when overloaded, rather than getting caught in an unrecoverable back-queue tarpit

I'm currently failing-over to a dedicated "backup" instance, so that I could measure exactly how much traffic is being offloaded. This means my benchmarks w/o failover are 1 server, but with failover is actually 2 servers. We're reconfiguring to something more like the original diagram at which point I'll do some cluster-wide stress-tests & share data/scripts/process. 

BTW, this configuration needs a cool name!

-jamie
http://jamiedubs.com
http://fffff.at


From normalperson at yhbt.net  Tue Jun 22 00:53:46 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 22 Jun 2010 04:53:46 +0000
Subject: scaling unicorn
In-Reply-To: <F596AC52-4CD1-4543-9B04-7A19762995DC@tramchase.com>
References: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
	<20100622001632.GA10082@dcvr.yhbt.net>
	<F596AC52-4CD1-4543-9B04-7A19762995DC@tramchase.com>
Message-ID: <20100622045346.GA23710@dcvr.yhbt.net>

Jamie Wilkinson <jamie at tramchase.com> wrote:
> On Jun 21, 2010, at 5:16 PM, Eric Wong wrote:
> >> overloading the server or hitting memory bandwidth issues.  The
> >> backlog is at the somaxconn default of 128, I'm still not sure if
> >> we will bump that up or not.
> > 
> > The default backlog we try to specify is actually 1024 (same as
> > Mongrel).  But it's always a murky value anyways, as it's
> > kernel/sysctl-dependent.  With Unix domain sockets, some folks use
> > crazy values like 2048 to look better on synthetic benchmarks :)
>  
> Somewhat related -- I've been meaning to discuss the finer points of
> backlog tuning.
>  
> I've been experimenting with the multi-server socket+TCP megaunicorn
> configuration from your CDT:
> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000033.html
> 
> Which I think is what this sentence from TUNING is talking about?
>  
> 	"Setting a very low value for the :backlog parameter in ?listen?
> 	directives can allow failover to happen more quickly if your
> 	cluster is configured for it."

Yes.

<snip>

Thanks for sharing, and good to this is working well for you.

I'm still unlikely to have the chance to test this anywhere soon, but
maybe more folks can give it a try now that we've had one successful
report.   More reports (success or not) would definitely be good
to hear.

> BTW, this configuration needs a cool name!

Since you're the first person brave enough to try (or at least report
about it), you shall have the honor of naming it :)

-- 
Eric Wong

From snacktime at gmail.com  Tue Jun 22 14:03:20 2010
From: snacktime at gmail.com (snacktime)
Date: Tue, 22 Jun 2010 11:03:20 -0700
Subject: scaling unicorn
In-Reply-To: <20100622045346.GA23710@dcvr.yhbt.net>
References: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
	<20100622001632.GA10082@dcvr.yhbt.net>
	<F596AC52-4CD1-4543-9B04-7A19762995DC@tramchase.com>
	<20100622045346.GA23710@dcvr.yhbt.net>
Message-ID: <AANLkTilWNicXuA0N5ouCvt3Pt1Wl0Fu9PEX_GQApzQn4@mail.gmail.com>

>> Somewhat related -- I've been meaning to discuss the finer points of
>> backlog tuning.
>>
>> I've been experimenting with the multi-server socket+TCP megaunicorn
>> configuration from your CDT:
>> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000033.html

So I'm in the position of launching a web app in a couple of weeks
that is pretty much guaranteed to get huge traffic.  I'm working with
ops people who are very good but this is not how they would normally
setup load balancing and scale out.  I'm having a meeting with our
network ops lead tomorrow to talk about this.  I like the idea of this
approach, it seems like it gives you more fine grained control over
how much load you put on individual servers as well as how individual
requests are handled.  But I'm not too keen on using something like
this at scale when we simply don't have the chance to test it out at a
smaller scale.  I have yet to see anyone with this setup running at
scale.  That of course doesn't mean it's not a great idea, only that I
doubt our ops guys are going to want to be the first.  They are
already overworked as it is:)

So assuming we will scale out the 'normal' way by not having a short
backlog, any info on how to manage that?   Should we control the
backlog queue in nginx (not sure exactly how I would do that) or via
the listen backlog?  I was looking around last night and couldn't find
a way to actually poll the listen backlog queue size.

Also, any ideas on how you would practically manage this type of load
balancing setup?  Seems like you would have some type of 'reserve'
cluster for requests that hit the listen backlog, and when you start
seeing too much traffic going to the reserve, you add more servers to
your main pool.  How else would you manage the configuration for
something like this when you are working with 100 - 200 servers?  You
can't be changing the nginx configs every time you add servers, that's
just not practical.

Chris

From snacktime at gmail.com  Tue Jun 22 13:30:06 2010
From: snacktime at gmail.com (snacktime)
Date: Tue, 22 Jun 2010 10:30:06 -0700
Subject: [Mongrel] scaling unicorn
In-Reply-To: <20100622001632.GA10082@dcvr.yhbt.net>
References: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
	<20100622001632.GA10082@dcvr.yhbt.net>
Message-ID: <AANLkTimMAcSSt-ieRbgGqdSljsVwWx74h191V4QIuFUK@mail.gmail.com>

On Mon, Jun 21, 2010 at 5:16 PM, Eric Wong <normalperson at yhbt.net> wrote:
> snacktime <snacktime at gmail.com> wrote:
>> Interested in some feeback on this (does it sound right?), or maybe
>> this might be of interest to others.
>
> Hi Chris,
>
> I think you meant to post this to the mongrel-unicorn at rubyforge.org
> list, not mongrel-users at rubyforge.org :>
>
Yes, not sure how that got mixed up...


>
> That's "8 way xeon" _before_ hyperthreading, right? ?Which family of
> Xeons are you using, the Pentium4-based crap or the awesome new ones?
>
Two quad core Nehalems on each server.

> How much memory is each Unicorn worker using for your app?
>
Undoubtedly this is lower then it will be under a real load, but under
our load tests they stabilize at around 160mb.

> Do you have actual performance numbers you're able to share?
> Mean/median request times/rates would be very useful. ?If your requests
> run very quickly, you may be limited by contention with the accept()
> syscall on the listen socket, too.
>

I had two different types of requests to test that I did in varying
combinations.  One takes on average 600ms, and the other 40ms.  98% of
our requests will be the faster one.  Deviations were really low.

> I assume you're using nginx as the proxy, is this with Unix domain
> sockets or TCP sockets? ?Unix domain sockets should give a small
> performance over TCP if it's all on the same box.
>

Yes nginx with domain sockets.


Chris

From jamie at tramchase.com  Tue Jun 22 14:57:58 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Tue, 22 Jun 2010 11:57:58 -0700
Subject: scaling unicorn
In-Reply-To: <AANLkTilWNicXuA0N5ouCvt3Pt1Wl0Fu9PEX_GQApzQn4@mail.gmail.com>
References: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
	<20100622001632.GA10082@dcvr.yhbt.net>
	<F596AC52-4CD1-4543-9B04-7A19762995DC@tramchase.com>
	<20100622045346.GA23710@dcvr.yhbt.net>
	<AANLkTilWNicXuA0N5ouCvt3Pt1Wl0Fu9PEX_GQApzQn4@mail.gmail.com>
Message-ID: <9A7FCF99-DE2A-4D7F-A8ED-5B9302BD4B11@tramchase.com>

>> Somewhat related -- I've been meaning to discuss the finer points of
>> backlog tuning.
>> 
>> I've been experimenting with the multi-server socket+TCP megaunicorn
>> configuration from your CDT:
>> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000033.html

On Jun 22, 2010, at 11:03 AM, snacktime wrote:

> Seems like you would have some type of 'reserve'
> cluster for requests that hit the listen backlog, and when you start
> seeing too much traffic going to the reserve, you add more servers to
> your main pool.  How else would you manage the configuration for
> something like this when you are working with 100 - 200 servers?  You
> can't be changing the nginx configs every time you add servers, that's
> just not practical.

We are using chef for machine configuration which makes these kinds of  numbers doable
http://wiki.opscode.com/display/chef/Home

I would love to see a nginx module for distributed configuration mgmnt

Right now we are running 6 frontend machines, 4 in use & 2 in reserve like you described. We are doing about 5000rpm with this, almost all dynamic. 10-30% of requests might be 'slow' (1+s) depending on usage patterns. 

To measure health I am using munin to watch system load, nginx requests & nginx errors. In this configuration 502 Bad Gateways from frontend nginx indicate a busy unicorn socket & thus a handoff of the request to the backups. Then we measure the rails production.log for request counts + speed on each server as well as using NewRelic RPM

monit also emails us when 502s show up. 
In theory monit could  be automatically spinning up another backup server, provisioning it using chef, then reprovisioning the rest of the cluster to start handing over traffic. Alternately the new server could just act as backup for the one overloaded machine, which could make isolating performance issues easier.

-jamie


From jamie at tramchase.com  Tue Jun 22 15:18:00 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Tue, 22 Jun 2010 12:18:00 -0700
Subject: scaling unicorn
In-Reply-To: <20100622045346.GA23710@dcvr.yhbt.net>
References: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
	<20100622001632.GA10082@dcvr.yhbt.net>
	<F596AC52-4CD1-4543-9B04-7A19762995DC@tramchase.com>
	<20100622045346.GA23710@dcvr.yhbt.net>
Message-ID: <D8BC44A8-A904-487C-A4D5-75292430B016@tramchase.com>

On Jun 21, 2010, at 9:53 PM, Eric Wong wrote:

> Thanks for sharing, and good to this is working well for you.
> 
> I'm still unlikely to have the chance to test this anywhere soon, but
> maybe more folks can give it a try now that we've had one successful
> report.   More reports (success or not) would definitely be good
> to hear.
> 
>> BTW, this configuration needs a cool name!
> 
> Since you're the first person brave enough to try (or at least report
> about it), you shall have the honor of naming it :)

The all-knowing WikiAnswers says "a group of unicorns is a blessing" :)

http://wiki.answers.com/Q/What_is_a_group_of_Unicorns_called

Some great fan art out there:

http://www.elfwood.com/~ara-tun/Unicorn-Herd.2537340.html

But my coworkers & I are voting "pegacorn"

http://images.elfwood.com/art/m/i/michelle16/pegacorn.jpg

-jamie

From jamie at tramchase.com  Tue Jun 22 19:14:03 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Tue, 22 Jun 2010 16:14:03 -0700
Subject: mongrel_proctitle like support in Unicorn
In-Reply-To: <20100617232149.GB25862@dcvr.yhbt.net>
References: <ae96cae30912021352l170af773k275b14377138202c@mail.gmail.com>
	<944a03770912021404s189a3686sb8b024a21d5a7c7a@mail.gmail.com>
	<loom.20100618T002015-365@post.gmane.org>
	<20100617232149.GB25862@dcvr.yhbt.net>
Message-ID: <2BBAE655-2636-4B32-B854-50A8EB7CA2A1@tramchase.com>

On Jun 17, 2010, at 4:21 PM, Eric Wong wrote:

>>> http://coderack.org/users/arya/entries/3-rack-proctitle
>> 
>> FWIW this doesn't seem to work out-of-box with Unicorn. Eric, do you know
>> offhand if it's possible to use middleware to call unicorn's proc_name? I'd love
>> to be able to watch what my unicorns are doing in realtime
> 
> That coderack link no longer works for me, but I see no reason why
> Rack::ProcTitle in rack-contrib wouldn't work (based on my reading of
> the code):
> 
> http://github.com/rack/rack-contrib/blob/master/lib/rack/contrib/proctitle.rb

This works great . Thanks Eric.

For anyone else: I just dropped the above into lib/rack_proctitle.rb and added "use Rack::ProcTitle" in my Rails config.ru

Now me & my unicorns are like totally BFFs

-jamie

From me at craigdavey.ca  Wed Jun 23 01:57:00 2010
From: me at craigdavey.ca (Craig Davey)
Date: Wed, 23 Jun 2010 05:57:00 +0000 (UTC)
Subject: Purpose of "Status" header in HTTP responses?
Message-ID: <loom.20100623T073622-750@post.gmane.org>

Hi folks

On line #63 of unicorn/http_response.rb a "Status" header is written to the
socket. A comment in the code explains that some broken clients require this 
header and unicorn generously accommodates them.

We?re having the opposite problem. One of our clients using Microsoft Windows
and ASP haven?t been able to connect to our HTTP API since we moved it to 
unicorn from passenger. They receive the following error message when they 
try to connect to our servers:

msxml3.dll error '80072f78' server returned an invalid or unrecognized response

Our client thinks this error is caused by the "Status" header that is added 
to responses by unicorn. We don?t know of any other instances where this header 
is causing problems so we?re pretty confused about why it?s a problem for them.

Does anyone remember why this "Status" header was added to HttpResponse? Which 
broken clients was the change trying to accommodate?

Craig


From normalperson at yhbt.net  Wed Jun 23 05:07:31 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 23 Jun 2010 09:07:31 +0000
Subject: Purpose of "Status" header in HTTP responses?
In-Reply-To: <loom.20100623T073622-750@post.gmane.org>
References: <loom.20100623T073622-750@post.gmane.org>
Message-ID: <20100623090731.GA19233@dcvr.yhbt.net>

Craig Davey <me at craigdavey.ca> wrote:
> Hi folks
> 
> On line #63 of unicorn/http_response.rb a "Status" header is written
> to the socket. A comment in the code explains that some broken clients
> require this header and unicorn generously accommodates them.
> 
> We?re having the opposite problem. One of our clients using Microsoft
> Windows and ASP haven?t been able to connect to our HTTP API since we
> moved it to unicorn from passenger. They receive the following error
> message when they try to connect to our servers:
> 
> msxml3.dll error '80072f78' server returned an invalid or unrecognized
> response

Hi Craig,

Interesting and strange...

Looking at lib/phusion_passenger/rack/request_handler.rb (blob ad22dfa)
line 94, they also set the Status: header, too (but just the numeric
code, no text).

You can try "proxy_hide_header Status;" in your nginx config
to suppress it.


Another theory: You are running nginx in front of Unicorn, right?

If not (but you really should be), the lack of a Server header may throw
off some clients...

I also don't ever want folks to be forced to reveal they use which
server they use for security concerns, so Unicorn won't ever force the
Server: header on you.  And since nginx overwrites any Server header
Unicorn would set, Unicorn won't bother, either.  However, it's easy to
setup Rack middleware to write anything you want in the Server header.

rainbows.git (unreleased) allows using the Rainbows::ServerToken
middleware, and if you really need it, it should be easy to port to
Unicorn:

  http://git.bogomips.org/cgit/rainbows.git/tree/lib/rainbows/server_token.rb

> Our client thinks this error is caused by the "Status" header that is
> added to responses by unicorn. We don?t know of any other instances
> where this header is causing problems so we?re pretty confused about
> why it?s a problem for them.

Passenger also adds X-Powered-By, but that's completely non-standard and
probably used to get around proxies (like nginx) that overwrite the
standard Server: header.  You can also make middleware (or your app) add
that header, too, and even go as far to make Unicorn pretend to be
Passenger :>

> Does anyone remember why this "Status" header was added to
> HttpResponse? Which broken clients was the change trying to
> accommodate?

I seem to recall some JavaScript libraries relied on it at some point,
and possibly some versions of Firebug.  Maybe some browser plugins do,
too.  Some folks here with more experience on client-side stuff ought
to chime in, since I generally stay away from GUI/DOM things.

However, even with my lack of JS experience (or because of) I realize
it's very easy to fall into the trap of writing JavaScript that relies
on the Status: header.  The Status: header has been with us as a
de-facto standard since the CGI days.  Older cgi.rb-based versions of
Rails set it, too.

-- 
Eric Wong

From normalperson at yhbt.net  Wed Jun 23 05:32:35 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 23 Jun 2010 09:32:35 +0000
Subject: scaling unicorn
In-Reply-To: <AANLkTilWNicXuA0N5ouCvt3Pt1Wl0Fu9PEX_GQApzQn4@mail.gmail.com>
References: <AANLkTilv4e_DPDKy440xotrlE7ucFIFXs74uHyGrzCKL@mail.gmail.com>
	<20100622001632.GA10082@dcvr.yhbt.net>
	<F596AC52-4CD1-4543-9B04-7A19762995DC@tramchase.com>
	<20100622045346.GA23710@dcvr.yhbt.net>
	<AANLkTilWNicXuA0N5ouCvt3Pt1Wl0Fu9PEX_GQApzQn4@mail.gmail.com>
Message-ID: <20100623093235.GB19233@dcvr.yhbt.net>

snacktime <snacktime at gmail.com> wrote:
> >> Somewhat related -- I've been meaning to discuss the finer points of
> >> backlog tuning.
> >>
> >> I've been experimenting with the multi-server socket+TCP megaunicorn
> >> configuration from your CDT:
> >> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000033.html
> 
> So I'm in the position of launching a web app in a couple of weeks
> that is pretty much guaranteed to get huge traffic.  I'm working with
> ops people who are very good but this is not how they would normally
> setup load balancing and scale out.  I'm having a meeting with our
> network ops lead tomorrow to talk about this.  I like the idea of this
> approach, it seems like it gives you more fine grained control over
> how much load you put on individual servers as well as how individual
> requests are handled.  But I'm not too keen on using something like
> this at scale when we simply don't have the chance to test it out at a
> smaller scale.  I have yet to see anyone with this setup running at
> scale.  That of course doesn't mean it's not a great idea, only that I
> doubt our ops guys are going to want to be the first.  They are
> already overworked as it is:)

No worries.  Don't ever feel obligated to try something you're not
comfortable with.  Heck, it took months before anybody besides myself
was comfortable with Unicorn.

> So assuming we will scale out the 'normal' way by not having a short
> backlog, any info on how to manage that?   Should we control the
> backlog queue in nginx (not sure exactly how I would do that) or via
> the listen backlog?  I was looking around last night and couldn't find
> a way to actually poll the listen backlog queue size.

nginx lets you specify a backlog=num with the "listen" directive
much like Unicorn does (Unicorn steals most configuration parameter
names/options from nginx):

  http://wiki.nginx.org/NginxHttpCoreModule#listen

If you use Linux, you can poll the current listen queue
using Raindrops (http://raindrops.bogomips.org/), the ss(8) utility,
or parsing /proc/net/tcp and/or /proc/net/unix.  Unfortunately,
checking the listen queue for Unix domain sockets is expensive,
Raindrops and ss(8) both need to parse /proc/net/unix because
that info isn't available via netlink.

> Also, any ideas on how you would practically manage this type of load
> balancing setup?  Seems like you would have some type of 'reserve'
> cluster for requests that hit the listen backlog, and when you start
> seeing too much traffic going to the reserve, you add more servers to
> your main pool.  How else would you manage the configuration for
> something like this when you are working with 100 - 200 servers?  You
> can't be changing the nginx configs every time you add servers, that's
> just not practical.

I've never tried this setup, so what Jamie said :)

One extra note, 100-200 hosts in an upstream {} block makes a very long
nginx config file.  You could use ERB or something else to template,
but based on a previous reading of the nginx source code, you can
also setup a round-robin DNS entry for all the servers.

nginx only does DNS lookups for upstreams at load time.  For round-robin
DNS entries, nginx adds an entry for every IP address a name resolves
to, so just specify the one DNS name in the upstream block instead of
the list of IP(s).

Just remember to HUP the nginxes (or if you're forgetful, make an
occasional cronjob to HUP them) when you make DNS changes and add/remove
a box.

-- 
Eric Wong

From uzugbo1 at btinternet.com  Wed Jun 23 09:41:39 2010
From: uzugbo1 at btinternet.com (DR. BEN UZUGDO.)
Date: Wed, 23 Jun 2010 15:41:39 +0200
Subject: =?iso-8859-1?Q?Re:_Information=92s_regarding_your_fund.?=
Message-ID: <cc22adff01e401cf8826a2a176f4b533@julietta-barrientos.de>

================================
================================

Attention: Sir/Madam,

This message is to inform you regarding your payment; this is to inform you that Central Bank of Nigeria (CBN) has to open an account for you and paid your Funds into your account.

Your fund is ready, kindly contact me and forward to me your full name, account details and your full address where your fund will be remitted to you, and also your cell phone number for discussion.

Please contact officer/Central Bank of Nigeria immediately with the required information:-

Best Regards, 

Yours Sincerely,
DR. BEN UZUGDO.
PUBLIC RELATION DIRECTOR
CENTRAL BANK OF NIGERIA.

================================
================================

From me at craigdavey.ca  Thu Jun 24 10:56:31 2010
From: me at craigdavey.ca (Craig Davey)
Date: Thu, 24 Jun 2010 14:56:31 +0000 (UTC)
Subject: Purpose of &quot;Status&quot; header in HTTP responses?
References: <loom.20100623T073622-750@post.gmane.org>
	<20100623090731.GA19233@dcvr.yhbt.net>
Message-ID: <loom.20100624T165510-593@post.gmane.org>

Eric, thanks for the detailed reply and the proxy_hide_header tip. 

We used proxy_hide_header to remove the Status header from our responses. 
After the change our client continued to receive the same error so I?m
pretty sure that the Status header was never the problem in this case. 
Sorry for raising this issue unnecessarily. Thanks again for your help.

Craig


From andrewmilkowski at gmail.com  Fri Jun 25 11:06:42 2010
From: andrewmilkowski at gmail.com (Andrew Milkowski)
Date: Fri, 25 Jun 2010 11:06:42 -0400
Subject: test failure on Mac OSX
Message-ID: <AANLkTimu7M4wWrVxSijwu_qJI4b0JTkYlhl25eDHHu8W@mail.gmail.com>

Hi there,

ran to a snag building unicorn from sources (git master at
git://git.bogomips.org/unicorn.git)

  1) Failure:
: test_working_directory_rel_path_config_file(ExecTest)
[test/exec/test_exec.rb:106]:
: <"/var/folders/gy/gyjECZCGGxKv3I3mRTmhB++++TI/-Tmp-/unicorn_exec_test20100625-1679-1x3ks9d-0">
expected but was
: <"/private/var/folders/gy/gyjECZCGGxKv3I3mRTmhB++++TI/-Tmp-/unicorn_exec_test20100625-1679-1x3ks9d-0">.
:
: 1 tests, 2 assertions, 1 failures, 0 errors
make: *** [test/exec/test_exec.rb--test_working_directory_rel_path_config_file.n]
Error 1

running following configuration (session console dump below)

bash-3.2$ uname -v
Darwin Kernel Version 10.4.0: Fri Apr 23 18:27:12 PDT 2010;
root:xnu-1504.7.4~1/RELEASE_X86_64
bash-3.2$ ruby -v
ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-darwin10.2.0], MBARI
0x6770, Ruby Enterprise Edition 2010.01
bash-3.2$


thanks in advance!


bash-3.2$ make
* test/unit/test_configurator.rb
* test/unit/test_http_parser.rb
* test/unit/test_http_parser_ng.rb
* test/unit/test_request.rb
* test/unit/test_response.rb
* test/unit/test_socket_helper.rb
* test/unit/test_tee_input.rb
* test/unit/test_util.rb
* test/unit/test_server.rb -n test_preload_app_config
* test/unit/test_server.rb -n test_broken_app
* test/unit/test_server.rb -n test_simple_server
* test/unit/test_server.rb -n test_client_shutdown_writes
* test/unit/test_server.rb -n test_client_shutdown_write_truncates
* test/unit/test_server.rb -n test_client_malformed_body
* test/unit/test_server.rb -n test_trickle_attack
* test/unit/test_server.rb -n test_close_client
* test/unit/test_server.rb -n test_bad_client
* test/unit/test_server.rb -n test_logger_set
* test/unit/test_server.rb -n test_logger_changed
* test/unit/test_server.rb -n test_bad_client_400
* test/unit/test_server.rb -n test_http_0_9
* test/unit/test_server.rb -n test_header_is_too_long
* test/unit/test_server.rb -n test_file_streamed_request
* test/unit/test_server.rb -n test_file_streamed_request_bad_body
* test/unit/test_server.rb -n test_listener_names
* test/exec/test_exec.rb -n test_working_directory_rel_path_config_file
: /opt/local/src/ruby-enterprise-server/ruby-enterprise-1.8.7-2010.01/build/lib/ruby/1.8/pathname.rb:263:
warning: `*' interpreted as argument prefix
: Loaded suite test/exec/test_exec
: Started
: test_working_directory_rel_path_config_file(ExecTest): F
:
: Finished in 0.629278 seconds.
:
:   1) Failure:
: test_working_directory_rel_path_config_file(ExecTest)
[test/exec/test_exec.rb:106]:
: <"/var/folders/gy/gyjECZCGGxKv3I3mRTmhB++++TI/-Tmp-/unicorn_exec_test20100625-1679-1x3ks9d-0">
expected but was
: <"/private/var/folders/gy/gyjECZCGGxKv3I3mRTmhB++++TI/-Tmp-/unicorn_exec_test20100625-1679-1x3ks9d-0">.
:
: 1 tests, 2 assertions, 1 failures, 0 errors
make: *** [test/exec/test_exec.rb--test_working_directory_rel_path_config_file.n]
Error 1
bash-3.2$

From normalperson at yhbt.net  Fri Jun 25 14:35:18 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 25 Jun 2010 11:35:18 -0700
Subject: test failure on Mac OSX
In-Reply-To: <AANLkTimu7M4wWrVxSijwu_qJI4b0JTkYlhl25eDHHu8W@mail.gmail.com>
References: <AANLkTimu7M4wWrVxSijwu_qJI4b0JTkYlhl25eDHHu8W@mail.gmail.com>
Message-ID: <20100625183518.GA32646@dcvr.yhbt.net>

Andrew Milkowski <andrewmilkowski at gmail.com> wrote:
> Hi there,
> 
> ran to a snag building unicorn from sources (git master at
> git://git.bogomips.org/unicorn.git)
> 
>   1) Failure:
> : test_working_directory_rel_path_config_file(ExecTest)
> [test/exec/test_exec.rb:106]:
> : <"/var/folders/gy/gyjECZCGGxKv3I3mRTmhB++++TI/-Tmp-/unicorn_exec_test20100625-1679-1x3ks9d-0">
> expected but was
> : <"/private/var/folders/gy/gyjECZCGGxKv3I3mRTmhB++++TI/-Tmp-/unicorn_exec_test20100625-1679-1x3ks9d-0">.

Hi Andrew,

Not being an OSX user, I had to look this up, but apparently /var on OSX
is a symlink to /private/var?  Weird...

We have to work around a similar issue for upgrades in Capistrano
deployments by updating ENV['PWD'], might as well do it here.

I've just pushed out the following patch, which is a more thorough test
anyways:

>From cf63db66bca9acfd3416ab8fc8a7fd4f07927342 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Fri, 25 Jun 2010 11:29:13 -0700
Subject: [PATCH] test-exec: prefer ENV['PWD'] in working_directory tests

We do an extra check in the application dispatch to ensure
ENV['PWD'] is set correctly to match Dir.pwd (even if the
string path is different) as this is required for Capistrano
deployments.

These tests should now pass under OSX where /var is apparently
a symlink to /private/var.
---
 test/exec/test_exec.rb |   29 +++++++++++++++++------------
 1 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 9830683..1d24ca3 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -54,6 +54,20 @@ before_fork do |server, worker|
 end
   EOS
 
+  WORKING_DIRECTORY_CHECK_RU = <<-EOS
+use Rack::ContentLength
+run lambda { |env|
+  pwd = ENV['PWD']
+  a = ::File.stat(pwd)
+  b = ::File.stat(Dir.pwd)
+  if (a.ino == b.ino && a.dev == b.dev)
+    [ 200, { 'Content-Type' => 'text/plain' }, [ pwd ] ]
+  else
+    [ 404, { 'Content-Type' => 'text/plain' }, [] ]
+  end
+}
+  EOS
+
   def setup
     @pwd = Dir.pwd
     @tmpfile = Tempfile.new('unicorn_exec_test')
@@ -87,10 +101,7 @@ end
     File.unlink(other.path)
     Dir.mkdir(other.path)
     File.open("config.ru", "wb") do |fp|
-      fp.syswrite <<EOF
-use Rack::ContentLength
-run proc { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ Dir.pwd ] ] }
-EOF
+      fp.syswrite WORKING_DIRECTORY_CHECK_RU
     end
     FileUtils.cp("config.ru", other.path + "/config.ru")
     Dir.chdir(@tmpdir)
@@ -138,10 +149,7 @@ EOF
     File.unlink(other.path)
     Dir.mkdir(other.path)
     File.open("config.ru", "wb") do |fp|
-      fp.syswrite <<EOF
-use Rack::ContentLength
-run proc { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ Dir.pwd ] ] }
-EOF
+      fp.syswrite WORKING_DIRECTORY_CHECK_RU
     end
     FileUtils.cp("config.ru", other.path + "/config.ru")
     tmp = Tempfile.new('unicorn.config')
@@ -177,10 +185,7 @@ EOF
     File.unlink(other.path)
     Dir.mkdir(other.path)
     File.open("config.ru", "wb") do |fp|
-      fp.syswrite <<EOF
-use Rack::ContentLength
-run proc { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ Dir.pwd ] ] }
-EOF
+      fp.syswrite WORKING_DIRECTORY_CHECK_RU
     end
     FileUtils.cp("config.ru", other.path + "/config.ru")
     system('mkfifo', "#{other.path}/fifo")
-- 
Eric Wong

From andrewmilkowski at gmail.com  Fri Jun 25 19:51:01 2010
From: andrewmilkowski at gmail.com (Andrew Milkowski)
Date: Fri, 25 Jun 2010 19:51:01 -0400
Subject: test failure on Mac OSX
In-Reply-To: <20100625183518.GA32646@dcvr.yhbt.net>
References: <AANLkTimu7M4wWrVxSijwu_qJI4b0JTkYlhl25eDHHu8W@mail.gmail.com>
	<20100625183518.GA32646@dcvr.yhbt.net>
Message-ID: <AANLkTil7W1AGZOuOK0jrtPm11HO_JvbtlYZKJ2aoyG3D@mail.gmail.com>

Thanks Eric!

wonderful, all test passed (with very minimal CPU strain) I also
enabled full 64 bit kernel so that helps I am sure.

and yes, very true var is symlinked under 10.6 (not sure if this was
the case <)

thanks that was wonderful weekend gift (I am now unblocked) and try
other things with this container

have a wonderful weekend!

bash-3.2$ ls -l var
lrwxr-xr-x@ 1 root  wheel  11 Feb  1 18:44 var -> private/var
bash-3.2$


bash-3.2$ make
* test/unit/test_configurator.rb
* test/unit/test_http_parser.rb
* test/unit/test_http_parser_ng.rb
* test/unit/test_request.rb
* test/unit/test_response.rb
* test/unit/test_socket_helper.rb
* test/unit/test_tee_input.rb
* test/unit/test_util.rb
* test/unit/test_server.rb -n test_preload_app_config
* test/unit/test_server.rb -n test_broken_app
* test/unit/test_server.rb -n test_simple_server
* test/unit/test_server.rb -n test_client_shutdown_writes
* test/unit/test_server.rb -n test_client_shutdown_write_truncates
* test/unit/test_server.rb -n test_client_malformed_body
* test/unit/test_server.rb -n test_trickle_attack
* test/unit/test_server.rb -n test_close_client
* test/unit/test_server.rb -n test_bad_client
* test/unit/test_server.rb -n test_logger_set
* test/unit/test_server.rb -n test_logger_changed
* test/unit/test_server.rb -n test_bad_client_400
* test/unit/test_server.rb -n test_http_0_9
* test/unit/test_server.rb -n test_header_is_too_long
* test/unit/test_server.rb -n test_file_streamed_request
* test/unit/test_server.rb -n test_file_streamed_request_bad_body
* test/unit/test_server.rb -n test_listener_names
* test/exec/test_exec.rb -n test_working_directory_rel_path_config_file
* test/exec/test_exec.rb -n test_working_directory
* test/exec/test_exec.rb -n test_working_directory_controls_relative_paths
* test/exec/test_exec.rb -n test_exit_signals
* test/exec/test_exec.rb -n test_basic
* test/exec/test_exec.rb -n test_rack_env_unset
* test/exec/test_exec.rb -n test_rack_env_cli_set
* test/exec/test_exec.rb -n test_rack_env_ENV_set
* test/exec/test_exec.rb -n test_rack_env_cli_override_ENV
* test/exec/test_exec.rb -n test_ttin_ttou
* test/exec/test_exec.rb -n test_help
* test/exec/test_exec.rb -n test_broken_reexec_config
* test/exec/test_exec.rb -n test_broken_reexec_ru
* test/exec/test_exec.rb -n test_unicorn_config_listener_swap
* test/exec/test_exec.rb -n test_unicorn_config_listen_with_options
* test/exec/test_exec.rb -n test_unicorn_config_per_worker_listen
* test/exec/test_exec.rb -n test_unicorn_config_listen_augments_cli
* test/exec/test_exec.rb -n test_weird_config_settings
* test/exec/test_exec.rb -n test_read_embedded_cli_switches
* test/exec/test_exec.rb -n test_config_ru_alt_path
* test/exec/test_exec.rb -n test_load_module
* test/exec/test_exec.rb -n test_reexec
* test/exec/test_exec.rb -n test_reexec_alt_config
* test/exec/test_exec.rb -n test_socket_unlinked_restore
* test/exec/test_exec.rb -n test_unicorn_config_file
* test/exec/test_exec.rb -n test_daemonize_reexec
* test/exec/test_exec.rb -n test_daemonize_redirect_fail
* test/exec/test_exec.rb -n test_reexec_fd_leak
* test/exec/test_exec.rb -n test_preload_app_hup
* test/exec/test_exec.rb -n test_hup
* test/exec/test_exec.rb -n test_default_listen_hup_holds_listener
* test/exec/test_exec.rb -n test_default_listen_upgrade_holds_listener
* test/unit/test_signals.rb -n test_worker_dies_on_dead_master
* test/unit/test_signals.rb -n test_sleepy_kill
* test/unit/test_signals.rb -n test_timeout_slow_response
* test/unit/test_signals.rb -n test_response_write
* test/unit/test_signals.rb -n test_request_read
* test/unit/test_upload.rb -n test_put
* test/unit/test_upload.rb -n test_put_content_md5
* test/unit/test_upload.rb -n test_put_trickle_small
* test/unit/test_upload.rb -n test_put_keepalive_truncates_small_overwrite
* test/unit/test_upload.rb -n test_put_excessive_overwrite_closed
* test/unit/test_upload.rb -n test_uncomfortable_with_onenine_encodings
* test/unit/test_upload.rb -n test_chunked_upload_via_curl
* test/unit/test_upload.rb -n test_curl_chunked_small

179 tests, 63093 assertions, 0 failures, 0 errors
bash-3.2$


On Fri, Jun 25, 2010 at 2:35 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Andrew Milkowski <andrewmilkowski at gmail.com> wrote:
>> Hi there,
>>
>> ran to a snag building unicorn from sources (git master at
>> git://git.bogomips.org/unicorn.git)
>>
>> ? 1) Failure:
>> : test_working_directory_rel_path_config_file(ExecTest)
>> [test/exec/test_exec.rb:106]:
>> : <"/var/folders/gy/gyjECZCGGxKv3I3mRTmhB++++TI/-Tmp-/unicorn_exec_test20100625-1679-1x3ks9d-0">
>> expected but was
>> : <"/private/var/folders/gy/gyjECZCGGxKv3I3mRTmhB++++TI/-Tmp-/unicorn_exec_test20100625-1679-1x3ks9d-0">.
>
> Hi Andrew,
>
> Not being an OSX user, I had to look this up, but apparently /var on OSX
> is a symlink to /private/var? ?Weird...
>
> We have to work around a similar issue for upgrades in Capistrano
> deployments by updating ENV['PWD'], might as well do it here.
>
> I've just pushed out the following patch, which is a more thorough test
> anyways:
>
> >From cf63db66bca9acfd3416ab8fc8a7fd4f07927342 Mon Sep 17 00:00:00 2001
> From: Eric Wong <normalperson at yhbt.net>
> Date: Fri, 25 Jun 2010 11:29:13 -0700
> Subject: [PATCH] test-exec: prefer ENV['PWD'] in working_directory tests
>
> We do an extra check in the application dispatch to ensure
> ENV['PWD'] is set correctly to match Dir.pwd (even if the
> string path is different) as this is required for Capistrano
> deployments.
>
> These tests should now pass under OSX where /var is apparently
> a symlink to /private/var.
> ---
> ?test/exec/test_exec.rb | ? 29 +++++++++++++++++------------
> ?1 files changed, 17 insertions(+), 12 deletions(-)
>
> diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
> index 9830683..1d24ca3 100644
> --- a/test/exec/test_exec.rb
> +++ b/test/exec/test_exec.rb
> @@ -54,6 +54,20 @@ before_fork do |server, worker|
> ?end
> ? EOS
>
> + ?WORKING_DIRECTORY_CHECK_RU = <<-EOS
> +use Rack::ContentLength
> +run lambda { |env|
> + ?pwd = ENV['PWD']
> + ?a = ::File.stat(pwd)
> + ?b = ::File.stat(Dir.pwd)
> + ?if (a.ino == b.ino && a.dev == b.dev)
> + ? ?[ 200, { 'Content-Type' => 'text/plain' }, [ pwd ] ]
> + ?else
> + ? ?[ 404, { 'Content-Type' => 'text/plain' }, [] ]
> + ?end
> +}
> + ?EOS
> +
> ? def setup
> ? ? @pwd = Dir.pwd
> ? ? @tmpfile = Tempfile.new('unicorn_exec_test')
> @@ -87,10 +101,7 @@ end
> ? ? File.unlink(other.path)
> ? ? Dir.mkdir(other.path)
> ? ? File.open("config.ru", "wb") do |fp|
> - ? ? ?fp.syswrite <<EOF
> -use Rack::ContentLength
> -run proc { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ Dir.pwd ] ] }
> -EOF
> + ? ? ?fp.syswrite WORKING_DIRECTORY_CHECK_RU
> ? ? end
> ? ? FileUtils.cp("config.ru", other.path + "/config.ru")
> ? ? Dir.chdir(@tmpdir)
> @@ -138,10 +149,7 @@ EOF
> ? ? File.unlink(other.path)
> ? ? Dir.mkdir(other.path)
> ? ? File.open("config.ru", "wb") do |fp|
> - ? ? ?fp.syswrite <<EOF
> -use Rack::ContentLength
> -run proc { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ Dir.pwd ] ] }
> -EOF
> + ? ? ?fp.syswrite WORKING_DIRECTORY_CHECK_RU
> ? ? end
> ? ? FileUtils.cp("config.ru", other.path + "/config.ru")
> ? ? tmp = Tempfile.new('unicorn.config')
> @@ -177,10 +185,7 @@ EOF
> ? ? File.unlink(other.path)
> ? ? Dir.mkdir(other.path)
> ? ? File.open("config.ru", "wb") do |fp|
> - ? ? ?fp.syswrite <<EOF
> -use Rack::ContentLength
> -run proc { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ Dir.pwd ] ] }
> -EOF
> + ? ? ?fp.syswrite WORKING_DIRECTORY_CHECK_RU
> ? ? end
> ? ? FileUtils.cp("config.ru", other.path + "/config.ru")
> ? ? system('mkfifo', "#{other.path}/fifo")
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>

From sgoetz58 at gmail.com  Wed Jun 30 15:42:25 2010
From: sgoetz58 at gmail.com (Sharon Massey)
Date: Wed, 30 Jun 2010 14:42:25 -0500
Subject: =?utf-8?Q?God_bless_you?=
Message-ID: <4c2b9e21795de@gmail.com>

My Dear,
 With Due Respect and Humanity, I was compelled to write to you under a
humanitarian ground, however, this is not mandatory nor will I in any manner
compel you to honour against your will, believing you will be honest and
trustworthy for the sake of this secret I am releasing to you. I believe you
will not betray the confidence I have reposed in you, after going through
your profile.
Though i was lead in the spirit to write you, for me to have contacted you
even though we've not met or known each other before.
 
Please I am contacting you to let you know my desire to establish a charity
foundation in your country with this sum of ? 3.8Million (Three Million
Eight hundred thousand Euros) which I inherited from my late husband. 
 
It is my desire to see that this money is invested to any organization of
your choice in your country and distributed each year among the charity
organizations, motherless baby?s home, mosques, churches, Schools,
supporting destitute aged men and women or whatever you may have in mind
that will be to the benefit of the less fortunate.
 
I took this decision because I was raised from a motherless baby?s home
and presently, I'm hospitalized here in (London) where I am undergoing
treatment for my up coming breast cancer surgery operation. I am suffering
from a long time cancer of the breast,from all indication my conditions is
really deteriorating and it is quite obvious that i can't work or do any
stressful thing, according to my doctors they have advised me that i may not
live for the next two months,this is because the cancer stage has gotten to
a very bad stage.
 
All i need from you is a confidential assurance that the funds when
recieved by you will be used for the said purpose. it is diffcult for me but
i dont know why the my spirit approved me to ask for your help in
establishing my will.
Please reply as soon as possible, confirming your acceptance to this
proposition so that I will give you all the relevant information that will
authorize the release and transfer of the fund to you as my duly assigned
representative.
 
 I look forward hearing from you.
Regards,
Mrs Saron G. Massey (sharongoetz58 at aol.co.uk)