From kamil.kukura at gmail.com  Wed Dec  2 09:36:20 2009
From: kamil.kukura at gmail.com (Kamil Kukura)
Date: Wed, 02 Dec 2009 15:36:20 +0100
Subject: Restart with HUP signal can't bind listener
Message-ID: <hf5u15$lkj$1@ger.gmane.org>

I have "preload_app false" in my config, but when trying to restart 
workers with sending HUP to master process Im getting (4 times) the error:

ERROR -- : retrying in 0.5 seconds (4 tries left)
ERROR -- : adding listener failed addr=0.0.0.0:8080 (in use)
...
...

Complete config file to see here: http://gist.github.com/247235


Thanks in advance,

-- 
Kamil


From jhahn at rbmtechnologies.com  Wed Dec  2 16:52:04 2009
From: jhahn at rbmtechnologies.com (Justin Hahn)
Date: Wed, 2 Dec 2009 16:52:04 -0500
Subject: mongrel_proctitle like support in Unicorn
Message-ID: <ae96cae30912021352l170af773k275b14377138202c@mail.gmail.com>

Before I go off ans try cutting my own horrible monkey patch to
unicorn, has anyone tried implementing something like
mongrel_proctitle for Unicorn?

We've been playing with Unicorn in our test environments and it looks
very promising, but something we're really addicted to is the ability
to see what each mongrel is doing "Right Now" via top and ps.

I don't think this would be terribly hard to add to Unicorn, but I
figured I'd ask the public list before reinventing the wheel.

---
Justin Hahn
Systems Architect

RBM Technologies, Inc.
25 Mt. Auburn St  ?  Cambridge, MA 02138
call  (617) 576-1234
visit www.rbmtechnologies.com

From mguterl at gmail.com  Wed Dec  2 17:04:20 2009
From: mguterl at gmail.com (Michael Guterl)
Date: Wed, 2 Dec 2009 17:04:20 -0500
Subject: mongrel_proctitle like support in Unicorn
In-Reply-To: <ae96cae30912021352l170af773k275b14377138202c@mail.gmail.com>
References: <ae96cae30912021352l170af773k275b14377138202c@mail.gmail.com>
Message-ID: <944a03770912021404s189a3686sb8b024a21d5a7c7a@mail.gmail.com>

On Wed, Dec 2, 2009 at 4:52 PM, Justin Hahn <jhahn at rbmtechnologies.com> wrote:
> Before I go off ans try cutting my own horrible monkey patch to
> unicorn, has anyone tried implementing something like
> mongrel_proctitle for Unicorn?
>
> We've been playing with Unicorn in our test environments and it looks
> very promising, but something we're really addicted to is the ability
> to see what each mongrel is doing "Right Now" via top and ps.
>
> I don't think this would be terribly hard to add to Unicorn, but I
> figured I'd ask the public list before reinventing the wheel.
>
There's some rack middleware out there, RackProctitle, which provides
this functionality.  I have not used it with Unicorn, but I don't see
why it wouldn't work.

http://coderack.org/users/arya/entries/3-rack-proctitle

Best regards,
Michael Guterl

From normalperson at yhbt.net  Wed Dec  2 17:28:12 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 2 Dec 2009 14:28:12 -0800
Subject: Restart with HUP signal can't bind listener
In-Reply-To: <hf5u15$lkj$1@ger.gmane.org>
References: <hf5u15$lkj$1@ger.gmane.org>
Message-ID: <20091202222812.GA25556@dcvr.yhbt.net>

Kamil Kukura <kamil.kukura at gmail.com> wrote:
> I have "preload_app false" in my config, but when trying to restart  
> workers with sending HUP to master process Im getting (4 times) the 
> error:
>
> ERROR -- : retrying in 0.5 seconds (4 tries left)
> ERROR -- : adding listener failed addr=0.0.0.0:8080 (in use)
> ...
> ...
>
> Complete config file to see here: http://gist.github.com/247235
>
>
> Thanks in advance,

Hi Kamil,

I can't seem to reproduce this.  Is anything else taking up port
8080 on your box?  Which version of Unicorn is this?

Is there any chance you somehow got two Unicorn masters running?

If you're on Linux, running "netstat -anp | grep :8080" can tell you
what PID is hogging up port 8080.

-- 
Eric Wong

From hukl at berlin.ccc.de  Mon Dec  7 03:51:51 2009
From: hukl at berlin.ccc.de (John-Paul Bader)
Date: Mon, 7 Dec 2009 09:51:51 +0100
Subject: FreeBSD rc script
Message-ID: <9D2B34EC-FDF7-435D-A9D9-26AAB5C037F8@berlin.ccc.de>

Hey Eric,

would you be interested in including a FreeBSD rc.d script ? Here is what my current one looks like:

in /usr/local/etc/rc.d/unicorn 

#!/bin/sh
#
# PROVIDE: unicorn

. /etc/rc.subr

name=unicorn
rcvar=`set_rcvar`

command=/usr/local/bin/unicorn_rails
command_args="-c /usr/local/etc/unicorn.rb -E production -D"

load_rc_config ${name}

: ${unicorn_dir:=/usr/local/www/}

procname=ruby18

unicorn_chdir=${unicorn_dir}
required_dirs=${unicorn_dir}

extra_commands=reload
sig_reload=USR2

run_rc_command $1


and in /etc/rc.conf

unicorn_enable="YES"
unicorn_dir="/usr/local/www/myprojectroot"


This loads unicorn at startup, gives the usual start stop and restart commands and has a reload command which sends USR2 to the master process.

Kind regards, John

From normalperson at yhbt.net  Mon Dec  7 04:50:27 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 7 Dec 2009 09:50:27 +0000
Subject: FreeBSD rc script
In-Reply-To: <9D2B34EC-FDF7-435D-A9D9-26AAB5C037F8@berlin.ccc.de>
References: <9D2B34EC-FDF7-435D-A9D9-26AAB5C037F8@berlin.ccc.de>
Message-ID: <20091207095027.GA6725@dcvr.yhbt.net>

John-Paul Bader <hukl at berlin.ccc.de> wrote:
> Hey Eric,
> 
> would you be interested in including a FreeBSD rc.d script ? Here is
> what my current one looks like:

Hi Paul,

Sure, a few comments below.

> command=/usr/local/bin/unicorn_rails
> command_args="-c /usr/local/etc/unicorn.rb -E production -D"

<snip>

> extra_commands=reload
> sig_reload=USR2

<snip>

> This loads unicorn at startup, gives the usual start stop and restart
> commands and has a reload command which sends USR2 to the master
> process.

Since it's using a Unicorn config file with USR2, wouldn't it have to
use QUIT to complete the actual upgrade?

Not too knowledgeable with FreeBSD init scripts on my end here,
but I'm guessing at how it'd work:

	extra_commands="reexec graceful reload"
	sig_reexec=USR2
	sig_graceful=QUIT
	sig_reload=HUP

	...

-- 
Eric Wong

From normalperson at yhbt.net  Mon Dec  7 05:01:22 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 7 Dec 2009 10:01:22 +0000
Subject: [ANN] unicorn 0.95.2 - small HTTP parser fixes
Message-ID: <20091207100122.GA8688@dcvr.yhbt.net>

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* email: mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git
* finger: unicorn at bogomips.org

Changes:

Small fixes to our HTTP parser to allows semicolons in PATH_INFO
as allowed by RFC 2396, section 3.3.  This is low impact for
existing apps as semicolons are rarely seen in URIs.  Our HTTP
parser runs properly under Rubinius 0.13.0 and 1.0.0-rc1 again
(though not yet the rest of the server since we rely heavily on
signals).

Another round of small documentation tweaks and minor cleanups.

-- 
Eric Wong

From kamil.kukura at gmail.com  Tue Dec  8 18:35:33 2009
From: kamil.kukura at gmail.com (Kamil Kukura)
Date: Wed, 09 Dec 2009 00:35:33 +0100
Subject: Restart with HUP signal can't bind listener
In-Reply-To: <20091202222812.GA25556@dcvr.yhbt.net>
References: <hf5u15$lkj$1@ger.gmane.org> <20091202222812.GA25556@dcvr.yhbt.net>
Message-ID: <hfmns5$v20$1@ger.gmane.org>

It is running in FreeBSD jail so I finally figured out I had to bind to 
jail's specific IP address, that is in my case:
listen "10.100.10.3:8080"

Dne 2.12.09 23:28, Eric Wong napsal(a):
> I can't seem to reproduce this.  Is anything else taking up port
> 8080 on your box?  Which version of Unicorn is this?
>
> Is there any chance you somehow got two Unicorn masters running?
>
> If you're on Linux, running "netstat -anp | grep :8080" can tell you
> what PID is hogging up port 8080.
>



From jan.dvorak at kraxnet.cz  Wed Dec  9 16:05:21 2009
From: jan.dvorak at kraxnet.cz (Jan Dvorak)
Date: Wed, 09 Dec 2009 22:05:21 +0100
Subject: Expect 100-continue errors
Message-ID: <4B201111.2030701@kraxnet.cz>

Hello,

i'm trying to implement handling of http Expect: 100-continue with
unicorn in my rails app, but when i return [100] as a response (which
should force unicorn to return 100 status and rerun my app with the same
connection) i get this error:

Read error: #<ThreadError: stopping only thread
    note: use sleep to stop forever>
/var/lib/gems/1.8/gems/actionpack-2.3.5/lib/action_controller/reloader.rb:31:in
`lock'
/var/lib/gems/1.8/gems/actionpack-2.3.5/lib/action_controller/reloader.rb:31:in
`run'
/var/lib/gems/1.8/gems/actionpack-2.3.5/lib/action_controller/dispatcher.rb:108:in
`call'
/var/lib/gems/1.8/gems/rails-2.3.5/lib/rails/rack/static.rb:31:in `call'
/var/lib/gems/1.8/gems/rack-1.0.1/lib/rack/urlmap.rb:46:in `call'
/var/lib/gems/1.8/gems/rack-1.0.1/lib/rack/urlmap.rb:40:in `each'
/var/lib/gems/1.8/gems/rack-1.0.1/lib/rack/urlmap.rb:40:in `call'
/var/lib/gems/1.8/gems/rails-2.3.5/lib/rails/rack/log_tailer.rb:17:in `call'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:576:in `process_client'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:643:in `worker_loop'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:641:in `each'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:641:in `worker_loop'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:534:in
`spawn_missing_workers'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:534:in `fork'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:534:in
`spawn_missing_workers'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:530:in `each'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:530:in
`spawn_missing_workers'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:540:in
`maintain_worker_count'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:215:in `start'
/var/lib/gems/1.8/gems/unicorn-0.95.1/lib/unicorn.rb:28:in `run'
/var/lib/gems/1.8/gems/unicorn-0.95.1/bin/unicorn_rails:207
/var/lib/gems/1.8/bin/unicorn_rails:19:in `load'
/var/lib/gems/1.8/bin/unicorn_rails:19

Any idea about what's wrong ?

Thanks,
Jan

From dvdplm at gmail.com  Wed Dec  9 16:54:32 2009
From: dvdplm at gmail.com (David Palm)
Date: Wed, 9 Dec 2009 22:54:32 +0100
Subject: Expect 100-continue errors
In-Reply-To: <4B201111.2030701@kraxnet.cz>
References: <4B201111.2030701@kraxnet.cz>
Message-ID: <20091209225432345785.f364eed1@gmail.com>

On Wed, 09 Dec 2009 22:05:21 +0100, Jan Dvorak wrote:
> Hello,
> 
> i'm trying to implement handling of http Expect: 100-continue with
> unicorn in my rails app, but when i return [100] as a response (which

Just out of curiosity: what does 100 Continue mean and how can you use it (usefully)?

From normalperson at yhbt.net  Wed Dec  9 17:16:09 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 9 Dec 2009 22:16:09 +0000
Subject: Expect 100-continue errors
In-Reply-To: <4B201111.2030701@kraxnet.cz>
References: <4B201111.2030701@kraxnet.cz>
Message-ID: <20091209221608.GA17364@dcvr.yhbt.net>

Jan Dvorak <jan.dvorak at kraxnet.cz> wrote:
> Hello,
> 
> i'm trying to implement handling of http Expect: 100-continue with
> unicorn in my rails app, but when i return [100] as a response (which
> should force unicorn to return 100 status and rerun my app with the same
> connection) i get this error:
> 
> Read error: #<ThreadError: stopping only thread
>     note: use sleep to stop forever>
> /var/lib/gems/1.8/gems/actionpack-2.3.5/lib/action_controller/reloader.rb:31:in
> `lock'

Hi Jan,

Try disabling automatic code reloading under Rails development mode.

Unicorn (and Rainbows!) are the only Ruby servers I know of that allow
Rack applications to properly return and continue with 100 responses.

Unicorn calls your Rack app twice in this case: it sees a 100 response,
writes the 100-continue response and then does app.call(env) again after
deleting the "Expect: 100-continue" header from env.

I haven't tested this under Rails so I don't know how well it works with
what Rails does under the covers, but I've done this heavily with bare
Rack applications.

-- 
Eric Wong

From normalperson at yhbt.net  Wed Dec  9 17:20:16 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 9 Dec 2009 22:20:16 +0000
Subject: Expect 100-continue errors
In-Reply-To: <20091209225432345785.f364eed1@gmail.com>
References: <4B201111.2030701@kraxnet.cz>
	<20091209225432345785.f364eed1@gmail.com>
Message-ID: <20091209222016.GB17364@dcvr.yhbt.net>

David Palm <dvdplm at gmail.com> wrote:
> On Wed, 09 Dec 2009 22:05:21 +0100, Jan Dvorak wrote:
> > Hello,
> > 
> > i'm trying to implement handling of http Expect: 100-continue with
> > unicorn in my rails app, but when i return [100] as a response
> > (which
>
> Just out of curiosity: what does 100 Continue mean and how can you use
> it (usefully)?

Hi David,

It's described in the HTTP/1.1 spec
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3

curl uses it by default:
http://curl.haxx.se/docs/faq.html#My_HTTP_POST_or_PUT_requests_are

There are several uses of them in the Rainbows! test cases:

$ git clone git://git.bogomips.org/rainbows
$ grep HTTP_EXPECT rainbows/t/*.ru

-- 
Eric Wong

From jan.dvorak at kraxnet.cz  Wed Dec  9 18:46:05 2009
From: jan.dvorak at kraxnet.cz (Jan Dvorak)
Date: Thu, 10 Dec 2009 00:46:05 +0100
Subject: Expect 100-continue errors
In-Reply-To: <20091209221608.GA17364@dcvr.yhbt.net>
References: <4B201111.2030701@kraxnet.cz>
	<20091209221608.GA17364@dcvr.yhbt.net>
Message-ID: <4B2036BD.60200@kraxnet.cz>

Eric Wong wrote:
> Hi Jan,
>
> Try disabling automatic code reloading under Rails development mode.
>
>   
Thanks, that worked.
> Unicorn (and Rainbows!) are the only Ruby servers I know of that allow
> Rack applications to properly return and continue with 100 responses.
>
> Unicorn calls your Rack app twice in this case: it sees a 100 response,
> writes the 100-continue response and then does app.call(env) again after
> deleting the "Expect: 100-continue" header from env.
>
> I haven't tested this under Rails so I don't know how well it works with
> what Rails does under the covers, but I've done this heavily with bare
> Rack applications.
>
>   
After googling, it seems like rails error, it creates locks but never
properly releases them, and deadlocks upon class reloading.


Thanks,
Jan

From normalperson at yhbt.net  Thu Dec 10 17:30:37 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 10 Dec 2009 22:30:37 +0000
Subject: PATH_INFO spec (with regard to ";")
Message-ID: <20091210223037.GB22867@dcvr.yhbt.net>

Hi all,

I've been notified privately that my changes for PATH_INFO in Unicorn
0.95.2 (which also got into Thin) may not be completely kosher, but I'm
also asking for the Rack team to clarify PATH_INFO for HTTP parser
implementers.


Upon further reading (and also of the
related-but-not-necessarily-true-for-Rack RFC 3875 section 4.1.5),
I came across this:

   Unlike a URI path, the PATH_INFO is not URL-encoded, and cannot
   contain path-segment parameters.

First off, Rack already directly contradicts the "the PATH_INFO is not
URL-encoded" part, so Unicorn conforms to Rack specs over RFC 3875.

*But* Rack does not address the "cannot contain path-segment parameters"
part at all.  So I (and probably a few other people) would like
clarification on how to handle PATH_INFO when it comes to ";"


Things to keep in mind:

  * URI.parse keeps ";" in URI::HTTP#path
    This point may not be relevant to us, as PATH_INFO and
    URI::HTTP#path should not necessarily be treated as equals

  * WEBrick keeps ";" in PATH_INFO

  * PEP333 (which Rack is based on) does not go into this level of
    detail regarding PATH_INFO and path segments

  * PATH_INFO in Rack appears to be based on CGI/1.1 (RFC 3875)

  * Again, Rack already contradicts the URL encoding rules of RFC 3875
    for PATH_INFO, so there is precedence for Rack contradicting more
    of RFC 3875...

  * Rack::Request#full_path only looks at PATH_INFO + QUERY_STRING,
    this means many Rack applications may never see the ";" parts
    if Thin and Unicorn revert to old behavior.

  * Rack does not require REQUEST_URI, this is an extension Unicorn
    and Thin both carried over from Mongrel.

  * None of the official rack/rack-contrib middleware use REQUEST_URI


Of course, in the grand scheme of things, hardly anybody uses ";" in
paths.  Yay for rare corner cases making our lives difficult.

-- 
Eric Wong

From chewbranca at gmail.com  Mon Dec 14 22:48:27 2009
From: chewbranca at gmail.com (Russell Branca)
Date: Mon, 14 Dec 2009 19:48:27 -0800
Subject: Hardcoded ruby interpreter location
Message-ID: <b9c5d3a30912141948i4d1b3471n20027951d4d31b89@mail.gmail.com>

---Had some issues posting this, reposting, sorry if it does duplicate.

Hello,

I'm not sure if this is the correct send this bug report, but the 'Bug
Tracker' link on gemcutter links to this mailing list.
I just saw this in gem version 0.95.2, in the shebang line of both
bin/unicorn and bin/unicorn_rails the ruby?interpreter?is hard coded
to:

#!/home/ew/bin/ruby

Which obviously isn't correct.


-Russell

From normalperson at yhbt.net  Mon Dec 14 23:09:36 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 15 Dec 2009 04:09:36 +0000
Subject: Hardcoded ruby interpreter location
In-Reply-To: <b9c5d3a30912141948i4d1b3471n20027951d4d31b89@mail.gmail.com>
References: <b9c5d3a30912141948i4d1b3471n20027951d4d31b89@mail.gmail.com>
Message-ID: <20091215040936.GA29127@dcvr.yhbt.net>

Russell Branca <chewbranca at gmail.com> wrote:
> ---Had some issues posting this, reposting, sorry if it does duplicate.
> 
> Hello,
> 
> I'm not sure if this is the correct send this bug report, but the 'Bug
> Tracker' link on gemcutter links to this mailing list.

Hi Russell, yes, the mailing list is a bug tracker here.

> I just saw this in gem version 0.95.2, in the shebang line of both
> bin/unicorn and bin/unicorn_rails the ruby?interpreter?is hard coded
> to:
> 
> #!/home/ew/bin/ruby
> 
> Which obviously isn't correct.

It doesn't matter, no matter what the source tree sets it to it's wrong
for somebody.  Which is why RubyGems installs its own bin wrapper for
all bins, and setup.rb rewrites the shebang line for you.

And using the generic:

  #!/usr/bin/env ruby

...is even worse, since it can setup an incorrect $0 on some platforms
so SIGUSR2 upgrade method fails.

I might as well make it:

  #!/shebang/will/be/rewritten/or/wrapped/anyways/ruby

So it's very wrong for everyone :)

-- 
Eric Wong

From superjoe at gmail.com  Tue Dec 15 20:37:13 2009
From: superjoe at gmail.com (Joseph McDonald)
Date: Tue, 15 Dec 2009 17:37:13 -0800
Subject: workers created when request comes in?
Message-ID: <73096a160912151737t48f6684fhbb5f047bc187c0eb@mail.gmail.com>

Is it possible to have unicorn spawn a new worker when a request comes
in, and only allow that child to service the one request?

In development mode it would be nice if I could just save my changes
and know that I will see them right away without having to HUP the
server.

thanks,
-joe

From normalperson at yhbt.net  Tue Dec 15 21:37:01 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 15 Dec 2009 18:37:01 -0800
Subject: workers created when request comes in?
In-Reply-To: <73096a160912151737t48f6684fhbb5f047bc187c0eb@mail.gmail.com>
References: <73096a160912151737t48f6684fhbb5f047bc187c0eb@mail.gmail.com>
Message-ID: <20091216023701.GA6398@dcvr.yhbt.net>

Joseph McDonald <superjoe at gmail.com> wrote:
> Is it possible to have unicorn spawn a new worker when a request comes
> in, and only allow that child to service the one request?
> 
> In development mode it would be nice if I could just save my changes
> and know that I will see them right away without having to HUP the
> server.

Hi Joseph,

Yes, just send SIGQUIT to yourself anywhere inside the application
dispatch and it'll defer the exit until your application is done
processing.

You should be able to just make it middleware like this:

  # name inspired by the "shotgun" gem
  class Unicorn::Shotgun < Struct.new(:app)
    def call(env)
      Process.kill(:QUIT, 0)
      app.call(env)
    end
  end

-- 
Eric Wong

From normalperson at yhbt.net  Wed Dec 16 01:05:39 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 16 Dec 2009 06:05:39 +0000
Subject: workers created when request comes in?
In-Reply-To: <73096a160912152148l4eea211eq972fcea3ebdf9112@mail.gmail.com>
References: <73096a160912151737t48f6684fhbb5f047bc187c0eb@mail.gmail.com>
	<20091216023701.GA6398@dcvr.yhbt.net>
	<73096a160912152148l4eea211eq972fcea3ebdf9112@mail.gmail.com>
Message-ID: <20091216060539.GB1654@dcvr.yhbt.net>

Joseph McDonald <superjoe at gmail.com> wrote:
> On Tue, Dec 15, 2009 at 6:37 PM, Eric Wong <normalperson at yhbt.net> wrote:
> 
> > Yes, just send SIGQUIT to yourself anywhere inside the application
> > dispatch and it'll defer the exit until your application is done
> > processing.
> >
> > You should be able to just make it middleware like this:
> >
> > ?# name inspired by the "shotgun" gem
> > ?class Unicorn::Shotgun < Struct.new(:app)
> > ? ?def call(env)
> > ? ? ?Process.kill(:QUIT, 0)
                             ^-- Oops, that should be a $$, not 0
> > ? ? ?app.call(env)
> > ? ?end
> > ?end
> 
> Thanks Eric,
> I added that middleware, but now unicorn exits after servicing one
> request.  I'd like unicorn to keep running and fork off a new child if
> a new request comes in.  Is that possible?

Yeah, just change the 0 to a $$ there, brain fart :x

With that middleware, it should prefork a worker, but then kill it
after every request (and a new one will be preforked).

-- 
Eric Wong

From schacon at gmail.com  Thu Dec 17 19:20:17 2009
From: schacon at gmail.com (Scott Chacon)
Date: Thu, 17 Dec 2009 16:20:17 -0800
Subject: Usernames in the http_URL
Message-ID: <d411cc4a0912171620k3ff74554i6c5ad278e10c4db@mail.gmail.com>

I just ran into a stupid client that put the username in the http_URL
field, making the first line of the HTTP request look like this:

GET http://username at localhost:8080/mojombo/grit HTTP/1.1

Unicorn 500s on this, saying it can't parse the headers.  I'm
including a unit test that will die on this, but my question is should
Unicorn handle this gracefully by just stripping off the username -
parsing it as a 'server' instead of a 'host'?  It seems that most
other webservers do, even though it doesn't appear to be the spec.

Thanks,
Scott

---
?test/unit/test_http_parser.rb | ? 18 ++++++++++++++++++
?1 files changed, 18 insertions(+), 0 deletions(-)

diff --git a/test/unit/test_http_parser.rb b/test/unit/test_http_parser.rb
index 1b3faaf..7d32e5e 100644
--- a/test/unit/test_http_parser.rb
+++ b/test/unit/test_http_parser.rb
@@ -298,6 +298,24 @@ class HttpParserTest < Test::Unit::TestCase
? ? assert ! parser.keepalive?
? end

+ ?# some dumb clients add users because they're stupid
+ ?def test_absolute_uri_w_user
+ ? ?parser = HttpParser.new
+ ? ?req = {}
+ ? ?http = "GET http://user at example.com/foo?q=bar HTTP/1.0\r\n\r\n"
+ ? ?assert_equal req, parser.headers(req, http)
+ ? ?assert_equal 'http', req['rack.url_scheme']
+ ? ?assert_equal '/foo?q=bar', req['REQUEST_URI']
+ ? ?assert_equal '/foo', req['REQUEST_PATH']
+ ? ?assert_equal 'q=bar', req['QUERY_STRING']
+
+ ? ?assert_equal 'example.com', req['HTTP_HOST']
+ ? ?assert_equal 'example.com', req['SERVER_NAME']
+ ? ?assert_equal '80', req['SERVER_PORT']
+ ? ?assert_equal "", http
+ ? ?assert ! parser.keepalive?
+ ?end
+
? def test_absolute_uri
? ? parser = HttpParser.new
? ? req = {}
--
1.6.6.rc1

From normalperson at yhbt.net  Thu Dec 17 20:23:32 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 17 Dec 2009 17:23:32 -0800
Subject: Usernames in the http_URL
In-Reply-To: <d411cc4a0912171620k3ff74554i6c5ad278e10c4db@mail.gmail.com>
References: <d411cc4a0912171620k3ff74554i6c5ad278e10c4db@mail.gmail.com>
Message-ID: <20091218012332.GA30328@dcvr.yhbt.net>

Scott Chacon <schacon at gmail.com> wrote:
> I just ran into a stupid client that put the username in the http_URL
> field, making the first line of the HTTP request look like this:
> 
> GET http://username at localhost:8080/mojombo/grit HTTP/1.1
> 
> Unicorn 500s on this, saying it can't parse the headers.  I'm
> including a unit test that will die on this, but my question is should
> Unicorn handle this gracefully by just stripping off the username -
> parsing it as a 'server' instead of a 'host'?  It seems that most
> other webservers do, even though it doesn't appear to be the spec.

Hi Scott,

Other servers (Mongrel) fell back to URI.parse which allowed this.
Since Mongrel allowed it (possibly on accident), Unicorn should probably
allow it, too...

The following change should fix things for you, but I'm not sure about
the list of allowed characters for the user and don't have time to check
the RFCs right now.  Which client is doing this?  Any hope of fixing it
there?  But yeah, definitely not in rfc2616 from what I remember.

Also scp-ed the C source up to
http://unicorn.bogomips.org/unicorn_parser.c in case you don't have
Ragel.

diff --git a/ext/unicorn_http/unicorn_http_common.rl b/ext/unicorn_http/unicorn_http_common.rl
index 041dfec..4842972 100644
--- a/ext/unicorn_http/unicorn_http_common.rl
+++ b/ext/unicorn_http/unicorn_http_common.rl
@@ -28,6 +28,7 @@
   scheme = ( "http"i ("s"i)? ) $downcase_char >mark %scheme;
   hostname = (alnum | "-" | "." | "_")+;
   host_with_port = (hostname (":" digit*)?) >mark %host;
+  user = ((alnum | "_" | ".")+ "@")*;
 
   path = ( pchar+ ( "/" pchar* )* ) ;
   query = ( uchar | reserved )* %query_string ;
@@ -36,7 +37,7 @@
   rel_path = (path? (";" params)? %request_path) ("?" %start_query query)?;
   absolute_path = ( "/"+ rel_path );
   path_uri = absolute_path > mark %request_uri;
-  Absolute_URI = (scheme "://" host_with_port path_uri);
+  Absolute_URI = (scheme "://" user host_with_port path_uri);
 
   Request_URI = ((absolute_path | "*") >mark %request_uri) | Absolute_URI;
   Fragment = ( uchar | reserved )* >mark %fragment;

-- 
Eric Wong

From normalperson at yhbt.net  Thu Dec 17 20:32:24 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 18 Dec 2009 01:32:24 +0000
Subject: Usernames in the http_URL
In-Reply-To: <20091218012332.GA30328@dcvr.yhbt.net>
References: <d411cc4a0912171620k3ff74554i6c5ad278e10c4db@mail.gmail.com>
	<20091218012332.GA30328@dcvr.yhbt.net>
Message-ID: <20091218013224.GA12328@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> Also scp-ed the C source up to
> http://unicorn.bogomips.org/unicorn_parser.c in case you don't have
> Ragel.

Err, http://unicorn.bogomips.org/unicorn_http.c  that is

From schacon at gmail.com  Thu Dec 17 21:31:54 2009
From: schacon at gmail.com (Scott Chacon)
Date: Thu, 17 Dec 2009 18:31:54 -0800
Subject: Usernames in the http_URL
In-Reply-To: <20091218012332.GA30328@dcvr.yhbt.net>
References: <d411cc4a0912171620k3ff74554i6c5ad278e10c4db@mail.gmail.com>
	<20091218012332.GA30328@dcvr.yhbt.net>
Message-ID: <d411cc4a0912171831h8cbeef2wfed676bd11fa2b19@mail.gmail.com>

Hey,

On Thu, Dec 17, 2009 at 5:23 PM, Eric Wong <normalperson at yhbt.net> wrote:
> The following change should fix things for you, but I'm not sure about
> the list of allowed characters for the user and don't have time to check
> the RFCs right now. ?Which client is doing this? ?Any hope of fixing it
> there? ?But yeah, definitely not in rfc2616 from what I remember.

Thanks a lot.  I do have Ragel, I'll test this out tomorrow.  It is
the Versions SVN client in DAV mode that is doing this, and I hate
them a little for it.  However, I figured other clients must be doing
it too.  Thanks a bunch for the patch.

Scott

From hukl at berlin.ccc.de  Fri Dec 18 04:48:43 2009
From: hukl at berlin.ccc.de (John-Paul Bader)
Date: Fri, 18 Dec 2009 10:48:43 +0100
Subject: Usernames in the http_URL
In-Reply-To: <d411cc4a0912171831h8cbeef2wfed676bd11fa2b19@mail.gmail.com>
References: <d411cc4a0912171620k3ff74554i6c5ad278e10c4db@mail.gmail.com>
	<20091218012332.GA30328@dcvr.yhbt.net>
	<d411cc4a0912171831h8cbeef2wfed676bd11fa2b19@mail.gmail.com>
Message-ID: <1191F787-4539-4877-86A1-C62C8D58DC27@berlin.ccc.de>

Hey guys,


I think the <resource_type>://<username>:<password>@<host>/<path> scheme is not "illegal". There are examples of this in the URL RFC, just no explicit HTTP example.

This probably a vague area. Its not in the http rfc and its not explicitly mentioned in the http auth rfc either but in combination with the URL RFC there is at least room for it. I haven't found the paragraph yet which says: no username:password stuff allowed in http urls. But I was just searching through these things ? there are good chances I missed it.

http://en.wikipedia.org/wiki/URI_scheme
http://tools.ietf.org/html/rfc2617
http://www.ietf.org/rfc/rfc1738.txt

Anyway, I came across such urls a lot. Often I use them for giving people easy access to an otherwise basic authed resource - in  a chat conversation for example. I know apache and nginx support this - IIS does not. 

Hrm - tough call ;)

Kind regards, John

From normalperson at yhbt.net  Sat Dec 19 05:04:52 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Sat, 19 Dec 2009 02:04:52 -0800
Subject: Usernames in the http_URL
In-Reply-To: <1191F787-4539-4877-86A1-C62C8D58DC27@berlin.ccc.de>
References: <d411cc4a0912171620k3ff74554i6c5ad278e10c4db@mail.gmail.com>
	<20091218012332.GA30328@dcvr.yhbt.net>
	<d411cc4a0912171831h8cbeef2wfed676bd11fa2b19@mail.gmail.com>
	<1191F787-4539-4877-86A1-C62C8D58DC27@berlin.ccc.de>
Message-ID: <20091219100452.GA13394@dcvr.yhbt.net>

John-Paul Bader <hukl at berlin.ccc.de> wrote:
> Hey guys,
> 
> I think the <resource_type>://<username>:<password>@<host>/<path>
> scheme is not "illegal". There are examples of this in the URL RFC,
> just no explicit HTTP example.
> 
> This probably a vague area. Its not in the http rfc and its not
> explicitly mentioned in the http auth rfc either but in combination
> with the URL RFC there is at least room for it. I haven't found the
> paragraph yet which says: no username:password stuff allowed in http
> urls. But I was just searching through these things ? there are good
> chances I missed it.

Hi,

Yes, I've come to the same conclusion.  rfc2616 just seems to defer
to rfc2396 (which superceded rfc1738 and is superceded by rfc3986).

> http://en.wikipedia.org/wiki/URI_scheme
> http://tools.ietf.org/html/rfc2617
> http://www.ietf.org/rfc/rfc1738.txt
> 
> Anyway, I came across such urls a lot. Often I use them for giving
> people easy access to an otherwise basic authed resource - in  a chat
> conversation for example. I know apache and nginx support this - IIS
> does not. 
> 
> Hrm - tough call ;)

Yup, definitely precedence for supporting it (along with Mongrel).  I've
updated the Ragel parser with everything URI.parse("http://..") supports
and pushed out the change.

I've been meaning to make a few more small documentation updates and do
a 0.95.3 release tomorrow when I'm more awake.

-- 
Eric Wong

From normalperson at yhbt.net  Mon Dec 21 17:21:02 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 21 Dec 2009 22:21:02 +0000
Subject: [ANN] unicorn 0.95.3 - userinfo in absoluteURIs
Message-ID: <20091221222102.GA13223@dcvr.yhbt.net>

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

The HTTP parser now allows (but does not parse) the userinfo
component in the very rare requests that send absoluteURIs.
Thanks to Scott Chacon for reporting and submitting a test case
for this fix.

There are also minor documentation updates and tiny cleanups.

-- 
Eric Wong

From ibc at aliax.net  Tue Dec 22 21:20:59 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Wed, 23 Dec 2009 03:20:59 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
Message-ID: <200912230320.59469.ibc@aliax.net>

Hi, I'm writing a Debian init script for unicorn and realized that when 
starting unicorn with daemonize option, the command always returns 0, even if 
the start action failed (due for example Errno::EADDRINUSE).

Returning 0 in such case is not good as it breaks service init scripts or 
service controllers (as HeartBeat) that fully rely on the appropriate exit 
code.

Is there some way to determine if unicorn failed to start when using "-D"?



Another related issue: When the Rack config.ru file contains some error (as a 
typo) the worker(s) returns 1 (at the moment usually). Then unicorn master 
process reapes the terminated worker process and restarts it. Of course it 
would fail again and again. Anyhow "unicorn -D" returns 0 again (success).

Usually if a worker (all the workers) fail to start at the moment of running 
it, it obviously means that there is some error in the application with 
prevents it to run. It could be great if Unicorn could detect it.

For that I suggest something as a new option "--validation-time TIME". Let's 
suppose TIME is 5 seconds. In case *all* the workers fail within 5 seconds 
after starting unicorn, then unicorn understands that the Rack application is 
wrong (or any other error as Errno::EADDRINUSE) so terminates all the workers 
and itself (and hopefully returns 1 or any other non-zero exit status).

Of course, all the above means that Unicorn should wait TIME seconds before 
being daemonized (so after TIME seconds it can decide which code to return).

Does it make sense? Thanks a lot.

-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Wed Dec 23 02:26:17 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 23 Dec 2009 07:26:17 +0000
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912230320.59469.ibc@aliax.net>
References: <200912230320.59469.ibc@aliax.net>
Message-ID: <20091223072617.GA28704@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> Hi, I'm writing a Debian init script for unicorn and realized that when 
> starting unicorn with daemonize option, the command always returns 0, even if 
> the start action failed (due for example Errno::EADDRINUSE).
> 
> Returning 0 in such case is not good as it breaks service init scripts or 
> service controllers (as HeartBeat) that fully rely on the appropriate exit 
> code.
> 
> Is there some way to determine if unicorn failed to start when using "-D"?

Hi I?aki,

No way to determine that currently, as I've never encountered the need.

For validating startups, most folks I know have specific endpoint(s) in
application dedicated to checks.  That way they can get way more info
and all the way down into stack including things like database/memcached
connections.

Anything less is superficial because they can fail to detect other
misconfigurations (including stuff like wrong RAILS_ENV); not just
startup errors.

> Another related issue: When the Rack config.ru file contains some error (as a 
> typo) the worker(s) returns 1 (at the moment usually). Then unicorn master 
> process reapes the terminated worker process and restarts it. Of course it 
> would fail again and again. Anyhow "unicorn -D" returns 0 again (success).
> 
> Usually if a worker (all the workers) fail to start at the moment of running 
> it, it obviously means that there is some error in the application with 
> prevents it to run. It could be great if Unicorn could detect it.

I'm generally hesitant to introduce code/features/bloat that aren't
helpful to people who properly test configurations before deploying :)

Adding this code doesn't solve problems.  Either way the site can become
inaccessible/unusable and problems are noticeable very quickly.

If there's sufficient interest, I'll consider accepting a small patch
for this.  Right now I'm unconvinced...

> For that I suggest something as a new option "--validation-time TIME". Let's 
> suppose TIME is 5 seconds. In case *all* the workers fail within 5 seconds 
> after starting unicorn, then unicorn understands that the Rack application is 
> wrong (or any other error as Errno::EADDRINUSE) so terminates all the workers 
> and itself (and hopefully returns 1 or any other non-zero exit status).
> 
> Of course, all the above means that Unicorn should wait TIME seconds before 
> being daemonized (so after TIME seconds it can decide which code to return).
> 
> Does it make sense? Thanks a lot.

We avoid introducing command-line options since they're unlikely to be
compatible with "rackup" parsing of the "#\" lines in .ru files.

-- 
Eric Wong

From ibc at aliax.net  Wed Dec 23 05:22:27 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Wed, 23 Dec 2009 11:22:27 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091223072617.GA28704@dcvr.yhbt.net>
References: <200912230320.59469.ibc@aliax.net>
	<20091223072617.GA28704@dcvr.yhbt.net>
Message-ID: <200912231122.28331.ibc@aliax.net>

El Mi?rcoles, 23 de Diciembre de 2009, Eric Wong escribi?:

> No way to determine that currently, as I've never encountered the need.
> 
> For validating startups, most folks I know have specific endpoint(s) in
> application dedicated to checks.  That way they can get way more info
> and all the way down into stack including things like database/memcached
> connections.

Yes, you are right and I agree now. I was thinking in Unicorn as a final 
application rather than a HTTP applications container. It's like Apache, no 
body expects Apache to return 1 and exit in case a PHP application has a typo 
:)



> > Usually if a worker (all the workers) fail to start at the moment of
> > running it, it obviously means that there is some error in the
> > application with prevents it to run. It could be great if Unicorn could
> > detect it.
> 
> I'm generally hesitant to introduce code/features/bloat that aren't
> helpful to people who properly test configurations before deploying :)
> 
> Adding this code doesn't solve problems.  Either way the site can become
> inaccessible/unusable and problems are noticeable very quickly.

Sure, but how to explain that to Hearteat? :)

 
> If there's sufficient interest, I'll consider accepting a small patch
> for this.  Right now I'm unconvinced...

I've taken a look to the code and such a feature would definitively make it 
ugly (IMHO).

So I'm thinking in a different approach: a custom script to check the status 
of the application. Such script would communicate with the application (maybe 
using DRB). If the DRB connection fails (because the workers fail to start) 
then it means that something wrong occurs. And such script would also return 
the whole status (DB connections and so).
I would include such script as "status" option in a service init script. The 
"start" action would call "status" after N seconds and decide if the 
*application* is running or not (if not then kill unicorn and exit 1).

PS: Since Unicorn makes usage of signals, is there any way to determine (by 
using a signal) if the process running with some PID is an unicorn process?

This is: usually to verify the process status the PID file/value is inspected. 
If there is a process running under such PID then we "assume" that process is 
Unicorn. In case that PID is stolen by any other process (since PID file 
wasn't deleted) nobody realizes of it.

A way to determine it could be asking to Unicorn master process (using i.e. 
DRB) its PID and match it against the PID value stored in the pidfile. Of 
course it would require some kind of communication with unicorn master process 
to get its PID, is it possible now in some way?



 
> > For that I suggest something as a new option "--validation-time TIME".
> > Let's suppose TIME is 5 seconds. In case *all* the workers fail within 5
> > seconds after starting unicorn, then unicorn understands that the Rack
> > application is wrong (or any other error as Errno::EADDRINUSE) so
> > terminates all the workers and itself (and hopefully returns 1 or any
> > other non-zero exit status).
> >
> > Of course, all the above means that Unicorn should wait TIME seconds
> > before being daemonized (so after TIME seconds it can decide which code
> > to return).
> >
> > Does it make sense? Thanks a lot.
> 
> We avoid introducing command-line options since they're unlikely to be
> compatible with "rackup" parsing of the "#\" lines in .ru files.

I didn't know about such options in .ru file. Are those options supposed to be 
read by the backend?


Thanks a lot.

-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Wed Dec 23 07:17:14 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Wed, 23 Dec 2009 13:17:14 +0100
Subject: How to override "Worker#proc_name" method?
Message-ID: <200912231317.14330.ibc@aliax.net>

Hi, I'm trying to override Unicorn::HttpServer::Worker#proc_name method. For 
that, I've copied bin/unicorn with other name and added the following code 
before "Unicorn::Launcher.daemonize! if daemonize" line:

module Unicorn
  class HttpServer
    class Worker
      def proc_name(tag)
          ... some proc name ...
      end
    end
  end
end

However it doesn't work and workers still read the original method. I've also 
tryed by using "-r" option to load a library containing such modifications.

I suspect what is happening: When Unicorn creates the forked workers each of 
them is a new process which performs "require 'unicorn'" so the workers don't 
read my customizations. Am I right? If so, is there some way to achieve it?

Thanks a lot.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Wed Dec 23 07:36:28 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Wed, 23 Dec 2009 13:36:28 +0100
Subject: How to override "Worker#proc_name" method?
In-Reply-To: <200912231317.14330.ibc@aliax.net>
References: <200912231317.14330.ibc@aliax.net>
Message-ID: <200912231336.28336.ibc@aliax.net>

El Mi?rcoles, 23 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> I suspect what is happening: When Unicorn creates the forked workers each
>  of  them is a new process which performs "require 'unicorn'" so the
>  workers don't read my customizations. Am I right? If so, is there some way
>  to achieve it?

Humm, but now I've realized that "netstat" just gets the master process. 
Shouldn't the master process read my changes? The "issue" occurs running 
unicorn in both foreground and background (daemonized).

Thanks.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Wed Dec 23 07:42:28 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Wed, 23 Dec 2009 13:42:28 +0100
Subject: How to override "Worker#proc_name" method?
In-Reply-To: <200912231317.14330.ibc@aliax.net>
References: <200912231317.14330.ibc@aliax.net>
Message-ID: <200912231342.28718.ibc@aliax.net>

El Mi?rcoles, 23 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> Hi, I'm trying to override Unicorn::HttpServer::Worker#proc_name method.

I want to do it because by default Unicorn processes are called as:

  unicorn worker[0] -E production -c unicorn.conf.rb config.ru

Unfortunatelly netstat takes the last word and shows "config.ru":

  ~# netstat -putan | grep "0.0.0.0:8080"
  tcp   0   0   0.0.0.0:9292   0.0.0.0:*   LISTEN   31575/config.ru

This is not very good for the case in which the user/admin wants to know in 
which port the process is listening so it does:

  ~# netstat -putan | grep "unicorn"

and gets nothing.

Of course this is more an issue of "netstat" and how it determines to display 
the process name. However I would like to override the way unicorn displays 
the process name (without changing unicorn code).


Regards.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Wed Dec 23 09:29:11 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Wed, 23 Dec 2009 15:29:11 +0100
Subject: How to trap USR2 for custom ussage ?
Message-ID: <200912231529.11880.ibc@aliax.net>

Hi, I've modified bin/unicorn to start a DRb server before last lines:

  MyApp::DRbServer.start(Process.pid)
  Unicorn::Launcher.daemonize! if daemonize
  Unicorn.run(app, options)

DRb server listens in port 5555. When I send a USR2 signal to unicorn master 
process I get an error because DRb is started again so:

  /usr/local/lib/ruby1.9/1.9.1/drb/drb.rb:861:in `initialize': Address already
  in use - bind(2) (Errno::EADDRINUSE)

The only solution I can imagine is to trap USR2 signal:

  MyApp::DRbServer.start(Process.pid)
  trap("USR2") { ::DRb.stop_service }
  Unicorn::Launcher.daemonize! if daemonize
  Unicorn.run(app, options)

Unfortunatelly it doesn't work because Unicorn overrides the 'trap' call so I 
don't get the USR2 signal.


Is there any way to achive this?
I could imagine a new config option "on_usr2" (or "before_reexec") so the 
passsed block would be executed upon receipt of USR2 (before the real "reexec" 
method).

Does it make sense?

Thanks a lot.

-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Wed Dec 23 15:35:45 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 23 Dec 2009 20:35:45 +0000
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912231122.28331.ibc@aliax.net>
References: <200912230320.59469.ibc@aliax.net>
	<20091223072617.GA28704@dcvr.yhbt.net>
	<200912231122.28331.ibc@aliax.net>
Message-ID: <20091223203544.GB13735@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Mi?rcoles, 23 de Diciembre de 2009, Eric Wong escribi?:
> > > Usually if a worker (all the workers) fail to start at the moment of
> > > running it, it obviously means that there is some error in the
> > > application with prevents it to run. It could be great if Unicorn could
> > > detect it.
> > 
> > I'm generally hesitant to introduce code/features/bloat that aren't
> > helpful to people who properly test configurations before deploying :)
> > 
> > Adding this code doesn't solve problems.  Either way the site can become
> > inaccessible/unusable and problems are noticeable very quickly.
> 
> Sure, but how to explain that to Hearteat? :)

I had a lot of trouble coming to terms with this one myself :)

Eventually I reasoned that the heartbeat mechanism is to protect against
unforeseen failures way down the line (and sometimes outside of the
application's control).

It's (hopefully) common and expected to check applications shortly
following deployments, so heartbeat isn't for that.

However applications have a tendency to break in unforseen ways, and
only in production, and at the worst times.  These failures happen well
after the app is deployed :)

In my experience, it's often the least used/trafficked parts of the
application that fail, too.

Of course if you write perfect software that handles every possible
failure case and don't need heartbeat (nor multiple processes because
your code is super-efficient :), check out Zbatery[1]

[1] http://zbatery.bogomip.org/

> > If there's sufficient interest, I'll consider accepting a small patch
> > for this.  Right now I'm unconvinced...
> 
> I've taken a look to the code and such a feature would definitively make it 
> ugly (IMHO).
> 
> So I'm thinking in a different approach: a custom script to check the status 
> of the application. Such script would communicate with the application (maybe 
> using DRB). If the DRB connection fails (because the workers fail to start) 
> then it means that something wrong occurs. And such script would also return 
> the whole status (DB connections and so).
> I would include such script as "status" option in a service init script. The 
> "start" action would call "status" after N seconds and decide if the 
> *application* is running or not (if not then kill unicorn and exit 1).
> 
> PS: Since Unicorn makes usage of signals, is there any way to determine (by 
> using a signal) if the process running with some PID is an unicorn process?
> 
> This is: usually to verify the process status the PID file/value is inspected. 
> If there is a process running under such PID then we "assume" that process is 
> Unicorn. In case that PID is stolen by any other process (since PID file 
> wasn't deleted) nobody realizes of it.

Under modern Linux systems, you have several options...

  tr '\0' ' ' < /proc/$PID/cmdline # show the command-line

  pmap $PID | grep unicorn_http.so # not many other things load this

  lsof -p $PID  # can check listening ports/log files

Unfortunately I don't use other OSes enough to remember the
equivalents...

> A way to determine it could be asking to Unicorn master process (using i.e. 
> DRB) its PID and match it against the PID value stored in the pidfile. Of 
> course it would require some kind of communication with unicorn master process 
> to get its PID, is it possible now in some way?

I don't see how being under a specific PID actually matters.  Why not
just write a script that sends an HTTP request to check?

Workers will die soon after the first request, or (timeout/2) seconds if
a master dies anyways.

> > We avoid introducing command-line options since they're unlikely to be
> > compatible with "rackup" parsing of the "#\" lines in .ru files.
> 
> I didn't know about such options in .ru file. Are those options
> supposed to be read by the backend?

I don't believe it's well-documented, I only found out from reading the
rackup code and I'm not really a fan of magic comments, but they're only
for the basic command-line arguments rackup handles:

  $ rackup -h
  Usage: rackup [ruby options] [rack options] [rackup config]
  Ruby options:
    -e, --eval LINE          evaluate a LINE of code
    -d, --debug              set debugging flags (set $DEBUG to true)
    -w, --warn               turn warnings on for your script
    -I, --include PATH       specify $LOAD_PATH (may be used more than once)
    -r, --require LIBRARY    require the library, before executing your script
  Rack options:
    -s, --server SERVER      serve using SERVER (webrick/mongrel)
    -o, --host HOST          listen on HOST (default: 0.0.0.0)
    -p, --port PORT          use PORT (default: 9292)
    -E, --env ENVIRONMENT    use ENVIRONMENT for defaults (default: development)
    -D, --daemonize          run daemonized in the background
    -P, --pid FILE           file to store PID (default: rack.pid)
  Common options:
    -h, --help               Show this message
        --version            Show version

-- 
Eric Wong

From normalperson at yhbt.net  Wed Dec 23 15:39:27 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 23 Dec 2009 20:39:27 +0000
Subject: How to override "Worker#proc_name" method?
In-Reply-To: <200912231317.14330.ibc@aliax.net>
References: <200912231317.14330.ibc@aliax.net>
Message-ID: <20091223203927.GC13735@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> Hi, I'm trying to override Unicorn::HttpServer::Worker#proc_name method. For 
> that, I've copied bin/unicorn with other name and added the following code 
> before "Unicorn::Launcher.daemonize! if daemonize" line:
> 
> module Unicorn
>   class HttpServer
>     class Worker
>       def proc_name(tag)
>           ... some proc name ...
>       end
>     end
>   end
> end

Hi, proc_name is in HttpServer, not Worker

-- 
Eric Wong

From normalperson at yhbt.net  Wed Dec 23 15:47:05 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 23 Dec 2009 20:47:05 +0000
Subject: How to trap USR2 for custom ussage ?
In-Reply-To: <200912231529.11880.ibc@aliax.net>
References: <200912231529.11880.ibc@aliax.net>
Message-ID: <20091223204705.GD13735@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> Hi, I've modified bin/unicorn to start a DRb server before last lines:
> 
>   MyApp::DRbServer.start(Process.pid)
>   Unicorn::Launcher.daemonize! if daemonize
>   Unicorn.run(app, options)
> 
> DRb server listens in port 5555. When I send a USR2 signal to unicorn master 
> process I get an error because DRb is started again so:
> 
>   /usr/local/lib/ruby1.9/1.9.1/drb/drb.rb:861:in `initialize': Address already
>   in use - bind(2) (Errno::EADDRINUSE)

First off I really don't think you need to be using DRb for this or
(generally) sticking things into the master process that don't belong
there.

> Is there any way to achive this?
> I could imagine a new config option "on_usr2" (or "before_reexec") so the 
> passsed block would be executed upon receipt of USR2 (before the real "reexec" 
> method).
> 
> Does it make sense?

There's already a similar before_exec hook documented in
http://unicorn.bogomips.org/Unicorn/Configurator.html
But it is called after forking, so the listener will be shared.

But again, you don't need to use DRb for this.

-- 
Eric Wong

From ibc at aliax.net  Thu Dec 24 03:55:08 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Thu, 24 Dec 2009 09:55:08 +0100
Subject: How to override "Worker#proc_name" method?
In-Reply-To: <20091223203927.GC13735@dcvr.yhbt.net>
References: <200912231317.14330.ibc@aliax.net>
	<20091223203927.GC13735@dcvr.yhbt.net>
Message-ID: <200912240955.08780.ibc@aliax.net>

El Mi?rcoles, 23 de Diciembre de 2009, Eric Wong escribi?:
> I?aki Baz Castillo <ibc at aliax.net> wrote:
> > Hi, I'm trying to override Unicorn::HttpServer::Worker#proc_name method.
> > For that, I've copied bin/unicorn with other name and added the following
> > code before "Unicorn::Launcher.daemonize! if daemonize" line:
> >
> > module Unicorn
> >   class HttpServer
> >     class Worker
> >       def proc_name(tag)
> >           ... some proc name ...
> >       end
> >     end
> >   end
> > end
> 
> Hi, proc_name is in HttpServer, not Worker

Great! oh my god... I was trying everything and didn't realize of that XDD

Thanks a lot, it just works :)
 


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Thu Dec 24 04:30:01 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Thu, 24 Dec 2009 10:30:01 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091223203544.GB13735@dcvr.yhbt.net>
References: <200912230320.59469.ibc@aliax.net>
	<200912231122.28331.ibc@aliax.net>
	<20091223203544.GB13735@dcvr.yhbt.net>
Message-ID: <200912241030.02160.ibc@aliax.net>

El Mi?rcoles, 23 de Diciembre de 2009, Eric Wong escribi?:
> > > Adding this code doesn't solve problems.  Either way the site can
> > > become inaccessible/unusable and problems are noticeable very quickly.
> >
> > Sure, but how to explain that to Hearteat? :)
> 
> I had a lot of trouble coming to terms with this one myself :)
> 
> Eventually I reasoned that the heartbeat mechanism is to protect against
> unforeseen failures way down the line (and sometimes outside of the
> application's control).
> 
> It's (hopefully) common and expected to check applications shortly
> following deployments, so heartbeat isn't for that.

Yes, you are right.

 
> However applications have a tendency to break in unforseen ways, and
> only in production, and at the worst times.  These failures happen well
> after the app is deployed :)

Yes, every application breaks just in the moment that it shouldn't :)

However I'm building my app now and when start it it usually has errors, and 
that's good as it means I'm working :)
But sincerelly I would like to see the error just once rather than many times 
due to workers respawn :)

Also, is there any "interval" parameter so a worker is not re-spawned until 
such timeout (after die).




> > This is: usually to verify the process status the PID file/value is
> > inspected. If there is a process running under such PID then we "assume"
> > that process is Unicorn. In case that PID is stolen by any other process
> > (since PID file wasn't deleted) nobody realizes of it.
> 
> Under modern Linux systems, you have several options...
> 
>   tr '\0' ' ' < /proc/$PID/cmdline # show the command-line
> 
>   pmap $PID | grep unicorn_http.so # not many other things load this
> 
>   lsof -p $PID  # can check listening ports/log files

Yes! this is much better! :)

 
> Unfortunately I don't use other OSes enough to remember the
> equivalents...

The server will run just on Linux (Unix) server :)


> > A way to determine it could be asking to Unicorn master process (using
> > i.e. DRB) its PID and match it against the PID value stored in the
> > pidfile. Of course it would require some kind of communication with
> > unicorn master process to get its PID, is it possible now in some way?
> 
> I don't see how being under a specific PID actually matters.  Why not
> just write a script that sends an HTTP request to check?
> 
> Workers will die soon after the first request, or (timeout/2) seconds if
> a master dies anyways.

Well, in fact I want to code some console utility to get information from the 
application (loaded configuration, check the database(s) connection and so).
For that I was thinking in a DRb server running in the master because it must 
bind in some TCP port.

The problem is that I should stop the DRb server before master reexec to avoid 
EADDINUSE. Well, I could code some rescue block...




> > I didn't know about such options in .ru file. Are those options
> > supposed to be read by the backend?
> 
> I don't believe it's well-documented, I only found out from reading the
> rackup code and I'm not really a fan of magic comments, but they're only
> for the basic command-line arguments rackup handles:
> 
>   $ rackup -h
...
>  -D, --daemonize          run daemonized in the background

Why "daemonize" is not present in unicornf configuration file?


Really thanks a lot for all your help.



-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Thu Dec 24 04:40:25 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Thu, 24 Dec 2009 10:40:25 +0100
Subject: How to trap USR2 for custom ussage ?
In-Reply-To: <20091223204705.GD13735@dcvr.yhbt.net>
References: <200912231529.11880.ibc@aliax.net>
	<20091223204705.GD13735@dcvr.yhbt.net>
Message-ID: <200912241040.25841.ibc@aliax.net>

El Mi?rcoles, 23 de Diciembre de 2009, Eric Wong escribi?:

> First off I really don't think you need to be using DRb for this or
> (generally) sticking things into the master process that don't belong
> there.

The problem is: how to start a server (which binds in a TCP port) in just one 
worker? humm, would the following make sense?:

  after_fork do |server, worker|
    # Start DRb server just in worker[0]
    if worker.nr == 0
      ... start DRb server ...
    end
  end

But it would also fail upon receipt of USR2 as there would be two instances of 
the DRb server trying to bind on same port...


> > Is there any way to achive this?
> > I could imagine a new config option "on_usr2" (or "before_reexec") so the
> > passsed block would be executed upon receipt of USR2 (before the real
> > "reexec" method).
> >
> > Does it make sense?
> 
> There's already a similar before_exec hook documented in
> http://unicorn.bogomips.org/Unicorn/Configurator.html
> But it is called after forking, so the listener will be shared.

Hummm, perhaps I could use the above code plus:

  before_exec do |server|
    .... stop DRb server ...
  end

In this way the DRb server is stopped before exec and started in the new 
worker[0].
Just a problem, how to access to existing/old worker[0] from "before_exec" in 
order to stop the DRb server?


 
> But again, you don't need to use DRb for this.

As I've said in other thread I want DRb to get some info related to the 
application. A DRb client console could be used to check application 
database(s) connection and so.
What would you suggest for this rather than DRb server?

 
Again thanks a lot for your help.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Thu Dec 24 08:36:26 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Thu, 24 Dec 2009 14:36:26 +0100
Subject: Why "--pid" is deprecated?
Message-ID: <200912241436.26761.ibc@aliax.net>

By inspecting bin/unicorn I've realized that "--pid" file exists but it's 
deprecated:

  opts.on("-P", "--pid FILE", "DEPRECATED") do |f|
    warn %q{Use of --pid/-P is strongly discouraged}
    warn %q{Use the 'pid' directive in the Unicorn config file instead}
    options[:pid] = f
  end

Could I know why? In order to build a service init script is really common the 
init script to handle the PID file so it can start, restart or stop the 
process runnining under that PID value.

Which is the real advantage of using "pid" directive in config file rather 
than "--pid"?

Thanks.

-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Thu Dec 24 14:17:35 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 24 Dec 2009 11:17:35 -0800
Subject: Why "--pid" is deprecated?
In-Reply-To: <200912241436.26761.ibc@aliax.net>
References: <200912241436.26761.ibc@aliax.net>
Message-ID: <20091224191735.GA14191@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> By inspecting bin/unicorn I've realized that "--pid" file exists but it's 
> deprecated:
> 
>   opts.on("-P", "--pid FILE", "DEPRECATED") do |f|
>     warn %q{Use of --pid/-P is strongly discouraged}
>     warn %q{Use the 'pid' directive in the Unicorn config file instead}
>     options[:pid] = f
>   end
> 
> Could I know why? In order to build a service init script is really common the 
> init script to handle the PID file so it can start, restart or stop the 
> process runnining under that PID value.

Hi I?aki,

It conflicts/confuses the -P/--path setting in bin/unicorn_rails.  An
oversight on my part since I based unicorn_rails on script/server
in Rails.  So I suppose we're stuck with it for a while.

> Which is the real advantage of using "pid" directive in config file rather 
> than "--pid"?

Not much, pid files suck either way.  Sadly there aren't many other
options that's universally used/accepted.

Using ENV["UNICORN_PID"] could be an option in the config file if you
want to share it between the config and init script.

-- 
Eric Wong

From normalperson at yhbt.net  Thu Dec 24 14:34:33 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 24 Dec 2009 11:34:33 -0800
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912241030.02160.ibc@aliax.net>
References: <200912230320.59469.ibc@aliax.net>
	<200912231122.28331.ibc@aliax.net>
	<20091223203544.GB13735@dcvr.yhbt.net>
	<200912241030.02160.ibc@aliax.net>
Message-ID: <20091224193433.GB14191@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Mi?rcoles, 23 de Diciembre de 2009, Eric Wong escribi?:
> However I'm building my app now and when start it it usually has errors, and 
> that's good as it means I'm working :)
> But sincerelly I would like to see the error just once rather than many times 
> due to workers respawn :)
> 
> Also, is there any "interval" parameter so a worker is not re-spawned until 
> such timeout (after die).

You can sleep in the before_fork hook.

> Well, in fact I want to code some console utility to get information from the 
> application (loaded configuration, check the database(s) connection and so).
> For that I was thinking in a DRb server running in the master because it must 
> bind in some TCP port.
 
DRb can bind to UNIX domain sockets, too.  But ask yourself:

  Is your app really so broken that you always need DRb running on it?

You may also want to checkout Hijack, I haven't used/needed it myself,
but it could be interesting if an app is really in a lot of trouble:

http://www.rubyinside.com/hijack-get-a-live-irb-prompt-for-any-existing-ruby-process-2232.html

> The problem is that I should stop the DRb server before master reexec to avoid 
> EADDINUSE. Well, I could code some rescue block...

rescuing EADDRINUSE in a loop is also how the "listen" directive
implements the :tries/:delay parameters.

> >   $ rackup -h
> ...
> >  -D, --daemonize          run daemonized in the background
> 
> Why "daemonize" is not present in unicornf configuration file?

Mainly it was cleaner and easier to test/implement it the way it is now.
It's also easier for somebody testing a setup to toggle it from the
command line.

I actually didn't want to support --daemonize at all since setsid(8),
is standard most GNU/Linux distros nowadays, but we support more than
just GNU/Linux.

> Really thanks a lot for all your help.

No problem.

-- 
Eric Wong

From normalperson at yhbt.net  Thu Dec 24 14:41:47 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 24 Dec 2009 11:41:47 -0800
Subject: How to trap USR2 for custom ussage ?
In-Reply-To: <200912241040.25841.ibc@aliax.net>
References: <200912231529.11880.ibc@aliax.net>
	<20091223204705.GD13735@dcvr.yhbt.net>
	<200912241040.25841.ibc@aliax.net>
Message-ID: <20091224194147.GC14191@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Mi?rcoles, 23 de Diciembre de 2009, Eric Wong escribi?:
> > But again, you don't need to use DRb for this.
> 
> As I've said in other thread I want DRb to get some info related to the 
> application. A DRb client console could be used to check application 
> database(s) connection and so.
> What would you suggest for this rather than DRb server?

As I said before, write an endpoint in your HTTP application that
calls and displays all the needed info you need.  Something that
hits the database(s) and/or whatever else you're using.

There's no need to introduce another socket protocol into that process.

-- 
Eric Wong

From ibc at aliax.net  Fri Dec 25 17:16:46 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Fri, 25 Dec 2009 23:16:46 +0100
Subject: How to trap USR2 for custom ussage ?
In-Reply-To: <20091224194147.GC14191@dcvr.yhbt.net>
References: <200912231529.11880.ibc@aliax.net>
	<200912241040.25841.ibc@aliax.net>
	<20091224194147.GC14191@dcvr.yhbt.net>
Message-ID: <200912252316.46495.ibc@aliax.net>

El Jueves, 24 de Diciembre de 2009, Eric Wong escribi?:

> > As I've said in other thread I want DRb to get some info related to the
> > application. A DRb client console could be used to check application
> > database(s) connection and so.
> > What would you suggest for this rather than DRb server?
> 
> As I said before, write an endpoint in your HTTP application that
> calls and displays all the needed info you need.  Something that
> hits the database(s) and/or whatever else you're using.

Can Rack/Unicorn behave different depending on the request received port?
This is: I could run Unicorn in port 80 for clients access and port 5000 for 
admin access (status and so).
However I couldn't find in Rack specification, neither in Unicorn, a way to 
inspect the port in which the request was received by the http server. Is it 
possible?


> There's no need to introduce another socket protocol into that process.

Yes, I'm realizing that openging another socet protocol gets in conflict with 
Unicorn signals usage. For example USR2 + QUIT is really great as it allows to 
reload/restart a server and go back to the previos running instance if the new 
one fails in some way.
So I'll use HTTP instead of DRb protocol to display the server status.

Thanks a lot again.
 


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Fri Dec 25 17:09:23 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Fri, 25 Dec 2009 23:09:23 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091224193433.GB14191@dcvr.yhbt.net>
References: <200912230320.59469.ibc@aliax.net>
	<200912241030.02160.ibc@aliax.net>
	<20091224193433.GB14191@dcvr.yhbt.net>
Message-ID: <200912252309.24084.ibc@aliax.net>

El Jueves, 24 de Diciembre de 2009, Eric Wong escribi?:
> I?aki Baz Castillo <ibc at aliax.net> wrote:
> > El Mi?rcoles, 23 de Diciembre de 2009, Eric Wong escribi?:
> > However I'm building my app now and when start it it usually has errors,
> > and that's good as it means I'm working :)
> > But sincerelly I would like to see the error just once rather than many
> > times due to workers respawn :)
> >
> > Also, is there any "interval" parameter so a worker is not re-spawned
> > until such timeout (after die).
> 
> You can sleep in the before_fork hook.

You mean "before_exec", rigth? :)

Yes, it's the solution :)




> I actually didn't want to support --daemonize at all since setsid(8),
> is standard most GNU/Linux distros nowadays, but we support more than
> just GNU/Linux.

AFAIK setsid is not 100% valid to daemonize a process because:

- It doesn't handle the exit return code of the process execution (it just 
returns 0 even if the process returns 1 at startup).

- It remains writting in the current stdout (the terminal in which you run 
"setsid program_name".


Thanks a lot.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Fri Dec 25 17:21:23 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Fri, 25 Dec 2009 23:21:23 +0100
Subject: Why "--pid" is deprecated?
In-Reply-To: <20091224191735.GA14191@dcvr.yhbt.net>
References: <200912241436.26761.ibc@aliax.net>
	<20091224191735.GA14191@dcvr.yhbt.net>
Message-ID: <200912252321.23817.ibc@aliax.net>

El Jueves, 24 de Diciembre de 2009, Eric Wong escribi?:
> > Which is the real advantage of using "pid" directive in config file
> > rather  than "--pid"?
> 
> Not much, pid files suck either way.  Sadly there aren't many other
> options that's universally used/accepted.

Ubuntu upstart system (now also adopted by Debian) is a great improvement over 
classic init system as it behaves as a daemon which already knows the PID 
number used for each service (so it doesn't need to look for it in a file).
However I haven't experiment with upstart yet.


> Using ENV["UNICORN_PID"] could be an option in the config file if you
> want to share it between the config and init script.

The problem is that config file is usually a XML/INI/YAML/Ruby file, while a 
common init script is writen in Bash (99% of times), so reading a config 
parameter from the init script involves dirty parsing and so... :(

Well, I agree that using PID file is ugly and not cool at all, but it's widely 
used and it's what we have now :)




-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Fri Dec 25 18:58:13 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Sat, 26 Dec 2009 00:58:13 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091223072617.GA28704@dcvr.yhbt.net>
References: <200912230320.59469.ibc@aliax.net>
	<20091223072617.GA28704@dcvr.yhbt.net>
Message-ID: <200912260058.14046.ibc@aliax.net>

El Mi?rcoles, 23 de Diciembre de 2009, Eric Wong escribi?:
> > Usually if a worker (all the workers) fail to start at the moment of
> > running  it, it obviously means that there is some error in the
> > application with prevents it to run. It could be great if Unicorn could
> > detect it.
> 
> I'm generally hesitant to introduce code/features/bloat that aren't
> helpful to people who properly test configurations before deploying :)
> 
> Adding this code doesn't solve problems.  Either way the site can become
> inaccessible/unusable and problems are noticeable very quickly.

Ok, I should explain better why I need this feature:

I'm not coding a HTTP app on top of Unicorn to host in a server. Instead I'm 
building a XCAP server (RFC 4825) using Rack and Unicorn. I'll also publish a 
Ruby gem containing all the server (Unicorn + Rack application).

So, any user could install the gem, set the database(s), fill the 
configuration file and star the server in background by running the provided 
init script ("/etc/init.d/xcapserver start") which runs the server daemonized.

Imagine that the user wrote a typo, i.e:

  after_fork do |server, worker|
    # Note the space typo:
    worker.u ser("www-data", "www-data") if Process.euid == 0
  end

Then the init script would return 0 (success) but the server wouldn't work. 
The user would check "ps aux | grep xcapserver" and would see the xcapserver 
running. It would be complex to understand why the server it's failing.

So it would be great the following:

- The user start the init script (which calls "unicorn -D ...") with a 
hypotethical new option ("--no-error-time 2").

- The command "unicorn -D" remains in foreground for 2 seconds before master 
process going to background.

- Unicorn fails to create the workers at the moment (so within 2 seconds).

- Instead of re-spawning them, the command (still in foreground) exits with 
return code 1 (error).




But there is other case which is much wrose IMHO. Imagine the user writes a 
space typo:

    stde rr_path "/var/log/xcapserver.err.log"

or imagine it uses a path that doesn't exist (/var/log/xcapserver/ doesn't 
exist):

    stderr_path "/var/log/xcapserver/err.log"

In both cases "unicorn -D" returns 0 but the server is not running (no unicorn 
process running). So, why did it return 0? It's not an error when creating the 
workers, but when running also the master process. IMHO in this case Unicorn 
should return non zero (even if it called with "-D").


I'm playing right now with unicorn/launcher.rb (daemonize! method)but I get 
nothing. Unfortunatelly I think that the usage of fork makes very difficult 
the main command to behave as I desire (exit status).

If you could give me some tips I'd try to implement it.

Reall thanks a lot again and again :)




-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Fri Dec 25 20:30:31 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Sat, 26 Dec 2009 02:30:31 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912252309.24084.ibc@aliax.net>
References: <200912230320.59469.ibc@aliax.net>
	<20091224193433.GB14191@dcvr.yhbt.net>
	<200912252309.24084.ibc@aliax.net>
Message-ID: <200912260230.32277.ibc@aliax.net>

El Viernes, 25 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> AFAIK setsid is not 100% valid to daemonize a process because:
> 
> - It doesn't handle the exit return code of the process execution (it just 
> returns 0 even if the process returns 1 at startup).
> 
> - It remains writting in the current stdout (the terminal in which you run 
> "setsid program_name".

Forget my last comment as the redirection must be done for the program to run. 


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Fri Dec 25 23:29:45 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Sat, 26 Dec 2009 05:29:45 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
Message-ID: <200912260529.45530.ibc@aliax.net>

El S?bado, 26 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> I'm playing right now with unicorn/launcher.rb (daemonize! method)but I
>  get  nothing. Unfortunatelly I think that the usage of fork makes very
>  difficult the main command to behave as I desire (exit status).
> 
> If you could give me some tips I'd try to implement it.

I've implemented it! :)

If you are interested please review this file (I cannot attach it in the mail 
as it's rejected by the maillist):

  http://oversip.net/public/min_time_running.rb

It contains a modification for bin/unicorn and a rewritten 
lib/unicorn/launcher.rb.

The code works for me for the two cases I explained in my previous mail.
Of course it's an optional feature (I've implemented it with "-M --min-
running-time SECONDS" in the options parser).

Regards.

-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Sat Dec 26 00:53:18 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 25 Dec 2009 21:53:18 -0800
Subject: How to trap USR2 for custom ussage ?
In-Reply-To: <200912252316.46495.ibc@aliax.net>
References: <200912231529.11880.ibc@aliax.net>
	<200912241040.25841.ibc@aliax.net>
	<20091224194147.GC14191@dcvr.yhbt.net>
	<200912252316.46495.ibc@aliax.net>
Message-ID: <20091226055317.GA22433@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Jueves, 24 de Diciembre de 2009, Eric Wong escribi?:
> > > As I've said in other thread I want DRb to get some info related to the
> > > application. A DRb client console could be used to check application
> > > database(s) connection and so.
> > > What would you suggest for this rather than DRb server?
> > 
> > As I said before, write an endpoint in your HTTP application that
> > calls and displays all the needed info you need.  Something that
> > hits the database(s) and/or whatever else you're using.
> 
> Can Rack/Unicorn behave different depending on the request received port?
> This is: I could run Unicorn in port 80 for clients access and port 5000 for 
> admin access (status and so).
> However I couldn't find in Rack specification, neither in Unicorn, a way to 
> inspect the port in which the request was received by the http server. Is it 
> possible?

Unicorn should always be running behind nginx if you're dealing with
untrusted clients, so you should just have nginx protecting the admin
endpoints.  The SERVER_PORT Rack environment variable is
available, but not considered trustable since it reads the value
of the Host: HTTP header.

-- 
Eric Wong

From normalperson at yhbt.net  Sat Dec 26 01:16:52 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 25 Dec 2009 22:16:52 -0800
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912260529.45530.ibc@aliax.net>
References: <200912260529.45530.ibc@aliax.net>
Message-ID: <20091226061652.GB22433@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El S?bado, 26 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> > I'm playing right now with unicorn/launcher.rb (daemonize! method)but I
> >  get  nothing. Unfortunatelly I think that the usage of fork makes very
> >  difficult the main command to behave as I desire (exit status).
> > 
> > If you could give me some tips I'd try to implement it.
> 
> I've implemented it! :)
> 
> If you are interested please review this file (I cannot attach it in the mail 
> as it's rejected by the maillist):
> 
>   http://oversip.net/public/min_time_running.rb
> 
> It contains a modification for bin/unicorn and a rewritten 
> lib/unicorn/launcher.rb.

Interesting, I could be tempted to accept this with a few changes...

Process.detach() spawns a background Thread.  Having running threads
before forking won't fly with certain OS threading libraries (some
versions of FreeBSD, I believe, check MRI Redmine for some open bugs).

What you're trying to accomplish does not require threads.

Use trap(Symbol), trap(Integer) is harder to read and has a likelyhood
of being non-portable for certain signals.  Likewise with Process.kill
(0 being the only exception, of course).

I would trap() in the parent before any forking, in case the
sleep time is ever so low there's a race condition.  Probably no
need to handle USR1, either, just let the parent die on its own,
or alternatively, have the parent sleep forever in case something
unforseen happens in the children...

> The code works for me for the two cases I explained in my previous mail.
> Of course it's an optional feature (I've implemented it with "-M --min-
> running-time SECONDS" in the options parser).

I believe the nginx grace period is hard-coded at 5 seconds, which should
be enough.

-- 
Eric Wong

From ibc at aliax.net  Sat Dec 26 10:47:51 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Sat, 26 Dec 2009 16:47:51 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091226061652.GB22433@dcvr.yhbt.net>
References: <200912260529.45530.ibc@aliax.net>
	<20091226061652.GB22433@dcvr.yhbt.net>
Message-ID: <200912261647.51764.ibc@aliax.net>

El S?bado, 26 de Diciembre de 2009, Eric Wong escribi?:

> >   http://oversip.net/public/min_time_running.rb
> >
> > It contains a modification for bin/unicorn and a rewritten
> > lib/unicorn/launcher.rb.
> 
> Interesting, I could be tempted to accept this with a few changes...
> 
> Process.detach() spawns a background Thread.  Having running threads
> before forking won't fly with certain OS threading libraries (some
> versions of FreeBSD, I believe, check MRI Redmine for some open bugs).

Yes, but there is a problem:
Imagine I don't use Process.detach and the master process ends when loads the 
rack application (due to a typo or due to an error in unicorn conf file).
Then the master process would remain visible/alive but as zombie.
If so, when the controller process sends signal 0 to check the master process 
it will always receive 'true' (a zombie process is a process which can receive 
signals, it's does exist).

If I could check if the master process is zombie then I wouldn't need 
Process.detach, but I don't know how to check if a process is zombie, no way 
in:
  http://ruby-doc.org/core/classes/Process.html
  http://ruby-doc.org/core/classes/Process/Status.html

Any suggestion here?

 

> Use trap(Symbol), trap(Integer) is harder to read and has a likelyhood
> of being non-portable for certain signals.  Likewise with Process.kill
> (0 being the only exception, of course).

Sure, I'll change it.

 
> I would trap() in the parent before any forking, in case the
> sleep time is ever so low there's a race condition.

Ok.


> Probably no need to handle USR1, either, just let the parent die on its own,

Note that the parent sleeps for 'min_running_time' + 1 (to avoid exiting 
before receiving USR1 or USR2 from the controller process). But if the 
controller process sends USR1 then the parent can already exit without waiting 
one second more.
Well, this is not very interesting, but I think it's good in order to get a 
faster exit code (it's not a good idea that a daemon takes long time to start 
and return ans exit status).


> or alternatively, have the parent sleep forever in case something
> unforseen happens in the children...

But then the parent process doesn't end and continues in foreground. This is 
not valid for running as daemon. The parent process must exit with the 
appropriate status code after a few seconds to behave as any service running 
in background.


> > The code works for me for the two cases I explained in my previous mail.
> > Of course it's an optional feature (I've implemented it with "-M --min-
> > running-time SECONDS" in the options parser).
> 
> I believe the nginx grace period is hard-coded at 5 seconds, which should
> be enough.

Sorry, I don't know ngix. I need to use other reverse proxy (Pound) but let me 
open a new thread about it :)

Anyhow I don't understand what you mean with "nginx grace period". How does 
affect to Unicorn? Do you mean that, in case of implementing the above, 
Unicorn wouldn't need a --min-running-time option? (I expect it would be 
required since Unicorn is 100% independent of ngix)



Really thanks a lot. I'll do the changes you suggest and show you again (but 
still don't know how to avoid Process.detach for now...).



-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Sat Dec 26 13:23:24 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Sat, 26 Dec 2009 19:23:24 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912261647.51764.ibc@aliax.net>
References: <200912260529.45530.ibc@aliax.net>
	<20091226061652.GB22433@dcvr.yhbt.net>
	<200912261647.51764.ibc@aliax.net>
Message-ID: <200912261923.24626.ibc@aliax.net>

El S?bado, 26 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> El S?bado, 26 de Diciembre de 2009, Eric Wong escribi?:
> > >   http://oversip.net/public/min_time_running.rb
> > >
> > > It contains a modification for bin/unicorn and a rewritten
> > > lib/unicorn/launcher.rb.
> >
> > Interesting, I could be tempted to accept this with a few changes...
> >
> > Process.detach() spawns a background Thread.  Having running threads
> > before forking won't fly with certain OS threading libraries (some
> > versions of FreeBSD, I believe, check MRI Redmine for some open bugs).
> 
> Yes, but there is a problem:
> Imagine I don't use Process.detach and the master process ends when loads
>  the rack application (due to a typo or due to an error in unicorn conf
>  file). Then the master process would remain visible/alive but as zombie.
> If so, when the controller process sends signal 0 to check the master
>  process it will always receive 'true' (a zombie process is a process which
>  can receive signals, it's does exist).


Hi, I've improved it a bit with your suggestions and also simplified it a lot.
Now there is no a "controller process". Instead, if "Unicorn.run" raises then 
master process rescues the exception and sends USR1 to parent process so it 
exists with 1.
So, Process.detach is not used anymore :)

Please check it from here:

  http://oversip.net/public/unicorn_addons.rb



> > Use trap(Symbol), trap(Integer) is harder to read and has a likelyhood
> > of being non-portable for certain signals.  Likewise with Process.kill
> > (0 being the only exception, of course).

Done in new code.

 
> > I would trap() in the parent before any forking, in case the
> > sleep time is ever so low there's a race condition.

Done in new code.

 
> > Probably no need to handle USR1, either, just let the parent die on its
> > own,

Now just one signal plays: USR1. It is sent by master process to parent 
process if Unicorn.run raised an exception.


Regards.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Sat Dec 26 20:31:15 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Sat, 26 Dec 2009 17:31:15 -0800
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912261923.24626.ibc@aliax.net>
References: <200912260529.45530.ibc@aliax.net>
	<20091226061652.GB22433@dcvr.yhbt.net>
	<200912261647.51764.ibc@aliax.net>
	<200912261923.24626.ibc@aliax.net>
Message-ID: <20091227013115.GA28140@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El S?bado, 26 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> > El S?bado, 26 de Diciembre de 2009, Eric Wong escribi?:
> > > Interesting, I could be tempted to accept this with a few changes...
> > >
> > > Process.detach() spawns a background Thread.  Having running threads
> > > before forking won't fly with certain OS threading libraries (some
> > > versions of FreeBSD, I believe, check MRI Redmine for some open bugs).
> > 
> > Yes, but there is a problem:
> > Imagine I don't use Process.detach and the master process ends when loads
> >  the rack application (due to a typo or due to an error in unicorn conf
> >  file). Then the master process would remain visible/alive but as zombie.
> > If so, when the controller process sends signal 0 to check the master
> >  process it will always receive 'true' (a zombie process is a process which
> >  can receive signals, it's does exist).
> 
> Hi, I've improved it a bit with your suggestions and also simplified
> it a lot.  Now there is no a "controller process". Instead, if
> "Unicorn.run" raises then master process rescues the exception and
> sends USR1 to parent process so it exists with 1.

Hi, I realized Unicorn.run inside the daemonize method doesn't work.
The daemonize method is also used by Rainbows! (and Zbatery).

However, I've reimplemented much of your idea here so it does not
involve sleeping at all, but rather shares a pipe with the grandparent
(started by the user) and Unicorn master process.  The added benefit of
using a pipe is that there's no fuzzy sleeping involved at or grace
periods to worry about, too.

Also pushed out to the "ready_pipe" branch of
git://git.bogomips.org/unicorn.git

Let me know how it goes.

If everything looks good, I'll consider merging for 0.96.0.

>From 5eea32764571b721cd1a89cf9ebfa853c621ac9e Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Sat, 26 Dec 2009 17:04:57 -0800
Subject: [PATCH] exit with failure if master dies when daemonized
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This behavior change also means our grandparent (launched
from a controlling terminal or script) will wait until
the master process is ready before returning.

Thanks to I?aki Baz Castillo for the initial implementations
and inspiration.
---
 bin/unicorn             |    2 +-
 bin/unicorn_rails       |    2 +-
 lib/unicorn.rb          |    9 +++++++--
 lib/unicorn/launcher.rb |   37 +++++++++++++++++++++++++++++--------
 test/exec/test_exec.rb  |    2 +-
 5 files changed, 39 insertions(+), 13 deletions(-)

diff --git a/bin/unicorn b/bin/unicorn
index 651c2ff..5af021d 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -161,5 +161,5 @@ if $DEBUG
   })
 end
 
-Unicorn::Launcher.daemonize! if daemonize
+Unicorn::Launcher.daemonize!(options) if daemonize
 Unicorn.run(app, options)
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 4a22a8c..b1458fc 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -202,6 +202,6 @@ end
 
 if daemonize
   options[:pid] = rails_pid
-  Unicorn::Launcher.daemonize!
+  Unicorn::Launcher.daemonize!(options)
 end
 Unicorn.run(app, options)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 71d5994..ae05f03 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -25,7 +25,8 @@ module Unicorn
 
   class << self
     def run(app, options = {})
-      HttpServer.new(app, options).start.join
+      ready_pipe = options.delete(:ready_pipe)
+      HttpServer.new(app, options).start.join(ready_pipe)
     end
   end
 
@@ -313,7 +314,7 @@ module Unicorn
     # (or until a termination signal is sent).  This handles signals
     # one-at-a-time time and we'll happily drop signals in case somebody
     # is signalling us too often.
-    def join
+    def join(ready_pipe = nil)
       # this pipe is used to wake us up from select(2) in #join when signals
       # are trapped.  See trap_deferred
       init_self_pipe!
@@ -324,6 +325,10 @@ module Unicorn
       trap(:CHLD) { |sig_nr| awaken_master }
       proc_name 'master'
       logger.info "master process ready" # test_exec.rb relies on this message
+      if ready_pipe
+        ready_pipe.syswrite($$.to_s)
+        ready_pipe.close
+      end
       begin
         loop do
           reap_all_workers
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 1229b84..2d6ad97 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -19,21 +19,42 @@ class Unicorn::Launcher
   #     the directory it was started in when being re-executed
   #     to pickup code changes if the original deployment directory
   #     is a symlink or otherwise got replaced.
-  def self.daemonize!
+  def self.daemonize!(options = {})
     $stdin.reopen("/dev/null")
+    $stdin.sync = true # may not do anything...
 
     # We only start a new process group if we're not being reexecuted
     # and inheriting file descriptors from our parent
     unless ENV['UNICORN_FD']
-      exit if fork
-      Process.setsid
-      exit if fork
+      # grandparent - reads pipe, exits when master is ready
+      #  \_ parent  - exits immediately ASAP
+      #      \_ unicorn master - writes to pipe when ready
 
-      # $stderr/$stderr can/will be redirected separately in the Unicorn config
-      Unicorn::Configurator::DEFAULTS[:stderr_path] = "/dev/null"
-      Unicorn::Configurator::DEFAULTS[:stdout_path] = "/dev/null"
+      rd, wr = IO.pipe
+      grandparent = $$
+      if fork
+        wr.close # grandparent does not write
+      else
+        rd.close # unicorn master does not read
+        Process.setsid
+        exit if fork # parent dies now
+      end
+
+      if grandparent == $$
+        # this will block until HttpServer#join runs (or it dies)
+        master_pid = rd.sysread(16).to_i
+        exit!(1) unless master_pid > 1
+        exit 0
+      else # unicorn master process
+        options[:ready_pipe] = wr
+        # $stderr/$stderr can/will be redirected separately in the
+        # Unicorn config
+        Unicorn::Configurator::DEFAULTS[:stderr_path] = "/dev/null"
+        Unicorn::Configurator::DEFAULTS[:stdout_path] = "/dev/null"
+
+        # returns so Unicorn.run can happen
+      end
     end
-    $stdin.sync = $stdout.sync = $stderr.sync = true
   end
 
 end
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index fc0719b..24ba856 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -805,7 +805,7 @@ EOF
       exec($unicorn_bin, "-D", "-l#{@addr}:#{@port}", "-c#{ucfg.path}")
     end
     pid, status = Process.waitpid2(pid)
-    assert status.success?, "original process exited successfully"
+    assert ! status.success?, "original process exited successfully"
     sleep 1 # can't waitpid on a daemonized process :<
     assert err.stat.size > 0
   end
-- 
Eric Wong

From ibc at aliax.net  Sat Dec 26 22:06:39 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Sun, 27 Dec 2009 04:06:39 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091227013115.GA28140@dcvr.yhbt.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912261923.24626.ibc@aliax.net>
	<20091227013115.GA28140@dcvr.yhbt.net>
Message-ID: <200912270406.39535.ibc@aliax.net>

El Domingo, 27 de Diciembre de 2009, Eric Wong escribi?:
> > Hi, I've improved it a bit with your suggestions and also simplified
> > it a lot.  Now there is no a "controller process". Instead, if
> > "Unicorn.run" raises then master process rescues the exception and
> > sends USR1 to parent process so it exists with 1.
> 
> Hi, I realized Unicorn.run inside the daemonize method doesn't work.

Strange, it works for me...

 
> However, I've reimplemented much of your idea here so it does not
> involve sleeping at all, but rather shares a pipe with the grandparent
> (started by the user) and Unicorn master process.  The added benefit of
> using a pipe is that there's no fuzzy sleeping involved at or grace
> periods to worry about, too.
> 
> Also pushed out to the "ready_pipe" branch of
> git://git.bogomips.org/unicorn.git
> 
> Let me know how it goes.
> 
> If everything looks good, I'll consider merging for 0.96.0.

It's really awesome! I've tested it and it works great, and avoids the 
sleeping stuff and a new commandlline option. Congratulations :)

However there is still a not solved case. Let me please explain it with two 
examples:

1) In "after_fork" I set:
     worker.user("non_existing_user","non_existing_group")

The master process would start properly and would notify "success" to 
grandparent. (so the init script returns 0). But the fact is that all the 
workers fail to start and are respawned again and again.


2) I use "Clogger" in my Rack config.ru. But I set a log file in a path that 
doesn't exist. Clogger raises due to this (it doesn't try to create the full 
path).
Again the master process has notified "success" to grandparent (exis status 0 
then) but the workers are respawned again and again.


In both cases, if "preload_app" is false then doing a "ps" I see new master 
processes being created by first master process (with new workers) and all of 
them die while new ones born.
If "preload_app" is false then there are no borning master processes as just 
the workers fail to start when loading the Rack app.


It would be great if the same mechanims (master notifies to grandparent) would 
be implemented between first process loading the Rack app (a worker if 
"preload_app") and master process, so if the first worker fails when loading 
the Rack app then master process doesn't notify "success" to grandparent and 
exists (instead of respawning again and again the workers).

If "preload_app" is true then I don't understand why your new code doesn't 
work right now. I expect that master process notifies "success" to grandparent 
before loading the Rack app, am I right? would make sense master process to 
wait until Rack app is loaded and if it fails then notify "error" and exit 
completely?


Kind regards and thanks a lot for your great work.











-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Sat Dec 26 22:07:53 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Sun, 27 Dec 2009 04:07:53 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912270406.39535.ibc@aliax.net>
References: <200912260529.45530.ibc@aliax.net>
	<20091227013115.GA28140@dcvr.yhbt.net>
	<200912270406.39535.ibc@aliax.net>
Message-ID: <200912270407.53163.ibc@aliax.net>

El Domingo, 27 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> In both cases, if "preload_app" is false then doing a "ps" I see new
>  master  processes being created by first master process (with new workers)
>  and all of them die while new ones born.

Here I meant:

  if "preload_app" is true ...


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Sun Dec 27 14:58:33 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Sun, 27 Dec 2009 20:58:33 +0100
Subject: Do we let Nginx to handle response Transfer-Encoding?
Message-ID: <200912272058.33744.ibc@aliax.net>

Hi, I've checked that in case my Rack application (under Unicorn) doesn't set 
"Content-Lentgh" neither "Transfer-Encoding", then Nginx automatically adds 
"Transfer-Encoding: chunked" to the response (and edits the body properly to 
be in chunked format).

However Unicorn is sending a response with body but without "Content-Lentgh" 
or "Transfer-Encoding", so IMHO it's incorrect according to RFC 2616.

So, should my Rack application handle this stuff? or is it better if I let 
Nginx "fixing" it for me?

Thanks a lot.

-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Sun Dec 27 19:37:58 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Mon, 28 Dec 2009 01:37:58 +0100
Subject: Listening UNIX socket is not deleted when stopping Unicorn?
Message-ID: <200912280137.59162.ibc@aliax.net>

Hi, I listen into an UNIX socket but after stopping Unicorn the socket is not 
removed. Is it the expected behavior?

This causes problem of permissions when changing the user/group running the 
workers (as they cannot remove the old socket with different owner).

Regards.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Sun Dec 27 22:15:46 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Sun, 27 Dec 2009 19:15:46 -0800
Subject: Listening UNIX socket is not deleted when stopping Unicorn?
In-Reply-To: <200912280137.59162.ibc@aliax.net>
References: <200912280137.59162.ibc@aliax.net>
Message-ID: <20091228031546.GA4349@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> Hi, I listen into an UNIX socket but after stopping Unicorn the socket
> is not removed. Is it the expected behavior?

Yes.

Otherwise it's subject to race conditions where the socket
owned by a new/replacement process gets its socket unlinked.

Currently Unicorn unlinks any existing socket on the FS before
attempting to bind to it for the following reasons:

* I consider this less error-prone, especially when people aren't
  storing sockets in a directory that's cleared on reboots (like /tmp).

* This can also be desirable behavior since it can be used to do
  transparent upgrades/binary replacements in cases where it's less
  convenient to use USR2+QUIT,  if you're switching between Ruby
  installations for example.

> This causes problem of permissions when changing the user/group
> running the workers (as they cannot remove the old socket with
> different owner).

Since there's absolutely no point in running Unicorn on port 80/443, you
should just avoid user switching entirely since it'll significantly
simplify your setup(s).

-- 
Eric Wong

From normalperson at yhbt.net  Sun Dec 27 22:17:00 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Sun, 27 Dec 2009 19:17:00 -0800
Subject: Do we let Nginx to handle response Transfer-Encoding?
In-Reply-To: <200912272058.33744.ibc@aliax.net>
References: <200912272058.33744.ibc@aliax.net>
Message-ID: <20091228031700.GB4349@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> Hi, I've checked that in case my Rack application (under Unicorn) doesn't set 
> "Content-Lentgh" neither "Transfer-Encoding", then Nginx automatically adds 
> "Transfer-Encoding: chunked" to the response (and edits the body properly to 
> be in chunked format).
> 
> However Unicorn is sending a response with body but without "Content-Lentgh" 
> or "Transfer-Encoding", so IMHO it's incorrect according to RFC 2616.
> 
> So, should my Rack application handle this stuff? or is it better if I let 
> Nginx "fixing" it for me?

Just load the Rack::ContentLength middleware, if you're building a Rack
app from scratch.  Some frameworks already handle Content-Length for
you.

-- 
Eric Wong

From normalperson at yhbt.net  Sun Dec 27 22:29:02 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Sun, 27 Dec 2009 19:29:02 -0800
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912270406.39535.ibc@aliax.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912261923.24626.ibc@aliax.net>
	<20091227013115.GA28140@dcvr.yhbt.net>
	<200912270406.39535.ibc@aliax.net>
Message-ID: <20091228032902.GC4349@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Domingo, 27 de Diciembre de 2009, Eric Wong escribi?:
> > > Hi, I've improved it a bit with your suggestions and also simplified
> > > it a lot.  Now there is no a "controller process". Instead, if
> > > "Unicorn.run" raises then master process rescues the exception and
> > > sends USR1 to parent process so it exists with 1.
> > 
> > Hi, I realized Unicorn.run inside the daemonize method doesn't work.
> 
> Strange, it works for me...

"Doesn't work" as in it's not friendly to servers based on Unicorn (like
Rainbows!).

> > However, I've reimplemented much of your idea here so it does not
> > involve sleeping at all, but rather shares a pipe with the grandparent
> > (started by the user) and Unicorn master process.  The added benefit of
> > using a pipe is that there's no fuzzy sleeping involved at or grace
> > periods to worry about, too.
> > 
> > Also pushed out to the "ready_pipe" branch of
> > git://git.bogomips.org/unicorn.git
> > 
> > Let me know how it goes.
> > 
> > If everything looks good, I'll consider merging for 0.96.0.
> 
> It's really awesome! I've tested it and it works great, and avoids the 
> sleeping stuff and a new commandlline option. Congratulations :)

The current version is actually slightly buggy in that it leaks
the pipe descriptor...

> However there is still a not solved case. Let me please explain it with two 
> examples:
> 
> 1) In "after_fork" I set:
>      worker.user("non_existing_user","non_existing_group")
> 
> The master process would start properly and would notify "success" to 
> grandparent. (so the init script returns 0). But the fact is that all the 
> workers fail to start and are respawned again and again.

For that particular case there'll be a Unicorn::Configurator#user
directive.

But really, there's absolutely no good reason to use user switching in a
backend application server like Unicorn.

I only added that feature to support derivative servers like Rainbows!,
and even then it's debatable since using things like iptables or load
balancers can be used to redirect port 80 to arbitrary ports anyways.

> 2) I use "Clogger" in my Rack config.ru. But I set a log file in a path that 
> doesn't exist. Clogger raises due to this (it doesn't try to create the full 
> path).
> Again the master process has notified "success" to grandparent (exis status 0 
> then) but the workers are respawned again and again.

There's really an infinite number of ways to screw things up badly in
workers and cause them to flap.  It's not possible to protect careless
users from all of them, so attempting to do so would be a waste of
effort.

Avoiding user-switching in an app server is a great first step, as it
eliminates an entire class of problems :)

-- 
Eric Wong

From ibc at aliax.net  Mon Dec 28 05:23:50 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 28 Dec 2009 11:23:50 +0100
Subject: Do we let Nginx to handle response Transfer-Encoding?
In-Reply-To: <20091228031700.GB4349@dcvr.yhbt.net>
References: <200912272058.33744.ibc@aliax.net>
	<20091228031700.GB4349@dcvr.yhbt.net>
Message-ID: <200912281123.50741.ibc@aliax.net>

El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> I?aki Baz Castillo <ibc at aliax.net> wrote:
> > Hi, I've checked that in case my Rack application (under Unicorn) doesn't
> > set "Content-Lentgh" neither "Transfer-Encoding", then Nginx
> > automatically adds "Transfer-Encoding: chunked" to the response (and
> > edits the body properly to be in chunked format).
> >
> > However Unicorn is sending a response with body but without
> > "Content-Lentgh" or "Transfer-Encoding", so IMHO it's incorrect according
> > to RFC 2616.
> >
> > So, should my Rack application handle this stuff? or is it better if I
> > let Nginx "fixing" it for me?
> 
> Just load the Rack::ContentLength middleware, if you're building a Rack
> app from scratch.  Some frameworks already handle Content-Length for
> you.

The fact is that to realizing about the above, I already added Content-Length 
by myself in the Rck application :)

Thanks. 


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Mon Dec 28 05:39:08 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 28 Dec 2009 11:39:08 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091228032902.GC4349@dcvr.yhbt.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912270406.39535.ibc@aliax.net>
	<20091228032902.GC4349@dcvr.yhbt.net>
Message-ID: <200912281139.09065.ibc@aliax.net>

El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:

> > The master process would start properly and would notify "success" to
> > grandparent. (so the init script returns 0). But the fact is that all the
> > workers fail to start and are respawned again and again.
> 
> For that particular case there'll be a Unicorn::Configurator#user
> directive.
> 
> But really, there's absolutely no good reason to use user switching in a
> backend application server like Unicorn.
> 
> I only added that feature to support derivative servers like Rainbows!,
> and even then it's debatable since using things like iptables or load
> balancers can be used to redirect port 80 to arbitrary ports anyways.

Well, chaning the running user it's common in most of servers. I've already 
found lots of cases of attacks to Apache servers running some "cool" PHP 
application (so we get exploits in /tmp or/var/tmp as they are the only 
writable paths for "www-data" user running apache).

However it's true that Unicorn approach (worker.user) is different as the 
master process remains as root (but since the master process doesn't listen it 
shouldn't matter).

So, do you mean that there will be a new configuration option called "user" 
(and "group") so also themaster process would run as such user?

Thanks.

-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Mon Dec 28 05:39:56 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 28 Dec 2009 11:39:56 +0100
Subject: Listening UNIX socket is not deleted when stopping Unicorn?
In-Reply-To: <20091228031546.GA4349@dcvr.yhbt.net>
References: <200912280137.59162.ibc@aliax.net>
	<20091228031546.GA4349@dcvr.yhbt.net>
Message-ID: <200912281139.56857.ibc@aliax.net>

El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> Otherwise it's subject to race conditions where the socket
> owned by a new/replacement process gets its socket unlinked.
> 
> Currently Unicorn unlinks any existing socket on the FS before
> attempting to bind to it for the following reasons:
> 
> * I consider this less error-prone, especially when people aren't
>   storing sockets in a directory that's cleared on reboots (like /tmp).
> 
> * This can also be desirable behavior since it can be used to do
>   transparent upgrades/binary replacements in cases where it's less
>   convenient to use USR2+QUIT,  if you're switching between Ruby
>   installations for example.


ok, understood. Thanks. 


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Mon Dec 28 05:48:51 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 28 Dec 2009 11:48:51 +0100
Subject: Listening UNIX socket is not deleted when stopping Unicorn?
In-Reply-To: <20091228031546.GA4349@dcvr.yhbt.net>
References: <200912280137.59162.ibc@aliax.net>
	<20091228031546.GA4349@dcvr.yhbt.net>
Message-ID: <200912281148.51362.ibc@aliax.net>

El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> Since there's absolutely no point in running Unicorn on port 80/443, you
> should just avoid user switching entirely since it'll significantly
> simplify your setup(s).

The problem are init scripts as they are executed by init process (as root).
There are some ways to run a commandwith other user (as "su") but AFAIK most 
of the servers implement full user switching (not just worker processes) by 
themself.

Regards. 


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Mon Dec 28 06:51:57 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 28 Dec 2009 12:51:57 +0100
Subject: Listening UNIX socket is not deleted when stopping Unicorn?
In-Reply-To: <200912281148.51362.ibc@aliax.net>
References: <200912280137.59162.ibc@aliax.net>
	<20091228031546.GA4349@dcvr.yhbt.net>
	<200912281148.51362.ibc@aliax.net>
Message-ID: <200912281251.57176.ibc@aliax.net>

El Lunes, 28 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > Since there's absolutely no point in running Unicorn on port 80/443, you
> > should just avoid user switching entirely since it'll significantly
> > simplify your setup(s).
> 
> The problem are init scripts as they are executed by init process (as
>  root). There are some ways to run a commandwith other user (as "su") but
>  AFAIK most of the servers implement full user switching (not just worker
>  processes) by themself.

I've found this workaround by adding the following (extracted from 
Worker#user) at the top of unicorn.conf.rb:

if Process.euid == 0 && (user = MyApp::Config[:user])
  group = MyApp::Config[:group]
  uid = Etc.getpwnam(user).uid
  gid = Etc.getgrnam(group).gid if group
  if gid && Process.egid != gid
    Process.initgroups(user, gid)
    Process::GID.change_privilege(gid)
  end
  Process.euid != uid and Process::UID.change_privilege(uid)
end

Of course using it a worker cannot listen in port <= 1024, but as you said, 
what's the problem with that? :)

Regards.



-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Mon Dec 28 07:50:44 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 28 Dec 2009 13:50:44 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091228032902.GC4349@dcvr.yhbt.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912270406.39535.ibc@aliax.net>
	<20091228032902.GC4349@dcvr.yhbt.net>
Message-ID: <200912281350.44760.ibc@aliax.net>

El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > It's really awesome! I've tested it and it works great, and avoids the 
> > sleeping stuff and a new commandlline option. Congratulations :)
> 
> The current version is actually slightly buggy in that it leaks
> the pipe descriptor...

I've detected some other "issue":

Imagine port 80 is used by other application (as apache) and Unicorn is 
configured to also bind in same port.


When running it in foreground all is great:

  ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
  ERROR -- : retrying in 3 seconds (1 tries left)
  ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
  ERROR -- : retrying in 3 seconds (0 tries left)
  ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
  /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/socket_helper.rb:110:in  
  `initialize': Address already in use - bind(2) (Errno::EADDRINUSE)


But when running in background an ugly error is displayed:

  /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/launcher.rb:45:in `sysread':
  end of file reached (EOFError)
    from /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/launcher.rb:45:in
    `daemonize!'


In both cases $? is 1 (error) but in the last case the error message is not 
very useful.

I suspect what is happening: the master tries several times to bind and after 
N retries it terminates (so it closes the pipe and grandparent gets EOFError).

As a suggestion, could the grandparent rescue such exception and display some 
kind of error message?:

  "The master couldn't be started. Inspect the log error or run in foreground"

Regards.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Mon Dec 28 14:06:33 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 28 Dec 2009 11:06:33 -0800
Subject: Listening UNIX socket is not deleted when stopping Unicorn?
In-Reply-To: <200912281148.51362.ibc@aliax.net>
References: <200912280137.59162.ibc@aliax.net>
	<20091228031546.GA4349@dcvr.yhbt.net>
	<200912281148.51362.ibc@aliax.net>
Message-ID: <20091228190633.GA4230@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > Since there's absolutely no point in running Unicorn on port 80/443, you
> > should just avoid user switching entirely since it'll significantly
> > simplify your setup(s).
> 
> The problem are init scripts as they are executed by init process (as root).
> There are some ways to run a commandwith other user (as "su") but AFAIK most 
> of the servers implement full user switching (not just worker processes) by 
> themself.

Hi I?aki,

User switching in the master isn't possible because we need to support
USR2 upgrades (and even some/most cases of HUP/USR1 handling).

But yes, just use su to become a regular user in an init script.

Since you're using a Linux box, maybe you can try getting something with
capabilities and enabling CAP_NET_BIND_SERVICE somehow... I never took
the time to get that working right.

-- 
Eric Wong

From normalperson at yhbt.net  Mon Dec 28 14:25:12 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 28 Dec 2009 11:25:12 -0800
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912281350.44760.ibc@aliax.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912270406.39535.ibc@aliax.net>
	<20091228032902.GC4349@dcvr.yhbt.net>
	<200912281350.44760.ibc@aliax.net>
Message-ID: <20091228192512.GB4230@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > > It's really awesome! I've tested it and it works great, and avoids the 
> > > sleeping stuff and a new commandlline option. Congratulations :)
> > 
> > The current version is actually slightly buggy in that it leaks
> > the pipe descriptor...
> 
> I've detected some other "issue":
> 
> Imagine port 80 is used by other application (as apache) and Unicorn is 
> configured to also bind in same port.
> 
> 
> When running it in foreground all is great:
> 
>   ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
>   ERROR -- : retrying in 3 seconds (1 tries left)
>   ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
>   ERROR -- : retrying in 3 seconds (0 tries left)
>   ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
>   /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/socket_helper.rb:110:in  
>   `initialize': Address already in use - bind(2) (Errno::EADDRINUSE)
> 
> 
> But when running in background an ugly error is displayed:
> 
>   /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/launcher.rb:45:in `sysread':
>   end of file reached (EOFError)
>     from /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/launcher.rb:45:in
>     `daemonize!'
> 
> 
> In both cases $? is 1 (error) but in the last case the error message is not 
> very useful.
> 
> I suspect what is happening: the master tries several times to bind and after 
> N retries it terminates (so it closes the pipe and grandparent gets EOFError).
> 
> As a suggestion, could the grandparent rescue such exception and display some 
> kind of error message?:
> 
>   "The master couldn't be started. Inspect the log error or run in foreground"

Thanks, pushed out to the ready_pipe branch of unicorn.git

>From 52eee4e424198a3c80793ee9c930fd3bb0285168 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Mon, 28 Dec 2009 11:16:00 -0800
Subject: [PATCH] launcher: descriptive error message on startup failure
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Rather than erroring out with a non-descript EOFError,
show a warning message telling users to check the logs
instead.

Reported-by: I?aki Baz Castillo mid=200912281350.44760.ibc at aliax.net
---
 lib/unicorn/launcher.rb |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 0ea836b..1871420 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -42,8 +42,11 @@ class Unicorn::Launcher
 
       if grandparent == $$
         # this will block until HttpServer#join runs (or it dies)
-        master_pid = rd.readpartial(16).to_i
-        exit!(1) unless master_pid > 1
+        master_pid = (rd.readpartial(16) rescue nil).to_i
+        unless master_pid > 1
+          warn "master failed to start, check stderr log for details"
+          exit!(1)
+        end
         exit 0
       else # unicorn master process
         options[:ready_pipe] = wr
-- 
Eric Wong

From ibc at aliax.net  Mon Dec 28 15:17:27 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 28 Dec 2009 21:17:27 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091228192512.GB4230@dcvr.yhbt.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912281350.44760.ibc@aliax.net>
	<20091228192512.GB4230@dcvr.yhbt.net>
Message-ID: <200912282117.28125.ibc@aliax.net>

El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > As a suggestion, could the grandparent rescue such exception and display
> > some  kind of error message?:
> > 
> >   "The master couldn't be started. Inspect the log error or run in
> > foreground"
> 
> Thanks, pushed out to the ready_pipe branch of unicorn.git

Hi, I attach a minor patch to make the error more verbose when Unicorn cannot
listen in a socket.

For example, as normal user (not root) I try to listen in a UNIX socket (ok)
and also on TCP port 84 (permission error). Actually I would see this error log:

I, [2009-12-28T21:05:31.503533 #26137]  INFO -- : unlinking existing socket=/tmp/unicorn.sock
I, [2009-12-28T21:05:31.504019 #26137]  INFO -- : listening on addr=/tmp/unicorn.sock fd=3
/usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/socket_helper.rb:110:in `initialize': Permission denied - bind(2) (Errno::EACCES)

By inspecting the error it seems that the problem is in the UNIX socket.

With my patch the log would be:

I, [2009-12-28T21:11:18.165918 #26589]  INFO -- : unlinking existing socket=/tmp/openxdms.sock
I, [2009-12-28T21:11:18.166228 #26589]  INFO -- : listening on addr=/tmp/openxdms.sock fd=3
F, [2009-12-28T21:11:18.166632 #26589] FATAL -- : error adding listener addr=0.0.0.0:84
/usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/socket_helper.rb:110:in `initialize': Permission denied - bind(2) (Errno::EACCES)


I hope it's useful. Regards.

-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Mon Dec 28 15:32:59 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 28 Dec 2009 12:32:59 -0800
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912282117.28125.ibc@aliax.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912281350.44760.ibc@aliax.net>
	<20091228192512.GB4230@dcvr.yhbt.net>
	<200912282117.28125.ibc@aliax.net>
Message-ID: <20091228203259.GB23032@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > > As a suggestion, could the grandparent rescue such exception and display
> > > some  kind of error message?:
> > > 
> > >   "The master couldn't be started. Inspect the log error or run in
> > > foreground"
> > 
> > Thanks, pushed out to the ready_pipe branch of unicorn.git
> 
> Hi, I attach a minor patch to make the error more verbose when Unicorn cannot
> listen in a socket.

Hi, it seems like you didn't attach the patch.  Attaching patches is
strongly discouraged, though, inline patches are far easier to
review/edit/apply.

-- 
Eric Wong

From ibc at aliax.net  Mon Dec 28 15:41:51 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 28 Dec 2009 21:41:51 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091228203259.GB23032@dcvr.yhbt.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912282117.28125.ibc@aliax.net>
	<20091228203259.GB23032@dcvr.yhbt.net>
Message-ID: <200912282141.51953.ibc@aliax.net>

El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> I?aki Baz Castillo <ibc at aliax.net> wrote:
> > El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > > > As a suggestion, could the grandparent rescue such exception and
> > > > display some  kind of error message?:
> > > >
> > > >   "The master couldn't be started. Inspect the log error or run in
> > > > foreground"
> > >
> > > Thanks, pushed out to the ready_pipe branch of unicorn.git
> >
> > Hi, I attach a minor patch to make the error more verbose when Unicorn
> > cannot listen in a socket.
> 
> Hi, it seems like you didn't attach the patch.

Ops, usually my mail client (Kmail) warns me when I write tha word "attach" 
but attach nothing... not sure why it didn't it this time.


> Attaching patches is
> strongly discouraged, though, inline patches are far easier to
> review/edit/apply.

Ok, here it's (note that it's done over 'ready_pipe' branch):


--- unicorn.rb.orig     2009-12-28 21:02:47.000000000 +0100
+++ unicorn.rb  2009-12-28 21:11:12.000000000 +0100
@@ -307,6 +307,9 @@
                      "(#{tries < 0 ? 'infinite' : tries} tries left)"
         sleep(delay)
         retry
+      rescue => err
+        logger.fatal "error adding listener addr=#{address}"
+        raise err
       end
     end


-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Mon Dec 28 19:17:00 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 28 Dec 2009 16:17:00 -0800
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <200912282141.51953.ibc@aliax.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912282117.28125.ibc@aliax.net>
	<20091228203259.GB23032@dcvr.yhbt.net>
	<200912282141.51953.ibc@aliax.net>
Message-ID: <20091229001700.GA25599@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > I?aki Baz Castillo <ibc at aliax.net> wrote:
> > > El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > > > > As a suggestion, could the grandparent rescue such exception and
> > > > > display some  kind of error message?:
> > > > >
> > > > >   "The master couldn't be started. Inspect the log error or run in
> > > > > foreground"
> > > >
> > > > Thanks, pushed out to the ready_pipe branch of unicorn.git
> > >
> > > Hi, I attach a minor patch to make the error more verbose when Unicorn
> > > cannot listen in a socket.
> > 
> > Hi, it seems like you didn't attach the patch.
> 
> Ops, usually my mail client (Kmail) warns me when I write tha word "attach" 
> but attach nothing... not sure why it didn't it this time.
> 
> > Attaching patches is
> > strongly discouraged, though, inline patches are far easier to
> > review/edit/apply.
> 
> Ok, here it's (note that it's done over 'ready_pipe' branch):

Thanks, applied with your name and pushed out to both ready_pipe and
master in unicorn.git

> --- unicorn.rb.orig     2009-12-28 21:02:47.000000000 +0100
> +++ unicorn.rb  2009-12-28 21:11:12.000000000 +0100

It should be easier to make a commit using git and then just
"git format-patch -M" to output it, proposed commit message and all.

In the HACKING doc, I've been recommending people follow the
SubmittingPatches document distributed with git when submitting
patches to Unicorn:

 http://git.bogomips.org/cgit/mirrors/git.git/tree/Documentation/SubmittingPatches

-- 
Eric Wong

From ibc at aliax.net  Mon Dec 28 19:32:54 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Tue, 29 Dec 2009 01:32:54 +0100
Subject: "unicorn -D" always returns 0 "success" (even when failed to load)
In-Reply-To: <20091229001700.GA25599@dcvr.yhbt.net>
References: <200912260529.45530.ibc@aliax.net>
	<200912282141.51953.ibc@aliax.net>
	<20091229001700.GA25599@dcvr.yhbt.net>
Message-ID: <200912290132.54471.ibc@aliax.net>

El Martes, 29 de Diciembre de 2009, Eric Wong escribi?:
> Thanks, applied with your name and pushed out to both ready_pipe and
> master in unicorn.git

Thanks a lot :)

 
> > --- unicorn.rb.orig     2009-12-28 21:02:47.000000000 +0100
> > +++ unicorn.rb  2009-12-28 21:11:12.000000000 +0100
> 
> It should be easier to make a commit using git and then just
> "git format-patch -M" to output it, proposed commit message and all.
> 
> In the HACKING doc, I've been recommending people follow the
> SubmittingPatches document distributed with git when submitting
> patches to Unicorn:
> 
>  http://git.bogomips.org/cgit/mirrors/git.git/tree/Documentation/Submitting
> Patches

Ok, I'll read it.
Thanks againg. 


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Thu Dec 31 09:57:57 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Thu, 31 Dec 2009 15:57:57 +0100
Subject: Listening UNIX socket is not deleted when stopping Unicorn?
In-Reply-To: <20091228031546.GA4349@dcvr.yhbt.net>
References: <200912280137.59162.ibc@aliax.net>
	<20091228031546.GA4349@dcvr.yhbt.net>
Message-ID: <200912311557.57499.ibc@aliax.net>

El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > Hi, I listen into an UNIX socket but after stopping Unicorn the socket
> > is not removed. Is it the expected behavior?
> 
> Yes.
> 
> Otherwise it's subject to race conditions where the socket
> owned by a new/replacement process gets its socket unlinked.
> 
> Currently Unicorn unlinks any existing socket on the FS before
> attempting to bind to it for the following reasons:
> 
> * I consider this less error-prone, especially when people aren't
>   storing sockets in a directory that's cleared on reboots (like /tmp).
> 
> * This can also be desirable behavior since it can be used to do
>   transparent upgrades/binary replacements in cases where it's less
>   convenient to use USR2+QUIT,  if you're switching between Ruby
>   installations for example.

Hi, just a question:

Then is it totaly safe to delete the socket before starting Unicorn? 
I expect the answer is yes as Unicorn does it.

Thanks.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Thu Dec 31 13:01:14 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Thu, 31 Dec 2009 19:01:14 +0100
Subject: PID is not regenerated when using "-P" and the master receives USR2
Message-ID: <200912311901.14620.ibc@aliax.net>

In case of using "unicorn -P PIDFILE" when the master receives a USR2 the 
pidfile is deleted and not regenerated anymore.

In case of using "pid" mehotd in config file then it works well.

It seems a bug, do I miss something?

Regards.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Thu Dec 31 13:23:41 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Thu, 31 Dec 2009 19:23:41 +0100
Subject: PID is not regenerated when using "-P" and the master receives
	USR2
In-Reply-To: <200912311901.14620.ibc@aliax.net>
References: <200912311901.14620.ibc@aliax.net>
Message-ID: <200912311923.41504.ibc@aliax.net>

El Jueves, 31 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> In case of using "unicorn -P PIDFILE" when the master receives a USR2 the
> pidfile is deleted and not regenerated anymore.
> 
> In case of using "pid" mehotd in config file then it works well.
> 
> It seems a bug, do I miss something?

Definitively I think this is the expected behaviour as the new binary has no 
way to know the PIDFILE passed as commandline argument to original binary, am 
I right?

-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Thu Dec 31 14:42:45 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 31 Dec 2009 11:42:45 -0800
Subject: Listening UNIX socket is not deleted when stopping Unicorn?
In-Reply-To: <200912311557.57499.ibc@aliax.net>
References: <200912280137.59162.ibc@aliax.net>
	<20091228031546.GA4349@dcvr.yhbt.net>
	<200912311557.57499.ibc@aliax.net>
Message-ID: <20091231194245.GB24888@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Lunes, 28 de Diciembre de 2009, Eric Wong escribi?:
> > > Hi, I listen into an UNIX socket but after stopping Unicorn the socket
> > > is not removed. Is it the expected behavior?
> > 
> > Yes.
> > 
> > Otherwise it's subject to race conditions where the socket
> > owned by a new/replacement process gets its socket unlinked.
> > 
> > Currently Unicorn unlinks any existing socket on the FS before
> > attempting to bind to it for the following reasons:
> > 
> > * I consider this less error-prone, especially when people aren't
> >   storing sockets in a directory that's cleared on reboots (like /tmp).
> > 
> > * This can also be desirable behavior since it can be used to do
> >   transparent upgrades/binary replacements in cases where it's less
> >   convenient to use USR2+QUIT,  if you're switching between Ruby
> >   installations for example.
> 
> Hi, just a question:
> 
> Then is it totaly safe to delete the socket before starting Unicorn? 
> I expect the answer is yes as Unicorn does it.

Yes.

-- 
Eric Wong

From normalperson at yhbt.net  Thu Dec 31 15:01:17 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 31 Dec 2009 12:01:17 -0800
Subject: PID is not regenerated when using "-P" and the master receives
	USR2
In-Reply-To: <200912311923.41504.ibc@aliax.net>
References: <200912311901.14620.ibc@aliax.net>
	<200912311923.41504.ibc@aliax.net>
Message-ID: <20091231200117.GC24888@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Jueves, 31 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> > In case of using "unicorn -P PIDFILE" when the master receives a USR2 the
> > pidfile is deleted and not regenerated anymore.
> > 
> > In case of using "pid" mehotd in config file then it works well.
> > 
> > It seems a bug, do I miss something?
> 
> Definitively I think this is the expected behaviour as the new binary has no 
> way to know the PIDFILE passed as commandline argument to original binary, am 
> I right?

All command-line arguments are passed to the child process.  We capture
the command-line as soon as possible before option parsing happens.

When USR2 is received, we create a new pid file for the old
binary and add the ".oldbin" suffix to the pathname.

That said, "-P" in the command-line and "pid" in the config file should
work the same.  Despite being officially deprecated, "-P" is used
extensively and interchangeably with "pid" in the existing tests.

-- 
Eric Wong

From ibc at aliax.net  Thu Dec 31 15:40:50 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Thu, 31 Dec 2009 21:40:50 +0100
Subject: PID is not regenerated when using "-P" and the master receives
	USR2
In-Reply-To: <20091231200117.GC24888@dcvr.yhbt.net>
References: <200912311901.14620.ibc@aliax.net>
	<200912311923.41504.ibc@aliax.net>
	<20091231200117.GC24888@dcvr.yhbt.net>
Message-ID: <200912312140.50446.ibc@aliax.net>

El Jueves, 31 de Diciembre de 2009, Eric Wong escribi?:
> I?aki Baz Castillo <ibc at aliax.net> wrote:
> > El Jueves, 31 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> > > In case of using "unicorn -P PIDFILE" when the master receives a USR2
> > > the pidfile is deleted and not regenerated anymore.
> > >
> > > In case of using "pid" mehotd in config file then it works well.
> > >
> > > It seems a bug, do I miss something?
> >
> > Definitively I think this is the expected behaviour as the new binary has
> > no way to know the PIDFILE passed as commandline argument to original
> > binary, am I right?
> 
> All command-line arguments are passed to the child process.  We capture
> the command-line as soon as possible before option parsing happens.
> 
> When USR2 is received, we create a new pid file for the old
> binary and add the ".oldbin" suffix to the pathname.
> 
> That said, "-P" in the command-line and "pid" in the config file should
> work the same.  Despite being officially deprecated, "-P" is used
> extensively and interchangeably with "pid" in the existing tests.

Yes, I confirm that using pure "unicorn" command with "-P" it works (both 
"name.pid" and "name.pid.oldbin" exist after USR2).

However I use a modified version of "unicorn" executable, which is the same 
with just two differences:

- The name is different.

- The commandline "-c" and "rackup" are removed. Instead these values are 
fixed and I set them into the executable script by adding:

  options[:config_file]  = "PATH_TO_UNICORN_CONFIG.rb"
  config                 = "PATH_TO_CONFIG.ru"


So to run as daemon I just type:

  ~# myapp -D -P /tmp/myapp.pid


Could it explain my issue? I see "/tmp/myapp.pid.oldbin" after USR2, but I 
don't see a new "/tmp/myapp.pid".


Thanks a lot for your help.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From ibc at aliax.net  Thu Dec 31 16:26:39 2009
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Thu, 31 Dec 2009 22:26:39 +0100
Subject: PID is not regenerated when using "-P" and the master receives
	USR2
In-Reply-To: <200912312140.50446.ibc@aliax.net>
References: <200912311901.14620.ibc@aliax.net>
	<20091231200117.GC24888@dcvr.yhbt.net>
	<200912312140.50446.ibc@aliax.net>
Message-ID: <200912312226.39253.ibc@aliax.net>

El Jueves, 31 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> El Jueves, 31 de Diciembre de 2009, Eric Wong escribi?:
> > I?aki Baz Castillo <ibc at aliax.net> wrote:
> > > El Jueves, 31 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> > > > In case of using "unicorn -P PIDFILE" when the master receives a USR2
> > > > the pidfile is deleted and not regenerated anymore.
> > > >
> > > > In case of using "pid" mehotd in config file then it works well.
> > > >
> > > > It seems a bug, do I miss something?
> > >
> > > Definitively I think this is the expected behaviour as the new binary
> > > has no way to know the PIDFILE passed as commandline argument to
> > > original binary, am I right?
> >
> > All command-line arguments are passed to the child process.  We capture
> > the command-line as soon as possible before option parsing happens.
> >
> > When USR2 is received, we create a new pid file for the old
> > binary and add the ".oldbin" suffix to the pathname.
> >
> > That said, "-P" in the command-line and "pid" in the config file should
> > work the same.  Despite being officially deprecated, "-P" is used
> > extensively and interchangeably with "pid" in the existing tests.
> 
> Yes, I confirm that using pure "unicorn" command with "-P" it works (both
> "name.pid" and "name.pid.oldbin" exist after USR2).
> 
> However I use a modified version of "unicorn" executable, which is the same
> with just two differences:
> 
> - The name is different.
> 
> - The commandline "-c" and "rackup" are removed. Instead these values are
> fixed and I set them into the executable script by adding:
> 
>   options[:config_file]  = "PATH_TO_UNICORN_CONFIG.rb"
>   config                 = "PATH_TO_CONFIG.ru"
> 
> 
> So to run as daemon I just type:
> 
>   ~# myapp -D -P /tmp/myapp.pid
> 
> 
> Could it explain my issue? I see "/tmp/myapp.pid.oldbin" after USR2, but I
> don't see a new "/tmp/myapp.pid".
> 
> 
> Thanks a lot for your help.


I've some issue somewhere as when sending the USR2 I see:

  INFO: executing ["/usr/local/bin/myapp"] (in /)

while it should be ["/usr/local/bin/myapp","-P","/tmp/myapp.pid","-D"]


-- 
I?aki Baz Castillo <ibc at aliax.net>

From normalperson at yhbt.net  Thu Dec 31 16:41:04 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 31 Dec 2009 21:41:04 +0000
Subject: PID is not regenerated when using "-P" and the master receives
	USR2
In-Reply-To: <200912312226.39253.ibc@aliax.net>
References: <200912311901.14620.ibc@aliax.net>
	<20091231200117.GC24888@dcvr.yhbt.net>
	<200912312140.50446.ibc@aliax.net>
	<200912312226.39253.ibc@aliax.net>
Message-ID: <20091231214104.GA19467@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> El Jueves, 31 de Diciembre de 2009, I?aki Baz Castillo escribi?:
> > El Jueves, 31 de Diciembre de 2009, Eric Wong escribi?:
> > > I?aki Baz Castillo <ibc at aliax.net> wrote:
> > > > El Jueves, 31 de Diciembre de 2009, I?aki Baz Castillo escribi?:

<snip>

> > > All command-line arguments are passed to the child process.  We capture
> > > the command-line as soon as possible before option parsing happens.
> > 
> > However I use a modified version of "unicorn" executable, which is the same
> > with just two differences:
> > 
> > - The name is different.
> > 
> > - The commandline "-c" and "rackup" are removed. Instead these values are
> > fixed and I set them into the executable script by adding:
> > 
> >   options[:config_file]  = "PATH_TO_UNICORN_CONFIG.rb"
> >   config                 = "PATH_TO_CONFIG.ru"
> > 
> > 
> > So to run as daemon I just type:
> > 
> >   ~# myapp -D -P /tmp/myapp.pid
> > 
> > 
> > Could it explain my issue? I see "/tmp/myapp.pid.oldbin" after USR2, but I
> > don't see a new "/tmp/myapp.pid".
> > 
> > 
> > Thanks a lot for your help.
> 
> 
> I've some issue somewhere as when sending the USR2 I see:
> 
>   INFO: executing ["/usr/local/bin/myapp"] (in /)
> 
> while it should be ["/usr/local/bin/myapp","-P","/tmp/myapp.pid","-D"]

You need to require 'unicorn/launcher' or 'unicorn' before you do any
option parsing to stash ARGV in a safe place.

You can also play with the Unicorn::HttpServer::START_CTX hash
and modify the :argv array value if needed...

-- 
Eric Wong