From normalperson at yhbt.net Wed Jul 1 18:58:44 2009 From: normalperson at yhbt.net (Eric Wong) Date: Wed, 1 Jul 2009 15:58:44 -0700 Subject: [ANN] unicorn 0.9.0 (experimental release) Message-ID: <20090701225843.GA7475@dcvr.yhbt.net> Unicorn is a Rack HTTP server for Unix, fast clients and nothing else[1] We now have support for "Transfer-Encoding: chunked" bodies in requests. Not only that, Rack applications reading input bodies get that data streamed off to the client socket on an as-needed basis. This allows the application to do things like upload progress notification and even tunneling of arbitrary stream protocols via bidirectional chunked encoding. See Unicorn::App::Inetd and examples/git.ru (including the comments) for an example of tunneling the git:// protocol over HTTP. This release also gives applications the ability to respond positively to "Expect: 100-continue" headers before being rerun without closing the socket connection. See Unicorn::App::Inetd for an example of how this is used. This release is NOT recommended for production use. Eric Wong (43): http_request: no need to reset the request http_request: StringIO is binary for empty bodies (1.9) http_request: fix typo for 1.9 Transfer-Encoding: chunked streaming input support Unicorn::App::Inetd: reinventing Unix, poorly :) README: update with mailing list info local.mk.sample: publish_doc gzips all html, js, css Put copyright text in new files, include GPL2 text examples/cat-chunk-proxy: link to proposed curl(1) patch Update TODO Avoid duplicating the "Z" constant Optimize body-less GET/HEAD requests (again) tee_input: Don't expose the @rd object as a return value exec_cgi: small cleanups README: another note about older Sinatra tee_input: avoid defining a @rd.size method Make TeeInput easier to use test_upload: add tests for chunked encoding GNUmakefile: more stringent error checking in tests test_upload: fix ECONNRESET with 1.9 GNUmakefile: allow TRACER= to be specified for tests test_rails: workaround long-standing 1.9 bug tee_input: avoid rereading fresh data "Fix" tests that break with stream_input=false inetd: fix broken constant references configurator: provide stream_input (true|false) option chunked_reader: simpler interface http_request: force BUFFER to be Encoding::BINARY ACK clients on "Expect: 100-continue" header Only send "100 Continue" when no body has been sent http_request: tighter Transfer-Encoding: "chunked" check Add trailer_parser for parsing trailers chunked_reader: Add test for chunk parse failure TeeInput: use only one IO for tempfile trailer_parser: set keys with "HTTP_" prefix TrailerParser integration into ChunkedReader Unbind listeners as before stopping workers Retry listen() on EADDRINUSE 5 times ever 500ms Re-add support for non-portable socket options Move "Expect: 100-continue" handling to the app tee_input: avoid ignoring initial body blob Force streaming input onto apps by default unicorn 0.9.0 * site: http://unicorn.bogomips.org/ * git: git://git.bogomips.org/unicorn.git * http+git: http://git.bogomips.org:8080/unicorn.git [1] * cgit: http://git.bogomips.org/cgit/unicorn.git/ * list: mongrel-unicorn at rubyforge.org [1] - Actually, most of the new features in this release should in fact work on non-Unix OSes so they can be adapted for other servers with wider audiences. [2] - This is the git:// protocol tunneled inside a bidirectional "Transfer-Encoding: chunked" request/response using this latest release of Unicorn. -- Eric Wong From paul at sponagl.de Mon Jul 6 04:21:56 2009 From: paul at sponagl.de (Paul Sponagl) Date: Mon, 6 Jul 2009 10:21:56 +0200 Subject: Unicorn Message-ID: <218F5913-4D49-4F6C-B334-CF264120D61A@sponagl.de> Hi Eric, the Featurelist sounds great ! I was using qrp for a long time and now i am eagerly awaiting a production ready release of unicorn :) Btw. I tried it on OS-X with -l /tmp/thin.0.sock and it took a while for nginx to spit out the page... ... (60: Operation timed out) while reading upstream, client: 127.0.0.1, server: w..., request: "GET / HTTP/1.1", upstream: "http://unix :/tmp/thin.0.sock:/" ... but finally it appeared (for now it is not development ready for me ;) But this evening i'll give it a try with tcp sockets. thank you for your efforts in this great project and all the best Paul !DSPAM:4a51b6975171980815732! From normalperson at yhbt.net Mon Jul 6 19:16:46 2009 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 6 Jul 2009 16:16:46 -0700 Subject: Unicorn In-Reply-To: <218F5913-4D49-4F6C-B334-CF264120D61A@sponagl.de> References: <218F5913-4D49-4F6C-B334-CF264120D61A@sponagl.de> Message-ID: <20090706231646.GA22306@dcvr.yhbt.net> Paul Sponagl wrote: > Hi Eric, > > the Featurelist sounds great ! > > I was using qrp for a long time and now i am eagerly awaiting a > production ready release of unicorn :) > > Btw. I tried it on OS-X with -l /tmp/thin.0.sock and it took a while for > nginx to spit out the page... > > ... (60: Operation timed out) while reading upstream, client: 127.0.0.1, > server: w..., request: "GET / HTTP/1.1", upstream: "http://unix > :/tmp/thin.0.sock:/" ... > > but finally it appeared (for now it is not development ready for me ;) > But this evening i'll give it a try with tcp sockets. Weird. I'm doubtful TCP sockets would solve your problems but let me know if it fixes it. UNIX and TCP sockets follow exactly the same code paths once the server is running so I'm not sure what could be the issue (especially since you're sending a body-less GET request). Maybe OSX has some weird rules/bugs with UNIX sockets (I've heard of issues with FIFOs on OSX), but I don't have access to an OSX machine. Is your "GET / HTTP/1.1" request supposed to be taking over 60s? 60 seconds is the default (and IMHO very conservative) timeout for when Unicorn decides to terminate a request. 0.8.1 should actually be considered production-ready. It doesn't support chunked transfer-encoding; but otherwise it's been running fine for some internal APIs for quite a while now. > thank you for your efforts in this great project and all the best Thanks for giving it a shot! -- Eric Wong From paul at sponagl.de Tue Jul 7 04:23:49 2009 From: paul at sponagl.de (Paul Sponagl) Date: Tue, 7 Jul 2009 10:23:49 +0200 Subject: Unicorn In-Reply-To: <20090706231646.GA22306@dcvr.yhbt.net> References: <218F5913-4D49-4F6C-B334-CF264120D61A@sponagl.de> <20090706231646.GA22306@dcvr.yhbt.net> Message-ID: Hi Eric, unicorn does not close the connection on osx after sending the content. (It acts as the client would send a Connection: Keep-Alive) The connection will be aborted by nginx after about 60 secs and the content will be send out. Macintosh:~> telnet 127.0.0.1 3001 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET / HTTP/1.1 HTTP/1.1 200 OK Date: Tue, 07 Jul 2009 08:11:17 GMT Status: 200 OK Connection: close X-Runtime: 293 ETag: "2659d611050f70fc65c6c052687e5a13" Content-Type: text/html; charset=utf-8 Cache-Control: private, max-age=0, must-revalidate Content-Length: 24328 ... the rest of the content Hope that helps to track it down further. Best Paul Am 07.07.2009 um 01:16 schrieb Eric Wong: > Paul Sponagl wrote: >> Hi Eric, >> >> the Featurelist sounds great ! >> >> I was using qrp for a long time and now i am eagerly awaiting a >> production ready release of unicorn :) >> >> Btw. I tried it on OS-X with -l /tmp/thin.0.sock and it took a >> while for >> nginx to spit out the page... >> >> ... (60: Operation timed out) while reading upstream, client: >> 127.0.0.1, >> server: w..., request: "GET / HTTP/1.1", upstream: "http://unix >> :/tmp/thin.0.sock:/" ... >> >> but finally it appeared (for now it is not development ready for >> me ;) >> But this evening i'll give it a try with tcp sockets. > > Weird. I'm doubtful TCP sockets would solve your problems but let me > know if it fixes it. UNIX and TCP sockets follow exactly the same > code > paths once the server is running so I'm not sure what could be the > issue > (especially since you're sending a body-less GET request). Maybe OSX > has some weird rules/bugs with UNIX sockets (I've heard of issues with > FIFOs on OSX), but I don't have access to an OSX machine. > > Is your "GET / HTTP/1.1" request supposed to be taking over 60s? 60 > seconds is the default (and IMHO very conservative) timeout for when > Unicorn decides to terminate a request. > > 0.8.1 should actually be considered production-ready. It doesn't > support chunked transfer-encoding; but otherwise it's been running > fine for some internal APIs for quite a while now. > >> thank you for your efforts in this great project and all the best > > Thanks for giving it a shot! > > -- > Eric Wong > > > > !DSPAM:4a53088a5179781588283! From normalperson at yhbt.net Tue Jul 7 16:48:39 2009 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 7 Jul 2009 13:48:39 -0700 Subject: Unicorn In-Reply-To: References: <218F5913-4D49-4F6C-B334-CF264120D61A@sponagl.de> <20090706231646.GA22306@dcvr.yhbt.net> Message-ID: <20090707204839.GA2650@dcvr.yhbt.net> Paul Sponagl wrote: > Hi Eric, > > unicorn does not close the connection on osx after sending the content. > (It acts as the client would send a Connection: Keep-Alive) > The connection will be aborted by nginx after about 60 > secs and the content will be send out. > > Macintosh:~> telnet 127.0.0.1 3001 > Trying 127.0.0.1... > Connected to localhost. > Escape character is '^]'. > GET / HTTP/1.1 > > HTTP/1.1 200 OK > Date: Tue, 07 Jul 2009 08:11:17 GMT > Status: 200 OK > Connection: close > X-Runtime: 293 > ETag: "2659d611050f70fc65c6c052687e5a13" > Content-Type: text/html; charset=utf-8 > Cache-Control: private, max-age=0, must-revalidate > Content-Length: 24328 > ... > the rest of the content Hi Paul, This is strange. Can you tell that you're getting all 24328 bytes of the content? And no exception is thrown to stderr, either? Unicorn should've closed the socket even on an exception. Which version of OSX and Ruby are you running? Is there anything special/weird about the Rack response body you're sending? Does Rack::Lint complain about anything? Unicorn is fairly strict about Rack::Lint compatibility. Does the following patch help? diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb index bfaa33d..acf4d04 100644 --- a/lib/unicorn/http_response.rb +++ b/lib/unicorn/http_response.rb @@ -60,6 +60,7 @@ module Unicorn "Connection: close\r\n" \ "#{OUT.join(Z)}\r\n") body.each { |chunk| socket.write(chunk) } + socket.flush socket.close # flushes and uncorks the socket immediately ensure body.respond_to?(:close) and body.close rescue nil ------------------------------------------------------------------------ Or you can replace the "socket.close" with "exit!(0)" and really force it to close :) Thanks again for trying this! -- Eric Wong From paul at sponagl.de Thu Jul 9 04:06:03 2009 From: paul at sponagl.de (Paul Sponagl) Date: Thu, 9 Jul 2009 10:06:03 +0200 Subject: Unicorn In-Reply-To: <20090708220413.GA32416@dcvr.yhbt.net> References: <218F5913-4D49-4F6C-B334-CF264120D61A@sponagl.de> <20090706231646.GA22306@dcvr.yhbt.net> <20090707204839.GA2650@dcvr.yhbt.net> <20090708220413.GA32416@dcvr.yhbt.net> Message-ID: <3D052B79-A43B-4A7C-9ED4-032B31CBE664@sponagl.de> Eric - you're a genius ! This patch solved it! Indeed i am forking a sleeping cache refresh process at requests to '/' via a system call. Now i am ready to give it a try. Thank you very much for your help! All the best for you and unicorn Paul PS: Last time i forgot to cc to the list - i added my previous mail below to keep searchers- and engines happy. Am 09.07.2009 um 00:04 schrieb Eric Wong: > Paul Sponagl wrote: > > > >> hmmm - everything works as it should... > > Very odd, nothing out of the ordinary there. Is there any chance your > application forks/execs long-running processes in the background? I'm > relying on the FD_CLOEXEC flag to be inherited by accepted sockets > from > the original listen socket. Maybe that's not the case with OSX... > > Perhaps this patch can fix things under OSX? > > diff --git a/lib/unicorn.rb b/lib/unicorn.rb > index 281aa7d..f3073f4 100644 > --- a/lib/unicorn.rb > +++ b/lib/unicorn.rb > @@ -444,6 +444,7 @@ module Unicorn > # once a client is accepted, it is processed in its entirety here > # in 3 easy steps: read request, call app, write app response > def process_client(app, client) > + client.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) > response = app.call(env = REQUEST.read(client)) > > if 100 == response.first.to_i > >> i also tried this config.ru >> ---8<--- >> # config.ru >> require "config/environment" >> use Rack::Lint >> run ActionController::Dispatcher.new >> ---8<--- >> >> does this activate Rack::Lint ? > > Yes it should. > > Thanks for looking into this issue and doing detailed analysis > on it! > > -- > Eric Wong > > > > Hi Eric, ok, let me write down a few tests: unicorn_rails with static content: ------------------------------------------- Macintosh-2:> echo -n "7 bytes" > public/test.html Macintosh-2:> unicorn_rails I, [2009-07-08T09:46:53.445542 #25037] INFO -- : listening on addr=0.0.0.0:8080 fd=3 I, [2009-07-08T09:46:53.647616 #25037] INFO -- : worker=0 spawning... I, [2009-07-08T09:46:53.648553 #25037] INFO -- : master process ready I, [2009-07-08T09:46:53.651324 #25039] INFO -- : worker=0 spawned pid=25039 I, [2009-07-08T09:46:53.651812 #25039] INFO -- : Refreshing Gem list worker=0 ready Macintosh-2:~> telnet 127.0.0.1 8080 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /test.html HTTP/1.1 Host: test.local HTTP/1.1 200 OK Date: Wed, 08 Jul 2009 07:47:33 GMT Status: 200 OK Connection: close Last-Modified: Wed, 08 Jul 2009 07:46:42 GMT Content-Type: text/html Content-Length: 7 7 bytesConnection closed by foreign host. Worked ... unicorn_rails with simple rails content: -------------------------------------------------- class HomeController < ApplicationController layout 'home' def test render :text => '7 bytes' end Macintosh-2:~> telnet 127.0.0.1 8080 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /home/test HTTP/1.1 Host: test.local HTTP/1.1 200 OK Date: Wed, 08 Jul 2009 07:50:23 GMT Status: 200 OK Connection: close X-Runtime: 2059 ETag: "4028bd3c2c4b2eec87994d1a46c86878" Content-Type: text/html; charset=utf-8 Cache-Control: private, max-age=0, must-revalidate Content-Length: 7 Set-Cookie: flash=%7B%7D; path=/ Set-Cookie: _bilderbuch_session=...; path=/; HttpOnly 7 bytesConnection closed by foreign host. worked... thin with more complex rails content: -------------------------------------------------- Macintosh-2:~> wget --server-response http://127.0.0.1:8080/ --10:01:52-- http://127.0.0.1:8080/ => `index.html.3' Verbindungsaufbau zu 127.0.0.1:8080... verbunden. HTTP Anforderung gesendet, warte auf Antwort... 1 HTTP/1.1 200 OK 2 X-Runtime: 446 3 ETag: "0eda35d2574d23011a57cff8043800c8" 4 Content-Type: text/html; charset=utf-8 5 Cache-Control: private, max-age=0, must-revalidate 6 Set-Cookie: flash=%7B%7D; path=/ 7 Set-Cookie: _bilderbuch_session=...; path=/; HttpOnly 8 Content-Length: 24340 9 Connection: keep-alive 10 Server: thin 1.0.0 codename That's What She Said 100%[======>] 24,340 23.21M/s ETA 00:00 10:01:53 (23.21 MB/s) - ?index.html.3? gespeichert [24340/24340] Macintosh-2:~> ls -la index.html.3 -rw-r--r-- 1 paulspon paulspon 24340 8. Jul 10:01 index.html.3 worked (content-length and filesize matches)... unicorn_rails with more complex rails content: ------------------------------------------------------------- Macintosh-2:~> wget --server-response http://127.0.0.1:8080/ --10:02:50-- http://127.0.0.1:8080/ => `index.html.4' Verbindungsaufbau zu 127.0.0.1:8080... verbunden. HTTP Anforderung gesendet, warte auf Antwort... 1 HTTP/1.1 200 OK 2 Date: Wed, 08 Jul 2009 08:02:51 GMT 3 Status: 200 OK 4 Connection: close 5 X-Runtime: 1541 6 ETag: "e750924df3f2890ab7d8a560d0088c1a" 7 Content-Type: text/html; charset=utf-8 8 Cache-Control: private, max-age=0, must-revalidate 9 Content-Length: 24376 10 Set-Cookie: flash=%7B%7D; path=/ 11 Set-Cookie: _bilderbuch_session=...; path=/; HttpOnly 0% [ ] 0 --.--K/s ETA --:-- does not work (waiting forever...) but - after killing unicorn_rails Macintosh-2:~> ls -la index.html.4 -rw-r--r-- 1 paulspon paulspon 24376 8. Jul 10:02 index.html.4 its there... unicorn_rails with more complex rails content: ( after adding your suggested socket.flush: ) ------------------------------------------------------------- no changes - still timeouts... a few more tests: ---------------------------- socket.write("HTTP/1.1 #{status}\r\n" \ "Date: #{Time.now.httpdate}\r\n" \ "Status: #{status}\r\n" \ "Connection: close\r\n" \ "#{OUT.join(Z)}\r\n") size = 0 puts "after socket.write HEADER" body.each { |chunk| size += chunk.length socket.write(chunk) } puts "after socket.write BODY #{size}" socket.flush exit!(0) Content lengt matches output wget :... 9 Content-Length: 25257 ... after socket.write HEADER after socket.write BODY 25257 I, [2009-07-08T10:18:16.137773 #25334] INFO -- : reaped # worker=0 I, [2009-07-08T10:18:16.161050 #25334] INFO -- : worker=0 spawning... ... hmmm - everything works as it should... i also tried this config.ru ---8<--- # config.ru require "config/environment" use Rack::Lint run ActionController::Dispatcher.new ---8<--- does this activate Rack::Lint ? this is all really strange - do i miss something simple in here ??? Best Paul !DSPAM:4a55a7665173716248820! From normalperson at yhbt.net Thu Jul 9 05:09:51 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 9 Jul 2009 02:09:51 -0700 Subject: [ANN] unicorn 0.8.2 and 0.9.1 Message-ID: <20090709090951.GA3173@dcvr.yhbt.net> Unicorn is a Rack HTTP server for Unix, fast clients and nothing else Socket handling bugfixes and socket-related usability and performance tweaks. We no longer trust FD_CLOEXEC to be inherited across accept(); thanks to Paul Sponagl for diagnosing this issue on OSX. There are also minor tweaks backported from 0.9.0 to make non-graceful restarts/upgrades go more smoothly. The 0.9.x series (including 0.9.1) is NOT recommended for production use, try 0.8.2 instead. * site: http://unicorn.bogomips.org/ * git: git://git.bogomips.org/unicorn.git * cgit: http://git.bogomips.org/cgit/unicorn.git/ * list: mongrel-unicorn at rubyforge.org -- Eric Wong From normalperson at yhbt.net Thu Jul 9 06:10:44 2009 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 9 Jul 2009 03:10:44 -0700 Subject: Unicorn In-Reply-To: <3D052B79-A43B-4A7C-9ED4-032B31CBE664@sponagl.de> References: <218F5913-4D49-4F6C-B334-CF264120D61A@sponagl.de> <20090706231646.GA22306@dcvr.yhbt.net> <20090707204839.GA2650@dcvr.yhbt.net> <20090708220413.GA32416@dcvr.yhbt.net> <3D052B79-A43B-4A7C-9ED4-032B31CBE664@sponagl.de> Message-ID: <20090709101044.GB32416@dcvr.yhbt.net> Paul Sponagl wrote: > Eric - you're a genius ! This patch solved it! Cool, but a genius would've avoided this problem entirely :) > Indeed i am forking a sleeping cache refresh process at requests to '/' > via a system call. > Now i am ready to give it a try. Thank you very much for your help! > > All the best for you and unicorn > Paul No problem, thanks for reporting the issues. I've just released 0.8.2 (stable) and 0.9.1 (experimental) with the fix. Hopefully that's the last 0.8.x release since I've bigger plans for 0.9.x (or 0.10.x). > PS: Last time i forgot to cc to the list - i added my previous mail > below to keep searchers- and engines happy. Thanks again, btw can you avoid top-posting here? I'm really old school like that :) -- Eric Wong From normalperson at yhbt.net Fri Jul 10 15:52:20 2009 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 10 Jul 2009 12:52:20 -0700 Subject: unicorn 0.8.2 and 0.9.1 In-Reply-To: <3670459b7de642ba5d5a0ce074534b83@ruby-forum.com> References: <20090709090951.GA3173@dcvr.yhbt.net> <3670459b7de642ba5d5a0ce074534b83@ruby-forum.com> Message-ID: <20090710195220.GA13208@dcvr.yhbt.net> Roger Pack wrote: > Eric Wong wrote: > > Unicorn is a Rack HTTP server for Unix, fast clients and nothing else > > So who exactly would benefit most from this library? What is the target > audience? Just wondering more background on it. The target audience is people deploying/maintaining/managing HTTP apps that only clients within their LAN are expected/allowed to hit. Of course, with a decent reverse proxy in front, all traffic from the reverse proxy can be considered internal and fast (unless your LAN is horrible). Clients don't have to be on a Unix-like OS, of course. It's also a testing ground for features that may make it into Mongrel 2 (assuming portability requirements are met). Unicorn gives us more freedom to exploit features unique to Unix-like systems (and maybe even (optionally) Linux 2.6+ in the future). Fwiw, I've known about (and implemented similar (but meaner :)) things like the recently-disclosed Slowloris for several years now. That's why we refuse to support Unicorn talking directly to the Internet at large. There's also some philosophy and design notes here that describe why we did some of the things we did: http://unicorn.bogomips.org/PHILOSOPHY.html http://unicorn.bogomips.org/DESIGN.html -- Eric Wong From normalperson at yhbt.net Sun Jul 19 21:51:38 2009 From: normalperson at yhbt.net (Eric Wong) Date: Sun, 19 Jul 2009 18:51:38 -0700 Subject: [ANN] unicorn 0.8.3 and 0.9.2 Message-ID: <20090720015130.GA6145@dcvr.yhbt.net> Unicorn is a Rack HTTP server for Unix, fast clients and nothing else These releases fixes compatibility issues the Ruby 1.9.2 preview1 release (and one existing 1.9.x issue). Note that Rails 2.3.2.1 does NOT appear to work with Ruby 1.9.2 preview1, but that is outside the scope of this project. The 0.9.x series (including 0.9.2) is only recommended for development/experimental use. This series is NOT recommended for production use, use 0.8.3 instead. * site: http://unicorn.bogomips.org/ * git: git://git.bogomips.org/unicorn.git * cgit: http://git.bogomips.org/cgit/unicorn.git/ * list: mongrel-unicorn at rubyforge.org -- Eric Wong