From normalperson at yhbt.net  Mon Oct  5 06:58:28 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 5 Oct 2009 03:58:28 -0700
Subject: [ANN] Rainbows! 0.1.0 initial release
Message-ID: <20091005105828.GA19193@dcvr.yhbt.net>

Rainbows! Unicorn for slow apps and slow clients

Rainbows! is a HTTP server for Rack applications.  It is based on
Unicorn, but designed to handle applications that expect long
request/response times and/or slow clients.  For Rack applications not
heavily bound by slow external dependencies, consider Unicorn instead as
it simpler and easier to debug.

Rainbows! includes previous work from the "gossamer" and "rainbows"
branches of Unicorn and will support additional network concurrency
models.  It is basically for all the things that Unicorn sucks
at :)

* http://rainbows.rubyforge.org/
* rainbows-talk at rubyforge.org
* git://git.bogomips.org/rainbows.git

Changes:

Initial release

This release is currently highly experimental and is still
missing a lot of test coverage.

-- 
Eric Wong

From jftucker at gmail.com  Mon Oct  5 18:10:34 2009
From: jftucker at gmail.com (James Tucker)
Date: Mon, 5 Oct 2009 23:10:34 +0100
Subject: Single Threaded Async Responses
Message-ID: <0956CE27-14FE-42DF-BAD9-CED31B09D85D@gmail.com>

Hi Eric and anyone else listening in,

I completed an async API for Thin quite some time ago well suited to  
some of the design goals you have here. If you're interested we could  
have a chat about supporting the same API designs that allow single  
threaded reactor based concurrent responses. I believe such an API  
would fit well in rainbows too.

Kind regards,

James Tucker

From normalperson at yhbt.net  Mon Oct  5 18:56:33 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 5 Oct 2009 15:56:33 -0700
Subject: Single Threaded Async Responses
In-Reply-To: <0956CE27-14FE-42DF-BAD9-CED31B09D85D@gmail.com>
References: <0956CE27-14FE-42DF-BAD9-CED31B09D85D@gmail.com>
Message-ID: <20091005225633.GA23138@dcvr.yhbt.net>

James Tucker <jftucker at gmail.com> wrote:
> Hi Eric and anyone else listening in,
>
> I completed an async API for Thin quite some time ago well suited to  
> some of the design goals you have here. If you're interested we could  
> have a chat about supporting the same API designs that allow single  
> threaded reactor based concurrent responses. I believe such an API would 
> fit well in rainbows too.

Hi James,

This sounds interesting.

Does this mean it does synchronous request reading?  That'll mean
it requires nginx in front to work well.

So the response is written asynchronously, meaning it can be bigger than
the outgoing TCP buffers and still not block, right?  I considered
something like this for Unicorn, too, but then I realised that all the
responses I see managed to fit into the TCP buffers without blocking
already.

But for weird OSes with tiny buffers or apps dealing with large
responses it would probably make sense for Rainbows! (especially
if you plan to help me support it into the future :)

Is it merged into Thin already?  Where can/should I take a look?

One bit to be careful for is the body of the Rack response may not be a
simple Array of String objects, so that requires extra effort/cycles to
stringify into something that can be buffered for async writes.  But
then that could be a huge chunk to keep around in memory (since we
mainly need this to support larger responses).  What nginx does is
it will buffer into a temporary file and then sendfile() it over
as the socket becomes readable.

-- 
Eric Wong

From normalperson at yhbt.net  Mon Oct  5 23:57:41 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 5 Oct 2009 20:57:41 -0700
Subject: [ANN] Rainbows! 0.1.1 - with Ruby 1.8 support!
Message-ID: <20091006035741.GA9318@dcvr.yhbt.net>

Rainbows! Unicorn for slow apps and slow clients

Rainbows! is a HTTP server for sleepy Rack applications.  It is based on
Unicorn, but designed to handle applications that expect long
request/response times and/or slow clients.  For Rack applications not
heavily bound by slow external network dependencies, consider Unicorn
instead as it simpler and easier to debug.

* http://rainbows.rubyforge.org/
* rainbows-talk at rubyforge.org
* git://git.bogomips.org/rainbows.git

Fixed Ruby 1.8 support (and all 1.9 systems without Revactor).
Process-wide timeout handling for the ThreadSpawn concurrency
model should now work properly.  Small cleanups everywhere.

-- 
Eric Wong

From normalperson at yhbt.net  Wed Oct  7 20:09:42 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 7 Oct 2009 17:09:42 -0700
Subject: why Unicorn doesn't do slow clients
Message-ID: <20091008000942.GA25054@dcvr.yhbt.net>

You may have heard of Slowloris and Nkiller2 (if not check them out),
but there is also david:

  http://git.bogomips.org/cgit/david.git

I wasn't comfortable with announcing this two years ago when I wrote it.
I'm OK now since Slowloris and Nkiller2 are similar (ok, Nkiller2 is
lower-level and meaner) and attacks of this type should be well-known by
now.  I'm sure some folks have known about these types of attack since
the 1990s, even.

I haven't touched david in over two years and don't have any desire to
maintain or support it.  Feel free to take and hack on it for your own
testing or even make it into a real project, but please be nice and
don't hurt people with it.

This is different from Slowloris in that it throttles the entire request
including the POST/PUT body.  Most proxies (including haproxy) don't
buffer bodies before sending them on whereas nginx will buffer large
ones to temporary files.


I originally wrote david to convince some friends to stick nginx in
front of anything that wasn't nginx; including Apache +
mod_perl|mod_php|mongrel and Tomcat setups.

And it was quite convincing at the time :>


Implementation details:

  * david combines fork() with select() to multiplex file descriptors
    and get around select() and non-priviledged process FD limits.

  * The defaults at the top of the davic.c file are all pretty tame,
    but they're all commented on and you can tweak them.

  * david mmaps a raw file that is an HTTP request, some small
    samples are provided but you can/should generate your own large
    POST/PUT requests to really stress request body handling.

-- 
Eric Wong

From sde at linuxlinks.com  Thu Oct  8 13:58:05 2009
From: sde at linuxlinks.com (sde at linuxlinks.com)
Date: Thu, 8 Oct 2009 18:58:05 +0100 (added by postmaster@april.london.02.net)
Subject: Addition: Rainbows!
Message-ID: <2082671588.1255024688133.JavaMail.sde@vostro>

Hi

One of our visitors has submitted your entry Rainbows! to LinuxLinks.com, a leading portal site for Linux, Mac OS X, and Java users.

This entry has been approved.  It will appear after the next update (under 24 hours), and will be listed in both the category "Software/Internet/WebServers" and in the What's New section, http://www.linuxlinks.com/New/

New additions are not listed immediately at the site as we prefer using static pages for our links database.

To help the community you can let us know of any changes to your listing, including any change in URL.  The link and description can be modified at http://www.linuxlinks.com/portal/cgi-bin/modify.cgi

New releases of Rainbows! can be announced by emailing webmaster at linuxlinks.com, or by posting details at http://www.linuxlinks.com/portal/news/submit.php

If you have any questions, please do not hesitate to contact me.

Regards,
Steve Emms
LinuxLinks.com - The Linux Portal Site

Tel: 323.284.7240
Skype: sde at linuxlinks.com




From normalperson at yhbt.net  Tue Oct 13 02:26:02 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 12 Oct 2009 23:26:02 -0700
Subject: per-client timeouts disabled in rainbows.git
Message-ID: <20091013062602.GA13128@dcvr.yhbt.net>

Since we can handle hundreds (if not thousands of concurrent clients),
per-process and spawn multiple processes, I think I'll leave those
timeouts disabled for now to keep the code simple.

Even if timeouts were enabled, it's still possible to write a simple
client that can keep a connection alive forever on end with periodic
requests...

I haven't seen a lot of idle/sleepy client attacks so I suspect
kiddies have better things to do than to DoS in this fashion.

Of course, the per-process timeout is still enabled in case your
app hits a blocking syscall/loop that freezes your entire process.

-- 
Eric Wong

From normalperson at yhbt.net  Tue Oct 13 02:42:14 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 12 Oct 2009 23:42:14 -0700
Subject: [PATCH] README: alter reply conventions for the mailing list
Message-ID: <20091013064214.GA16042@dcvr.yhbt.net>

Mailman is now configured to munge Reply-To: to point back to
the mailing list.  This might make things easier for folks
on low traffic mailing lists like ours.
---
 README |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/README b/README
index c801efd..8924891 100644
--- a/README
+++ b/README
@@ -129,8 +129,8 @@ and we'll try our best to fix it.
 All feedback (bug reports, user/development dicussion, patches, pull
 requests) go to the mailing list/newsgroup.  Patches must be sent inline
 (git format-patch -M + git send-email).  No subscription is necessary
-to post on the mailing list.  No top posting.  Address replies +To:+ (or
-+Cc:+) the original sender and +Cc:+ the mailing list.
+to post on the mailing list.  No top posting.  Address replies +To:+
+the mailing list.
 
 * email: mailto:rainbows-talk at rubyforge.org
 * archives: http://rubyforge.org/pipermail/rainbows-talk
-- 
Eric Wong

From normalperson at yhbt.net  Thu Oct 15 04:18:37 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 15 Oct 2009 01:18:37 -0700
Subject: [ANN] Rainbows! 0.2.0 w/ bare Rev support and AppPool middleware
Message-ID: <20091015081837.GA7692@dcvr.yhbt.net>

Rainbows! is a HTTP server for sleepy Rack applications.  It is based on
Unicorn, but designed to handle applications that expect long
request/response times and/or slow clients.  For Rack applications not
heavily bound by slow external network dependencies, consider Unicorn
instead as it simpler and easier to debug.

* http://rainbows.rubyforge.org/
* rainbows-talk at rubyforge.org
* git://git.bogomips.org/rainbows.git

Changes:

This release adds preliminary Rev support for network
concurrency under Ruby 1.8 and Ruby 1.9.  There are caveats to
this model and reading the RDoc for Rainbows::Rev is
recommended.

  http://rainbows.rubyforge.org/Rainbows/Rev.html


Rainbows::AppPool Rack middleware is now available to limit
application concurrency on a per-process basis independently of
network concurrency.  See the RDoc for this class for further
details.

  http://rainbows.rubyforge.org/Rainbows/AppPool.html


Per-client timeouts have been removed, see
http://mid.gmane.org/20091013062602.GA13128 at dcvr.yhbt.net
for the reasoning.

Rack environment changes:

* "rack.multithread" is now only true for models with "Thread"
  in their name.  Enabling thread-safe (but not reentrant) code
  may actually be harmful for Revactor.

* "rainbows.model" is now exposed so the application can easily
  figure out which network concurrency model is in use.

Bugfixes include better shutdown and error handling for all
existing models, OpenBSD compatibility for the per-process
heartbeat (same as found in unicorn v0.93.3).

-- 
Eric Wong

From normalperson at yhbt.net  Mon Oct 19 14:27:12 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 19 Oct 2009 11:27:12 -0700
Subject: [ANN] Rainbows! 0.3.0 w/ DeferredResponse for Rev
Message-ID: <20091019182712.GA11442@dcvr.yhbt.net>

Rainbows! is a HTTP server for sleepy Rack applications.  It is based on
Unicorn, but designed to handle applications that expect long
request/response times and/or slow clients.  For Rack applications not
heavily bound by slow external network dependencies, consider Unicorn
instead as it simpler and easier to debug.

* http://rainbows.rubyforge.org/
* rainbows-talk at rubyforge.org
* git://git.bogomips.org/rainbows.git

Changes:

The major feature of this release is the new DeferredResponse
middleware for the Rev-based concurrency model.  It should be
transparently compatible with non-Rev models, as well.  As a
pleasant side effect, this change also allows large files to be
streamed to the client with Rev as the socket becomes writable
instead of slurping the entire file into an IO::Buffer first.

Bugfixes to graceful shutdowns support for all concurrency
models.  The Rev-based model also gets a working heartbeat
mechanism (oops!) and fixed HTTP/1.1 pipelining support.

-- 
Eric Wong

From normalperson at yhbt.net  Mon Oct 19 14:33:39 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 19 Oct 2009 11:33:39 -0700
Subject: Single Threaded Async Responses
In-Reply-To: <0956CE27-14FE-42DF-BAD9-CED31B09D85D@gmail.com>
References: <0956CE27-14FE-42DF-BAD9-CED31B09D85D@gmail.com>
Message-ID: <20091019183339.GB11442@dcvr.yhbt.net>

James Tucker <jftucker at gmail.com> wrote:
> Hi Eric and anyone else listening in,
>
> I completed an async API for Thin quite some time ago well suited to  
> some of the design goals you have here. If you're interested we could  
> have a chat about supporting the same API designs that allow single  
> threaded reactor based concurrent responses. I believe such an API would 
> fit well in rainbows too.

Hi James,

I've looked at Thin and added this as a TODO item for Rainbows! once
EventMachine support gets added.

For now, we also have an a (more Rack-compatible[1] but certain
Unices-only[2]) API which relies on passing file descriptors to Rev in
Rainbows! 0.3.0.

[1] - The only thing that keeps it from running correctly under
      Rack::Lint is that Rack::Lint doesn't pass body#to_path calls
      to up the stack even though it _allows_ it.

[2] - Most Unices seem to support /dev/fd/#{io.fileno}
      to lookup the contents of the file descriptor table.

-- 
Eric Wong

From normalperson at yhbt.net  Mon Oct 19 17:30:52 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 19 Oct 2009 14:30:52 -0700
Subject: [ANN] Rainbows! 0.3.0 w/ DeferredResponse for Rev
In-Reply-To: <20091019182712.GA11442@dcvr.yhbt.net>
References: <20091019182712.GA11442@dcvr.yhbt.net>
Message-ID: <20091019213052.GB17783@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> Changes:
> 
> The major feature of this release is the new DeferredResponse
                That should be DevFdResponse --^^^^^^^^^^^^^^^^
> middleware for the Rev-based concurrency model.  It should be

-- 
Eric Wong

From normalperson at yhbt.net  Tue Oct 27 04:58:05 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 27 Oct 2009 01:58:05 -0700
Subject: [ANN] Rainbows! 0.4.0 EventMachine+async_sinatra support
Message-ID: <20091027085805.GB29122@dcvr.yhbt.net>

Rainbows! is a HTTP server for sleepy Rack applications.  It is based on
Unicorn, but designed to handle applications that expect long
request/response times and/or slow clients.  For Rack applications not
heavily bound by slow external network dependencies, consider Unicorn
instead as it simpler and easier to debug.

* http://rainbows.rubyforge.org/
* rainbows-talk at rubyforge.org
* git://git.bogomips.org/rainbows.git

Changes:

Basic single-threaded EventMachine support is now included.  It
supports async_synatra[1] via the "async.callback" Rack
environment[2].  For EventMachine, we rely on the updated
attach/watch API in EventMachine 0.12.10.

As Revactor 0.1.5 is now available, our Revactor support now
depends on it as it adds the ability to listen on UNIX domain
sockets.

Of course, all dependencies (besides Unicorn and Rack) are
soft and only loaded if your configured concurrency model
requires it.

For developers/QA folks, the integration tests are completely
revamped for easier maintenance when new concurrency models are
introduced and should also produce TAP-compliant output.  The
test suite remains highly parallelizable using GNU make.

There are immediate plans to expand support for both Rev and
EventMachine to support use with threaded application dispatch.

Eric Wong (41):
      rev: remove Revactor-specific workaround
      README: change ordering of concurrency model listing
      tests: more correct HTTP/0.9 test
      test-lib: avoid stalling due to bad FIFO handling
      rev: fix static file responses under HTTP/0.9
      add news bodies to site NEWS.atom.xml
      tests: avoid needlessly remaking "rainbows"
      initial EventMachine support
      tests: hopefully fix stalls in input trailer tests
      tests: avoid race condition in reopen logs test
      tests: prefer "RUBY" to lowercased "ruby"
      tests: common setup and wait_start functions
      tests: add a TAP producer shell library
      tests: port all existing tests to TAP library
      tests: remove symlinks and small files, use Make
      t9000: bail if run with an unsupported/pointless model
      tests: allow "make $model" to run tests for that model
      rev: spell ECONNABORTED correctly
      rev/evma: move common code for event models into ev_core
      ev_core: do not drop deferred bodies on graceful quits
      eventmachine: get basic tests working
      rev: do not File.expand_path on result of body.to_path
      eventmachine 0.12.8 passes all tests
      tests: make large file memory tests more reliable
      eventmachine: require EM 0.12.10
      update gem dependencies in comments/local.mk.sample
      rev: enforce Rev::VERSION >= 0.3.0
      eventmachine: add async_sinatra support
      tests: only load Revactor tests under 1.9.1
      tests: gracefully exit if EventMachine is not available
      tests: error out if socat + curl aren't reachable
      thread*: fix MRI 1.8.6 compatibility
      local.mk.sample: cleanups and minor reorg
      eventmachine: remove unnecessary ivar assignment
      eventmachine: document our support of "async_synatra"
      doc: Update TODO and README
      tests: generate all dependencies atomically
      app_pool: update RDoc
      test-lib: DWIM handling of temp UNIX sockets
      revactor: require 0.1.5, remove 0.1.4 workarounds
      gemspec: bump up Unicorn dep version to 0.93.4

[1] http://github.com/raggi/async_sinatra
[2] this is not 100% Rack::Lint compatible, but we'll let it
    slide since there are already folks depending on
    the async_sinatra gem

-- 
Eric Wong

From normalperson at yhbt.net  Thu Oct 29 18:44:26 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 29 Oct 2009 15:44:26 -0700
Subject: Unicorn/Rainbows! mailing list migration
Message-ID: <20091029224426.GA12314@dcvr.yhbt.net>

As I'm sure you've all heard by now, RubyForge will be moving to a
read-only state and we'll have to migrate our mailing lists somewhere.

== librelist

I'm leaning strongly towards librelist.com.  I've had good, but limited
experiences with it.  I like that librelist has no logins or accounts to
deal with; just email the list you want to join and reply to the
automated message to subscribe.  It also has downloadable archives
via rsync.

librelist is run by Zed Shaw, the same guy who started Mongrel (which
eventually led us to Unicorn).  While I don't believe in giving any
single entity too much power and influence, Zed's done a lot of good for
Mongrel and handed it off gracefully when he didn't want to work on it
anymore.  For these reason, I think librelist will be in good hands for
years to come.

I probably would've picked librelist if it had been around when I
started this RubyForge list in May 2009.  I started Rainbows! more
recently, but continued using RubyForge because I wanted everything
Rainbows!-related on RubyForge.org (oops!).


== Other options

Google Groups is out.  Spam has been a big problem in the few groups
I've joined and I'd rather trust Zed than Google for mailing lists.
Also I don't know of a way to get archives from Google without scraping.

SourceForge - I've had spam problems in the past, mailman...

Savannah - no idea about the quality, but uses mailman...

I've also considered hosting the list ourselves, but that might be
too much work and spreading ourselves too thin at the moment.


Any other list hosting options out there we should consider?  It must be
a mailing list that does not mangle plain-text patches.

Or any votes in favor of migrating to librelist sooner rather than
later?  I'll work on getting the gmane NNTP mirrors moved over as
well.


== Other stuff related to RubyForge going read-only...

Gem hosting will (obviously) be taken care of by our new
Gemcut^WRubygems.org overlords.

The tarballs for non-RubyGems users will probably just be moved to a
files/ section of the websites; it'll be even be easier to find and
download tarballs in automated scripts that way, too.

Since we're using git, our repositories are mirrored everywhere already
and was never centrally-hosted on RubyForge in the first place.

http://rainbows.rubyforge.org/ will probably be redirected to
http://rainbows.bogomips.org/ (same box as the Unicorn website).

Should you ever want to access either of the sites offline, just make
sure you have RDoc 2.4.x ("gem install rdoc" if you're on 1.8) and run
`make doc' or `gmake doc' in the latest source tree.

-- 
Eric Wong

From normalperson at yhbt.net  Fri Oct 30 18:17:18 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 30 Oct 2009 15:17:18 -0700
Subject: Unicorn/Rainbows! mailing list migration
In-Reply-To: <20091029224426.GA12314@dcvr.yhbt.net>
References: <20091029224426.GA12314@dcvr.yhbt.net>
Message-ID: <20091030221718.GA18576@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> As I'm sure you've all heard by now, RubyForge will be moving to a
> read-only state and we'll have to migrate our mailing lists somewhere.

OK, it looks like we'll be able to keep our mailing lists on RubyForge.

http://tomcopeland.blogs.com/juniordeveloper/2009/10/things-to-keep-from-rubyforge.html

On the other hand, if anybody just *wants* to move to librelist, let us
know and we can consider it together, too.

Thanks.

-- 
Eric Wong