From normalperson at yhbt.net  Thu Nov 12 02:53:21 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 11 Nov 2009 23:53:21 -0800
Subject: [Revactor-talk] TIA: Rainbows! code review please
Message-ID: <20091112075321.GA767@dcvr.yhbt.net>

I've been having some issues with the http://upr-demo.bogomips.org/
site running Rainbows! 0.5.0 + Revactor 0.1.5 + Ruby 1.9.1 p243 +
Rails 2.3.4 (no sessions or backend connections).

You can check out git://git.bogomips.org/rainbows.git and look in
lib/rainbows/revactor.rb and lib/rainbows/revactor/tee_input.rb

There were a couple of minor fixes to the Revactor-using code since the
0.5.0 release but I don't think they significantly affect what I'm
seeing here.  "git log -p v0.5.0.. lib/rainbows/revactor*" just to
be sure :>

It may be related to Actor scheduling during large uploads, but I'm not
sure.  It could also be something odd that libev is doing, too.  It
seems to stop accepting connections entirely but I also don't get any
"busy: " messages in the logs, either...  When I truss the process, it
still seems to be correctly calling the fchmod heartbeat every second,
which runs inside a "filter.after(1)" block, so I know the worker
process isn't blocked/dead.

It also doesn't help that I'm running this on an OS I'm unfamiliar with
for debugging (FreeBSD 7.2) and also with high network latency.  I'll
setup another demo running on a closer box running GNU/Linux, soon, too,
since I'm much more comfortable with GNU/Linux.

Any comments would be much appreciated, thanks again!  I'll also post my
findings if I find anything.

-- 
Eric Wong

From tony at medioh.com  Fri Nov 13 23:43:21 2009
From: tony at medioh.com (Tony Arcieri)
Date: Fri, 13 Nov 2009 21:43:21 -0700
Subject: [Revactor-talk] TIA: Rainbows! code review please
In-Reply-To: <20091112075321.GA767@dcvr.yhbt.net>
References: <20091112075321.GA767@dcvr.yhbt.net>
Message-ID: <c7e6b2b00911132043x7b79e249m423ee580238ed26b@mail.gmail.com>

Hi Eric,

I haven't tried Revactor on FreeBSD.  The only platforms I've tried it on
are Linux and OS X.  However, Rev (Revactor's I/O layer) is based on libev,
and libev's author is a stickler for providing identical semantics across
multiple operating systems, to the point of ignoring features which would
provide better performance if they can't provide identical semantics.

I'll say that Revactor was the basis of some rock-solid web-spidering code
we had deployed at my employer.  This spider was running 24/7 (albeit
periodically restarted after retrieving N URLs due to heap growth from
Ruby's GC) for many months without any observed bugs in its behaviour.

Can you reproduce this problem on Linux?  I see nothing obviously wrong in
your code.

Are you going to be at RubyConf?  If so I'd like to say hi.

On Thu, Nov 12, 2009 at 12:53 AM, Eric Wong <normalperson at yhbt.net> wrote:

> I've been having some issues with the http://upr-demo.bogomips.org/
> site running Rainbows! 0.5.0 + Revactor 0.1.5 + Ruby 1.9.1 p243 +
> Rails 2.3.4 (no sessions or backend connections).
>
> You can check out git://git.bogomips.org/rainbows.git and look in
> lib/rainbows/revactor.rb and lib/rainbows/revactor/tee_input.rb
>
> There were a couple of minor fixes to the Revactor-using code since the
> 0.5.0 release but I don't think they significantly affect what I'm
> seeing here.  "git log -p v0.5.0.. lib/rainbows/revactor*" just to
> be sure :>
>
> It may be related to Actor scheduling during large uploads, but I'm not
> sure.  It could also be something odd that libev is doing, too.  It
> seems to stop accepting connections entirely but I also don't get any
> "busy: " messages in the logs, either...  When I truss the process, it
> still seems to be correctly calling the fchmod heartbeat every second,
> which runs inside a "filter.after(1)" block, so I know the worker
> process isn't blocked/dead.
>
> It also doesn't help that I'm running this on an OS I'm unfamiliar with
> for debugging (FreeBSD 7.2) and also with high network latency.  I'll
> setup another demo running on a closer box running GNU/Linux, soon, too,
> since I'm much more comfortable with GNU/Linux.
>
> Any comments would be much appreciated, thanks again!  I'll also post my
> findings if I find anything.
>
> --
> Eric Wong
> _______________________________________________
> Revactor-talk mailing list
> Revactor-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/revactor-talk
>



-- 
Tony Arcieri
Medioh/Nagravision
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/revactor-talk/attachments/20091113/b9515cb2/attachment.html>

From normalperson at yhbt.net  Sat Nov 14 01:17:52 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Sat, 14 Nov 2009 06:17:52 +0000
Subject: [Revactor-talk] TIA: Rainbows! code review please
In-Reply-To: <c7e6b2b00911132043x7b79e249m423ee580238ed26b@mail.gmail.com>
References: <20091112075321.GA767@dcvr.yhbt.net>
	<c7e6b2b00911132043x7b79e249m423ee580238ed26b@mail.gmail.com>
Message-ID: <20091114061752.GA3573@dcvr.yhbt.net>

Tony Arcieri <tony at medioh.com> wrote:
> Hi Eric,
> 
> I haven't tried Revactor on FreeBSD.  The only platforms I've tried it on
> are Linux and OS X.  However, Rev (Revactor's I/O layer) is based on libev,
> and libev's author is a stickler for providing identical semantics across
> multiple operating systems, to the point of ignoring features which would
> provide better performance if they can't provide identical semantics.
> 
> I'll say that Revactor was the basis of some rock-solid web-spidering code
> we had deployed at my employer.  This spider was running 24/7 (albeit
> periodically restarted after retrieving N URLs due to heap growth from
> Ruby's GC) for many months without any observed bugs in its behaviour.

Good to know!  The other thing is that it's also running the behemoth
which is Rails.   I wonder if it's also something weird going on with
Rails and some thread-safety features being incompatible with Fibers
switching.

Holding a Mutex during I/O would ruin our day, but a quick look says
it's not happening.  Unfortunately I need the reentrancy/lack-of-globals
that Rails thread-safety provides.  But yeah, the TeeInput class means
Revactor is doing I/O inside the Rack/Rails application dispatch.
Lots of new ground there :>

> Can you reproduce this problem on Linux?  I see nothing obviously wrong in
> your code.

Not yet, that said I haven't tried hitting it very hard...

> Are you going to be at RubyConf?  If so I'd like to say hi.

Nope, I'm not in the area and can't travel at the moment.  Maybe next
year.

> On Thu, Nov 12, 2009 at 12:53 AM, Eric Wong <normalperson at yhbt.net> wrote:
> 
> > I've been having some issues with the http://upr-demo.bogomips.org/
> > site running Rainbows! 0.5.0 + Revactor 0.1.5 + Ruby 1.9.1 p243 +
> > Rails 2.3.4 (no sessions or backend connections).
> >
> > You can check out git://git.bogomips.org/rainbows.git and look in
> > lib/rainbows/revactor.rb and lib/rainbows/revactor/tee_input.rb
> >
> > There were a couple of minor fixes to the Revactor-using code since the
> > 0.5.0 release but I don't think they significantly affect what I'm
> > seeing here.  "git log -p v0.5.0.. lib/rainbows/revactor*" just to
> > be sure :>
> >
> > It may be related to Actor scheduling during large uploads, but I'm not
> > sure.  It could also be something odd that libev is doing, too.  It
> > seems to stop accepting connections entirely but I also don't get any
> > "busy: " messages in the logs, either...  When I truss the process, it
> > still seems to be correctly calling the fchmod heartbeat every second,
> > which runs inside a "filter.after(1)" block, so I know the worker
> > process isn't blocked/dead.
> >
> > It also doesn't help that I'm running this on an OS I'm unfamiliar with
> > for debugging (FreeBSD 7.2) and also with high network latency.  I'll
> > setup another demo running on a closer box running GNU/Linux, soon, too,
> > since I'm much more comfortable with GNU/Linux.
> >
> > Any comments would be much appreciated, thanks again!  I'll also post my
> > findings if I find anything.
> >
> > --
> > Eric Wong
> > _______________________________________________

From tony at medioh.com  Sat Nov 14 01:23:17 2009
From: tony at medioh.com (Tony Arcieri)
Date: Fri, 13 Nov 2009 23:23:17 -0700
Subject: [Revactor-talk] TIA: Rainbows! code review please
In-Reply-To: <20091114061752.GA3573@dcvr.yhbt.net>
References: <20091112075321.GA767@dcvr.yhbt.net>
	<c7e6b2b00911132043x7b79e249m423ee580238ed26b@mail.gmail.com>
	<20091114061752.GA3573@dcvr.yhbt.net>
Message-ID: <c7e6b2b00911132223g133b5888jf208969f0f35c6e9@mail.gmail.com>

On Fri, Nov 13, 2009 at 11:17 PM, Eric Wong <normalperson at yhbt.net> wrote:

> Good to know!  The other thing is that it's also running the behemoth
> which is Rails.   I wonder if it's also something weird going on with
> Rails and some thread-safety features being incompatible with Fibers
> switching.
>

I never actually provided full support for multithreaded environments in
Revactor.  It was something I was working towards but never completed.  In
Ruby 1.9 every thread needs its own Rev event loop, and to communicate
across threads Actors need a mechanism for waking up another thread when
it's blocking on I/O.


> Holding a Mutex during I/O would ruin our day, but a quick look says
> it's not happening.  Unfortunately I need the reentrancy/lack-of-globals
> that Rails thread-safety provides.  But yeah, the TeeInput class means
> Revactor is doing I/O inside the Rack/Rails application dispatch.
> Lots of new ground there :>
>

Rev's AsyncWatchers were to form the groundwork of interthread
communication.  libev provides its own async watchers, but I had problems
utilizing them.  I wrote my own using Rev's IO watchers and pipes.  These
allowed one thread to send another a "wake up" event over a pipe.  However
this work was only partially done and I have no tests for true multithreaded
environments like Rack/Rails.

-- 
Tony Arcieri
Medioh/Nagravision
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/revactor-talk/attachments/20091113/87afca3f/attachment.html>

From normalperson at yhbt.net  Sat Nov 14 02:00:21 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Sat, 14 Nov 2009 07:00:21 +0000
Subject: [Revactor-talk] TIA: Rainbows! code review please
In-Reply-To: <c7e6b2b00911132223g133b5888jf208969f0f35c6e9@mail.gmail.com>
References: <20091112075321.GA767@dcvr.yhbt.net>
	<c7e6b2b00911132043x7b79e249m423ee580238ed26b@mail.gmail.com>
	<20091114061752.GA3573@dcvr.yhbt.net>
	<c7e6b2b00911132223g133b5888jf208969f0f35c6e9@mail.gmail.com>
Message-ID: <20091114070020.GB3573@dcvr.yhbt.net>

Tony Arcieri <tony at medioh.com> wrote:
> On Fri, Nov 13, 2009 at 11:17 PM, Eric Wong <normalperson at yhbt.net> wrote:
> 
> > Good to know!  The other thing is that it's also running the behemoth
> > which is Rails.   I wonder if it's also something weird going on with
> > Rails and some thread-safety features being incompatible with Fibers
> > switching.
> 
> I never actually provided full support for multithreaded environments in
> Revactor.  It was something I was working towards but never completed.  In
> Ruby 1.9 every thread needs its own Rev event loop, and to communicate
> across threads Actors need a mechanism for waking up another thread when
> it's blocking on I/O.

Actually, it's not multithreaded, it's just that the config.threadsafe!
option for Rails needed to be enabled because otherwise the Fibers could
get stuck behind the Rack::Lock middleware.

> > Holding a Mutex during I/O would ruin our day, but a quick look says
> > it's not happening.  Unfortunately I need the reentrancy/lack-of-globals
> > that Rails thread-safety provides.  But yeah, the TeeInput class means
> > Revactor is doing I/O inside the Rack/Rails application dispatch.
> > Lots of new ground there :>
> >
> 
> Rev's AsyncWatchers were to form the groundwork of interthread
> communication.  libev provides its own async watchers, but I had problems
> utilizing them.  I wrote my own using Rev's IO watchers and pipes.  These
> allowed one thread to send another a "wake up" event over a pipe.  However
> this work was only partially done and I have no tests for true multithreaded
> environments like Rack/Rails.

Yup, I've actually been playing around with Rev + Threads in Rainbows!
lately as well and found AsyncWatchers extremely useful there.  The
current rainbows.git RevThread* stuff still needs work, but I hope
to have time next week.

-- 
Eric Wong

From normalperson at yhbt.net  Sun Nov 15 20:45:50 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 16 Nov 2009 01:45:50 +0000
Subject: [Revactor-talk] TIA: Rainbows! code review please
In-Reply-To: <20091112075321.GA767@dcvr.yhbt.net>
References: <20091112075321.GA767@dcvr.yhbt.net>
Message-ID: <20091116014550.GA3301@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> It seems to stop accepting connections entirely

Durr...  I HUP-ed for a reload when there was only a single worker
process (I was lazy and didn't want to setup drb/memcached).  So when
the HUP signals a graceful reload, it stops the workers from accepting
so they can exit gracefully after all clients have disconnected.

Gah, this is going to be tricky to solve in Rainbows! :<

-- 
Eric Wong

From normalperson at yhbt.net  Mon Nov 16 02:29:51 2009
From: normalperson at yhbt.net (Eric Wong)
Date: Sun, 15 Nov 2009 23:29:51 -0800
Subject: [Revactor-talk] [pull request] UNIX socket cleanups and optional
	:timeout
Message-ID: <20091116072951.GA1081@dcvr.yhbt.net>

Since :timeout got into 0.1.5 for TCP sockets, I've also created these
as well, the first one is just a trivial cleanup and the other one just
makes UNIX sockets duck-type compatible with the TCP ones as far as the
#read/#write calls go.

I plan on using these to implement keepalive_timeout in Rainbows! since
it looks like too many clients like doing excessive HTTP keepalive :<

Pushed out to git://yhbt.net/revactor as usual.

commit e2628eaab2312251845bff63e0e64f11150087ca
Author: Eric Wong <normalperson at yhbt.net>
Date:   Sun Nov 15 23:03:05 2009 -0800

    UNIX: remove unused CONNECT_TIMEOUT constant
    
    Blind copy and paste error from the TCP code on my
    part, it's not possible to timeout path resolution
    on a UNIX filesystem.

commit 2c7d5a4a5f0a23485e53cfdd6267e456a7381bc5
Author: Eric Wong <normalperson at yhbt.net>
Date:   Sun Nov 15 23:09:53 2009 -0800

    UNIX: allow optional :timeout arguments in a hash
    
    This is mostly to be duck-type compatible with the
    Revactor::TCP::Socket :timeout feature.  This benefits servers
    that serve UNIX and TCP sockets and want to share as much code
    as possible.  Maybe occasionally it will be useful for
    untrusted/misbehaving clients on the local machine, too.

-- 
Eric Wong

From tony at medioh.com  Mon Nov 23 13:34:52 2009
From: tony at medioh.com (Tony Arcieri)
Date: Mon, 23 Nov 2009 11:34:52 -0700
Subject: [Revactor-talk] [pull request] UNIX socket cleanups and
	optional :timeout
In-Reply-To: <20091116072951.GA1081@dcvr.yhbt.net>
References: <20091116072951.GA1081@dcvr.yhbt.net>
Message-ID: <c7e6b2b00911231034k7fed2851xe758dd6f20c5f5fb@mail.gmail.com>

Hi,

Sorry about not getting to this... been preoccupied with RubyConf :)

I can give you commit access to my branch if you want.  Just send me your
pubkey.  That will stop me from MITMing :)

On Mon, Nov 16, 2009 at 12:29 AM, Eric Wong <normalperson at yhbt.net> wrote:

> Since :timeout got into 0.1.5 for TCP sockets, I've also created these
> as well, the first one is just a trivial cleanup and the other one just
> makes UNIX sockets duck-type compatible with the TCP ones as far as the
> #read/#write calls go.
>
> I plan on using these to implement keepalive_timeout in Rainbows! since
> it looks like too many clients like doing excessive HTTP keepalive :<
>
> Pushed out to git://yhbt.net/revactor as usual.
>
> commit e2628eaab2312251845bff63e0e64f11150087ca
> Author: Eric Wong <normalperson at yhbt.net>
> Date:   Sun Nov 15 23:03:05 2009 -0800
>
>    UNIX: remove unused CONNECT_TIMEOUT constant
>
>    Blind copy and paste error from the TCP code on my
>    part, it's not possible to timeout path resolution
>    on a UNIX filesystem.
>
> commit 2c7d5a4a5f0a23485e53cfdd6267e456a7381bc5
> Author: Eric Wong <normalperson at yhbt.net>
> Date:   Sun Nov 15 23:09:53 2009 -0800
>
>    UNIX: allow optional :timeout arguments in a hash
>
>    This is mostly to be duck-type compatible with the
>    Revactor::TCP::Socket :timeout feature.  This benefits servers
>    that serve UNIX and TCP sockets and want to share as much code
>    as possible.  Maybe occasionally it will be useful for
>    untrusted/misbehaving clients on the local machine, too.
>
> --
> Eric Wong
> _______________________________________________
> Revactor-talk mailing list
> Revactor-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/revactor-talk
>



-- 
Tony Arcieri
Medioh/Nagravision
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/revactor-talk/attachments/20091123/c21a18d4/attachment.html>