[Mongrel] mongrel garbage collection

[Luis Lavena]

On Mon, Mar 24, 2008 at 4:59 PM, Scott Windsor <swindsor at gmail.com> wrote:
> On Mon, Mar 24, 2008 at 12:18 PM, Luis Lavena <luislavena at gmail.com> wrote:
>
> >
> >
> > On Mon, Mar 24, 2008 at 3:58 PM, Scott Windsor <swindsor at gmail.com> wrote:
> >
> >
> >
> > You're using *RMagick*, not ImageMagick directly. If you used the
> > later (via system calls) there will no be memory leakage you can worry
> > about.
>
> You're correct - I'm using 'RMagick' - and it uses a large amount of memory.
> But that's not really the overall point.  My overall point is how to
> properly handle a rails app that uses a great deal of memory during each
> request.  I'm pretty sure this happens in other rails applications that
> don't happen to use 'RMagick'.
>

Yes, I faced huge memory usage issues with other things non related to
image processing and found that a good thing was move them out of the
request-response cycle and into a out-of-bound background job.

>
> So far, running the GC under fastcgi has given me pretty good results.  The
> zombing issue with fast cgi is a known issue with mod_fastcgi and I'm pretty
> sure unrelated to RMagick or garbage collection.
>

Yes, but even you "reclaim" the memory with GC, there will be pieces
that wouldn't be GC'ed ever, since the leaked in the C side, outside
GC control (some of the RMagick and ImageMagick mysteries).

>
> > Can you tell me how you addressed the "schedule" of the garbage
> > collection execution on your previous scenario? AFAIK most of the
> > frameworks or servers don't impose to the user how often GC should be
> > performed.
> >
> >
>
> In the previous scenario I was using fast_cgi with rails.  In my previous
> reply I provided a link to the rails fastcgi dispatcher.
>
> http://dev.rubyonrails.org/browser/trunk/railties/dispatches/dispatch.fcgi
>
> In addtion, in other languages and other language web frameworks there are
> provisions to control garbage collection (for languages that have garbage
> collections, of course).
>
>
> > I'll bet is rails specific, or you should take a look at the fcgi ruby
> > extension, since it is responsible, ruby-side, of bridging both
> > worlds.
> >
>
> This is done in the Rails FastCGI dispatcher.  I believe that the equivalent
> of this in Mongrel is the Mongrel Rails dispatcher.  Since the Mongrel Rails
> dispatcher is distributed as a part of Mongrel, I'd say this code is owned
> by Mongrel, which bridges these two worlds when using mongrel as a
> webserver.
>

Then you could provide a different Mongrel Handler that could perform
that, or even a series of GemPlugins that provide a gc:start instead
of plain 'start' command mongrel_rails scripts provides.

> >
> > On a personal note, I believe is not responsibility of Mongrel, as a
> > webserver, take care of the garbage collection and leakage issues of
> > the Vm on which your application runs. In any case, the GC of the VM
> > (MRI Ruby) should be enhanced to work better with heavy load and long
> > running environments.
> >
>
> Ruby provides an API to access and call the Garbage Collector.  This
> provides ruby application developers the ability to control when the garbage
> collection is run because in some cases, there may be an
> application-specific reason to prevent or explicity run the GC.  Web servers
> are a good example of these applications where state may help determine a
> better time to run the GC.  As you're serving each request, you're generally
> allocating a number of objects, then rendering output, then moving on to the
> next request.
>
> By limiting the GC to run in between requests rather than during requests
> you are trading request time for latency between requests.  This is a
> trade-off that I think web application developers should deciede, but by no
> means should this be a default or silver bullet for all.  My position is
> that this just be an option within Mongrel as a web server.
>

--gc-interval maybe?

Now that you convinced me and proved your point, having the option to
perform it (optionally, not forced) will be something good to have.

Patches are Welcome ;-)

-- 
Luis Lavena
Multimedia systems
-
Human beings, who are almost unique in having the ability to learn from
the experience of others, are also remarkable for their apparent
disinclination to do so.
Douglas Adams