From bryant.doug at gmail.com  Mon Jun 11 10:06:09 2007
From: bryant.doug at gmail.com (Doug Bryant)
Date: Mon, 11 Jun 2007 10:06:09 -0400
Subject: [libxml-devel] Memory Leaks
Message-ID: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>

What's the status on the memory leaks?  They seem to be documented in the
issue tracker and mailing list and I experience them in my application as
well.  They seem to come from when you create & merge xml nodes &
documents.  I'm using 0.4.0 pre1.

I unfortunately don't have the time nor the C expertise to track down the
problems.

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/libxml-devel/attachments/20070611/15d32f4d/attachment.html 

From tom at compton.nu  Mon Jun 11 10:09:07 2007
From: tom at compton.nu (Tom Hughes)
Date: Mon, 11 Jun 2007 15:09:07 +0100
Subject: [libxml-devel] Memory Leaks
In-Reply-To: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>
	(Doug Bryant's message of "Mon\, 11 Jun 2007 10\:06\:09 -0400")
References: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>
Message-ID: <yeksl8yvb0c.fsf@dellow.uk.cyberscience.com>

In message <177c686f0706110706j2d9e386bja2521b76258b2e46 at mail.gmail.com>
        Doug Bryant <bryant.doug at gmail.com> wrote:

> What's the status on the memory leaks?  They seem to be documented in the
> issue tracker and mailing list and I experience them in my application as
> well.  They seem to come from when you create & merge xml nodes &
> documents.  I'm using 0.4.0 pre1.

There is a mail from me with patches attaches stuck in the moderation
queue for this list as it was too large to go through without help.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://www.compton.nu/

From doug at netinlet.com  Mon Jun 11 11:33:20 2007
From: doug at netinlet.com (Doug Bryant)
Date: Mon, 11 Jun 2007 11:33:20 -0400
Subject: [libxml-devel] Memory Leaks
In-Reply-To: <yeksl8yvb0c.fsf@dellow.uk.cyberscience.com>
References: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>
	<yeksl8yvb0c.fsf@dellow.uk.cyberscience.com>
Message-ID: <78cf1ade0706110833h3b92b75ct57b06d92bfbe44eb@mail.gmail.com>

thanks Tom.  I'll give it a shot & let you know.  It is consuming all the
memory on the production box every other day.  (All being all physical &
swap memory.)

Doug

On 6/11/07, Tom Hughes <tom at compton.nu> wrote:
>
> In message <177c686f0706110706j2d9e386bja2521b76258b2e46 at mail.gmail.com>
>         Doug Bryant <bryant.doug at gmail.com> wrote:
>
> > What's the status on the memory leaks?  They seem to be documented in
> the
> > issue tracker and mailing list and I experience them in my application
> as
> > well.  They seem to come from when you create & merge xml nodes &
> > documents.  I'm using 0.4.0 pre1.
>
> There is a mail from me with patches attaches stuck in the moderation
> queue for this list as it was too large to go through without help.
>
> Tom
>
> --
> Tom Hughes (tom at compton.nu)
> http://www.compton.nu/
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/libxml-devel/attachments/20070611/8adce3ff/attachment.html 

From bryant.doug at gmail.com  Mon Jun 11 15:05:05 2007
From: bryant.doug at gmail.com (Doug Bryant)
Date: Mon, 11 Jun 2007 15:05:05 -0400
Subject: [libxml-devel] Memory Leaks
In-Reply-To: <yeksl8yvb0c.fsf@dellow.uk.cyberscience.com>
References: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>
	<yeksl8yvb0c.fsf@dellow.uk.cyberscience.com>
Message-ID: <177c686f0706111205l64831a9fn754392dd276859dc@mail.gmail.com>

Tom,

Are the two patches referred to here (
http://rubyforge.org/pipermail/libxml-devel/2007-May/thread.html ) by Jon
Burgess* *the ones which should be applied?  Looking back through the
archive, there seems to be a  patch you are referring, but I can't seem to
track it down.

Doug

On 6/11/07, Tom Hughes <tom at compton.nu> wrote:
>
> In message <177c686f0706110706j2d9e386bja2521b76258b2e46 at mail.gmail.com>
>         Doug Bryant <bryant.doug at gmail.com> wrote:
>
> > What's the status on the memory leaks?  They seem to be documented in
> the
> > issue tracker and mailing list and I experience them in my application
> as
> > well.  They seem to come from when you create & merge xml nodes &
> > documents.  I'm using 0.4.0 pre1.
>
> There is a mail from me with patches attaches stuck in the moderation
> queue for this list as it was too large to go through without help.
>
> Tom
>
> --
> Tom Hughes (tom at compton.nu)
> http://www.compton.nu/
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/libxml-devel/attachments/20070611/2e74ad1c/attachment.html 

From tom at compton.nu  Mon Jun 11 15:07:04 2007
From: tom at compton.nu (Tom Hughes)
Date: Mon, 11 Jun 2007 20:07:04 +0100
Subject: [libxml-devel] Memory Leaks
In-Reply-To: <177c686f0706111205l64831a9fn754392dd276859dc@mail.gmail.com>
References: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>
	<yeksl8yvb0c.fsf@dellow.uk.cyberscience.com> 
	<177c686f0706111205l64831a9fn754392dd276859dc@mail.gmail.com>
Message-ID: <7be06af14e.tom@loxley.compton.nu>

In message <177c686f0706111205l64831a9fn754392dd276859dc at mail.gmail.com>
          "Doug Bryant" <bryant.doug at gmail.com> wrote:

> Are the two patches referred to here (
> http://rubyforge.org/pipermail/libxml-devel/2007-May/thread.html ) by Jon
> Burgess* *the ones which should be applied?  Looking back through the
> archive, there seems to be a  patch you are referring, but I can't seem to
> track it down.

That was the point - my patches never made it to the list because
they were held for moderation and they still haven't been approved.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://www.compton.nu/

From bryant.doug at gmail.com  Mon Jun 11 16:37:57 2007
From: bryant.doug at gmail.com (Doug Bryant)
Date: Mon, 11 Jun 2007 16:37:57 -0400
Subject: [libxml-devel] Memory Leaks
In-Reply-To: <7be06af14e.tom@loxley.compton.nu>
References: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>
	<yeksl8yvb0c.fsf@dellow.uk.cyberscience.com>
	<177c686f0706111205l64831a9fn754392dd276859dc@mail.gmail.com>
	<7be06af14e.tom@loxley.compton.nu>
Message-ID: <177c686f0706111337y28605acev686de5f0b53c7dfc@mail.gmail.com>

My brain interpreted it as the patch had not been applied to the trunk by a
committer (which is true), not that it never even made  it to the list in
the first place.

Does the list block all attachments or just if they are a certain size.
Perhaps you could gzip or zip the patches if the latter.  I had an issue
earlier where I tried to post to the list with the incorrect email address.
Could  something like that have happened?

Ross - you still monitoring the list?

Doug

On 6/11/07, Tom Hughes <tom at compton.nu> wrote:
>
> In message <177c686f0706111205l64831a9fn754392dd276859dc at mail.gmail.com>
>           "Doug Bryant" <bryant.doug at gmail.com> wrote:
>
> > Are the two patches referred to here (
> > http://rubyforge.org/pipermail/libxml-devel/2007-May/thread.html ) by
> Jon
> > Burgess* *the ones which should be applied?  Looking back through the
> > archive, there seems to be a  patch you are referring, but I can't seem
> to
> > track it down.
>
> That was the point - my patches never made it to the list because
> they were held for moderation and they still haven't been approved.
>
> Tom
>
> --
> Tom Hughes (tom at compton.nu)
> http://www.compton.nu/
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/libxml-devel/attachments/20070611/cf29b461/attachment.html 

From chris at octopod.info  Tue Jun 12 04:05:31 2007
From: chris at octopod.info (Chris McGrath)
Date: Tue, 12 Jun 2007 09:05:31 +0100
Subject: [libxml-devel] Memory Leaks
In-Reply-To: <7be06af14e.tom@loxley.compton.nu>
References: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>
	<yeksl8yvb0c.fsf@dellow.uk.cyberscience.com>
	<177c686f0706111205l64831a9fn754392dd276859dc@mail.gmail.com>
	<7be06af14e.tom@loxley.compton.nu>
Message-ID: <4B9E6897-2EF6-4E66-B8DD-F3412546B7B3@octopod.info>


On 11 Jun 2007, at 20:07, Tom Hughes wrote:

> That was the point - my patches never made it to the list because
> they were held for moderation and they still haven't been approved.

Stick them up on http://pastie.caboo.se and post the links here.

Cheers,

Chris

From tom at compton.nu  Tue Jun 12 05:04:06 2007
From: tom at compton.nu (Tom Hughes)
Date: Tue, 12 Jun 2007 10:04:06 +0100
Subject: [libxml-devel] Memory Leaks
In-Reply-To: <4B9E6897-2EF6-4E66-B8DD-F3412546B7B3@octopod.info> (Chris
	McGrath's message of "Tue\, 12 Jun 2007 09\:05\:31 +0100")
References: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>
	<yeksl8yvb0c.fsf@dellow.uk.cyberscience.com>
	<177c686f0706111205l64831a9fn754392dd276859dc@mail.gmail.com>
	<7be06af14e.tom@loxley.compton.nu>
	<4B9E6897-2EF6-4E66-B8DD-F3412546B7B3@octopod.info>
Message-ID: <yek8xapv915.fsf@dellow.uk.cyberscience.com>

In message <4B9E6897-2EF6-4E66-B8DD-F3412546B7B3 at octopod.info>
        Chris McGrath <chris at octopod.info> wrote:

> On 11 Jun 2007, at 20:07, Tom Hughes wrote:
>
>> That was the point - my patches never made it to the list because
>> they were held for moderation and they still haven't been approved.
>
> Stick them up on http://pastie.caboo.se and post the links here.

Right... First up, here's the simple one, which is a leak in
document.to_s when converting a document to a string where it
leaks a copy of the document text:

  http://trac.openstreetmap.org/attachment/ticket/482/libxml-leak2.patch

The one that caused real grief trying to fix it is this:

  http://trac.openstreetmap.org/attachment/ticket/482/libxml-leak1.patch

The basic problem is the failure to free nodes when copying them
into a document, but fixing that reveals lots of problems with
libxml nodes and attributes being used via ruby objects after libxml
has already freed them.

The problem is that when a node is attached to another node you wind
up with a libxml level C object that is owned both by another libxml C
object (which is itself owned by a ruby object) and by a ruby object.

If the parent C object goes away first then it frees the other object
that is still be used by a ruby object...

The patch therefore winds up doing a major rework of the way the
libxml wrapper is handling node and attribute objects.

Both those patches have been in use on www.openstreetmap.org for
several weeks now and have helped substantially with the problems
we were having.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://www.compton.nu/

From has.sox at gmail.com  Tue Jun 12 10:41:40 2007
From: has.sox at gmail.com (Daniel N)
Date: Wed, 13 Jun 2007 00:41:40 +1000
Subject: [libxml-devel] Memory Leaks
In-Reply-To: <yek8xapv915.fsf@dellow.uk.cyberscience.com>
References: <177c686f0706110706j2d9e386bja2521b76258b2e46@mail.gmail.com>
	<yeksl8yvb0c.fsf@dellow.uk.cyberscience.com>
	<177c686f0706111205l64831a9fn754392dd276859dc@mail.gmail.com>
	<7be06af14e.tom@loxley.compton.nu>
	<4B9E6897-2EF6-4E66-B8DD-F3412546B7B3@octopod.info>
	<yek8xapv915.fsf@dellow.uk.cyberscience.com>
Message-ID: <2fff50390706120741r50b5806fw41104a0d3f315c82@mail.gmail.com>

On 6/12/07, Tom Hughes <tom at compton.nu> wrote:
> In message <4B9E6897-2EF6-4E66-B8DD-F3412546B7B3 at octopod.info>
>         Chris McGrath <chris at octopod.info> wrote:
>
> > On 11 Jun 2007, at 20:07, Tom Hughes wrote:
> >
> >> That was the point - my patches never made it to the list because
> >> they were held for moderation and they still haven't been approved.
> >
> > Stick them up on http://pastie.caboo.se and post the links here.
>
> Right... First up, here's the simple one, which is a leak in
> document.to_s when converting a document to a string where it
> leaks a copy of the document text:
>
>   http://trac.openstreetmap.org/attachment/ticket/482/libxml-leak2.patch
>
> The one that caused real grief trying to fix it is this:
>
>   http://trac.openstreetmap.org/attachment/ticket/482/libxml-leak1.patch
>
> The basic problem is the failure to free nodes when copying them
> into a document, but fixing that reveals lots of problems with
> libxml nodes and attributes being used via ruby objects after libxml
> has already freed them.
>
> The problem is that when a node is attached to another node you wind
> up with a libxml level C object that is owned both by another libxml C
> object (which is itself owned by a ruby object) and by a ruby object.
>
> If the parent C object goes away first then it frees the other object
> that is still be used by a ruby object...
>
> The patch therefore winds up doing a major rework of the way the
> libxml wrapper is handling node and attribute objects.
>
> Both those patches have been in use on www.openstreetmap.org for
> several weeks now and have helped substantially with the problems
> we were having.
>
> Tom
>
> --
> Tom Hughes (tom at compton.nu)
> http://www.compton.nu/
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel
>

I tried to use libxml-ruby recently but suffered serious memory leaks.
 I was trying to replace elements / attributes, modify attribute
values and remove elements.  Not knowing any C to speak of I could not
fix these memory issues.

Would these patches have any chance of fixing these issues?

I would love to use libxml as an engine for a whitelister for arbitrary html.

Cheers
Daniel