From jasoninclass at googlemail.com  Mon Feb  1 02:00:31 2010
From: jasoninclass at googlemail.com (jason franklin-stokes)
Date: Mon, 1 Feb 2010 08:00:31 +0100
Subject: [Celerity-users] locating all text nodes in a page/frame
In-Reply-To: <3a8841391001311411i4221347fmef08aeeda88ae096@mail.gmail.com>
References: <F1F57AA7-B0F6-4608-B519-6D75CE090A51@googlemail.com>
	<4B656E69.2070503@gmail.com>
	<2D8439B2-4A63-4D75-AC9F-0D99CAF8F791@googlemail.com>
	<eae5f6f91001310530t28663b7bvb43de034ff789220@mail.gmail.com>
	<2D51A57E-BFB7-47F8-8118-E17BC7DF09E1@googlemail.com>
	<3a8841391001311411i4221347fmef08aeeda88ae096@mail.gmail.com>
Message-ID: <A4F9BF03-A726-49D2-B895-03919CCA854D@googlemail.com>


thanks for the feedback. 

As Jari pointed out, what i am looking for is not just the text representation for the page frame - but the text nodes - dom objects that i can reflect on so that i can find out for example what objects are encapsulating the text nodes, find lists and group objects by their text by xpath and parent node (tag) etc - so unfortunately the browser.text function in celerity does not help me in this case. 

but thanks anyway. 

jason.


On Jan 31, 2010, at 11:11 PM, doridori Jo wrote:

> is it possible to do
> 
> browser.iframe.text
> 
> On Sun, Jan 31, 2010 at 6:07 AM, jason franklin-stokes <jasoninclass at googlemail.com> wrote:
> Hi Jari,
> 
> Thanks and point taken - i guess iteration is needed one way or the the other.
> 
> I think that the thought behind the thought is that what I have done in my helper class already exists somewhere in celerity or htmlunit and i might be repeating code where i dont need to.
> 
> I will ask the guys at htmlunit and see if they have any ideas.
> 
> thanks for the feedback.....
> 
> jason.
> 
> On Jan 31, 2010, at 2:30 PM, Jari Bakken wrote:
> 
> > On Sun, Jan 31, 2010 at 2:15 PM, jason franklin-stokes
> > <jasoninclass at googlemail.com> wrote:
> >
> >>
> >> however this is also not very efficient as it touches every node in the dom tree.
> >>
> >> what i am really looking for is a way to collect just the text leafs in the dom tree with something like
> >>
> >> browser.text_nodes
> >>
> >> and ideas???
> >>
> >
> > There's nothing like this in Celerity's API. I'm curious how you think
> > this could be implemented without touching every node in the DOM tree.
> > If I needed this, I would probably start by looking at how HtmlUnit
> > implements DomNode.getTextContent().
> > _______________________________________________
> > Celerity-users mailing list
> > Celerity-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/celerity-users
> 
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users
> 
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/celerity-users/attachments/20100201/f00b01dd/attachment.html>

From peter at hexagile.com  Wed Feb  3 12:43:41 2010
From: peter at hexagile.com (Peter Szinek)
Date: Wed, 3 Feb 2010 18:43:41 +0100
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
	space" after crawling a lot
Message-ID: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>

Hi guys,

I have a quite simple celerity script which tries to collect links  
from a job portal:

https://gist.github.com/31cb77955c8240d37d2b

After around the 110th page I am getting

java.lang.OutOfMemoryError: Java heap space

using the standard -J-Xmx512m - I am not sure setting it to more is  
the way to go, as opposed to finding out why the memleak is happening  
in the first place.

I am quite new to both jRuby and Celerity (so I am stumbling in the  
dark), but not to Ruby/Java so I tried a few things - especially  
making sure I am not holding onto any variable and allowing the GC to  
do it's thing - but nothing helped.

So from this point I am just wildly guessing. My first wild guess is  
that this might be because celerity's history is growing unbounded (I  
have no clue how is it solved in celerity, but there has to be some  
kind of data structure that holds the browsing history).

I don't need the history at all, ever, so I am ok with dropping it in  
each iteration if that would help (couldn't google out how to do that  
- any pointers?)

Any other ideas? Maybe I am missing the obvious and it's something  
trivial? How can I find out what's going on here?

Cheers,
Peter



From jari.bakken at gmail.com  Wed Feb  3 13:17:35 2010
From: jari.bakken at gmail.com (Jari Bakken)
Date: Wed, 3 Feb 2010 19:17:35 +0100
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
	space" after crawling a lot
In-Reply-To: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
Message-ID: <eae5f6f91002031017j1555f6b7mcc77eab5f33ce437@mail.gmail.com>

On Wed, Feb 3, 2010 at 6:43 PM, Peter Szinek <peter at hexagile.com> wrote:
> https://gist.github.com/31cb77955c8240d37d2b
>
> After around the 110th page I am getting
>
> java.lang.OutOfMemoryError: Java heap space
>

What version of Celerity, JRuby and Java do you have?
I let your script run for a while and couldn't reproduce the problem -
visited >1000 pages and heap memory stayed ~150Mb. I'm on OS X 10.6,
Java 6 and Celerity 0.7.8.

> I don't need the history at all, ever, so I am ok with dropping it in each
> iteration if that would help (couldn't google out how to do that - any
> pointers?)
>

Celerity doesn't store any history, but wraps HtmlUnit - they might do
it (I don't know).
Since you're familiar with Java, I would try rewriting the script in
Java using the HtmlUnit API directly and see if you still have the
problem (might take a while longer before it occurs since JRuby's
objects are heavier). If you can't reproduce, it's likely to be an
issue in Celerity. If you can, you should bring it to the HtmlUnit
list/tracker.


> Any other ideas? Maybe I am missing the obvious and it's something trivial?
> How can I find out what's going on here?
>

If you can, disable JavaScript execution by passing this option:

  Browser.new(:javascript_enabled => false))

That should give you a lower memory footprint (and a nice speed bump!)

From caius at brightbox.co.uk  Wed Feb  3 13:18:41 2010
From: caius at brightbox.co.uk (Caius Durling)
Date: Wed, 3 Feb 2010 18:18:41 +0000
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
	space" after crawling a lot
In-Reply-To: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
Message-ID: <83E92ACC-CE90-4F00-A01A-1923CA7F1302@brightbox.co.uk>

On 3 Feb 2010, at 17:43, Peter Szinek wrote:
> My first wild guess is that this might be because celerity's history is growing unbounded (I have no clue how is it solved in celerity, but there has to be some kind of data structure that holds the browsing history).
> 
> I don't need the history at all, ever, so I am ok with dropping it in each iteration if that would help (couldn't google out how to do that - any pointers?)


I'm not sure about how it fits into your problem, but you got me curious about the history as it doesn't appear to be exposed directly in celerity.

Turns out you can get to it, but only by delving into the Celerity::Browser instance. From there you can then drill down and talk directly to HTMLUnit to get the history as an array for each "webWindow".

> irb(main):001:0> b = Celerity::Browser.new
> => #<Celerity::Browser:0x17510d96 @page=nil @render_type=:html @charset="UTF-8" @viewer=Celerity::DefaultViewer @error_checkers=[]>
> irb(main):029:0> b.webclient.webWindows.map {|w| w.history.to_s }  
> => ["[]"]
> irb(main):030:0> b.goto "http://google.com/"
> => "http://www.google.co.uk/"
> irb(main):031:0> b.goto "http://google.co.uk/"
> => "http://www.google.co.uk/"
> irb(main):032:0> b.webclient.webWindows.map {|w| w.history.to_s }
> => ["[http://www.google.co.uk/, http://www.google.co.uk/]", "[about:blank]"]


http://htmlunit.sourceforge.net/apidocs/com/gargoylesoftware/htmlunit/History.html

Can't see any way to reset that though, seems like the window class would need setHistory to assign it a new instance of History, or History class would need an 'empty' or 'reset' method adding.

Hope that helps -somehow-,
C
---
Caius Durling
Brightbox
caius at brightbox.co.uk
http://brightbox.co.uk/


From jari.bakken at gmail.com  Wed Feb  3 13:37:38 2010
From: jari.bakken at gmail.com (Jari Bakken)
Date: Wed, 3 Feb 2010 19:37:38 +0100
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
	space" after crawling a lot
In-Reply-To: <83E92ACC-CE90-4F00-A01A-1923CA7F1302@brightbox.co.uk>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
	<83E92ACC-CE90-4F00-A01A-1923CA7F1302@brightbox.co.uk>
Message-ID: <eae5f6f91002031037pedb3065x635f1c8b07716c99@mail.gmail.com>

On Wed, Feb 3, 2010 at 7:18 PM, Caius Durling <caius at brightbox.co.uk> wrote:
>
> Can't see any way to reset that though, seems like the window class would need setHistory to assign it a new instance of History, or History class would need an 'empty' or 'reset' method adding.
>

Nice find. A workaround for this would be to close and create a new
browser for, say, every 100th page:

if i % 100 == 0
  @browser.close
  @browser = Browser.new(...)
end

From peter at hexagile.com  Wed Feb  3 13:54:52 2010
From: peter at hexagile.com (Peter Szinek)
Date: Wed, 3 Feb 2010 19:54:52 +0100
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
	space" after crawling a lot
In-Reply-To: <eae5f6f91002031017j1555f6b7mcc77eab5f33ce437@mail.gmail.com>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
	<eae5f6f91002031017j1555f6b7mcc77eab5f33ce437@mail.gmail.com>
Message-ID: <AF604DE3-4F1B-4DC7-9EC6-FC2F67823C68@hexagile.com>

Thanks for the quick reply - the celerity list is one of the  
friendliest MLs I am subscribed to!

> What version of Celerity, JRuby and Java do you have?
> I let your script run for a while and couldn't reproduce the problem -
> visited >1000 pages and heap memory stayed ~150Mb. I'm on OS X 10.6,
> Java 6 and Celerity 0.7.8.

OSX 10.5.8
Java 1.5.0_20
JRuby 1.3.1
jarib-celerity (0.0.7.1)

maybe it's time to upgrade a few things?

> If you can, disable JavaScript execution by passing this option:
>
>  Browser.new(:javascript_enabled => false))
>
> That should give you a lower memory footprint (and a nice speed bump!)

Holy crap, this is like 7857564*57E10 faster!!!1!!one! I  
think :javascript_enabled => false should be the default...
This setting fixed ll my problems immediately...

Cheers,
Peter 
  

From peter at hexagile.com  Wed Feb  3 14:00:11 2010
From: peter at hexagile.com (Peter Szinek)
Date: Wed, 3 Feb 2010 20:00:11 +0100
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
	space" after crawling a lot
In-Reply-To: <83E92ACC-CE90-4F00-A01A-1923CA7F1302@brightbox.co.uk>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
	<83E92ACC-CE90-4F00-A01A-1923CA7F1302@brightbox.co.uk>
Message-ID: <FCFE3D98-5E69-4CB7-A0E6-DCDCE6FADCE5@hexagile.com>

Thanks Caius! Fortunately Jari's :javascript_enabled => false solved  
my problem, but being able to access the history will definitely come  
handy!

Cheers,
Peter

>> irb(main):001:0> b = Celerity::Browser.new
>> => #<Celerity::Browser:0x17510d96 @page=nil @render_type=:html  
>> @charset="UTF-8" @viewer=Celerity::DefaultViewer @error_checkers=[]>
>> irb(main):029:0> b.webclient.webWindows.map {|w| w.history.to_s }
>> => ["[]"]
>> irb(main):030:0> b.goto "http://google.com/"
>> => "http://www.google.co.uk/"
>> irb(main):031:0> b.goto "http://google.co.uk/"
>> => "http://www.google.co.uk/"
>> irb(main):032:0> b.webclient.webWindows.map {|w| w.history.to_s }
>> => ["[http://www.google.co.uk/, http://www.google.co.uk/]",  
>> "[about:blank]"]
>
>
> http://htmlunit.sourceforge.net/apidocs/com/gargoylesoftware/htmlunit/History.html
>
> Can't see any way to reset that though, seems like the window class  
> would need setHistory to assign it a new instance of History, or  
> History class would need an 'empty' or 'reset' method adding.
>
> Hope that helps -somehow-,
> C
> ---
> Caius Durling
> Brightbox
> caius at brightbox.co.uk
> http://brightbox.co.uk/
>
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users


From phillipkoebbe at gmail.com  Wed Feb  3 15:59:35 2010
From: phillipkoebbe at gmail.com (Phillip Koebbe)
Date: Wed, 03 Feb 2010 14:59:35 -0600
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
 space" after crawling a lot
In-Reply-To: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
Message-ID: <4B69E3B7.2060500@gmail.com>

I ran into the same error the other day after I started using Scenario 
Outlines in Cucumber.

http://gist.github.com/294011

I worked around it by restarting the culerity server for each feature 
file and moving the outlines into a separate file (they were originally 
in with a number of other scenarios). Earlier today I happened to have 
Activity Monitor open while running all of my tests, and I noticed that 
the Java VM climbed to over 400Mb of memory when running these outlines. 
Not sure why it needs that much, though.

I have JavaScript support on by default, but I think I will look into 
ways to enable it only when I need it.

My environment is:

OS X 10.6.2

java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04-248-10M3025)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01-101, mixed mode)

jruby 1.4.0 (ruby 1.8.7 patchlevel 174) (2009-11-02 69fbfa3) (Java 
HotSpot(TM) 64-Bit Server VM 1.6.0_17) [x86_64-java]

celerity (0.7.6)

Peace,
Phillip



From peter at hexagile.com  Wed Feb  3 16:35:10 2010
From: peter at hexagile.com (Peter Szinek)
Date: Wed, 3 Feb 2010 22:35:10 +0100
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
	space" after crawling a lot
In-Reply-To: <4B69E3B7.2060500@gmail.com>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
	<4B69E3B7.2060500@gmail.com>
Message-ID: <1B79226E-426C-4AE1-B94E-E398B47E5010@hexagile.com>

Actually I realized that I can't switch off JavaScript... so I had to  
find a different solution which proved to be much easier than I hoped.  
It was enough to switch from

Java 1.5
JRuby 1.3.1
celerity 0.0.7.1.

to

java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-226)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-92, mixed mode)

jruby 1.4.0 (ruby 1.8.7 patchlevel 174) (2009-11-02 69fbfa3) (Java  
HotSpot(TM) 64-Bit Server VM 1.6.0_15) [x86_64-java]

Celerity 0.7.8

and instead of throwing an exception on about every 30th page, I did  
not get a *single* exception after 1058 pages, without actually  
touching the code itself.

The biggest difference compared to Philip's setup is that I am on  
Leopard...

Peter

On 2010.02.03., at 21:59, Phillip Koebbe wrote:

> I ran into the same error the other day after I started using  
> Scenario Outlines in Cucumber.
>
> http://gist.github.com/294011
>
> I worked around it by restarting the culerity server for each  
> feature file and moving the outlines into a separate file (they were  
> originally in with a number of other scenarios). Earlier today I  
> happened to have Activity Monitor open while running all of my  
> tests, and I noticed that the Java VM climbed to over 400Mb of  
> memory when running these outlines. Not sure why it needs that much,  
> though.
>
> I have JavaScript support on by default, but I think I will look  
> into ways to enable it only when I need it.
>
> My environment is:
>
> OS X 10.6.2
>
> java version "1.6.0_17"
> Java(TM) SE Runtime Environment (build 1.6.0_17-b04-248-10M3025)
> Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01-101, mixed mode)
>
> jruby 1.4.0 (ruby 1.8.7 patchlevel 174) (2009-11-02 69fbfa3) (Java  
> HotSpot(TM) 64-Bit Server VM 1.6.0_17) [x86_64-java]
>
> celerity (0.7.6)
>
> Peace,
> Phillip
>
>
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users


From phillipkoebbe at gmail.com  Thu Feb  4 17:30:09 2010
From: phillipkoebbe at gmail.com (Phillip Koebbe)
Date: Thu, 04 Feb 2010 16:30:09 -0600
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
 space" after crawling a lot
In-Reply-To: <4B6AFF7F.9030508@gmail.com>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
	<4B69E3B7.2060500@gmail.com> <4B6AFF7F.9030508@gmail.com>
Message-ID: <4B6B4A71.3000402@gmail.com>

I'll answer my own questions with Jari's advice:

turn off javascript support when possible.

I just ran those same outlines without JS support and about 40% of the 
memory was used and the outline finished in about a 1/3 of the time.

With Javascript 500Mb in 50 seconds
Without Javascript 200Mb in 17 seconds

I'm currently experimenting with using before hooks and tags to identify 
scenarios that require javascript support with the goal of creating a 
new browser when I switch.

Peace,
Phillip

Phillip Koebbe wrote:
> I just updated celerity to 0.7.8 and ran the scenario outlines I 
> shared in the gist and the java process grew to 448 Mb of memory. Any 
> ideas as to why and what can be done to limit it?
>
> Thanks,
> Phillip
>

From markus.kohler at gmail.com  Fri Feb  5 03:42:13 2010
From: markus.kohler at gmail.com (Markus Kohler)
Date: Fri, 5 Feb 2010 09:42:13 +0100
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
	space" after crawling a lot
In-Reply-To: <4B6B4A71.3000402@gmail.com>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
	<4B69E3B7.2060500@gmail.com> <4B6AFF7F.9030508@gmail.com>
	<4B6B4A71.3000402@gmail.com>
Message-ID: <771905291002050042l1491cb32g68d57ac958e85aee@mail.gmail.com>

Hi all,

If someone has a simple test,I would be willing to take a heap dump and
check what is going on using the
Eclipse Memory Analyzer ( http://eclipse.org/mat/).

BTW, I think tools like celerity are the future of testing web
applications.
I hope a can completly stay away from Loadrunner-like tools in the near
future ;-)


Regards,
Markus (http://kohlerm.blogspot.com/)
"The best way to predict the future is to invent it" -- Alan Kay


On Thu, Feb 4, 2010 at 11:30 PM, Phillip Koebbe <phillipkoebbe at gmail.com>wrote:

> I'll answer my own questions with Jari's advice:
>
> turn off javascript support when possible.
>
> I just ran those same outlines without JS support and about 40% of the
> memory was used and the outline finished in about a 1/3 of the time.
>
> With Javascript 500Mb in 50 seconds
> Without Javascript 200Mb in 17 seconds
>
> I'm currently experimenting with using before hooks and tags to identify
> scenarios that require javascript support with the goal of creating a new
> browser when I switch.
>
> Peace,
> Phillip
>
> Phillip Koebbe wrote:
>
>> I just updated celerity to 0.7.8 and ran the scenario outlines I shared in
>> the gist and the java process grew to 448 Mb of memory. Any ideas as to why
>> and what can be done to limit it?
>>
>> Thanks,
>> Phillip
>>
>>  _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/celerity-users/attachments/20100205/3932858a/attachment.html>

From phillipkoebbe at gmail.com  Fri Feb  5 11:41:46 2010
From: phillipkoebbe at gmail.com (Phillip Koebbe)
Date: Fri, 05 Feb 2010 10:41:46 -0600
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
 space" after crawling a lot
In-Reply-To: <771905291002050042l1491cb32g68d57ac958e85aee@mail.gmail.com>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>	<4B69E3B7.2060500@gmail.com>
	<4B6AFF7F.9030508@gmail.com>	<4B6B4A71.3000402@gmail.com>
	<771905291002050042l1491cb32g68d57ac958e85aee@mail.gmail.com>
Message-ID: <4B6C4A4A.3040701@gmail.com>

That's very generous of you, Markus. Thanks. I would create a simple 
project, but won't have time for a while. I spent the whole day 
yesterday working out enabling/disabling JavaScript automatically 
through Culerity and didn't get any paid work done. I can't afford to 
take more time right now for something that doesn't yield a check. If no 
one else creates one in the next week or so, I'll see if I can whip 
something together.

Peace,
Phillip

Markus Kohler wrote:
> Hi all,
>
> If someone has a simple test,I would be willing to take a heap dump 
> and check what is going on using the
> Eclipse Memory Analyzer ( http://eclipse.org/mat/).
>
> BTW, I think tools like celerity are the future of testing web 
> applications.
> I hope a can completly stay away from Loadrunner-like tools in the 
> near future ;-)
>
>
> Regards,
> Markus (http://kohlerm.blogspot.com/)
> "The best way to predict the future is to invent it" -- Alan Kay

From markus.kohler at gmail.com  Fri Feb  5 11:58:08 2010
From: markus.kohler at gmail.com (Markus Kohler)
Date: Fri, 5 Feb 2010 17:58:08 +0100
Subject: [Celerity-users] Getting "java.lang.OutOfMemoryError: Java heap
	space" after crawling a lot
In-Reply-To: <4B6C4A4A.3040701@gmail.com>
References: <7FD77CFD-E0B0-4B48-A1EB-B773A057EA69@hexagile.com>
	<4B69E3B7.2060500@gmail.com> <4B6AFF7F.9030508@gmail.com>
	<4B6B4A71.3000402@gmail.com>
	<771905291002050042l1491cb32g68d57ac958e85aee@mail.gmail.com>
	<4B6C4A4A.3040701@gmail.com>
Message-ID: <771905291002050858l690adf08lb23a5222691aaa4d@mail.gmail.com>

No hurry,
I'm pretty busy these days as well.
Just let me know as soon as you have something.

Markus
"The best way to predict the future is to invent it" -- Alan Kay


On Fri, Feb 5, 2010 at 5:41 PM, Phillip Koebbe <phillipkoebbe at gmail.com>wrote:

> That's very generous of you, Markus. Thanks. I would create a simple
> project, but won't have time for a while. I spent the whole day yesterday
> working out enabling/disabling JavaScript automatically through Culerity and
> didn't get any paid work done. I can't afford to take more time right now
> for something that doesn't yield a check. If no one else creates one in the
> next week or so, I'll see if I can whip something together.
>
> Peace,
> Phillip
>
>
> Markus Kohler wrote:
>
>> Hi all,
>>
>> If someone has a simple test,I would be willing to take a heap dump and
>> check what is going on using the
>> Eclipse Memory Analyzer ( http://eclipse.org/mat/).
>>
>> BTW, I think tools like celerity are the future of testing web
>> applications.
>> I hope a can completly stay away from Loadrunner-like tools in the near
>> future ;-)
>>
>>
>> Regards,
>> Markus (http://kohlerm.blogspot.com/)
>> "The best way to predict the future is to invent it" -- Alan Kay
>>
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/celerity-users/attachments/20100205/cc59d733/attachment-0001.html>

From p.olli at luukku.com  Tue Feb 16 06:24:13 2010
From: p.olli at luukku.com (Pasi Olli)
Date: Tue, 16 Feb 2010 13:24:13 +0200 (EET)
Subject: [Celerity-users] JS plugin problem
Message-ID: <1266319453472.p.olli.45579.-v2GacwIEggn7vM0sX_LCA@luukku.com>

Hi,

I need to load one mozilla plugin to execute my javascript command through celerity. Does anyone know how that would be possible ? 

I create the browser object now in this way:

@browser = Celerity::Browser.new(:browser => :firefox3, :log_level => :all, :javascript_exceptions => true, :javascript_enabled => true, :css => true)

but it seems not to be enough to imitate real browser.

After browser object creation I goto html-page which has couple of JS included and try execute the js-functions inside but they cannot because of the missing plugin.

Functions execute fine if real browser is used ?? Any ideas ??







....................................................................
Luukku Plus -paketilla p??set eroon tila- ja turvallisuusongelmista.
Hanki Luukku Plus ja helpotat el?m??si. http://www.mtv3.fi/luukku

From peter at hexagile.com  Tue Feb 16 07:37:11 2010
From: peter at hexagile.com (Peter Szinek)
Date: Tue, 16 Feb 2010 13:37:11 +0100
Subject: [Celerity-users] JS plugin problem
In-Reply-To: <1266319453472.p.olli.45579.-v2GacwIEggn7vM0sX_LCA@luukku.com>
References: <1266319453472.p.olli.45579.-v2GacwIEggn7vM0sX_LCA@luukku.com>
Message-ID: <97FF7EEC-08BE-49D2-935F-0A842551047D@hexagile.com>

Hi Pasi,

I am not sure about your problem, but I am almost sure that without  
further tweaks, celerity does NOT look like a 'real' browser...  a  
colleague of mine had to modify mechanize to look like a real browser  
(because they are heavily scraping google, and google hates to be  
heavily scraped), and they had to do a LOT of tweaks to look like a  
real browser (among others: download all assets, process JS/CSS to  
include any referenced assets within, retaining session info and re- 
sending cookies etc, obeying expires headers, doing appropriate HEAD  
request to get an updated expiry for any assets that may have expired,  
ensuring that each mechanize session is bound to a specific proxy/use  
agent/cookies/expire headers/etc for the duration of that session....  
- this is about 20% of the changes) and of course rotate the calls  
through a broad range of private IPs...

What I am trying to convey is that mechanize (and probably celerity  
too, though I don't know the detailed differences) is far from a  
'real' browser in the strict sense.

That said, beating google and this component are entirely different  
stories - I doubt this one bothers whether you are downloading and  
processing CSS or not, I would suspect some kind of security measures  
in Firefox...

P

On 2010.02.16., at 12:24, Pasi Olli wrote:

> Hi,
>
> I need to load one mozilla plugin to execute my javascript command  
> through celerity. Does anyone know how that would be possible ?
>
> I create the browser object now in this way:
>
> @browser = Celerity::Browser.new(:browser => :firefox3, :log_level  
> => :all, :javascript_exceptions => true, :javascript_enabled =>  
> true, :css => true)
>
> but it seems not to be enough to imitate real browser.
>
> After browser object creation I goto html-page which has couple of  
> JS included and try execute the js-functions inside but they cannot  
> because of the missing plugin.
>
> Functions execute fine if real browser is used ?? Any ideas ??
>
>
>
>
>
>
>
> ....................................................................
> Luukku Plus -paketilla p??set eroon tila- ja turvallisuusongelmista.
> Hanki Luukku Plus ja helpotat el?m??si. http://www.mtv3.fi/luukku
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users


From minger at gmail.com  Tue Feb 16 08:40:09 2010
From: minger at gmail.com (Ming)
Date: Tue, 16 Feb 2010 08:40:09 -0500
Subject: [Celerity-users] JS plugin problem
In-Reply-To: <97FF7EEC-08BE-49D2-935F-0A842551047D@hexagile.com>
References: <1266319453472.p.olli.45579.-v2GacwIEggn7vM0sX_LCA@luukku.com>
	<97FF7EEC-08BE-49D2-935F-0A842551047D@hexagile.com>
Message-ID: <fa42d3331002160540q79eed09dobd1eeb082aab6e37@mail.gmail.com>

The browser parameter merely sets the user-agent string.

If you need Mozilla specific functionality, you'll need a solution that
scripts Mozilla.  I haven't used any but have come across at least one
during my research.

2010/2/16 Peter Szinek <peter at hexagile.com>

> Hi Pasi,
>
> I am not sure about your problem, but I am almost sure that without further
> tweaks, celerity does NOT look like a 'real' browser...  a colleague of mine
> had to modify mechanize to look like a real browser (because they are
> heavily scraping google, and google hates to be heavily scraped), and they
> had to do a LOT of tweaks to look like a real browser (among others:
> download all assets, process JS/CSS to include any referenced assets within,
> retaining session info and re-sending cookies etc, obeying expires headers,
> doing appropriate HEAD request to get an updated expiry for any assets that
> may have expired, ensuring that each mechanize session is bound to a
> specific proxy/use agent/cookies/expire headers/etc for the duration of that
> session.... - this is about 20% of the changes) and of course rotate the
> calls through a broad range of private IPs...
>
> What I am trying to convey is that mechanize (and probably celerity too,
> though I don't know the detailed differences) is far from a 'real' browser
> in the strict sense.
>
> That said, beating google and this component are entirely different stories
> - I doubt this one bothers whether you are downloading and processing CSS or
> not, I would suspect some kind of security measures in Firefox...
>
> P
>
>
> On 2010.02.16., at 12:24, Pasi Olli wrote:
>
>  Hi,
>>
>> I need to load one mozilla plugin to execute my javascript command through
>> celerity. Does anyone know how that would be possible ?
>>
>> I create the browser object now in this way:
>>
>> @browser = Celerity::Browser.new(:browser => :firefox3, :log_level =>
>> :all, :javascript_exceptions => true, :javascript_enabled => true, :css =>
>> true)
>>
>> but it seems not to be enough to imitate real browser.
>>
>> After browser object creation I goto html-page which has couple of JS
>> included and try execute the js-functions inside but they cannot because of
>> the missing plugin.
>>
>> Functions execute fine if real browser is used ?? Any ideas ??
>>
>>
>>
>>
>>
>>
>>
>> ....................................................................
>> Luukku Plus -paketilla p??set eroon tila- ja turvallisuusongelmista.
>> Hanki Luukku Plus ja helpotat el?m??si. http://www.mtv3.fi/luukku
>> _______________________________________________
>> Celerity-users mailing list
>> Celerity-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/celerity-users
>>
>
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/celerity-users/attachments/20100216/c8cb4a9b/attachment.html>

From phillipkoebbe at gmail.com  Fri Feb 19 11:41:35 2010
From: phillipkoebbe at gmail.com (Phillip Koebbe)
Date: Fri, 19 Feb 2010 10:41:35 -0600
Subject: [Celerity-users] Timing Question
Message-ID: <4B7EBF3F.4080003@gmail.com>

I've been having some seemingly random failures in my cucumber 
scenarios, and I believe I have isolated them down to timing problems. I 
have a number of scenarios that submit forms and then check to see that 
a particular message is present. Periodically, I get failures in 
scenarios that don't normally fail. For example, I just ran my whole 
feature set four or five times, each time different tests failed. I 
noticed that the failures came after a button click, so I inserted a 1 
second wait after clicking the button. Then I ran my whole feature set 
again, and got no failures. I then ran it four more times right in a 
row, all with no failures. So it seems that pausing briefly after a 
button click gives things times to process properly.

My question, then, is does Celerity (or HtmlUnit, I'm not sure where 
this behavior would be) wait until a response has been received and 
processed before it passes status back through the chain? In my case, 
I'm using Culerity as a bridge to Celerity in Cucumber, and if Celerity 
is wrapped around HtmlUnit, it seems HtmlUnit would need to update the 
status to Celerity, which would need to update Culerity, which would 
need to update Cucumber, which then advances to the next step. Is this 
understanding correct?

Is there any way to tell Celerity/HtmlUnit to wait until the response is 
complete before moving on? I'm thinking that's essentially what I'm 
doing with the 1 second wait.

Peace,
Phillip

From jari.bakken at gmail.com  Fri Feb 19 12:43:57 2010
From: jari.bakken at gmail.com (Jari Bakken)
Date: Fri, 19 Feb 2010 18:43:57 +0100
Subject: [Celerity-users] Timing Question
In-Reply-To: <4B7EBF3F.4080003@gmail.com>
References: <4B7EBF3F.4080003@gmail.com>
Message-ID: <eae5f6f91002190943j2b21af1ewf804e790d9589c89@mail.gmail.com>

On Fri, Feb 19, 2010 at 5:41 PM, Phillip Koebbe <phillipkoebbe at gmail.com> wrote:
>
> Is there any way to tell Celerity/HtmlUnit to wait until the response is
> complete before moving on? I'm thinking that's essentially what I'm doing
> with the 1 second wait.
>

Celerity/HtmlUnit has a blocking API for most things, but Ajax
requests are not synchronized by default. See this page on the wiki
for more:

http://wiki.github.com/jarib/celerity/ajax

From phillipkoebbe at gmail.com  Fri Feb 19 12:52:35 2010
From: phillipkoebbe at gmail.com (Phillip Koebbe)
Date: Fri, 19 Feb 2010 11:52:35 -0600
Subject: [Celerity-users] Timing Question
In-Reply-To: <eae5f6f91002190943j2b21af1ewf804e790d9589c89@mail.gmail.com>
References: <4B7EBF3F.4080003@gmail.com>
	<eae5f6f91002190943j2b21af1ewf804e790d9589c89@mail.gmail.com>
Message-ID: <4B7ECFE3.1080504@gmail.com>

Jari Bakken wrote:
> On Fri, Feb 19, 2010 at 5:41 PM, Phillip Koebbe<phillipkoebbe at gmail.com>  wrote:
>    
>> Is there any way to tell Celerity/HtmlUnit to wait until the response is
>> complete before moving on? I'm thinking that's essentially what I'm doing
>> with the 1 second wait.
>>
>>      
>
> Celerity/HtmlUnit has a blocking API for most things, but Ajax
> requests are not synchronized by default. See this page on the wiki
> for more:
>
>    

Hi Jari,

These aren't ajax submissions. Just standard forms (form_tag or form_for 
in Rails). And I already pass resynchronize => true when creating a new 
browser.

Phillip

From jari.bakken at gmail.com  Fri Feb 19 13:05:01 2010
From: jari.bakken at gmail.com (Jari Bakken)
Date: Fri, 19 Feb 2010 19:05:01 +0100
Subject: [Celerity-users] Timing Question
In-Reply-To: <4B7ECFE3.1080504@gmail.com>
References: <4B7EBF3F.4080003@gmail.com>
	<eae5f6f91002190943j2b21af1ewf804e790d9589c89@mail.gmail.com>
	<4B7ECFE3.1080504@gmail.com>
Message-ID: <eae5f6f91002191005l7eaccef3u7ed3d667a0f45fed@mail.gmail.com>

>
> These aren't ajax submissions. Just standard forms (form_tag or form_for in
> Rails). And I already pass resynchronize => true when creating a new
> browser.
>

In that case it's hard to help unless you can provide a reproducible
example. I'm not aware of any bugs in Celerity that would cause your
problem.

From phillipkoebbe at gmail.com  Fri Feb 19 13:11:36 2010
From: phillipkoebbe at gmail.com (Phillip Koebbe)
Date: Fri, 19 Feb 2010 12:11:36 -0600
Subject: [Celerity-users] Timing Question
In-Reply-To: <eae5f6f91002191005l7eaccef3u7ed3d667a0f45fed@mail.gmail.com>
References: <4B7EBF3F.4080003@gmail.com>	<eae5f6f91002190943j2b21af1ewf804e790d9589c89@mail.gmail.com>	<4B7ECFE3.1080504@gmail.com>
	<eae5f6f91002191005l7eaccef3u7ed3d667a0f45fed@mail.gmail.com>
Message-ID: <4B7ED458.1040906@gmail.com>



Jari Bakken wrote:
>> These aren't ajax submissions. Just standard forms (form_tag or form_for in
>> Rails). And I already pass resynchronize =>  true when creating a new
>> browser.
>>
>>      
>
> In that case it's hard to help unless you can provide a reproducible
> example. I'm not aware of any bugs in Celerity that would cause your
> problem.
>    

Hm. Actually, I was just looking for answers to the questions. At this 
point, I don't know if there even is a bug, and I'm not sure that I'm 
suggesting there is one. I really just want to know if Celerity/HtmlUnit 
*should* be waiting for the server's response before passing status back 
through the chain.

Thanks, Jari.


From jari.bakken at gmail.com  Fri Feb 19 14:09:01 2010
From: jari.bakken at gmail.com (Jari Bakken)
Date: Fri, 19 Feb 2010 20:09:01 +0100
Subject: [Celerity-users] Timing Question
In-Reply-To: <4B7ED458.1040906@gmail.com>
References: <4B7EBF3F.4080003@gmail.com>
	<eae5f6f91002190943j2b21af1ewf804e790d9589c89@mail.gmail.com>
	<4B7ECFE3.1080504@gmail.com>
	<eae5f6f91002191005l7eaccef3u7ed3d667a0f45fed@mail.gmail.com>
	<4B7ED458.1040906@gmail.com>
Message-ID: <eae5f6f91002191109u522e6914qcc33f4984a5fe8cb@mail.gmail.com>

On Fri, Feb 19, 2010 at 7:11 PM, Phillip Koebbe <phillipkoebbe at gmail.com> wrote:
>
> Hm. Actually, I was just looking for answers to the questions. At this
> point, I don't know if there even is a bug, and I'm not sure that I'm
> suggesting there is one. I really just want to know if Celerity/HtmlUnit
> *should* be waiting for the server's response before passing status back
> through the chain.
>

Yes, it should. However I can imagine there would be issues if Rails
has clever submit handlers and the actual POST request isn't triggered
until after several iterations of the JS event loop. What happens if
you replace your sleep()s with browser.wait() calls?

From phillipkoebbe at gmail.com  Fri Feb 19 14:31:21 2010
From: phillipkoebbe at gmail.com (Phillip Koebbe)
Date: Fri, 19 Feb 2010 13:31:21 -0600
Subject: [Celerity-users] Timing Question
In-Reply-To: <eae5f6f91002191109u522e6914qcc33f4984a5fe8cb@mail.gmail.com>
References: <4B7EBF3F.4080003@gmail.com>	<eae5f6f91002190943j2b21af1ewf804e790d9589c89@mail.gmail.com>	<4B7ECFE3.1080504@gmail.com>	<eae5f6f91002191005l7eaccef3u7ed3d667a0f45fed@mail.gmail.com>	<4B7ED458.1040906@gmail.com>
	<eae5f6f91002191109u522e6914qcc33f4984a5fe8cb@mail.gmail.com>
Message-ID: <4B7EE709.5070109@gmail.com>


Jari Bakken wrote:
> On Fri, Feb 19, 2010 at 7:11 PM, Phillip Koebbe<phillipkoebbe at gmail.com>  wrote:
>    
>> Hm. Actually, I was just looking for answers to the questions. At this
>> point, I don't know if there even is a bug, and I'm not sure that I'm
>> suggesting there is one. I really just want to know if Celerity/HtmlUnit
>> *should* be waiting for the server's response before passing status back
>> through the chain.
>>
>>      
>
> Yes, it should. However I can imagine there would be issues if Rails
> has clever submit handlers and the actual POST request isn't triggered
> until after several iterations of the JS event loop. What happens if
> you replace your sleep()s with browser.wait() calls?
>
>    

Funny you should mention that. Browser.wait was the first thing I tried. 
It didn't seem to affect it much, if at all. At least I couldn't tell a 
difference. There were still unexpected failures.

I'll try to put together an example, but it might be difficult. I don't 
really want to reproduce most of my current application. Maybe I can 
strip out the unnecessary parts and use just what's failing.

Phillip