[Celerity-users] Parallel running test?

[jason franklin-stokes]

Markus, 

just to clarify - htmlunit is thread safe!! no doubts about that - but its not just about htmlunit - its going to be a combi of your code and htmlunit - and the resources you have available for your goals and the strategy you go for. I am spawning a lot of instances that go out doing their own thing in unknown territory -  as quickly as possible -using as little new code as possible - which is what I am assuming what you wanna be doing - as you are looking at multithreading. At the end of the day it is mostly about your htmlunit error handling and thread stratgey/recovery/monitoring and resource management. I personally did not find this trivial. 

I am planning to blog the details in after xmas - but for now here is a shopping list: 

- atomize the tasks you want htmlunit to do - i.e. no not use huge loops that do a ton of things. 
- use rails to encapsulate your htmlunit tasks
- trigger the tasks (i use restclient) that sends the task information to a rails controller that in turn triggers your htmlunit tasks.
- put rails in multithread mode so that you trigger the htmlunit in parralell
- launch the app in some application server (i use glassfish V3) 
- monitor the threads and memory useage accordingly (I use webvm for debugging and monitoring). 
- always always always close browser when you are finished with an instance. 
- I needed to add a method to celerity in class browser and class clickable element that forrces a page to open in a new window when you click it - it has the same effect as click_and_attach but without the overhead (i could not get click_and_attach working properly for what i needed in my environment - but this might not apply to you - it depends what you want to do)

It might seem a little strange to encapsulate htmlunit in the rails framework - but rails is thread safe and so is htmlunit - and you have database persistance at hand if you need it - all you have to do is put it together, rather reinventing the wheel - which means a lot less can go wrong - and its not that much additional overhead.

You can expect to need between 30MB and 100MB per browser instance depending on the website and the tasks it performs etc - and the time needed before GC kicks in for you to clean up the mess after you.

P.S the other changes I made to htmlunit and celerity all have to do with persistance in a multithreading environment and less with mutithreading itself. 

I hope that makes some sence and is of some help. 

Jason.



On Dec 15, 2009, at 1:12 AM, Markus Kohler wrote:

> Hi,
> Thanks!
> I tried to just spawn threads from jruby but that didn't really work properly.
> What kind of changes where necessary to htmlunit? Isn't it thread safe? 
> Markus
> 
> 
>> On Dec 15, 2009 12:44 AM, "jason franklin-stokes" <jasoninclass at googlemail.com> wrote:
>> 
>> hmm,
>> 
>> I have 100 running in Parallel....
>> 
>> I can't say that I can show you how that works in a few lines of code.
>> 
>> I needed to make some changes to htmlunit and to celerity to getting running properly (and I am still not too sure that is).
>> 
>> I ended up using rails in multithreded mode to encapsulate the browser instances and a seperate server to trigger the browser instances with tasks.
>> 
>> The browser instances are running as a jar file in glassfish V3
>> 
>> I definitely did not find this an easy task.... but its fun.
>> 
>> Jason
>> On Dec 15, 2009, at 12:00 AM, Markus Kohler wrote: > Hi all, > Does someone have simple example o...
>> 
>> > _______________________________________________
>> > Celerity-users mailing list
>> > Celerity-users at rubyforge.org
>> > http://rubyforge.org/mailman/listinfo/celerity-users
>> 
>> _______________________________________________
>> Celerity-users mailing list
>> Celerity-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/celerity-users
> 
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/celerity-users/attachments/20091215/0ef0675d/attachment.html>