[Celerity-users] Parallel running test?

[Markus Kohler]

Hi Jason,
Thanks a lot for your reply!
Makes a lot of sense to me.
I'm looking forward to your blog post, where should I look for it?  :-)

BTW your link on twitter is broken.

Markus
"The best way to predict the future is to invent it" -- Alan Kay


On Tue, Dec 15, 2009 at 8:40 AM, jason franklin-stokes <
jasoninclass at googlemail.com> wrote:

>
> Markus,
>
> just to clarify - htmlunit is thread safe!! no doubts about that - but its
> not just about htmlunit - its going to be a combi of your code and htmlunit
> - and the resources you have available for your goals and the strategy you
> go for. I am spawning a lot of instances that go out doing their own thing
> in unknown territory -  as quickly as possible -using as little new code as
> possible - which is what I am assuming what you wanna be doing - as you are
> looking at multithreading. At the end of the day it is mostly about your
> htmlunit error handling and thread stratgey/recovery/monitoring and resource
> management. I personally did not find this trivial.
>
> I am planning to blog the details in after xmas - but for now here is a
> shopping list:
>
> - atomize the tasks you want htmlunit to do - i.e. no not use huge loops
> that do a ton of things.
> - use rails to encapsulate your htmlunit tasks
> - trigger the tasks (i use restclient) that sends the task information to a
> rails controller that in turn triggers your htmlunit tasks.
> - put rails in multithread mode so that you trigger the htmlunit in
> parralell
> - launch the app in some application server (i use glassfish V3)
> - monitor the threads and memory useage accordingly (I use webvm for
> debugging and monitoring).
> - always always always close browser when you are finished with an
> instance.
> - I needed to add a method to celerity in class browser and class clickable
> element that forrces a page to open in a new window when you click it - it
> has the same effect as click_and_attach but without the overhead (i could
> not get click_and_attach working properly for what i needed in my
> environment - but this might not apply to you - it depends what you want to
> do)
>
> It might seem a little strange to encapsulate htmlunit in the rails
> framework - but rails is thread safe and so is htmlunit - and you have
> database persistance at hand if you need it - all you have to do is put it
> together, rather reinventing the wheel - which means a lot less can go wrong
> - and its not that much additional overhead.
>
> You can expect to need between 30MB and 100MB per browser instance
> depending on the website and the tasks it performs etc - and the time needed
> before GC kicks in for you to clean up the mess after you.
>
> P.S the other changes I made to htmlunit and celerity all have to do with
> persistance in a multithreading environment and less with mutithreading
> itself.
>
> I hope that makes some sence and is of some help.
>
> Jason.
>
>
>
> On Dec 15, 2009, at 1:12 AM, Markus Kohler wrote:
>
> Hi,
> Thanks!
> I tried to just spawn threads from jruby but that didn't really work
> properly.
> What kind of changes where necessary to htmlunit? Isn't it thread safe?
> Markus
>
> On Dec 15, 2009 12:44 AM, "jason franklin-stokes" <
> jasoninclass at googlemail.com> wrote:
>
> hmm,
>
> I have 100 running in Parallel....
>
> I can't say that I can show you how that works in a few lines of code.
>
> I needed to make some changes to htmlunit and to celerity to getting
> running properly (and I am still not too sure that is).
>
> I ended up using rails in multithreded mode to encapsulate the browser
> instances and a seperate server to trigger the browser instances with tasks.
>
> The browser instances are running as a jar file in glassfish V3
>
> I definitely did not find this an easy task.... but its fun.
>
> Jason
>
> On Dec 15, 2009, at 12:00 AM, Markus Kohler wrote: > Hi all, > Does someone
> have simple example o...
> > _______________________________________________
> > Celerity-users mailing list
> > Celerity-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/celerity-users
>
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users
>
>
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users
>
>
>
> _______________________________________________
> Celerity-users mailing list
> Celerity-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/celerity-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/celerity-users/attachments/20091215/a372c456/attachment.html>