You can use the built build thread pool to process more than one feed within the same worker. So within the worker, you'd do,<br><br>def parse_feeds<br> loop do<br> feed = Feed.find_feed_to_process<br> thread_pool.defer do <br>
feed.parse<br> end<br> end<br>end<br><br>I think the default pool size is 20. You can control the size of the thread pool using a class level method, as I recall it is<br><br>pool_size x<br><br>Paul<br><br><div class="gmail_quote">
On Wed, Apr 23, 2008 at 7:30 AM, Julien Genestoux <<a href="mailto:julien.genestoux@gmail.com">julien.genestoux@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Thanks Adam,<br>
<br>
That sounded weird to me as well to have one worker for each feed...<br>
However, if I only have one worker, that also means that I am parsing<br>
one feed only at any moment. An option, is maybe to have a few workers<br>
(denpending on the number of feeds) that parse feeds concurrently?<br>
<br>
If I only have one worker, according to you what should be the<br>
winnning strategy to choose the "right" parse to feed? Obviously some<br>
feeds need to be parsed one every few minutes, while some other might<br>
no need to be parse more than every hour...<br>
<br>
Any idea/tip on this?<br>
<div><div></div><div class="Wj3C7c"><br>
<br>
<br>
<br>
On 4/23/08, Adam Williams <<a href="mailto:adam@thewilliams.ws">adam@thewilliams.ws</a>> wrote:<br>
> On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote:<br>
><br>
> > I still have a few questions : shoud I have one worker for each feed<br>
> > that is called periodically (add_periodic_timer) or rather one single<br>
> > worker that calls every feed one by one?<br>
> ><br>
> > What is the best solution, perfomance-wise?<br>
><br>
><br>
> Good question... I don't suppose I know exactly. I would start by<br>
> processing all the feeds in one worker invocation - that is what I<br>
> have done for sending an unknown amount of email. It just seems wrong<br>
> to me to invoke a worker for one email at a time.<br>
><br>
> The right answer likely lies in understanding the whole MasterWorker,<br>
> Packet::Reactor/handler_instance.ask_work bits of the puzzle...<br>
><br>
><br>
> adam<br>
><br>
> _______________________________________________<br>
> Backgroundrb-devel mailing list<br>
> <a href="mailto:Backgroundrb-devel@rubyforge.org">Backgroundrb-devel@rubyforge.org</a><br>
> <a href="http://rubyforge.org/mailman/listinfo/backgroundrb-devel" target="_blank">http://rubyforge.org/mailman/listinfo/backgroundrb-devel</a><br>
><br>
<br>
<br>
</div></div><div class="Ih2E3d">--<br>
--<br>
Julien Genestoux<br>
<a href="mailto:julien.genestoux@gmail.com">julien.genestoux@gmail.com</a><br>
<a href="http://www.ouvre-boite.com" target="_blank">http://www.ouvre-boite.com</a><br>
+1 (415) 254 7340<br>
+33 (0)8 70 44 76 29<br>
_______________________________________________<br>
</div><div><div></div><div class="Wj3C7c">Backgroundrb-devel mailing list<br>
<a href="mailto:Backgroundrb-devel@rubyforge.org">Backgroundrb-devel@rubyforge.org</a><br>
<a href="http://rubyforge.org/mailman/listinfo/backgroundrb-devel" target="_blank">http://rubyforge.org/mailman/listinfo/backgroundrb-devel</a><br>
</div></div></blockquote></div><br>