From leomayleomay at gmail.com Wed Sep 3 05:06:42 2008 From: leomayleomay at gmail.com (Hao Liu) Date: Wed, 03 Sep 2008 17:06:42 +0800 Subject: [Backgroundrb-devel] [Help] How to pass multiple arguments to a method in a worker? Message-ID: <48BE53A2.7050707@gmail.com> Hi, guys, I ran into a problem while I tried to pass more than one parameters to a method in the worker class, it fails, from the backgroundrb log, I found that: You have a nil object when you didn't expect it! You might have expected an instance of ActiveRecord::Base. The error occurred while evaluating nil.[] Seems the args is nil, but a hash I passed in my controller. From the post: http://rubyforge.org/pipermail/backgroundrb-devel/2008-April/001668.html, it should be a hash if there are multiple arguments. Here's what I did: In my controller: def blah MiddleMan.worker(:hard_worker).send_mail(:args => {:recipient => 'blahblah at gmail.com', :subject => 'BlahBlah'}) end In my worker class: def send_mail(args) emailer.send_mail(args[:recipient], args[:subject]) end Is there anything wrong? Additional info: ruby-1.8.6 backgroundrb 1.0.4 All the best, Hao Liu From gethemant at gmail.com Wed Sep 3 06:50:03 2008 From: gethemant at gmail.com (hemant kumar) Date: Wed, 03 Sep 2008 16:20:03 +0530 Subject: [Backgroundrb-devel] [Help] How to pass multiple arguments to a method in a worker? In-Reply-To: <48BE53A2.7050707@gmail.com> References: <48BE53A2.7050707@gmail.com> Message-ID: <1220439003.7824.1.camel@shire> On Wed, 2008-09-03 at 17:06 +0800, Hao Liu wrote: > Hi, guys, > I ran into a problem while I tried to pass more than one parameters to > a method in the worker class, it fails, from the backgroundrb log, I > found that: > > You have a nil object when you didn't expect it! > You might have expected an instance of ActiveRecord::Base. > The error occurred while evaluating nil.[] > > Seems the args is nil, but a hash I passed in my controller. From the > post: > http://rubyforge.org/pipermail/backgroundrb-devel/2008-April/001668.html, > it should be a hash if there are multiple arguments. Here's what I did: > > In my controller: > def blah > MiddleMan.worker(:hard_worker).send_mail(:args => {:recipient => > 'blahblah at gmail.com', :subject => 'BlahBlah'}) > end > > In my worker class: > def send_mail(args) > emailer.send_mail(args[:recipient], args[:subject]) > end Err, because its ":arg" not ":args" , hence: MiddleMan.worker(:hard_worker).send_mail(:arg => {:recipient => 'blahblah at gmail.com', :subject => 'BlahBlah'}) I thought, I had it documented here: http://backgroundrb.rubyforge.org/rails/ From zach at plugthegap.co.uk Wed Sep 3 20:48:53 2008 From: zach at plugthegap.co.uk (Zachary Powell) Date: Wed, 3 Sep 2008 20:48:53 -0400 Subject: [Backgroundrb-devel] TypeError: Class (expected Thread) [updated] Message-ID: My backgroundrb server has just started returning these errors. Its happening on all workers (in random places), but the below are examples where its happening in the server on its own (not in worker code). I've tried restarting, has anyone else experienced this? The only other lead is that alot of one of the workers are building up (I guess after they crash they hang around). It is happening every few minutes... as far as I know nothing's changed in the system. # (expected Thread)> /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:127:in `start_reactor' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `loop' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `start_reactor' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in `run' /home/rails/sites/arenaflowers.net-4/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in `initialize' script/backgroundrb:106:in `new' script/backgroundrb:106 # /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:19:in `load' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:19:in `receive_data' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:37:in `call' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:37:in `extract' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:18:in `receive_data' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:64:in `handle_internal_messages' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in `handle_read_event' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in `each' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in `handle_read_event' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in `start_reactor' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `loop' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `start_reactor' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in `run' /home/rails/sites/arenaflowers.net-4/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in `initialize' script/backgroundrb:106:in `new' script/backgroundrb:106 # /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:19:in `load' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:19:in `receive_data' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:52:in `call' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:52:in `extract' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:18:in `receive_data' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:64:in `handle_internal_messages' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in `handle_read_event' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in `each' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in `handle_read_event' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in `start_reactor' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `loop' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `start_reactor' /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in `run' /home/rails/sites/arenaflowers.net-4/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in `initialize' script/backgroundrb:106:in `new' script/backgroundrb:106 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ramon.tayag at gmail.com Thu Sep 4 06:01:31 2008 From: ramon.tayag at gmail.com (Ramon Miguel M. Tayag) Date: Thu, 4 Sep 2008 18:01:31 +0800 Subject: [Backgroundrb-devel] Capistrano and bdrb? Message-ID: Hey all. I ended up using monit to monitor backgroundrb. It works great - it starts and restarts it (or so I thought) when specified. However, just now, I noticed that I was getting an invalid worker name error when trying to send out mail. I did some digging around and saw that my "script/backgroundrb start" has been on for quite a while (I did a "ps -ef | grep backgroundrb") and even if I did a "ruby script/backgroundrb stop" it still came out when I grepped it. I came to thinking that maybe when Capistrano changes the symlinks of the "current" folder to the latest release, the current backgroundrb script still points to the old release? So when monit tries to restart backgroundrb after the deploy of code, it's telling the _new_ script to stop and start, not the old currently running one. Is my theory right? If so, how can I fix this? To fix this, for now, I rebooted the server. However, this shouldn't be the case in the future :o I'm pasting my backgroundrb.monitrc file just in case (note I'm quite amateur): http://pastie.org/265841 -- Ramon Tayag From reynard.list at gmail.com Thu Sep 4 15:24:15 2008 From: reynard.list at gmail.com (Reynard) Date: Thu, 4 Sep 2008 15:24:15 -0400 Subject: [Backgroundrb-devel] Invalid worker with name ... Message-ID: <26e148920809041224i749daf7dtfff3bb8a5f94c5c6@mail.gmail.com> when trying this out using the latest from svn or git: MiddleMan.worker(:testa_worker).async_test() --- class TestaWorker < BackgrounDRb::MetaWorker set_worker_name :testa_worker def create(args = nil) end def test logger.info "sleep" sleep(5) logger.info "wake up" end end --- I get this error: Invalid worker with name testa_worker and key /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_connection.rb:52:in `ask_worker' /home/reynard/opt/aptana/workspace/test/vendor/plugins/backgroundrb/server/lib/master_worker.rb:103:in `async_method_invoke' /home/reynard/opt/aptana/workspace/test/vendor/plugins/backgroundrb/server/lib/master_worker.rb:37:in `receive_data' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_parser.rb:44:in `extract' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_parser.rb:26:in `loop' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_parser.rb:26:in `extract' /home/reynard/opt/aptana/workspace/test/vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in `receive_data' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:232:in `read_external_socket' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:222:in `handle_external_messages' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:196:in `handle_read_event' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:192:in `each' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:192:in `handle_read_event' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:146:in `start_reactor' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:139:in `loop' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:139:in `start_reactor' /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_master.rb:21:in `run' /home/reynard/opt/aptana/workspace/test/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in `initialize' script/backgroundrb:46:in `new' script/backgroundrb:46 Client disconected ---- I'm just using the example from the website. did I miss something? oh btw, using 1.0.3, I get undefined method async_test. I suppose that's new in 1.0.4? - reynard -------------- next part -------------- An HTML attachment was scrubbed... URL: From reynard.list at gmail.com Thu Sep 4 17:20:15 2008 From: reynard.list at gmail.com (Reynard) Date: Thu, 4 Sep 2008 17:20:15 -0400 Subject: [Backgroundrb-devel] Undefined method async_[some_method] Message-ID: <26e148920809041420p573b972uf07688919c6684bd@mail.gmail.com> So on the version 1.0.3 I get this when trying to run async_some_method Undefined method async_some_method called on worker some_worker I think it would be nice to have in the documentation what version is needed for a particular method/feature. I suppose I have to upgrade to 1.0.4, but as in the previous email the upgrade didn't go well either. heeelp ..... - reynard -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Thu Sep 4 17:25:15 2008 From: gethemant at gmail.com (gethemant at gmail.com) Date: Fri, 5 Sep 2008 02:55:15 +0530 Subject: [Backgroundrb-devel] Invalid worker with name ... In-Reply-To: <26e148920809041224i749daf7dtfff3bb8a5f94c5c6@mail.gmail.com> References: <26e148920809041224i749daf7dtfff3bb8a5f94c5c6@mail.gmail.com> Message-ID: Yes, thats only available in 1.0.4. On 9/5/08, Reynard wrote: > when trying this out using the latest from svn or git: > > MiddleMan.worker(:testa_worker).async_test() > > --- > class TestaWorker < BackgrounDRb::MetaWorker > set_worker_name :testa_worker > > def create(args = nil) > end > > def test > logger.info "sleep" > sleep(5) > logger.info "wake up" > end > end > --- > > > I get this error: > > Invalid worker with name testa_worker and key > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_connection.rb:52:in > `ask_worker' > /home/reynard/opt/aptana/workspace/test/vendor/plugins/backgroundrb/server/lib/master_worker.rb:103:in > `async_method_invoke' > /home/reynard/opt/aptana/workspace/test/vendor/plugins/backgroundrb/server/lib/master_worker.rb:37:in > `receive_data' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_parser.rb:44:in > `extract' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_parser.rb:26:in > `loop' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_parser.rb:26:in > `extract' > /home/reynard/opt/aptana/workspace/test/vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in > `receive_data' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:232:in > `read_external_socket' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:222:in > `handle_external_messages' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:196:in > `handle_read_event' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:192:in > `each' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:192:in > `handle_read_event' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:146:in > `start_reactor' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:139:in > `loop' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_core.rb:139:in > `start_reactor' > /usr/lib/ruby/gems/1.8/gems/packet-0.1.12/lib/packet/packet_master.rb:21:in > `run' > /home/reynard/opt/aptana/workspace/test/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in > `initialize' > script/backgroundrb:46:in `new' > script/backgroundrb:46 > Client disconected > > ---- > I'm just using the example from the website. > did I miss something? > > oh btw, using 1.0.3, I get undefined method async_test. I suppose that's new > in 1.0.4? > > > - reynard > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From reynard.list at gmail.com Thu Sep 4 17:44:03 2008 From: reynard.list at gmail.com (Reynard) Date: Thu, 4 Sep 2008 17:44:03 -0400 Subject: [Backgroundrb-devel] Invalid worker with name ... In-Reply-To: References: <26e148920809041224i749daf7dtfff3bb8a5f94c5c6@mail.gmail.com> Message-ID: <26e148920809041444n68c79360y2880ee24bc496373@mail.gmail.com> but when I use the latest from svn, running: MiddleMan.worker(:testa_worker).async_test() gives me this: Invalid worker with name testa_worker and key any idea? - reynard On Thu, Sep 4, 2008 at 5:25 PM, wrote: > Yes, thats only available in 1.0.4. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From reynard.list at gmail.com Thu Sep 4 17:56:12 2008 From: reynard.list at gmail.com (Reynard) Date: Thu, 4 Sep 2008 17:56:12 -0400 Subject: [Backgroundrb-devel] Invalid worker with name ... In-Reply-To: <26e148920809041444n68c79360y2880ee24bc496373@mail.gmail.com> References: <26e148920809041224i749daf7dtfff3bb8a5f94c5c6@mail.gmail.com> <26e148920809041444n68c79360y2880ee24bc496373@mail.gmail.com> Message-ID: <26e148920809041456k3325e4em9e3645a36bd66e53@mail.gmail.com> ah, maybe the problem is with the database table not getting created? when you run rake backgroundrb:setup it just create the database immediately so during deployment I just have to manually create the table on prod site? wouldn't it be nice if the rake task create a migration file and run it, so when deploying we just run the normal migration? - reynard On Thu, Sep 4, 2008 at 5:44 PM, Reynard wrote: > but when I use the latest from svn, running: > > MiddleMan.worker(:testa_worker).async_test() > > gives me this: > > Invalid worker with name testa_worker and key > > any idea? > > > - reynard > > > > On Thu, Sep 4, 2008 at 5:25 PM, wrote: > >> Yes, thats only available in 1.0.4. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From epugh at opensourceconnections.com Fri Sep 5 14:39:53 2008 From: epugh at opensourceconnections.com (Eric Pugh) Date: Fri, 5 Sep 2008 14:39:53 -0400 Subject: [Backgroundrb-devel] Upgrading to latest.. Message-ID: Folks, I am playing with writing a daemon_controller for backgroundrb, and realized I am using a 6 month old version! Do you recommend I grab the background rb from git? Or the from svn? Eric From gethemant at gmail.com Fri Sep 5 14:47:13 2008 From: gethemant at gmail.com (hemant kumar) Date: Sat, 06 Sep 2008 00:17:13 +0530 Subject: [Backgroundrb-devel] Upgrading to latest.. In-Reply-To: References: Message-ID: <1220640433.8926.1.camel@shire> On Fri, 2008-09-05 at 14:39 -0400, Eric Pugh wrote: > Folks, > > I am playing with writing a daemon_controller for backgroundrb, and > realized I am using a 6 month old version! Do you recommend I grab > the background rb from git? Or the from svn? > from git. Also, upgrade to packet 0.1.12, it has bits of nice little things now (unhandled/unexpected worker crashes are logged now) From epugh at opensourceconnections.com Fri Sep 5 15:14:49 2008 From: epugh at opensourceconnections.com (Eric Pugh) Date: Fri, 5 Sep 2008 15:14:49 -0400 Subject: [Backgroundrb-devel] Upgrading to latest.. In-Reply-To: <1220640433.8926.1.camel@shire> References: <1220640433.8926.1.camel@shire> Message-ID: Okay, I grabbed what was in svn listed as stable, but I'll move on up to git.. Thanks for the heads up on packet, I added to environment.rb: config.gem 'packet', :version => '>= 0.1.12' :-) Eric On Fri, Sep 5, 2008 at 2:47 PM, hemant kumar wrote: > On Fri, 2008-09-05 at 14:39 -0400, Eric Pugh wrote: >> Folks, >> >> I am playing with writing a daemon_controller for backgroundrb, and >> realized I am using a 6 month old version! Do you recommend I grab >> the background rb from git? Or the from svn? >> > > from git. Also, upgrade to packet 0.1.12, it has bits of nice little > things now (unhandled/unexpected worker crashes are logged now) > > > From funnie_16 at hotmail.com Sat Sep 6 16:04:22 2008 From: funnie_16 at hotmail.com (Jochen D.) Date: Sat, 6 Sep 2008 22:04:22 +0200 Subject: [Backgroundrb-devel] Worker_key invalid worker problem Message-ID: I am running the latest version of backgroundrb (git) with packet-0.1.12 and I am having problems using a key together with a worker. Without using a key, the worker works just fine, however when I do: worker = MiddleMan.worker(:hard_worker, 'mykey') worker.async_begin(:arg => 1) I receive an error: Invalid worker with name hard_worker_mykey and key mykey Looks like it appends the key to the worker name? Any idea what I might be doing wrong? Thanks! _________________________________________________________________ Vanaf nu heb je je vrienden overal bij! http://www.windowslivemobile.msn.com/?mkt=nl-be -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Sat Sep 6 16:40:39 2008 From: gethemant at gmail.com (hemant) Date: Sun, 7 Sep 2008 02:10:39 +0530 Subject: [Backgroundrb-devel] Worker_key invalid worker problem In-Reply-To: References: Message-ID: So, do you have worker "hard_worker" running with key "my_key" ? Whats the output of: MiddleMan.all_worker_info On Sun, Sep 7, 2008 at 1:34 AM, Jochen D. wrote: > > I am running the latest version of backgroundrb (git) with packet-0.1.12 and > I am having problems using a key together with a worker. > Without using a key, the worker works just fine, however when I do: > > worker = MiddleMan.worker(:hard_worker, 'mykey') > worker.async_begin(:arg => 1) > > I receive an error: > > Invalid worker with name hard_worker_mykey and key mykey > > Looks like it appends the key to the worker name? > > Any idea what I might be doing wrong? > > Thanks! > > ________________________________ > Dit schooljaar ben jij gratis de held online! Windows Live > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From brent.hargrave at gmail.com Sun Sep 7 17:07:44 2008 From: brent.hargrave at gmail.com (Brent Hargrave) Date: Sun, 7 Sep 2008 14:07:44 -0700 Subject: [Backgroundrb-devel] Backgroundrb incompatible with acts_as_readyonlyable? Message-ID: Hi all, I've setup MySQL replication using acts_as_readonlyable. The thin servers work fine, as does backgroundrb on non-read-only models. However, when it tries to read an ActiveRecord model that is "acts-as_readonlyable", I get the following error: ActiveRecord::ConnectionNotEstablished. To reiterate, I'm pretty sure it's not my configuration because the core rails app works as expected, as does backgroundrb for non-replicated models. Has anyone else experienced this problem, or know what may be the cause of it? Here's a gist of the backtrace: http://gist.github.com/9319 Note; I'm running rails 2.0.2 and a backgroundrb version from a little after 2008-02-28 (last changelog update). I have not upgraded since then for stability reasons. Any suggestions would be much appreciated. Thanks, Brent -------------- next part -------------- An HTML attachment was scrubbed... URL: From brentmc79 at gmail.com Mon Sep 8 12:26:21 2008 From: brentmc79 at gmail.com (Brent Collier) Date: Mon, 8 Sep 2008 12:26:21 -0400 Subject: [Backgroundrb-devel] Problems with async worker request Message-ID: Sorry if this comes through twice. I already sent this once, before joining the mailing list. I'm attempting to use Backgroundrb to handle asynchronous pdf creation, but in doing so, I've run into a very strange problem. Below is a method that's called from the controller which creates a new worker, then grabs the worker and calls the 'build_pdf' method asynchronously. def make_pdf(template_path, worker_key) with_empty_asset_id do html_string = render_to_string(:template => template_path, :layout => 'pdf') key = MiddleMan.new_worker(:worker => :prince_xml_worker, :worker_key => worker_key) MiddleMan.worker(:prince_xml_worker, key).async_build_pdf(:arg => html_string) end end Then there's another method which polls the worker like so. def ask_worker_status(worker_key) MiddleMan.worker(:prince_xml_worker, worker_key).ask_result(:pdf) end It didn't seem to be working properly, so I started doing a little debugging. Suddenly, it worked! So I removed my debugger statements and tried again. It stopped working. I kept going back and forth like this, trying differnet scenarios, poking at the workers, and checking their results. Finally, I narrowed it down to one debugger statement in the make_pdf method. def make_pdf(template_path, worker_key) with_empty_asset_id do html_string = render_to_string(:template => template_path, :layout => 'pdf') key = MiddleMan.new_worker(:worker => :prince_xml_worker, :worker_key => worker_key) worker = MiddleMan.worker(:prince_xml_worker, key) >> debugger worker.async_build_pdf(:arg => html_string) end end When that debugger statement is in the code, everything works perfectly. When it hits the debugger during the request, it doesn't matter whether I poke around at a few objects, or just continue immediately. I even tried replacing the 'debugger' with a 'sleep(1)' and everything worked perfectly. When I removed the sleep call, no worky. If I look at the backgroundrb_debug_11006 log, I see "Client disconnected" each time the app polls the worker, and occasionally, I see this: Packet::InvalidWorker /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_connection.rb:52:in `ask_worker' /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:123:in `get_result_object' /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:39:in `receive_data' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_parser.rb:44:in `extract' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_parser.rb:26:in `loop' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_parser.rb:26:in `extract' /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in `receive_data' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:228:in `read_external_socket' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:220:in `handle_external_messages' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:194:in `handle_read_event' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:190:in `each' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:190:in `handle_read_event' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:144:in `start_reactor' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:137:in `loop' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:137:in `start_reactor' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_master.rb:21:in `run' /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in `initialize' script/backgroundrb:46:in `new' script/backgroundrb:46 This doesn't make much sense, and I'm at a loss. Does anybody have any clue what might be going on here? FYI - I'm on the latest (as of this morning) git version of backgroundrb. thanks, -Brent -- Brent Collier | www.acts-as-blogr.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Mon Sep 8 13:46:05 2008 From: gethemant at gmail.com (hemant) Date: Mon, 8 Sep 2008 23:16:05 +0530 Subject: [Backgroundrb-devel] Problems with async worker request In-Reply-To: References: Message-ID: This is one known bug/issue which shows up only on Mac OSX. I haven't had time to chase this thing (mostly because I don't have a Mac machine for testing), but behaviour is, for a newly spawned worker, it takes sometime before worker becomes usable. I am very sorry about this and will try to fix on next possible chance(time), btw this whole thing works flawlessly on Linux. On Mon, Sep 8, 2008 at 9:56 PM, Brent Collier wrote: > Sorry if this comes through twice. I already sent this once, before joining > the mailing list. > > I'm attempting to use Backgroundrb to handle asynchronous pdf creation, but > in doing so, I've run into a very strange problem. Below is a method that's > called from the controller which creates a new worker, then grabs the worker > and calls the 'build_pdf' method asynchronously. > > def make_pdf(template_path, worker_key) > with_empty_asset_id do > html_string = render_to_string(:template => template_path, :layout => > 'pdf') > key = MiddleMan.new_worker(:worker => :prince_xml_worker, :worker_key > => worker_key) > MiddleMan.worker(:prince_xml_worker, key).async_build_pdf(:arg => > html_string) > end > end > > Then there's another method which polls the worker like so. > > def ask_worker_status(worker_key) > MiddleMan.worker(:prince_xml_worker, worker_key).ask_result(:pdf) > end > > It didn't seem to be working properly, so I started doing a little > debugging. Suddenly, it worked! So I removed my debugger statements and > tried again. It stopped working. I kept going back and forth like this, > trying differnet scenarios, poking at the workers, and checking their > results. Finally, I narrowed it down to one debugger statement in the > make_pdf method. > > def make_pdf(template_path, worker_key) > with_empty_asset_id do > html_string = render_to_string(:template => template_path, :layout => > 'pdf') > key = MiddleMan.new_worker(:worker => :prince_xml_worker, :worker_key > => worker_key) > worker = MiddleMan.worker(:prince_xml_worker, key) >>> debugger > worker.async_build_pdf(:arg => html_string) > end > end > > When that debugger statement is in the code, everything works perfectly. > When it hits the debugger during the request, it doesn't matter whether I > poke around at a few objects, or just continue immediately. I even tried > replacing the 'debugger' with a 'sleep(1)' and everything worked perfectly. > When I removed the sleep call, no worky. > > If I look at the backgroundrb_debug_11006 log, I see "Client disconnected" > each time the app polls the worker, and occasionally, I see this: > > Packet::InvalidWorker > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_connection.rb:52:in > `ask_worker' > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:123:in > `get_result_object' > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:39:in > `receive_data' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_parser.rb:44:in > `extract' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_parser.rb:26:in > `loop' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_parser.rb:26:in > `extract' > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in > `receive_data' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:228:in > `read_external_socket' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:220:in > `handle_external_messages' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:194:in > `handle_read_event' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:190:in > `each' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:190:in > `handle_read_event' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:144:in > `start_reactor' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:137:in > `loop' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:137:in > `start_reactor' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_master.rb:21:in > `run' > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in > `initialize' > script/backgroundrb:46:in `new' > script/backgroundrb:46 > > This doesn't make much sense, and I'm at a loss. Does anybody have any clue > what might be going on here? > > FYI - I'm on the latest (as of this morning) git version of backgroundrb. > > thanks, > -Brent > > -- > Brent Collier | www.acts-as-blogr.com > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Mon Sep 8 13:48:12 2008 From: gethemant at gmail.com (hemant) Date: Mon, 8 Sep 2008 23:18:12 +0530 Subject: [Backgroundrb-devel] Backgroundrb incompatible with acts_as_readyonlyable? In-Reply-To: References: Message-ID: Can you send a sample app(tar.gz) thats pre-configured with acts_as_readonlyable, so as I can reproduce this thing? On Mon, Sep 8, 2008 at 2:37 AM, Brent Hargrave wrote: > Hi all, > > I've setup MySQL replication using acts_as_readonlyable. The thin servers > work fine, as does backgroundrb on non-read-only models. However, when it > tries to read an ActiveRecord model that is "acts-as_readonlyable", I get > the following error: ActiveRecord::ConnectionNotEstablished. To reiterate, > I'm pretty sure it's not my configuration because the core rails app works > as expected, as does backgroundrb for non-replicated models. > > Has anyone else experienced this problem, or know what may be the cause of > it? Here's a gist of the backtrace: > http://gist.github.com/9319 > > Note; I'm running rails 2.0.2 and a backgroundrb version from a little after > 2008-02-28 (last changelog update). I have not upgraded since then for > stability reasons. > > Any suggestions would be much appreciated. > > Thanks, > Brent > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From brentmc79 at gmail.com Mon Sep 8 14:02:11 2008 From: brentmc79 at gmail.com (Brent Collier) Date: Mon, 8 Sep 2008 14:02:11 -0400 Subject: [Backgroundrb-devel] Problems with async worker request In-Reply-To: References: Message-ID: Yep, I'm on OSX. Well, I guess that solves it. Thanks for the quick response guys! -Brent On Mon, Sep 8, 2008 at 1:46 PM, hemant wrote: > This is one known bug/issue which shows up only on Mac OSX. I haven't > had time to chase this thing (mostly because I don't have a Mac > machine for testing), but behaviour is, for a newly spawned worker, it > takes sometime before worker becomes usable. > > I am very sorry about this and will try to fix on next possible > chance(time), btw this whole thing works flawlessly on Linux. > > > On Mon, Sep 8, 2008 at 9:56 PM, Brent Collier wrote: > > Sorry if this comes through twice. I already sent this once, before > joining > > the mailing list. > > > > I'm attempting to use Backgroundrb to handle asynchronous pdf creation, > but > > in doing so, I've run into a very strange problem. Below is a method > that's > > called from the controller which creates a new worker, then grabs the > worker > > and calls the 'build_pdf' method asynchronously. > > > > def make_pdf(template_path, worker_key) > > with_empty_asset_id do > > html_string = render_to_string(:template => template_path, :layout > => > > 'pdf') > > key = MiddleMan.new_worker(:worker => :prince_xml_worker, > :worker_key > > => worker_key) > > MiddleMan.worker(:prince_xml_worker, key).async_build_pdf(:arg => > > html_string) > > end > > end > > > > Then there's another method which polls the worker like so. > > > > def ask_worker_status(worker_key) > > MiddleMan.worker(:prince_xml_worker, worker_key).ask_result(:pdf) > > end > > > > It didn't seem to be working properly, so I started doing a little > > debugging. Suddenly, it worked! So I removed my debugger statements and > > tried again. It stopped working. I kept going back and forth like this, > > trying differnet scenarios, poking at the workers, and checking their > > results. Finally, I narrowed it down to one debugger statement in the > > make_pdf method. > > > > def make_pdf(template_path, worker_key) > > with_empty_asset_id do > > html_string = render_to_string(:template => template_path, :layout > => > > 'pdf') > > key = MiddleMan.new_worker(:worker => :prince_xml_worker, > :worker_key > > => worker_key) > > worker = MiddleMan.worker(:prince_xml_worker, key) > >>> debugger > > worker.async_build_pdf(:arg => html_string) > > end > > end > > > > When that debugger statement is in the code, everything works perfectly. > > When it hits the debugger during the request, it doesn't matter whether I > > poke around at a few objects, or just continue immediately. I even tried > > replacing the 'debugger' with a 'sleep(1)' and everything worked > perfectly. > > When I removed the sleep call, no worky. > > > > If I look at the backgroundrb_debug_11006 log, I see "Client > disconnected" > > each time the app polls the worker, and occasionally, I see this: > > > > Packet::InvalidWorker > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_connection.rb:52:in > > `ask_worker' > > > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:123:in > > `get_result_object' > > > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:39:in > > `receive_data' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_parser.rb:44:in > > `extract' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_parser.rb:26:in > > `loop' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_parser.rb:26:in > > `extract' > > > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in > > `receive_data' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:228:in > > `read_external_socket' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:220:in > > `handle_external_messages' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:194:in > > `handle_read_event' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:190:in > > `each' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:190:in > > `handle_read_event' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:144:in > > `start_reactor' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:137:in > > `loop' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_core.rb:137:in > > `start_reactor' > > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.10/lib/packet/packet_master.rb:21:in > > `run' > > > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in > > `initialize' > > script/backgroundrb:46:in `new' > > script/backgroundrb:46 > > > > This doesn't make much sense, and I'm at a loss. Does anybody have any > clue > > what might be going on here? > > > > FYI - I'm on the latest (as of this morning) git version of backgroundrb. > > > > thanks, > > -Brent > > > > -- > > Brent Collier | www.acts-as-blogr.com > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org > -- Brent Collier | www.acts-as-blogr.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ramon.tayag at gmail.com Tue Sep 9 05:38:06 2008 From: ramon.tayag at gmail.com (Ramon Miguel M. Tayag) Date: Tue, 9 Sep 2008 17:38:06 +0800 Subject: [Backgroundrb-devel] add_timer Message-ID: Hi everyone, I found this post about reading more about add_timer from a while back: http://www.mail-archive.com/backgroundrb-devel at rubyforge.org/msg00507.html However... I really can't find it :o I looked at the page and the manual. Also, the "Look at at beginning of the docs" -- is that the manual? Coz I looked there too :o My problem is that I can't seem to get this set_non_winners_as_rejected method to work: http://pastie.org/268758. When I run it, nothing gets written into the bdrb_job_queues table in the database. I also tried typing this straight into the console: >> v = VotingPeriod.first => # >> MiddleMan.worker(:voting_period_worker).async_set_non_winners_as_rejected(:arg => v.id) => nil v.ends_at - Time.now returns the proper time in seconds that I want the timer to execute. What should I do next to debug it? Thanks, -- Ramon Tayag From gethemant at gmail.com Tue Sep 9 07:56:36 2008 From: gethemant at gmail.com (hemant) Date: Tue, 9 Sep 2008 17:26:36 +0530 Subject: [Backgroundrb-devel] add_timer In-Reply-To: References: Message-ID: This should defintely work, did you check if block inside is throwing some errors? Run backgroundrb without daemonizing it and add some puts statements inside the block and see if they are printing. Also, you should check the timer value (how many seconds!) On Tue, Sep 9, 2008 at 3:08 PM, Ramon Miguel M. Tayag wrote: > Hi everyone, > > I found this post about reading more about add_timer from a while > back: http://www.mail-archive.com/backgroundrb-devel at rubyforge.org/msg00507.html > > However... I really can't find it :o I looked at the page and the > manual. Also, the "Look at at beginning of the docs" -- is that the > manual? Coz I looked there too :o > > My problem is that I can't seem to get this > set_non_winners_as_rejected method to work: http://pastie.org/268758. > When I run it, nothing gets written into the bdrb_job_queues table in > the database. I also tried typing this straight into the console: >>> v = VotingPeriod.first > => # "2008-09-09 09:45:00", created_at: "2008-09-08 18:34:31", updated_at: > "2008-09-09 09:29:12", user_id: 1> >>> MiddleMan.worker(:voting_period_worker).async_set_non_winners_as_rejected(:arg => v.id) > => nil > > v.ends_at - Time.now returns the proper time in seconds that I want > the timer to execute. > > What should I do next to debug it? > > Thanks, > -- > Ramon Tayag > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From grant.blakeman at gmail.com Tue Sep 9 15:32:49 2008 From: grant.blakeman at gmail.com (Grant Blakeman) Date: Tue, 9 Sep 2008 13:32:49 -0600 Subject: [Backgroundrb-devel] "Client disconected" error - trying to upgrade to latest version of backgroundrb Message-ID: <51a0cb8c0809091232y6f32d0es795892b84741b907@mail.gmail.com> Hi, I've had a project using a really old version of backgroundrb (don't remember which one) to simply send out some e-mails in the background. I'm currently trying to upgrade the project so that I can learn more about backgroundrb and use some of its other features. I've tried rewriting my worker call to work with the latest version (see below), but every time I try it, the debug log simply prints 'Client disconected' and nothing happens. Am I doing something wrong? In my controller: MiddleMan.worker(:david_worker).async_do_work(:args => { :email => @user.email, :url => @user.reset_password_code, :mail_type => :forgot_password_email }) ------------------- In my worker: set_worker_name :david_worker def create(args = nil) # this method is called, when worker is loaded for the first time end def do_work(args) Notifier.send("deliver_#{args[:mail_type]}", args) end Thanks in advance for any tips! -- -Grant Blakeman grant.blakeman at gmail.com http://grantblakeman.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.hendley at plymouthsystems.com Wed Sep 10 10:00:25 2008 From: sam.hendley at plymouthsystems.com (Sam Hendley) Date: Wed, 10 Sep 2008 10:00:25 -0400 Subject: [Backgroundrb-devel] Invalid worker with name... Message-ID: Hello all, I have been researching background processing options and I think BackgroundDRb is a good fit for our project. I set up a test project to play around and prototype the behavior I want but I ran into the same problem as Reynard. I created a skeleton worker using the generator and I am trying to call a test function on the worker. (basically following the tutorial) Here's the code I'm trying to run: >> MiddleMan.new_worker(:worker => :upload_worker, :worker_key => :test_key) => "work" >> MiddleMan.worker(:upload_worker, :test_key).upload() => nil And heres what I see in the logs: Invalid worker with name upload_worker_test_work and key test_work /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_connection.r b:52:in `ask_worker' /home/sam/bdb_test/vendor/plugins/backgroundrb/server/lib/master_worker. rb:103:in `async_method_invoke' /home/sam/bdb_test/vendor/plugins/backgroundrb/server/lib/master_worker. rb:37:in `receive_data' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:44 :in `extract' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:26 :in `loop' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:26 :in `extract' /home/sam/bdb_test/vendor/plugins/backgroundrb/server/lib/master_worker. rb:32:in `receive_data' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:232: in `read_external_socket' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:222: in `handle_external_messages' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:196: in `handle_read_event' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:192: in `each' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:192: in `handle_read_event' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:146: in `start_reactor' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:139: in `loop' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:139: in `start_reactor' /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_master.rb:21 :in `run' /home/sam/bdb_test/vendor/plugins/backgroundrb/server/lib/master_proxy.r b:14:in `initialize' script/backgroundrb:46:in `new' script/backgroundrb:46 Reynard did you ever solve this problem? It must be something basic I am missing can anyone point me in the right direction? Thanks Sam Hendley -------------- next part -------------- An HTML attachment was scrubbed... URL: From brentmc79 at gmail.com Wed Sep 10 10:23:07 2008 From: brentmc79 at gmail.com (Brent Collier) Date: Wed, 10 Sep 2008 10:23:07 -0400 Subject: [Backgroundrb-devel] Invalid Worker Problem Message-ID: I'm having a problem where whenever I try to call a worker method, something blows up saying that the worker can't be found. In debugging my code, I can see that the worker was found and that the problem doesn't arrise until I attempt to call a method on it, async or not. My code looks like this: MiddleMan.new_worker(:worker => :geocode_worker, :worker_key => "123") sleep(1) # OSX hack! geocode_worker = MiddleMan.worker(:geocode_worker, :worker_key => "123") geocode_worker.async_geocode_address(:arg => data) Querying the geocode_worker in the debugger, I get: # ip="127.0.0.1", port=11006>], @round_robin=[0], @last_polled_time=Wed Sep 10 13:58:03 +0000 2008, @disconnected_connections={}, @backend_connections=[#, @server_port=11006, @mutex=#>]>, @worker_name=:geocode_worker, @tried_connections=[], @worker_key={:worker_key=>"123"}> So it would seem that the worker was found, and it has the proper name and worker_key. But when I call async_geocode_address on the worker, I see this in the backgroundrb log: Invalid worker with name geocode_worker_worker_key123 and key worker_key123 /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_connection.rb:52:in `ask_worker' /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:140:in `method_invoke' /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:41:in `receive_data' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:44:in `extract' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:26:in `loop' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:26:in `extract' /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in `receive_data' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:230:in `read_external_socket' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:222:in `handle_external_messages' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:196:in `handle_read_event' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:192:in `each' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:192:in `handle_read_event' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:146:in `start_reactor' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:139:in `loop' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:139:in `start_reactor' /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_master.rb:21:in `run' /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in `initialize' script/backgroundrb:46:in `new' script/backgroundrb:46 What's up with that first line? The worker name and key are both totally wrong, and I'm not sure why. Does anybody have any idea what's going on here? FYI, I'm on the lates Git code as of a couple of days ago and Packet v0.1.13. -Brent -- Brent Collier | www.BrentCollier.com | www.acts-as-blogr.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.clymer at gmail.com Wed Sep 10 13:34:31 2008 From: michael.clymer at gmail.com (Mike Clymer) Date: Wed, 10 Sep 2008 11:34:31 -0600 Subject: [Backgroundrb-devel] async_some_task vs new_worker & deleting a unique worker Message-ID: <65F516F9-F89E-48F4-A439-5F6A0635E566@gmail.com> Hi All, I am using custom data collection agents to periodically poll a given data source on the internet, for a given user of our application. I would like to create a backgroundrb worker, per type of data source, per user, to kick off the associated data collection agent in it's own background process. I have looked at the docs and I am wondering if using MiddleMan.new_worker would suffice. I could have the worker's "create" method kick off the data collection agent. Is there some additional benefit to moving the agent creation logic to the "async_some_task" worker functionality instead? I would like each worker and associated agent to run in their own process / thread to unload the burden from the Rails application. On a related note, I also need to kill the data collection agent when the user is no longer interested in collecting data from a particular source. I plan to create the worker with a unique worker_key. Will using that worker_key along with the worker name / type in the delete method only kill the worker of that name associated with the given key? I need workers of the same type with different keys to keep processing. By killing the worker, will the data collection agent created by that worker also be destroyed? I hope my questions are clear enough. I greatly appreciate any help. --Mike From alberto.perdomo at gmail.com Thu Sep 11 16:05:36 2008 From: alberto.perdomo at gmail.com (Alberto Perdomo) Date: Thu, 11 Sep 2008 21:05:36 +0100 Subject: [Backgroundrb-devel] persistent_disabled = true but still querying the DB Message-ID: <7247614c0809111305i30baa767r1cffe32cb7a0bc59@mail.gmail.com> Hi, i have set this in my config file: :persistent_disabled: true But backgrounDrb still queries the DB every few seconds. I have also tried setting the following: :persistent_delay: 60 :persistent_disabled: false I would expect it to check now the DB every 60 seconds but it counting the time it actually does it every 5 seconds. I find it somehow annoying since i dont need persistent queues at all... Im using: backgroundrb from trunk packet (0.1.12) chronic (0.2.3) rails (2.1.1) ruby 1.8.6 x64 patchlevel 11 Any ideas? Cheers, Alberto. From gethemant at gmail.com Thu Sep 11 16:25:06 2008 From: gethemant at gmail.com (hemant kumar) Date: Fri, 12 Sep 2008 01:55:06 +0530 Subject: [Backgroundrb-devel] persistent_disabled = true but still querying the DB In-Reply-To: <7247614c0809111305i30baa767r1cffe32cb7a0bc59@mail.gmail.com> References: <7247614c0809111305i30baa767r1cffe32cb7a0bc59@mail.gmail.com> Message-ID: <1221164706.25357.0.camel@shire> You need backgroundrb from git for this. On Thu, 2008-09-11 at 21:05 +0100, Alberto Perdomo wrote: > Hi, > > i have set this in my config file: > > :persistent_disabled: true > > But backgrounDrb still queries the DB every few seconds. > I have also tried setting the following: > > :persistent_delay: 60 > :persistent_disabled: false > > I would expect it to check now the DB every 60 seconds but it counting > the time it actually does it every 5 seconds. > > I find it somehow annoying since i dont need persistent queues at > all... > > Im using: > backgroundrb from trunk > packet (0.1.12) > chronic (0.2.3) > rails (2.1.1) > ruby 1.8.6 x64 patchlevel 11 > > Any ideas? > > Cheers, > Alberto. > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From gethemant at gmail.com Thu Sep 11 16:27:15 2008 From: gethemant at gmail.com (hemant kumar) Date: Fri, 12 Sep 2008 01:57:15 +0530 Subject: [Backgroundrb-devel] Invalid worker with name... In-Reply-To: References: Message-ID: <1221164835.25357.3.camel@shire> Can you post your worker code? There is a known problem with OSX where a newly spawned worker becomes usable only after fraction of second. Although it works okay on Linux. On Wed, 2008-09-10 at 10:00 -0400, Sam Hendley wrote: > Hello all, I have been researching background processing options and I > think BackgroundDRb is a good fit for our project. > > > > I set up a test project to play around and prototype the behavior I > want but I ran into the same problem as Reynard. I created a skeleton > worker using the generator and I am trying to call a test function on > the worker. (basically following the tutorial) > > > > Here?s the code I?m trying to run: > > > > >> MiddleMan.new_worker(:worker => :upload_worker, :worker_key > => :test_key) > > => ?work? > > >> MiddleMan.worker(:upload_worker, :test_key).upload() > > => nil > > > > And heres what I see in the logs: > > Invalid worker with name upload_worker_test_work and key test_work > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_connection.rb:52:in `ask_worker' > > /home/sam/bdb_test/vendor/plugins/backgroundrb/server/lib/master_worker.rb:103:in `async_method_invoke' > > /home/sam/bdb_test/vendor/plugins/backgroundrb/server/lib/master_worker.rb:37:in `receive_data' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:44:in `extract' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:26:in `loop' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:26:in `extract' > > /home/sam/bdb_test/vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in `receive_data' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:232:in `read_external_socket' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:222:in `handle_external_messages' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:196:in `handle_read_event' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:192:in `each' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:192:in `handle_read_event' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:146:in `start_reactor' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:139:in `loop' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:139:in `start_reactor' > > /usr/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_master.rb:21:in `run' > > /home/sam/bdb_test/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in `initialize' > > script/backgroundrb:46:in `new' > > script/backgroundrb:46 > > > > Reynard did you ever solve this problem? It must be something basic I > am missing can anyone point me in the right direction? > > > > Thanks > > Sam Hendley > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From gethemant at gmail.com Thu Sep 11 16:35:35 2008 From: gethemant at gmail.com (hemant kumar) Date: Fri, 12 Sep 2008 02:05:35 +0530 Subject: [Backgroundrb-devel] Invalid Worker Problem In-Reply-To: References: Message-ID: <1221165335.25357.6.camel@shire> Hi, Sorry for late reply, but second call where you are invoking a method in worker should be: geocode_worker = MiddleMan.worker(:geocode_worker,"123") This should fix your problem. On Wed, 2008-09-10 at 10:23 -0400, Brent Collier wrote: > I'm having a problem where whenever I try to call a worker method, > something blows up saying that the worker can't be found. In > debugging my code, I can see that the worker was found and that the > problem doesn't arrise until I attempt to call a method on it, async > or not. My code looks like this: > > MiddleMan.new_worker(:worker => :geocode_worker, :worker_key => "123") > sleep(1) # OSX hack! > geocode_worker = MiddleMan.worker(:geocode_worker, :worker_key => > "123") > geocode_worker.async_geocode_address(:arg => data) > > > Querying the geocode_worker in the debugger, I get: > > # @middle_man=# @request_count=2, @bdrb_servers=[# > ip="127.0.0.1", port=11006>], @round_robin=[0], @last_polled_time=Wed > Sep 10 13:58:03 +0000 2008, @disconnected_connections={}, > @backend_connections=[# @server_ip="127.0.0.1", @connection_status=true, @connection=nil, > @cluster_conn=#, > @server_port=11006, @mutex=#>]>, > @worker_name=:geocode_worker, @tried_connections=[], > @worker_key={:worker_key=>"123"}> > > > So it would seem that the worker was found, and it has the proper name > and worker_key. But when I call async_geocode_address on the worker, > I see this in the backgroundrb log: > > Invalid worker with name geocode_worker_worker_key123 and key > worker_key123 > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_connection.rb:52:in `ask_worker' > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:140:in `method_invoke' > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:41:in `receive_data' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:44:in `extract' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:26:in `loop' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_parser.rb:26:in `extract' > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in `receive_data' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:230:in `read_external_socket' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:222:in `handle_external_messages' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:196:in `handle_read_event' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:192:in `each' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:192:in `handle_read_event' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:146:in `start_reactor' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:139:in `loop' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_core.rb:139:in `start_reactor' > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/packet_master.rb:21:in `run' > /Users/brent/near-time/near-time.net-exp-rescue-princexml-5504/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in `initialize' > script/backgroundrb:46:in `new' > script/backgroundrb:46 > > > What's up with that first line? The worker name and key are both > totally wrong, and I'm not sure why. Does anybody have any idea > what's going on here? > > FYI, I'm on the lates Git code as of a couple of days ago and Packet > v0.1.13. > > -Brent > > > > -- > Brent Collier | www.BrentCollier.com | www.acts-as-blogr.com > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From alberto.perdomo at gmail.com Thu Sep 11 16:50:22 2008 From: alberto.perdomo at gmail.com (Alberto Perdomo) Date: Thu, 11 Sep 2008 21:50:22 +0100 Subject: [Backgroundrb-devel] persistent_disabled = true but still querying the DB In-Reply-To: <1221164706.25357.0.camel@shire> References: <7247614c0809111305i30baa767r1cffe32cb7a0bc59@mail.gmail.com> <1221164706.25357.0.camel@shire> Message-ID: <7247614c0809111350v439c95bye8b8451acbe3af17@mail.gmail.com> thanks! On Thu, Sep 11, 2008 at 9:25 PM, hemant kumar wrote: > You need backgroundrb from git for this. > > > On Thu, 2008-09-11 at 21:05 +0100, Alberto Perdomo wrote: >> Hi, >> >> i have set this in my config file: >> >> :persistent_disabled: true >> >> But backgrounDrb still queries the DB every few seconds. >> I have also tried setting the following: >> >> :persistent_delay: 60 >> :persistent_disabled: false >> >> I would expect it to check now the DB every 60 seconds but it counting >> the time it actually does it every 5 seconds. >> >> I find it somehow annoying since i dont need persistent queues at >> all... >> >> Im using: >> backgroundrb from trunk >> packet (0.1.12) >> chronic (0.2.3) >> rails (2.1.1) >> ruby 1.8.6 x64 patchlevel 11 >> >> Any ideas? >> >> Cheers, >> Alberto. >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > From jmillr at umich.edu Fri Sep 12 12:43:10 2008 From: jmillr at umich.edu (John Miller) Date: Fri, 12 Sep 2008 12:43:10 -0400 Subject: [Backgroundrb-devel] more on leopard problem Message-ID: <61A36466-1189-44B6-BF3D-D7ED72E4051C@umich.edu> All, Just joined, so can't reply to earlier thread regarding the problem with packet on leopard. Here is some data detailing the situation on my machine. I have only one worker defined (in addition to the default log_worker). If I manually "script/backgroundrb start" from my rails root, everything works fine. However, I'm trying to bring this to a production level, using god. If I manually start god with "god -c /path/to/config/background.god" then god will start backgroundrb if it isn't currently running, and everything works fine. The problem occurs when I use launchd to automatically start god on reboot. When that happens, backgroundrb starts, but I get the following error in backgroundrb_debug_11006.log: ------------------------------------------- /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:116:in `exec': No such file or directory - packet_worker_runner 8:7:log_worker:17:/Users/jmillr/rails/ mchanntesting/lib/workers:/Users/jmillr/rails/mchanntesting/script/ load_worker_env (Errno::ENOENT) from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:116:in `fork_and_load' from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:80:in `start_worker' from /Users/jmillr/rails/mchanntesting/vendor/plugins/backgroundrb/ server/lib/master_proxy.rb:16:in `initialize' from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:19:in `run' from /Users/jmillr/rails/mchanntesting/vendor/plugins/backgroundrb/ server/lib/master_proxy.rb:14:in `initialize' from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46:in `new' from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46 /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:116:in `exec': No such file or directory - packet_worker_runner 11:10:symbol_data_worker:4:/Users/jmillr/rails/ mchanntesting/lib/workers:/Users/jmillr/rails/mchanntesting/script/ load_worker_env (Errno::ENOENT) from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:116:in `fork_and_load' from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:68:in `load_workers' from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:63:in `each' from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:63:in `load_workers' from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ packet_master.rb:20:in `run' from /Users/jmillr/rails/mchanntesting/vendor/plugins/backgroundrb/ server/lib/master_proxy.rb:14:in `initialize' from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46:in `new' from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46 Client disconected ------------------------------------------- Neither worker is found. Let me repeat: It all works fine when I manually start background.rb or manually start god which then starts backgroundrb. The problem only occurs when I use launchd to start god. (And, by the way, the problem is the same if I use launchd to autostart backgroundrb; this was before I tried god.) The problem appears to be in packet (on MacOS X 10.5). I can send any of the launch plist, god config, or worker files, but as I've said, it works great if I invoke the system manually. If anyone has any suggestions, I'd be glad to try them out. If anyone can suggest changes to packet to try, I'll make those changes to see if helps. (It's beyond my ken to try to understand what packet is doing...) If I can't get this resolved, I'll have to go on to something else (beanstalkd?) Thanks for any help, John Miller From jonathan.wallace at gmail.com Fri Sep 12 13:29:11 2008 From: jonathan.wallace at gmail.com (Jonathan Wallace) Date: Fri, 12 Sep 2008 13:29:11 -0400 Subject: [Backgroundrb-devel] Capistrano and bdrb? In-Reply-To: References: Message-ID: Your theory does not sound correct to me unless there is some funkiness where monit is resolving the 'current' directory symlink somehow. Maybe its possible a worker died on you? Jonathan On Thu, Sep 4, 2008 at 6:01 AM, Ramon Miguel M. Tayag wrote: > Hey all. I ended up using monit to monitor backgroundrb. It works > great - it starts and restarts it (or so I thought) when specified. > However, just now, I noticed that I was getting an invalid worker name > error when trying to send out mail. > > I did some digging around and saw that my "script/backgroundrb start" > has been on for quite a while (I did a "ps -ef | grep backgroundrb") > and even if I did a "ruby script/backgroundrb stop" it still came out > when I grepped it. > > I came to thinking that maybe when Capistrano changes the symlinks of > the "current" folder to the latest release, the current backgroundrb > script still points to the old release? So when monit tries to > restart backgroundrb after the deploy of code, it's telling the _new_ > script to stop and start, not the old currently running one. > > Is my theory right? If so, how can I fix this? > > To fix this, for now, I rebooted the server. However, this shouldn't > be the case in the future :o > > I'm pasting my backgroundrb.monitrc file just in case (note I'm quite > amateur): http://pastie.org/265841 > -- > Ramon Tayag > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From seth at glyde.com Fri Sep 12 20:04:48 2008 From: seth at glyde.com (Seth Morabito) Date: Fri, 12 Sep 2008 17:04:48 -0700 Subject: [Backgroundrb-devel] Patch submission for BackgrounDRb Message-ID: Hello Hemant, and others! Our team here has been using BackgrounDRb with great success, it's been very useful for us! We're using release version 1.0.3, but we're interested in moving to the git version for the new persistent job queues, which I've been playing around with. We prefer to keep all of our database changes checked into our source repository as migrations, so we have some concerns about how the current git version of backgroundrb creates the queue table through a rake task. I've made a fork of the project on github with a couple of small changes to this, and I'd like to ask you to review them, if you're interested. The fork is here: http://github.com/sethm/backgroundrb/tree/master The changes I've made do the following: * Added a generator that produces a db migration for the queue database table. The generator is run with "./script/generate bdrb_migration". It takes an optional argument, which is passed to the generator as the name to use for the migration class, either CamelCase or snake_case. For example, "./script/generate bdrb_migration MakeQueueDb" will create the file "./db/migrate/20080912164900_make_queue_db.rb". The default name for the migration class is "CreateBackgroundrbQueueTable". * Added a rake task that can be used to call the generator, if desired. "rake backgroundrb:queue_migration" * Removed the queue table creation from the "backgroundrb:setup" rake task. Changed it so that it calls the migration generator instead. This means anyone using the plugin will need to do a rake "db:migrate" before they can start using persistent queues. I'd love feedback on these changes. The migration would be very useful for us -- but would anyone else find it helpful? I'm perfectly happy to make changes or enhancements. Regards, -Seth -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Fri Sep 12 21:13:50 2008 From: gethemant at gmail.com (hemant kumar) Date: Sat, 13 Sep 2008 06:43:50 +0530 Subject: [Backgroundrb-devel] more on leopard problem In-Reply-To: <61A36466-1189-44B6-BF3D-D7ED72E4051C@umich.edu> References: <61A36466-1189-44B6-BF3D-D7ED72E4051C@umich.edu> Message-ID: <1221268430.31161.1.camel@shire> Looks like, PATH is not proper when god is automatically starting it. Check if "packet_worker_runner" script is in PATH.Its a executable, that packet installs when gem gets installed. On Fri, 2008-09-12 at 12:43 -0400, John Miller wrote: > All, > > Just joined, so can't reply to earlier thread regarding the problem > with packet on leopard. Here is some data detailing the situation on > my machine. > > I have only one worker defined (in addition to the default log_worker). > > If I manually "script/backgroundrb start" from my rails root, > everything works fine. > > However, I'm trying to bring this to a production level, using god. > > If I manually start god with "god -c /path/to/config/background.god" > then god will start backgroundrb if it isn't currently running, and > everything works fine. > > The problem occurs when I use launchd to automatically start god on > reboot. When that happens, backgroundrb starts, but I get the > following error in backgroundrb_debug_11006.log: > > ------------------------------------------- > /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:116:in `exec': No such file or directory - > packet_worker_runner 8:7:log_worker:17:/Users/jmillr/rails/ > mchanntesting/lib/workers:/Users/jmillr/rails/mchanntesting/script/ > load_worker_env (Errno::ENOENT) > from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:116:in `fork_and_load' > from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:80:in `start_worker' > from /Users/jmillr/rails/mchanntesting/vendor/plugins/backgroundrb/ > server/lib/master_proxy.rb:16:in `initialize' > from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:19:in `run' > from /Users/jmillr/rails/mchanntesting/vendor/plugins/backgroundrb/ > server/lib/master_proxy.rb:14:in `initialize' > from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46:in `new' > from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46 > /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:116:in `exec': No such file or directory - > packet_worker_runner 11:10:symbol_data_worker:4:/Users/jmillr/rails/ > mchanntesting/lib/workers:/Users/jmillr/rails/mchanntesting/script/ > load_worker_env (Errno::ENOENT) > from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:116:in `fork_and_load' > from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:68:in `load_workers' > from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:63:in `each' > from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:63:in `load_workers' > from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ > packet_master.rb:20:in `run' > from /Users/jmillr/rails/mchanntesting/vendor/plugins/backgroundrb/ > server/lib/master_proxy.rb:14:in `initialize' > from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46:in `new' > from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46 > Client disconected > ------------------------------------------- > > Neither worker is found. > > Let me repeat: It all works fine when I manually start background.rb > or manually start god which then starts backgroundrb. The problem only > occurs when I use launchd to start god. (And, by the way, the problem > is the same if I use launchd to autostart backgroundrb; this was > before I tried god.) > > The problem appears to be in packet (on MacOS X 10.5). > > I can send any of the launch plist, god config, or worker files, but > as I've said, it works great if I invoke the system manually. > > If anyone has any suggestions, I'd be glad to try them out. If anyone > can suggest changes to packet to try, I'll make those changes to see > if helps. (It's beyond my ken to try to understand what packet is > doing...) If I can't get this resolved, I'll have to go on to > something else (beanstalkd?) > > Thanks for any help, > > John Miller > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From epugh at opensourceconnections.com Sat Sep 13 07:03:40 2008 From: epugh at opensourceconnections.com (Eric Pugh) Date: Sat, 13 Sep 2008 07:03:40 -0400 Subject: [Backgroundrb-devel] Patch submission for BackgrounDRb In-Reply-To: References: Message-ID: <1C3F7EC2-3185-4336-9CBD-AA0299719E2A@opensourceconnections.com> Seth, I posted a bug about this exact issue here: http://backgroundrb.devjavu.com/ticket/103 , however you did the "right" thing and actually forked the code and made the fix! Thanks for being a fixer versus just a complainer like myself! At any rate, Hemant, here's my +1 to this as well. Also, using migrations this way will be really nice as we move to 1.0.5 and 1.0.6 and have more db changes! Eric On Sep 12, 2008, at 8:04 PM, Seth Morabito wrote: > Hello Hemant, and others! > > Our team here has been using BackgrounDRb with great success, it's > been very useful for us! We're using release version 1.0.3, but > we're interested in moving to the git version for the new persistent > job queues, which I've been playing around with. > > We prefer to keep all of our database changes checked into our > source repository as migrations, so we have some concerns about how > the current git version of backgroundrb creates the queue table > through a rake task. I've made a fork of the project on github with > a couple of small changes to this, and I'd like to ask you to review > them, if you're interested. The fork is here: http://github.com/sethm/backgroundrb/tree/master > > The changes I've made do the following: > > * Added a generator that produces a db migration for the queue > database table. The generator is run with "./script/generate > bdrb_migration". It takes an optional argument, which is passed to > the generator as the name to use for the migration class, either > CamelCase or snake_case. For example, "./script/generate > bdrb_migration MakeQueueDb" will create the file "./db/migrate/ > 20080912164900_make_queue_db.rb". The default name for the > migration class is "CreateBackgroundrbQueueTable". > > * Added a rake task that can be used to call the generator, if > desired. "rake backgroundrb:queue_migration" > > * Removed the queue table creation from the "backgroundrb:setup" > rake task. Changed it so that it calls the migration generator > instead. This means anyone using the plugin will need to do a rake > "db:migrate" before they can start using persistent queues. > > I'd love feedback on these changes. The migration would be very > useful for us -- but would anyone else find it helpful? I'm > perfectly happy to make changes or enhancements. > > Regards, > > -Seth > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel ----------------------------------------------------- Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rue at thinlayer.co.uk Sat Sep 13 09:03:57 2008 From: rue at thinlayer.co.uk (Rue Turner) Date: Sat, 13 Sep 2008 14:03:57 +0100 Subject: [Backgroundrb-devel] Breaking up complex jobs Message-ID: <200809131403.58538.rue@thinlayer.co.uk> I've got some complex work to do and parts of it are heavy so I want to break it up into smaller jobs. What's the best way for a worker to pass jobs off to other workers? Should we use MiddleMan or enqueue_task() ? I can't find any docs on enqueue_task but if someone will point me in the right direction, I'll gladly write it up and pass it on to Hemant for inclusion when I've got it working. thanks all From gethemant at gmail.com Sat Sep 13 10:30:15 2008 From: gethemant at gmail.com (hemant) Date: Sat, 13 Sep 2008 20:00:15 +0530 Subject: [Backgroundrb-devel] Breaking up complex jobs In-Reply-To: <200809131403.58538.rue@thinlayer.co.uk> References: <200809131403.58538.rue@thinlayer.co.uk> Message-ID: On Sat, Sep 13, 2008 at 6:33 PM, Rue Turner wrote: > > I've got some complex work to do and parts of it are heavy so I want to break > it up into smaller jobs. > > What's the best way for a worker to pass jobs off to other workers? > Should we use MiddleMan or enqueue_task() ? > > I can't find any docs on enqueue_task but if someone will point me in the > right direction, I'll gladly write it up and pass it on to Hemant for > inclusion when I've got it working. > I think, its here: http://backgroundrb.rubyforge.org/workers/ -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Sat Sep 13 10:35:35 2008 From: gethemant at gmail.com (hemant) Date: Sat, 13 Sep 2008 20:05:35 +0530 Subject: [Backgroundrb-devel] Patch submission for BackgrounDRb In-Reply-To: <1C3F7EC2-3185-4336-9CBD-AA0299719E2A@opensourceconnections.com> References: <1C3F7EC2-3185-4336-9CBD-AA0299719E2A@opensourceconnections.com> Message-ID: Hmm, I think, I don't have big time problem with this, I will try to get this merged. On Sat, Sep 13, 2008 at 4:33 PM, Eric Pugh wrote: > Seth, > I posted a bug about this exact issue > here: http://backgroundrb.devjavu.com/ticket/103, however you did the > "right" thing and actually forked the code and made the fix! Thanks for > being a fixer versus just a complainer like myself! > At any rate, Hemant, here's my +1 to this as well. Also, using migrations > this way will be really nice as we move to 1.0.5 and 1.0.6 and have more db > changes! > Eric > On Sep 12, 2008, at 8:04 PM, Seth Morabito wrote: > > Hello Hemant, and others! > Our team here has been using BackgrounDRb with great success, it's been very > useful for us! We're using release version 1.0.3, but we're interested in > moving to the git version for the new persistent job queues, which I've been > playing around with. > We prefer to keep all of our database changes checked into our source > repository as migrations, so we have some concerns about how the current git > version of backgroundrb creates the queue table through a rake task. I've > made a fork of the project on github with a couple of small changes to this, > and I'd like to ask you to review them, if you're interested. The fork is > here: http://github.com/sethm/backgroundrb/tree/master > The changes I've made do the following: > * Added a generator that produces a db migration for the queue database > table. The generator is run with "./script/generate bdrb_migration". It > takes an optional argument, which is passed to the generator as the name to > use for the migration class, either CamelCase or snake_case. For example, > "./script/generate bdrb_migration MakeQueueDb" will create the file > "./db/migrate/20080912164900_make_queue_db.rb". The default name for the > migration class is "CreateBackgroundrbQueueTable". > * Added a rake task that can be used to call the generator, if desired. > "rake backgroundrb:queue_migration" > * Removed the queue table creation from the "backgroundrb:setup" rake task. > Changed it so that it calls the migration generator instead. This means > anyone using the plugin will need to do a rake "db:migrate" before they can > start using persistent queues. > I'd love feedback on these changes. The migration would be very useful for > us -- but would anyone else find it helpful? I'm perfectly happy to make > changes or enhancements. > Regards, > -Seth > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > ----------------------------------------------------- > Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | > http://www.opensourceconnections.com > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From jmillr at umich.edu Sat Sep 13 10:53:42 2008 From: jmillr at umich.edu (John Miller) Date: Sat, 13 Sep 2008 10:53:42 -0400 Subject: [Backgroundrb-devel] more on leopard problem In-Reply-To: <1221268430.31161.1.camel@shire> References: <61A36466-1189-44B6-BF3D-D7ED72E4051C@umich.edu> <1221268430.31161.1.camel@shire> Message-ID: <66DEDC41-9274-451F-96FF-947DA29598E9@umich.edu> Thanks so much for the pointer. Indeed, all I had to do was add the following to the .plist: EnvironmentVariables PATH /usr/local/bin and voila, no problems running the worker! Thanks again, hemant; I'm looking forward to adding more workers as the app develops. John On Sep 12, 2008, at 9:13 PM, hemant kumar wrote: > Looks like, PATH is not proper when god is automatically starting it. > Check if "packet_worker_runner" script is in PATH.Its a executable, > that > packet installs when gem gets installed. > > > On Fri, 2008-09-12 at 12:43 -0400, John Miller wrote: >> All, >> >> Just joined, so can't reply to earlier thread regarding the problem >> with packet on leopard. Here is some data detailing the situation on >> my machine. >> >> I have only one worker defined (in addition to the default >> log_worker). >> >> If I manually "script/backgroundrb start" from my rails root, >> everything works fine. >> >> However, I'm trying to bring this to a production level, using god. >> >> If I manually start god with "god -c /path/to/config/background.god" >> then god will start backgroundrb if it isn't currently running, and >> everything works fine. >> >> The problem occurs when I use launchd to automatically start god on >> reboot. When that happens, backgroundrb starts, but I get the >> following error in backgroundrb_debug_11006.log: >> >> ------------------------------------------- >> /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:116:in `exec': No such file or directory - >> packet_worker_runner 8:7:log_worker:17:/Users/jmillr/rails/ >> mchanntesting/lib/workers:/Users/jmillr/rails/mchanntesting/script/ >> load_worker_env (Errno::ENOENT) >> from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:116:in `fork_and_load' >> from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:80:in `start_worker' >> from /Users/jmillr/rails/mchanntesting/vendor/plugins/backgroundrb/ >> server/lib/master_proxy.rb:16:in `initialize' >> from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:19:in `run' >> from /Users/jmillr/rails/mchanntesting/vendor/plugins/backgroundrb/ >> server/lib/master_proxy.rb:14:in `initialize' >> from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46:in >> `new' >> from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46 >> /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:116:in `exec': No such file or directory - >> packet_worker_runner 11:10:symbol_data_worker:4:/Users/jmillr/rails/ >> mchanntesting/lib/workers:/Users/jmillr/rails/mchanntesting/script/ >> load_worker_env (Errno::ENOENT) >> from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:116:in `fork_and_load' >> from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:68:in `load_workers' >> from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:63:in `each' >> from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:63:in `load_workers' >> from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.13/lib/packet/ >> packet_master.rb:20:in `run' >> from /Users/jmillr/rails/mchanntesting/vendor/plugins/backgroundrb/ >> server/lib/master_proxy.rb:14:in `initialize' >> from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46:in >> `new' >> from /Users/jmillr/rails/mchanntesting/script/backgroundrb:46 >> Client disconected >> ------------------------------------------- >> >> Neither worker is found. >> >> Let me repeat: It all works fine when I manually start background.rb >> or manually start god which then starts backgroundrb. The problem >> only >> occurs when I use launchd to start god. (And, by the way, the problem >> is the same if I use launchd to autostart backgroundrb; this was >> before I tried god.) >> >> The problem appears to be in packet (on MacOS X 10.5). >> >> I can send any of the launch plist, god config, or worker files, but >> as I've said, it works great if I invoke the system manually. >> >> If anyone has any suggestions, I'd be glad to try them out. If anyone >> can suggest changes to packet to try, I'll make those changes to see >> if helps. (It's beyond my ken to try to understand what packet is >> doing...) If I can't get this resolved, I'll have to go on to >> something else (beanstalkd?) >> >> Thanks for any help, >> >> John Miller >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > From Alan.Hayes at onevision.com Sat Sep 13 16:10:23 2008 From: Alan.Hayes at onevision.com (Alan.Hayes at onevision.com) Date: Sat, 13 Sep 2008 16:10:23 -0400 Subject: [Backgroundrb-devel] Determining whether shared code is running in a worker Message-ID: Hello Hemant et al - I have a module which is common to my foreground Rails code and some of my workers. Is there a way to determine whether I am running in a worker? I can't use the session because not all my foreground code has a session; some is run as callbacks to the foreground and has no session. There may have been a question about this in the past, but, if so, I haven't found it. Any suggestions are appreciated. Regards, alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Sat Sep 13 22:22:50 2008 From: gethemant at gmail.com (hemant kumar) Date: Sun, 14 Sep 2008 07:52:50 +0530 Subject: [Backgroundrb-devel] Determining whether shared code is running in a worker In-Reply-To: References: Message-ID: <1221358970.712.4.camel@shire> Sure, There are couple of ways: 1. worker_name method will be available inside your worker code and should be non nil. 2. If inside worker code, you will have access to many worker unique attributes (like thread_pool), you can check for those and make conditions. On Sat, 2008-09-13 at 16:10 -0400, Alan.Hayes at onevision.com wrote: > > Hello Hemant et al - > > I have a module which is common to my foreground Rails code > and some of my workers. Is there a way to determine whether I am > running in a worker? I can't use the session because not all my > foreground code has a session; some is run as callbacks to the > foreground and has no session. There may have been a question about > this in the past, but, if so, I haven't found it. Any suggestions are > appreciated. > > Regards, alan > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From rmatei at gmail.com Sun Sep 14 22:34:55 2008 From: rmatei at gmail.com (Robert Matei) Date: Sun, 14 Sep 2008 19:34:55 -0700 Subject: [Backgroundrb-devel] Best way to run hundreds of concurrent tasks? Message-ID: <31e3a0430809141934k5cdc4605o2073e81fd83202d0@mail.gmail.com> Hi all, I'm trying to figure out to run some asynchronous tasks in a Facebook app. I've got things working, but BRB crashes after a little while, and I'm not sure if my setup is ideal. Here's the scenario: - whenever a user visits my app, I need to fire off a bunch of API calls. - these calls need to start right away, because the user sees a 'loading' screen until they return. - the job is not calculation or memory-intensive on my end, but it does take 15-20 seconds to get all the responses from Facebook. I'm on a fairly small server, so it's important to keep memory use low (and stability high), while still being able to handle thousands of simultaneous visitors. Right now I'm using a single worker, and each visitor generates a new thread in the thread pool. After a little while tasks only get a few calls in before failing with a "Connection reset by peer" error. Eventually my worker just disappears altogether. What's going on? What kind of setup should I be running? Thanks a lot for a great plugin... Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From zol at icyte.com Sun Sep 14 22:39:47 2008 From: zol at icyte.com (Zoltan Olah) Date: Mon, 15 Sep 2008 12:39:47 +1000 Subject: [Backgroundrb-devel] not finding activerecord data in worker Message-ID: Hey guys, Firstly, I'm new to the list and great work with backgroundrb, it's sweet!!! I've just started noticing an issue and i'm wondering if there is a standard fix. The situation is as follows: 1. In my model's after_save I call my worker asynchronously with the active record object's id as a parameter (the record that has just been saved). 2. In my worker, the first thing I do is find the record from the database using the id which has been passed in. What i'm finding is occasionally, in step 2. the worker cannot find the record in the db. Having log_bin set in my mysql configuration seems to drastically increase the frequency of this happening. It is quite clear from the rails logs that the INSERT always happens before the call to the backgroundrb worker. I'm puzzled as I thought the mysql insert query would not return until the inserted data is actually in the database and ready to be retrieved. It seems to be not the case. The current workaround i'm thinking is to put some retrying/sleeping in the backgroundrb worker in order to try to wait it out until the record is there. This seems to me a last resort. Would love to hear what your experiences are and of a better solution if it exists. many thanks, zol From gethemant at gmail.com Mon Sep 15 00:26:32 2008 From: gethemant at gmail.com (hemant kumar) Date: Mon, 15 Sep 2008 09:56:32 +0530 Subject: [Backgroundrb-devel] not finding activerecord data in worker In-Reply-To: References: Message-ID: <1221452792.18545.8.camel@shire> On Mon, 2008-09-15 at 12:39 +1000, Zoltan Olah wrote: > Hey guys, > > Firstly, I'm new to the list and great work with backgroundrb, it's > sweet!!! I've just started noticing an issue and i'm wondering if > there is a standard fix. The situation is as follows: > > 1. In my model's after_save I call my worker asynchronously with the > active record object's id as a parameter (the record that has just > been saved). > 2. In my worker, the first thing I do is find the record from the > database using the id which has been passed in. > > What i'm finding is occasionally, in step 2. the worker cannot find > the record in the db. Having log_bin set in my mysql configuration > seems to drastically increase the frequency of this happening. It is > quite clear from the rails logs that the INSERT always happens before > the call to the backgroundrb worker. > > I'm puzzled as I thought the mysql insert query would not return until > the inserted data is actually in the database and ready to be > retrieved. It seems to be not the case. The current workaround i'm > thinking is to put some retrying/sleeping in the backgroundrb worker > in order to try to wait it out until the record is there. This seems > to me a last resort. Would love to hear what your experiences are and > of a better solution if it exists. > Yes, this issue came up before on the list and mostly went unresolved. I think, this is especially true for databases thats remotely located (I am unable to reproduce this with my local test setup). Currently your best bet is to retry, something like this can be a good idea: def foobar ar_id t = User.find(ar_id) unless t # better than sleep, because your worker can # process other requests while this timer gets ready add_timer(1) { foobar(ar_id) } else # do the normal processing here end end From gethemant at gmail.com Mon Sep 15 00:30:29 2008 From: gethemant at gmail.com (hemant kumar) Date: Mon, 15 Sep 2008 10:00:29 +0530 Subject: [Backgroundrb-devel] Best way to run hundreds of concurrent tasks? In-Reply-To: <31e3a0430809141934k5cdc4605o2073e81fd83202d0@mail.gmail.com> References: <31e3a0430809141934k5cdc4605o2073e81fd83202d0@mail.gmail.com> Message-ID: <1221453029.18545.11.camel@shire> On Sun, 2008-09-14 at 19:34 -0700, Robert Matei wrote: > Hi all, > > I'm trying to figure out to run some asynchronous tasks in a Facebook > app. I've got things working, but BRB crashes after a little while, > and I'm not sure if my setup is ideal. > > Here's the scenario: > - whenever a user visits my app, I need to fire off a bunch of API > calls. > - these calls need to start right away, because the user sees a > 'loading' screen until they return. > - the job is not calculation or memory-intensive on my end, but it > does take 15-20 seconds to get all the responses from Facebook. > > I'm on a fairly small server, so it's important to keep memory use low > (and stability high), while still being able to handle thousands of > simultaneous visitors. Right now I'm using a single worker, and each > visitor generates a new thread in the thread pool. After a little > while tasks only get a few calls in before failing with a "Connection > reset by peer" error. Eventually my worker just disappears > altogether. > > What's going on? What kind of setup should I be running? What version of BDRB? What version of packet gem? Which OS? Are you using mechanize or something for interacting with facebook? or just normal RFacebook or something? From rmatei at gmail.com Mon Sep 15 00:58:57 2008 From: rmatei at gmail.com (Robert Matei) Date: Sun, 14 Sep 2008 21:58:57 -0700 Subject: [Backgroundrb-devel] Best way to run hundreds of concurrent tasks? In-Reply-To: <1221453029.18545.11.camel@shire> References: <31e3a0430809141934k5cdc4605o2073e81fd83202d0@mail.gmail.com> <1221453029.18545.11.camel@shire> Message-ID: <31e3a0430809142158y6a6b4b9ds6369e2e9518b0137@mail.gmail.com> BDRB is the latest version from the SVN repository (supposed to mirror stable git releases) Packet 0.1.13 using Facebooker to interact with Facebook. On Sun, Sep 14, 2008 at 9:30 PM, hemant kumar wrote: > On Sun, 2008-09-14 at 19:34 -0700, Robert Matei wrote: > > Hi all, > > > > I'm trying to figure out to run some asynchronous tasks in a Facebook > > app. I've got things working, but BRB crashes after a little while, > > and I'm not sure if my setup is ideal. > > > > Here's the scenario: > > - whenever a user visits my app, I need to fire off a bunch of API > > calls. > > - these calls need to start right away, because the user sees a > > 'loading' screen until they return. > > - the job is not calculation or memory-intensive on my end, but it > > does take 15-20 seconds to get all the responses from Facebook. > > > > I'm on a fairly small server, so it's important to keep memory use low > > (and stability high), while still being able to handle thousands of > > simultaneous visitors. Right now I'm using a single worker, and each > > visitor generates a new thread in the thread pool. After a little > > while tasks only get a few calls in before failing with a "Connection > > reset by peer" error. Eventually my worker just disappears > > altogether. > > > > What's going on? What kind of setup should I be running? > > What version of BDRB? What version of packet gem? Which OS? > > Are you using mechanize or something for interacting with facebook? or > just normal RFacebook or something? > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zol at icyte.com Mon Sep 15 01:45:13 2008 From: zol at icyte.com (Zoltan Olah) Date: Mon, 15 Sep 2008 15:45:13 +1000 Subject: [Backgroundrb-devel] not finding activerecord data in worker In-Reply-To: <1221452792.18545.8.camel@shire> References: <1221452792.18545.8.camel@shire> Message-ID: <9037E8E1-AF08-4A25-B366-602EA6103F97@icyte.com> Thanks for your reply. After doing some more research, it seems likely that the cause of the problem is after_save being called from inside the transaction which wraps save. After some very short testing, the problem seems fixed by downloading the after_commit plugin from git:// github.com/GUI/after_commit.git . This adds an after_commit AR callback which happens after the save transaction has been commited to the db. Make the async call to backgroundrb from the after_commit callback instead after_save . This ensures that the save transaction has been committed to the db and subsequent selects *should* work barring any mysql bugs :) zol On 15/09/2008, at 2:26 PM, hemant kumar wrote: > On Mon, 2008-09-15 at 12:39 +1000, Zoltan Olah wrote: >> Hey guys, >> >> Firstly, I'm new to the list and great work with backgroundrb, it's >> sweet!!! I've just started noticing an issue and i'm wondering if >> there is a standard fix. The situation is as follows: >> >> 1. In my model's after_save I call my worker asynchronously with the >> active record object's id as a parameter (the record that has just >> been saved). >> 2. In my worker, the first thing I do is find the record from the >> database using the id which has been passed in. >> >> What i'm finding is occasionally, in step 2. the worker cannot find >> the record in the db. Having log_bin set in my mysql configuration >> seems to drastically increase the frequency of this happening. It is >> quite clear from the rails logs that the INSERT always happens before >> the call to the backgroundrb worker. >> >> I'm puzzled as I thought the mysql insert query would not return >> until >> the inserted data is actually in the database and ready to be >> retrieved. It seems to be not the case. The current workaround i'm >> thinking is to put some retrying/sleeping in the backgroundrb worker >> in order to try to wait it out until the record is there. This seems >> to me a last resort. Would love to hear what your experiences are and >> of a better solution if it exists. >> > > Yes, this issue came up before on the list and mostly went unresolved. > I think, this is especially true for databases thats remotely > located (I > am unable to reproduce this with my local test setup). Currently your > best bet is to retry, something like this can be a good idea: > > def foobar ar_id > t = User.find(ar_id) > unless t > # better than sleep, because your worker can > # process other requests while this timer gets ready > add_timer(1) { foobar(ar_id) } > else > # do the normal processing here > end > end > > > From gethemant at gmail.com Mon Sep 15 07:36:00 2008 From: gethemant at gmail.com (hemant kumar) Date: Mon, 15 Sep 2008 17:06:00 +0530 Subject: [Backgroundrb-devel] Breaking up complex jobs In-Reply-To: <200809151040.27233.rue@thinlayer.co.uk> References: <200809131403.58538.rue@thinlayer.co.uk> <200809151040.27233.rue@thinlayer.co.uk> Message-ID: <1221478560.18545.16.camel@shire> MiddleMan object is certainly available in workers.Also, persistent_job is only available in worker, when task has been enqueued through enq_xxx method. Can you show the code, where you are trying to invoke the worker method using enq_xx?? On Mon, 2008-09-15 at 10:40 +0100, Rue Turner wrote: > > I think, its here: > > > > http://backgroundrb.rubyforge.org/workers/ > > Thanks Hermant. I did try those bits first but it seems MiddleMan isn't > available from inside a worker. I also tried various way to enq_ and > enqueue_ but still no luck. > > Also, persistent_job seems to return a nil object when calling 'finish!'. > > source: > http://pastie.org/272591 > log output: > http://pastie.org/272595 > > any help is really appreciated. > > Rue From gethemant at gmail.com Mon Sep 15 07:37:29 2008 From: gethemant at gmail.com (hemant kumar) Date: Mon, 15 Sep 2008 17:07:29 +0530 Subject: [Backgroundrb-devel] Best way to run hundreds of concurrent tasks? In-Reply-To: <31e3a0430809142158y6a6b4b9ds6369e2e9518b0137@mail.gmail.com> References: <31e3a0430809141934k5cdc4605o2073e81fd83202d0@mail.gmail.com> <1221453029.18545.11.camel@shire> <31e3a0430809142158y6a6b4b9ds6369e2e9518b0137@mail.gmail.com> Message-ID: <1221478649.18545.18.camel@shire> Can you paste the worker code? I am not sure, whats causing this. Which OS btw? On Sun, 2008-09-14 at 21:58 -0700, Robert Matei wrote: > BDRB is the latest version from the SVN repository (supposed to mirror > stable git releases) > Packet 0.1.13 > using Facebooker to interact with Facebook. > > > On Sun, Sep 14, 2008 at 9:30 PM, hemant kumar > wrote: > > On Sun, 2008-09-14 at 19:34 -0700, Robert Matei wrote: > > Hi all, > > > > I'm trying to figure out to run some asynchronous tasks in a > Facebook > > app. I've got things working, but BRB crashes after a little > while, > > and I'm not sure if my setup is ideal. > > > > Here's the scenario: > > - whenever a user visits my app, I need to fire off a bunch > of API > > calls. > > - these calls need to start right away, because the user > sees a > > 'loading' screen until they return. > > - the job is not calculation or memory-intensive on my end, > but it > > does take 15-20 seconds to get all the responses from > Facebook. > > > > I'm on a fairly small server, so it's important to keep > memory use low > > (and stability high), while still being able to handle > thousands of > > simultaneous visitors. Right now I'm using a single worker, > and each > > visitor generates a new thread in the thread pool. After a > little > > while tasks only get a few calls in before failing with a > "Connection > > reset by peer" error. Eventually my worker just disappears > > altogether. > > > > What's going on? What kind of setup should I be running? > > > What version of BDRB? What version of packet gem? Which OS? > > Are you using mechanize or something for interacting with > facebook? or > just normal RFacebook or something? > > > > > > From mrryancase at gmail.com Tue Sep 16 12:11:38 2008 From: mrryancase at gmail.com (Ryan Case) Date: Tue, 16 Sep 2008 12:11:38 -0400 Subject: [Backgroundrb-devel] trouble stopping backgroundrb Message-ID: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> Hi folks - I'm having trouble getting backgroundrb to stop after one of the packet_worker_r processes dies. If backgroundrb is running properly, "/path/to/application/script/backgroundrb stop" works fine, but often one of the packet_worker_r processes dies, and the stop command no longer works after that (it runs, but it does not stop the processes, and so then start doesn't work). The only thing that seems to work at that point is to manually kill the processes that are still running, and then the start works, but that is going to make restarting via monit a lot less clean. Any ideas would be much appreciated! I'm using github version of backgroundrb, and packet 0.1.13 running on ubuntu. Thanks! Ryan From jonathan.wallace at gmail.com Wed Sep 17 11:42:35 2008 From: jonathan.wallace at gmail.com (Jonathan Wallace) Date: Wed, 17 Sep 2008 11:42:35 -0400 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> Message-ID: Hi Ryan, I recently ran into the same issue where the backgroundrb process would not respond to ./script/backgroundrb stop command. The pid file was being deleted but the actual process was not being killed. I'm running packet 0.1.12 on gentoo. I'm not exactly sure what conditions put backgroundrb into such a state but I've decided to modify the script/backgroundrb to behave a little differently. My hypothesis is that if one of the Process.kill method calls in script/backgroundrb raises an exception, the pid file is deleted even though the kill signal is never sent. At this point, running starting and stopping backgroundrb never affects the original still running backgroundrb process. There are a couple of reasons that I believe an exception could be raised. Either the Process.getpgid(pid), Process.kill('TERM', pid) or the PRocess.kill('-TERM', pgid) raise an exception or the effective uid of the user running script/backgroundrb stop does not have permission to kill those processes. To fix this, we've removed the Process.getpgid and the two Process.kill's that are sending the TERM signal. Since we've architected our backgroundrb jobs to be persistent and idempotent (a db backed queue written before the feature appeared in bdrb), we'll just use the KILL signal. Thoughts? Thanks, Jonathan On Tue, Sep 16, 2008 at 12:11 PM, Ryan Case wrote: > Hi folks - > > I'm having trouble getting backgroundrb to stop after one of the > packet_worker_r processes dies. > > If backgroundrb is running properly, > "/path/to/application/script/backgroundrb stop" works fine, but often > one of the packet_worker_r processes dies, and the stop command no > longer works after that (it runs, but it does not stop the processes, > and so then start doesn't work). > > The only thing that seems to work at that point is to manually kill > the processes that are still running, and then the start works, but > that is going to make restarting via monit a lot less clean. > > Any ideas would be much appreciated! > > I'm using github version of backgroundrb, and packet 0.1.13 running on ubuntu. > > Thanks! > Ryan > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From joshea at nooked.com Wed Sep 17 12:35:37 2008 From: joshea at nooked.com (John O'Shea) Date: Wed, 17 Sep 2008 17:35:37 +0100 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> Message-ID: <48D131D9.9000703@nooked.com> Jonathan, Glad you raised this, I've been spending some time trying to diagnose this exact same problem. The exception handling code in the "when 'stop'" block (in script/backgroundrb) could definitely could be improved somewhat - check that the process with 'pid' exists before trying to kill it - rescue permission exceptions (Errno::EPERM) - only delete the pid file if the process pid does not still exist (in ensure block) - be a little more verbose to stdout/stderr While we are on the subject of shutdown, - when the backgroundrb process gets a HUP signal does it wait for existing workers to complete any work methods that are executing or is the 'Process.kill('-TERM', pgid)' call intended to make the OS handle this? We use capistrano to deploy our application (stopping and restarting backgroundrb after the rails app has been updated). It would be great if we could have more predictability regarding shutting down backgroundrb (i.e. have the backgroundrb disable the reactor loop in idle workers and wait for all active workers to finish methods, then shutdown"). John. Jonathan Wallace wrote: > Hi Ryan, > > I recently ran into the same issue where the backgroundrb process > would not respond to ./script/backgroundrb stop command. The pid file > was being deleted but the actual process was not being killed. I'm > running packet 0.1.12 on gentoo. > > I'm not exactly sure what conditions put backgroundrb into such a > state but I've decided to modify the script/backgroundrb to behave a > little differently. > > My hypothesis is that if one of the Process.kill method calls in > script/backgroundrb raises an exception, the pid file is deleted even > though the kill signal is never sent. At this point, running starting > and stopping backgroundrb never affects the original still running > backgroundrb process. > > There are a couple of reasons that I believe an exception could be > raised. Either the Process.getpgid(pid), Process.kill('TERM', pid) or > the PRocess.kill('-TERM', pgid) raise an exception or the effective > uid of the user running script/backgroundrb stop does not have > permission to kill those processes. > > To fix this, we've removed the Process.getpgid and the two > Process.kill's that are sending the TERM signal. Since we've > architected our backgroundrb jobs to be persistent and idempotent (a > db backed queue written before the feature appeared in bdrb), we'll > just use the KILL signal. > > Thoughts? > > Thanks, > Jonathan > > On Tue, Sep 16, 2008 at 12:11 PM, Ryan Case wrote: > >> Hi folks - >> >> I'm having trouble getting backgroundrb to stop after one of the >> packet_worker_r processes dies. >> >> If backgroundrb is running properly, >> "/path/to/application/script/backgroundrb stop" works fine, but often >> one of the packet_worker_r processes dies, and the stop command no >> longer works after that (it runs, but it does not stop the processes, >> and so then start doesn't work). >> >> The only thing that seems to work at that point is to manually kill >> the processes that are still running, and then the start works, but >> that is going to make restarting via monit a lot less clean. >> >> Any ideas would be much appreciated! >> >> I'm using github version of backgroundrb, and packet 0.1.13 running on ubuntu. >> >> Thanks! >> Ryan >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> >> > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- John O'Shea, CTO at Nooked www: http://www.nooked.com/ cell: +353 87 992 9959 skype: joshea From gethemant at gmail.com Wed Sep 17 13:06:57 2008 From: gethemant at gmail.com (hemant kumar) Date: Wed, 17 Sep 2008 22:36:57 +0530 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> Message-ID: <1221671217.11714.4.camel@shire> Hi Jonathan & Ryan, Problem is cross platform issues, just sending "KILL" won't work across all the platforms.Also we are sending a Process.kill("-TERM",pgid), because, this will ensure that corresponding workers also get killed. What was the exception, you got when this happened, that can shed some light on the issue. On my ubuntu box, On Wed, 2008-09-17 at 11:42 -0400, Jonathan Wallace wrote: > Hi Ryan, > > I recently ran into the same issue where the backgroundrb process > would not respond to ./script/backgroundrb stop command. The pid file > was being deleted but the actual process was not being killed. I'm > running packet 0.1.12 on gentoo. > > I'm not exactly sure what conditions put backgroundrb into such a > state but I've decided to modify the script/backgroundrb to behave a > little differently. > > My hypothesis is that if one of the Process.kill method calls in > script/backgroundrb raises an exception, the pid file is deleted even > though the kill signal is never sent. At this point, running starting > and stopping backgroundrb never affects the original still running > backgroundrb process. > > There are a couple of reasons that I believe an exception could be > raised. Either the Process.getpgid(pid), Process.kill('TERM', pid) or > the PRocess.kill('-TERM', pgid) raise an exception or the effective > uid of the user running script/backgroundrb stop does not have > permission to kill those processes. > > To fix this, we've removed the Process.getpgid and the two > Process.kill's that are sending the TERM signal. Since we've > architected our backgroundrb jobs to be persistent and idempotent (a > db backed queue written before the feature appeared in bdrb), we'll > just use the KILL signal. > > Thoughts? > > Thanks, > Jonathan > > On Tue, Sep 16, 2008 at 12:11 PM, Ryan Case wrote: > > Hi folks - > > > > I'm having trouble getting backgroundrb to stop after one of the > > packet_worker_r processes dies. > > > > If backgroundrb is running properly, > > "/path/to/application/script/backgroundrb stop" works fine, but often > > one of the packet_worker_r processes dies, and the stop command no > > longer works after that (it runs, but it does not stop the processes, > > and so then start doesn't work). > > > > The only thing that seems to work at that point is to manually kill > > the processes that are still running, and then the start works, but > > that is going to make restarting via monit a lot less clean. > > > > Any ideas would be much appreciated! > > > > I'm using github version of backgroundrb, and packet 0.1.13 running on ubuntu. > > > > Thanks! > > Ryan > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From mark at sabado.com Wed Sep 17 15:04:57 2008 From: mark at sabado.com (Mark Lynn) Date: Wed, 17 Sep 2008 15:04:57 -0400 Subject: [Backgroundrb-devel] ask_result usage Message-ID: <4A150217-E72B-497B-9231-70EF7374B1EA@sabado.com> I am new to Backgroundrb and am struggling to get a progress bar working on a long running job. I have it setup with memcache which is working. The code works when I check status using MiddleMan.worker(:transfer_worker).ask_result(:percent_complete) However, things get messed up when I run a second instance of this task and percent_complete is still 100% from the last run. So, I assume I need the ask_result version that uses the job_key so that I am looking at a separate instance from the cache. I tried result = MiddleMan.ask_result(:worker => :transfer_worker, :job_key => job_key) result[:percent_complete] which is similar to how ask_status was used in the Advanced Rails Recipes book, but this gives undefined method `ask_result' for # Any hints as to what I am screwing up? Thanks! Mark Lynn Sabado Technologies mark at sabado.com From woody at crystalcommerce.com Wed Sep 17 16:14:18 2008 From: woody at crystalcommerce.com (Woody Peterson) Date: Wed, 17 Sep 2008 13:14:18 -0700 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> Message-ID: <4925EB2C-A52F-48BB-83CC-74FDC2B2BDA6@crystalcommerce.com> I too have been having the same issue. Every time I try to restart backgroundrb after an update to our application (about once a day), I have to forcefully kill it myself. However, I haven't been able to reproduce it in a controlled setting. After I kill and start it, it all works ok. I tried killing packet_worker processes (even with -9), but it still shuts down correctly on the stop command. I'll let it run for a while and try tomorrow, but has anyone been able to predictably reproduce the issue? -Woody (debian etch, packet 0.1.10) On Sep 16, 2008, at 9:11 AM, Ryan Case wrote: > Hi folks - > > I'm having trouble getting backgroundrb to stop after one of the > packet_worker_r processes dies. > > If backgroundrb is running properly, > "/path/to/application/script/backgroundrb stop" works fine, but often > one of the packet_worker_r processes dies, and the stop command no > longer works after that (it runs, but it does not stop the processes, > and so then start doesn't work). > > The only thing that seems to work at that point is to manually kill > the processes that are still running, and then the start works, but > that is going to make restarting via monit a lot less clean. > > Any ideas would be much appreciated! > > I'm using github version of backgroundrb, and packet 0.1.13 running > on ubuntu. > > Thanks! > Ryan > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From gethemant at gmail.com Wed Sep 17 16:43:51 2008 From: gethemant at gmail.com (hemant kumar) Date: Thu, 18 Sep 2008 02:13:51 +0530 Subject: [Backgroundrb-devel] ask_result usage In-Reply-To: <4A150217-E72B-497B-9231-70EF7374B1EA@sabado.com> References: <4A150217-E72B-497B-9231-70EF7374B1EA@sabado.com> Message-ID: <1221684231.11714.9.camel@shire> Couple of things: Are you using memcache for session storage? (please, consider using it, if your rails app is already using memcache someway, its pretty cheap). Also, only when you are using memcache for result storage, your results will be available even if worker is processing some task, otherwise, result will be only available at the end of task (or if you are using thread pool). On Wed, 2008-09-17 at 15:04 -0400, Mark Lynn wrote: > I am new to Backgroundrb and am struggling to get a progress bar > working on a long running job. I have it setup with memcache which is > working. The code works when I check status using > > MiddleMan.worker(:transfer_worker).ask_result(:percent_complete) > > However, things get messed up when I run a second instance of this > task and percent_complete is still 100% from the last run. You need to reset the counter at the beginning of the task. > > So, I assume I need the ask_result version that uses the job_key so > that I am looking at a separate instance from the cache. I tried > > result = MiddleMan.ask_result(:worker > => :transfer_worker, :job_key => job_key) > result[:percent_complete] > > which is similar to how ask_status was used in the Advanced Rails > Recipes book, but this gives > > undefined method `ask_result' for # 0x1f496a0> Error is obvious, you can't use ask_result like that. You need to go through the worker proxy only. From gethemant at gmail.com Wed Sep 17 17:47:47 2008 From: gethemant at gmail.com (hemant kumar) Date: Thu, 18 Sep 2008 03:17:47 +0530 Subject: [Backgroundrb-devel] ask_result usage In-Reply-To: References: <4A150217-E72B-497B-9231-70EF7374B1EA@sabado.com> <1221684231.11714.9.camel@shire> Message-ID: <1221688067.16986.1.camel@shire> Here is a small patch that lets you reset memcache data using: MiddleMan.worker(:foo_worker).reset_memcache_result(key,value) diff --git a/lib/backgroundrb/rails_worker_proxy.rb b/lib/backgroundrb/rails_worker_proxy.rb index 0205f07..2b0a080 100644 --- a/lib/backgroundrb/rails_worker_proxy.rb +++ b/lib/backgroundrb/rails_worker_proxy.rb @@ -105,6 +105,13 @@ module BackgrounDRb middle_man.cache[gen_key(options)] end + def reset_memcache_result(job_key,value) + options = compact(:worker => worker_name,:worker_key => worker_key,:job_key => job_key) + key = gen_key(options) + middle_man.cache[key] = value + value + end + def return_result result result = Array(result) result.size <= 1 ? result[0] : result I will be commiting this to git, let me know,how it goes. On Wed, 2008-09-17 at 17:14 -0400, Mark Lynn wrote: > Thanks for the reply. > > I was using memcache for session storage, but stopped because we had > some special needs for session expiration. > > As to my issue, I am already resetting the counter at the beginning of > the backgroundrb task. However, this does not work because the rails > process often reaches the place where it reads :percent_complete > before the backgroundrb process has reset it. I see the memcached > "get" before I see any "set" operations. Is there a backgroundrb > method to clear the memcache value before starting the worker? I can > clear the memcache directly from Rails, but that means I am building > some assumptions about the memcache namespace being used into my code. > > I knew that the second ask_result was not correct? I don't understand > the difference between the ask_result(p_data) method and the > ask_result(job_key) method. I am trying to get up to speed without > working through too much of the backgroundrb source code. I started > with the Advanced Rails Recipes book which is apparently totally > outdated and have looked through some of your code. I very much > appreciate your efforts on backgroundrb, but I think users could > really benefit from some examples - perhaps an updated version of the > Advanced Rails Recipe chapter. > > Thanks again. > > - Mark > > On Sep 17, 2008, at 4:43 PM, hemant kumar wrote: > > > Couple of things: > > > > Are you using memcache for session storage? (please, consider using > > it, > > if your rails app is already using memcache someway, its pretty > > cheap). > > Also, only when you are using memcache for result storage, your > > results > > will be available even if worker is processing some task, otherwise, > > result will be only available at the end of task (or if you are using > > thread pool). > > > > > > On Wed, 2008-09-17 at 15:04 -0400, Mark Lynn wrote: > >> I am new to Backgroundrb and am struggling to get a progress bar > >> working on a long running job. I have it setup with memcache which is > >> working. The code works when I check status using > >> > >> MiddleMan.worker(:transfer_worker).ask_result(:percent_complete) > >> > >> However, things get messed up when I run a second instance of this > >> task and percent_complete is still 100% from the last run. > > > > You need to reset the counter at the beginning of the task. > > > >> > >> So, I assume I need the ask_result version that uses the job_key so > >> that I am looking at a separate instance from the cache. I tried > >> > >> result = MiddleMan.ask_result(:worker > >> => :transfer_worker, :job_key => job_key) > >> result[:percent_complete] > >> > >> which is similar to how ask_status was used in the Advanced Rails > >> Recipes book, but this gives > >> > >> undefined method `ask_result' for # >> 0x1f496a0> > > > > Error is obvious, you can't use ask_result like that. You need to go > > through the worker proxy only. > > > > > > > > > > Mark Lynn > Sabado Technologies > mark at sabado.com > 616-935-9910 > > > From mrryancase at gmail.com Wed Sep 17 17:48:22 2008 From: mrryancase at gmail.com (Ryan Case) Date: Wed, 17 Sep 2008 17:48:22 -0400 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: <4925EB2C-A52F-48BB-83CC-74FDC2B2BDA6@crystalcommerce.com> References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> <4925EB2C-A52F-48BB-83CC-74FDC2B2BDA6@crystalcommerce.com> Message-ID: <983e153a0809171448w53eeb3a7vc00bda065812180e@mail.gmail.com> I'm not able to reproduce the issue consistently. Often killing (-9) a packet_worker will create the issue, but not always. Jonathan - thanks for the info! skipping the pgid sounds interesting, since what I do to manually fix it is usually just kill the pid for the backgroundrb process (however, as Hemant mentioned, I guess that might leave workers still running). And it definitely seems like it is failing on the kill block and deleting the pid file even tho the process is still running. Hemant - I don't see too much as far as exceptions when I run into this issue. The debug log does show the "Address already in use - bind(2) (Errno::EADDRINUSE)" error when start tries to run, but I don't see exceptions for the stop in the log. When I run stop however, I do get the "Deleting pid file" output, which looks like Errno::ESRCH is being rescued. (Not sure if that is correct, or if there is a way to see more detail on the exception...) Thanks everyone, Ryan On Wed, Sep 17, 2008 at 4:14 PM, Woody Peterson wrote: > I too have been having the same issue. Every time I try to restart > backgroundrb after an update to our application (about once a day), I have > to forcefully kill it myself. However, I haven't been able to reproduce it > in a controlled setting. After I kill and start it, it all works ok. I tried > killing packet_worker processes (even with -9), but it still shuts down > correctly on the stop command. I'll let it run for a while and try tomorrow, > but has anyone been able to predictably reproduce the issue? > > -Woody > > (debian etch, packet 0.1.10) > > On Sep 16, 2008, at 9:11 AM, Ryan Case wrote: > >> Hi folks - >> >> I'm having trouble getting backgroundrb to stop after one of the >> packet_worker_r processes dies. >> >> If backgroundrb is running properly, >> "/path/to/application/script/backgroundrb stop" works fine, but often >> one of the packet_worker_r processes dies, and the stop command no >> longer works after that (it runs, but it does not stop the processes, >> and so then start doesn't work). >> >> The only thing that seems to work at that point is to manually kill >> the processes that are still running, and then the start works, but >> that is going to make restarting via monit a lot less clean. >> >> Any ideas would be much appreciated! >> >> I'm using github version of backgroundrb, and packet 0.1.13 running on >> ubuntu. >> >> Thanks! >> Ryan >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From funnie_16 at hotmail.com Wed Sep 17 18:05:04 2008 From: funnie_16 at hotmail.com (Jochen D.) Date: Thu, 18 Sep 2008 00:05:04 +0200 Subject: [Backgroundrb-devel] ask_result usage In-Reply-To: <4A150217-E72B-497B-9231-70EF7374B1EA@sabado.com> References: <4A150217-E72B-497B-9231-70EF7374B1EA@sabado.com> Message-ID: Try usingkey = MiddleMan.new_worker(:worker => :transfer_worker, :worker_key => random_key)sleep 0.3 #hack if you're running OS XMiddleMan.worker(:transfer_worker, key).method(:arg => args)Then use MiddleMan.worker(:transfer_worker, key).ask_result> From: mark at sabado.com> To: backgroundrb-devel at rubyforge.org> Date: Wed, 17 Sep 2008 15:04:57 -0400> Subject: [Backgroundrb-devel] ask_result usage> > > I am new to Backgroundrb and am struggling to get a progress bar > working on a long running job. I have it setup with memcache which is > working. The code works when I check status using> > MiddleMan.worker(:transfer_worker).ask_result(:percent_complete)> > However, things get messed up when I run a second instance of this > task and percent_complete is still 100% from the last run.> > So, I assume I need the ask_result version that uses the job_key so > that I am looking at a separate instance from the cache. I tried> > result = MiddleMan.ask_result(:worker > => :transfer_worker, :job_key => job_key)> result[:percent_complete]> > which is similar to how ask_status was used in the Advanced Rails > Recipes book, but this gives> > undefined method `ask_result' for # 0x1f496a0>> > Any hints as to what I am screwing up? Thanks!> > > Mark Lynn> Sabado Technologies> mark at sabado.com> > > > > _______________________________________________> Backgroundrb-devel mailing list> Backgroundrb-devel at rubyforge.org> http://rubyforge.org/mailman/listinfo/backgroundrb-devel _________________________________________________________________ Start het nieuwe schooljaar sterk en voordelig! http://www.startsterk.be From gethemant at gmail.com Wed Sep 17 18:08:52 2008 From: gethemant at gmail.com (hemant kumar) Date: Thu, 18 Sep 2008 03:38:52 +0530 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: <48D131D9.9000703@nooked.com> References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> <48D131D9.9000703@nooked.com> Message-ID: <1221689332.16986.5.camel@shire> Okay folks here is a patch to "backgroundrb" script, which should fix some issues: diff --git a/script/backgroundrb b/script/backgroundrb index dabf80b..8d4bb78 100755 --- a/script/backgroundrb +++ b/script/backgroundrb @@ -49,18 +49,9 @@ when 'stop' def kill_process arg_pid_file pid = nil File.open(arg_pid_file, "r") { |pid_handle| pid = pid_handle.gets.strip.chomp.to_i } - begin - pgid = Process.getpgid(pid) - Process.kill('TERM', pid) - Process.kill('-TERM', pgid) - Process.kill('KILL', pid) - rescue Errno::ESRCH => e - puts "Deleting pid file" - rescue - puts $! - ensure - File.delete(arg_pid_file) if File.exists?(arg_pid_file) - end + pgid = Process.getpgid(pid) + Process.kill('-TERM', pgid) + File.delete(arg_pid_file) if File.exists?(arg_pid_file) end pid_files = Dir["#{RAILS_HOME}/tmp/pids/backgroundrb_*.pid"] pid_files.each { |x| kill_process(x) } What it does is: 1. Deleting by group id is enough for master process. 2. Do not delete the pid file if, there was an exception while stopping the daemon. 3. Do not handle exceptions silently. Please try this and let me know, how it goes. On Wed, 2008-09-17 at 17:35 +0100, John O'Shea wrote: > Jonathan, > Glad you raised this, I've been spending some time trying to > diagnose this exact same problem. > The exception handling code in the "when 'stop'" block (in > script/backgroundrb) could definitely could be improved somewhat > - check that the process with 'pid' exists before trying to kill it > - rescue permission exceptions (Errno::EPERM) > - only delete the pid file if the process pid does not still exist (in > ensure block) > - be a little more verbose to stdout/stderr > > While we are on the subject of shutdown, - when the backgroundrb process > gets a HUP signal does it wait for existing workers to complete any work > methods that are executing or is the 'Process.kill('-TERM', pgid)' call > intended to make the OS handle this? > > We use capistrano to deploy our application (stopping and restarting > backgroundrb after the rails app has been updated). It would be great > if we could have more predictability regarding shutting down > backgroundrb (i.e. have the backgroundrb disable the reactor loop in > idle workers and wait for all active workers to finish methods, then > shutdown"). > > John. > > Jonathan Wallace wrote: > > Hi Ryan, > > > > I recently ran into the same issue where the backgroundrb process > > would not respond to ./script/backgroundrb stop command. The pid file > > was being deleted but the actual process was not being killed. I'm > > running packet 0.1.12 on gentoo. > > > > I'm not exactly sure what conditions put backgroundrb into such a > > state but I've decided to modify the script/backgroundrb to behave a > > little differently. > > > > My hypothesis is that if one of the Process.kill method calls in > > script/backgroundrb raises an exception, the pid file is deleted even > > though the kill signal is never sent. At this point, running starting > > and stopping backgroundrb never affects the original still running > > backgroundrb process. > > > > There are a couple of reasons that I believe an exception could be > > raised. Either the Process.getpgid(pid), Process.kill('TERM', pid) or > > the PRocess.kill('-TERM', pgid) raise an exception or the effective > > uid of the user running script/backgroundrb stop does not have > > permission to kill those processes. > > > > To fix this, we've removed the Process.getpgid and the two > > Process.kill's that are sending the TERM signal. Since we've > > architected our backgroundrb jobs to be persistent and idempotent (a > > db backed queue written before the feature appeared in bdrb), we'll > > just use the KILL signal. > > > > Thoughts? > > > > Thanks, > > Jonathan > > > > On Tue, Sep 16, 2008 at 12:11 PM, Ryan Case wrote: > > > >> Hi folks - > >> > >> I'm having trouble getting backgroundrb to stop after one of the > >> packet_worker_r processes dies. > >> > >> If backgroundrb is running properly, > >> "/path/to/application/script/backgroundrb stop" works fine, but often > >> one of the packet_worker_r processes dies, and the stop command no > >> longer works after that (it runs, but it does not stop the processes, > >> and so then start doesn't work). > >> > >> The only thing that seems to work at that point is to manually kill > >> the processes that are still running, and then the start works, but > >> that is going to make restarting via monit a lot less clean. > >> > >> Any ideas would be much appreciated! > >> > >> I'm using github version of backgroundrb, and packet 0.1.13 running on ubuntu. > >> > >> Thanks! > >> Ryan > >> _______________________________________________ > >> Backgroundrb-devel mailing list > >> Backgroundrb-devel at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > >> > >> > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > From sweiss at stylesight.com Wed Sep 17 18:03:11 2008 From: sweiss at stylesight.com (Stephen Weiss) Date: Wed, 17 Sep 2008 18:03:11 -0400 Subject: [Backgroundrb-devel] Backgroundrb-devel Digest, Vol 28, Issue 9 In-Reply-To: References: Message-ID: <4E34D20B-58C6-4533-A77C-17E33A6A367F@stylesight.com> > > > From: Woody Peterson > Date: September 17, 2008 4:14:18 PM EDT > To: backgroundrb-devel at rubyforge.org > Subject: Re: [Backgroundrb-devel] trouble stopping backgroundrb > > > I too have been having the same issue. Every time I try to restart > backgroundrb after an update to our application (about once a day), > I have to forcefully kill it myself. However, I haven't been able to > reproduce it in a controlled setting. After I kill and start it, it > all works ok. I tried killing packet_worker processes (even with > -9), but it still shuts down correctly on the stop command. I'll let > it run for a while and try tomorrow, but has anyone been able to > predictably reproduce the issue? > > -Woody > > (debian etch, packet 0.1.10) > Me three, on OS X Server 10.4.11. Always just figured it was because my backgroundrb was a month old but I can see the issue hasn't been resolved yet (so good thing I didn't upgrade to fix it!). If I go in and turn it off myself, everything works just fine maybe 80% of the time, the other 20% of the time backgroundrb stops but the workers keep running (and therefore my StartDaemon errors out and suddenly there's no background processing anymore). Then I have to go in and kill them off one by one until there's no more ruby in my process list (super-unfortunately, I can't even do a killall ruby - something about the way the name of the process is registered, but it would be far more convenient because these are the only ruby programs on these servers that get run directly and not through lighty). My codebase is relatively stable (changes maybe once a month, if that often), so this is not a huge deal. If the server dies on its own for whatever reason (after a few weeks of uptime, which is downright commendable by the way - nice job guys, with previous versions I'd be lucky to get a day of uptime - it starts to freak out a little bit), it almost always leaves the workers sitting there running, and I have to go through this whole process then as well. This is double bad because if they don't terminate cleanly, they leave junk laying behind in my DB queue and then people have to resubmit their jobs. I mean, I can usually pull the list of jobs from the rails logs (just re-run the URL's that trigger them again) but it's a bit of a pain. All in all though, nothing close to the problems I've had with previous versions of backgroundrb. I'd take this issue any day. -- Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From miriam at nite.org.il Thu Sep 18 07:09:55 2008 From: miriam at nite.org.il (Miriam Lavian) Date: Thu, 18 Sep 2008 13:09:55 +0200 Subject: [Backgroundrb-devel] Problem in running BackgroundRB server Message-ID: <057C9E541CD75D4E8B0B10D3D6AD242548E622@jasper.nite.org.il> Hello everyone, Lately I updated the backgroundrb version to 1.0.4 I run it from the command line: ./script/backgroundrb start And I can see in the processes running, I also defined the environment: production in backgroundrb.yml file, But from the server I get the following error: BackgrounDRb::NoServerAvailable (No BackgrounDRb server is found running): Do you have any idea? Thanks Miriam -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshea at nooked.com Thu Sep 18 06:21:00 2008 From: joshea at nooked.com (John O'Shea) Date: Thu, 18 Sep 2008 11:21:00 +0100 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: <1221689332.16986.5.camel@shire> References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> <48D131D9.9000703@nooked.com> <1221689332.16986.5.camel@shire> Message-ID: <48D22B8C.50507@nooked.com> Slight variation that - deletes pid for already-gone processes - exits (with errror code -1) without deleting the pid file if there was a permission problem begin - pgid = Process.getpgid(pid) - Process.kill('TERM', pid) - Process.kill('-TERM', pgid) - Process.kill('KILL', pid) - rescue Errno::ESRCH => e - puts "Deleting pid file" - rescue + pgid = Process.getpgid(pid) + Process.kill('-TERM', pgid) + rescue Errno::ESRCH + puts $! + # No process - Do nothing. + rescue Errno::EPERM + # Permission denied. + puts $! + Process.exit! ensure File.delete(arg_pid_file) if File.exists?(arg_pid_file) end hemant kumar wrote: > Okay folks here is a patch to "backgroundrb" script, which should fix > some issues: > > diff --git a/script/backgroundrb b/script/backgroundrb > index dabf80b..8d4bb78 100755 > --- a/script/backgroundrb > +++ b/script/backgroundrb > @@ -49,18 +49,9 @@ when 'stop' > def kill_process arg_pid_file > pid = nil > File.open(arg_pid_file, "r") { |pid_handle| pid = > pid_handle.gets.strip.chomp.to_i } > - begin > - pgid = Process.getpgid(pid) > - Process.kill('TERM', pid) > - Process.kill('-TERM', pgid) > - Process.kill('KILL', pid) > - rescue Errno::ESRCH => e > - puts "Deleting pid file" > - rescue > - puts $! > - ensure > - File.delete(arg_pid_file) if File.exists?(arg_pid_file) > - end > + pgid = Process.getpgid(pid) > + Process.kill('-TERM', pgid) > + File.delete(arg_pid_file) if File.exists?(arg_pid_file) > end > pid_files = Dir["#{RAILS_HOME}/tmp/pids/backgroundrb_*.pid"] > pid_files.each { |x| kill_process(x) } > > What it does is: > 1. Deleting by group id is enough for master process. > 2. Do not delete the pid file if, there was an exception while stopping > the daemon. > 3. Do not handle exceptions silently. > > Please try this and let me know, how it goes. > > > > On Wed, 2008-09-17 at 17:35 +0100, John O'Shea wrote: > >> Jonathan, >> Glad you raised this, I've been spending some time trying to >> diagnose this exact same problem. >> The exception handling code in the "when 'stop'" block (in >> script/backgroundrb) could definitely could be improved somewhat >> - check that the process with 'pid' exists before trying to kill it >> - rescue permission exceptions (Errno::EPERM) >> - only delete the pid file if the process pid does not still exist (in >> ensure block) >> - be a little more verbose to stdout/stderr >> >> While we are on the subject of shutdown, - when the backgroundrb process >> gets a HUP signal does it wait for existing workers to complete any work >> methods that are executing or is the 'Process.kill('-TERM', pgid)' call >> intended to make the OS handle this? >> >> We use capistrano to deploy our application (stopping and restarting >> backgroundrb after the rails app has been updated). It would be great >> if we could have more predictability regarding shutting down >> backgroundrb (i.e. have the backgroundrb disable the reactor loop in >> idle workers and wait for all active workers to finish methods, then >> shutdown"). >> >> John. >> >> Jonathan Wallace wrote: >> >>> Hi Ryan, >>> >>> I recently ran into the same issue where the backgroundrb process >>> would not respond to ./script/backgroundrb stop command. The pid file >>> was being deleted but the actual process was not being killed. I'm >>> running packet 0.1.12 on gentoo. >>> >>> I'm not exactly sure what conditions put backgroundrb into such a >>> state but I've decided to modify the script/backgroundrb to behave a >>> little differently. >>> >>> My hypothesis is that if one of the Process.kill method calls in >>> script/backgroundrb raises an exception, the pid file is deleted even >>> though the kill signal is never sent. At this point, running starting >>> and stopping backgroundrb never affects the original still running >>> backgroundrb process. >>> >>> There are a couple of reasons that I believe an exception could be >>> raised. Either the Process.getpgid(pid), Process.kill('TERM', pid) or >>> the PRocess.kill('-TERM', pgid) raise an exception or the effective >>> uid of the user running script/backgroundrb stop does not have >>> permission to kill those processes. >>> >>> To fix this, we've removed the Process.getpgid and the two >>> Process.kill's that are sending the TERM signal. Since we've >>> architected our backgroundrb jobs to be persistent and idempotent (a >>> db backed queue written before the feature appeared in bdrb), we'll >>> just use the KILL signal. >>> >>> Thoughts? >>> >>> Thanks, >>> Jonathan >>> >>> On Tue, Sep 16, 2008 at 12:11 PM, Ryan Case wrote: >>> >>> >>>> Hi folks - >>>> >>>> I'm having trouble getting backgroundrb to stop after one of the >>>> packet_worker_r processes dies. >>>> >>>> If backgroundrb is running properly, >>>> "/path/to/application/script/backgroundrb stop" works fine, but often >>>> one of the packet_worker_r processes dies, and the stop command no >>>> longer works after that (it runs, but it does not stop the processes, >>>> and so then start doesn't work). >>>> >>>> The only thing that seems to work at that point is to manually kill >>>> the processes that are still running, and then the start works, but >>>> that is going to make restarting via monit a lot less clean. >>>> >>>> Any ideas would be much appreciated! >>>> >>>> I'm using github version of backgroundrb, and packet 0.1.13 running on ubuntu. >>>> >>>> Thanks! >>>> Ryan >>>> _______________________________________________ >>>> Backgroundrb-devel mailing list >>>> Backgroundrb-devel at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>>> >>>> >>>> >>> _______________________________________________ >>> Backgroundrb-devel mailing list >>> Backgroundrb-devel at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >>> >> > > -- John O'Shea, CTO at Nooked www: http://www.nooked.com/ cell: +353 87 992 9959 skype: joshea From grant.blakeman at gmail.com Thu Sep 18 13:06:58 2008 From: grant.blakeman at gmail.com (Grant Blakeman) Date: Thu, 18 Sep 2008 11:06:58 -0600 Subject: [Backgroundrb-devel] "Client disconected" error - trying to upgrade to latest version of backgroundrb Message-ID: <51a0cb8c0809181006q262b3e98ub556f59dae7ef6f8@mail.gmail.com> Hi, I've had a project using a really old version of backgroundrb (don't remember which one) to simply send out some e-mails in the background. I'm currently trying to upgrade the project so that I can learn more about backgroundrb and use some of its other features. I've tried rewriting my worker call to work with the latest version (see below), but every time I try it, the debug log simply prints 'Client disconected' and nothing happens. Am I doing something wrong? In my controller: MiddleMan.worker(:david_worker).async_do_work(:args => { :email => @user.email, :url => @user.reset_password_code, :mail_type => :forgot_password_email }) ------------------- In my worker: set_worker_name :david_worker def create(args = nil) # this method is called, when worker is loaded for the first time end def do_work(args) Notifier.send("deliver_#{args[:mail_type]}", args) end Thanks in advance for any tips! -- -Grant Blakeman grant.blakeman at gmail.com http://grantblakeman.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From grant.blakeman at gmail.com Thu Sep 18 13:52:41 2008 From: grant.blakeman at gmail.com (Grant Blakeman) Date: Thu, 18 Sep 2008 11:52:41 -0600 Subject: [Backgroundrb-devel] Client disconected error - trying to upgrade to latest version of backgroundrb Message-ID: <51a0cb8c0809181052t1825b5c9s77a769a7e08ad76e@mail.gmail.com> Hi, I've had a project using a really old version of backgroundrb (don't remember which one) to simply send out some e-mails in the background. I'm currently trying to upgrade the project so that I can learn more about backgroundrb and use some of its other features. I've tried rewriting my worker call to work with the latest version (see below), but every time I try it, the debug log simply prints 'Client disconected' and nothing happens. Am I doing something wrong? In my controller: MiddleMan.worker(:david_worker).async_do_work(:args => { :email => @user.email, :url => @user.reset_password_code, :mail_type => :forgot_password_email }) ------------------- In my worker: set_worker_name :david_worker def create(args = nil) # this method is called, when worker is loaded for the first time end def do_work(args) Notifier.send("deliver_#{args[:mail_type]}", args) end Thanks in advance for any tips! -- -Grant Blakeman grant.blakeman at gmail.com http://grantblakeman.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Thu Sep 18 14:22:26 2008 From: gethemant at gmail.com (hemant) Date: Thu, 18 Sep 2008 23:52:26 +0530 Subject: [Backgroundrb-devel] Client disconected error - trying to upgrade to latest version of backgroundrb In-Reply-To: <51a0cb8c0809181052t1825b5c9s77a769a7e08ad76e@mail.gmail.com> References: <51a0cb8c0809181052t1825b5c9s77a769a7e08ad76e@mail.gmail.com> Message-ID: To pass arguments you use :arg not :args. On Thu, Sep 18, 2008 at 11:22 PM, Grant Blakeman wrote: > Hi, I've had a project using a really old version of backgroundrb (don't > remember which one) to simply send out some e-mails in the background. I'm > currently trying to upgrade the project so that I can learn more about > backgroundrb and use some of its other features. > I've tried rewriting my worker call to work with the latest version (see > below), but every time I try it, the debug log simply prints 'Client > disconected' and nothing happens. Am I doing something wrong? > In my controller: > MiddleMan.worker(:david_worker).async_do_work(:args => { :email => > @user.email, > :url => @user.reset_password_code, > :mail_type => :forgot_password_email }) > ------------------- > In my worker: > set_worker_name :david_worker > > def create(args = nil) > # this method is called, when worker is loaded for the first time > end > > def do_work(args) > Notifier.send("deliver_#{args[:mail_type]}", args) > end > Thanks in advance for any tips! > -- > -Grant Blakeman > grant.blakeman at gmail.com > http://grantblakeman.com > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From woody at crystalcommerce.com Thu Sep 18 14:24:44 2008 From: woody at crystalcommerce.com (Woody Peterson) Date: Thu, 18 Sep 2008 11:24:44 -0700 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: <48D22B8C.50507@nooked.com> References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> <48D131D9.9000703@nooked.com> <1221689332.16986.5.camel@shire> <48D22B8C.50507@nooked.com> Message-ID: In my particular case I know it's not a permissions issue, as I'm always using the same user. I just tried restarting it, and with Hemant's patch I got: script/backgroundrb:52:in `getpgid': No such process (Errno::ESRCH) Via the above I found that in this particular case what happened is that my logrotate wasn't calling stop, only start (it meant to call stop, but was in a failing if statement checking if the pid existed). When you call start, it doesn't check to see if it's already running, so it starts backgroundrb, overwrites the pid file, then backgroundrb fails to start but has had it's pid file changed. The original process is still running, but can't stop because it doesn't have the correct pid in the pid file. Thus, I rewrote script/backgroundrb to be more LSB compliant (http://refspecs.linux-foundation.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html ) so I don't have to check for existing pid files myself. I made a patch, but it's almost as big as the script itself and Hemants patch didn't apply for me (I must have changed something earlier in the file), so the whole thing is at the end of the email. While we're on the topic, is there a place to load all the requirements other than this file? backgroundrb status takes a matter of seconds to do a simple File.exists?(pid) 'cuz it has to load all the backgroundrb requirements. Not that it really matters... -Woody #!/usr/bin/env ruby RAILS_HOME = File.expand_path(File.join(File.dirname(__FILE__),"..")) BDRB_HOME = File.join(RAILS_HOME,"vendor","plugins","backgroundrb") WORKER_ROOT = File.join(RAILS_HOME,"lib","workers") WORKER_LOAD_ENV = File.join(RAILS_HOME,"script","load_worker_env") ["server","server/lib","lib","lib/backgroundrb"].each { |x| $LOAD_PATH.unshift(BDRB_HOME + "/#{x}")} $LOAD_PATH.unshift(WORKER_ROOT) require "rubygems" require "yaml" require "erb" require "logger" require "packet" require "optparse" require "bdrb_config" require RAILS_HOME + "/config/boot" require "active_support" BackgrounDRb::Config.parse_cmd_options ARGV BDRB_CONFIG = BackgrounDRb::Config.read_config("#{RAILS_HOME}/config/ backgroundrb.yml") require RAILS_HOME + "/config/environment" require "bdrb_job_queue" require "backgroundrb_server" PID_FILE = "#{RAILS_HOME}/tmp/pids/ backgroundrb_#{BDRB_CONFIG[:backgroundrb][:port]}.pid" SERVER_LOGGER = "#{RAILS_HOME}/log/ backgroundrb_debug_#{BDRB_CONFIG[:backgroundrb][:port]}.log" def kill_process arg_pid_file pid = nil File.open(arg_pid_file, "r") { |pid_handle| pid = pid_handle.gets.strip.chomp.to_i } pgid = Process.getpgid(pid) puts "stopping backgroundrb" Process.kill('-TERM', pgid) File.delete(arg_pid_file) if File.exists?(arg_pid_file) end def status File.exists?(PID_FILE) end def start if fork sleep(5) exit else if status puts "already running" exit end puts "starting backgroundrb" op = File.open(PID_FILE, "w") op.write(Process.pid().to_s) op.close if BDRB_CONFIG[:backgroundrb][:log].nil? or BDRB_CONFIG[:backgroundrb][:log] != 'foreground' log_file = File.open(SERVER_LOGGER,"w+") [STDIN, STDOUT, STDERR].each {|desc| desc.reopen(log_file)} end BackgrounDRb::MasterProxy.new() end end def stop pid_files = Dir["#{RAILS_HOME}/tmp/pids/backgroundrb_*.pid"] pid_files.each { |x| kill_process(x) } end case ARGV[0] when 'start' start when 'stop' stop when 'restart' stop start when 'status' if status puts "running" exit else puts "not running" exit!(3) end else BackgrounDRb::MasterProxy.new() end On Sep 18, 2008, at 3:21 AM, John O'Shea wrote: > Slight variation that > - deletes pid for already-gone processes > - exits (with errror code -1) without deleting the pid file if there > was a permission problem > > begin > - pgid = Process.getpgid(pid) > - Process.kill('TERM', pid) > - Process.kill('-TERM', pgid) > - Process.kill('KILL', pid) > - rescue Errno::ESRCH => e > - puts "Deleting pid file" > - rescue > + pgid = Process.getpgid(pid) + Process.kill('-TERM', > pgid) + rescue Errno::ESRCH > + puts $! > + # No process - Do nothing. > + rescue Errno::EPERM > + # Permission denied. + puts $! > + Process.exit! > ensure File.delete(arg_pid_file) if File.exists? > (arg_pid_file) > end > hemant kumar wrote: >> Okay folks here is a patch to "backgroundrb" script, which should fix >> some issues: >> >> diff --git a/script/backgroundrb b/script/backgroundrb >> index dabf80b..8d4bb78 100755 >> --- a/script/backgroundrb >> +++ b/script/backgroundrb >> @@ -49,18 +49,9 @@ when 'stop' >> def kill_process arg_pid_file >> pid = nil >> File.open(arg_pid_file, "r") { |pid_handle| pid = >> pid_handle.gets.strip.chomp.to_i } >> - begin >> - pgid = Process.getpgid(pid) >> - Process.kill('TERM', pid) >> - Process.kill('-TERM', pgid) >> - Process.kill('KILL', pid) >> - rescue Errno::ESRCH => e >> - puts "Deleting pid file" >> - rescue >> - puts $! >> - ensure >> - File.delete(arg_pid_file) if File.exists?(arg_pid_file) >> - end >> + pgid = Process.getpgid(pid) >> + Process.kill('-TERM', pgid) >> + File.delete(arg_pid_file) if File.exists?(arg_pid_file) >> end >> pid_files = Dir["#{RAILS_HOME}/tmp/pids/backgroundrb_*.pid"] >> pid_files.each { |x| kill_process(x) } >> >> What it does is: >> 1. Deleting by group id is enough for master process. 2. Do not >> delete the pid file if, there was an exception while stopping >> the daemon. >> 3. Do not handle exceptions silently. >> >> Please try this and let me know, how it goes. >> >> >> >> On Wed, 2008-09-17 at 17:35 +0100, John O'Shea wrote: >> >>> Jonathan, >>> Glad you raised this, I've been spending some time trying to >>> diagnose this exact same problem. The exception handling code >>> in the "when 'stop'" block (in script/backgroundrb) could >>> definitely could be improved somewhat >>> - check that the process with 'pid' exists before trying to kill it >>> - rescue permission exceptions (Errno::EPERM) >>> - only delete the pid file if the process pid does not still exist >>> (in ensure block) >>> - be a little more verbose to stdout/stderr >>> >>> While we are on the subject of shutdown, - when the backgroundrb >>> process gets a HUP signal does it wait for existing workers to >>> complete any work methods that are executing or is the >>> 'Process.kill('-TERM', pgid)' call intended to make the OS handle >>> this? >>> We use capistrano to deploy our application (stopping and >>> restarting backgroundrb after the rails app has been updated). It >>> would be great if we could have more predictability regarding >>> shutting down backgroundrb (i.e. have the backgroundrb disable the >>> reactor loop in idle workers and wait for all active workers to >>> finish methods, then shutdown"). >>> >>> John. >>> >>> Jonathan Wallace wrote: >>> >>>> Hi Ryan, >>>> >>>> I recently ran into the same issue where the backgroundrb process >>>> would not respond to ./script/backgroundrb stop command. The pid >>>> file >>>> was being deleted but the actual process was not being killed. I'm >>>> running packet 0.1.12 on gentoo. >>>> >>>> I'm not exactly sure what conditions put backgroundrb into such a >>>> state but I've decided to modify the script/backgroundrb to >>>> behave a >>>> little differently. >>>> >>>> My hypothesis is that if one of the Process.kill method calls in >>>> script/backgroundrb raises an exception, the pid file is deleted >>>> even >>>> though the kill signal is never sent. At this point, running >>>> starting >>>> and stopping backgroundrb never affects the original still running >>>> backgroundrb process. >>>> >>>> There are a couple of reasons that I believe an exception could be >>>> raised. Either the Process.getpgid(pid), Process.kill('TERM', >>>> pid) or >>>> the PRocess.kill('-TERM', pgid) raise an exception or the effective >>>> uid of the user running script/backgroundrb stop does not have >>>> permission to kill those processes. >>>> >>>> To fix this, we've removed the Process.getpgid and the two >>>> Process.kill's that are sending the TERM signal. Since we've >>>> architected our backgroundrb jobs to be persistent and idempotent >>>> (a >>>> db backed queue written before the feature appeared in bdrb), we'll >>>> just use the KILL signal. >>>> >>>> Thoughts? >>>> >>>> Thanks, >>>> Jonathan >>>> >>>> On Tue, Sep 16, 2008 at 12:11 PM, Ryan Case >>>> wrote: >>>> >>>>> Hi folks - >>>>> >>>>> I'm having trouble getting backgroundrb to stop after one of the >>>>> packet_worker_r processes dies. >>>>> >>>>> If backgroundrb is running properly, >>>>> "/path/to/application/script/backgroundrb stop" works fine, but >>>>> often >>>>> one of the packet_worker_r processes dies, and the stop command no >>>>> longer works after that (it runs, but it does not stop the >>>>> processes, >>>>> and so then start doesn't work). >>>>> >>>>> The only thing that seems to work at that point is to manually >>>>> kill >>>>> the processes that are still running, and then the start works, >>>>> but >>>>> that is going to make restarting via monit a lot less clean. >>>>> >>>>> Any ideas would be much appreciated! >>>>> >>>>> I'm using github version of backgroundrb, and packet 0.1.13 >>>>> running on ubuntu. >>>>> >>>>> Thanks! >>>>> Ryan >>>>> _______________________________________________ >>>>> Backgroundrb-devel mailing list >>>>> Backgroundrb-devel at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>>>> >>>>> >>>> _______________________________________________ >>>> Backgroundrb-devel mailing list >>>> Backgroundrb-devel at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>>> >>> >> >> > > > -- > John O'Shea, CTO at Nooked > www: http://www.nooked.com/ > cell: +353 87 992 9959 > skype: joshea > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From gethemant at gmail.com Thu Sep 18 23:14:30 2008 From: gethemant at gmail.com (hemant kumar) Date: Fri, 19 Sep 2008 08:44:30 +0530 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> <48D131D9.9000703@nooked.com> <1221689332.16986.5.camel@shire> <48D22B8C.50507@nooked.com> Message-ID: <1221794070.23629.0.camel@shire> Okay, So, Did you find out, why "stop" didn't work from logrotate, in first place. I think, thats rather critical. On Thu, 2008-09-18 at 11:24 -0700, Woody Peterson wrote: > In my particular case I know it's not a permissions issue, as I'm > always using the same user. > > I just tried restarting it, and with Hemant's patch I got: > > script/backgroundrb:52:in `getpgid': No such process (Errno::ESRCH) > > Via the above I found that in this particular case what happened is > that my logrotate wasn't calling stop, only start (it meant to call > stop, but was in a failing if statement checking if the pid existed). > When you call start, it doesn't check to see if it's already running, > so it starts backgroundrb, overwrites the pid file, then backgroundrb > fails to start but has had it's pid file changed. The original process > is still running, but can't stop because it doesn't have the correct > pid in the pid file. > > Thus, I rewrote script/backgroundrb to be more LSB compliant (http://refspecs.linux-foundation.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html > ) so I don't have to check for existing pid files myself. I made a > patch, but it's almost as big as the script itself and Hemants patch > didn't apply for me (I must have changed something earlier in the > file), so the whole thing is at the end of the email. > > While we're on the topic, is there a place to load all the > requirements other than this file? backgroundrb status takes a matter > of seconds to do a simple File.exists?(pid) 'cuz it has to load all > the backgroundrb requirements. Not that it really matters... > > -Woody > > #!/usr/bin/env ruby > > RAILS_HOME = File.expand_path(File.join(File.dirname(__FILE__),"..")) > BDRB_HOME = File.join(RAILS_HOME,"vendor","plugins","backgroundrb") > WORKER_ROOT = File.join(RAILS_HOME,"lib","workers") > WORKER_LOAD_ENV = File.join(RAILS_HOME,"script","load_worker_env") > > ["server","server/lib","lib","lib/backgroundrb"].each { |x| > $LOAD_PATH.unshift(BDRB_HOME + "/#{x}")} > $LOAD_PATH.unshift(WORKER_ROOT) > > require "rubygems" > require "yaml" > require "erb" > require "logger" > require "packet" > require "optparse" > > require "bdrb_config" > require RAILS_HOME + "/config/boot" > require "active_support" > > BackgrounDRb::Config.parse_cmd_options ARGV > BDRB_CONFIG = BackgrounDRb::Config.read_config("#{RAILS_HOME}/config/ > backgroundrb.yml") > > require RAILS_HOME + "/config/environment" > require "bdrb_job_queue" > require "backgroundrb_server" > > PID_FILE = "#{RAILS_HOME}/tmp/pids/ > backgroundrb_#{BDRB_CONFIG[:backgroundrb][:port]}.pid" > SERVER_LOGGER = "#{RAILS_HOME}/log/ > backgroundrb_debug_#{BDRB_CONFIG[:backgroundrb][:port]}.log" > > def kill_process arg_pid_file > pid = nil > File.open(arg_pid_file, "r") { |pid_handle| pid = > pid_handle.gets.strip.chomp.to_i } > pgid = Process.getpgid(pid) > puts "stopping backgroundrb" > Process.kill('-TERM', pgid) > File.delete(arg_pid_file) if File.exists?(arg_pid_file) > end > > def status > File.exists?(PID_FILE) > end > > def start > if fork > sleep(5) > exit > else > if status > puts "already running" > exit > end > > puts "starting backgroundrb" > > op = File.open(PID_FILE, "w") > op.write(Process.pid().to_s) > op.close > if BDRB_CONFIG[:backgroundrb][:log].nil? or > BDRB_CONFIG[:backgroundrb][:log] != 'foreground' > log_file = File.open(SERVER_LOGGER,"w+") > [STDIN, STDOUT, STDERR].each {|desc| desc.reopen(log_file)} > end > > BackgrounDRb::MasterProxy.new() > end > end > > def stop > pid_files = Dir["#{RAILS_HOME}/tmp/pids/backgroundrb_*.pid"] > pid_files.each { |x| kill_process(x) } > end > > case ARGV[0] > when 'start' > start > when 'stop' > stop > when 'restart' > stop > start > when 'status' > if status > puts "running" > exit > else > puts "not running" > exit!(3) > end > else > BackgrounDRb::MasterProxy.new() > end > > > On Sep 18, 2008, at 3:21 AM, John O'Shea wrote: > > > Slight variation that > > - deletes pid for already-gone processes > > - exits (with errror code -1) without deleting the pid file if there > > was a permission problem > > > > begin > > - pgid = Process.getpgid(pid) > > - Process.kill('TERM', pid) > > - Process.kill('-TERM', pgid) > > - Process.kill('KILL', pid) > > - rescue Errno::ESRCH => e > > - puts "Deleting pid file" > > - rescue > > + pgid = Process.getpgid(pid) + Process.kill('-TERM', > > pgid) + rescue Errno::ESRCH > > + puts $! > > + # No process - Do nothing. > > + rescue Errno::EPERM > > + # Permission denied. + puts $! > > + Process.exit! > > ensure File.delete(arg_pid_file) if File.exists? > > (arg_pid_file) > > end > > hemant kumar wrote: > >> Okay folks here is a patch to "backgroundrb" script, which should fix > >> some issues: > >> > >> diff --git a/script/backgroundrb b/script/backgroundrb > >> index dabf80b..8d4bb78 100755 > >> --- a/script/backgroundrb > >> +++ b/script/backgroundrb > >> @@ -49,18 +49,9 @@ when 'stop' > >> def kill_process arg_pid_file > >> pid = nil > >> File.open(arg_pid_file, "r") { |pid_handle| pid = > >> pid_handle.gets.strip.chomp.to_i } > >> - begin > >> - pgid = Process.getpgid(pid) > >> - Process.kill('TERM', pid) > >> - Process.kill('-TERM', pgid) > >> - Process.kill('KILL', pid) > >> - rescue Errno::ESRCH => e > >> - puts "Deleting pid file" > >> - rescue > >> - puts $! > >> - ensure > >> - File.delete(arg_pid_file) if File.exists?(arg_pid_file) > >> - end > >> + pgid = Process.getpgid(pid) > >> + Process.kill('-TERM', pgid) > >> + File.delete(arg_pid_file) if File.exists?(arg_pid_file) > >> end > >> pid_files = Dir["#{RAILS_HOME}/tmp/pids/backgroundrb_*.pid"] > >> pid_files.each { |x| kill_process(x) } > >> > >> What it does is: > >> 1. Deleting by group id is enough for master process. 2. Do not > >> delete the pid file if, there was an exception while stopping > >> the daemon. > >> 3. Do not handle exceptions silently. > >> > >> Please try this and let me know, how it goes. > >> > >> > >> > >> On Wed, 2008-09-17 at 17:35 +0100, John O'Shea wrote: > >> > >>> Jonathan, > >>> Glad you raised this, I've been spending some time trying to > >>> diagnose this exact same problem. The exception handling code > >>> in the "when 'stop'" block (in script/backgroundrb) could > >>> definitely could be improved somewhat > >>> - check that the process with 'pid' exists before trying to kill it > >>> - rescue permission exceptions (Errno::EPERM) > >>> - only delete the pid file if the process pid does not still exist > >>> (in ensure block) > >>> - be a little more verbose to stdout/stderr > >>> > >>> While we are on the subject of shutdown, - when the backgroundrb > >>> process gets a HUP signal does it wait for existing workers to > >>> complete any work methods that are executing or is the > >>> 'Process.kill('-TERM', pgid)' call intended to make the OS handle > >>> this? > >>> We use capistrano to deploy our application (stopping and > >>> restarting backgroundrb after the rails app has been updated). It > >>> would be great if we could have more predictability regarding > >>> shutting down backgroundrb (i.e. have the backgroundrb disable the > >>> reactor loop in idle workers and wait for all active workers to > >>> finish methods, then shutdown"). > >>> > >>> John. > >>> > >>> Jonathan Wallace wrote: > >>> > >>>> Hi Ryan, > >>>> > >>>> I recently ran into the same issue where the backgroundrb process > >>>> would not respond to ./script/backgroundrb stop command. The pid > >>>> file > >>>> was being deleted but the actual process was not being killed. I'm > >>>> running packet 0.1.12 on gentoo. > >>>> > >>>> I'm not exactly sure what conditions put backgroundrb into such a > >>>> state but I've decided to modify the script/backgroundrb to > >>>> behave a > >>>> little differently. > >>>> > >>>> My hypothesis is that if one of the Process.kill method calls in > >>>> script/backgroundrb raises an exception, the pid file is deleted > >>>> even > >>>> though the kill signal is never sent. At this point, running > >>>> starting > >>>> and stopping backgroundrb never affects the original still running > >>>> backgroundrb process. > >>>> > >>>> There are a couple of reasons that I believe an exception could be > >>>> raised. Either the Process.getpgid(pid), Process.kill('TERM', > >>>> pid) or > >>>> the PRocess.kill('-TERM', pgid) raise an exception or the effective > >>>> uid of the user running script/backgroundrb stop does not have > >>>> permission to kill those processes. > >>>> > >>>> To fix this, we've removed the Process.getpgid and the two > >>>> Process.kill's that are sending the TERM signal. Since we've > >>>> architected our backgroundrb jobs to be persistent and idempotent > >>>> (a > >>>> db backed queue written before the feature appeared in bdrb), we'll > >>>> just use the KILL signal. > >>>> > >>>> Thoughts? > >>>> > >>>> Thanks, > >>>> Jonathan > >>>> > >>>> On Tue, Sep 16, 2008 at 12:11 PM, Ryan Case > >>>> wrote: > >>>> > >>>>> Hi folks - > >>>>> > >>>>> I'm having trouble getting backgroundrb to stop after one of the > >>>>> packet_worker_r processes dies. > >>>>> > >>>>> If backgroundrb is running properly, > >>>>> "/path/to/application/script/backgroundrb stop" works fine, but > >>>>> often > >>>>> one of the packet_worker_r processes dies, and the stop command no > >>>>> longer works after that (it runs, but it does not stop the > >>>>> processes, > >>>>> and so then start doesn't work). > >>>>> > >>>>> The only thing that seems to work at that point is to manually > >>>>> kill > >>>>> the processes that are still running, and then the start works, > >>>>> but > >>>>> that is going to make restarting via monit a lot less clean. > >>>>> > >>>>> Any ideas would be much appreciated! > >>>>> > >>>>> I'm using github version of backgroundrb, and packet 0.1.13 > >>>>> running on ubuntu. > >>>>> > >>>>> Thanks! > >>>>> Ryan > >>>>> _______________________________________________ > >>>>> Backgroundrb-devel mailing list > >>>>> Backgroundrb-devel at rubyforge.org > >>>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > >>>>> > >>>>> > >>>> _______________________________________________ > >>>> Backgroundrb-devel mailing list > >>>> Backgroundrb-devel at rubyforge.org > >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > >>>> > >>> > >> > >> > > > > > > -- > > John O'Shea, CTO at Nooked > > www: http://www.nooked.com/ > > cell: +353 87 992 9959 > > skype: joshea > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From jnutting at gmail.com Fri Sep 19 08:33:08 2008 From: jnutting at gmail.com (Jack Nutting) Date: Fri, 19 Sep 2008 14:33:08 +0200 Subject: [Backgroundrb-devel] ask_work Message-ID: I'm working on upgrading to the latest backgroundrb (from svn, not the bleeding edge from github), and have been able to work through each problem I've encountered until this. I've got a set of workers running, and one 'master' worker that assigns them tasks like this: MiddleMan.ask_work(:worker=>worker, :job_key=>job_key, :worker_method=>:work_wrapper) (worker and job_key are references to already-existing workers) In the latest version this doesn't work, I get this at the top of my backtrace: undefined method `ask_work' for # So now that the ask_work method is gone, what's the right way for me to do this? I've looked through the documentation on the website and haven't seen anything that deals with this usage. -- // jack // http://www.nuthole.com From jnutting at gmail.com Fri Sep 19 08:43:29 2008 From: jnutting at gmail.com (Jack Nutting) Date: Fri, 19 Sep 2008 14:43:29 +0200 Subject: [Backgroundrb-devel] version numbering policy Message-ID: IMHO, the amount of changes that happen from one release to the next should warrant much bigger version numbering changes. E.g. I am in the process of upgrading from (I think) 1.0.1 to 1.0.4, and had to do a substantial amount of poring over code and making changes due to API changes. It seems to me that that amount of changes in the codebase should at least warrant a bump to 1.1. I guess that to my way of thinking, a change in the first number should indicate major changes in a project's structure or direction, a change in the the second number should indicate significant functional additions, changes, or API changes, while a change in the third number should really just be about bug-fixes and minor enhancements. What do the rest of you think? -- // jack // http://www.nuthole.com From gethemant at gmail.com Fri Sep 19 09:11:52 2008 From: gethemant at gmail.com (hemant kumar) Date: Fri, 19 Sep 2008 18:41:52 +0530 Subject: [Backgroundrb-devel] ask_work In-Reply-To: References: Message-ID: <1221829912.28380.2.camel@shire> Hi Jack, New method is simply: MiddleMan.worker(:foo_worker,"worker_key").async_work_wrapper() and for synchronous tasks MiddleMan.worker(:foo_worker,"worker_key").work_wrapper() More details you can find at: http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001880.html On Fri, 2008-09-19 at 14:33 +0200, Jack Nutting wrote: > I'm working on upgrading to the latest backgroundrb (from svn, not the > bleeding edge from github), and have been able to work through each > problem I've encountered until this. > > I've got a set of workers running, and one 'master' worker that > assigns them tasks like this: > > MiddleMan.ask_work(:worker=>worker, :job_key=>job_key, > :worker_method=>:work_wrapper) > > (worker and job_key are references to already-existing workers) > > In the latest version this doesn't work, I get this at the top of my backtrace: > > undefined method `ask_work' for # > > So now that the ask_work method is gone, what's the right way for me > to do this? I've looked through the documentation on the website and > haven't seen anything that deals with this usage. > From gethemant at gmail.com Fri Sep 19 09:16:06 2008 From: gethemant at gmail.com (hemant kumar) Date: Fri, 19 Sep 2008 18:46:06 +0530 Subject: [Backgroundrb-devel] version numbering policy In-Reply-To: References: Message-ID: <1221830166.28380.8.camel@shire> I was hoping for two things to fix before bumping the version to 1.1: * Problem with running packet as frozen gem. * close to 100% coverage (currently at 67%) I couldn't manage that with 1.0.4 release and hence even though 1.0.4 involved API changes and was really way better than previous versions, I didn't do a major version bump. But I don't want this to be my decision. If folks are willing to put some time and effort in helping out, we can discuss this beforehand and make such decisions. On Fri, 2008-09-19 at 14:43 +0200, Jack Nutting wrote: > IMHO, the amount of changes that happen from one release to the next > should warrant much bigger version numbering changes. E.g. I am in > the process of upgrading from (I think) 1.0.1 to 1.0.4, and had to do > a substantial amount of poring over code and making changes due to API > changes. It seems to me that that amount of changes in the codebase > should at least warrant a bump to 1.1. I guess that to my way of > thinking, a change in the first number should indicate major changes > in a project's structure or direction, a change in the the second > number should indicate significant functional additions, changes, or > API changes, while a change in the third number should really just be > about bug-fixes and minor enhancements. What do the rest of you > think? > From curtis.hatter at insightbb.com Fri Sep 19 23:14:57 2008 From: curtis.hatter at insightbb.com (Mitchell Curtis Hatter) Date: Fri, 19 Sep 2008 23:14:57 -0400 Subject: [Backgroundrb-devel] Worker's not created quick enough? Message-ID: Just starting out with BackgroundDRb and have some troubles with calling a worker's async process right after I try to create it. Here's my code: In my book model I have the following: # Starts a new BackgroundDRb worker for searching def start_search(worker_key, job_key, search_params) MiddleMan.new_worker(:worker => :texis_worker, :worker_key => worker_key.to_s) MiddleMan.worker(:texis_worker, worker_key.to_s).async_search(:arg => {:book => self, :search_params => search_params}, :job_key => job_key) end # Get Status of Worker def self.search_status(worker_key, job_key) status = MiddleMan.worker(:texis_worker, worker_key.to_s).ask_result("#{worker_key.to_s}_#{job_key.to_s}") end My controller's search method has this line: @book.start_search(session[:user], @book.id, search_params) Here's my worker: class TexisWorker < BackgrounDRb::MetaWorker set_worker_name :texis_worker set_no_auto_load true def create(args = nil) logger.info "Creating #{worker_key}" end def search(arg) logger.info "Testing #{worker_key} -- #{job_key} -- #{arg[:book].name}" cache["#{worker_key.to_s}_#{job_key.to_s}"] = arg[:book].name end end When a user runs a search in my books controller I call start_search passing in a user id as the worker_key, book_id as job_key and a hash of search parameters. I tail the log and then browse to the controller. In my log I will get: Creating 293460168 But nothing else. Eventually, say 30 seconds to a minute and switching between different books I finally start to get something like: Testing 293460168 -- 3806 -- Workers Compensation If I do this from script/console everything works fine. I'm running mongrel in development mode, using packet 0.1.13, backgroundrb 1.0.4, cent os 5 It seems initially creating the worker is not done for a while? and so the calls to async are failing; once the worker has been intialized everything is fine? Any help would be appreciated. Thanks, Curtis From brentmc79 at gmail.com Fri Sep 19 23:58:52 2008 From: brentmc79 at gmail.com (Brent Collier) Date: Fri, 19 Sep 2008 23:58:52 -0400 Subject: [Backgroundrb-devel] Worker's not created quick enough? In-Reply-To: References: Message-ID: Yeah, you pretty much have to throw a sleep call in between creating the worker and actually asking it to do work. It only takes a fraction of a second. I was told this was only a problem on OS X, but I've also witnessed it on Gentoo. I see this question on the mailing list enough that I think it might as well be included in the docs somewhere. -Brent On Fri, Sep 19, 2008 at 11:14 PM, Mitchell Curtis Hatter < curtis.hatter at insightbb.com> wrote: > Just starting out with BackgroundDRb and have some troubles with calling a > worker's async process right after I try to create it. > > Here's my code: > > In my book model I have the following: > > # Starts a new BackgroundDRb worker for searching > def start_search(worker_key, job_key, search_params) > MiddleMan.new_worker(:worker => :texis_worker, :worker_key => > worker_key.to_s) > MiddleMan.worker(:texis_worker, worker_key.to_s).async_search(:arg => > {:book => self, :search_params => search_params}, :job_key => job_key) > end > > # Get Status of Worker > def self.search_status(worker_key, job_key) > status = MiddleMan.worker(:texis_worker, > worker_key.to_s).ask_result("#{worker_key.to_s}_#{job_key.to_s}") > end > > My controller's search method has this line: > > @book.start_search(session[:user], @book.id, search_params) > > Here's my worker: > > class TexisWorker < BackgrounDRb::MetaWorker > set_worker_name :texis_worker > set_no_auto_load true > > def create(args = nil) > logger.info "Creating #{worker_key}" > end > > def search(arg) > logger.info "Testing #{worker_key} -- #{job_key} -- #{arg[:book].name}" > cache["#{worker_key.to_s}_#{job_key.to_s}"] = arg[:book].name > end > end > > > When a user runs a search in my books controller I call start_search > passing in a user id as the worker_key, book_id as job_key and a hash of > search parameters. > > I tail the log and then browse to the controller. In my log I will get: > Creating 293460168 > > But nothing else. > > Eventually, say 30 seconds to a minute and switching between different > books I finally start to get something like: > Testing 293460168 -- 3806 -- Workers Compensation > > If I do this from script/console everything works fine. > > I'm running mongrel in development mode, using packet 0.1.13, backgroundrb > 1.0.4, cent os 5 > > It seems initially creating the worker is not done for a while? and so the > calls to async are failing; once the worker has been intialized everything > is fine? > > Any help would be appreciated. > > Thanks, > Curtis > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Brent Collier | 919.564.6915 | www.BrentCollier.com | www.acts-as-blogr.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From curtis.hatter at insightbb.com Sat Sep 20 00:11:25 2008 From: curtis.hatter at insightbb.com (Mitchell Curtis Hatter) Date: Sat, 20 Sep 2008 00:11:25 -0400 Subject: [Backgroundrb-devel] Worker's not created quick enough? In-Reply-To: References: Message-ID: Thanks, yeah I specifically moved my development off my laptop to the CentOS system because I saw a thread that said it was only a problem on OS X. This is unfortunate, as when it does work it's great. I'll try the sleep call and see what starts to happen. Thanks again, Curtis On Sep 19, 2008, at 11:58 PM, Brent Collier wrote: > Yeah, you pretty much have to throw a sleep call in between creating > the worker and actually asking it to do work. It only takes a > fraction of a second. I was told this was only a problem on OS X, > but I've also witnessed it on Gentoo. I see this question on the > mailing list enough that I think it might as well be included in the > docs somewhere. > > -Brent > > > On Fri, Sep 19, 2008 at 11:14 PM, Mitchell Curtis Hatter > wrote: > Just starting out with BackgroundDRb and have some troubles with > calling a worker's async process right after I try to create it. > > Here's my code: > > In my book model I have the following: > > # Starts a new BackgroundDRb worker for searching > def start_search(worker_key, job_key, search_params) > MiddleMan.new_worker(:worker => :texis_worker, :worker_key => > worker_key.to_s) > MiddleMan.worker(:texis_worker, worker_key.to_s).async_search(:arg > => {:book => self, :search_params => search_params}, :job_key => > job_key) > end > > # Get Status of Worker > def self.search_status(worker_key, job_key) > status = MiddleMan.worker(:texis_worker, > worker_key.to_s).ask_result("#{worker_key.to_s}_#{job_key.to_s}") > end > > My controller's search method has this line: > > @book.start_search(session[:user], @book.id, search_params) > > Here's my worker: > > class TexisWorker < BackgrounDRb::MetaWorker > set_worker_name :texis_worker > set_no_auto_load true > > def create(args = nil) > logger.info "Creating #{worker_key}" > end > > def search(arg) > logger.info "Testing #{worker_key} -- #{job_key} -- > #{arg[:book].name}" > cache["#{worker_key.to_s}_#{job_key.to_s}"] = arg[:book].name > end > end > > > When a user runs a search in my books controller I call start_search > passing in a user id as the worker_key, book_id as job_key and a > hash of search parameters. > > I tail the log and then browse to the controller. In my log I will > get: > Creating 293460168 > > But nothing else. > > Eventually, say 30 seconds to a minute and switching between > different books I finally start to get something like: > Testing 293460168 -- 3806 -- Workers Compensation > > If I do this from script/console everything works fine. > > I'm running mongrel in development mode, using packet 0.1.13, > backgroundrb 1.0.4, cent os 5 > > It seems initially creating the worker is not done for a while? and > so the calls to async are failing; once the worker has been > intialized everything is fine? > > Any help would be appreciated. > > Thanks, > Curtis > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -- > Brent Collier | 919.564.6915 | www.BrentCollier.com | www.acts-as-blogr.com From gethemant at gmail.com Sat Sep 20 00:36:44 2008 From: gethemant at gmail.com (hemant kumar) Date: Sat, 20 Sep 2008 10:06:44 +0530 Subject: [Backgroundrb-devel] Worker's not created quick enough? In-Reply-To: References: Message-ID: <1221885404.18769.12.camel@shire> On Fri, 2008-09-19 at 23:58 -0400, Brent Collier wrote: > Yeah, you pretty much have to throw a sleep call in between creating > the worker and actually asking it to do work. It only takes a > fraction of a second. I was told this was only a problem on OS X, but > I've also witnessed it on Gentoo. I see this question on the mailing > list enough that I think it might as well be included in the docs > somewhere. First what exactly is async_xxx "failed"? As far as my thinking goes, since in a worker, usually full rails environment is loaded, it is naturally a bit slow to start, but any sync/async task execution should not get "lost" between the time worker is loading the rails environment. For example, on my ubuntu box, with packet-0.1.13 and bdrb from git: require File.join(File.dirname(__FILE__) + "/../config/environment") a = MiddleMan.new_worker(:worker => :wow_worker,:worker_key => "hello_world") 10.times do |i| MiddleMan.worker(:wow_worker,"hello_world").async_run_crap(:arg => "hello world : #{i}") end I won't loose any of the "async_run_crap" calls, even though there is no "sleep" between starting a worker and asking it to execute a method. Sure, worker may take time, to be fully ready, but by that time, tasks should be "queued" up and thats what happens in Linux and behaviour is correct. On OSX though, you will loose async_xxx calls and hence its a bug.So, mitchell, are you loosing async_xxx method calls? Or it gets invoked eventually but slightly delayed (while its loading rails env)? If a method call is lost, its a bug, I will need to fix it. On OSX, ruby nonblocking APIs aren't working well and hence tasks aren't getting enqueued in socket buffer too and hence we are loosing messages. But this works, quite well in Linux, in my experience. From curtis.hatter at insightbb.com Sat Sep 20 01:25:32 2008 From: curtis.hatter at insightbb.com (Mitchell Curtis Hatter) Date: Sat, 20 Sep 2008 01:25:32 -0400 Subject: [Backgroundrb-devel] Worker's not created quick enough? In-Reply-To: <1221885404.18769.12.camel@shire> References: <1221885404.18769.12.camel@shire> Message-ID: I'm actually losing the call to the async_* method. Anytime I first create a new worker I lose any async_* calls to it for approximately 10 to 15 seconds. I even tried this: MiddleMan.new_worker(:worker => :texis_worker, :worker_key => worker_key.to_s) sleep(5) MiddleMan.worker(:texis_worker, worker_key.to_s).async_search(:arg => {:book => self, :search_params => search_params}, :job_key => job_key) And got nothing. After the worker creates it's fine. But I tried just one call and waited about 30 seconds, got nothing, then made another call and got results right away. Again, when I do it from script/console it never fails. On Sep 20, 2008, at 12:36 AM, hemant kumar wrote: > On Fri, 2008-09-19 at 23:58 -0400, Brent Collier wrote: >> Yeah, you pretty much have to throw a sleep call in between creating >> the worker and actually asking it to do work. It only takes a >> fraction of a second. I was told this was only a problem on OS X, >> but >> I've also witnessed it on Gentoo. I see this question on the mailing >> list enough that I think it might as well be included in the docs >> somewhere. > > > First what exactly is async_xxx "failed"? As far as my thinking goes, > since in a worker, usually full rails environment is loaded, it is > naturally a bit slow to start, but any sync/async task execution > should > not get "lost" between the time worker is loading the rails > environment. > > For example, on my ubuntu box, with packet-0.1.13 and bdrb from git: > > require File.join(File.dirname(__FILE__) + "/../config/environment") > > a = MiddleMan.new_worker(:worker => :wow_worker,:worker_key => > "hello_world") > > 10.times do |i| > MiddleMan.worker(:wow_worker,"hello_world").async_run_crap(:arg => > "hello world : #{i}") > end > > I won't loose any of the "async_run_crap" calls, even though there > is no > "sleep" between starting a worker and asking it to execute a method. > Sure, worker may take time, to be fully ready, but by that time, tasks > should be "queued" up and thats what happens in Linux and behaviour is > correct. > > On OSX though, you will loose async_xxx calls and hence its a bug.So, > mitchell, are you loosing async_xxx method calls? Or it gets invoked > eventually but slightly delayed (while its loading rails env)? If a > method call is lost, its a bug, I will need to fix it. > > On OSX, ruby nonblocking APIs aren't working well and hence tasks > aren't > getting enqueued in socket buffer too and hence we are loosing > messages. > But this works, quite well in Linux, in my experience. > > > > > From gethemant at gmail.com Sat Sep 20 02:18:41 2008 From: gethemant at gmail.com (hemant) Date: Sat, 20 Sep 2008 11:48:41 +0530 Subject: [Backgroundrb-devel] Worker's not created quick enough? In-Reply-To: References: <1221885404.18769.12.camel@shire> Message-ID: Mitchel, can you send me tar.gz file of your app, which is sufficient to reproduce mentioned behaviour? I mean, don't send me entire app (if it contains confidential info), just enough of code, so as I can reproduce this. Also, in the meanwhile, you can try git version and see if it fixes this. On Sat, Sep 20, 2008 at 10:55 AM, Mitchell Curtis Hatter wrote: > I'm actually losing the call to the async_* method. > > Anytime I first create a new worker I lose any async_* calls to it for > approximately 10 to 15 seconds. > > I even tried this: > > MiddleMan.new_worker(:worker => :texis_worker, :worker_key => > worker_key.to_s) > sleep(5) > MiddleMan.worker(:texis_worker, worker_key.to_s).async_search(:arg => > {:book => self, :search_params => search_params}, :job_key => job_key) > > And got nothing. After the worker creates it's fine. But I tried just one > call and waited about 30 seconds, got nothing, then made another call and > got results right away. > > Again, when I do it from script/console it never fails. > > On Sep 20, 2008, at 12:36 AM, hemant kumar wrote: > >> On Fri, 2008-09-19 at 23:58 -0400, Brent Collier wrote: >>> >>> Yeah, you pretty much have to throw a sleep call in between creating >>> the worker and actually asking it to do work. It only takes a >>> fraction of a second. I was told this was only a problem on OS X, but >>> I've also witnessed it on Gentoo. I see this question on the mailing >>> list enough that I think it might as well be included in the docs >>> somewhere. >> >> >> First what exactly is async_xxx "failed"? As far as my thinking goes, >> since in a worker, usually full rails environment is loaded, it is >> naturally a bit slow to start, but any sync/async task execution should >> not get "lost" between the time worker is loading the rails environment. >> >> For example, on my ubuntu box, with packet-0.1.13 and bdrb from git: >> >> require File.join(File.dirname(__FILE__) + "/../config/environment") >> >> a = MiddleMan.new_worker(:worker => :wow_worker,:worker_key => >> "hello_world") >> >> 10.times do |i| >> MiddleMan.worker(:wow_worker,"hello_world").async_run_crap(:arg => >> "hello world : #{i}") >> end >> >> I won't loose any of the "async_run_crap" calls, even though there is no >> "sleep" between starting a worker and asking it to execute a method. >> Sure, worker may take time, to be fully ready, but by that time, tasks >> should be "queued" up and thats what happens in Linux and behaviour is >> correct. >> >> On OSX though, you will loose async_xxx calls and hence its a bug.So, >> mitchell, are you loosing async_xxx method calls? Or it gets invoked >> eventually but slightly delayed (while its loading rails env)? If a >> method call is lost, its a bug, I will need to fix it. >> >> On OSX, ruby nonblocking APIs aren't working well and hence tasks aren't >> getting enqueued in socket buffer too and hence we are loosing messages. >> But this works, quite well in Linux, in my experience. >> >> >> >> >> > > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From curtis.hatter at insightbb.com Sat Sep 20 02:27:21 2008 From: curtis.hatter at insightbb.com (Mitchell Curtis Hatter) Date: Sat, 20 Sep 2008 02:27:21 -0400 Subject: [Backgroundrb-devel] Worker's not created quick enough? In-Reply-To: <1221885404.18769.12.camel@shire> References: <1221885404.18769.12.camel@shire> Message-ID: <93E079F2-FBCD-4E15-B160-A5117CB149B3@insightbb.com> Found this: http://gnufied.org/2008/02/12/backgroundrb-best-practises/ Which you wrote of course. Point #2 seems to have solved my problem. I removed where I was passing the entire Book AR object and just passed the Book's id and things start working as expected within the full Rails environment. Any chance a "best practices" could be added somewhere to the backgroundrb.rubyforge.org site and have those comments added. If I have any more problems with this I'll post to this thread again but for now seems to be working. Thanks, Curtis On Sep 20, 2008, at 12:36 AM, hemant kumar wrote: > On Fri, 2008-09-19 at 23:58 -0400, Brent Collier wrote: >> Yeah, you pretty much have to throw a sleep call in between creating >> the worker and actually asking it to do work. It only takes a >> fraction of a second. I was told this was only a problem on OS X, >> but >> I've also witnessed it on Gentoo. I see this question on the mailing >> list enough that I think it might as well be included in the docs >> somewhere. > > > First what exactly is async_xxx "failed"? As far as my thinking goes, > since in a worker, usually full rails environment is loaded, it is > naturally a bit slow to start, but any sync/async task execution > should > not get "lost" between the time worker is loading the rails > environment. > > For example, on my ubuntu box, with packet-0.1.13 and bdrb from git: > > require File.join(File.dirname(__FILE__) + "/../config/environment") > > a = MiddleMan.new_worker(:worker => :wow_worker,:worker_key => > "hello_world") > > 10.times do |i| > MiddleMan.worker(:wow_worker,"hello_world").async_run_crap(:arg => > "hello world : #{i}") > end > > I won't loose any of the "async_run_crap" calls, even though there > is no > "sleep" between starting a worker and asking it to execute a method. > Sure, worker may take time, to be fully ready, but by that time, tasks > should be "queued" up and thats what happens in Linux and behaviour is > correct. > > On OSX though, you will loose async_xxx calls and hence its a bug.So, > mitchell, are you loosing async_xxx method calls? Or it gets invoked > eventually but slightly delayed (while its loading rails env)? If a > method call is lost, its a bug, I will need to fix it. > > On OSX, ruby nonblocking APIs aren't working well and hence tasks > aren't > getting enqueued in socket buffer too and hence we are loosing > messages. > But this works, quite well in Linux, in my experience. > > > > > From ramon.tayag at gmail.com Sun Sep 21 06:23:09 2008 From: ramon.tayag at gmail.com (Ramon Miguel M. Tayag) Date: Sun, 21 Sep 2008 18:23:09 +0800 Subject: [Backgroundrb-devel] Monit; BackgroundRB; environment variables Message-ID: I started emailing monit mailing list because I started getting this problem where I couldn't start backgroundrb with monit anymore, thinking it was a monit problem. However, I back tracked a bit since I had added code to my project, and monit was able to start backgroundrb again. I seem to have zeroed in on the problem. I'm not sure where the problem lies - maybe monit or maybe backgroundrb, or very likely, my code :) The problem of monit not being able to start backgroundrb started when I added 2 lines to my Rails configuration file: "require 'lib/hodel_3000_compliant_logger'" AND "config.logger = Hodel3000CompliantLogger.new(config.log_path)" (See http://pastie.org/276618 for better formatting. I had just installed a logger tool.) My speculation: via monit backgroundrb tried to load the logger lib but can't find it. Since monit does away with environment variables, maybe the backgroundrb script expects some of these to be there (maybe to find where the RAILS_ROOT is). I'm stuck because I don't know what environment variables the backgroundrb script needs to start properly. If I know what it needs then I can add it to the monit start command. Thanks, -- Ramon Tayag From gethemant at gmail.com Sun Sep 21 15:25:18 2008 From: gethemant at gmail.com (hemant kumar) Date: Mon, 22 Sep 2008 00:55:18 +0530 Subject: [Backgroundrb-devel] Monit; BackgroundRB; environment variables In-Reply-To: References: Message-ID: <1222025118.9729.1.camel@shire> BackgrounDRb doesn't depends on any environment variable, but it depends on correct path (meaning, location from where you are starting backgroundrb matters. On Sun, 2008-09-21 at 18:23 +0800, Ramon Miguel M. Tayag wrote: > I started emailing monit mailing list because I started getting this > problem where I couldn't start backgroundrb with monit anymore, > thinking it was a monit problem. However, I back tracked a bit since > I had added code to my project, and monit was able to start > backgroundrb again. > > I seem to have zeroed in on the problem. I'm not sure where the > problem lies - maybe monit or maybe backgroundrb, or very likely, my > code :) > > The problem of monit not being able to start backgroundrb started when > I added 2 lines to my Rails configuration file: > > "require 'lib/hodel_3000_compliant_logger'" AND "config.logger = > Hodel3000CompliantLogger.new(config.log_path)" > (See http://pastie.org/276618 for better formatting. I had just > installed a logger tool.) > > My speculation: via monit backgroundrb tried to load the logger lib > but can't find it. Since monit does away with environment variables, > maybe the backgroundrb script expects some of these to be there (maybe > to find where the RAILS_ROOT is). > > I'm stuck because I don't know what environment variables the > backgroundrb script needs to start properly. If I know what it needs > then I can add it to the monit start command. > > Thanks, From jnutting at gmail.com Sun Sep 21 18:37:02 2008 From: jnutting at gmail.com (Jack Nutting) Date: Mon, 22 Sep 2008 00:37:02 +0200 Subject: [Backgroundrb-devel] ask_work In-Reply-To: <1221829912.28380.2.camel@shire> References: <1221829912.28380.2.camel@shire> Message-ID: On Fri, Sep 19, 2008 at 3:11 PM, hemant kumar wrote: > Hi Jack, > > New method is simply: > > MiddleMan.worker(:foo_worker,"worker_key").async_work_wrapper() > > and for synchronous tasks > > MiddleMan.worker(:foo_worker,"worker_key").work_wrapper() > > More details you can find at: > > http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001880.html Thanks, that seems to do the trick! I must have missed your announcement about 1.0.4 and its new features and change back in July, because this week I've been "discovering" some of them one at a time ;) -- // jack // http://www.nuthole.com From tim at pivotib.com Mon Sep 22 09:14:47 2008 From: tim at pivotib.com (Tim Glen) Date: Mon, 22 Sep 2008 09:14:47 -0400 Subject: [Backgroundrb-devel] Monit; BackgroundRB; environment variables In-Reply-To: <1222025118.9729.1.camel@shire> References: <1222025118.9729.1.camel@shire> Message-ID: <65B733DA-6C12-4C94-99D2-5B5240AADB4B@pivotib.com> Are you saying that instead of calling: `/path/to/my/app/script/backgroundrb start` from a monit script, I should be calling: `cd /path/to/my/app && script/backgroundrb start` ? Should it make any difference? if it actually does, that could be the cause of one of my problems... On 21-Sep-08, at 3:25 PM, hemant kumar wrote: > BackgrounDRb doesn't depends on any environment variable, but it > depends > on correct path (meaning, location from where you are starting > backgroundrb matters. > > On Sun, 2008-09-21 at 18:23 +0800, Ramon Miguel M. Tayag wrote: >> I started emailing monit mailing list because I started getting this >> problem where I couldn't start backgroundrb with monit anymore, >> thinking it was a monit problem. However, I back tracked a bit since >> I had added code to my project, and monit was able to start >> backgroundrb again. >> >> I seem to have zeroed in on the problem. I'm not sure where the >> problem lies - maybe monit or maybe backgroundrb, or very likely, my >> code :) >> >> The problem of monit not being able to start backgroundrb started >> when >> I added 2 lines to my Rails configuration file: >> >> "require 'lib/hodel_3000_compliant_logger'" AND "config.logger = >> Hodel3000CompliantLogger.new(config.log_path)" >> (See http://pastie.org/276618 for better formatting. I had just >> installed a logger tool.) >> >> My speculation: via monit backgroundrb tried to load the logger lib >> but can't find it. Since monit does away with environment variables, >> maybe the backgroundrb script expects some of these to be there >> (maybe >> to find where the RAILS_ROOT is). >> >> I'm stuck because I don't know what environment variables the >> backgroundrb script needs to start properly. If I know what it needs >> then I can add it to the monit start command. >> >> Thanks, > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From gethemant at gmail.com Mon Sep 22 15:58:26 2008 From: gethemant at gmail.com (hemant) Date: Tue, 23 Sep 2008 01:28:26 +0530 Subject: [Backgroundrb-devel] Monit; BackgroundRB; environment variables In-Reply-To: <65B733DA-6C12-4C94-99D2-5B5240AADB4B@pivotib.com> References: <1222025118.9729.1.camel@shire> <65B733DA-6C12-4C94-99D2-5B5240AADB4B@pivotib.com> Message-ID: On Mon, Sep 22, 2008 at 6:44 PM, Tim Glen wrote: > Are you saying that instead of calling: > `/path/to/my/app/script/backgroundrb start` > from a monit script, I should be calling: > `cd /path/to/my/app && script/backgroundrb start` ? > > Should it make any difference? if it actually does, that could be the cause > of one of my problems... No that won't make a difference (usually), but from my experience, monit screws up paths badly, if say for instance, you use relative paths to start your backgroundrb server (or any daemon basically), hence usually I specify full path whenever adding daemons to monitor in monit. When backgroundrb server is starting, it expects, File.join(File.dirname(__FILE__) + "..") to be RAILS_ROOT, thats the only fundamental constant it expects to be correct. All the [other] environment variables(that bdrb needs) it sets itself during startup. Another thing to keep in mind is, "packet_worker_runner" executable should be in PATH, when you are using monit. For interactive shells, its usually in path, but in some cases, when gems binaries are installed in non standard location (or location thats not in PATH), can cause problems. From jnutting at gmail.com Mon Sep 22 18:29:07 2008 From: jnutting at gmail.com (Jack Nutting) Date: Tue, 23 Sep 2008 00:29:07 +0200 Subject: [Backgroundrb-devel] further struggles with 1.0.4 Message-ID: So I've got my background workers up and running, but hit trouble as soon as one of them tries to talk to another. For example, I've got a worker that was created like this: MiddleMan.new_worker(:worker=>:receiver, :job_key=>:r, :data=>{:job_key=>:r}) Later, I try to get access to this, in order to asynchronously call the method "work_wrapper" on it, like this: MiddleMan.worker(:receiver, :r).async_work_wrapper() Unfortunately, this gaves me a backtrace in the debug log, with this at the top: Invalid worker with name receiver_r and key r /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.7/lib/packet/packet_connection.rb:52:in `ask_worker' (at first I encountered this while running with packet 0.1.13, and downgraded to 0.1.7 in case something had changed, but the problem is just the same) There seems to be a mismatch between what I'm passing to new_worker and later to worker. I do need to distinguish between multiple instances of some of my workers, so I do need the job_key, but I guess I'm passing it incorrectly. Any help would be appreciated! -- // jack // http://www.nuthole.com From grant.blakeman at gmail.com Mon Sep 22 21:09:59 2008 From: grant.blakeman at gmail.com (Grant Blakeman) Date: Mon, 22 Sep 2008 19:09:59 -0600 Subject: [Backgroundrb-devel] Client disconected error - trying to upgrade to latest version of backgroundrb In-Reply-To: References: <51a0cb8c0809181052t1825b5c9s77a769a7e08ad76e@mail.gmail.com> Message-ID: <51a0cb8c0809221809n3d4b6afdx54720c8dbf918a43@mail.gmail.com> Hi, I've tried it both ways (:arg & :args) and still get the same 'Client disconected' in the debug log. I also tried simplifying the call to see if any of the arguments were the problem: MiddleMan.worker(:danny_worker).async_do_work(:arg => " grant.blakeman at gmail.com") and I still get the error. It seems like the disconnect happens before it even reads anything in the worker file. I tried calling an argument that wasn't in the worker: MiddleMan.worker(:danny_worker).async_should_not_work(:arg => " grant.blakeman at gmail.com") and simply got the 'Client disconected' error. Any ideas? -gb On Thu, Sep 18, 2008 at 12:22 PM, hemant wrote: > To pass arguments you use :arg not :args. > > > On Thu, Sep 18, 2008 at 11:22 PM, Grant Blakeman > wrote: > > Hi, I've had a project using a really old version of backgroundrb (don't > > remember which one) to simply send out some e-mails in the background. > I'm > > currently trying to upgrade the project so that I can learn more about > > backgroundrb and use some of its other features. > > I've tried rewriting my worker call to work with the latest version (see > > below), but every time I try it, the debug log simply prints 'Client > > disconected' and nothing happens. Am I doing something wrong? > > In my controller: > > MiddleMan.worker(:david_worker).async_do_work(:args => { :email => > > @user.email, > > :url => @user.reset_password_code, > > :mail_type => :forgot_password_email }) > > ------------------- > > In my worker: > > set_worker_name :david_worker > > > > def create(args = nil) > > # this method is called, when worker is loaded for the first time > > end > > > > def do_work(args) > > Notifier.send("deliver_#{args[:mail_type]}", args) > > end > > Thanks in advance for any tips! > > -- > > -Grant Blakeman > > grant.blakeman at gmail.com > > http://grantblakeman.com > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org > -- -Grant Blakeman grant.blakeman at gmail.com http://grantblakeman.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kieran776 at gmail.com Mon Sep 22 22:30:37 2008 From: kieran776 at gmail.com (Kieran P) Date: Tue, 23 Sep 2008 14:30:37 +1200 Subject: [Backgroundrb-devel] Client disconected error - trying to upgrade to latest version of backgroundrb In-Reply-To: <51a0cb8c0809221809n3d4b6afdx54720c8dbf918a43@mail.gmail.com> References: <51a0cb8c0809181052t1825b5c9s77a769a7e08ad76e@mail.gmail.com> <51a0cb8c0809221809n3d4b6afdx54720c8dbf918a43@mail.gmail.com> Message-ID: Hello Grant, Out of curiosity, are you using a Mac or Linux? If Mac, try downgrading the packet gem to 0.1.7. Later versions include a raise line that I found causes similar problems for me. And if Linux, try upgrading to the latest. I believe 0.1.13 contains fixes related to packet on linux..... Regards Kieran On Tue, Sep 23, 2008 at 1:09 PM, Grant Blakeman wrote: > Hi, I've tried it both ways (:arg & :args) and still get the same 'Client > disconected' in the debug log. I also tried simplifying the call to see if > any of the arguments were the problem: > MiddleMan.worker(:danny_worker).async_do_work(:arg => " > grant.blakeman at gmail.com") > > and I still get the error. It seems like the disconnect happens before it > even reads anything in the worker file. I tried calling an argument that > wasn't in the worker: > > MiddleMan.worker(:danny_worker).async_should_not_work(:arg => " > grant.blakeman at gmail.com") > > and simply got the 'Client disconected' error. > > Any ideas? > > -gb > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Tue Sep 23 00:44:55 2008 From: gethemant at gmail.com (hemant) Date: Tue, 23 Sep 2008 10:14:55 +0530 Subject: [Backgroundrb-devel] further struggles with 1.0.4 In-Reply-To: References: Message-ID: The problem is, #new_worker, syntax has changed as well: http://backgroundrb.rubyforge.org/rails/ MiddleMan.new_worker(:worker => :foo_worker,:worker_key => "my_secret_job_key") On Tue, Sep 23, 2008 at 3:59 AM, Jack Nutting wrote: > So I've got my background workers up and running, but hit trouble as > soon as one of them tries to talk to another. For example, I've got a > worker that was created like this: > > MiddleMan.new_worker(:worker=>:receiver, :job_key=>:r, :data=>{:job_key=>:r}) > > Later, I try to get access to this, in order to asynchronously call > the method "work_wrapper" on it, like this: > > MiddleMan.worker(:receiver, :r).async_work_wrapper() > > Unfortunately, this gaves me a backtrace in the debug log, with this at the top: > > Invalid worker with name receiver_r and key r > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.7/lib/packet/packet_connection.rb:52:in > `ask_worker' > > (at first I encountered this while running with packet 0.1.13, and > downgraded to 0.1.7 in case something had changed, but the problem is > just the same) > > There seems to be a mismatch between what I'm passing to new_worker > and later to worker. I do need to distinguish between multiple > instances of some of my workers, so I do need the job_key, but I guess > I'm passing it incorrectly. Any help would be appreciated! > > -- > // jack > // http://www.nuthole.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From csarva at operative.com Tue Sep 23 10:37:41 2008 From: csarva at operative.com (Chetan Sarva) Date: Tue, 23 Sep 2008 10:37:41 -0400 Subject: [Backgroundrb-devel] No more kill() ? Message-ID: Hi, We just upgraded to 1.0.4 (from a way old version) and noticed that MetaWorker::kill() no longer exists. Is there a suggested replacement or has the error handling idiom changed? For example, we had something like this: def do_work(args) kill() if not args.to_i restore = Restore.find(args) kill() if not restore # job not found [....] The understanding being that kill() made sure that the worker process cleaned up properly. Should we now just return or raise an exception? What is the best practice here? regards, chetan From gethemant at gmail.com Tue Sep 23 11:22:00 2008 From: gethemant at gmail.com (hemant kumar) Date: Tue, 23 Sep 2008 20:52:00 +0530 Subject: [Backgroundrb-devel] No more kill() ? In-Reply-To: References: Message-ID: <1222183320.2849.0.camel@shire> On Tue, 2008-09-23 at 10:37 -0400, Chetan Sarva wrote: > Hi, > > We just upgraded to 1.0.4 (from a way old version) and noticed that > MetaWorker::kill() no longer exists. Is there a suggested replacement > or has the error handling idiom changed? For example, we had something > like this: > > def do_work(args) > kill() if not args.to_i > restore = Restore.find(args) > kill() if not restore # job not found > [....] > > The understanding being that kill() made sure that the worker process > cleaned up properly. Should we now just return or raise an exception? > What is the best practice here? > call "exit" from your workers. From grant.blakeman at gmail.com Tue Sep 23 18:51:54 2008 From: grant.blakeman at gmail.com (Grant Blakeman) Date: Tue, 23 Sep 2008 16:51:54 -0600 Subject: [Backgroundrb-devel] Client disconected error - trying to upgrade to latest version of backgroundrb In-Reply-To: References: <51a0cb8c0809181052t1825b5c9s77a769a7e08ad76e@mail.gmail.com> <51a0cb8c0809221809n3d4b6afdx54720c8dbf918a43@mail.gmail.com> Message-ID: <51a0cb8c0809231551y30629367ld628844892a30bf2@mail.gmail.com> Hi Kieran, I'm on a Mac. I was running packet 0.1.12, so I tried downgrading to 0.1.7.But I'm still getting the same behavior: an immediate "Client disconected" in the log whenever I try to call a new worker or ask an existing one to do something. Could there be more info somewhere other than the debug log that I might be able to check and see what's going on? -gb On Mon, Sep 22, 2008 at 8:30 PM, Kieran P wrote: > Hello Grant, > > Out of curiosity, are you using a Mac or Linux? > > If Mac, try downgrading the packet gem to 0.1.7. Later versions include a > raise line that I found causes similar problems for me. > > And if Linux, try upgrading to the latest. I believe 0.1.13 contains fixes > related to packet on linux..... > > Regards > Kieran > > > On Tue, Sep 23, 2008 at 1:09 PM, Grant Blakeman wrote: > >> Hi, I've tried it both ways (:arg & :args) and still get the same 'Client >> disconected' in the debug log. I also tried simplifying the call to see if >> any of the arguments were the problem: >> MiddleMan.worker(:danny_worker).async_do_work(:arg => " >> grant.blakeman at gmail.com") >> >> and I still get the error. It seems like the disconnect happens before it >> even reads anything in the worker file. I tried calling an argument that >> wasn't in the worker: >> >> MiddleMan.worker(:danny_worker).async_should_not_work(:arg => " >> grant.blakeman at gmail.com") >> >> and simply got the 'Client disconected' error. >> >> Any ideas? >> >> -gb >> > -- -Grant Blakeman grant.blakeman at gmail.com http://grantblakeman.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From grant.blakeman at gmail.com Tue Sep 23 19:11:09 2008 From: grant.blakeman at gmail.com (Grant Blakeman) Date: Tue, 23 Sep 2008 17:11:09 -0600 Subject: [Backgroundrb-devel] Client disconected error - trying to upgrade to latest version of backgroundrb In-Reply-To: <51a0cb8c0809231551y30629367ld628844892a30bf2@mail.gmail.com> References: <51a0cb8c0809181052t1825b5c9s77a769a7e08ad76e@mail.gmail.com> <51a0cb8c0809221809n3d4b6afdx54720c8dbf918a43@mail.gmail.com> <51a0cb8c0809231551y30629367ld628844892a30bf2@mail.gmail.com> Message-ID: <51a0cb8c0809231611t53c17c89q3c7c633097a22be7@mail.gmail.com> I take it back. Downgrading the packet version does seem to have helped. It was still showing "Client disconected" in the debug log, but in my development log I can see it's now executing the code in the worker file. Yay. Thanks for the help! -gb On Tue, Sep 23, 2008 at 4:51 PM, Grant Blakeman wrote: > Hi Kieran, > I'm on a Mac. I was running packet 0.1.12, so I tried downgrading to > 0.1.7. But I'm still getting the same behavior: an immediate "Client > disconected" in the log whenever I try to call a new worker or ask an > existing one to do something. > > Could there be more info somewhere other than the debug log that I might be > able to check and see what's going on? > > -gb > > > On Mon, Sep 22, 2008 at 8:30 PM, Kieran P wrote: > >> Hello Grant, >> >> Out of curiosity, are you using a Mac or Linux? >> >> If Mac, try downgrading the packet gem to 0.1.7. Later versions include a >> raise line that I found causes similar problems for me. >> >> And if Linux, try upgrading to the latest. I believe 0.1.13 contains fixes >> related to packet on linux..... >> >> Regards >> Kieran >> >> >> On Tue, Sep 23, 2008 at 1:09 PM, Grant Blakeman > > wrote: >> >>> Hi, I've tried it both ways (:arg & :args) and still get the same 'Client >>> disconected' in the debug log. I also tried simplifying the call to see if >>> any of the arguments were the problem: >>> MiddleMan.worker(:danny_worker).async_do_work(:arg => " >>> grant.blakeman at gmail.com") >>> >>> and I still get the error. It seems like the disconnect happens before it >>> even reads anything in the worker file. I tried calling an argument that >>> wasn't in the worker: >>> >>> MiddleMan.worker(:danny_worker).async_should_not_work(:arg => " >>> grant.blakeman at gmail.com") >>> >>> and simply got the 'Client disconected' error. >>> >>> Any ideas? >>> >>> -gb >>> >> > > > -- > -Grant Blakeman > grant.blakeman at gmail.com > http://grantblakeman.com > -- -Grant Blakeman grant.blakeman at gmail.com http://grantblakeman.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From csarva at operative.com Tue Sep 23 19:35:49 2008 From: csarva at operative.com (Chetan Sarva) Date: Tue, 23 Sep 2008 19:35:49 -0400 Subject: [Backgroundrb-devel] Persistent queue and scheduled jobs Message-ID: <29F1C6AB-DB52-4A2C-B0F3-7C844DC13533@operative.com> Hi, While playing with scheduled jobs, I ran into a couple of issues and made a few notes. 1) Times MUST be UTC: I've got the following: BdrbJobQueue.insert_job(:worker_name => 'backup_worker', :worker_method => 'do_work', :args => @backup.id, :scheduled_at => Time.now + 30) resulting in: submitted_at: 2008-09-23 17:57:26 started_at: 2008-09-23 17:57:30 finished_at: NULL archived_at: NULL scheduled_at: 2008-09-23 17:57:56 changing it to Time.now.utc + 30 fixes it. 2. job_key is mandatory when using the persistent queue since there is a unique key check on this field. This was not immediately obvious, and no key is automatically generated. The above code inserts NULL for job_key. I can submit doc patches for the above if that helps. 3. When firing the job at the scheduled time, MetaWorker calls Marshal.load() on `args` though they were not previously marshalled by ActiveRecord. I made the following change in my WC in order to move passed this problem: Index: vendor/plugins/backgroundrb/server/lib/meta_worker.rb =================================================================== --- vendor/plugins/backgroundrb/server/lib/meta_worker.rb (revision 114) +++ vendor/plugins/backgroundrb/server/lib/meta_worker.rb (working copy) @@ -287,10 +287,9 @@ Thread.current[:persistent_job_id] = task[:id] Thread.current[:job_key] = task[:job_key] called_method_arity = self.method(task.worker_method).arity - args = load_data(task.args) begin if called_method_arity != 0 - self.send(task.worker_method,args) + self.send(task.worker_method,task.args) else self.send(task.worker_method) end regards, chetan From curtis.hatter at insightbb.com Wed Sep 24 11:29:28 2008 From: curtis.hatter at insightbb.com (Mitchell Curtis Hatter) Date: Wed, 24 Sep 2008 11:29:28 -0400 Subject: [Backgroundrb-devel] Clearing cached results Message-ID: <5144A692-3A9A-4093-82B4-DE550A30EEA7@insightbb.com> I'm trying to figure out how to clear the for a worker from my controller. In my aysnc_xxx process I set cache[key]. However, I don't see an easy way to clear the cache from a controller with invoking something on the worker. What I was looking for was something like MiddleMan(:worker_name, :worker_key_name).cache[key]= Is there something like that and I'm just missing it or do I need to create a "clear_cache" function on my worker and call that when I need to clear the cache? Thanks, Curtis From csarva at operative.com Wed Sep 24 12:03:54 2008 From: csarva at operative.com (Chetan Sarva) Date: Wed, 24 Sep 2008 12:03:54 -0400 Subject: [Backgroundrb-devel] Client disconected error - trying to upgrade to latest version of backgroundrb In-Reply-To: <51a0cb8c0809231611t53c17c89q3c7c633097a22be7@mail.gmail.com> References: <51a0cb8c0809181052t1825b5c9s77a769a7e08ad76e@mail.gmail.com> <51a0cb8c0809221809n3d4b6afdx54720c8dbf918a43@mail.gmail.com> <51a0cb8c0809231551y30629367ld628844892a30bf2@mail.gmail.com> <51a0cb8c0809231611t53c17c89q3c7c633097a22be7@mail.gmail.com> Message-ID: <7D3C92B1-0672-4CE0-9902-D5A38D87797C@operative.com> For the record, I see the same behaviour on linux. "Client disconnected" in debug, but workers execute successfully. I tried grepping for that string in bdrb and packet, but it's not there or I'm looking in the wrong place... On Sep 23, 2008, at 7:11 PM, Grant Blakeman wrote: > I take it back. Downgrading the packet version does seem to have > helped. It was still showing "Client disconected" in the debug log, > but in my development log I can see it's now executing the code in > the worker file. Yay. > > Thanks for the help! > > -gb > > On Tue, Sep 23, 2008 at 4:51 PM, Grant Blakeman > wrote: > Hi Kieran, > > I'm on a Mac. I was running packet 0.1.12, so I tried downgrading to > MailScanner warning: numerical links are often malicious: 0.1.7. But > I'm still getting the same behavior: an immediate "Client > disconected" in the log whenever I try to call a new worker or ask > an existing one to do something. > > Could there be more info somewhere other than the debug log that I > might be able to check and see what's going on? > > -gb > > > On Mon, Sep 22, 2008 at 8:30 PM, Kieran P wrote: > Hello Grant, > > Out of curiosity, are you using a Mac or Linux? > > If Mac, try downgrading the packet gem to MailScanner warning: > numerical links are often malicious: 0.1.7. Later versions include a > raise line that I found causes similar problems for me. > > And if Linux, try upgrading to the latest. I believe 0.1.13 contains > fixes related to packet on linux..... > > Regards > Kieran > > > > On Tue, Sep 23, 2008 at 1:09 PM, Grant Blakeman > wrote: > Hi, I've tried it both ways (:arg & :args) and still get the same > 'Client disconected' in the debug log. I also tried simplifying the > call to see if any of the arguments were the problem: > > MiddleMan.worker(:danny_worker).async_do_work(:arg => "grant.blakeman at gmail.com > ") > > and I still get the error. It seems like the disconnect happens > before it even reads anything in the worker file. I tried calling an > argument that wasn't in the worker: > > MiddleMan.worker(:danny_worker).async_should_not_work(:arg => "grant.blakeman at gmail.com > ") > > and simply got the 'Client disconected' error. > > Any ideas? > > -gb > > > > -- > -Grant Blakeman > grant.blakeman at gmail.com > http://grantblakeman.com > > > > -- > -Grant Blakeman > grant.blakeman at gmail.com > http://grantblakeman.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Wed Sep 24 13:02:50 2008 From: gethemant at gmail.com (hemant) Date: Wed, 24 Sep 2008 22:32:50 +0530 Subject: [Backgroundrb-devel] Client disconected error - trying to upgrade to latest version of backgroundrb In-Reply-To: <7D3C92B1-0672-4CE0-9902-D5A38D87797C@operative.com> References: <51a0cb8c0809181052t1825b5c9s77a769a7e08ad76e@mail.gmail.com> <51a0cb8c0809221809n3d4b6afdx54720c8dbf918a43@mail.gmail.com> <51a0cb8c0809231551y30629367ld628844892a30bf2@mail.gmail.com> <51a0cb8c0809231611t53c17c89q3c7c633097a22be7@mail.gmail.com> <7D3C92B1-0672-4CE0-9902-D5A38D87797C@operative.com> Message-ID: That message is nothing to worry, since client usually clients and disconnects for sending a message and that message is because of that. (think some print statement is left somewhere). On Wed, Sep 24, 2008 at 9:33 PM, Chetan Sarva wrote: > For the record, I see the same behaviour on linux. "Client disconnected" in > debug, but workers execute successfully. I tried grepping for that string in > bdrb and packet, but it's not there or I'm looking in the wrong place... > On Sep 23, 2008, at 7:11 PM, Grant Blakeman wrote: > > I take it back. Downgrading the packet version does seem to have helped. It > was still showing "Client disconected" in the debug log, but in my > development log I can see it's now executing the code in the worker file. > Yay. > Thanks for the help! > -gb > > On Tue, Sep 23, 2008 at 4:51 PM, Grant Blakeman > wrote: >> >> Hi Kieran, >> I'm on a Mac. I was running packet 0.1.12, so I tried downgrading to >> MailScanner has detected a possible fraud attempt from "0.1.7" claiming to >> be MailScanner warning: numerical links are often malicious: 0.1.7. But I'm >> still getting the same behavior: an immediate "Client disconected" in the >> log whenever I try to call a new worker or ask an existing one to do >> something. >> Could there be more info somewhere other than the debug log that I might >> be able to check and see what's going on? >> -gb >> >> On Mon, Sep 22, 2008 at 8:30 PM, Kieran P wrote: >>> >>> Hello Grant, >>> >>> Out of curiosity, are you using a Mac or Linux? >>> >>> If Mac, try downgrading the packet gem to MailScanner has detected a >>> possible fraud attempt from "0.1.7" claiming to be MailScanner warning: >>> numerical links are often malicious: 0.1.7. Later versions include a raise >>> line that I found causes similar problems for me. >>> >>> And if Linux, try upgrading to the latest. I believe 0.1.13 contains >>> fixes related to packet on linux..... >>> >>> Regards >>> Kieran >>> >>> >>> On Tue, Sep 23, 2008 at 1:09 PM, Grant Blakeman >>> wrote: >>>> >>>> Hi, I've tried it both ways (:arg & :args) and still get the same >>>> 'Client disconected' in the debug log. I also tried simplifying the call to >>>> see if any of the arguments were the problem: >>>> MiddleMan.worker(:danny_worker).async_do_work(:arg => >>>> "grant.blakeman at gmail.com") >>>> and I still get the error. It seems like the disconnect happens before >>>> it even reads anything in the worker file. I tried calling an argument that >>>> wasn't in the worker: >>>> MiddleMan.worker(:danny_worker).async_should_not_work(:arg => >>>> "grant.blakeman at gmail.com") >>>> and simply got the 'Client disconected' error. >>>> Any ideas? >>>> -gb >> >> >> -- >> -Grant Blakeman >> grant.blakeman at gmail.com >> http://grantblakeman.com > > > > -- > -Grant Blakeman > grant.blakeman at gmail.com > http://grantblakeman.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Wed Sep 24 13:05:34 2008 From: gethemant at gmail.com (hemant) Date: Wed, 24 Sep 2008 22:35:34 +0530 Subject: [Backgroundrb-devel] Persistent queue and scheduled jobs In-Reply-To: <29F1C6AB-DB52-4A2C-B0F3-7C844DC13533@operative.com> References: <29F1C6AB-DB52-4A2C-B0F3-7C844DC13533@operative.com> Message-ID: Thanks, Chetan for the patch, I will be commiting them soonish. On Wed, Sep 24, 2008 at 5:05 AM, Chetan Sarva wrote: > Hi, > > While playing with scheduled jobs, I ran into a couple of issues and made a > few notes. > > > 1) Times MUST be UTC: > > I've got the following: > > BdrbJobQueue.insert_job(:worker_name => 'backup_worker', > :worker_method => 'do_work', > :args => @backup.id, > :scheduled_at => Time.now + 30) > > > resulting in: > > submitted_at: 2008-09-23 17:57:26 > started_at: 2008-09-23 17:57:30 > finished_at: NULL > archived_at: NULL > scheduled_at: 2008-09-23 17:57:56 > > > changing it to Time.now.utc + 30 fixes it. > > > 2. job_key is mandatory when using the persistent queue since there is a > unique key check on this field. This was not immediately obvious, and no key > is automatically generated. The above code inserts NULL for job_key. > > > I can submit doc patches for the above if that helps. > > > 3. When firing the job at the scheduled time, MetaWorker calls > Marshal.load() on `args` though they were not previously marshalled by > ActiveRecord. I made the following change in my WC in order to move passed > this problem: > > Index: vendor/plugins/backgroundrb/server/lib/meta_worker.rb > =================================================================== > --- vendor/plugins/backgroundrb/server/lib/meta_worker.rb (revision > 114) > +++ vendor/plugins/backgroundrb/server/lib/meta_worker.rb (working > copy) > @@ -287,10 +287,9 @@ > Thread.current[:persistent_job_id] = task[:id] > Thread.current[:job_key] = task[:job_key] > called_method_arity = self.method(task.worker_method).arity > - args = load_data(task.args) > begin > if called_method_arity != 0 > - self.send(task.worker_method,args) > + self.send(task.worker_method,task.args) > else > self.send(task.worker_method) > end > > > > regards, > > chetan > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Wed Sep 24 17:07:18 2008 From: gethemant at gmail.com (hemant kumar) Date: Thu, 25 Sep 2008 02:37:18 +0530 Subject: [Backgroundrb-devel] Clearing cached results In-Reply-To: <5144A692-3A9A-4093-82B4-DE550A30EEA7@insightbb.com> References: <5144A692-3A9A-4093-82B4-DE550A30EEA7@insightbb.com> Message-ID: <1222290438.18994.1.camel@shire> Add following to rails_worker_proxy.rb file: diff --git a/lib/backgroundrb/rails_worker_proxy.rb b/lib/backgroundrb/rails_worker_proxy.rb index 0205f07..2b0a080 100644 --- a/lib/backgroundrb/rails_worker_proxy.rb +++ b/lib/backgroundrb/rails_worker_proxy.rb @@ -105,6 +105,13 @@ module BackgrounDRb middle_man.cache[gen_key(options)] end + def reset_memcache_result(job_key,value) + options = compact(:worker => worker_name,:worker_key => worker_key,:job_key => job_key) + key = gen_key(options) + middle_man.cache[key] = value + value + end + def return_result result result = Array(result) result.size <= 1 ? result[0] : result And use it with: MiddleMan.worker(:foo_worker).reset_memcache_result("key",value) It will be checked in soonish (need to write testcases and stuff) On Wed, 2008-09-24 at 11:29 -0400, Mitchell Curtis Hatter wrote: > I'm trying to figure out how to clear the for a worker from my > controller. > > In my aysnc_xxx process I set cache[key]. However, I don't see an easy > way to clear the cache from a controller with invoking something on > the worker. > > What I was looking for was something like > > MiddleMan(:worker_name, :worker_key_name).cache[key]= > > Is there something like that and I'm just missing it or do I need to > create a "clear_cache" function on my worker and call that when I need > to clear the cache? > > Thanks, > Curtis > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From csarva at operative.com Thu Sep 25 13:53:47 2008 From: csarva at operative.com (Chetan Sarva) Date: Thu, 25 Sep 2008 13:53:47 -0400 Subject: [Backgroundrb-devel] Persistent queue and scheduled jobs In-Reply-To: References: <29F1C6AB-DB52-4A2C-B0F3-7C844DC13533@operative.com> Message-ID: <7F64EDDF-A6F2-4BA8-986D-D43F278C6AE8@operative.com> Note that my patch below was for the case where you have a single, simple argument like a string or int, as it is in our case. If an array or hash is passed, I'm not sure what the behaviour would be. On Sep 24, 2008, at 1:05 PM, hemant wrote: > Thanks, Chetan for the patch, I will be commiting them soonish. > > On Wed, Sep 24, 2008 at 5:05 AM, Chetan Sarva > wrote: >> Hi, >> >> While playing with scheduled jobs, I ran into a couple of issues >> and made a >> few notes. >> >> >> 1) Times MUST be UTC: >> >> I've got the following: >> >> BdrbJobQueue.insert_job(:worker_name => 'backup_worker', >> :worker_method => 'do_work', >> :args => @backup.id, >> :scheduled_at => Time.now + 30) >> >> >> resulting in: >> >> submitted_at: 2008-09-23 17:57:26 >> started_at: 2008-09-23 17:57:30 >> finished_at: NULL >> archived_at: NULL >> scheduled_at: 2008-09-23 17:57:56 >> >> >> changing it to Time.now.utc + 30 fixes it. >> >> >> 2. job_key is mandatory when using the persistent queue since there >> is a >> unique key check on this field. This was not immediately obvious, >> and no key >> is automatically generated. The above code inserts NULL for job_key. >> >> >> I can submit doc patches for the above if that helps. >> >> >> 3. When firing the job at the scheduled time, MetaWorker calls >> Marshal.load() on `args` though they were not previously marshalled >> by >> ActiveRecord. I made the following change in my WC in order to move >> passed >> this problem: >> >> Index: vendor/plugins/backgroundrb/server/lib/meta_worker.rb >> =================================================================== >> --- vendor/plugins/backgroundrb/server/lib/meta_worker.rb >> (revision >> 114) >> +++ vendor/plugins/backgroundrb/server/lib/meta_worker.rb >> (working >> copy) >> @@ -287,10 +287,9 @@ >> Thread.current[:persistent_job_id] = task[:id] >> Thread.current[:job_key] = task[:job_key] >> called_method_arity = self.method(task.worker_method).arity >> - args = load_data(task.args) >> begin >> if called_method_arity != 0 >> - self.send(task.worker_method,args) >> + self.send(task.worker_method,task.args) >> else >> self.send(task.worker_method) >> end >> >> >> >> regards, >> >> chetan >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> > > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org From mrryancase at gmail.com Fri Sep 26 19:30:47 2008 From: mrryancase at gmail.com (Ryan Case) Date: Fri, 26 Sep 2008 19:30:47 -0400 Subject: [Backgroundrb-devel] trouble stopping backgroundrb In-Reply-To: <1221689332.16986.5.camel@shire> References: <983e153a0809160911h3728361epc252d892de88bb98@mail.gmail.com> <48D131D9.9000703@nooked.com> <1221689332.16986.5.camel@shire> Message-ID: Thanks for the patch - this works much better. Occasionally, I still have to "pkill -9 -f backgroundrb", but most of the time just the stop script will clean up when one of the packet_worker processes dies. Thanks, Ryan On Sep 17, 2008, at 6:08 PM, hemant kumar wrote: > Okay folks here is a patch to "backgroundrb" script, which should fix > some issues: > > diff --git a/script/backgroundrb b/script/backgroundrb > index dabf80b..8d4bb78 100755 > --- a/script/backgroundrb > +++ b/script/backgroundrb > @@ -49,18 +49,9 @@ when 'stop' > def kill_process arg_pid_file > pid = nil > File.open(arg_pid_file, "r") { |pid_handle| pid = > pid_handle.gets.strip.chomp.to_i } > - begin > - pgid = Process.getpgid(pid) > - Process.kill('TERM', pid) > - Process.kill('-TERM', pgid) > - Process.kill('KILL', pid) > - rescue Errno::ESRCH => e > - puts "Deleting pid file" > - rescue > - puts $! > - ensure > - File.delete(arg_pid_file) if File.exists?(arg_pid_file) > - end > + pgid = Process.getpgid(pid) > + Process.kill('-TERM', pgid) > + File.delete(arg_pid_file) if File.exists?(arg_pid_file) > end > pid_files = Dir["#{RAILS_HOME}/tmp/pids/backgroundrb_*.pid"] > pid_files.each { |x| kill_process(x) } > > What it does is: > 1. Deleting by group id is enough for master process. > 2. Do not delete the pid file if, there was an exception while > stopping > the daemon. > 3. Do not handle exceptions silently. > > Please try this and let me know, how it goes. > > > > On Wed, 2008-09-17 at 17:35 +0100, John O'Shea wrote: >> Jonathan, >> Glad you raised this, I've been spending some time trying to >> diagnose this exact same problem. >> The exception handling code in the "when 'stop'" block (in >> script/backgroundrb) could definitely could be improved somewhat >> - check that the process with 'pid' exists before trying to kill it >> - rescue permission exceptions (Errno::EPERM) >> - only delete the pid file if the process pid does not still exist >> (in >> ensure block) >> - be a little more verbose to stdout/stderr >> >> While we are on the subject of shutdown, - when the backgroundrb >> process >> gets a HUP signal does it wait for existing workers to complete any >> work >> methods that are executing or is the 'Process.kill('-TERM', pgid)' >> call >> intended to make the OS handle this? >> >> We use capistrano to deploy our application (stopping and restarting >> backgroundrb after the rails app has been updated). It would be >> great >> if we could have more predictability regarding shutting down >> backgroundrb (i.e. have the backgroundrb disable the reactor loop in >> idle workers and wait for all active workers to finish methods, then >> shutdown"). >> >> John. >> >> Jonathan Wallace wrote: >>> Hi Ryan, >>> >>> I recently ran into the same issue where the backgroundrb process >>> would not respond to ./script/backgroundrb stop command. The pid >>> file >>> was being deleted but the actual process was not being killed. I'm >>> running packet 0.1.12 on gentoo. >>> >>> I'm not exactly sure what conditions put backgroundrb into such a >>> state but I've decided to modify the script/backgroundrb to behave a >>> little differently. >>> >>> My hypothesis is that if one of the Process.kill method calls in >>> script/backgroundrb raises an exception, the pid file is deleted >>> even >>> though the kill signal is never sent. At this point, running >>> starting >>> and stopping backgroundrb never affects the original still running >>> backgroundrb process. >>> >>> There are a couple of reasons that I believe an exception could be >>> raised. Either the Process.getpgid(pid), Process.kill('TERM', >>> pid) or >>> the PRocess.kill('-TERM', pgid) raise an exception or the effective >>> uid of the user running script/backgroundrb stop does not have >>> permission to kill those processes. >>> >>> To fix this, we've removed the Process.getpgid and the two >>> Process.kill's that are sending the TERM signal. Since we've >>> architected our backgroundrb jobs to be persistent and idempotent (a >>> db backed queue written before the feature appeared in bdrb), we'll >>> just use the KILL signal. >>> >>> Thoughts? >>> >>> Thanks, >>> Jonathan >>> >>> On Tue, Sep 16, 2008 at 12:11 PM, Ryan Case >>> wrote: >>> >>>> Hi folks - >>>> >>>> I'm having trouble getting backgroundrb to stop after one of the >>>> packet_worker_r processes dies. >>>> >>>> If backgroundrb is running properly, >>>> "/path/to/application/script/backgroundrb stop" works fine, but >>>> often >>>> one of the packet_worker_r processes dies, and the stop command no >>>> longer works after that (it runs, but it does not stop the >>>> processes, >>>> and so then start doesn't work). >>>> >>>> The only thing that seems to work at that point is to manually kill >>>> the processes that are still running, and then the start works, but >>>> that is going to make restarting via monit a lot less clean. >>>> >>>> Any ideas would be much appreciated! >>>> >>>> I'm using github version of backgroundrb, and packet 0.1.13 >>>> running on ubuntu. >>>> >>>> Thanks! >>>> Ryan >>>> _______________________________________________ >>>> Backgroundrb-devel mailing list >>>> Backgroundrb-devel at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>>> >>>> >>> _______________________________________________ >>> Backgroundrb-devel mailing list >>> Backgroundrb-devel at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >> >> > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From alias at pet-theory.com Mon Sep 29 11:23:03 2008 From: alias at pet-theory.com (Matt Garland) Date: Mon, 29 Sep 2008 08:23:03 -0700 Subject: [Backgroundrb-devel] periodic votes--threads or workers? Message-ID: <9EEFC139-D9FF-4B32-B53E-A5190BC1A48C@pet-theory.com> Hi, I'm just getting started with BackgroundRB, but I'm not sure how to do what I need to do. I have many chat rooms (Jabber) which are controlled or mediated by a Rails app. There are periodic elections carried out in these rooms. Just this would be easy. I could set up a chron job in the configuration which gets all the chat rooms from Rails and calls chat.do_election on each (Chat is a model). However, it is more complicated. Often there is not enough members to vote: no quorum. Sometimes the chatters can "recall" a winner with an early vote. So each chat room is essentially it's own story, with its own chronology. I know the signature of the election_worker class: start_all_elections()# for create trigger_election(chat_name)# tell Rails Model chat to do election with individual chat stop_voting(chat_name)# Rails tells election_worker no quorum start_voting(chat_name)# quorum is reached skip_to_vote(chat_name)# vote is taken immediately, and clock reset, when Rails detects a recall What I DO NOT know is how to deal with these multiple elections. Are these THREADS inside the election_worker class? If so, how would I set these up? (I'm new to threads). Or are they multiple instances of election_worker? (Where then would I create multiple instances of the worker, since Rails has no "entry point"?) Thanks, any help appreciated-- Matt