From nate at natemurray.com Thu May 3 18:27:34 2007 From: nate at natemurray.com (Nathan Murray) Date: Thu, 3 May 2007 15:27:34 -0700 Subject: [Backgroundrb-devel] Reloadable workers? (Rails Testing without DRb restart?) Message-ID: <8798EA3C-0269-48E8-B6D0-58BA84928C1F@natemurray.com> Hey guys, I just opened up my first installation of BackgrounDRb and so far I love it. I'll have to share this plugin at my next Ruby Brigade meeting! I have one question though: Is there a way that I can run the BackgrounDRb server in a "test" mode that will reload my changes without having to restart every time? (Similar to how mongrel does this with my ActiveRecords in development mode.) Currently I'm restarting my BackgrounDrb server everytime I make a change and need to test it. I'm guessing someone else has found a better way? Thanks, Nate Murray http://pasadenarb.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070503/2521e7e1/attachment.html From skaar at waste.org Fri May 4 10:47:53 2007 From: skaar at waste.org (skaar) Date: Fri, 4 May 2007 09:47:53 -0500 Subject: [Backgroundrb-devel] Reloadable workers? (Rails Testing without DRb restart?) In-Reply-To: <8798EA3C-0269-48E8-B6D0-58BA84928C1F@natemurray.com> References: <8798EA3C-0269-48E8-B6D0-58BA84928C1F@natemurray.com> Message-ID: <20070504144752.GA308@waste.org> * Nathan Murray (nate at natemurray.com) [070503 17:41]: > Hey guys, I just opened up my first installation of BackgrounDRb and so > far I love it. I'll have to share this plugin at my next Ruby Brigade > meeting! > I have one question though: Is there a way that I can run the BackgrounDRb > server in a "test" mode that will reload my changes without having to > restart every time? (Similar to how mongrel does this with my > ActiveRecords in development mode.) Currently I'm restarting my > BackgrounDrb server everytime I make a change and need to test it. I'm > guessing someone else has found a better way? Thanks, > Nate Murray > [1]http://pasadenarb.com If you connect with the console you can already issue a 'reload!' command. Note that this will not "update" currently running workers, but it does allow you to reload your worker classes without having to restart your server. /skaar > > References > > Visible links > 1. http://pasadenarb.com/ > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From nate at natemurray.com Fri May 4 15:41:32 2007 From: nate at natemurray.com (Nathan Murray) Date: Fri, 4 May 2007 12:41:32 -0700 Subject: [Backgroundrb-devel] Setting environment Message-ID: <2FFCEAEC-A4C2-41AA-B9DC-5ACD33436C32@natemurray.com> > Chris T wrote: > > Not sure this is known behaviour but it seems that if want to set > the > > environment to, for example, test, and you want to use Active > Record you > > have to both explicitly set the RAILS_ENV and the BackgrounDRb > environment. > > I ran into essentially the same problem (needing to run BRB in the > test > environment so AR would use the right DB). My "solution" was to > add entries > in backgroundrb.yml for environment, log_file and pid_file to allow > for > multiple instances of BRB to run simultaneously in different > environments. My > config now looks like: > > environment: <%= ENV['RAILS_ENV'] %> > port: <%= 22222 + ['production', 'development', 'test'].index(ENV > ['RAILS_ENV']) %> > log_file: <%= "log/brb-#{ENV['RAILS_ENV']}.log" %> > pid_file: <%= "log/brb-#{ENV['RAILS_ENV']}.pid" %> > ... > > Obviously, I had to hack BRB to use these settings, modify the > start/stop > scripts to take an --env argument, and add Rake tasks to (re)start > the test > instance. Add that restart task as a preqreq of 'test', and the > BRB test > instance will restart (and reload) before each test run. > > If interested, I can provide diffs ... > > -- Steve Steve, I'm interested in the diffs. I'd like to the same thing. Nate Murray http://pasadenarb.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070504/c8ee1d9f/attachment.html From nate at natemurray.com Mon May 7 16:13:33 2007 From: nate at natemurray.com (Nathan Murray) Date: Mon, 7 May 2007 13:13:33 -0700 Subject: [Backgroundrb-devel] Daemons closes files? (Breaks logging while testing) In-Reply-To: <2FFCEAEC-A4C2-41AA-B9DC-5ACD33436C32@natemurray.com> References: <2FFCEAEC-A4C2-41AA-B9DC-5ACD33436C32@natemurray.com> Message-ID: <5EE4D69D-5C5E-4F46-BC98-EEF4D6D0EA47@natemurray.com> Hey guys, I'm working really hard to get the unit tests for Workers to work with Rails. I'm hoping I can figure it out and submit a patch. I've made really great progress in so far as I can run basic unit tests. I am starting up the BRB server just for the test and then bringing it back down. I think this makes the most sense for an accurate test and it isn't very slow. The biggest issue I am having is that ActiveRecord::Base.logger's file descriptor keeps getting closed and throwing tons of errors anytime I try to do anything. (As a side note, the DB connection seems to be rolling back before my workers have a chance to get at it, but I think that's a different issue.) I think this is because Daemons is closing the file, but I'm not exactly sure. From the Daemons documentation: ... 6. Closes file descriptors (reopens STDOUT and STDERR to point to a logfile if possible). ... I'm guessing that Daemons is rightly trying to be its own independent process but while doing that it seems to close the Rails logger. Does anyone have any suggestions on how to reconcile these issues? Testing is very important to us here as this is a production-level business app. I really love the idea behind BackgrounDRb and I want to contribute my changes to the community so any help would be greatly appreciated! Nate Murray http://pasadenarb.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070507/ffc000cb/attachment-0001.html From nate at natemurray.com Mon May 7 20:02:51 2007 From: nate at natemurray.com (Nathan Murray) Date: Mon, 7 May 2007 17:02:51 -0700 Subject: [Backgroundrb-devel] [PATCH] Unit Testing Rails Workers - It works! Message-ID: <65E98CC0-A783-4F00-A2FD-45364307A215@natemurray.com> Hey guys, I've spent the last two days getting Unit Testing for Rails Workers to work. I've posted all the details at this ticket: http://backgroundrb.devjavu.com/projects/backgroundrb/ticket/53 Nate Murray http://pasadenarb.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070507/f27214e5/attachment.html From twr at frii.com Wed May 9 00:11:36 2007 From: twr at frii.com (todd Richmond) Date: Tue, 08 May 2007 22:11:36 -0600 Subject: [Backgroundrb-devel] Neophyte needs help! Message-ID: <1178683896.3225.149.camel@localhost.localdomain> Hi All, Real neophyte here, to both backgroundrb and ruby/rails in general. (Hey, we've all got to start somewhere, right!) :) I've just been playing around with the example code at http://backgroundrb.rubyforge.org/ , and trying to get all the pieces to work (btw, I'm using version 0.2.1). I did a 'generate worker example', which created an example_worker.rb file in my ./libs/workers directory. I fleshed it out so it looks like this (as per the example in the documentation): ##################################################################### class ExampleWorker < BackgrounDRb::Worker::RailsBase def do_work(args) logger.info('ExampleWorker do work') results[:do_work_time] = Time.now.to_s results[:done_with_do_work] ||= true end def other_method logger.info('other_method in ExampleWorker called') results[:extra_data] = "Just a plain old string" end def arg_method(arg) end end ExampleWorker.register ##################################################################### Next, I created a 'backgroundrb_example' file under ./script. It looks like this: ##################################################################### #!/usr/bin/env ruby require File.dirname(__FILE__) + '/../config/boot' require "#{RAILS_ROOT}/config/environment" key = MiddleMan.new_worker(:class => :example_worker) worker = MiddleMan.worker(key) worker.other_method worker.delete ##################################################################### I also started the backgroundrb server, and it and the other processes seem to be running fine: todd 6865 0.0 0.6 16484 13372 ? S 21:20 0:00 backgroundrb todd 6866 0.0 0.6 16320 13168 ? S 21:20 0:00 backgroundrb_logger todd 6867 0.0 0.6 16520 13384 ? S 21:20 0:00 backgroundrb_results My config file looks like this: :host: localhost :port: 22222 :worker_dir: lib/workers :rails_env: development :pool_size: 15 :acl: :deny: all :allow: localhost 127.0.0.1 :order: deny allow When I try to run the example, I get a NameError because it can't seem to find my ExampleWorker class: ruby backgroundrb_example (drbunix:///tmp/backgroundrbunix_localhost_22222) /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/dependencies.rb:266:in `load_missing_constant': uninitialized constant ExampleWorker (NameError) from (drbunix:///tmp/backgroundrbunix_localhost_22222) /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/dependencies.rb:452:in `const_missing' from (drbunix:///tmp/backgroundrbunix_localhost_22222) /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/dependencies.rb:464:in `const_missing' from (drbunix:///tmp/backgroundrbunix_localhost_22222) /home/todd/devel/pedperf/trunk/site_app/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:453:in `worker_klass_constant' from (drbunix:///tmp/backgroundrbunix_localhost_22222) /home/todd/devel/pedperf/trunk/site_app/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:178:in `new_worker' from (drbunix:///tmp/backgroundrbunix_localhost_22222) /usr/local/lib/ruby/1.8/drb/drb.rb:1552:in `perform_without_block' from (drbunix:///tmp/backgroundrbunix_localhost_22222) /usr/local/lib/ruby/1.8/drb/drb.rb:1512:in `perform' from (drbunix:///tmp/backgroundrbunix_localhost_22222) /usr/local/lib/ruby/1.8/drb/drb.rb:1586:in `main_loop' from (drbunix:///tmp/backgroundrbunix_localhost_22222) /usr/local/lib/ruby/1.8/drb/drb.rb:1582:in `main_loop' ... 13 levels... from (drbunix:///tmp/backgroundrbunix_localhost_22222) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in `run_proc' from (drbunix:///tmp/backgroundrbunix_localhost_22222) /home/todd/devel/pedperf/trunk/site_app/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in `run' from (drbunix:///tmp/backgroundrbunix_localhost_22222) ./script/backgroundrb:29 from backgroundrb_example:6 Any help would be greatly appreciated! Thanks, Todd Richmond -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070508/745cdaba/attachment.html From eden at mojiti.com Thu May 10 09:28:46 2007 From: eden at mojiti.com (Eden Li) Date: Thu, 10 May 2007 21:28:46 +0800 Subject: [Backgroundrb-devel] Password required for dejavu repo In-Reply-To: References: Message-ID: <1dd361e10705100628q721197d4pbce2338fed2de407@mail.gmail.com> I just got this myself. I've got svn:externals pointing to a backgroundrb from there and it's blocking my capistrano scripts. Has devjavu changed their password policy? On 4/4/07, Saimon Moore wrote: > Since this morning, the server at http://svn.devjavu.com/backgroundrb > is requiring authorized access. > > Could you guys look into it? > > Thanks, > > Saimon > > -- > Saimon Moore > Freelance Web Developer > (Available for hire - For details visit http://saimonmoore.net) > > Skype: saimonmoore > Yahoo IM: saimonmoore > Google IM: saimonmoore > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From Peer.Allan at canadadrugs.com Fri May 11 11:30:10 2007 From: Peer.Allan at canadadrugs.com (Peer Allan) Date: Fri, 11 May 2007 10:30:10 -0500 Subject: [Backgroundrb-devel] Scheduled Workers Message-ID: Hello all, I think I am having a problem with a scheduled worker. I believe I have it configured correctly to run on a cron schedule each morning at 1:50am. I start up the server and it runs correctly the first time and sometimes a second time, but after that it doesn't seem to run again. The log shows no activity and I have a link in my rails app to start it manually and that doesn't work either. To get it going again I have to restart the server. I don't know how else to explain what is happening, but will be happy to answer any questions if it will help. Just for starters here is my YML file that configures the schedules My_worker: :class: :my_worker_class :job_key: :my_job_key :trigger_args: 0 50 1 * * * * Thanks. Peer Allan Sr. Web Developer, e-Business CanadaDrugs.com Group of Companies 16 Terracon Place Winnipeg, MB, Canada R2J 4G7 Phone: (204) 654-7951 Cell: (204) 781-8719 Fax: (204) 654-7910 www.canadadrugs.com From jodi at nnovation.ca Mon May 14 11:52:19 2007 From: jodi at nnovation.ca (Jodi Showers) Date: Mon, 14 May 2007 11:52:19 -0400 Subject: [Backgroundrb-devel] (no subject) Message-ID: Greetings, I am seeing the following in my log quite a bit 20 + times per day. The stack-trace seems tin indicate an internal issue - is concurrency related? == 0.2.1 (Released November 28, 2006 - r162) My workers continue to function; on the next iteration I believe. Any thoughts are appreciated. cheers, Jodi allow_concurrency=true Scheduler.run failure: undefined method `initial_do_work' for []:Array Backtrace: /var/www/html/the-soup/vendor/plugins/backgroundrb/server/lib/ backgroundrb/middleman.rb:342:in `schedule_worker' /var/www/html/the-soup/vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:46:in `run' /var/www/html/the-soup/vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:40:in `run' /var/www/html/the-soup/vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:35:in `run' /var/www/html/the-soup/vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:98:in `run' /var/www/html/the-soup/vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:97:in `run' /var/www/html/the-soup/vendor/plugins/backgroundrb/server/lib/ backgroundrb/middleman.rb:106:in `setup' /var/www/html/the-soup/vendor/plugins/backgroundrb/server/lib/ backgroundrb_server.rb:306:in `run' /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ application.rb:187:in `start_proc' /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ daemonize.rb:192:in `call_as_daemon' /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ application.rb:191:in `start_proc' /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ application.rb:227:in `start' /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ controller.rb:69:in `run' /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in `run_proc' /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ cmdline.rb:105:in `catch_exceptions' /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in `run_proc' /var/www/html/the-soup/vendor/plugins/backgroundrb/server/lib/ backgroundrb_server.rb:301:in `run' script/backgroundrb:29 From isak.hansen at gmail.com Tue May 15 08:16:26 2007 From: isak.hansen at gmail.com (Isak Hansen) Date: Tue, 15 May 2007 14:16:26 +0200 Subject: [Backgroundrb-devel] Need help with singleton worker Message-ID: <6b9e1eb20705150516h21d7e7cel77889ab9c77b5121@mail.gmail.com> We have a fairly expensive task that we'd rather not schedule too many instances of in parallel. Couldn't get pool_size to limit the amount of workers, and I figure that's hardly optimal anyway, so would like to have the work queued up for a single named worker instead, running as soon as possible. Using MiddleMan.schedule_worker() with :job_key does start off a worker, but I can't manage to add another task to the same worker and have it queue up. It's not just a matter of adding :worker_method => :do_work/:worker_method_args => whatever to schedule_worker()..? (see example below) Ideally using the same interface for running the first and any subsequent tasks. Here's a debug run, which may or may not make any sense. Task a, b and c were started in 10-20 sec total, d was started after a and b finished. As you can see, b isn't queued up, and c/d never start. I've also had 'b' start but never finish. (May have been due to do_work deleting the worker as 'a' finished in that case.. figured delete was neccessary to have the other jobs start, but no..) lib/workers/my_worker: class MyWorker < BackgrounDRb::Worker::RailsBase def do_work(args) logger.info("MyWorker#do_work. args: #{args}") sleep(60) logger.info("MyWorker#work done, deleting self. args: #{args}") # probably shouldn't call self.delete here, as we reuse the worker..? end end MyWorker.register script/console: >> MiddleMan.schedule_worker(:class => :my_worker, :job_key => :foo, :worker_method => :do_work, :worker_method_args => "a", :args => "_a_", :trigger_args => { :start => Time.now }) => 0 >> MiddleMan.schedule_worker(:class => :my_worker, :job_key => :foo, :worker_method => :do_work, :worker_method_args => "b", :args => "_b_", :trigger_args => { :start => Time.now }) => 1 >> MiddleMan.schedule_worker(:class => :my_worker, :job_key => :foo, :worker_method => :do_work, :worker_method_args => "c", :args => "_c_", :trigger_args => { :start => Time.now }) => 2 >> MiddleMan.schedule_worker(:class => :my_worker, :job_key => :foo, :worker_method => :do_work, :worker_method_args => "d", :args => "_d_", :trigger_args => { :start => Time.now }) => 3 >> bdrb log #1: 20070515-13:55:27 (10865) MyWorker#do_work. args: _a_ 20070515-13:55:32 (10865) MyWorker#do_work. args: b 20070515-13:56:27 (10865) MyWorker#work done, deleting self. args: _a_ 20070515-13:56:32 (10865) MyWorker#work done, deleting self. args: b bdrb log #2: 20070515-13:55:22 (10864) Loading Sechedule: job_keyfooargs_a_classmy_worker trigger_typetriggertrigger_argsstartTue May 15 13:55:22 CEST 2007worker_methoddo_workworker_method_argsajob_keyfooclassmy_workerargs_a_ # 20070515-13:55:23 (10864) Schedule triggered: # job=#, trigger=#, earliest=Tue May 15 13:55:23 CEST 2007, last=Tue May 15 13:55:23 CEST 2007> 20070515-13:55:27 (10864) Starting worker: my_worker foo (my_worker_foo) (_a_) 20070515-13:55:31 (10864) Loading Sechedule: job_keyfooargs_b_classmy_worker trigger_typetriggertrigger_argsstartTue May 15 13:55:31 CEST 2007worker_methoddo_workworker_method_argsbjob_keyfooclassmy_workerargs_b_ # 20070515-13:55:32 (10864) Schedule triggered: # job=#, trigger=#, earliest=Tue May 15 13:55:32 CEST 2007, last=Tue May 15 13:55:32 CEST 2007> 20070515-13:55:40 (10864) Loading Sechedule: job_keyfooargs_c_classmy_worker trigger_typetriggertrigger_argsstartTue May 15 13:55:40 CEST 2007worker_methoddo_workworker_method_argscjob_keyfooclassmy_workerargs_c_ # 20070515-13:56:03 (10864) Loading Sechedule: job_keyfooargs_d_classmy_worker trigger_typetriggertrigger_argsstartTue May 15 13:56:03 CEST 2007worker_methoddo_workworker_method_argsdjob_keyfooclassmy_workerargs_d_ # From cschlaefcke at wms-network.de Tue May 15 14:37:06 2007 From: cschlaefcke at wms-network.de (Christian Schlaefcke) Date: Tue, 15 May 2007 20:37:06 +0200 Subject: [Backgroundrb-devel] Behaviour of pool_size setting Message-ID: <4649FDD2.10804@wms-network.de> Hi, I have backgroundrb running to decouple the execution of massive business logic from an ActionWebservice request. The service is designed to take some configuration parameters and fire a lot of background workers to do the requested work. Due to performance reasons I want to limit the number of workers to a maximum number of 30. But when I start a configuration that requires for example 300 worker executions I can see that the limit of 30 workers is not kept and a number of about 180 worker processes are filling up my process list. I start my workers like this: key = MiddleMan.new_worker(:class => :execution_worker, :args => {...some_args...}) Why do I see so much more than my declared number of 30 workers? Am I wrong somehow? How do I have to understand the behaviour of pool_size? What happens when I have 30 workers working and the 31st, 32nd, ..., 300th request to start a worker comes in? Thanks & Regards! Christian -------------- next part -------------- A non-text attachment was scrubbed... Name: cschlaefcke.vcf Type: text/x-vcard Size: 368 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070515/defa024a/attachment.vcf From isak.hansen at gmail.com Wed May 16 04:43:10 2007 From: isak.hansen at gmail.com (Isak Hansen) Date: Wed, 16 May 2007 10:43:10 +0200 Subject: [Backgroundrb-devel] Need help with singleton worker In-Reply-To: <6a6edeed0705150925o6ea2cfa4i7da5dab3af30b4d8@mail.gmail.com> References: <6b9e1eb20705150516h21d7e7cel77889ab9c77b5121@mail.gmail.com> <6a6edeed0705150925o6ea2cfa4i7da5dab3af30b4d8@mail.gmail.com> Message-ID: <6b9e1eb20705160143i655debasdd747f99a613ae61@mail.gmail.com> On 5/15/07, Michael Siebert wrote: > for queueing up work, here is what i use, it runs in its own named worker > and is very stable. currently, there is only one job at a time, but you > could also generate two workers so you could process two jobs at a time. > *snipped code where jobs are queued up in the worker* Interesting approach. I was depending on bdrb itself to take care of the queueing, but will give this a shot if nothing is changed by the new release. > does the list have any improvements or comments? > Hmm.. I'm thinking some kind of synchronization may be neccessary on the worker's job queue. Btw.. How does this stuff really work? The worker instance lives in the bdrb process, while middleman creates a remote-call-somehow-proxy to that instance for your rails app to use? Any limitation on argument type(s) for public methods in my worker? Being serializable is probably a must? Regards, Isak From isak.hansen at gmail.com Wed May 16 05:17:37 2007 From: isak.hansen at gmail.com (Isak Hansen) Date: Wed, 16 May 2007 11:17:37 +0200 Subject: [Backgroundrb-devel] Need help with singleton worker In-Reply-To: <6a6edeed0705160152p4d6ed7d5kce96cc51ee91f851@mail.gmail.com> References: <6b9e1eb20705150516h21d7e7cel77889ab9c77b5121@mail.gmail.com> <6a6edeed0705150925o6ea2cfa4i7da5dab3af30b4d8@mail.gmail.com> <6b9e1eb20705160133j24a9a9d7k364df22663490384@mail.gmail.com> <6a6edeed0705160152p4d6ed7d5kce96cc51ee91f851@mail.gmail.com> Message-ID: <6b9e1eb20705160217v1dc14c72sf5b0b7dc7a2f30bd@mail.gmail.com> On 5/16/07, Michael Siebert wrote: > 2007/5/16, Isak Hansen : > > Hmm.. I'm thinking some kind of synchronization may be neccessary on > > the worker's job queue. > > help me understand what you mean... :) there is only one instance of the > worker so i dont see any place that should need synchronization. > There's the thread running do_work, and I'd think there are additional thread(s) adding jobs to the worker? > > Btw.. How does this stuff really work? The worker instance lives in > > the bdrb process, while middleman creates a remote-call-somehow-proxy > > to that instance for your rails app to use? > > I use this thing in productio and it just works. there is no other magic > behind that (besides what i have shown you > Sorry, I meant bdrb in general. Your example was great, and easy to follow. :) > > Any limitation on argument type(s) for public methods in my worker? > > Being serializable is probably a must? > > they must be serializable since they are sent via drb. its the same like > when passing arguments to do_work. > That's one piece of the puzzle down, thanks. Regards, Isak > kind regards, > micha > > > > -- > Siebert Webdesign > Michael Siebert From siebertm85 at googlemail.com Wed May 16 05:23:58 2007 From: siebertm85 at googlemail.com (Michael Siebert) Date: Wed, 16 May 2007 11:23:58 +0200 Subject: [Backgroundrb-devel] Need help with singleton worker In-Reply-To: <6b9e1eb20705160217v1dc14c72sf5b0b7dc7a2f30bd@mail.gmail.com> References: <6b9e1eb20705150516h21d7e7cel77889ab9c77b5121@mail.gmail.com> <6a6edeed0705150925o6ea2cfa4i7da5dab3af30b4d8@mail.gmail.com> <6b9e1eb20705160133j24a9a9d7k364df22663490384@mail.gmail.com> <6a6edeed0705160152p4d6ed7d5kce96cc51ee91f851@mail.gmail.com> <6b9e1eb20705160217v1dc14c72sf5b0b7dc7a2f30bd@mail.gmail.com> Message-ID: <6a6edeed0705160223o234b0119x4758802801260ba2@mail.gmail.com> 2007/5/16, Isak Hansen : > > There's the thread running do_work, and I'd think there are additional > thread(s) adding jobs to the worker? no there is only one process running. for the exact things going on, ask ezra or skaar, for me, it just works great Sorry, I meant bdrb in general. as i already said: its like rails and ubuntu: it just works. Your example was great, and easy to follow. :) thank you. kind regards -- Siebert Webdesign Michael Siebert -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070516/df88ed15/attachment.html From skaar at waste.org Wed May 16 08:40:54 2007 From: skaar at waste.org (skaar) Date: Wed, 16 May 2007 07:40:54 -0500 Subject: [Backgroundrb-devel] Behaviour of pool_size setting In-Reply-To: <4649FDD2.10804@wms-network.de> References: <4649FDD2.10804@wms-network.de> Message-ID: <20070516124054.GB7613@waste.org> * Christian Schlaefcke (cschlaefcke at wms-network.de) [070515 23:49]: > Hi, > > I have backgroundrb running to decouple the execution of massive > business logic from an ActionWebservice request. The service is designed > to take some configuration parameters and fire a lot of background > workers to do the requested work. Due to performance reasons I want to > limit the number of workers to a maximum number of 30. But when I start > a configuration that requires for example 300 worker executions I can > see that the limit of 30 workers is not kept and a number of about 180 > worker processes are filling up my process list. ok, I might look at adding a 'max_worker' configuration parameter as something separate from the thread pool (which really just say something about the MiddleMan, not the exisiting workers) - changing the pool_size semantics could impact current installations. /skaar > > I start my workers like this: > > key = MiddleMan.new_worker(:class => :execution_worker, :args => > {...some_args...}) > > > Why do I see so much more than my declared number of 30 workers? Am I > wrong somehow? How do I have to understand the behaviour of pool_size? > What happens when I have 30 workers working and the 31st, 32nd, ..., > 300th request to start a worker comes in? > > Thanks & Regards! > > Christian > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From masonhale at gmail.com Wed May 16 09:27:43 2007 From: masonhale at gmail.com (Mason Hale) Date: Wed, 16 May 2007 08:27:43 -0500 Subject: [Backgroundrb-devel] Behaviour of pool_size setting In-Reply-To: <20070516124054.GB7613@waste.org> References: <4649FDD2.10804@wms-network.de> <20070516124054.GB7613@waste.org> Message-ID: <8bca3aa10705160627g3e8e6bb7w8047a49bdbeed8d4@mail.gmail.com> I resolved a similar situation by pre-allocating a fixed-size pool of workers. These workers are long-running and they individually poll the database for new work. My backgroundrb_schedules.yml includes this: <% pub_worker_pool_size = 5 %> <% pub_worker_pool_size.times do |i| %> pubworker<%= i %>: :class: :pub_worker :job_key: :pubworker<%= i %> :worker_method: :do_work :worker_method_args: :ignored: 1 :trigger_args: :start: <%= Time.now + (10 + (10 * i)).seconds %> <% end %> My PubWorker#do_work method is: def do_work(args) logger.debug("#{self.jobkey}: do_work called") loop do # loop forever unless do_publish sleep 60 # pause 1 minute, if no publications were found on last try else sleep 1 # pause 1 second otherwise end end end My do_publish method checks the queue for work, and if there is a publication due, it publishes it and returns true, if there was no work it returns false. In the do_work you'll see that if the queue is empty it gets polled once per minute per worker. Otherwise workers try to empty the queue as quickly as possible. I stagger the start up of the workers at 10 second intervals so as to distribute the polls to the database. I avoid multi-threading issues by making each worker responsible for its own loop. There is no external scheduler that calls the do_work method periodically. My previous implementation had a single long-running worker that would spawn additional workers as needed to handle the work in the queue. But my experience was that creating a worker from a worker leads to socket not found and connection reset by peer errors, and poor reliability in general. So I scrapped that in favor of the fixed-size pool above and it has been much more reliable. I do think there is potential to create a fixed-size pool of workers, and another long-running 'queue' manager worker, and implement it such that the workers request work from the queue manager worker rather than the database. This way the queue manager could get a block of work and parcel it out with fewer queries to the database. In addition the workers could check in with the manager when they are done, making it possible for the manager to compile statistics on the jobs. For me, using the database to manage the queue -- and the inherent multi-threading issues -- was the more expedient route. Mason On 5/16/07, skaar wrote: > > * Christian Schlaefcke (cschlaefcke at wms-network.de) [070515 23:49]: > > Hi, > > > > I have backgroundrb running to decouple the execution of massive > > business logic from an ActionWebservice request. The service is designed > > to take some configuration parameters and fire a lot of background > > workers to do the requested work. Due to performance reasons I want to > > limit the number of workers to a maximum number of 30. But when I start > > a configuration that requires for example 300 worker executions I can > > see that the limit of 30 workers is not kept and a number of about 180 > > worker processes are filling up my process list. > > ok, I might look at adding a 'max_worker' configuration parameter as > something separate from the thread pool (which really just say something > about the MiddleMan, not the exisiting workers) - changing the pool_size > semantics could impact current installations. > > /skaar > > > > > > > I start my workers like this: > > > > key = MiddleMan.new_worker(:class => :execution_worker, :args => > > {...some_args...}) > > > > > > Why do I see so much more than my declared number of 30 workers? Am I > > wrong somehow? How do I have to understand the behaviour of pool_size? > > What happens when I have 30 workers working and the 31st, 32nd, ..., > > 300th request to start a worker comes in? > > > > Thanks & Regards! > > > > Christian > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- > ---------------------------------------------------------------------- > |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n > | | >=========== W.A.S.T.E. | genarratologies > |/|/ (_) is the wisdom | skaar at waste.org > ---------------------------------------------------------------------- > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070516/f73bbf90/attachment-0001.html From ivorpaul at gmail.com Thu May 17 05:25:22 2007 From: ivorpaul at gmail.com (Ivor Paul) Date: Thu, 17 May 2007 11:25:22 +0200 Subject: [Backgroundrb-devel] exec format error Message-ID: Hi I have been using backgroundrb version 0.1 for a while. Unfortunately, I have to develop on windows :( It has worked fine for a while. However, this morning I started up and old app that worked fine and get this error. exec format error from running --trace I see that the error comes from backgroundrb_tasks.rb line 59 which reads: 59 `#{scripts_src}/start -d` Here is the context. desc 'Start backgroundrb server (default values)' task :start do scripts_src = "#{RAILS_ROOT}/script/backgroundrb" if File.exists?(scripts_src) `#{scripts_src}/start -d` else puts "Backgroundrb is not installed. Run 'rake backgroundrb:setup' first!" end end I have recently upgraded to rails 1.2 - could that be the problem? Regards Ivor -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070517/4e914304/attachment.html From ivorpaul at gmail.com Thu May 17 05:35:07 2007 From: ivorpaul at gmail.com (Ivor Paul) Date: Thu, 17 May 2007 11:35:07 +0200 Subject: [Backgroundrb-devel] exec format error In-Reply-To: References: Message-ID: solved. It has obviously been so long that I have forgotten that the rake tasks dont run under windows :) use ruby script/backgroundrb/start instead. regards Ivor On 5/17/07, Ivor Paul wrote: > > Hi > > I have been using backgroundrb version 0.1 for a while. Unfortunately, I > have to develop on windows :( > It has worked fine for a while. However, this morning I started up and old > app that worked fine and get this error. > > exec format error > from running --trace I see that the error comes from backgroundrb_tasks.rb > line 59 which reads: > > 59 `#{scripts_src}/start -d` > > Here is the context. > desc 'Start backgroundrb server (default values)' > task :start do > scripts_src = "#{RAILS_ROOT}/script/backgroundrb" > > if File.exists?(scripts_src) > `#{scripts_src}/start -d` > else > puts "Backgroundrb is not installed. Run 'rake backgroundrb:setup' > first!" > end > end > > I have recently upgraded to rails 1.2 - could that be the problem? > > Regards > Ivor > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070517/0c2fb7b5/attachment.html From chewi at aura-online.co.uk Sat May 19 08:14:19 2007 From: chewi at aura-online.co.uk (James Le Cuirot) Date: Sat, 19 May 2007 13:14:19 +0100 Subject: [Backgroundrb-devel] Worker cleanup and TTL Message-ID: <20070519131419.222a02df@rhapsody> Hello. I read in an old announcement by Ezra, relating to the older version of BackgrounDRb, about workers being able to clean up after themselves and a TTL feature. I'm not sure if these things are present in the newer version. I've tried overriding the delete method in my worker but when I call delete_worker, the method doesn't appear to be called. Is there a right way to do this? If not, could a callback be added? The :ttl option doesn't appear to be there anymore. With scheduling, does :end actually delete the worker or merely stop making calls to it? If it doesn't delete, I could achieve the same effect by setting up a scheduler that sends different args between the first and second calls and do_work could subsequently delete the worker on the second call. This seems rather awkward though. Is there a better way? Cheers, James From chewi at aura-online.co.uk Mon May 21 06:55:28 2007 From: chewi at aura-online.co.uk (James Le Cuirot) Date: Mon, 21 May 2007 11:55:28 +0100 Subject: [Backgroundrb-devel] Worker cleanup and TTL In-Reply-To: <20070519131419.222a02df@rhapsody> References: <20070519131419.222a02df@rhapsody> Message-ID: <20070521115528.6197b463@rhapsody> On Sat, 19 May 2007 13:14:19 +0100 James Le Cuirot wrote: > I've tried overriding the delete method in my worker but when I call > delete_worker, the method doesn't appear to be called. Is there a > right way to do this? If not, could a callback be added? I've now tried various other things but the only way I've been able to do it is by using at_exit, as mentioned in the Slave README. Simply calling at_exit from within the worker class doesn't seem to work. You have to call it when creating the slave. Here's a small patch against 0.2.1. I tried the same thing against trunk and it worked. Is it any good? I still would also appreciate info on the TTL issue. Cheers, James -------------- next part -------------- A non-text attachment was scrubbed... Name: backgroundrb-at-exit.patch Type: text/x-patch Size: 1050 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070521/c7b69d9f/attachment.bin From isak.hansen at gmail.com Mon May 21 08:27:47 2007 From: isak.hansen at gmail.com (Isak Hansen) Date: Mon, 21 May 2007 14:27:47 +0200 Subject: [Backgroundrb-devel] Need help with singleton worker In-Reply-To: <6a6edeed0705160223o234b0119x4758802801260ba2@mail.gmail.com> References: <6b9e1eb20705150516h21d7e7cel77889ab9c77b5121@mail.gmail.com> <6a6edeed0705150925o6ea2cfa4i7da5dab3af30b4d8@mail.gmail.com> <6b9e1eb20705160133j24a9a9d7k364df22663490384@mail.gmail.com> <6a6edeed0705160152p4d6ed7d5kce96cc51ee91f851@mail.gmail.com> <6b9e1eb20705160217v1dc14c72sf5b0b7dc7a2f30bd@mail.gmail.com> <6a6edeed0705160223o234b0119x4758802801260ba2@mail.gmail.com> Message-ID: <6b9e1eb20705210527pa8f20a2yf0a47442b3bad4e2@mail.gmail.com> On 5/16/07, Michael Siebert wrote: > 2007/5/16, Isak Hansen : > > There's the thread running do_work, and I'd think there are additional > > thread(s) adding jobs to the worker? > > no there is only one process running. for the exact things going on, ask > ezra or skaar, for me, it just works great > One process yes, but I believe add_job() is called from a different thread than do_work, which is stuck in an infinite loop. I'm not sure Array isn't thread safe (been getting conflicting feedback about that), but figure it can't hurt to use a mutex around code accessing the job queue. How about cleanup? Say we eventually get 5-6 different named workers, that's quite a bit of memory. Should we just trust the OS to swap those processes out, or would you guys recommend having the worker deleting itself and terminate after for instance X minutes of idle time? Regards, Isak From taylorsingletary at gmail.com Mon May 21 12:38:39 2007 From: taylorsingletary at gmail.com (D. Taylor Singletary) Date: Mon, 21 May 2007 09:38:39 -0700 Subject: [Backgroundrb-devel] Helpers not available to ActionMailer classes in backgroundrb Message-ID: <75d04d500705210938h2fc7a541p7de438e9ee959adc@mail.gmail.com> Hi there, I've got a problem I'm hoping someone else has run into before and might have some tips on. I have an ActionMailer class where I'm manually including my application helper along with simply_helpful for use in my html emal templates. This works fine when calling the mailer class from another model class.. (example.. @order.deliver_emails would be a method that would call the deliver methods of my ActionMailer object. But when I offload the delivery of emails to a Backgroundrb process, even though I have these include statements in my mailer model (and have even tried including them within my worker library), I get an error triggered when sending emails out that it couldn't utilize my helpers. Has anyone run into this and possibly have any advice to make my workers aware of the directives I have in my ActionMailer class? Thanks. D. Taylor Singletary, Reality Technician http://realitytechnicians.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070521/e8784479/attachment.html From jnutting at gmail.com Tue May 22 06:24:42 2007 From: jnutting at gmail.com (Jack Nutting) Date: Tue, 22 May 2007 12:24:42 +0200 Subject: [Backgroundrb-devel] Recommendations for eternally-running backgroundrb workers? Message-ID: I've got some workers that I want to have running all the time. Right now I'm just launching them manually, by requesting a special page in my rails app that includes lines like: MiddleMan.new_worker(:class=> :receiver, :job_key=>:r, :args=>{:sleep_time=>10}) This strikes me as a really weak way to fire up my workers. I basically want to automate things so that these will be started by a script in /etc/rc?.d. It seems like using the backgroundrb scheduler is not a possibility, since do_work never returns from my workers (each of them runs in an infinite loop, polling the database periodically to look for work). So it looks like I need another startup script, running after the rest of the system is up, that does one of these things: - use curl or wget to hit my special worker-starting page (how lame is that?) - pipe my MiddleMan.new_worker calls into a "script/backgroundrb console" or just "script/console" session (is that any better?) As you can see, neither of these options seems very appealing to me. I can't be the first person to do this... How are others handling this? -- // jack // http://www.nuthole.com From eden at mojiti.com Tue May 22 06:32:46 2007 From: eden at mojiti.com (Eden Li) Date: Tue, 22 May 2007 18:32:46 +0800 Subject: [Backgroundrb-devel] Recommendations for eternally-running backgroundrb workers? In-Reply-To: References: Message-ID: <1dd361e10705220332v5a85bbb1vb38f996f1bf6869c@mail.gmail.com> Can't you remove the periodic checker code that wraps the actual worker and let backgroundrb scheduler do the periodic thing for you? In other words, why do you have do_work run "in an infinite loop, ... periodically to look for work" when essentially that's what backgroundrb does for you? On 5/22/07, Jack Nutting wrote: > I've got some workers that I want to have running all the time. Right > now I'm just launching them manually, by requesting a special page in > my rails app that includes lines like: > > MiddleMan.new_worker(:class=> :receiver, :job_key=>:r, > :args=>{:sleep_time=>10}) > > This strikes me as a really weak way to fire up my workers. I > basically want to automate things so that these will be started by a > script in /etc/rc?.d. It seems like using the backgroundrb scheduler > is not a possibility, since do_work never returns from my workers > (each of them runs in an infinite loop, polling the database > periodically to look for work). So it looks like I need another > startup script, running after the rest of the system is up, that does > one of these things: > > - use curl or wget to hit my special worker-starting page (how lame is that?) > - pipe my MiddleMan.new_worker calls into a "script/backgroundrb > console" or just "script/console" session (is that any better?) > > As you can see, neither of these options seems very appealing to me. > I can't be the first person to do this... How are others handling > this? > > -- > // jack > // http://www.nuthole.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From Peer.Allan at canadadrugs.com Tue May 22 08:32:12 2007 From: Peer.Allan at canadadrugs.com (Peer Allan) Date: Tue, 22 May 2007 07:32:12 -0500 Subject: [Backgroundrb-devel] BackgroundRb Server not responding Message-ID: Hey, I have scheduled script that is not working very well. It will only run once or twice before I have to restart the server to get it running again. If I try to the worker manually after it has stopped exceuting the schedule I get this in the server log: 20070522-07:33:36 (5030) Connection reset by peer - (DRb::DRbConnError) 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:566:in `read' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:566:in `load' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:632:in `recv_reply' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:921:in `recv_reply' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1195:in `send_message' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1086:in `method_missing' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1170:in `open' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1085:in `method_missing' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1103:in `with_friend' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1084:in `method_missing' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1072:in `respond_to?' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:454:in `initialize' 20070522-07:33:36 (5030) /usr/local/apache2/htdocs/ruby/pharmacy_wire/vendor/plugins/backgroundrb/ser ver/lib/backgroundrb/middleman.rb:210:in `new' 20070522-07:33:36 (5030) /usr/local/apache2/htdocs/ruby/pharmacy_wire/vendor/plugins/backgroundrb/ser ver/lib/backgroundrb/middleman.rb:210:in `new_worker' 20070522-07:33:36 (5030) /usr/local/apache2/htdocs/ruby/pharmacy_wire/vendor/plugins/backgroundrb/ser ver/lib/backgroundrb/thread_pool.rb:36:in `dispatch' 20070522-07:33:36 (5030) /usr/local/apache2/htdocs/ruby/pharmacy_wire/vendor/plugins/backgroundrb/ser ver/lib/backgroundrb/thread_pool.rb:22:in `initialize' 20070522-07:33:36 (5030) /usr/local/apache2/htdocs/ruby/pharmacy_wire/vendor/plugins/backgroundrb/ser ver/lib/backgroundrb/thread_pool.rb:22:in `new' 20070522-07:33:36 (5030) /usr/local/apache2/htdocs/ruby/pharmacy_wire/vendor/plugins/backgroundrb/ser ver/lib/backgroundrb/thread_pool.rb:22:in `dispatch' 20070522-07:33:36 (5030) /usr/local/apache2/htdocs/ruby/pharmacy_wire/vendor/plugins/backgroundrb/ser ver/lib/backgroundrb/middleman.rb:199:in `new_worker' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1555:in `__send__' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1555:in `perform_without_block' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1515:in `perform' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1589:in `main_loop' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1585:in `loop' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1585:in `main_loop' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1581:in `start' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1581:in `main_loop' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1430:in `run' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1427:in `start' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1427:in `run' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1347:in `initialize' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1627:in `new' 20070522-07:33:36 (5030) /usr/local/lib/ruby/1.8/drb/drb.rb:1627:in `start_service' 20070522-07:33:36 (5030) /usr/local/apache2/htdocs/ruby/pharmacy_wire/vendor/plugins/backgroundrb/ser ver/lib/backgroundrb_server.rb:315:in `run' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:1 87:in `call' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:1 87:in `start_proc' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/daemonize.rb:192 :in `call' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/daemonize.rb:192 :in `call_as_daemon' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:1 91:in `start_proc' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:2 27:in `start' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/controller.rb:69 :in `run' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in `run_proc' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:i n `call' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:i n `catch_exceptions' 20070522-07:33:36 (5030) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in `run_proc' 20070522-07:33:36 (5030) /usr/local/apache2/htdocs/ruby/pharmacy_wire/vendor/plugins/backgroundrb/ser ver/lib/backgroundrb_server.rb:301:in `run' 20070522-07:33:36 (5030) script/backgroundrb:29 Anyone know what?s going on here? Thanks Peer Allan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070522/41803ee2/attachment-0001.html From jnutting at gmail.com Wed May 23 03:37:45 2007 From: jnutting at gmail.com (Jack Nutting) Date: Wed, 23 May 2007 09:37:45 +0200 Subject: [Backgroundrb-devel] Recommendations for eternally-running backgroundrb workers? In-Reply-To: References: <1dd361e10705220332v5a85bbb1vb38f996f1bf6869c@mail.gmail.com> Message-ID: Hmmm, I meant to reply all on this... ---------- Forwarded message ---------- From: Jack Nutting Date: May 22, 2007 1:39 PM Subject: Re: [Backgroundrb-devel] Recommendations for eternally-running backgroundrb workers? To: Eden Li On 5/22/07, Eden Li wrote: > Can't you remove the periodic checker code that wraps the actual > worker and let backgroundrb scheduler do the periodic thing for you? > In other words, why do you have do_work run "in an infinite loop, ... > periodically to look for work" when essentially that's what > backgroundrb does for you? Well, that's the first thing I tried. What I found is that if one worker was tied up doing something that took a long time, the other workers wouldn't be triggered to run at all. So, I've got 3 workers (let's call them A, B, and C), each set to pause about a minute between looking for work. At some point, worker A finds it has some tasks waiting for it, and starts doing them. If this ends up taking 15 minutes, workers B and C are completely idle for 15 minutes. I've never figured out if this is happening on accident or by design, but it seems that the scheduler runs in a single thread and waits for each worker call to finish before proceeding. I suppose I could get around that by breaking up the work, so that each call to A.real_work only handles a chunk of its work before returning and allowing the others to run. But even then, the effect is that I'd still be stuck serializing (though in smaller chunks) something that I want to run in parallel. In particular, both A and B spend a large amount of their working time waiting for results from http requests (which are independent of each other, coming from different hosts), so these are good candidates for running in parallel instead of saying, "OK, now A can make some requests.... OK, now it's B's turn...." etc. -- // jack // http://www.nuthole.com From masonhale at gmail.com Wed May 23 11:44:25 2007 From: masonhale at gmail.com (Mason Hale) Date: Wed, 23 May 2007 10:44:25 -0500 Subject: [Backgroundrb-devel] Recommendations for eternally-running backgroundrb workers? In-Reply-To: References: Message-ID: <8bca3aa10705230844r1f57693dha22cfa703d08be5e@mail.gmail.com> Jack -- you can use the scheduler to fire up long-running processes. Just don't specify a repeat interval. You backgroundrb_schedules.yml file should look something like: my_worker: :class: :polling_worker :job_key: :my_worker :worker_method: :do_work :worker_method_args: placeholder :trigger_args: :start: <%= Time.now + 1.seconds %> On 5/22/07, Jack Nutting wrote: > > I've got some workers that I want to have running all the time. Right > now I'm just launching them manually, by requesting a special page in > my rails app that includes lines like: > > MiddleMan.new_worker(:class=> :receiver, :job_key=>:r, > :args=>{:sleep_time=>10}) > > This strikes me as a really weak way to fire up my workers. I > basically want to automate things so that these will be started by a > script in /etc/rc?.d. It seems like using the backgroundrb scheduler > is not a possibility, since do_work never returns from my workers > (each of them runs in an infinite loop, polling the database > periodically to look for work). So it looks like I need another > startup script, running after the rest of the system is up, that does > one of these things: > > - use curl or wget to hit my special worker-starting page (how lame is > that?) > - pipe my MiddleMan.new_worker calls into a "script/backgroundrb > console" or just "script/console" session (is that any better?) > > As you can see, neither of these options seems very appealing to me. > I can't be the first person to do this... How are others handling > this? > > -- > // jack > // http://www.nuthole.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070523/60587e8f/attachment.html From masonhale at gmail.com Wed May 23 11:49:00 2007 From: masonhale at gmail.com (Mason Hale) Date: Wed, 23 May 2007 10:49:00 -0500 Subject: [Backgroundrb-devel] Recommendations for eternally-running backgroundrb workers? In-Reply-To: <8bca3aa10705230844r1f57693dha22cfa703d08be5e@mail.gmail.com> References: <8bca3aa10705230844r1f57693dha22cfa703d08be5e@mail.gmail.com> Message-ID: <8bca3aa10705230849n7e1f73a3p8b1de60043ee466c@mail.gmail.com> hit send prematurely... my_worker: :class: :polling_worker :job_key: :my_worker :worker_method: :do_work :worker_method_args: :arg1: placeholder :trigger_args: :start: <%= Time.now + 1.seconds %> Note there is not :repeate_interval: argument under :trigger_args: In your worker class it could look like: def do_work(args) logger.debug("#{self.jobkey}: do_work called") loop do # loop forever check_for_work # this is the method that really does the work sleep 60 end end end You should see the "do_work called" message once. The fact that do_work does not return should not matter, the other scheduled jobs will still run. I use the method above in production with a high level of reliability. Mason On 5/23/07, Mason Hale wrote: > > Jack -- you can use the scheduler to fire up long-running processes. > > Just don't specify a repeat interval. > > You backgroundrb_schedules.yml file should look something like: > > my_worker: > :class: :polling_worker > :job_key: :my_worker > :worker_method: :do_work > :worker_method_args: > placeholder > :trigger_args: > :start: <%= Time.now + 1.seconds %> > > On 5/22/07, Jack Nutting wrote: > > > > I've got some workers that I want to have running all the time. Right > > now I'm just launching them manually, by requesting a special page in > > my rails app that includes lines like: > > > > MiddleMan.new_worker (:class=> :receiver, :job_key=>:r, > > :args=>{:sleep_time=>10}) > > > > This strikes me as a really weak way to fire up my workers. I > > basically want to automate things so that these will be started by a > > script in /etc/rc?.d. It seems like using the backgroundrb scheduler > > is not a possibility, since do_work never returns from my workers > > (each of them runs in an infinite loop, polling the database > > periodically to look for work). So it looks like I need another > > startup script, running after the rest of the system is up, that does > > one of these things: > > > > - use curl or wget to hit my special worker-starting page (how lame is > > that?) > > - pipe my MiddleMan.new_worker calls into a "script/backgroundrb > > console" or just "script/console" session (is that any better?) > > > > As you can see, neither of these options seems very appealing to me. > > I can't be the first person to do this... How are others handling > > this? > > > > -- > > // jack > > // http://www.nuthole.com > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070523/960cad0f/attachment.html From wil at 3cglabs.com Sat May 26 10:51:31 2007 From: wil at 3cglabs.com (Wil Chung) Date: Sat, 26 May 2007 15:51:31 +0100 Subject: [Backgroundrb-devel] New worker not being created In-Reply-To: <4cdd0d160705241205q407d5cb9m9403ff00c83272b9@mail.gmail.com> References: <4cdd0d160705241205q407d5cb9m9403ff00c83272b9@mail.gmail.com> Message-ID: <4cdd0d160705260751j39dc4ac8w762173926743653d@mail.gmail.com> I figured out that in the backgroundrb.yml, I had the old setting of making the database RAILS_ENV. It wasn't a valid variable in backgroundrb 0.2.1, and it wasn't reporting an error, so I had no idea. Once I changed it to 'development' or 'production', it started working. I found that if a worker has an error, it usually logs the error trace in backgroundrb.log. However, when using a scheduled worker, it doesn't seem to log the error stack trace at all. What's the best practice on this? Should all methods in a worker catch exceptions and manually log the info? Or is this something I can change/hack in backgroundrb to force error reporting? On 5/24/07, Wil Chung wrote: > > Hi all, > > I've looked on the mailing lists and the web, and didn't find the answer, > so I'm posting for the first time to see if I'm missing something. > > I recently upgraded to BackgroundRB 0.2.1, but I can't seem to create new > workers now. To test things out again with a simple example, I tried > following the example in the README of the docs. > > ./script/generate worker example > > my example worker: > > class ExampleWorker < BackgrounDRb::Worker::RailsBase > > def do_work(args) > logger.info('ExampleWorker do work') > results[:do_work_time] = Time.now.to_s > results[:done_with_do_work] ||= true > end > > def other_method > logger.info('other_method in ExampleWorker called') > results[:extra_data] = "Just a plain old string" > end > > end > ExampleWorker.register > > Then in a controller, I have: > > def test_backgroundrb > key = MiddleMan.new_worker(:class => :example_worker) > worker = MiddleMan.worker(key) > worker.other_method > worker.delete > end > > When I point the browser to test_backgroundrb, I get: > > NoMethodError (undefined method `object' for nil:NilClass): > (drbunix:///tmp/backgroundrbunix_localhost_2000) > /home/wil/proj/3cgworkspace/thoughtless/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:396:in > `worker' > etc. etc. etc. > > When I try to use the version from the trunk, it looks like worker returns > nil if it doesn't find anything, as discussed per April 2007's post. > > NoMethodError (You have a nil object when you didn't expect it! > The error occurred while evaluating nil.other_method): > .//app/controllers/account_controller.rb:23:in `test_backgroundrb' > /usr/lib/ruby/gems/1.8/gems/actionpack-1.13.0/lib/action_controller/base.rb:1095:in > `perform_action_without_filters' > etc. etc. etc. > > What seems really odd is that while new_worker returns a key, a worker is > never created. I verified this with a Breakpoint: > > irb(#):001:0> key = MiddleMan.new_worker(:class => :example_worker) > => "a2dc70e7a1de822c4616113a9d602dec" > irb(#):002:0> MiddleMan.jobs.has_key?(key) > => false > > I don't get it. What's going on? If anyone could enlighten the matter, > I'd greatly appreciate it. Thanks. > > Wil > -- http://www.3cglabs.com http://thoughtl.us -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070526/5c1d119a/attachment.html From kamil.skalski at gmail.com Thu May 31 06:07:51 2007 From: kamil.skalski at gmail.com (Kamil Skalski) Date: Thu, 31 May 2007 12:07:51 +0200 Subject: [Backgroundrb-devel] Slave socket problem again Message-ID: <711aea6b0705310307l5fe24017qc207528e2eeadbfd@mail.gmail.com> Hello! I have been using backgroundrb for a small project. My worker task is run every minute, checks the processing queue in data-base and execute the business method from Rails if something is to be done. Unfortunately I was hit by the problem of backgroundrb server dying after some runs. The exception is: 20070531-11:06:05 (31816) Starting worker: s_e_work sql_exe (s_e_work_sql_exe) () 20070531-11:07:05 (31816) Schedule triggered: # job=#, trigger=#, earliest=Thu May 31 11:07:05 +0200 2007, last=Thu May 31 11:07:05 +0200 2007> 20070531-11:07:05 (31816) failed to find slave socket - (RuntimeError) 20070531-11:07:05 (31816) /usr/local/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:435:in `initialize' 20070531-11:07:05 (31816) /home/kskalski/railsapp/sars/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new' 20070531-11:07:05 (31816) /home/kskalski/railsapp/sars/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new_worker' 20070531-11:07:05 (31816) /home/kskalski/railsapp/sars/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in `dispatch' 20070531-11:07:05 (31816) /home/kskalski/railsapp/sars/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `initialize' I tried suggestions from http://rubyforge.org/pipermail/backgroundrb-devel/2007-March/000776.html but it didn't help... actually it even get worse: I shortened the class name of worker and I used druby protocol and after that server fails even faster than before. Also, what I have spotted - even though I enabled druby protocol (and in logs there is entry: 20070531-10:24:33 (31813) protocol: druby 20070531-10:24:33 (31813) uri: druby://localhost:2000 20070531-10:24:33 (31813) config: /home/kskalski/railsapp/sars/config/backgroundrb.yml 20070531-10:24:33 (31813) rails_env: development 20070531-10:24:33 (31813) socket_dir: /tmp/backgroundrb.31813 20070531-10:24:33 (31813) pool_size: 5 20070531-10:24:33 (31813) host: localhost 20070531-10:24:33 (31813) acl: denyallallowlocalhost 127.0.0.1orderdenyallow 20070531-10:24:33 (31813) temp_dir: /tmp 20070531-10:24:33 (31813) port: 2000 20070531-10:24:33 (31813) Installed DRb ACL ) the server still uses sockets!!! I have observed, that with every run of worker the sockets directory gets a new entry, which is never deleted... maybe this is a problem - new sockets are created, but never deleted and system runs out of resources? Currently my configuration is (backgroundrb.yml): :port: 2000 :rails_env: development :host: localhost :protocol: druby :pool_size: 5 :worker_dir: lib/workers (backgroundrb_schedules.yml) sql_scheduler: :class: :s_e_work :job_key: :sql_exe :trigger_args: 5 * * * * * * Worker code is: class SEWork < BackgrounDRb::Worker::RailsBase def do_work(args) # This method is called in it's own new thread begin script = SqlScript.find(:first, :conditions => [ 'state = ? AND scheduled_run_at IS NOT NULL AND scheduled_run_at < ? OR' + ' EXISTS (SELECT id FROM execution_params WHERE sql_script_id = sql_scripts.id) AND' + ' state <> ?', SqlScript::STATE_SCHEDULED, Time.now, SqlScript::STATE_RUNNING]) if script.nil? logger.debug "Nothing to run" else if script.execution_param.nil? ret, status = script.scheduled_execute else ret, status = script.execute end logger.info "Executed (#{ret}) #{script.name} - #{status}" end rescue Exception logger.error "Problems executing sql executor worker: #{$!}" ensure # worker must be finished, otherwise stale process it leaved self.delete end end end SEWork.register (note that I *have* self.delete at the end of worker code) -- Kamil Skalski http://nazgul.omega.pl From dtown22 at gmail.com Thu May 31 19:45:26 2007 From: dtown22 at gmail.com (dtown) Date: Thu, 31 May 2007 16:45:26 -0700 Subject: [Backgroundrb-devel] background worker just stops running after some time Message-ID: <778fbc270705311645o7a63374fuc6de788ca60c5b26@mail.gmail.com> Hi, I am fairly new to ruby/rails, but I thought I did a fair job of setting up my background worker, but after some time, maybe 4-5 hours of running, it just stops running, without any messages. Here is the yml file: backgroundrb.yml --- :port: 2000 :rails_env: production :host: localhost here is my schedules file: alerts_task: :class: :alerts_worker :job_key: :alerts_key :worker_method: :do_work :trigger_args: :start: <%= Time.now + 5.seconds %> :repeat_interval: <%= 1.hour %> and i start it by doing: />script/backgroundrb start everytime the worker is called, I print an Iteration Time to the logger. While checking on the worker, I noticed that it hadn't run in several hours (i.e. the Iteration printout was several hours old). I looked at the background_server.log, and everything looked fine, here is the last few entries: 20070531-12:47:44 (678) Schedule triggered: # job=#, trigger=#, earliest=Thu May 31 12:47:44 -0700 2007, last=Thu May 31 12:47:44 -0700 2007> 20070531-13:47:44 (678) Schedule triggered: # job=#, trigger=#, earliest=Thu May 31 13:47:44 -0700 2007, last=Thu May 31 13:47:44 -0700 2007> 20070531-14:47:48 (678) Schedule triggered: # job=#, trigger=#, earliest=Thu May 31 14:47:44 -0700 2007, last=Thu May 31 14:47:44 -0700 2007> and no error messages. So I checked to see if the processes are still running, and I have: $ps aux | grep ruby mongrel 639 0.5 3.2 36000 32176 ?? S 10:47AM 2:28.13/usr/local/bin/ruby /usr/local/bin/mongrel_rai mongrel 633 0.3 3.3 36820 32924 ?? S 10:47AM 2:41.17/usr/local/bin/ruby /usr/local/bin/mongrel_rai mongrel 636 0.0 3.2 35888 32040 ?? S 10:47AM 2:33.56/usr/local/bin/ruby /usr/local/bin/mongrel_rai root 678 0.0 0.2 39076 1816 ?? S 10:47AM 2:22.97 ruby: backgroundrb (ruby) root 679 0.0 0.0 20824 0 ?? IW - 0:00.00 ruby: backgroundrb_logger (ruby) root 681 0.0 0.0 20824 0 ?? IW - 0:00.00 ruby: backgroundrb_results (ruby) root 688 0.0 0.0 67288 0 ?? IW - 0:00.00 ruby: alerts_worker_alerts_key (ruby) root 2266 0.0 0.0 348 224 p0 R+ 4:42PM 0:00.00 grep ruby I have thought of having a sleep call inside my do_work, but I dont like that...i want to keep my design as simple as possible. Does anyone have any ideas? thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070531/378d6c75/attachment.html