From caifara.subscribe at gmail.com Mon Dec 3 15:19:14 2007 From: caifara.subscribe at gmail.com (Ivo Dancet) Date: Mon, 3 Dec 2007 21:19:14 +0100 Subject: [Backgroundrb-devel] stability of scheduler In-Reply-To: References: Message-ID: Hi I'm experiencing some stability problems in BackgroundRB with scheduling. I have some long running tasks that start up on client request. The app then schedules BackgroundRB to start working at 1AM. We noticed the task sometimes just didn't start. No error, no log, nothing. I also find it annoying that I can't query the scheduler, so I find it really hard to debug problems like this. So I'm posing myself the question if it wouldn't be better to just use a small cron at 1AM that starts up requests that are for example being logged in the db. Is the backgroundrb scheduler fit for the job, am I doing something wrong, is the new version (1.0RC1) ten times more stable? Thanks -c From gethemant at gmail.com Tue Dec 4 00:00:15 2007 From: gethemant at gmail.com (hemant) Date: Tue, 4 Dec 2007 10:30:15 +0530 Subject: [Backgroundrb-devel] BackgrounDRb 1.0 RC2 released Message-ID: Hi Folks, Thanks for your patience, new release of BackgrounDRb is available now. You can get it from here: http://svn.devjavu.com/backgroundrb/trunk/ Yes, older version is deprecated and not supported anymore. A brief list of changes since last release is: 1. Now, each worker has logger objects, which can be used to log stuff. 2. A worker can be started or scheduled dynamically. 3. Multiple copies of same worker can be started using different job_keys. 4. You can query status of all workers in oneshot. Detailed information about various changes that has gone in place is available from here: http://backgroundrb.rubyforge.org/ Give it a spin and see how it turns out. Post all your problems here, I would be happy to have a look. If someone can lend me a hand, then we can port it to framework agnostic as well, so as it can run in a standalone manner, or with merb. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Tue Dec 4 00:03:05 2007 From: gethemant at gmail.com (hemant) Date: Tue, 4 Dec 2007 10:33:05 +0530 Subject: [Backgroundrb-devel] stability of scheduler In-Reply-To: References: Message-ID: On Dec 4, 2007 1:49 AM, Ivo Dancet wrote: > Hi > > I'm experiencing some stability problems in BackgroundRB with > scheduling. I have some long running tasks that start up on client > request. The app then schedules BackgroundRB to start working at 1AM. > We noticed the task sometimes just didn't start. No error, no log, > nothing. I also find it annoying that I can't query the scheduler, so > I find it really hard to debug problems like this. > > So I'm posing myself the question if it wouldn't be better to just use > a small cron at 1AM that starts up requests that are for example being > logged in the db. Is the backgroundrb scheduler fit for the job, am I > doing something wrong, is the new version (1.0RC1) ten times more > stable? > Please try new version, scheduling has certainly been improved a lot. Mainly because, thats for what I use bdrb. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From caifara.subscribe at gmail.com Wed Dec 5 17:40:54 2007 From: caifara.subscribe at gmail.com (Ivo Dancet) Date: Wed, 5 Dec 2007 23:40:54 +0100 Subject: [Backgroundrb-devel] RC2 troubles Message-ID: <56D0F061-F0C5-4B9C-85C0-C56DC8B2816F@gmail.com> Hi I can't get backgroundrb working. I get the following error in backgroundrb_server.log: backgroundrb/framework/core.rb:117:in `bind': Address family not supported by protocol family - bind(2) (Errno::EAFNOSUPPORT) Seems to be a problem with my Mac (Leopard) and my ruby install (1.8.6 that came with the OS) according to some emails I found with google. Did anyone allready found a way to solve this? -c From gethemant at gmail.com Wed Dec 5 20:07:08 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 06 Dec 2007 06:37:08 +0530 Subject: [Backgroundrb-devel] RC2 troubles In-Reply-To: <56D0F061-F0C5-4B9C-85C0-C56DC8B2816F@gmail.com> References: <56D0F061-F0C5-4B9C-85C0-C56DC8B2816F@gmail.com> Message-ID: <1196903228.19587.1.camel@shire> On Wed, 2007-12-05 at 23:40 +0100, Ivo Dancet wrote: > Hi > > I can't get backgroundrb working. I get the following error in > backgroundrb_server.log: > > backgroundrb/framework/core.rb:117:in `bind': Address family not > supported by protocol family - bind(2) (Errno::EAFNOSUPPORT) > > Seems to be a problem with my Mac (Leopard) and my ruby install (1.8.6 > that came with the OS) according to some emails I found with google. > Did anyone allready found a way to solve this? In your config file, change "localhost" to "0.0.0.0" and it should work. There seems to be some problem with using "localhost" as IP address on Mac. From andy.tyra at gmail.com Wed Dec 5 20:51:11 2007 From: andy.tyra at gmail.com (Andy Tyra) Date: Wed, 5 Dec 2007 17:51:11 -0800 Subject: [Backgroundrb-devel] Feedback on RC2 Message-ID: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> I tried to upgrade my existing application to RC2 last night. Like many, I use this mostly for running scheduled tasks. For the moment, I've abandoned the effort, but am looking forward to being able to use this. Feedback below: First, the reason I was looking forward to this upgrade was to use the threaded scheduler. I have an application with long-running tasks. I found that scheduled tasks in previous versions of backgroundrb would not run if the worker scheduled before it was still running. This seemed counterintuitive to me (given the whole idea behind backgroundrb) but after reading documentation and forums, it seems many others ran into this issue too. RC2 planned to do away with this. The Good: - (Still) Easy Installattion - Well designed. Logging is centralized, that's nice. - Easy to hack up for use in an enterprise environment. The Bad: - Lots of arbitrary changes that I don't quite understand. Different function names to create workers, new YML formats, different config files for scheduler. Would have been great if none of the rails application-facing stuff changed so the new version could just be swapped in. - Logging detail has greatly diminished. It's difficult to troubleshoot what's happening when things go wrong. - For the moment, this seems less stable than the old version. My scheduled tasks are bringing down the whole server and I can't tell why (nothing showing up in the logs). This is why I've abandoned the upgrade. The scheduler is working fine for lightweight tasks, but the long-running ones just bring everything down. For what it's worth, the logic for the tasks still lives in my rails models. The workers basically just call class methods in the models. (Truly using this as a scheduler.) Perhaps this has something to do with it? - Documentation and resources for the new version are sparse (obviously) and seem a little inaccurate. For example, I learned that the MiddleMan.ask_worker() method detailed in the docs is actually called as MiddleMan.ask_work (). If anyone has any thoughts on what might be going wrong with my scheduled tasks, that'd be great. Thanks! Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071205/46073cc7/attachment.html From gethemant at gmail.com Wed Dec 5 21:54:05 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 06 Dec 2007 08:24:05 +0530 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> Message-ID: <1196909645.19587.7.camel@shire> On Wed, 2007-12-05 at 17:51 -0800, Andy Tyra wrote: > I tried to upgrade my existing application to RC2 last night. Like > many, I use this mostly for running scheduled tasks. For the moment, > I've abandoned the effort, but am looking forward to being able to use > this. Feedback below: > > First, the reason I was looking forward to this upgrade was to use the > threaded scheduler. I have an application with long-running tasks. I > found that scheduled tasks in previous versions of backgroundrb would > not run if the worker scheduled before it was still running. This > seemed counterintuitive to me (given the whole idea behind > backgroundrb) but after reading documentation and forums, it seems > many others ran into this issue too. RC2 planned to do away with > this. > > The Good: > > - (Still) Easy Installattion > - Well designed. Logging is centralized, that's nice. > - Easy to hack up for use in an enterprise environment. > > The Bad: > > - Lots of arbitrary changes that I don't quite understand. Different > function names to create workers, new YML formats, different config > files for scheduler. Would have been great if none of the rails > application-facing stuff changed so the new version could just be > swapped in. > - Logging detail has greatly diminished. It's difficult to > troubleshoot what's happening when things go wrong. > - For the moment, this seems less stable than the old version. My > scheduled tasks are bringing down the whole server and I can't tell > why (nothing showing up in the logs). This is why I've abandoned the > upgrade. The scheduler is working fine for lightweight tasks, but the > long-running ones just bring everything down. For what it's worth, > the logic for the tasks still lives in my rails models. The workers > basically just call class methods in the models. (Truly using this as > a scheduler.) Perhaps this has something to do with it? > - Documentation and resources for the new version are sparse > (obviously) and seem a little inaccurate. For example, I learned that > the MiddleMan.ask_worker() method detailed in the docs is actually > called as MiddleMan.ask_work (). I also saw some typos in the documentation and hence fixing them. > > If anyone has any thoughts on what might be going wrong with my > scheduled tasks, that'd be great. Thanks! There is definitely an exception being thrown in your worker thats not handled. if possible, please wrap your worker method in a begin rescue and see what happens. Currently, before daemonizing workers, i am redirecting their STDOUT,STDIN and STDERR to a file, so as unhandled exceptions gets logged. I will get this better. Long running schedulers are most important thing and hence any bug will be fixed with atmost care. Also, regarding multi tasksing that you want to achieve. New release of bdrb basically encourages cooperative multitasking not preemtpive multitasking of threads.I will code some examples for this. Thanks for your patience and please try the fix i suggested. From andy.tyra at gmail.com Thu Dec 6 19:31:58 2007 From: andy.tyra at gmail.com (Andy Tyra) Date: Thu, 6 Dec 2007 16:31:58 -0800 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: <1196909645.19587.7.camel@shire> References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> <1196909645.19587.7.camel@shire> Message-ID: <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> Hi There, Thanks for the help! I have made some progress. It looks like the worker is crashing because it is not getting passed the argument that it requires to run. That would explain why kicking this worker off manually worked just fine, but doing so in the scheduler did not. The problem is, I need to be able to pass a parameter to this worker's function using the scheduling YML and that doesn't seem to be working. Perhaps you can help troubleshoot? Here is the relevant method from the worker which is named ExpireAndRefreshWorker: def execute_report(report_name) myReport = Report.new(report_name) begin File.delete (File.join(RAILS_ROOT, 'public','reports', myReport.const_name + '.html')) rescue #do nothing end renderController = FakeTest.new renderController.render_report (myReport) end As you can see, it takes one parameter, report_name, which should be a string. Here's the YML I'm trying to use to schedule it: :schedules: :expire_and_refresh_worker: :worker_method: execute_report :data: unfilled_demand :trigger_args: 0 */15 * * * * * :job_key: unfilled_demand_schedule And here's the stack trace of the exception that is thrown when the schedule triggers: (Please note that the line numbers in my stack trace will not match yours for script/backgroundrb. I've added some additional tasks to this script that add YML scheduling rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:91:in `execute_report': wrong number of arguments (0 for 1) (ArgumentError) from rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:91:in `send' from rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:91:in `check_for_timer_events' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:129:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `loop' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/worker.rb:21:in `start_worker' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:152:in `fork_and_load' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:111:in `load_workers' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in `each' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in `load_workers' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in `run' from rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in `initialize' from script/backgroundrb:73:in `new' from script/backgroundrb:73 It appears the parameter specified as :data in the yml is not getting passed to the function. Is there some other way this should be done? I don't see it in the doco. Is it done as :args like in the old version? Also, this is unrelated, but I just thought I'd point it out. The following stack trace appears in the logs when stopping backgroundrb: rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:20:in `read_data': Packet::DisconnectError (Packet::DisconnectError) from rails-root/vendor/plugins/backgroundrb/framework/worker.rb:47:in `handle_internal_messages' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:138:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in `each' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `loop' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/worker.rb:21:in `start_worker' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:152:in `fork_and_load' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:128:in `start_worker' from rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:115:in `initialize' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:18:in `run' from rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in `initialize' from script/backgroundrb:73:in `new' from script/backgroundrb:73 And, a question. I think I've figured out where everything is getting logged. Will this version eventually log things like the old version? The old version would report when workers stopped and started, when schedules were loaded and triggered, etc. Is there somewhere you'd like this sort of thing submitted as a feature request? Thanks, Andy On Dec 5, 2007 6:54 PM, hemant kumar wrote: > > On Wed, 2007-12-05 at 17:51 -0800, Andy Tyra wrote: > > I tried to upgrade my existing application to RC2 last night. Like > > many, I use this mostly for running scheduled tasks. For the moment, > > I've abandoned the effort, but am looking forward to being able to use > > this. Feedback below: > > > > First, the reason I was looking forward to this upgrade was to use the > > threaded scheduler. I have an application with long-running tasks. I > > found that scheduled tasks in previous versions of backgroundrb would > > not run if the worker scheduled before it was still running. This > > seemed counterintuitive to me (given the whole idea behind > > backgroundrb) but after reading documentation and forums, it seems > > many others ran into this issue too. RC2 planned to do away with > > this. > > > > The Good: > > > > - (Still) Easy Installattion > > - Well designed. Logging is centralized, that's nice. > > - Easy to hack up for use in an enterprise environment. > > > > The Bad: > > > > - Lots of arbitrary changes that I don't quite understand. Different > > function names to create workers, new YML formats, different config > > files for scheduler. Would have been great if none of the rails > > application-facing stuff changed so the new version could just be > > swapped in. > > - Logging detail has greatly diminished. It's difficult to > > troubleshoot what's happening when things go wrong. > > - For the moment, this seems less stable than the old version. My > > scheduled tasks are bringing down the whole server and I can't tell > > why (nothing showing up in the logs). This is why I've abandoned the > > upgrade. The scheduler is working fine for lightweight tasks, but the > > long-running ones just bring everything down. For what it's worth, > > the logic for the tasks still lives in my rails models. The workers > > basically just call class methods in the models. (Truly using this as > > a scheduler.) Perhaps this has something to do with it? > > - Documentation and resources for the new version are sparse > > (obviously) and seem a little inaccurate. For example, I learned that > > the MiddleMan.ask_worker() method detailed in the docs is actually > > called as MiddleMan.ask_work (). > > I also saw some typos in the documentation and hence fixing them. > > > > If anyone has any thoughts on what might be going wrong with my > > scheduled tasks, that'd be great. Thanks! > > There is definitely an exception being thrown in your worker thats not > handled. if possible, please wrap your worker method in a begin rescue > and see what happens. Currently, before daemonizing workers, i am > redirecting their STDOUT,STDIN and STDERR to a file, so as unhandled > exceptions gets logged. I will get this better. Long running schedulers > are most important thing and hence any bug will be fixed with atmost > care. > > Also, regarding multi tasksing that you want to achieve. New release of > bdrb basically encourages cooperative multitasking not preemtpive > multitasking of threads.I will code some examples for this. > > Thanks for your patience and please try the fix i suggested. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071206/8b7f86e9/attachment.html From andy.tyra at gmail.com Thu Dec 6 20:31:58 2007 From: andy.tyra at gmail.com (Andy Tyra) Date: Thu, 6 Dec 2007 17:31:58 -0800 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> <1196909645.19587.7.camel@shire> <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> Message-ID: <4f5f5e510712061731j699387b7sccb768b8fc4932d1@mail.gmail.com> Another issue to report. When running the aforementioned worker manually, it runs and seems to work, but the worker process crashes at the end, leaving this stack trace: rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in `dump': can't dump anonymous class # (TypeError) from rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in `dump_object' from rails-root/vendor/plugins/backgroundrb/framework/worker.rb:32:in `send_data' from rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:79:in `send_response' from rails-root/lib/workers/expire_and_refresh_worker.rb:36:in `process_request' from rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:48:in `receive_data' from rails-root/vendor/plugins/backgroundrb/framework/worker.rb:54:in `receive_internal_data' from rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:30:in `extract' from rails-root/vendor/plugins/backgroundrb/framework/worker.rb:52:in `receive_internal_data' ... 11 levels... from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in `run' from rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in `initialize' from script/backgroundrb:73:in `new' from script/backgroundrb:73 It looked like this was because my worker method was not returning anything, so I added the line return "I am done." to the end of the worker method. I am now getting this stack trace: rails-root/vendor/plugins/backgroundrb/framework/meta_pimp.rb:55:in `process_response': You have a nil object when you didn't expect it! (NoMethodError) The error occurred while evaluating nil.instance from rails-root/vendor/plugins/backgroundrb/framework/meta_pimp.rb:28:in `handle_object' from rails-root/vendor/plugins/backgroundrb/framework/meta_pimp.rb:19:in `receive_data' from rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:30:in `extract' from rails-root/vendor/plugins/backgroundrb/framework/meta_pimp.rb:17:in `receive_data' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:81:in `handle_internal_messages' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:138:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in `each' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `loop' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:20:in `run' from rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in `initialize' from script/backgroundrb:73:in `new' from script/backgroundrb:73 On Dec 6, 2007 4:31 PM, Andy Tyra wrote: > Hi There, > > Thanks for the help! I have made some progress. > > It looks like the worker is crashing because it is not getting passed the > argument that it requires to run. That would explain why kicking this > worker off manually worked just fine, but doing so in the scheduler did > not. The problem is, I need to be able to pass a parameter to this worker's > function using the scheduling YML and that doesn't seem to be working. > Perhaps you can help troubleshoot? > > Here is the relevant method from the worker which is named > ExpireAndRefreshWorker: > > def execute_report(report_name) > myReport = Report.new(report_name) > begin > File.delete (File.join(RAILS_ROOT, 'public','reports', > myReport.const_name + '.html')) > rescue > #do nothing > end > renderController = FakeTest.new > renderController.render_report (myReport) > end > > As you can see, it takes one parameter, report_name, which should be a > string. > > Here's the YML I'm trying to use to schedule it: > > :schedules: > :expire_and_refresh_worker: > :worker_method: execute_report > :data: unfilled_demand > :trigger_args: 0 */15 * * * * * > :job_key: unfilled_demand_schedule > > And here's the stack trace of the exception that is thrown when the > schedule triggers: (Please note that the line numbers in my stack trace > will not match yours for script/backgroundrb. I've added some additional > tasks to this script that add YML scheduling > > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:91:in > `execute_report': wrong number of arguments (0 for 1) (ArgumentError) > from > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:91:in `send' > from > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:91:in > `check_for_timer_events' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:129:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `loop' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:21:in > `start_worker' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:152:in > `fork_and_load' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:111:in > `load_workers' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in > `each' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in > `load_workers' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > `run' > from > rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in > `initialize' > from script/backgroundrb:73:in `new' > from script/backgroundrb:73 > > It appears the parameter specified as :data in the yml is not getting > passed to the function. Is there some other way this should be done? I > don't see it in the doco. Is it done as :args like in the old version? > > Also, this is unrelated, but I just thought I'd point it out. The > following stack trace appears in the logs when stopping backgroundrb: > > rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:20:in > `read_data': Packet::DisconnectError (Packet::DisconnectError) > from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:47:in > `handle_internal_messages' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:138:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in `each' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `loop' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:21:in > `start_worker' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:152:in > `fork_and_load' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:128:in > `start_worker' > from > rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:115:in > `initialize' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:18:in > `run' > from > rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in > `initialize' > from script/backgroundrb:73:in `new' > from script/backgroundrb:73 > > And, a question. I think I've figured out where everything is getting > logged. Will this version eventually log things like the old version? The > old version would report when workers stopped and started, when schedules > were loaded and triggered, etc. Is there somewhere you'd like this sort of > thing submitted as a feature request? > > Thanks, > Andy > > > On Dec 5, 2007 6:54 PM, hemant kumar wrote: > > > > > On Wed, 2007-12-05 at 17:51 -0800, Andy Tyra wrote: > > > I tried to upgrade my existing application to RC2 last night. Like > > > many, I use this mostly for running scheduled tasks. For the moment, > > > I've abandoned the effort, but am looking forward to being able to use > > > this. Feedback below: > > > > > > First, the reason I was looking forward to this upgrade was to use the > > > threaded scheduler. I have an application with long-running tasks. I > > > > > found that scheduled tasks in previous versions of backgroundrb would > > > not run if the worker scheduled before it was still running. This > > > seemed counterintuitive to me (given the whole idea behind > > > backgroundrb) but after reading documentation and forums, it seems > > > many others ran into this issue too. RC2 planned to do away with > > > this. > > > > > > The Good: > > > > > > - (Still) Easy Installattion > > > - Well designed. Logging is centralized, that's nice. > > > - Easy to hack up for use in an enterprise environment. > > > > > > The Bad: > > > > > > - Lots of arbitrary changes that I don't quite understand. Different > > > function names to create workers, new YML formats, different config > > > files for scheduler. Would have been great if none of the rails > > > application-facing stuff changed so the new version could just be > > > swapped in. > > > - Logging detail has greatly diminished. It's difficult to > > > troubleshoot what's happening when things go wrong. > > > - For the moment, this seems less stable than the old version. My > > > scheduled tasks are bringing down the whole server and I can't tell > > > why (nothing showing up in the logs). This is why I've abandoned the > > > upgrade. The scheduler is working fine for lightweight tasks, but the > > > > > long-running ones just bring everything down. For what it's worth, > > > the logic for the tasks still lives in my rails models. The workers > > > basically just call class methods in the models. (Truly using this as > > > > > a scheduler.) Perhaps this has something to do with it? > > > - Documentation and resources for the new version are sparse > > > (obviously) and seem a little inaccurate. For example, I learned that > > > the MiddleMan.ask_worker() method detailed in the docs is actually > > > called as MiddleMan.ask_work (). > > > > I also saw some typos in the documentation and hence fixing them. > > > > > > If anyone has any thoughts on what might be going wrong with my > > > scheduled tasks, that'd be great. Thanks! > > > > There is definitely an exception being thrown in your worker thats not > > handled. if possible, please wrap your worker method in a begin rescue > > and see what happens. Currently, before daemonizing workers, i am > > redirecting their STDOUT,STDIN and STDERR to a file, so as unhandled > > exceptions gets logged. I will get this better. Long running schedulers > > are most important thing and hence any bug will be fixed with atmost > > care. > > > > Also, regarding multi tasksing that you want to achieve. New release of > > bdrb basically encourages cooperative multitasking not preemtpive > > multitasking of threads.I will code some examples for this. > > > > Thanks for your patience and please try the fix i suggested. > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071206/edfafb6d/attachment-0001.html From gethemant at gmail.com Thu Dec 6 21:22:34 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 07 Dec 2007 07:52:34 +0530 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> <1196909645.19587.7.camel@shire> <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> Message-ID: <1196994154.9223.4.camel@shire> On Thu, 2007-12-06 at 16:31 -0800, Andy Tyra wrote: > Hi There, > > Thanks for the help! I have made some progress. > > It looks like the worker is crashing because it is not getting passed > the argument that it requires to run. That would explain why kicking > this worker off manually worked just fine, but doing so in the > scheduler did not. The problem is, I need to be able to pass a > parameter to this worker's function using the scheduling YML and that > doesn't seem to be working. Perhaps you can help troubleshoot? > > Here is the relevant method from the worker which is named > ExpireAndRefreshWorker: > > def execute_report(report_name) > myReport = Report.new(report_name) > begin > File.delete (File.join(RAILS_ROOT, 'public','reports', > myReport.const_name + '.html')) > rescue > #do nothing > end > renderController = FakeTest.new > renderController.render_report (myReport) > end > > As you can see, it takes one parameter, report_name, which should be a > string. > > Here's the YML I'm trying to use to schedule it: > > :schedules: > :expire_and_refresh_worker: > :worker_method: execute_report > :data: unfilled_demand > :trigger_args: 0 */15 * * * * * > :job_key: unfilled_demand_schedule Thanks Andy.. with new version you were not able to pass arguments from configuration file and hence the error. But I have fixed this in trunk and your old code should work as usual. > > And here's the stack trace of the exception that is thrown when the > schedule triggers: (Please note that the line numbers in my stack > trace will not match yours for script/backgroundrb. I've added some > additional tasks to this script that add YML scheduling > > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:91:in > `execute_report': wrong number of arguments (0 for 1) (ArgumentError) > from > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:91:in > `send' > from > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:91:in > `check_for_timer_events' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:129:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in > `loop' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:21:in > `start_worker' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:152:in `fork_and_load' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:111:in `load_workers' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in `each' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in `load_workers' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in `run' > from > rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in > `initialize' > from script/backgroundrb:73:in `new' > from script/backgroundrb:73 > > It appears the parameter specified as :data in the yml is not getting > passed to the function. Is there some other way this should be done? > I don't see it in the doco. Is it done as :args like in the old > version? > > Also, this is unrelated, but I just thought I'd point it out. The > following stack trace appears in the logs when stopping backgroundrb: > > rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:20:in `read_data': Packet::DisconnectError (Packet::DisconnectError) > from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:47:in > `handle_internal_messages' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:138:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in > `each' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `loop' > from > rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in > `start_reactor' > from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:21:in > `start_worker' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:152:in `fork_and_load' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:128:in `start_worker' > from > rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:115:in > `initialize' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:18:in `run' > from > rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in > `initialize' > from script/backgroundrb:73:in `new' > from script/backgroundrb:73 > > And, a question. I think I've figured out where everything is getting > logged. Will this version eventually log things like the old version? > The old version would report when workers stopped and started, when > schedules were loaded and triggered, etc. Is there somewhere you'd > like this sort of thing submitted as a feature request? Ok, its not a big deal...We can change so as it logs all such things. Thanks for reports. From gethemant at gmail.com Thu Dec 6 21:24:06 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 07 Dec 2007 07:54:06 +0530 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: <4f5f5e510712061731j699387b7sccb768b8fc4932d1@mail.gmail.com> References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> <1196909645.19587.7.camel@shire> <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> <4f5f5e510712061731j699387b7sccb768b8fc4932d1@mail.gmail.com> Message-ID: <1196994246.9223.7.camel@shire> On Thu, 2007-12-06 at 17:31 -0800, Andy Tyra wrote: > Another issue to report. > > When running the aforementioned worker manually, it runs and seems to > work, but the worker process crashes at the end, leaving this stack > trace: > > rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in `dump': can't dump anonymous class # (TypeError) > from > rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in `dump_object' > from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:32:in > `send_data' > from > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:79:in > `send_response' > from rails-root/lib/workers/expire_and_refresh_worker.rb:36:in > `process_request' > from > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:48:in > `receive_data' > from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:54:in > `receive_internal_data' > from > rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:30:in `extract' > from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:52:in > `receive_internal_data' > ... 11 levels... > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in `run' > from > rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in > `initialize' > from script/backgroundrb:73:in `new' > from script/backgroundrb:73 Can I have a look at your worker script? If not possible..you can send me personally and I will have a look. I do think, its because of return issue. From andy.tyra at gmail.com Thu Dec 6 22:15:06 2007 From: andy.tyra at gmail.com (Andy Tyra) Date: Thu, 6 Dec 2007 19:15:06 -0800 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: <1196994246.9223.7.camel@shire> References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> <1196909645.19587.7.camel@shire> <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> <4f5f5e510712061731j699387b7sccb768b8fc4932d1@mail.gmail.com> <1196994246.9223.7.camel@shire> Message-ID: <4f5f5e510712061915r59c6f721qff286249d702d770@mail.gmail.com> Sure, here is the worker code in its entirety. For a little background, this is using the test framework to call a controller function which executes and renders a report. It's done this way because the previous version of backgroundrb did not support accessing and calling controllers. (If this has changed, that would be awesome, but I'm guessing it has not.) That's what the extra "FakeTest" class is for. require 'active_support' require 'action_controller' require 'action_controller/test_process' require 'action_view' class ExpireAndRefreshWorker < BackgrounDRb::MetaWorker set_worker_name :expire_and_refresh_worker def create(args = nil) # this method is called, when worker is loaded for the first time end def execute_report(report_name) myReport = Report.new(report_name) begin File.delete(File.join(RAILS_ROOT, 'public','reports', myReport.const_name + '.html')) rescue #do nothing end renderController = FakeTest.new renderController.render_report(myReport) return "I am done." end def process_request(p_data) user_input = p_data[:data] result = self.send(user_input[:worker_method],user_input[:data]) send_response(p_data,result) end end class FakeTest include ActionController::TestProcess def initialize require_dependency 'application' unless defined?(ApplicationController) @controller = ReportsController.new @request = ActionController::TestRequest.new @response = ActionController::TestResponse.new end def render_report(myReport) get :run, {:id => myReport.const_name} @response end end On Dec 6, 2007 6:24 PM, hemant kumar wrote: > > On Thu, 2007-12-06 at 17:31 -0800, Andy Tyra wrote: > > Another issue to report. > > > > When running the aforementioned worker manually, it runs and seems to > > work, but the worker process crashes at the end, leaving this stack > > trace: > > > > > rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > `dump': can't dump anonymous class # (TypeError) > > from > > > rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > `dump_object' > > from > > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:32:in > > `send_data' > > from > > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:79:in > > `send_response' > > from rails-root/lib/workers/expire_and_refresh_worker.rb:36:in > > `process_request' > > from > > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:48:in > > `receive_data' > > from > > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:54:in > > `receive_internal_data' > > from > > > rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:30:in > `extract' > > from > > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:52:in > > `receive_internal_data' > > ... 11 levels... > > from > > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > `run' > > from > > rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:114:in > > `initialize' > > from script/backgroundrb:73:in `new' > > from script/backgroundrb:73 > > Can I have a look at your worker script? If not possible..you can send > me personally and I will have a look. > > I do think, its because of return issue. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071206/a06bf598/attachment.html From gethemant at gmail.com Fri Dec 7 05:39:10 2007 From: gethemant at gmail.com (hemant) Date: Fri, 7 Dec 2007 16:09:10 +0530 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: <4f5f5e510712061915r59c6f721qff286249d702d770@mail.gmail.com> References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> <1196909645.19587.7.camel@shire> <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> <4f5f5e510712061731j699387b7sccb768b8fc4932d1@mail.gmail.com> <1196994246.9223.7.camel@shire> <4f5f5e510712061915r59c6f721qff286249d702d770@mail.gmail.com> Message-ID: On Dec 7, 2007 8:45 AM, Andy Tyra wrote: > Sure, here is the worker code in its entirety. For a little background, > this is using the test framework to call a controller function which > executes and renders a report. It's done this way because the previous > version of backgroundrb did not support accessing and calling controllers. > (If this has changed, that would be awesome, but I'm guessing it has not.) > That's what the extra "FakeTest" class is for. > > require 'active_support' > require 'action_controller' > require 'action_controller/test_process' > require 'action_view' > > class ExpireAndRefreshWorker < BackgrounDRb::MetaWorker > set_worker_name :expire_and_refresh_worker > def create(args = nil) > # this method is called, when worker is loaded for the first time > end > > > def execute_report(report_name) > myReport = Report.new(report_name) > begin > File.delete(File.join(RAILS_ROOT, 'public','reports', > myReport.const_name + '.html')) > rescue > #do nothing > end > renderController = FakeTest.new > renderController.render_report(myReport) > return "I am done." > end > > def process_request(p_data) > user_input = p_data[:data] > result = self.send(user_input[:worker_method],user_input[:data]) > send_response(p_data,result) > end > end > > class FakeTest > > include ActionController::TestProcess > > def initialize > require_dependency 'application' unless > defined?(ApplicationController) > @controller = ReportsController.new > @request = ActionController::TestRequest.new > @response = ActionController::TestResponse.new > end > > def render_report(myReport) > get :run, {:id => myReport.const_name} > @response > end > end > > Hi Andy, I am trying to reproduce this. Currently since, we do a require 'environment' all your controllers are available in workers too. However, the error you are reporting seems to have happen when there is communication back and forth between workers or master. But i do not see that case either, since your worker is running in a pretty stand alone manner. What i doubt is, either rails or some other lib that rails is using having a method called "send_response" defined..and thats causing some conflict or something. Let me see it further. From russellkt at gmail.com Fri Dec 7 17:11:10 2007 From: russellkt at gmail.com (Kevin Russell) Date: Fri, 7 Dec 2007 16:11:10 -0600 Subject: [Backgroundrb-devel] Working in linux, scheduler working mac 10.4, triggers don't work Message-ID: Thanks so much for backgroundrb, it's nice to be able to keep the moving parts to a minimum. I'm noting some odd behavior on ruby 1.8.5, rails 1.2.5, backroundrb trunk on a macbook. When scheduling a task everything works perfectly, but triggers are not calling the methods, only blocking the background drb instance. Logs show no errors. Even more interesting is the fact that everything works fine on linux. Out of curiosity, I ran wireshark and think I've found the problem, truncated data. Here's what was sent: 000000073..{.:.worker_method:.load_policies:.worker:.renewal_worker:.type:.do_wor which would explain why everything is blocked. Has anyone seen this behavior or have any thoughts on what is happening? Thanks, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071207/eee8ba5d/attachment-0001.html From gethemant at gmail.com Sat Dec 8 01:50:59 2007 From: gethemant at gmail.com (hemant kumar) Date: Sat, 08 Dec 2007 12:20:59 +0530 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> <1196909645.19587.7.camel@shire> <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> <4f5f5e510712061731j699387b7sccb768b8fc4932d1@mail.gmail.com> <1196994246.9223.7.camel@shire> <4f5f5e510712061915r59c6f721qff286249d702d770@mail.gmail.com> Message-ID: <1197096659.6645.6.camel@shire> On Fri, 2007-12-07 at 16:09 +0530, hemant wrote: > On Dec 7, 2007 8:45 AM, Andy Tyra wrote: > > Sure, here is the worker code in its entirety. For a little background, > > this is using the test framework to call a controller function which > > executes and renders a report. It's done this way because the previous > > version of backgroundrb did not support accessing and calling controllers. > > (If this has changed, that would be awesome, but I'm guessing it has not.) > > That's what the extra "FakeTest" class is for. > > > > require 'active_support' > > require 'action_controller' > > require 'action_controller/test_process' > > require 'action_view' > > > > class ExpireAndRefreshWorker < BackgrounDRb::MetaWorker > > set_worker_name :expire_and_refresh_worker > > def create(args = nil) > > # this method is called, when worker is loaded for the first time > > end > > > > > > def execute_report(report_name) > > myReport = Report.new(report_name) > > begin > > File.delete(File.join(RAILS_ROOT, 'public','reports', > > myReport.const_name + '.html')) > > rescue > > #do nothing > > end > > renderController = FakeTest.new > > renderController.render_report(myReport) > > return "I am done." > > end > > > > def process_request(p_data) > > user_input = p_data[:data] > > result = self.send(user_input[:worker_method],user_input[:data]) > > send_response(p_data,result) > > end > > end > > > > class FakeTest > > > > include ActionController::TestProcess > > > > def initialize > > require_dependency 'application' unless > > defined?(ApplicationController) > > @controller = ReportsController.new > > @request = ActionController::TestRequest.new > > @response = ActionController::TestResponse.new > > end > > > > def render_report(myReport) > > get :run, {:id => myReport.const_name} > > @response > > end > > end > > > > Hi Andy, Sync with latest source code. The bug you encountered should be fixed now. And get rid of older ./script/backgroundrb file and do a setup using: rake backgroundrb:setup Also, now in configuration file you can mention: backgroundrb.yml: :backgroundrb: :port: 11006 :ip: 0.0.0.0 :log: foreground and start backgroundrb in foreground mode using: ./script/backgroundrb this would make sure that all the exceptions appear on stdout. Give it a shot... From gethemant at gmail.com Sat Dec 8 01:52:47 2007 From: gethemant at gmail.com (hemant) Date: Sat, 8 Dec 2007 12:22:47 +0530 Subject: [Backgroundrb-devel] backgroundrb not cleaning up tmp files? In-Reply-To: <84F1E20A-7F11-4896-A054-F035A3B05C0C@gmail.com> References: <84F1E20A-7F11-4896-A054-F035A3B05C0C@gmail.com> Message-ID: Eric, Please try upgrading to latest version. I am putting lots of features and fixes there. On Dec 1, 2007 12:47 AM, Erik Morton wrote: > I just had an interesting issue on a production system that has been > running nicely for about a year. There were approximately 10,000 > backgroundrb temp files in the /tmp directory. It looks like > backgroundrb wasn't cleaning them up. > > I noticed it with the following error: > DRb::DRbConnError: drbunix:///tmp/backgroundrb.20015/ > backgroundrb_logger_0_0.101544829808665 - # file or directory - ///tmp/backgroundrb.20015/ > backgroundrb_logger_0_0.101544829808665> > /usr/lib/ruby/1.8/drb/drb.rb:736:in `open' > /usr/lib/ruby/1.8/drb/drb.rb:729:in `each' > /usr/lib/ruby/1.8/drb/drb.rb:729:in `open' > /usr/lib/ruby/1.8/drb/drb.rb:1189:in `initialize' > /usr/lib/ruby/1.8/drb/drb.rb:1169:in `new' > /usr/lib/ruby/1.8/drb/drb.rb:1169:in `open' > /usr/lib/ruby/1.8/drb/drb.rb:1085:in `method_missing' > /usr/lib/ruby/1.8/drb/drb.rb:1103:in `with_friend' > /usr/lib/ruby/1.8/drb/drb.rb:1084:in `method_missing' > /apps/eisweb/releases/20071120022316/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:42:in `initialize' > /apps/eisweb/releases/20071120022316/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:420:in `new' > /apps/eisweb/releases/20071120022316/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:420:in `worker' > /apps/eisweb/releases/20071120022316/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/worker.rb:37:in `logger' > #{RAILS_ROOT}/app/helpers/distribution_helper.rb:4:in > `do_distribution' > > Has anyone else seen this? > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From andy.tyra at gmail.com Sat Dec 8 03:26:40 2007 From: andy.tyra at gmail.com (Andy Tyra) Date: Sat, 8 Dec 2007 00:26:40 -0800 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: <1197096659.6645.6.camel@shire> References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> <1196909645.19587.7.camel@shire> <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> <4f5f5e510712061731j699387b7sccb768b8fc4932d1@mail.gmail.com> <1196994246.9223.7.camel@shire> <4f5f5e510712061915r59c6f721qff286249d702d770@mail.gmail.com> <1197096659.6645.6.camel@shire> Message-ID: <4f5f5e510712080026q27f9d78v3b82ce39a160ed82@mail.gmail.com> I'll try this over the weekend. Found one bug. Ironically, it occurs if you have NO schedule defined. :) Below is the stack trace and below that is the diff on the file that seems to fix it. rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:27:in `worker_init': You have a nil object when you didn't expect it! (NoMethodError) You might have expected an instance of Array. The error occurred while evaluating nil.[] from rails-root/vendor/plugins/backgroundrb/framework/worker.rb:20:in `start_worker' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:152:in `fork_and_load' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:111:in `load_workers' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in `each' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in `load_workers' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in `run' from rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:116:in `initialize' from script/backgroundrb:25:in `new' from script/backgroundrb:25 svn diff meta_worker.rb Index: meta_worker.rb =================================================================== --- meta_worker.rb (revision 227) +++ meta_worker.rb (working copy) @@ -24,7 +24,7 @@ @logger = PacketLogger.new(self) if(@worker_options[:schedule] && no_auto_load) load_schedule_from_args - elsif t_schedule = @config_file[:schedules][worker_name.to_sym] + elsif !@config_file[:schedules].nil? and t_schedule = @config_file[:schedules][worker_name.to_sym] @my_schedule = t_schedule @worker_method_arity = self.method (@my_schedule[:worker_method]).arity load_schedule if @my_schedule Thx, Andy On Dec 7, 2007 10:50 PM, hemant kumar wrote: > > On Fri, 2007-12-07 at 16:09 +0530, hemant wrote: > > On Dec 7, 2007 8:45 AM, Andy Tyra wrote: > > > Sure, here is the worker code in its entirety. For a little > background, > > > this is using the test framework to call a controller function which > > > executes and renders a report. It's done this way because the > previous > > > version of backgroundrb did not support accessing and calling > controllers. > > > (If this has changed, that would be awesome, but I'm guessing it has > not.) > > > That's what the extra "FakeTest" class is for. > > > > > > require 'active_support' > > > require 'action_controller' > > > require 'action_controller/test_process' > > > require 'action_view' > > > > > > class ExpireAndRefreshWorker < BackgrounDRb::MetaWorker > > > set_worker_name :expire_and_refresh_worker > > > def create(args = nil) > > > # this method is called, when worker is loaded for the first time > > > end > > > > > > > > > def execute_report(report_name) > > > myReport = Report.new(report_name) > > > begin > > > File.delete(File.join(RAILS_ROOT, 'public','reports', > > > myReport.const_name + '.html')) > > > rescue > > > #do nothing > > > end > > > renderController = FakeTest.new > > > renderController.render_report(myReport) > > > return "I am done." > > > end > > > > > > def process_request(p_data) > > > user_input = p_data[:data] > > > result = self.send(user_input[:worker_method],user_input[:data]) > > > send_response(p_data,result) > > > end > > > end > > > > > > class FakeTest > > > > > > include ActionController::TestProcess > > > > > > def initialize > > > require_dependency 'application' unless > > > defined?(ApplicationController) > > > @controller = ReportsController.new > > > @request = ActionController::TestRequest.new > > > @response = ActionController::TestResponse.new > > > end > > > > > > def render_report(myReport) > > > get :run, {:id => myReport.const_name} > > > @response > > > end > > > end > > > > > > > > Hi Andy, > > Sync with latest source code. The bug you encountered should be fixed > now. > > And get rid of older ./script/backgroundrb file and do a setup using: > > rake backgroundrb:setup > > Also, now in configuration > file you can mention: > > backgroundrb.yml: > > :backgroundrb: > :port: 11006 > :ip: 0.0.0.0 > :log: foreground > > and start backgroundrb in foreground mode using: > > ./script/backgroundrb > > this would make sure that all the exceptions appear on stdout. Give it a > shot... > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071208/120760c8/attachment.html From andy.tyra at gmail.com Sat Dec 8 05:03:19 2007 From: andy.tyra at gmail.com (Andy Tyra) Date: Sat, 8 Dec 2007 02:03:19 -0800 Subject: [Backgroundrb-devel] Feedback on RC2 In-Reply-To: <4f5f5e510712080026q27f9d78v3b82ce39a160ed82@mail.gmail.com> References: <4f5f5e510712051751i6be965e2ie1f392cad317b702@mail.gmail.com> <1196909645.19587.7.camel@shire> <4f5f5e510712061631x7f8d9676la9e5eae6f61d08dc@mail.gmail.com> <4f5f5e510712061731j699387b7sccb768b8fc4932d1@mail.gmail.com> <1196994246.9223.7.camel@shire> <4f5f5e510712061915r59c6f721qff286249d702d770@mail.gmail.com> <1197096659.6645.6.camel@shire> <4f5f5e510712080026q27f9d78v3b82ce39a160ed82@mail.gmail.com> Message-ID: <4f5f5e510712080203r6098ab96oc86d9dcee044903e@mail.gmail.com> Looks like this is working! Thanks so much! I'll try merging this into the app's mainline tomorrow. One other small bug to report. Backgroundrb doesn't seem to exit once it has been started and successfully run a worker. I checked for the pid file and was surprised to find it wasn't there. (This is *WHILE *backgroundrb was running - I never called stop.) Checked the logs and I found the stack below. It seems like backgroundrb is calling "stop" on itself at some point while it is running. Does this make any sense? rails-root/config/../vendor/plugins/backgroundrb/lib/../framework/nbio.rb:20:in `read_data': Packet::DisconnectError (Packet::DisconnectError) from rails-root/vendor/plugins/backgroundrb/framework/worker.rb:47:in `handle_internal_messages' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:138:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in `each' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:136:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `loop' from rails-root/vendor/plugins/backgroundrb/framework/core.rb:128:in `start_reactor' from rails-root/vendor/plugins/backgroundrb/framework/worker.rb:21:in `start_worker' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:152:in `fork_and_load' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:128:in `start_worker' from rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:117:in `initialize' from rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:18:in `run' from rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:116:in `initialize' from script/backgroundrb:75:in `new' from script/backgroundrb:75 gins/backgroundrb/server/master_worker.rb:116:in `initialize' from script/backgroundrb:75:in `new' from script/backgroundrb:75 On Dec 8, 2007 12:26 AM, Andy Tyra wrote: > I'll try this over the weekend. > > Found one bug. Ironically, it occurs if you have NO schedule defined. > :) Below is the stack trace and below that is the diff on the file that > seems to fix it. > > rails-root/vendor/plugins/backgroundrb/server/meta_worker.rb:27:in > `worker_init': You have a nil object when you didn't expect it! > (NoMethodError) > You might have expected an instance of Array. > The error occurred while evaluating nil.[] from > rails-root/vendor/plugins/backgroundrb/framework/worker.rb:20:in > `start_worker' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:152:in > `fork_and_load' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:111:in > `load_workers' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in > `each' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:106:in > `load_workers' > from > rails-root/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > `run' > from > rails-root/vendor/plugins/backgroundrb/server/master_worker.rb:116:in > `initialize' > from script/backgroundrb:25:in `new' > from script/backgroundrb:25 > > > svn diff meta_worker.rb > Index: meta_worker.rb > =================================================================== > --- meta_worker.rb (revision 227) > +++ meta_worker.rb (working copy) > @@ -24,7 +24,7 @@ > @logger = PacketLogger.new(self) > if(@worker_options[:schedule] && no_auto_load) > load_schedule_from_args > - elsif t_schedule = @config_file[:schedules][worker_name.to_sym] > + elsif !@config_file[:schedules].nil? and t_schedule = > @config_file[:schedules][worker_name.to_sym] > @my_schedule = t_schedule > @worker_method_arity = self.method(@my_schedule[:worker_method]).arity > > load_schedule if @my_schedule > > Thx, > Andy > > > On Dec 7, 2007 10:50 PM, hemant kumar wrote: > > > > > On Fri, 2007-12-07 at 16:09 +0530, hemant wrote: > > > On Dec 7, 2007 8:45 AM, Andy Tyra wrote: > > > > Sure, here is the worker code in its entirety. For a little > > background, > > > > this is using the test framework to call a controller function which > > > > executes and renders a report. It's done this way because the > > previous > > > > version of backgroundrb did not support accessing and calling > > controllers. > > > > (If this has changed, that would be awesome, but I'm guessing it has > > not.) > > > > That's what the extra "FakeTest" class is for. > > > > > > > > require 'active_support' > > > > require 'action_controller' > > > > require 'action_controller/test_process' > > > > require 'action_view' > > > > > > > > class ExpireAndRefreshWorker < BackgrounDRb::MetaWorker > > > > set_worker_name :expire_and_refresh_worker > > > > def create(args = nil) > > > > # this method is called, when worker is loaded for the first > > time > > > > end > > > > > > > > > > > > def execute_report(report_name) > > > > myReport = Report.new(report_name) > > > > begin > > > > File.delete(File.join(RAILS_ROOT, > > 'public','reports', > > > > myReport.const_name + '.html')) > > > > rescue > > > > #do nothing > > > > end > > > > renderController = FakeTest.new > > > > renderController.render_report(myReport) > > > > return "I am done." > > > > end > > > > > > > > def process_request(p_data) > > > > user_input = p_data[:data] > > > > result = self.send(user_input[:worker_method],user_input[:data]) > > > > send_response(p_data,result) > > > > end > > > > end > > > > > > > > class FakeTest > > > > > > > > include ActionController::TestProcess > > > > > > > > def initialize > > > > require_dependency 'application' unless > > > > defined?(ApplicationController) > > > > @controller = ReportsController.new > > > > @request = ActionController::TestRequest.new > > > > @response = ActionController::TestResponse.new > > > > end > > > > > > > > def render_report(myReport) > > > > get :run, {:id => myReport.const_name} > > > > @response > > > > end > > > > end > > > > > > > > > > > > Hi Andy, > > > > Sync with latest source code. The bug you encountered should be fixed > > now. > > > > And get rid of older ./script/backgroundrb file and do a setup using: > > > > rake backgroundrb:setup > > > > Also, now in configuration > > file you can mention: > > > > backgroundrb.yml: > > > > :backgroundrb: > > :port: 11006 > > :ip: 0.0.0.0 > > :log: foreground > > > > and start backgroundrb in foreground mode using: > > > > ./script/backgroundrb > > > > this would make sure that all the exceptions appear on stdout. Give it a > > > > shot... > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071208/eade788a/attachment-0001.html From gethemant at gmail.com Sat Dec 8 14:21:09 2007 From: gethemant at gmail.com (hemant kumar) Date: Sun, 09 Dec 2007 00:51:09 +0530 Subject: [Backgroundrb-devel] Another fix release for backgroundrb rc2 Message-ID: <1197141669.6272.3.camel@shire> Hi Folks, Thank you for feedback and error reports. I have fixed most of the errors that were reported and were valid. I have also removed some cruft that was way too clunky. Here is a brief summary of changes : - Now, you don't define method process_request method in your worker. Its no longer required. Whatever methods, you want to call from rails should be directly defined in your worker and you are good to go. Thus, please remove process_request method that was created by worker generator. - Worker generator has been updated to take care of case #1. - Some issues were found with bin_parser class and they are fixed now. - In most of the cases, now a invalid object dump won't result in worker crash and you should be moving happily even after that. But keep an eye on your log file. - Introduced a :log: foreground option for debugging purpose. Installation: - Get it from here : http://svn.devjavu.com/backgroundrb/trunk/ - Please remove script/backgroundrb and update your config file. And then run rake backgroundrb:setup Also, make sure to read the README file available at http://backgroundrb.rubyforge.org/ . I have updated the documenation according to fixes and changes. Again, thank you all for bug reports and feedbacks. PS: thanks Andy and Kevin and please check if this new version is working for you. From andy.tyra at gmail.com Sat Dec 8 18:13:17 2007 From: andy.tyra at gmail.com (Andy Tyra) Date: Sat, 8 Dec 2007 15:13:17 -0800 Subject: [Backgroundrb-devel] Multiple Worker Methods on Different Schedules Message-ID: <4f5f5e510712081513w1b39bcb8x166de1ac9d53f521@mail.gmail.com> Quick question: If I want to schedule different worker methods defined in the same worker to trigger at different times, is this possible? Would the following scheduler yaml work? :schedules: :foo_worker: :worker_method: method1 :trigger_args: 0 */10 * * * * * :job_key: some_key_1 :foo_worker: :worker_method: method2 :trigger_args: 0 */15 * * * * * :job_key: some_key_2 Would this work? Is there some other way to do it? Would the second method need to be pulled into a separate worker? Thanks, Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071208/369de149/attachment.html From gethemant at gmail.com Sun Dec 9 01:25:13 2007 From: gethemant at gmail.com (hemant kumar) Date: Sun, 09 Dec 2007 11:55:13 +0530 Subject: [Backgroundrb-devel] Multiple Worker Methods on Different Schedules In-Reply-To: <4f5f5e510712081513w1b39bcb8x166de1ac9d53f521@mail.gmail.com> References: <4f5f5e510712081513w1b39bcb8x166de1ac9d53f521@mail.gmail.com> Message-ID: <1197181513.6272.15.camel@shire> On Sat, 2007-12-08 at 15:13 -0800, Andy Tyra wrote: > Quick question: If I want to schedule different worker methods > defined in the same worker to trigger at different times, is this > possible? Would the following scheduler yaml work? > > :schedules: > :foo_worker: > :worker_method: method1 > :trigger_args: 0 */10 * * * * * > :job_key: some_key_1 > :foo_worker: > :worker_method: method2 > :trigger_args: 0 */15 * * * * * > :job_key: some_key_2 > > > Would this work? Is there some other way to do it? Would the second > method need to be pulled into a separate worker? > Currently you can't do that. But its a nice to have this. I would like to have something like this: :schedules: :foo_worker: :foobar: :trigger_args: */5 * * * * * * :data: hello_world :barbar: :trigger_args: */10 * * * * * * :data: Bar_World Can be easily implemented, but again, we will have to change format of backgroundrb.yml because of this. Is this ok? From andy.tyra at gmail.com Sun Dec 9 04:28:49 2007 From: andy.tyra at gmail.com (Andy Tyra) Date: Sun, 9 Dec 2007 01:28:49 -0800 Subject: [Backgroundrb-devel] Multiple Worker Methods on Different Schedules In-Reply-To: <1197181513.6272.15.camel@shire> References: <4f5f5e510712081513w1b39bcb8x166de1ac9d53f521@mail.gmail.com> <1197181513.6272.15.camel@shire> Message-ID: <4f5f5e510712090128r7399208bof3eba1d767150ec@mail.gmail.com> I think it's ideal. If it's easy, do it! This will make backgroundrb much more flexible. The old version (and when I say that, I mean 0.2.X) did something similar. I think your proposed YML would work just fine. Thanks! On Dec 8, 2007 10:25 PM, hemant kumar wrote: > On Sat, 2007-12-08 at 15:13 -0800, Andy Tyra wrote: > > Quick question: If I want to schedule different worker methods > > defined in the same worker to trigger at different times, is this > > possible? Would the following scheduler yaml work? > > > > :schedules: > > :foo_worker: > > :worker_method: method1 > > :trigger_args: 0 */10 * * * * * > > :job_key: some_key_1 > > :foo_worker: > > :worker_method: method2 > > :trigger_args: 0 */15 * * * * * > > :job_key: some_key_2 > > > > > > Would this work? Is there some other way to do it? Would the second > > method need to be pulled into a separate worker? > > > > Currently you can't do that. But its a nice to have this. I would like > to have something like this: > > :schedules: > :foo_worker: > :foobar: > :trigger_args: */5 * * * * * * > :data: hello_world > :barbar: > :trigger_args: */10 * * * * * * > :data: Bar_World > > Can be easily implemented, but again, we will have to change format of > backgroundrb.yml because of this. Is this ok? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071209/1db5f767/attachment.html From andy.tyra at gmail.com Sun Dec 9 15:45:12 2007 From: andy.tyra at gmail.com (Andy Tyra) Date: Sun, 9 Dec 2007 12:45:12 -0800 Subject: [Backgroundrb-devel] Multiple Worker Methods on Different Schedules In-Reply-To: <4f5f5e510712090128r7399208bof3eba1d767150ec@mail.gmail.com> References: <4f5f5e510712081513w1b39bcb8x166de1ac9d53f521@mail.gmail.com> <1197181513.6272.15.camel@shire> <4f5f5e510712090128r7399208bof3eba1d767150ec@mail.gmail.com> Message-ID: <4f5f5e510712091245s33473261weeb47554cb299d3@mail.gmail.com> Another thought on this. The proposed YML would not support invoking a single method on different schedules with different sets of data. What about this? :schedules: :foo_worker: :foo_job_key_1: :worker_method: foo_method :data: foo_data :trigger_args: 0 */10 * * * * * :foo_job_key_2: :worker_method: foo_method :data: bar_data :trigger_args: 0 */15 * * * * * Possible? On Dec 9, 2007 1:28 AM, Andy Tyra wrote: > I think it's ideal. If it's easy, do it! This will make backgroundrb > much more flexible. The old version (and when I say that, I mean 0.2.X) > did something similar. I think your proposed YML would work just fine. > > Thanks! > > > On Dec 8, 2007 10:25 PM, hemant kumar wrote: > > > On Sat, 2007-12-08 at 15:13 -0800, Andy Tyra wrote: > > > Quick question: If I want to schedule different worker methods > > > defined in the same worker to trigger at different times, is this > > > possible? Would the following scheduler yaml work? > > > > > > :schedules: > > > :foo_worker: > > > :worker_method: method1 > > > :trigger_args: 0 */10 * * * * * > > > :job_key: some_key_1 > > > :foo_worker: > > > :worker_method: method2 > > > :trigger_args: 0 */15 * * * * * > > > :job_key: some_key_2 > > > > > > > > > Would this work? Is there some other way to do it? Would the second > > > method need to be pulled into a separate worker? > > > > > > > Currently you can't do that. But its a nice to have this. I would like > > to have something like this: > > > > :schedules: > > :foo_worker: > > :foobar: > > :trigger_args: */5 * * * * * * > > :data: hello_world > > :barbar: > > :trigger_args: */10 * * * * * * > > :data: Bar_World > > > > Can be easily implemented, but again, we will have to change format of > > backgroundrb.yml because of this. Is this ok? > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071209/c84e251d/attachment.html From gethemant at gmail.com Mon Dec 10 18:47:51 2007 From: gethemant at gmail.com (hemant) Date: Tue, 11 Dec 2007 05:17:51 +0530 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now Message-ID: Hi Folks, We are glad to announce shiny new release of BackgrounDRb, which will soon become 1.0. A quick summary of changes: - BackgrounDRb is no londer DRb, its based on event driven network programming library packet ( http://www.packet.googlecode.com ) . - Since we moved to packet, many nasty thread issues, result hash corruption issues are totally gone. Lots of work has went in making scheduler rock solid stable. - Each worker, still runs in its own process, but each worker has a event loop of its own and all the events are triggered by the internal reactor loop. In a nutshell, you are not encouraged to use threads in your workers now. All the workers are already concurrent, but you are encouraged to use co-operative multitasking, rather than pre-emptive. A simple example is, For implement something like progress bar in old version of bdrb, you would: - start your processing a thread (so as your worker can receive further request from rails ) and have a instance variable ( protected by mutex ) which is updated on progress and can be send to rails. With new backgroundrb, progrss bar would be: - process your damn request and just use register_status() to register status of your worker. Just because you are doing some processing won't mean that your worker will block. It can still receive requests from rails. - Now, you can schedule mulitple methods with their own triggers. - Inside each worker, you can start tcp server or connect to a external server. Two important methods available in all workers are: start_server("localhost",port,ModuleName) connect("localhost",port,ModuleName) Connected client or outgoing connection would be integrated with Event Loop and you can process requests from these guys asynchronously. This mouse trap can allow you to build truly distributed workers across your network. The detailed list of changes can be found here: http://backgroundrb.rubyforge.org/ Please give it a try and let me know if you found any bugs. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From brandon at opensoul.org Mon Dec 10 23:02:38 2007 From: brandon at opensoul.org (Brandon Keepers) Date: Mon, 10 Dec 2007 23:02:38 -0500 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now In-Reply-To: References: Message-ID: <70A68453-B1F7-4FDF-BE52-4EA0146E4ADB@opensoul.org> On Dec 10, 2007, at 6:47 PM, hemant wrote: > Hi Folks, > > We are glad to announce shiny new release of BackgrounDRb, which will > soon become 1.0. > > A quick summary of changes: > > - BackgrounDRb is no londer DRb, its based on event driven network > programming library packet ( http://www.packet.googlecode.com ) . > > - Since we moved to packet, many nasty thread issues, result hash > corruption issues are totally gone. Lots of work has went in > making scheduler rock solid stable. > > - Each worker, still runs in its own process, but each worker has a > event loop of its own and all the events are triggered by the internal > reactor loop. In a nutshell, you are not encouraged to use threads > in your workers now. All the workers are already concurrent, but you > are encouraged to use co-operative multitasking, rather than > pre-emptive. A simple example is, > > For implement something like progress bar in old version of bdrb, > you would: > - start your processing a thread (so as your worker can receive > further request from rails ) and have a instance > variable ( protected by mutex ) which is updated on progress and > can be send to rails. > > With new backgroundrb, progrss bar would be: > - process your damn request and just use register_status() to > register status of your worker. Just because > you are doing some processing won't mean that your worker will > block. It can still receive requests from rails. > > > - Now, you can schedule mulitple methods with their own triggers. > > - Inside each worker, you can start tcp server or connect to a > external server. Two important methods available in all workers are: > > start_server("localhost",port,ModuleName) > connect("localhost",port,ModuleName) > > Connected client or outgoing connection would be integrated with > Event Loop and you can process requests from these guys > asynchronously. This mouse trap can allow you to build truly > distributed workers across your network. > > > The detailed list of changes can be found here: > > http://backgroundrb.rubyforge.org/ > > Please give it a try and let me know if you found any bugs. Thanks for all your hard work on this recently. At one point with the old version it was fairly straight-forward to test workers, but that broke at one point. Could you give any pointers writing tests for workers in the new version? Thanks, Brandon -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071210/22882edb/attachment-0001.bin From emarceta at gmail.com Tue Dec 11 04:53:13 2007 From: emarceta at gmail.com (Emil Marceta) Date: Tue, 11 Dec 2007 10:53:13 +0100 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now In-Reply-To: References: Message-ID: <74b000990712110153h7f6ef991p8034fb4d1ae1a281@mail.gmail.com> Hi there, On Dec 11, 2007 12:47 AM, hemant wrote: > Hi Folks, > > We are glad to announce shiny new release of BackgrounDRb, which will > soon become 1.0. > > A quick summary of changes: > > - BackgrounDRb is no londer DRb, its based on event driven network > programming library packet ( http://www.packet.googlecode.com ) . > > - Since we moved to packet, many nasty thread issues, result hash > corruption issues are totally gone. Lots of work has went in > making scheduler rock solid stable. Does this means that slave/daemons are not the dependency anymore? > - Each worker, still runs in its own process, but each worker has a > event loop of its own and all the events are triggered by the internal > reactor loop. In a nutshell, you are not encouraged to use threads > in your workers now. By 'not encouraged' do you mean that 1.0 is not supporting multiple threads in the worker or just as a general guidance? Could you please comment, how would you approach the following scenario with 1.0. Currently, we have a worker that creates threads that process financial payment transactions. An http request sends several 10s or 100s payment transaction records. They are handled by the single worker instance. Within the worker there is a pool of threads created that is calculated based on the number of transactions. For example for 200 transactions there will be 20 threads where each thread handles 10 requests in a squence. Each transaction takes about 3-5 seconds, so our throughput is significantly improved by internal worker parallelization with a thread pool. The worker periodically updates custom backgroundjob databse record, so that following ajax request from the client can read the status of the worker process. The job is identified with the worker key that is stored in the session. > All the workers are already concurrent, but you > are encouraged to use co-operative multitasking, rather than > pre-emptive. A simple example is, > > For implement something like progress bar in old version of bdrb, you would: > - start your processing a thread (so as your worker can receive > further request from rails ) and have a instance > variable ( protected by mutex ) which is updated on progress and > can be send to rails. > > With new backgroundrb, progrss bar would be: > - process your damn request and just use register_status() to > register status of your worker. Just because > you are doing some processing won't mean that your worker will > block. It can still receive requests from rails. How this works with fastcgi or multiple mongrel based engines where it is not guaranteed to hit the same process with the next request? We are using custom database tables and code for sharing the status information now but I was wandering whether the plumbing includes something to address this. thanks, emil From gethemant at gmail.com Tue Dec 11 05:50:48 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 11 Dec 2007 16:20:48 +0530 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now In-Reply-To: <74b000990712110153h7f6ef991p8034fb4d1ae1a281@mail.gmail.com> References: <74b000990712110153h7f6ef991p8034fb4d1ae1a281@mail.gmail.com> Message-ID: <1197370248.9002.13.camel@shire> Hi On Tue, 2007-12-11 at 10:53 +0100, Emil Marceta wrote: > Does this means that slave/daemons are not the dependency anymore? Yes, its gone. bdrb no longer depends on slave and daemons. > By 'not encouraged' do you mean that 1.0 is not supporting multiple > threads in the worker or just as a general guidance? > > Could you please comment, how would you approach the following > scenario with 1.0. Currently, we have a worker that creates threads > that process financial payment transactions. An http request sends > several 10s or 100s payment transaction records. They are handled by > the single worker instance. Within the worker there is a pool of > threads created that is calculated based on the number of > transactions. For example for 200 transactions there will be 20 > threads where each thread handles 10 requests in a squence. Each > transaction takes about 3-5 seconds, so our throughput is > significantly improved by internal worker parallelization with a > thread pool. The worker periodically updates custom backgroundjob > databse record, so that following ajax request from the client can > read the status of the worker process. The job is identified with the > worker key that is stored in the session. Its not encouraged, thats all. You can still have threads in your workers. However, I am planning to add thread pool feature in bdrb itself, that should simplify things a bit. Also ideally, when using EventDriven network programming, you want all your sockets within select loop for efficiency. So, you wouldn't need any damn threads, if you can use a HTTP handler that works in Evented manner. What i mean to say is, you don't do this: a = Net::HTTP.get("http://www.google.com") but you do, Backgroundrb::HTTP.open("http://www.google.com") do |data| process_data(data) end What I am trying to illustrate is, when you ask to open, google.com page, evented model allows you to attach callback ( the block in this case ), which will be called when data arrives from google.com, rather than waiting for it in a thread. So, BackgrounDRb::HTTP.open() returns immediately. And you are concurrent as hell. But this is not possible, because if you are charging cards, then you are probably using ActiveMerchant which is using Net::HTTP and which blocks when you make request. But trust me, writing a simple http client is not that difficult, there is already connect() available in all workers. > How this works with fastcgi or multiple mongrel based engines where it > is not guaranteed to hit the same process with the next request? We > are using custom database tables and code for sharing the status > information now but I was wandering whether the plumbing includes > something to address this. Thats no problem at all, BackgrounDRb is a TCP server, so if you have followed the README file, no matter from which machine, you are making the request if you are specifying worker X, then its guaranteed to hit the same worker(with optional job_key if you are starting your worker dynamically) From gethemant at gmail.com Tue Dec 11 06:24:46 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 11 Dec 2007 16:54:46 +0530 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now In-Reply-To: <70A68453-B1F7-4FDF-BE52-4EA0146E4ADB@opensoul.org> References: <70A68453-B1F7-4FDF-BE52-4EA0146E4ADB@opensoul.org> Message-ID: <1197372286.9002.16.camel@shire> > At one point with the old version it was fairly straight-forward to > test workers, but that broke at one point. Could you give any > pointers writing tests for workers in the new version? > Hi Brandon, update your bdrb copy from svn and run rake backgroundrb:setup and you should have a RAILS_ROOT/test/bdrb_test_helper.rb file. Now, all your worker test cases can go in RAILS_ROOT/test/unit directory, just make sure that you require bdrb_test_helper file, and you can write test cases. For example: require File.join(File.dirname(__FILE__) + "/../bdrb_test_helper") require "god_worker" context "When god worker starts" do setup do god_worker = GodWorker.new end end I hope this helps. From gethemant at gmail.com Tue Dec 11 07:40:56 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 11 Dec 2007 18:10:56 +0530 Subject: [Backgroundrb-devel] [Rails] Re: [ANN] BackgrounDRb 1.0 pre-release available now In-Reply-To: <635bd2180712110408q567fff7fyb7c88004d37215f4@mail.gmail.com> References: <635bd2180712110408q567fff7fyb7c88004d37215f4@mail.gmail.com> Message-ID: <1197376856.9002.19.camel@shire> On Tue, 2007-12-11 at 23:08 +1100, George Bray wrote: > Hemant, this looks great. Could one use BackgroundRb to have workers > interact programatically with a remote telnet service? Or would I > simply start a worker that does this interaction via a > shell/spawn/telnet/expect... Sure as hell.. with any tcp service actually in a evented manner. However, that area is not polished ( no one ever asked. :) ) From emarceta at gmail.com Tue Dec 11 08:52:22 2007 From: emarceta at gmail.com (Emil Marceta) Date: Tue, 11 Dec 2007 14:52:22 +0100 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now In-Reply-To: <1197370248.9002.13.camel@shire> References: <74b000990712110153h7f6ef991p8034fb4d1ae1a281@mail.gmail.com> <1197370248.9002.13.camel@shire> Message-ID: <74b000990712110552m496a0fdft8cdfbed5f75e9096@mail.gmail.com> On Dec 11, 2007 11:50 AM, hemant kumar wrote: > Hi > > On Tue, 2007-12-11 at 10:53 +0100, Emil Marceta wrote: > > > Does this means that slave/daemons are not the dependency anymore? > > Yes, its gone. bdrb no longer depends on slave and daemons. > > > By 'not encouraged' do you mean that 1.0 is not supporting multiple > > threads in the worker or just as a general guidance? > > > > Could you please comment, how would you approach the following > > scenario with 1.0. Currently, we have a worker that creates threads > > that process financial payment transactions. An http request sends > > several 10s or 100s payment transaction records. They are handled by > > the single worker instance. Within the worker there is a pool of > > threads created that is calculated based on the number of > > transactions. For example for 200 transactions there will be 20 > > threads where each thread handles 10 requests in a squence. Each > > transaction takes about 3-5 seconds, so our throughput is > > significantly improved by internal worker parallelization with a > > thread pool. The worker periodically updates custom backgroundjob > > databse record, so that following ajax request from the client can > > read the status of the worker process. The job is identified with the > > worker key that is stored in the session. > > Its not encouraged, thats all. You can still have threads in your > workers. Thanks for clarifying. snip ... > Also ideally, when using EventDriven network programming, you want all > your sockets within select loop for efficiency. So, you wouldn't need > any damn threads, if you can use a HTTP handler that works in Evented > manner. What i mean to say is, you don't do this: > > a = Net::HTTP.get("http://www.google.com") > > but you do, > > Backgroundrb::HTTP.open("http://www.google.com") do |data| > process_data(data) > end > > What I am trying to illustrate is, when you ask to open, google.com > page, evented model allows you to attach callback ( the block in this > case ), which will be called when data arrives from google.com, rather > than waiting for it in a thread. So, BackgrounDRb::HTTP.open() returns > immediately. And you are concurrent as hell. > > But this is not possible, because if you are charging cards, then you > are probably using ActiveMerchant which is using Net::HTTP and which > blocks when you make request. But trust me, writing a simple http client > is not that difficult, there is already connect() available in all > workers. We are actually on of the ActiveMerchant providers (E-Xact), so strictly we are talking what is actually behind ActiveMerchant. There are many protocols involved in financial networks, depending where the transaction is routed. We are very familiar with Reactor engines and patterns you are advocating, and they work great, especially in uniform scenarios without throttling, sequencing etc. In our case, I don't see a clear gain I'm afraid. While a thread pool was done in no-time and is dead simple maintain, test etc. > > > > How this works with fastcgi or multiple mongrel based engines where it > > is not guaranteed to hit the same process with the next request? We > > are using custom database tables and code for sharing the status > > information now but I was wandering whether the plumbing includes > > something to address this. > > Thats no problem at all, BackgrounDRb is a TCP server, so if you have > followed the README file, no matter from which machine, you are making > the request if you are specifying worker X, then its guaranteed to hit > the same worker(with optional job_key if you are starting your worker > dynamically) Our Rails cluster runs bdrb on each Rails server and uses domain sockets. This to avoid a single point of failure and have uniform architecture. Would that work too? That is, does bdrb now works sort of like memcache where each server knows of every other instance? But even with that in place, in fastcgi for example, fastcgi processor may recycle the Rails process where callback has been registered. thanks, emil From gethemant at gmail.com Tue Dec 11 09:58:12 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 11 Dec 2007 20:28:12 +0530 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now In-Reply-To: <74b000990712110552m496a0fdft8cdfbed5f75e9096@mail.gmail.com> References: <74b000990712110153h7f6ef991p8034fb4d1ae1a281@mail.gmail.com> <1197370248.9002.13.camel@shire> <74b000990712110552m496a0fdft8cdfbed5f75e9096@mail.gmail.com> Message-ID: <1197385092.9002.34.camel@shire> Hi On Tue, 2007-12-11 at 14:52 +0100, Emil Marceta wrote: > We are actually on of the ActiveMerchant providers (E-Xact), so > strictly we are talking what is actually behind ActiveMerchant. There > are many protocols involved in financial networks, depending where the > transaction is routed. We are very familiar with Reactor engines and > patterns you are advocating, and they work great, especially in > uniform scenarios without throttling, sequencing etc. In our case, I > don't see a clear gain I'm afraid. While a thread pool was done in > no-time and is dead simple maintain, test etc. Cool. You can use existing approach provided you handle your threads with as much care. I will get back to this in sometime. There are other ways also, that I am looking. For example: co-routines ( on top of fibers ) from Ruby1.9. Just watch bdrb mailing list, or submit some patch. As i guess, you guys are already running somewhat customized version of bdrb. > Our Rails cluster runs bdrb on each Rails server and uses domain > sockets. This to avoid a single point of failure and have uniform > architecture. Would that work too? That is, does bdrb now works sort > of like memcache where each server knows of every other instance? But > even with that in place, in fastcgi for example, fastcgi processor may > recycle the Rails process where callback has been registered. Hmm, this is cool. So, how did you handle this situation earlier? Prolly, what you can do is, have bdrb instances running on each cluster and have cluster specific backgroundrb configuration file. So as, requests from mongrels running on cluster1 will be served by bdrb running on cluster1 only, and update some db/memcache key to indicate it, so as even if next time request goes to another worker on another machine, you know the state. Again, I would love any patch, ideas from you and I am myself working on something like this, which would avoid logging to db and stuff. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From hutch at recursive.ca Wed Dec 12 11:33:19 2007 From: hutch at recursive.ca (Bob Hutchison) Date: Wed, 12 Dec 2007 11:33:19 -0500 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now In-Reply-To: References: Message-ID: Hi, Looking forward to a chance to use this library. Thanks for the work! On 10-Dec-07, at 6:47 PM, hemant wrote: > - BackgrounDRb is no londer DRb, its based on event driven network > programming library packet ( http://www.packet.googlecode.com ) . How does this affect the licensing of BackgrounDRb (not to mention the name of the project :-)? The packet library is GPLv2 (the url doesn't have the leading www by the way), while BackgrounDRb is dual licensed with the Ruby License or an MIT license. Cheers, Bob ---- Bob Hutchison -- tumblelog at http://www.recursive.ca/so/ Recursive Design Inc. -- weblog at http://www.recursive.ca/hutch http://www.recursive.ca/ -- works on http://www.raconteur.info/cms-for-static-content/home/ From gethemant at gmail.com Wed Dec 12 12:10:56 2007 From: gethemant at gmail.com (hemant kumar) Date: Wed, 12 Dec 2007 22:40:56 +0530 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now In-Reply-To: <7148CA1B-4C09-4D6F-A05A-4FD7293448CD@recursive.ca> References: <7148CA1B-4C09-4D6F-A05A-4FD7293448CD@recursive.ca> Message-ID: <1197479456.6320.7.camel@shire> Hi On Wed, 2007-12-12 at 11:35 -0500, Bob Hutchison wrote: > How does this affect the licensing of BackgrounDRb (not to mention the > name of the project :-)? The packet library is GPLv2 (the url doesn't > have the leading www by the way), while BackgrounDRb is dual licensed > with the Ruby License or an MIT license. Damn I realized it after posting the message. But then thought "packet" may be irrelevant anyways ( to rails guys i mean ) Regarding license issues, since packet is dual licensed under GPL2 and Ruby, you can take shit from packet and embed in your app and forget that its under GPL2, since Ruby license allows you do that. There is a clause from Ruby license that says: "place your modifications in the Public Domain or otherwise make them Freely Available, such as by posting said modifications to Usenet or an equivalent medium, or by allowing the author to include your modifications in the software." So, I guess its ok to have that. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From Ivan.Manida at Sun.COM Wed Dec 12 12:07:05 2007 From: Ivan.Manida at Sun.COM (Ivan S. Manida) Date: Wed, 12 Dec 2007 20:07:05 +0300 Subject: [Backgroundrb-devel] BDRB, new version questions Message-ID: <47601539.908@Sun.COM> Hi, we are trying to use BDRB for background processing of long-running reports, and have two issues - one is that worker does not see our Report class from Rails, throws a: /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:22:in `load': undefined class/module Report (ArgumentError) from /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:22:in `receive_data' And the other is - does this new release work on Windows which some of our developers use? I know that last releases did not work because of fork() issues, but the new rewrite has a new networking model, so do you think it might work with some workarounds? thanks for your help! From gethemant at gmail.com Wed Dec 12 13:11:30 2007 From: gethemant at gmail.com (hemant kumar) Date: Wed, 12 Dec 2007 23:41:30 +0530 Subject: [Backgroundrb-devel] BDRB, new version questions In-Reply-To: <47601539.908@Sun.COM> References: <47601539.908@Sun.COM> Message-ID: <1197483090.6320.11.camel@shire> On Wed, 2007-12-12 at 20:07 +0300, Ivan S. Manida wrote: > Hi, > > we are trying to use BDRB for background processing of long-running > reports, and have two issues - one is that worker does not see our > Report class from Rails, throws a: > > /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:22:in > `load': undefined class/module Report (ArgumentError) > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:22:in > `receive_data' > > And the other is - does this new release work on Windows which some > of our developers use? I know that last releases did not work > because of fork() issues, but the new rewrite has a new networking > model, so do you think it might work with some workarounds? No, It won't work on Windows still, because of process creation model. Although, one can probably fix it with Win32 emulation API. Where is this Report class defined? is it a model or loaded from plugins? any code that you are using would be helpful. Technically however, you can put a explicit require of file that defines the class and see if it works. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From jason.lapier at gmail.com Wed Dec 12 16:29:40 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Wed, 12 Dec 2007 13:29:40 -0800 Subject: [Backgroundrb-devel] worker cleanup Message-ID: <93b44c1f0712121329i41833a7cm5edb4e21eaca8663@mail.gmail.com> I was using an older version of backgroundrb and today I started to play with the most recent release. First up, I'd like to say, thanks so much for this library and thanks for making it more stable. From the looks of things, I think it's going to be easier to use now than it was several months ago when I first started using it. I'd like to make sure I'm using everything the way it was intended, so let me give a brief overview of what I use bdrb for: Periodically, a user will need to perform a specific function that's initiated by a long-running task. This task is specific to each user, so what I did before was create a new worker each time the task was started (from within a Rails Controller) and give the worker a key based on the user's id (something like :key => "task_worker_#{user.id}"). Then, in other controller methods, I could check the status of the task by looking up that key. If the task was complete, I would delete the worker. The biggest problem with this workflow is that I'm relying on a controller action to delete the worker. If, for some reason, the worker never gets deleted, the next time the user tries to perform the same task, my controller is going to see that the worker with that key already exists, causing me some problems. The only "fix" I can think of is to give each worker a unique ID, and store that in the User model, so I can avoid looking up an old worker. With the newest release of bdrb, I'm looking at how the workers are slightly different, and I think I still want to create a new instance of my worker for each user, each time this task is invoked. If I'm going to continue to do it this way, is there a way I can have the worker delete itself when the work is done? Is this a bad idea? Or should I be thinking about a way to use a single worker to handle the multiple user tasks? Thanks again, - Jason L. -- My Rails and Linux Blog: http://offtheline.net From gethemant at gmail.com Wed Dec 12 23:06:17 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 13 Dec 2007 09:36:17 +0530 Subject: [Backgroundrb-devel] worker cleanup In-Reply-To: <93b44c1f0712121329i41833a7cm5edb4e21eaca8663@mail.gmail.com> References: <93b44c1f0712121329i41833a7cm5edb4e21eaca8663@mail.gmail.com> Message-ID: <1197518777.6320.27.camel@shire> Hi Jason, On Wed, 2007-12-12 at 13:29 -0800, Jason LaPier wrote: > I was using an older version of backgroundrb and today I started to > play with the most recent release. First up, I'd like to say, thanks > so much for this library and thanks for making it more stable. From > the looks of things, I think it's going to be easier to use now than > it was several months ago when I first started using it. Thanks > > I'd like to make sure I'm using everything the way it was intended, so > let me give a brief overview of what I use bdrb for: > Periodically, a user will need to perform a specific function that's > initiated by a long-running task. This task is specific to each user, > so what I did before was create a new worker each time the task was > started (from within a Rails Controller) and give the worker a key > based on the user's id (something like :key => > "task_worker_#{user.id}"). Then, in other controller methods, I could > check the status of the task by looking up that key. If the task was > complete, I would delete the worker. > > The biggest problem with this workflow is that I'm relying on a > controller action to delete the worker. If, for some reason, the > worker never gets deleted, the next time the user tries to perform the > same task, my controller is going to see that the worker with that key > already exists, causing me some problems. The only "fix" I can think > of is to give each worker a unique ID, and store that in the User > model, so I can avoid looking up an old worker. > > With the newest release of bdrb, I'm looking at how the workers are > slightly different, and I think I still want to create a new instance > of my worker for each user, each time this task is invoked. If I'm > going to continue to do it this way, is there a way I can have the > worker delete itself when the work is done? Is this a bad idea? Or > should I be thinking about a way to use a single worker to handle the > multiple user tasks? How long does a task for one particular user takes to run? If it doesn't take much long, then you can keep your worker around forever and just pass, arguments from controller to appropriate worker method. It can be done async if we code carefully. however, if you do not handle all the user events in separate worker (which can get nasty pretty quickly), and want to delete workers when task is done, just call exit on them. For example: class FooWorker < BackgrounDRb::MetaWorker set_worker_name :foo_worker def create(args = nil) end def some_task(user_args = nil) # do your work exit end end Since, each worker is a process, calling exit on them would remove the worker. > > Thanks again, > > - Jason L. > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Thu Dec 13 10:44:35 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 13 Dec 2007 21:14:35 +0530 Subject: [Backgroundrb-devel] Some fixes and updates Message-ID: <1197560675.6320.64.camel@shire> Hi , If you are following commits, I have pushed following changes: 1. Now, you can specify :environment option in config file to load production environment and stuff. 2. Exeptions that were exposed to the user has been changed to BdrbConnError if rails is not able to connect to bdrb server. 3. loading mechanism has been changed, so remove backgroundrb script from your backgroundrb directory and do a rake backgroundrb:setup and you should be good to go. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From Ivan.Manida at Sun.COM Thu Dec 13 10:55:09 2007 From: Ivan.Manida at Sun.COM (Ivan Manida) Date: Thu, 13 Dec 2007 18:55:09 +0300 Subject: [Backgroundrb-devel] Some fixes and updates In-Reply-To: <1197560675.6320.64.camel@shire> References: <1197560675.6320.64.camel@shire> Message-ID: <476155DD.7090703@sun.com> hemant kumar wrote: > 3. loading mechanism has been changed, so remove backgroundrb script > from your backgroundrb directory and do a rake backgroundrb:setup and > you should be good to go. > Probably the new loading mechanism is causing this error after update: /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:114:in `initialize': uninitialized constant MasterProxy::YAML (NameError) from script/backgroundrb:22:in `new' from script/backgroundrb:22 requiring Yaml causes it to fail on ERB, then Packet. From gethemant at gmail.com Thu Dec 13 11:11:41 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 13 Dec 2007 21:41:41 +0530 Subject: [Backgroundrb-devel] Some fixes and updates In-Reply-To: <476155DD.7090703@sun.com> References: <1197560675.6320.64.camel@shire> <476155DD.7090703@sun.com> Message-ID: <1197562301.6320.69.camel@shire> On Thu, 2007-12-13 at 18:55 +0300, Ivan Manida wrote: > hemant kumar wrote: > > 3. loading mechanism has been changed, so remove backgroundrb script > > from your backgroundrb directory and do a rake backgroundrb:setup and > > you should be good to go. > > > > Probably the new loading mechanism is causing this error after update: > > /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:114:in > `initialize': uninitialized constant MasterProxy::YAML (NameError) > from script/backgroundrb:22:in `new' > from script/backgroundrb:22 > > requiring Yaml causes it to fail on ERB, then Packet. Probably, You have slightly old code. Do a svn up now and remove backgroundrb script from script directory and do a "rake backgroundrb:setup" ( this is important and should be always done , when you update your plugin ) -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From Ivan.Manida at Sun.COM Thu Dec 13 11:37:46 2007 From: Ivan.Manida at Sun.COM (Ivan Manida) Date: Thu, 13 Dec 2007 19:37:46 +0300 Subject: [Backgroundrb-devel] Some fixes and updates In-Reply-To: <1197562301.6320.69.camel@shire> References: <1197560675.6320.64.camel@shire> <476155DD.7090703@sun.com> <1197562301.6320.69.camel@shire> Message-ID: <47615FDA.5060601@sun.com> hemant kumar wrote: > Probably, You have slightly old code. Do a svn up now and remove > backgroundrb script from script directory and do a "rake > backgroundrb:setup" ( this is important and should be always done , when > you update your plugin ) > > Hemant, Yes, I did that, and sources were recent - just got another file updated few minutes ago, which fixed that problem, but now mongrels won't start: ** Starting Mongrel listening at 0.0.0.0:3000 ** Starting Rails with production environment... /usr/local/lib/ruby/gems/1.8/gems/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:266:in `load_missing_constant': uninitialized constant BackgrounDRb (NameError) from /usr/local/lib/ruby/gems/1.8/gems/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:452:in `const_missing' from /usr/local/lib/ruby/gems/1.8/gems/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:464:in `const_missing' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/lib/bdrb_conn_error.rb:1 <.. standard mongrel log cut ..> Is there some other configuration step that's needed now? From gethemant at gmail.com Thu Dec 13 11:51:56 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 13 Dec 2007 22:21:56 +0530 Subject: [Backgroundrb-devel] Some fixes and updates In-Reply-To: <47615FDA.5060601@sun.com> References: <1197560675.6320.64.camel@shire> <476155DD.7090703@sun.com> <1197562301.6320.69.camel@shire> <47615FDA.5060601@sun.com> Message-ID: <1197564716.6320.75.camel@shire> On Thu, 2007-12-13 at 19:37 +0300, Ivan Manida wrote: > hemant kumar wrote: > > Probably, You have slightly old code. Do a svn up now and remove > > backgroundrb script from script directory and do a "rake > > backgroundrb:setup" ( this is important and should be always done , when > > you update your plugin ) > > > > > > Hemant, > > Yes, I did that, and sources were recent - just got another file updated > few minutes ago, which fixed that problem, but now mongrels won't start: > > ** Starting Mongrel listening at 0.0.0.0:3000 > ** Starting Rails with production environment... > /usr/local/lib/ruby/gems/1.8/gems/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:266:in > `load_missing_constant': uninitialized constant BackgrounDRb (NameError) > from > /usr/local/lib/ruby/gems/1.8/gems/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:452:in > `const_missing' > from > /usr/local/lib/ruby/gems/1.8/gems/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:464:in > `const_missing' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/lib/bdrb_conn_error.rb:1 > <.. standard mongrel log cut ..> > > Is there some other configuration step that's needed now? No no, there was a problem with code base. Fixed now. update your code please. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From veraticus at gmail.com Thu Dec 13 12:13:19 2007 From: veraticus at gmail.com (Josh Symonds) Date: Thu, 13 Dec 2007 11:13:19 -0600 Subject: [Backgroundrb-devel] Possible Bug & Question Message-ID: <76341dbf0712130913r2abec143k9f7017a8f8cd49d5@mail.gmail.com> Hi there, Amazing changes! I really like what you've done with BackgrounDRb and I hope you continue developing it, since it was definitely somewhat languishing before. One small error stuck out for me, though. The create method requires an argument even if one isn't used, such that if you have: def create @results = Hash.new end The server throws an error about trying to send an argument to create when it doesn't accept any. Obviously the fix on our end is to just do def create(arg) and ignore the argument if one isn't used, but you might want to make the server friendlier to create methods that don't accept arguments. And now that I've made a bug report, I have a question for you. I have a very long running task (sometimes taking up to two minutes) that I offloaded onto the previous version of BackgrounDRb. I made a new worker (called correlator) for every user and killed it after it returned results for them. This was a pretty ugly method of doing things and occassionally I'd lose Result hashes or the workers wouldn't get created at all, so with the new version I'm trying to use only one worker and report individual correlations through a results hash, like so: def CorrelatorWorker < BackgrounDRb::MetaWorker set_worker_name :correlator_worker def create(arg) @results = Hash.new end def correlate(user_id) # This stuff takes awhile... @results[user_id] = progress register_status(@results) end end So in my code I can say MiddleMan.ask_status(:worker=>:correlator_worker)[user_id] to find the progress of correlations for a single user. But the previous version of BackgrounDRb did have one big advantage in that each worker was a separate thread. In this version, when I start correlations for one user, the correlator_worker blocks until the method is finished, and then starts in on the next correlation, and so on. I want all the correlations to happen concurrently, so I was thinking of replacing correlate with this: def correlate(user_id) Thread.new do # This stuff takes awhile... @results[user_id] = progress register_status(@results) end end And splitting each correlation into its own thread. I have no idea how the new BackgrounDRb acts with this sort of thing, though. Would you recommend this as the developer, or is there some more obvious method that I'm missing to allow one worker to do a method without blocking further calls to that method? Josh Symonds -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071213/1d91d126/attachment-0001.html From gethemant at gmail.com Thu Dec 13 12:42:35 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 13 Dec 2007 23:12:35 +0530 Subject: [Backgroundrb-devel] Possible Bug & Question In-Reply-To: <76341dbf0712130913r2abec143k9f7017a8f8cd49d5@mail.gmail.com> References: <76341dbf0712130913r2abec143k9f7017a8f8cd49d5@mail.gmail.com> Message-ID: <1197567755.6320.91.camel@shire> On Thu, 2007-12-13 at 11:13 -0600, Josh Symonds wrote: > Hi there, > > Amazing changes! I really like what you've done with BackgrounDRb and > I hope you continue developing it, since it was definitely somewhat > languishing before. > > One small error stuck out for me, though. The create method requires > an argument even if one isn't used, such that if you have: > > def create > @results = Hash.new > end > > The server throws an error about trying to send an argument to create > when it doesn't accept any. Obviously the fix on our end is to just do > def create(arg) and ignore the argument if one isn't used, but you > might want to make the server friendlier to create methods that don't > accept arguments. Ok, Will be fixed. > > And now that I've made a bug report, I have a question for you. I have > a very long running task (sometimes taking up to two minutes) that I > offloaded onto the previous version of BackgrounDRb. I made a new > worker (called correlator) for every user and killed it after it > returned results for them. This was a pretty ugly method of doing > things and occassionally I'd lose Result hashes or the workers > wouldn't get created at all, so with the new version I'm trying to use > only one worker and report individual correlations through a results > hash, like so: > > def CorrelatorWorker < BackgrounDRb::MetaWorker > set_worker_name :correlator_worker > > def create(arg) > @results = Hash.new > end > > def correlate(user_id) > # This stuff takes awhile... > @results[user_id] = progress > register_status(@results) > end > > end > > So in my code I can say > MiddleMan.ask_status(:worker=>:correlator_worker)[user_id] to find the > progress of correlations for a single user. But the previous version > of BackgrounDRb did have one big advantage in that each worker was a > separate thread. In this version, when I start correlations for one > user, the correlator_worker blocks until the method is finished, and > then starts in on the next correlation, and so on. I want all the > correlations to happen concurrently, so I was thinking of replacing > correlate with this: > > def correlate(user_id) > Thread.new do > # This stuff takes awhile... > @results[user_id] = progress > register_status(@results) > end > end > > And splitting each correlation into its own thread. I have no idea how > the new BackgrounDRb acts with this sort of thing, though. Would you > recommend this as the developer, or is there some more obvious method > that I'm missing to allow one worker to do a method without blocking > further calls to that method? Since ruby has green threads, you got to be careful about what you are doing in those threads. If your task is CPU bound then ruby threads will not help, but if its reading from network or something, then you may use threads. However, i am in process of finalizing usage of threads in bdrb and would update the code soon. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From Ivan.Manida at Sun.COM Thu Dec 13 13:30:53 2007 From: Ivan.Manida at Sun.COM (Ivan Manida) Date: Thu, 13 Dec 2007 21:30:53 +0300 Subject: [Backgroundrb-devel] BDRB, new version questions In-Reply-To: <1197483090.6320.11.camel@shire> References: <47601539.908@Sun.COM> <1197483090.6320.11.camel@shire> Message-ID: <47617A5D.7040508@sun.com> > On Wed, 2007-12-12 at 20:07 +0300, Ivan S. Manida wrote: >> we are trying to use BDRB for background processing of long-running >> reports, and have two issues - one is that worker does not see our >> Report class from Rails, throws a: >> >> /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:22:in >> `load': undefined class/module Report (ArgumentError) >> from >> /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:22:in >> `receive_data' hemant kumar wrote: > Where is this Report class defined? is it a model or loaded from > plugins? any code that you are using would be helpful. > > Technically however, you can put a explicit require of file that defines > the class and see if it works. > It's a standard ruby model, just not one inherited from the ActiveRecord::Base. I tried "require"'ing it explicitly but then it complains about other models (ActiveRecord ones) used in the report generation. Seems that bdrb is not picking up models in "app/models" at all. From gethemant at gmail.com Thu Dec 13 13:47:21 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 14 Dec 2007 00:17:21 +0530 Subject: [Backgroundrb-devel] BDRB, new version questions In-Reply-To: <47617A5D.7040508@sun.com> References: <47601539.908@Sun.COM> <1197483090.6320.11.camel@shire> <47617A5D.7040508@sun.com> Message-ID: <1197571641.27742.1.camel@shire> On Thu, 2007-12-13 at 21:30 +0300, Ivan Manida wrote: > > On Wed, 2007-12-12 at 20:07 +0300, Ivan S. Manida wrote: > >> we are trying to use BDRB for background processing of long-running > >> reports, and have two issues - one is that worker does not see our > >> Report class from Rails, throws a: > >> > >> /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:22:in > >> `load': undefined class/module Report (ArgumentError) > >> from > >> /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:22:in > >> `receive_data' > > hemant kumar wrote: > > Where is this Report class defined? is it a model or loaded from > > plugins? any code that you are using would be helpful. > > > > Technically however, you can put a explicit require of file that defines > > the class and see if it works. > > > > It's a standard ruby model, just not one inherited from the > ActiveRecord::Base. I tried "require"'ing it explicitly but > then it complains about other models (ActiveRecord ones) used > in the report generation. Seems that bdrb is not picking up > models in "app/models" at all. This is weird, I am using models in couple of my apps and it works fine. Anyways, just send me the worker code ( if possible model code too ) and complete backtrace of the error. It shouldn't be a problem. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From todd at industrialogic.com Fri Dec 14 01:26:19 2007 From: todd at industrialogic.com (Todd Persen) Date: Fri, 14 Dec 2007 01:26:19 -0500 Subject: [Backgroundrb-devel] Error when loading backgroundrb Message-ID: <3A50A200-19E5-453D-AC11-90A15A7C5A95@industrialogic.com> I'm getting the following error (when doing script/backgroundrb start) on my local machine, which is running os x 10.4.11. However, it works just fine on my linux server. RAILS_ROOT/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:64:in `dump_object': undefined method `write_nonblock' for # (NoMethodError) I just grabbed the latest (r274) from the repository, and my Packet gems are identical. I also deleted the backgroundrb script and re- ran rake backgroundrb:setup. Any thoughts? Thanks, Todd From gethemant at gmail.com Fri Dec 14 02:25:25 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 14 Dec 2007 12:55:25 +0530 Subject: [Backgroundrb-devel] Error when loading backgroundrb In-Reply-To: <3A50A200-19E5-453D-AC11-90A15A7C5A95@industrialogic.com> References: <3A50A200-19E5-453D-AC11-90A15A7C5A95@industrialogic.com> Message-ID: <1197617125.27742.13.camel@shire> On Fri, 2007-12-14 at 01:26 -0500, Todd Persen wrote: > I'm getting the following error (when doing script/backgroundrb start) > on my local machine, which is running os x 10.4.11. However, it > works just fine on my linux server. > > RAILS_ROOT/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:64:in > `dump_object': undefined method `write_nonblock' for # 0x135b5f4> (NoMethodError) > > I just grabbed the latest (r274) from the repository, and my Packet > gems are identical. I also deleted the backgroundrb script and re- > ran rake backgroundrb:setup. Grab latest code. Also, you are running older version of Ruby perhaps. You need Ruby > 1.8.5 for using bdrb. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From todd at industrialogic.com Fri Dec 14 02:56:58 2007 From: todd at industrialogic.com (Todd Persen) Date: Fri, 14 Dec 2007 02:56:58 -0500 Subject: [Backgroundrb-devel] Error when loading backgroundrb In-Reply-To: <1197617125.27742.13.camel@shire> References: <3A50A200-19E5-453D-AC11-90A15A7C5A95@industrialogic.com> <1197617125.27742.13.camel@shire> Message-ID: Hemant, That was the problem. I was still running 1.8.4. Running great now, thanks for all the hard work. Todd On Dec 14, 2007, at 2:25 AM, hemant kumar wrote: > > On Fri, 2007-12-14 at 01:26 -0500, Todd Persen wrote: >> I'm getting the following error (when doing script/backgroundrb >> start) >> on my local machine, which is running os x 10.4.11. However, it >> works just fine on my linux server. >> >> RAILS_ROOT/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:64:in >> `dump_object': undefined method `write_nonblock' for #> 0x135b5f4> (NoMethodError) >> >> I just grabbed the latest (r274) from the repository, and my Packet >> gems are identical. I also deleted the backgroundrb script and re- >> ran rake backgroundrb:setup. > > Grab latest code. > > Also, you are running older version of Ruby perhaps. You need Ruby > > 1.8.5 for using bdrb. > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org > From veraticus at gmail.com Fri Dec 14 11:42:02 2007 From: veraticus at gmail.com (Josh Symonds) Date: Fri, 14 Dec 2007 10:42:02 -0600 Subject: [Backgroundrb-devel] Logging & Threading Message-ID: <76341dbf0712140842x55d17a35ya5d1ec4b9a547a98@mail.gmail.com> Hey again, After playing with BackgrounDRb a bit longer, I have two more issues I'd like to bring up with it. 1. Logging... it seems like even when the environment is set to production in both Mongrel and the backgroundrb.yml file, all ActiveRecord SQL statements are written to the development log. If you're doing a lot of ActiveRecord stuff in your workers, then the development log can get extremely large. I'm not sure why this is happening -- is this a bug in BackgrounDRb, or maybe something I misconfigured in Rails? It doesn't do this at any other times though. 2. Threads. Hemant, you said you were going to change how BackgrounDRb handles threads. But what do we do for now if we want a worker to do a process without blocking? I try using threads for this and get constant errors in the server until it finally crashes. Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071214/68834b89/attachment.html From gethemant at gmail.com Fri Dec 14 12:07:58 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 14 Dec 2007 22:37:58 +0530 Subject: [Backgroundrb-devel] Logging & Threading In-Reply-To: <76341dbf0712140842x55d17a35ya5d1ec4b9a547a98@mail.gmail.com> References: <76341dbf0712140842x55d17a35ya5d1ec4b9a547a98@mail.gmail.com> Message-ID: <1197652078.14033.21.camel@shire> Hi On Fri, 2007-12-14 at 10:42 -0600, Josh Symonds wrote: > Hey again, > > After playing with BackgrounDRb a bit longer, I have two more issues > I'd like to bring up with it. > > 1. Logging... it seems like even when the environment is set to > production in both Mongrel and the backgroundrb.yml file, all > ActiveRecord SQL statements are written to the development log. If > you're doing a lot of ActiveRecord stuff in your workers, then the > development log can get extremely large. I'm not sure why this is > happening -- is this a bug in BackgrounDRb, or maybe something I > misconfigured in Rails? It doesn't do this at any other times though. This is fixed now. I will be pushing a release that addresses this issue. > > 2. Threads. Hemant, you said you were going to change how BackgrounDRb > handles threads. But what do we do for now if we want a worker to do a > process without blocking? I try using threads for this and get > constant errors in the server until it finally crashes. I did some tests with threads it seems to work. In my apps, I don't need threads and hence my tests may be trivial, but can you be specific and let me know about the errors that you got? If possible, paste the code as well so as I can see how you are using threads? Basically I was going to add a thread pool within bdrb itself so as: Bdrb.defer { your_long_task } So as, whatever you do in block would run in a new thread, and once processing is done, the thread is returned to pool. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From joevandyk at gmail.com Sat Dec 15 14:14:28 2007 From: joevandyk at gmail.com (Joe Van Dyk) Date: Sat, 15 Dec 2007 11:14:28 -0800 Subject: [Backgroundrb-devel] Error when loading backgroundrb In-Reply-To: <1197617125.27742.13.camel@shire> References: <3A50A200-19E5-453D-AC11-90A15A7C5A95@industrialogic.com> <1197617125.27742.13.camel@shire> Message-ID: Perhaps add a check for ruby 1.8.5 to bdrb? On 12/13/07, hemant kumar wrote: > > On Fri, 2007-12-14 at 01:26 -0500, Todd Persen wrote: > > I'm getting the following error (when doing script/backgroundrb start) > > on my local machine, which is running os x 10.4.11. However, it > > works just fine on my linux server. > > > > RAILS_ROOT/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:64:in > > `dump_object': undefined method `write_nonblock' for # > 0x135b5f4> (NoMethodError) > > > > I just grabbed the latest (r274) from the repository, and my Packet > > gems are identical. I also deleted the backgroundrb script and re- > > ran rake backgroundrb:setup. > > Grab latest code. > > Also, you are running older version of Ruby perhaps. You need Ruby > > 1.8.5 for using bdrb. > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From gethemant at gmail.com Sat Dec 15 14:19:27 2007 From: gethemant at gmail.com (hemant) Date: Sun, 16 Dec 2007 00:49:27 +0530 Subject: [Backgroundrb-devel] Error when loading backgroundrb In-Reply-To: References: <3A50A200-19E5-453D-AC11-90A15A7C5A95@industrialogic.com> <1197617125.27742.13.camel@shire> Message-ID: On Dec 16, 2007 12:44 AM, Joe Van Dyk wrote: > Perhaps add a check for ruby 1.8.5 to bdrb? Yeah... coming up... soon. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From faivrem at gmail.com Sat Dec 15 17:08:55 2007 From: faivrem at gmail.com (Mickael Faivre-Macon) Date: Sat, 15 Dec 2007 23:08:55 +0100 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker Message-ID: Hello ! 1) I have this error logged by bdrb: undefined method `send_request' for nil:NilClass framework/packet_master.rb:58:in `ask_worker' backgroundrb/server/master_worker.rb:59:in `process_work' backgroundrb/server/master_worker.rb:16:in `receive_data' backgroundrb/framework/bin_parser.rb:29:in `call' backgroundrb/framework/bin_parser.rb:29:in `extract' backgroundrb/server/master_worker.rb:12:in `receive_data' backgroundrb/framework/core.rb:191:in `read_external_socket' backgroundrb/framework/core.rb:183:in `handle_external_messages' backgroundrb/framework/core.rb:156:in `start_reactor' backgroundrb/framework/core.rb:152:in `each' backgroundrb/framework/core.rb:152:in `start_reactor' backgroundrb/framework/core.rb:144:in `loop' backgroundrb/framework/core.rb:144:in `start_reactor' backgroundrb/framework/packet_master.rb:20:in `run' backgroundrb/server/master_worker.rb:115:in `initialize' ./script/backgroundrb:39:in `new' ./script/backgroundrb:39 my code is : class MyController < ApplicationController def index end def login @worker = MiddleMan.new_worker(:worker => :xmpp_worker, :job_key => :my_key) MiddleMan.ask_work(:worker => :xmpp_worker, :worker_method => :login, :data => "test") render(:partial=>'window') end end class XmppWorker < BackgrounDRb::MetaWorker set_worker_name :xmpp_worker set_no_auto_load true def create(args = nil) end def login end end Can someone helps me ? 2) I am creating a worker with a key : MiddleMan.new_worker(:worker => :xmpp_worker, :job_key => :my_key) Is it possible to use this key in a request to send work to a specific worker ? Thanks for your help, Mickael. From gethemant at gmail.com Sat Dec 15 17:43:32 2007 From: gethemant at gmail.com (hemant) Date: Sun, 16 Dec 2007 04:13:32 +0530 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: References: Message-ID: On Dec 16, 2007 3:38 AM, Mickael Faivre-Macon wrote: > Hello ! > > 1) I have this error logged by bdrb: > undefined method `send_request' for nil:NilClass > framework/packet_master.rb:58:in `ask_worker' > backgroundrb/server/master_worker.rb:59:in `process_work' > backgroundrb/server/master_worker.rb:16:in `receive_data' > backgroundrb/framework/bin_parser.rb:29:in `call' > backgroundrb/framework/bin_parser.rb:29:in `extract' > backgroundrb/server/master_worker.rb:12:in `receive_data' > backgroundrb/framework/core.rb:191:in `read_external_socket' > backgroundrb/framework/core.rb:183:in `handle_external_messages' > backgroundrb/framework/core.rb:156:in `start_reactor' > backgroundrb/framework/core.rb:152:in `each' > backgroundrb/framework/core.rb:152:in `start_reactor' > backgroundrb/framework/core.rb:144:in `loop' > backgroundrb/framework/core.rb:144:in `start_reactor' > backgroundrb/framework/packet_master.rb:20:in `run' > backgroundrb/server/master_worker.rb:115:in `initialize' > ./script/backgroundrb:39:in `new' > ./script/backgroundrb:39 > > my code is : > > class MyController < ApplicationController > > def index > end > > def login > @worker = MiddleMan.new_worker(:worker => :xmpp_worker, :job_key => :my_key) > MiddleMan.ask_work(:worker => :xmpp_worker, :worker_method => > :login, :data => "test") > render(:partial=>'window') > end > > end > > class XmppWorker < BackgrounDRb::MetaWorker > set_worker_name :xmpp_worker > set_no_auto_load true > > def create(args = nil) > end > > def login > end > > end > > Can someone helps me ? > > 2) I am creating a worker with a key : MiddleMan.new_worker(:worker => > :xmpp_worker, :job_key => :my_key) > Is it possible to use this key in a request to send work to a specific worker ? > I suggest you remove set_no_auto_load(true). Why exactly, you don't want worker to be automatically loaded when bdrb starts? The problem as i see may be because, since new_worker forks new worker in separate process, but the worker is not available immediately and hence you get the error. If you insist on using dynamically starting workers, I will be see what can be done to prevent error. There are couple of options. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From joshea at nooked.com Sat Dec 15 17:50:11 2007 From: joshea at nooked.com (John O'Shea) Date: Sat, 15 Dec 2007 22:50:11 +0000 Subject: [Backgroundrb-devel] Release tags in SVN? Message-ID: Hi, Im just wondering what strategy I should use for checking out from your repo. I've seen posts here announcing BackrounDRb 1.0 RC 2, 1.0 but I don't see any tags in the repo for these releases - I only see two old tags (release-0.2.0 & release-0.2.1). I'm assuming the HEAD code might be a bit unstable at times so should I use timestamps for checking out/exporting? If so can anyone advise of the correct timestamps or revision numbers for the most recent release? Or better still, could someone retrospectively apply tags to the correct revision number(s)? Thanks, John. From faivrem at gmail.com Sat Dec 15 18:12:12 2007 From: faivrem at gmail.com (Mickael Faivre-Macon) Date: Sun, 16 Dec 2007 00:12:12 +0100 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: References: Message-ID: Hi hemant, Thanks for your reply. The reason why I don't see the advantage to let the worker start wiht bdrb may be related to my understanding of how it works, and specifically to my question about keys and specific worker instances. I am using bdrb to keeps connections to a XMPP (Jabber) server, while Rails renders the UI (web based chat application). So the way I see it, is that there is only one :xmpp_worker class but several instances of that class running, each with a specific login and connection to the Jabber server. That's why I am launching a new :xmpp_worker (for each chat user). And that's why I need to know how to send_request (or ask_work) to a specific worker instance. I thought it could be done with the concept of keys. Is that the right way of doing things ? How would you do it ? Thanks, Mickael. On Dec 15, 2007 11:43 PM, hemant wrote: > > On Dec 16, 2007 3:38 AM, Mickael Faivre-Macon wrote: > > Hello ! > > > > 1) I have this error logged by bdrb: > > undefined method `send_request' for nil:NilClass > > framework/packet_master.rb:58:in `ask_worker' > > backgroundrb/server/master_worker.rb:59:in `process_work' > > backgroundrb/server/master_worker.rb:16:in `receive_data' > > backgroundrb/framework/bin_parser.rb:29:in `call' > > backgroundrb/framework/bin_parser.rb:29:in `extract' > > backgroundrb/server/master_worker.rb:12:in `receive_data' > > backgroundrb/framework/core.rb:191:in `read_external_socket' > > backgroundrb/framework/core.rb:183:in `handle_external_messages' > > backgroundrb/framework/core.rb:156:in `start_reactor' > > backgroundrb/framework/core.rb:152:in `each' > > backgroundrb/framework/core.rb:152:in `start_reactor' > > backgroundrb/framework/core.rb:144:in `loop' > > backgroundrb/framework/core.rb:144:in `start_reactor' > > backgroundrb/framework/packet_master.rb:20:in `run' > > backgroundrb/server/master_worker.rb:115:in `initialize' > > ./script/backgroundrb:39:in `new' > > ./script/backgroundrb:39 > > > > my code is : > > > > class MyController < ApplicationController > > > > def index > > end > > > > def login > > @worker = MiddleMan.new_worker(:worker => :xmpp_worker, :job_key => :my_key) > > MiddleMan.ask_work(:worker => :xmpp_worker, :worker_method => > > :login, :data => "test") > > render(:partial=>'window') > > end > > > > end > > > > class XmppWorker < BackgrounDRb::MetaWorker > > set_worker_name :xmpp_worker > > set_no_auto_load true > > > > def create(args = nil) > > end > > > > def login > > end > > > > end > > > > Can someone helps me ? > > > > 2) I am creating a worker with a key : MiddleMan.new_worker(:worker => > > :xmpp_worker, :job_key => :my_key) > > Is it possible to use this key in a request to send work to a specific worker ? > > > > I suggest you remove set_no_auto_load(true). Why exactly, you don't > want worker to be automatically loaded when bdrb starts? > > The problem as i see may be because, since new_worker forks new worker > in separate process, but the worker is not available immediately and > hence you get the error. > > If you insist on using dynamically starting workers, I will be see > what can be done to prevent error. There are couple of options. From joevandyk at gmail.com Sat Dec 15 19:55:52 2007 From: joevandyk at gmail.com (Joe Van Dyk) Date: Sat, 15 Dec 2007 16:55:52 -0800 Subject: [Backgroundrb-devel] Release tags in SVN? In-Reply-To: References: Message-ID: On the version control topic, has anyone considered using git to manage the source code for bdrb? Rubinius is using it and it seems to be working great for them. I think git has huge promise especially for open source projects. Could keep up a svn mirror for people who don't want things to change, I suppose. On Dec 15, 2007 2:50 PM, John O'Shea wrote: > Hi, > Im just wondering what strategy I should use for checking out from > your repo. I've seen posts here announcing BackrounDRb 1.0 RC 2, 1.0 > but I don't see any tags in the repo for these releases - I only see > two old tags (release-0.2.0 & release-0.2.1). I'm assuming the HEAD > code might be a bit unstable at times so should I use timestamps for > checking out/exporting? > If so can anyone advise of the correct timestamps or revision numbers > for the most recent release? Or better still, could someone > retrospectively apply tags to the correct revision number(s)? > > Thanks, > > John. > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From gethemant at gmail.com Sun Dec 16 00:30:50 2007 From: gethemant at gmail.com (hemant) Date: Sun, 16 Dec 2007 11:00:50 +0530 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: References: Message-ID: On Dec 16, 2007 4:42 AM, Mickael Faivre-Macon wrote: > Hi hemant, > > Thanks for your reply. > > The reason why I don't see the advantage to let the worker start wiht > bdrb may be related to my understanding of how it works, and > specifically to my question about keys and specific worker instances. > > I am using bdrb to keeps connections to a XMPP (Jabber) server, while > Rails renders the UI (web based chat application). So the way I see > it, is that there is only one :xmpp_worker class but several instances > of that class running, each with a specific login and connection to > the Jabber server. > > That's why I am launching a new :xmpp_worker (for each chat user). > And that's why I need to know how to send_request (or ask_work) to a > specific worker instance. I thought it could be done with the concept > of keys. yes you can use job keys for that. However I am curious about the fact that, if you have 100 concurrent users, you are going to have 100 parallel workers running for each user? Is that what you want? > > Is that the right way of doing things ? > How would you do it ? I am cooking a smallish worker with detailed explanation on how to handle multiple XMPP clients from one instance of backgroundrb. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Sun Dec 16 00:35:01 2007 From: gethemant at gmail.com (hemant) Date: Sun, 16 Dec 2007 11:05:01 +0530 Subject: [Backgroundrb-devel] Release tags in SVN? In-Reply-To: References: Message-ID: On Dec 16, 2007 6:25 AM, Joe Van Dyk wrote: > On the version control topic, has anyone considered using git to > manage the source code for bdrb? Rubinius is using it and it seems to > be working great for them. I think git has huge promise especially > for open source projects. Could keep up a svn mirror for people who > don't want things to change, I suppose. Yes..git is awesome, but I think, people will have trouble managing their rails plugins using git. There are couple of reasosns: 1. piston and ./script/plugin would freakout. 2. Generally git is not easy. > > > On Dec 15, 2007 2:50 PM, John O'Shea wrote: > > Hi, > > Im just wondering what strategy I should use for checking out from > > your repo. I've seen posts here announcing BackrounDRb 1.0 RC 2, 1.0 > > but I don't see any tags in the repo for these releases - I only see > > two old tags (release-0.2.0 & release-0.2.1). I'm assuming the HEAD > > code might be a bit unstable at times so should I use timestamps for > > checking out/exporting? > > If so can anyone advise of the correct timestamps or revision numbers > > for the most recent release? Or better still, could someone > > retrospectively apply tags to the correct revision number(s)? > > Yes.. I have been careless about this in past and would certainly start tagging releases. 1.0 is coming fast with some additional features, bug fixes, more examples, documentation and stuff. I would start tagging releases then onwards. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From faivrem at gmail.com Sun Dec 16 03:12:13 2007 From: faivrem at gmail.com (Mickael Faivre-Macon) Date: Sun, 16 Dec 2007 09:12:13 +0100 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: References: Message-ID: On Dec 16, 2007 6:30 AM, hemant wrote: > On Dec 16, 2007 4:42 AM, Mickael Faivre-Macon wrote: > > Hi hemant, > > > > Thanks for your reply. > > > > The reason why I don't see the advantage to let the worker start wiht > > bdrb may be related to my understanding of how it works, and > > specifically to my question about keys and specific worker instances. > > > > I am using bdrb to keeps connections to a XMPP (Jabber) server, while > > Rails renders the UI (web based chat application). So the way I see > > it, is that there is only one :xmpp_worker class but several instances > > of that class running, each with a specific login and connection to > > the Jabber server. > > > > That's why I am launching a new :xmpp_worker (for each chat user). > > And that's why I need to know how to send_request (or ask_work) to a > > specific worker instance. I thought it could be done with the concept > > of keys. > > yes you can use job keys for that. However I am curious about the fact > that, if you have 100 concurrent users, you are going to have 100 > parallel workers running for each user? Is that what you want? > Well, isn't it the same than to have only one worker with 100 threads ? (Well at least on the end, you would have 100 threads, not 100 ruby processes, that's right) I don't know what's better. > > > > Is that the right way of doing things ? > > How would you do it ? > > I am cooking a smallish worker with detailed explanation on how to > handle multiple XMPP clients from one instance of backgroundrb. You're spoiling the fun ;-) But it will greatly helps me on understanding how bdrb works and how to use it, so I will not complain :-) I am looking forward to see your solution. Mickael. From jonathan.wallace at gmail.com Sun Dec 16 14:46:30 2007 From: jonathan.wallace at gmail.com (Jonathan Wallace) Date: Sun, 16 Dec 2007 14:46:30 -0500 Subject: [Backgroundrb-devel] Running default rake tasks? Message-ID: I installed via piston and when in vendor/plugins/backgroundrb, I attempt to run the default rake task: (in /home/wallace/new_bdrb-wheels/vendor/plugins/backgroundrb) rake aborted! no such file to load -- spec/rake/spectask /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require' /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require' /home/wallace/new_bdrb-wheels/vendor/plugins/backgroundrb/Rakefile:5 /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1940:in `load' /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1940:in `raw_load_rakefile' /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1727:in `load_rakefile' /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1761:in `standard_exception_handling' /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1726:in `load_rakefile' /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1710:in `run' /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1761:in `standard_exception_handling' /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1708:in `run' /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/bin/rake:7 /usr/bin/rake:16:in `load' /usr/bin/rake:16 Also, I get an error attempting to start backgroundrb: wallace at cretney-deskop[3067]~/new_bdrb-wheels% ./script/backgroundrb start ./script/backgroundrb:27:in `require': no such file to load -- backgroundrb_server (LoadError) from ./script/backgroundrb:27 What am I missing? Thanks, Jonathan -- Jonathan Wallace From chaupt at mobirobo.com Sun Dec 16 22:52:46 2007 From: chaupt at mobirobo.com (Christopher Haupt) Date: Sun, 16 Dec 2007 19:52:46 -0800 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: References: Message-ID: <150262FF-1177-46F1-B169-211D6E23D133@mobirobo.com> Hi Hemant, all: Good work to-date on the new bdrb. On Dec 15, 2007, at 9:30 PM, hemant wrote: > On Dec 16, 2007 4:42 AM, Mickael Faivre-Macon > wrote: >> >> >> Is that the right way of doing things ? >> How would you do it ? > > I am cooking a smallish worker with detailed explanation on how to > handle multiple XMPP clients from one instance of backgroundrb. > I've been developing a chat middle-ware solution for a client (bunnybot.com). This is a the approach I've taken. We make heavy use of XMPP/Jabber, run OpenFire on the back-end, and need to be able to maintain user presence information. I've created a light-weight client worker that manages pools of user proxy objects inspired by xmpp4r- simple. Using this approach, a single worker can scale to maintain a pretty good size collection of connections (I've tested with 100's, don't see why it couldn't be more pretty much limited to OS resources). The worker and the app communicate in the standard way, with simple keys that identify the instance of the proxy object, and serialize key data (e.g. rosters, presence updates, messages, etc). I've got a few more things to do on the implementation, being paused while finishing other deployments, but am happy to discuss. I had the first version of this running under the older bdrb code-base using a self-managed set of threads. I'm moving it to the HEAD and looking at the new tcp connections in the event loop Hemant mentions. If others are doing similar things and can share knowledge, gotchas, or even code, that would be cool. A few of us have exchanged some messages over at RailsForum and RubyOnRails-Talk, so check those places for other leads. -chris -- Christopher Haupt chaupt at mobirobo.com From gethemant at gmail.com Mon Dec 17 00:52:40 2007 From: gethemant at gmail.com (hemant) Date: Mon, 17 Dec 2007 11:22:40 +0530 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: <150262FF-1177-46F1-B169-211D6E23D133@mobirobo.com> References: <150262FF-1177-46F1-B169-211D6E23D133@mobirobo.com> Message-ID: Hi Chris, > I've been developing a chat middle-ware solution for a client > (bunnybot.com). This is a the approach I've taken. We make heavy use > of XMPP/Jabber, run OpenFire on the back-end, and need to be able to > maintain user presence information. I've created a light-weight client > worker that manages pools of user proxy objects inspired by xmpp4r- > simple. Using this approach, a single worker can scale to maintain a > pretty good size collection of connections (I've tested with 100's, > don't see why it couldn't be more pretty much limited to OS > resources). The worker and the app communicate in the standard way, > with simple keys that identify the instance of the proxy object, and > serialize key data (e.g. rosters, presence updates, messages, etc). > I've got a few more things to do on the implementation, being paused > while finishing other deployments, but am happy to discuss. > > I had the first version of this running under the older bdrb code-base > using a self-managed set of threads. I'm moving it to the HEAD and > looking at the new tcp connections in the event loop Hemant mentions. > > If others are doing similar things and can share knowledge, gotchas, > or even code, that would be cool. A few of us have exchanged some > messages over at RailsForum and RubyOnRails-Talk, so check those > places for other leads. > My idea of managing everything from single worker is pretty simple. Keep all sever connections in hash, something like in create method of worker: def create @open_connections = {} @open_connection_mutex = Mutex.new end def login(details) thread_pool.defer(details) do |details| login_resp = login_using_details @open_connection_mutex.sychronize do @open_connections[user_id] = connection_info end end end def send_message(message) thread_pool.defer(message) do |message| user_info = message.user_id connection = nil @open_connection_mutex.sychronize do @connection = @open_connections[user_info] end connection.send_message(message.text) end Where threa_pool is a ThreadPool that is available right inside backgroundrb workers. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Mon Dec 17 01:09:53 2007 From: gethemant at gmail.com (hemant) Date: Mon, 17 Dec 2007 11:39:53 +0530 Subject: [Backgroundrb-devel] Some more updates, enhancements and fixes Message-ID: Hi Folks, Well, Yesterday I pushed some more updates that I was cooking since last couple of days. Here is a brief list: 1. Results from worker stays even after a worker dies. 2. You can ask backgroundrb to store status objects in memcache cluster now. by default its stored in master process memory, but I have introduced an option to allow it to be stored in a memcache cluster: #backgroundbr.yml :result_storage: :memcache: "10.10.10.2.11211,10.10.10.6:11211" 3. Check for Ruby version. 4. Introduced Thread pool now, all the workers have access to "thread_pool" object, which can be used to run concurrent tasks in ruby threads: An example, def scrap_wikipedia(text) thread_pool.defer(text) do |text| scrap_and_store_text(text) end end Please consult backgroundrb documentation for more information about this. By default the thread pool is of size 20. 5. Wrote examples on, how to use "connect" and "start_server" methods from backgroundrb workers. Again, consult documentation for it. Lets try/run it for a while and it will be released as 1.0 . -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Mon Dec 17 01:25:18 2007 From: gethemant at gmail.com (hemant) Date: Mon, 17 Dec 2007 11:55:18 +0530 Subject: [Backgroundrb-devel] Some more updates, enhancements and fixes In-Reply-To: References: Message-ID: On Dec 17, 2007 11:39 AM, hemant wrote: > Hi Folks, > > Well, Yesterday I pushed some more updates that I was cooking since > last couple of days. Here is a brief list: > > 1. Results from worker stays even after a worker dies. > 2. You can ask backgroundrb to store status objects in memcache > cluster now. by default its stored in master process memory, but I > have introduced an option to allow it to be stored in a memcache > cluster: > > #backgroundbr.yml > :result_storage: > :memcache: "10.10.10.2.11211,10.10.10.6:11211" > > > 3. Check for Ruby version. > 4. Introduced Thread pool now, all the workers have access to > "thread_pool" object, which can be used to run concurrent tasks in > ruby threads: > > An example, > > def scrap_wikipedia(text) > thread_pool.defer(text) do |text| > scrap_and_store_text(text) > end > end > > Please consult backgroundrb documentation for more information about > this. By default the thread pool is of size 20. > > 5. Wrote examples on, how to use "connect" and "start_server" methods > from backgroundrb workers. Again, consult documentation for it. > > > Lets try/run it for a while and it will be released as 1.0 . I have also cleared up tickets on: http://backgroundrb.devjavu.com/ So, if your ticket is against an old version of backgroudrb its probably cleared out. If you feel, please post it again, with relevant details against new version. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Mon Dec 17 04:15:32 2007 From: gethemant at gmail.com (hemant) Date: Mon, 17 Dec 2007 14:45:32 +0530 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: References: Message-ID: On Dec 16, 2007 3:38 AM, Mickael Faivre-Macon wrote: > Hello ! > > 1) I have this error logged by bdrb: > undefined method `send_request' for nil:NilClass > framework/packet_master.rb:58:in `ask_worker' > backgroundrb/server/master_worker.rb:59:in `process_work' > backgroundrb/server/master_worker.rb:16:in `receive_data' > backgroundrb/framework/bin_parser.rb:29:in `call' > backgroundrb/framework/bin_parser.rb:29:in `extract' > backgroundrb/server/master_worker.rb:12:in `receive_data' > backgroundrb/framework/core.rb:191:in `read_external_socket' > backgroundrb/framework/core.rb:183:in `handle_external_messages' > backgroundrb/framework/core.rb:156:in `start_reactor' > backgroundrb/framework/core.rb:152:in `each' > backgroundrb/framework/core.rb:152:in `start_reactor' > backgroundrb/framework/core.rb:144:in `loop' > backgroundrb/framework/core.rb:144:in `start_reactor' > backgroundrb/framework/packet_master.rb:20:in `run' > backgroundrb/server/master_worker.rb:115:in `initialize' > ./script/backgroundrb:39:in `new' > ./script/backgroundrb:39 > > my code is : > > class MyController < ApplicationController > > def index > end > > def login > @worker = MiddleMan.new_worker(:worker => :xmpp_worker, :job_key => :my_key) > MiddleMan.ask_work(:worker => :xmpp_worker, :worker_method => > :login, :data => "test") > render(:partial=>'window') > end > > end > > class XmppWorker < BackgrounDRb::MetaWorker > set_worker_name :xmpp_worker > set_no_auto_load true > > def create(args = nil) > end > > def login > end > > end > > Can someone helps me ? > > 2) I am creating a worker with a key : MiddleMan.new_worker(:worker => > :xmpp_worker, :job_key => :my_key) > Is it possible to use this key in a request to send work to a specific worker ? > Hi Michale, There was a genuine bug with backgroundrb in job_key handling. If you don't want to use threads and would rather want to have processes, you can try your code now. It should be fixed and working. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Mon Dec 17 07:23:23 2007 From: gethemant at gmail.com (hemant) Date: Mon, 17 Dec 2007 17:53:23 +0530 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb release 1.0 available now Message-ID: Hi Folks, I am glad to announce 1.0 release of BackgrounDRb. This would be a major release since 0.2 release of BackgrounDRb. Here is a brief summary of changes: - BackgrounDRb is DRb no longer. It makes use of EventDriven network programming library packet ( http://packet.googlecode.com ). - Since we moved to packet, many nasty thread issues, result hash corruption issues are totally gone. Lots of work has went in making scheduler rock solid stable. - Each worker, still runs in its own process, but each worker has a event loop of its own and all the events are triggered by the internal reactor loop. In a nutshell, you are not encouraged to use threads in your workers now. All the workers are already concurrent, but you are encouraged to use co-operative multitasking, rather than pre-emptive. A simple example is, For implement something like progress bar in old version of bdrb, you would: - start your processing a thread (so as your worker can receive further request from rails ) and have a instance variable ( protected by mutex ) which is updated on progress and can be send to rails. - With new backgroundrb, progress bar would be: process your damn request and just use register_status() to register status of your worker. Just because you are doing some processing won't mean that your worker will block. It can still receive requests from rails. - Now, you can schedule multiple methods with their own triggers. | :schedules: | :foo_worker: | :foobar: | :trigger_args: */5 * * * * * * | :data: Hello World | :barbar: | :trigger_args: */10 * * * * * * - Inside each worker, you can start tcp server or connect to a external server. Two important methods available in all workers are: start_server("localhost",port,ModuleName) connect("localhost",port,ModuleName) Connected client or outgoing connection would be integrated with Event Loop and you can process requests from these guys asynchronously. This mouse trap can allow you to build truly distributed workers across your network. - Each worker comes with a "thread_pool" object, which can be used to run tasks concurrently. For example: thread_pool.defer(url) { |url| scrap_wiki_content(url) } - Each worker has access to method "register_status" which can be used to update status of worker or store results. Results of a worker can be retrieved even after a worker has died. By default the results would be saved in master process memory, but you can configure BackgrounDRb to store these results in a memcache server or a cluster using following option in configuration file: # backgroundrb.yml | :backgroundrb: | :port: 11006 | :ip: 0.0.0.0 | :log: foreground | :result_storage: | :memcache: "10.10.10.2:11211,10.10.10.6:11211" - Relevant URLs: ** Home Page: http://backgroundrb.rubyforge.org ** SVN : http://svn.devjavu.com/backgroundrb/trunk ** Bug Reports/Ticks: http://backgroundrb.devjavu.com/report - Credits : ** Ezra Zygmuntowicz,skaar for taking BackgrounDRb so far. ** Kevin for helping out with OSX issues. ** Andy for patches and initial testing. ** Paul for patching up README. ** Other initial users. ** Matz, Francis for general inspiration. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From veraticus at gmail.com Mon Dec 17 09:17:38 2007 From: veraticus at gmail.com (Josh Symonds) Date: Mon, 17 Dec 2007 08:17:38 -0600 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb release 1.0 available now In-Reply-To: References: Message-ID: <76341dbf0712170617i1dfa7000j4aa9bb77e108f481@mail.gmail.com> Awesome, thanks for all the work, Hemant! Does this release also fix the issue with BackgrounDRb writing ActiveRecord messages to the development log? On Dec 17, 2007 6:23 AM, hemant wrote: > Hi Folks, > > I am glad to announce 1.0 release of BackgrounDRb. > > This would be a major release since 0.2 release of BackgrounDRb. > > Here is a brief summary of changes: > > - BackgrounDRb is DRb no longer. It makes use of EventDriven network > programming library packet ( http://packet.googlecode.com ). > > - Since we moved to packet, many nasty thread issues, result hash > corruption issues are totally gone. Lots of work has went in > making scheduler rock solid stable. > > - Each worker, still runs in its own process, but each worker has a > event loop of its own and all the events are triggered by the internal > reactor loop. In a nutshell, you are not encouraged to use threads > in your workers now. All the workers are already concurrent, but you > are encouraged to use co-operative multitasking, rather than > pre-emptive. A simple example is, > > For implement something like progress bar in old version of bdrb, you > would: > - start your processing a thread (so as your worker can receive > further request from rails ) and have a instance > variable ( protected by mutex ) which is updated on progress and > can be send to rails. > > - With new backgroundrb, progress bar would be: > process your damn request and just use register_status() to > register status of your worker. Just because > you are doing some processing won't mean that your worker will > block. It can still receive requests from rails. > > > - Now, you can schedule multiple methods with their own triggers. > > | :schedules: > | :foo_worker: > | :foobar: > | :trigger_args: */5 * * * * * * > | :data: Hello World > | :barbar: > | :trigger_args: */10 * * * * * * > > - Inside each worker, you can start tcp server or connect to a > external server. Two important methods available in all workers are: > > start_server("localhost",port,ModuleName) > connect("localhost",port,ModuleName) > > Connected client or outgoing connection would be integrated with > Event Loop and you can process requests from these guys > asynchronously. This mouse trap can allow you to build truly > distributed workers across your network. > > - Each worker comes with a "thread_pool" object, which can be used > to run tasks concurrently. For example: > > thread_pool.defer(url) { |url| scrap_wiki_content(url) } > > - Each worker has access to method "register_status" which can be > used to update status of worker or store results. Results of a worker > can be retrieved even after a worker has died. > > By default the results would be saved in master process memory, but > you can configure BackgrounDRb to store these results in a memcache > server or a cluster using following option in configuration file: > > # backgroundrb.yml > > | :backgroundrb: > | :port: 11006 > | :ip: 0.0.0.0 > | :log: foreground > | :result_storage: > | :memcache: "10.10.10.2:11211,10.10.10.6:11211" > > > - Relevant URLs: > ** Home Page: http://backgroundrb.rubyforge.org > ** SVN : http://svn.devjavu.com/backgroundrb/trunk > ** Bug Reports/Ticks: http://backgroundrb.devjavu.com/report > > - Credits : > ** Ezra Zygmuntowicz,skaar for taking BackgrounDRb so far. > ** Kevin for helping out with OSX issues. > ** Andy for patches and initial testing. > ** Paul for patching up README. > ** Other initial users. > ** Matz, Francis for general inspiration. > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071217/47dc223e/attachment-0001.html From du at bestwaytech.com Mon Dec 17 09:40:07 2007 From: du at bestwaytech.com (Danila Ulyanov) Date: Mon, 17 Dec 2007 08:40:07 -0600 Subject: [Backgroundrb-devel] Multiple copies of same worker In-Reply-To: <76341dbf0712170617i1dfa7000j4aa9bb77e108f481@mail.gmail.com> References: <76341dbf0712170617i1dfa7000j4aa9bb77e108f481@mail.gmail.com> Message-ID: <47668A47.4070208@bestwaytech.com> Hello! I remember there was a thread sometime last month about multiple copies of the same worker. Is that still impossible to do? What I am trying to accomplish is to limit number of concurrent connections for a lengthy search. I used to put search requests in a queue, then as slots freed up, they spawned off a new copy of the "search worker" which killed itself upon completion. That way no more than 8 searches were conducted at the same time. Will it be possible to do this in the new version or is there an alternative solution to this problem? Thanks! Danila From faivrem at gmail.com Mon Dec 17 09:50:19 2007 From: faivrem at gmail.com (Mickael Faivre-Macon) Date: Mon, 17 Dec 2007 15:50:19 +0100 Subject: [Backgroundrb-devel] Multiple copies of same worker In-Reply-To: <47668A47.4070208@bestwaytech.com> References: <76341dbf0712170617i1dfa7000j4aa9bb77e108f481@mail.gmail.com> <47668A47.4070208@bestwaytech.com> Message-ID: Hermant replied to me yesterday that it is still possible. Mickael. On Dec 17, 2007 3:40 PM, Danila Ulyanov wrote: > Hello! > > I remember there was a thread sometime last month about multiple copies > of the same worker. Is that still impossible to do? > > What I am trying to accomplish is to limit number of concurrent > connections for a lengthy search. I used to put search requests in a > queue, then as slots freed up, they spawned off a new copy of the > "search worker" which killed itself upon completion. That way no more > than 8 searches were conducted at the same time. > > Will it be possible to do this in the new version or is there an > alternative solution to this problem? > > Thanks! > > Danila From gethemant at gmail.com Mon Dec 17 09:59:22 2007 From: gethemant at gmail.com (hemant) Date: Mon, 17 Dec 2007 20:29:22 +0530 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb release 1.0 available now In-Reply-To: <76341dbf0712170617i1dfa7000j4aa9bb77e108f481@mail.gmail.com> References: <76341dbf0712170617i1dfa7000j4aa9bb77e108f481@mail.gmail.com> Message-ID: Hi On Dec 17, 2007 7:47 PM, Josh Symonds wrote: > Awesome, thanks for all the work, Hemant! Does this release also fix the > issue with BackgrounDRb writing ActiveRecord messages to the development > log? Yes, John, The issue you mention has been fixed in the latest release. > > > > On Dec 17, 2007 6:23 AM, hemant < gethemant at gmail.com> wrote: > > > > > > > > Hi Folks, > > > > I am glad to announce 1.0 release of BackgrounDRb. > > > > This would be a major release since 0.2 release of BackgrounDRb. > > > > Here is a brief summary of changes: > > > > - BackgrounDRb is DRb no longer. It makes use of EventDriven network > > programming library packet ( http://packet.googlecode.com ). > > > > - Since we moved to packet, many nasty thread issues, result hash > > corruption issues are totally gone. Lots of work has went in > > making scheduler rock solid stable. > > > > - Each worker, still runs in its own process, but each worker has a > > event loop of its own and all the events are triggered by the internal > > reactor loop. In a nutshell, you are not encouraged to use threads > > in your workers now. All the workers are already concurrent, but you > > are encouraged to use co-operative multitasking, rather than > > pre-emptive. A simple example is, > > > > For implement something like progress bar in old version of bdrb, you > would: > > - start your processing a thread (so as your worker can receive > > further request from rails ) and have a instance > > variable ( protected by mutex ) which is updated on progress and > > can be send to rails. > > > > - With new backgroundrb, progress bar would be: > > process your damn request and just use register_status() to > > register status of your worker. Just because > > you are doing some processing won't mean that your worker will > > block. It can still receive requests from rails. > > > > > > - Now, you can schedule multiple methods with their own triggers. > > > > | :schedules: > > | :foo_worker: > > | :foobar: > > | :trigger_args: */5 * * * * * * > > | :data: Hello World > > | :barbar: > > | :trigger_args: */10 * * * * * * > > > > - Inside each worker, you can start tcp server or connect to a > > external server. Two important methods available in all workers are: > > > > start_server("localhost",port,ModuleName) > > connect("localhost",port,ModuleName) > > > > Connected client or outgoing connection would be integrated with > > Event Loop and you can process requests from these guys > > asynchronously. This mouse trap can allow you to build truly > > distributed workers across your network. > > > > - Each worker comes with a "thread_pool" object, which can be used > > to run tasks concurrently. For example: > > > > thread_pool.defer(url) { |url| scrap_wiki_content(url) } > > > > - Each worker has access to method "register_status" which can be > > used to update status of worker or store results. Results of a worker > > can be retrieved even after a worker has died. > > > > By default the results would be saved in master process memory, but > > you can configure BackgrounDRb to store these results in a memcache > > server or a cluster using following option in configuration file: > > > > # backgroundrb.yml > > > > | :backgroundrb: > > | :port: 11006 > > | :ip: 0.0.0.0 > > | :log: foreground > > | :result_storage: > > | :memcache: "10.10.10.2:11211,10.10.10.6:11211" > > > > > > - Relevant URLs: > > ** Home Page: http://backgroundrb.rubyforge.org > > ** SVN : http://svn.devjavu.com/backgroundrb/trunk > > ** Bug Reports/Ticks: http://backgroundrb.devjavu.com/report > > > > - Credits : > > ** Ezra Zygmuntowicz,skaar for taking BackgrounDRb so far. > > ** Kevin for helping out with OSX issues. > > ** Andy for patches and initial testing. > > ** Paul for patching up README. > > ** Other initial users. > > ** Matz, Francis for general inspiration. > > > > > > -- > > Let them talk of their oriental summer climes of everlasting > > conservatories; give me the privilege of making my own summer with my > > own coals. > > > > http://gnufied.org > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Mon Dec 17 10:03:15 2007 From: gethemant at gmail.com (hemant) Date: Mon, 17 Dec 2007 20:33:15 +0530 Subject: [Backgroundrb-devel] Multiple copies of same worker In-Reply-To: References: <76341dbf0712170617i1dfa7000j4aa9bb77e108f481@mail.gmail.com> <47668A47.4070208@bestwaytech.com> Message-ID: Hi > On Dec 17, 2007 3:40 PM, Danila Ulyanov wrote: > > Hello! > > > > I remember there was a thread sometime last month about multiple copies > > of the same worker. Is that still impossible to do? > > > > What I am trying to accomplish is to limit number of concurrent > > connections for a lengthy search. I used to put search requests in a > > queue, then as slots freed up, they spawned off a new copy of the > > "search worker" which killed itself upon completion. That way no more > > than 8 searches were conducted at the same time. > > > > Will it be possible to do this in the new version or is there an > > alternative solution to this problem? > > Yes, you can have multiple copies of same worker running. But there are two catches: 1. You have to specify a unique job_key for each worker, when you are making a call to a start worker. 2. To start a worker, you will have to use MiddleMan.new_worker(:worker => :foo_worker, :job_key => "whoa_man") To exit a worker, once its done with search, simply call "exit" on them. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From du at bestwaytech.com Mon Dec 17 10:10:05 2007 From: du at bestwaytech.com (Danila Ulyanov) Date: Mon, 17 Dec 2007 09:10:05 -0600 Subject: [Backgroundrb-devel] Multiple copies of same worker In-Reply-To: References: <76341dbf0712170617i1dfa7000j4aa9bb77e108f481@mail.gmail.com> <47668A47.4070208@bestwaytech.com> Message-ID: <4766914D.4030404@bestwaytech.com> Is there any special reason that creating worker does not return a unique job_key itself upon creation like it used to in the old backgroundrb? :-) Would there be much overhead creating/destroying up to 8 workers versus having 8 workers running continuously and just reusing them? Thanks! hemant wrote: > Hi > >> On Dec 17, 2007 3:40 PM, Danila Ulyanov wrote: >>> Hello! >>> >>> I remember there was a thread sometime last month about multiple copies >>> of the same worker. Is that still impossible to do? >>> >>> What I am trying to accomplish is to limit number of concurrent >>> connections for a lengthy search. I used to put search requests in a >>> queue, then as slots freed up, they spawned off a new copy of the >>> "search worker" which killed itself upon completion. That way no more >>> than 8 searches were conducted at the same time. >>> >>> Will it be possible to do this in the new version or is there an >>> alternative solution to this problem? >>> > > Yes, you can have multiple copies of same worker running. But there > are two catches: > > 1. You have to specify a unique job_key for each worker, when you are > making a call to a start worker. > 2. To start a worker, you will have to use > MiddleMan.new_worker(:worker => :foo_worker, :job_key => "whoa_man") > > To exit a worker, once its done with search, simply call "exit" on them. > > From rob at iotum.com Mon Dec 17 11:47:08 2007 From: rob at iotum.com (Rob Nielsen) Date: Mon, 17 Dec 2007 11:47:08 -0500 Subject: [Backgroundrb-devel] Get "some read error" on calls to worker Message-ID: <48C76E56FDD0DF44B6585A82E5DD27AEEE8410@iotum-sbs.IotumCorp.local> I'm running the latest from svn (rev 285). I've been having a problem launching a process repeatedly. What happens is that I get a debug message "some read error" and then the worker refuses to run again. I've stripped my worker down to just doing a puts and it still happens. Like the other threads I want to launch my workers as needed, however I get the same behavior whether use the auto_load and ask_work or use new_worker with no_auto_load. If I remove the job_key I can get it to work, but I still get the "Some read error" message. I need the job to keep one worker per id. I'd rather have separate processes than go the thread route - for now. class UpdateFacebookAccountInfoWorker < BackgrounDRb::MetaWorker set_worker_name :update_facebook_account_info_worker set_no_auto_load true require "facebook_rails_controller_extensions" include RFacebook require 'activemessaging/processor' include ActiveMessaging::MessageSender def create(args) puts "create=>"+ args.inspect exit! end end and the call... MiddleMan.new_worker( :worker => :update_facebook_account_info_worker, :job_key => @some.id, # Only run one worker per id :data => { :session_key => @session_key, :requesting_fb_id => @organizer.facebook_id, :facebook_ids => @fb_ids, :params => params } ) The output... create=>{:params=>REDACTED]} Some read error Some read error Some read error Thanks Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071217/a2b14747/attachment-0001.html From gethemant at gmail.com Mon Dec 17 12:14:04 2007 From: gethemant at gmail.com (hemant kumar) Date: Mon, 17 Dec 2007 22:44:04 +0530 Subject: [Backgroundrb-devel] Get "some read error" on calls to worker In-Reply-To: <48C76E56FDD0DF44B6585A82E5DD27AEEE8410@iotum-sbs.IotumCorp.local> References: <48C76E56FDD0DF44B6585A82E5DD27AEEE8410@iotum-sbs.IotumCorp.local> Message-ID: <1197911644.6207.17.camel@shire> Hi Rob, On Mon, 2007-12-17 at 11:47 -0500, Rob Nielsen wrote: > I?m running the latest from svn (rev 285). I?ve been having a problem > launching a process repeatedly. What happens is that I get a debug > message ?some read error? and then the worker refuses to run again. > > > > I?ve stripped my worker down to just doing a puts and it still > happens. Like the other threads I want to launch my workers as needed, > however I get the same behavior whether use the auto_load and ask_work > or use new_worker with no_auto_load. If I remove the job_key I can > get it to work, but I still get the ?Some read error? message. I need > the job to keep one worker per id. I?d rather have separate processes > than go the thread route ? for now. > > > > class UpdateFacebookAccountInfoWorker < BackgrounDRb::MetaWorker > > set_worker_name :update_facebook_account_info_worker > > set_no_auto_load true > > > > require "facebook_rails_controller_extensions" > > include RFacebook > > > > require 'activemessaging/processor' > > include ActiveMessaging::MessageSender > > > > def create(args) > > puts "create=>"+ args.inspect > > exit! > > end > > end > > > > and the call? > > > > MiddleMan.new_worker( > > :worker => :update_facebook_account_info_worker, > > :job_key => @some.id, # Only run one worker per id > > :data => { > > :session_key => @session_key, > > :requesting_fb_id => @organizer.facebook_id, > > :facebook_ids => @fb_ids, > > :params => params > > } > > ) > > > > The output? > > > > create=>{:params=>REDACTED]} > > Some read error > > Some read error > > Some read error > > Ok so, create method is getting invoked when you invoke: MiddleMan.new_worker(...) right? Now, next i see an "exit!" statement there, which basically asks the worker to quit. Now when a worker quits, master process gets an error when reading from that socket, so thats why you are getting "some read error" message. But as far as I can see, create method is getting called alright. Correct me, If i am wrong. > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From adam at holonyx.com Mon Dec 17 11:55:36 2007 From: adam at holonyx.com (Adam Lebsack) Date: Mon, 17 Dec 2007 09:55:36 -0700 Subject: [Backgroundrb-devel] wday not honored in cron trigger Message-ID: <65B8EC13-DDE9-4E10-A356-446CAD04D6FA@holonyx.com> Hey guys, I was playing with the cron trigger, and I noticed that wday is not honored. Looking at the code confirms this. Has anyone else seen this? Thanks Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071217/44d3a8a6/attachment.html From rob at iotum.com Mon Dec 17 12:34:13 2007 From: rob at iotum.com (Rob Nielsen) Date: Mon, 17 Dec 2007 12:34:13 -0500 Subject: [Backgroundrb-devel] Get "some read error" on calls to worker In-Reply-To: <1197911644.6207.17.camel@shire> Message-ID: <48C76E56FDD0DF44B6585A82E5DD27AEEE8412@iotum-sbs.IotumCorp.local> -----Original Message----- From: hemant kumar [mailto:gethemant at gmail.com] Sent: Monday, December 17, 2007 12:14 PM To: Rob Nielsen Cc: backgroundrb-devel at rubyforge.org Subject: Re: [Backgroundrb-devel] Get "some read error" on calls to worker Hi Rob, [SNIP] Ok so, create method is getting invoked when you invoke: MiddleMan.new_worker(...) right? Now, next i see an "exit!" statement there, which basically asks the worker to quit. Now when a worker quits, master process gets an error when reading from that socket, so thats why you are getting "some read error" message. But as far as I can see, create method is getting called alright. Correct me, If i am wrong. ---- Yes it gets called once. Subsequent calls fail for some reason. If I take out the job_key, then I can call it again. From gethemant at gmail.com Mon Dec 17 12:36:10 2007 From: gethemant at gmail.com (hemant kumar) Date: Mon, 17 Dec 2007 23:06:10 +0530 Subject: [Backgroundrb-devel] Get "some read error" on calls to worker In-Reply-To: <48C76E56FDD0DF44B6585A82E5DD27AEEE8412@iotum-sbs.IotumCorp.local> References: <48C76E56FDD0DF44B6585A82E5DD27AEEE8412@iotum-sbs.IotumCorp.local> Message-ID: <1197912970.6207.20.camel@shire> On Mon, 2007-12-17 at 12:34 -0500, Rob Nielsen wrote: > > -----Original Message----- > From: hemant kumar [mailto:gethemant at gmail.com] > Sent: Monday, December 17, 2007 12:14 PM > To: Rob Nielsen > Cc: backgroundrb-devel at rubyforge.org > Subject: Re: [Backgroundrb-devel] Get "some read error" on calls to > worker > > Hi Rob, > > > [SNIP] > > Ok so, create method is getting invoked when you invoke: > > MiddleMan.new_worker(...) > > right? > > Now, next i see an "exit!" statement there, which basically asks the > worker to quit. Now when a worker quits, master process gets an error > when reading from that socket, so thats why you are getting "some read > error" message. But as far as I can see, create method is getting called > alright. Correct me, If i am wrong. > > ---- > > Yes it gets called once. Subsequent calls fail for some reason. If I > take out the job_key, then I can call it again. When you can exit! on a worker..its dead, its gone. You can't access it any more. You shouldn't call exit! on it, if you want to make further calls to it. > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From rob at iotum.com Mon Dec 17 12:39:04 2007 From: rob at iotum.com (Rob Nielsen) Date: Mon, 17 Dec 2007 12:39:04 -0500 Subject: [Backgroundrb-devel] Get "some read error" on calls to worker In-Reply-To: <1197912970.6207.20.camel@shire> Message-ID: <48C76E56FDD0DF44B6585A82E5DD27AEEE8413@iotum-sbs.IotumCorp.local> -----Original Message----- From: hemant kumar [mailto:gethemant at gmail.com] Sent: Monday, December 17, 2007 12:36 PM To: Rob Nielsen Cc: backgroundrb-devel at rubyforge.org Subject: RE: [Backgroundrb-devel] Get "some read error" on calls to worker On Mon, 2007-12-17 at 12:34 -0500, Rob Nielsen wrote: > > -----Original Message----- > From: hemant kumar [mailto:gethemant at gmail.com] > Sent: Monday, December 17, 2007 12:14 PM > To: Rob Nielsen > Cc: backgroundrb-devel at rubyforge.org > Subject: Re: [Backgroundrb-devel] Get "some read error" on calls to > worker > > Hi Rob, > > > [SNIP] > > Ok so, create method is getting invoked when you invoke: > > MiddleMan.new_worker(...) > > right? > > Now, next i see an "exit!" statement there, which basically asks the > worker to quit. Now when a worker quits, master process gets an error > when reading from that socket, so thats why you are getting "some read > error" message. But as far as I can see, create method is getting called > alright. Correct me, If i am wrong. > > ---- > > Yes it gets called once. Subsequent calls fail for some reason. If I > take out the job_key, then I can call it again. When you can exit! on a worker..its dead, its gone. You can't access it any more. You shouldn't call exit! on it, if you want to make further calls to it. ---- Right. But I'm calling new_worker. So that should start a new instance. Right? > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Mon Dec 17 13:08:33 2007 From: gethemant at gmail.com (hemant kumar) Date: Mon, 17 Dec 2007 23:38:33 +0530 Subject: [Backgroundrb-devel] Get "some read error" on calls to worker In-Reply-To: <48C76E56FDD0DF44B6585A82E5DD27AEEE8413@iotum-sbs.IotumCorp.local> References: <48C76E56FDD0DF44B6585A82E5DD27AEEE8413@iotum-sbs.IotumCorp.local> Message-ID: <1197914913.6207.21.camel@shire> On Mon, 2007-12-17 at 12:39 -0500, Rob Nielsen wrote: > > -----Original Message----- > From: hemant kumar [mailto:gethemant at gmail.com] > Sent: Monday, December 17, 2007 12:36 PM > To: Rob Nielsen > Cc: backgroundrb-devel at rubyforge.org > Subject: RE: [Backgroundrb-devel] Get "some read error" on calls to > worker > > > On Mon, 2007-12-17 at 12:34 -0500, Rob Nielsen wrote: > > > > -----Original Message----- > > From: hemant kumar [mailto:gethemant at gmail.com] > > Sent: Monday, December 17, 2007 12:14 PM > > To: Rob Nielsen > > Cc: backgroundrb-devel at rubyforge.org > > Subject: Re: [Backgroundrb-devel] Get "some read error" on calls to > > worker > > > > Hi Rob, > > > > > > [SNIP] > > > > Ok so, create method is getting invoked when you invoke: > > > > MiddleMan.new_worker(...) > > > > right? > > > > Now, next i see an "exit!" statement there, which basically asks the > > worker to quit. Now when a worker quits, master process gets an error > > when reading from that socket, so thats why you are getting "some read > > error" message. But as far as I can see, create method is getting > called > > alright. Correct me, If i am wrong. > > > > ---- > > > > Yes it gets called once. Subsequent calls fail for some reason. If I > > take out the job_key, then I can call it again. > > When you can exit! on a worker..its dead, its gone. You can't access it > any more. You shouldn't call exit! on it, if you want to make further > calls to it. > > ---- > Right. But I'm calling new_worker. So that should start a new instance. > Right? > > > Ok, sync with latest svn code and see if its fixed. If not then, come to #backgroundrb on freenode. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From Ivan.Manida at Sun.COM Mon Dec 17 13:35:03 2007 From: Ivan.Manida at Sun.COM (Ivan S. Manida) Date: Mon, 17 Dec 2007 21:35:03 +0300 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb release 1.0 available now In-Reply-To: References: Message-ID: <4766C157.1090008@Sun.COM> Hemant, "ruby script/backgroundrb stop" does not stop the currently running server. What is a good way to restart the server, besides doing a kill -9 `cat backgroundrb.pid`? hemant wrote: > Hi Folks, > > I am glad to announce 1.0 release of BackgrounDRb. > > This would be a major release since 0.2 release of BackgrounDRb. From gethemant at gmail.com Mon Dec 17 13:52:52 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 18 Dec 2007 00:22:52 +0530 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb release 1.0 available now In-Reply-To: <4766C157.1090008@Sun.COM> References: <4766C157.1090008@Sun.COM> Message-ID: <1197917572.6207.30.camel@shire> On Mon, 2007-12-17 at 21:35 +0300, Ivan S. Manida wrote: > Hemant, > > "ruby script/backgroundrb stop" does not stop the currently running > server. What is a good way to restart the server, besides doing a > kill -9 `cat backgroundrb.pid`? > > You got to be kidding. :) It does stops backgroundrb server for me. Here are possible fixes: 1. Remove old "backgroundrb" script lying in your script directory. 2. run: rake backgroundrb:setup 3. Remove any :log: foreground option if you are using 3. start bdrb server: ./script/backgroundrb start 4. stop bdrb server: ./script/backgroundrb stop 5. File a bug with super critical priority if it doesn't work. Mention, which OS, which version of bdrb. :) > hemant wrote: > > Hi Folks, > > > > I am glad to announce 1.0 release of BackgrounDRb. > > > > This would be a major release since 0.2 release of BackgrounDRb. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Mon Dec 17 15:09:14 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 18 Dec 2007 01:39:14 +0530 Subject: [Backgroundrb-devel] Get "some read error" on calls to worker In-Reply-To: <48C76E56FDD0DF44B6585A82E5DD27AEEE8417@iotum-sbs.IotumCorp.local> References: <48C76E56FDD0DF44B6585A82E5DD27AEEE8417@iotum-sbs.IotumCorp.local> Message-ID: <1197922154.6207.34.camel@shire> On Mon, 2007-12-17 at 14:48 -0500, Rob Nielsen wrote: > > -----Original Message----- > From: hemant kumar [mailto:gethemant at gmail.com] > Sent: Monday, December 17, 2007 1:09 PM > To: Rob Nielsen > Cc: backgroundrb-devel at rubyforge.org > Subject: RE: [Backgroundrb-devel] Get "some read error" on calls to > worker > > > On Mon, 2007-12-17 at 12:39 -0500, Rob Nielsen wrote: > > > > -----Original Message----- > > From: hemant kumar [mailto:gethemant at gmail.com] > > Sent: Monday, December 17, 2007 12:36 PM > > To: Rob Nielsen > > Cc: backgroundrb-devel at rubyforge.org > > Subject: RE: [Backgroundrb-devel] Get "some read error" on calls to > > worker > > > > > > On Mon, 2007-12-17 at 12:34 -0500, Rob Nielsen wrote: > > > > > > -----Original Message----- > > > From: hemant kumar [mailto:gethemant at gmail.com] > > > Sent: Monday, December 17, 2007 12:14 PM > > > To: Rob Nielsen > > > Cc: backgroundrb-devel at rubyforge.org > > > Subject: Re: [Backgroundrb-devel] Get "some read error" on calls to > > > worker > > > > > > Hi Rob, > > > > > > > > > [SNIP] > > > > > > Ok so, create method is getting invoked when you invoke: > > > > > > MiddleMan.new_worker(...) > > > > > > right? > > > > > > Now, next i see an "exit!" statement there, which basically asks the > > > worker to quit. Now when a worker quits, master process gets an > error > > > when reading from that socket, so thats why you are getting "some > read > > > error" message. But as far as I can see, create method is getting > > called > > > alright. Correct me, If i am wrong. > > > > > > ---- > > > > > > Yes it gets called once. Subsequent calls fail for some reason. If I > > > take out the job_key, then I can call it again. > > > > When you can exit! on a worker..its dead, its gone. You can't access > it > > any more. You shouldn't call exit! on it, if you want to make further > > calls to it. > > > > ---- > > Right. But I'm calling new_worker. So that should start a new > instance. > > Right? > > > > > > > Ok, sync with latest svn code and see if its fixed. If not then, come to > #backgroundrb on freenode. > > ---- > OK that seems to have fixed the issue. Thanks. > > A couple questions: > 1. What happens if you do a new_worker with a given job_key while a > worker with the same key is already running? Does the second task get > queued or does the request simply wait for the first process to finish? Currently none of the above happens. Your request will be simply ignored. > > 2. Does ask_status throw an exception if the worker isn't running? I > would like to be able to ask if the worker is running before starting > another one. Nope, It won't throw any exception in user code. It does throw an exception internally. However, you can query status of all workers that are currently running, using MiddleMan.query_all_workers() See, if that output is helpful. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From rob at iotum.com Mon Dec 17 15:18:34 2007 From: rob at iotum.com (Rob Nielsen) Date: Mon, 17 Dec 2007 15:18:34 -0500 Subject: [Backgroundrb-devel] Get "some read error" on calls to worker In-Reply-To: <1197922154.6207.34.camel@shire> Message-ID: <48C76E56FDD0DF44B6585A82E5DD27AEEE8418@iotum-sbs.IotumCorp.local> -----Original Message----- From: hemant kumar [mailto:gethemant at gmail.com] Sent: Monday, December 17, 2007 3:09 PM To: Rob Nielsen Cc: backgroundrb-devel Subject: RE: [Backgroundrb-devel] Get "some read error" on calls to worker [SNIP] > > 2. Does ask_status throw an exception if the worker isn't running? I > would like to be able to ask if the worker is running before starting > another one. Nope, It won't throw any exception in user code. It does throw an exception internally. However, you can query status of all workers that are currently running, using MiddleMan.query_all_workers() See, if that output is helpful. ---- I do get an exception: BackgrounDRb::BdrbConnError when I do the following and the worker is not running. worker_status = MiddleMan.ask_status(:worker => :update_facebook_account_info_worker, :job_key => @id) I don't mind using a begin rescue block if that is the way to verify that the worker is not running. Is there a better way? P.S. I'm on IRC now but no one seems to be answering (I'm an IRC noob - sorry). -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From Ivan.Manida at Sun.COM Mon Dec 17 16:40:20 2007 From: Ivan.Manida at Sun.COM (Ivan Manida) Date: Tue, 18 Dec 2007 00:40:20 +0300 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb release 1.0 available now In-Reply-To: <1197917572.6207.30.camel@shire> References: <4766C157.1090008@Sun.COM> <1197917572.6207.30.camel@shire> Message-ID: <4766ECC4.9010509@sun.com> hemant kumar wrote: > On Mon, 2007-12-17 at 21:35 +0300, Ivan S. Manida wrote: >> Hemant, >> >> "ruby script/backgroundrb stop" does not stop the currently running >> server. What is a good way to restart the server, besides doing a >> kill -9 `cat backgroundrb.pid`? >> >> > > You got to be kidding. :) > It does stops backgroundrb server for me. Here are possible fixes: > > 1. Remove old "backgroundrb" script lying in your script directory. > 2. run: rake backgroundrb:setup > 3. Remove any :log: foreground option if you are using > 3. start bdrb server: ./script/backgroundrb start > 4. stop bdrb server: ./script/backgroundrb stop > 5. File a bug with super critical priority if it doesn't work. Mention, > which OS, which version of bdrb. > > :) OS is Solaris, bdrb is up-to-the-minute. I did recreate the control scripts. After further investigation, it *maybe* happens only while there is a worker that does something. But the fact is there - it reports "deleting pid file" to console and the process is not removed - start fails since port is taken, have to kill. I'll test more tomorrow and will file a bug with more details. From gethemant at gmail.com Mon Dec 17 21:18:47 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 18 Dec 2007 07:48:47 +0530 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb release 1.0 available now In-Reply-To: <4766ECC4.9010509@sun.com> References: <4766C157.1090008@Sun.COM> <1197917572.6207.30.camel@shire> <4766ECC4.9010509@sun.com> Message-ID: <1197944327.6207.37.camel@shire> Hi, On Tue, 2007-12-18 at 00:40 +0300, Ivan Manida wrote: > > OS is Solaris, bdrb is up-to-the-minute. I did recreate the control > scripts. After further investigation, it *maybe* happens only while > there is a worker that does something. But the fact is there - it > reports "deleting pid file" to console and the process is not removed - > start fails since port is taken, have to kill. I'll test more tomorrow > and will file a bug with more details. Could be a solaris specific issue. I will check it up. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Mon Dec 17 21:52:41 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 18 Dec 2007 08:22:41 +0530 Subject: [Backgroundrb-devel] wday not honored in cron trigger In-Reply-To: <65B8EC13-DDE9-4E10-A356-446CAD04D6FA@holonyx.com> References: <65B8EC13-DDE9-4E10-A356-446CAD04D6FA@holonyx.com> Message-ID: <1197946361.6207.41.camel@shire> On Mon, 2007-12-17 at 09:55 -0700, Adam Lebsack wrote: > , and I noticed that wday is not honored. Looking at the code > confirms this. Has anyone else seen this? patch? -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From reynard.list at gmail.com Tue Dec 18 10:27:35 2007 From: reynard.list at gmail.com (Reynard) Date: Tue, 18 Dec 2007 10:27:35 -0500 Subject: [Backgroundrb-devel] querying status of worker with job_key Message-ID: <26e148920712180727n53aa23e8q5cea749c1959da1c@mail.gmail.com> Hi, Is it possible to start a worker with job_key and then ask_status of that specific worker with the job_key? MiddleMan.ask_work(:worker => :auth_worker, :job_key=>'1', :worker_method => :auth) and then MiddleMan.ask_status(:worker => :auth_worker, :job_key=>'1') >From my testing of the new backgroundrb, asking status using job key always return blank. Is it possible to do this? thanks, - reynard -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071218/bc2f7038/attachment.html From Ivan.Manida at Sun.COM Tue Dec 18 11:50:54 2007 From: Ivan.Manida at Sun.COM (Ivan S. Manida) Date: Tue, 18 Dec 2007 19:50:54 +0300 Subject: [Backgroundrb-devel] querying status of worker with job_key In-Reply-To: <26e148920712180727n53aa23e8q5cea749c1959da1c@mail.gmail.com> References: <26e148920712180727n53aa23e8q5cea749c1959da1c@mail.gmail.com> Message-ID: <4767FA6E.8000704@Sun.COM> Reynard wrote: > Is it possible to start a worker with job_key and then ask_status of that > specific worker with the job_key? > > MiddleMan.ask_work(:worker => :auth_worker, :job_key=>'1', > :worker_method => :auth) > > and then > > MiddleMan.ask_status(:worker => :auth_worker, :job_key=>'1') You should use register_status() from your worker - this works for me (using job_key as you describe). Of course, code is not yet well tested, but at least the status part is working fine :) I have another problem - meta_pimp dies sometimes, in about 50% of the jobs. With the following stack trace. Worker is trivial but is using job_key. Looking through the code, it seems that connection is dropped while working, but which one - master <-> worker? /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/meta_pimp.rb:58:in `process_response': undefined method `instance' for nil:NilClass (NoMethodError) from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/meta_pimp.rb:29:in `handle_object' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/meta_pimp.rb:20:in `receive_data' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in `call' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in `extract' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:42:in `extract' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/meta_pimp.rb:18:in `receive_data' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/packet_master.rb:63:in `handle_internal_messages' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:154:in `start_reactor' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:152:in `each' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:152:in `start_reactor' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:143:in `loop' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:143:in `start_reactor' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/packet_master.rb:20:in `run' from /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:133:in `initialize' from script/backgroundrb:37:in `new' from script/backgroundrb:37 Some read error Some read error Some read error Some read error Some read error Some read error Some read error Some read error Some read error Some read error Some read error Some read error Some read error From reynard.list at gmail.com Tue Dec 18 12:10:31 2007 From: reynard.list at gmail.com (Reynard) Date: Tue, 18 Dec 2007 12:10:31 -0500 Subject: [Backgroundrb-devel] querying status of worker with job_key In-Reply-To: <4767FA6E.8000704@Sun.COM> References: <26e148920712180727n53aa23e8q5cea749c1959da1c@mail.gmail.com> <4767FA6E.8000704@Sun.COM> Message-ID: <26e148920712180910n1293a2a7i60b5def2a91b620f@mail.gmail.com> > You should use register_status() from your worker - this works for > me (using job_key as you describe). Of course, code is not yet well > tested, but at least the status part is working fine :) > I did register the status from the worker just like this register_status('success') do you have to pass the job key again when registering the status? and how do you ask_status for that job_key? is this the right way? MiddleMan.ask_status(:worker => :auth_worker, :job_key=>'1') anyway, everytime I call ask_status with job_key it returns nil, If I don't pass the job_key it returns something (which is the default process I think) - reynard -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071218/a666be2b/attachment.html From gethemant at gmail.com Tue Dec 18 12:12:40 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 18 Dec 2007 22:42:40 +0530 Subject: [Backgroundrb-devel] querying status of worker with job_key In-Reply-To: <4767FA6E.8000704@Sun.COM> References: <26e148920712180727n53aa23e8q5cea749c1959da1c@mail.gmail.com> <4767FA6E.8000704@Sun.COM> Message-ID: <1197997960.22386.17.camel@shire> On Tue, 2007-12-18 at 19:50 +0300, Ivan S. Manida wrote: > Reynard wrote: > > Is it possible to start a worker with job_key and then ask_status > of that > > specific worker with the job_key? > > > > MiddleMan.ask_work(:worker => :auth_worker, :job_key=>'1', > > :worker_method => :auth) > > > > and then > > > > MiddleMan.ask_status(:worker => :auth_worker, :job_key=>'1') > > > You should use register_status() from your worker - this works for > me (using job_key as you describe). Of course, code is not yet well > tested, but at least the status part is working fine :) yes thats correct, before being able to query status of a worker, you need to set the status object first and it will work afterwards. For example: class ErrorWorker < BackgrounDRb::MetaWorker set_worker_name :error_worker set_no_auto_load(true) def create(args = nil) logger.info "login worker started" register_status("Started") end end >> MiddleMan.new_worker(:worker => :error_worker, :job_key => "hello") => 71 >> MiddleMan.ask_status(:worker => :error_worker, :job_key => "hello") => "Started" > > I have another problem - meta_pimp dies sometimes, in about 50% of > the jobs. With the following stack trace. Worker is trivial but is > using job_key. Looking through the code, it seems that connection is > dropped while working, but which one - master <-> worker? > > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/meta_pimp.rb:58:in > `process_response': undefined method `instance' for nil:NilClass > (NoMethodError) > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/meta_pimp.rb:29:in > `handle_object' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/meta_pimp.rb:20:in > `receive_data' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > `call' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > `extract' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:42:in > `extract' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/meta_pimp.rb:18:in > `receive_data' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/packet_master.rb:63:in > `handle_internal_messages' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:154:in > `start_reactor' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:152:in > `each' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:152:in > `start_reactor' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:143:in > `loop' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/core.rb:143:in > `start_reactor' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/framework/packet_master.rb:20:in > `run' > from > /var/apps/ror_trunk/vendor/plugins/backgroundrb/server/master_worker.rb:133:in > `initialize' > from script/backgroundrb:37:in `new' > from script/backgroundrb:37 > Some read error Ok Ivan, This is happening because, bdrb worker is trying to send data to rails, even when rails has disconncted. It should be fixed now. Check it out. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Tue Dec 18 12:20:11 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 18 Dec 2007 22:50:11 +0530 Subject: [Backgroundrb-devel] querying status of worker with job_key In-Reply-To: <26e148920712180910n1293a2a7i60b5def2a91b620f@mail.gmail.com> References: <26e148920712180727n53aa23e8q5cea749c1959da1c@mail.gmail.com> <4767FA6E.8000704@Sun.COM> <26e148920712180910n1293a2a7i60b5def2a91b620f@mail.gmail.com> Message-ID: <1197998411.22386.19.camel@shire> On Tue, 2007-12-18 at 12:10 -0500, Reynard wrote: > > You should use register_status() from your worker - this works > for > me (using job_key as you describe). Of course, code is not yet > well > tested, but at least the status part is working fine :) > > I did register the status from the worker just like this > > register_status('success') > > do you have to pass the job key again when registering the status? > > and how do you ask_status for that job_key? is this the right way? > > MiddleMan.ask_status(:worker => :auth_worker, :job_key=>'1') > > anyway, everytime I call ask_status with job_key it returns nil, If I > don't pass the job_key it returns something (which is the default > process I think) Sync with upto the minute latest code of backgroundrb. paste your worker code. What you are doing, seem to be correct. I would like to see, what can go wrong. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From Ivan.Manida at Sun.COM Tue Dec 18 12:25:49 2007 From: Ivan.Manida at Sun.COM (Ivan S. Manida) Date: Tue, 18 Dec 2007 20:25:49 +0300 Subject: [Backgroundrb-devel] querying status of worker with job_key In-Reply-To: <26e148920712180910n1293a2a7i60b5def2a91b620f@mail.gmail.com> References: <26e148920712180727n53aa23e8q5cea749c1959da1c@mail.gmail.com> <4767FA6E.8000704@Sun.COM> <26e148920712180910n1293a2a7i60b5def2a91b620f@mail.gmail.com> Message-ID: <4768029D.4000605@Sun.COM> Reynard wrote: > I did register the status from the worker just like this > > register_status('success') > > do you have to pass the job key again when registering the status? > > and how do you ask_status for that job_key? is this the right way? > > MiddleMan.ask_status(:worker => :auth_worker, :job_key=>'1') > That seems right. I suspect you don't do a MiddleMan.new_worker(:worker => :auth_worker, :job_key=>'1') which would be needed to create a separate worker instance with your job_key. From adam at holonyx.com Tue Dec 18 13:32:16 2007 From: adam at holonyx.com (Adam Lebsack) Date: Tue, 18 Dec 2007 11:32:16 -0700 Subject: [Backgroundrb-devel] wday not honored in cron trigger In-Reply-To: <65B8EC13-DDE9-4E10-A356-446CAD04D6FA@holonyx.com> References: <65B8EC13-DDE9-4E10-A356-446CAD04D6FA@holonyx.com> Message-ID: <78D6B090-4B5F-4713-B266-A8EDCEF406C9@holonyx.com> I got a patch for this. I think I figured out why wday processing was omitted. :) It's attached. -------------- next part -------------- A non-text attachment was scrubbed... Name: cron_trigger.patch Type: application/octet-stream Size: 4588 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071218/e3ddd507/attachment.obj -------------- next part -------------- Adam Lebsack Engineering Manager Ruffdogs Software, a division of Holonyx, Inc. adam at holonyx.com 970-232-2050 x2010 On Dec 17, 2007, at 9:55 AM, Adam Lebsack wrote: > Hey guys, > > I was playing with the cron trigger, and I noticed that wday is not > honored. Looking at the code confirms this. Has anyone else seen > this? > > Thanks > > Adam_______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From gethemant at gmail.com Tue Dec 18 13:39:12 2007 From: gethemant at gmail.com (hemant kumar) Date: Wed, 19 Dec 2007 00:09:12 +0530 Subject: [Backgroundrb-devel] wday not honored in cron trigger In-Reply-To: <78D6B090-4B5F-4713-B266-A8EDCEF406C9@holonyx.com> References: <65B8EC13-DDE9-4E10-A356-446CAD04D6FA@holonyx.com> <78D6B090-4B5F-4713-B266-A8EDCEF406C9@holonyx.com> Message-ID: <1198003152.22386.21.camel@shire> On Tue, 2007-12-18 at 11:32 -0700, Adam Lebsack wrote: > I got a patch for this. I think I figured out why wday processing > was omitted. :) > > > It's attached. Thanks Adam, I will have a look and commit it. > > > > Adam Lebsack > Engineering Manager > Ruffdogs Software, a division of Holonyx, Inc. > adam at holonyx.com > 970-232-2050 x2010 > > > > > On Dec 17, 2007, at 9:55 AM, Adam Lebsack wrote: > > > Hey guys, > > > > I was playing with the cron trigger, and I noticed that wday is not > > honored. Looking at the code confirms this. Has anyone else seen > > this? > > > > Thanks > > > > Adam_______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From rob at iotum.com Tue Dec 18 13:58:11 2007 From: rob at iotum.com (Rob Nielsen) Date: Tue, 18 Dec 2007 13:58:11 -0500 Subject: [Backgroundrb-devel] Backgroundrb crash calling ask_status when worker not running Message-ID: <48C76E56FDD0DF44B6585A82E5DD27AEEE843D@iotum-sbs.IotumCorp.local> When I call ask_status on a worker that has not been started I get and exception of type BackgrounDRb::BdrbConnError, which is fine. I can deal with that, but I also find that the backgroundrb process dies as well. The scenario is simple: Call ask_status on a worker that is not auto loaded. I used a job_key, in case that makes a difference. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071218/0fd2301e/attachment.html From reynard.list at gmail.com Tue Dec 18 15:21:00 2007 From: reynard.list at gmail.com (Reynard) Date: Tue, 18 Dec 2007 15:21:00 -0500 Subject: [Backgroundrb-devel] where does process status get stored? Message-ID: <26e148920712181221m42a2c08by20cf32449aabef51@mail.gmail.com> I'm just curious where does the process status get stored? Is it in the backgroundrb process, or some external storage? The reason I'm asking is because I have been doing stress testing by creating thousands of workers (with uniq job_key), each does register_status and exit after it's done. I have been monitoring the memory usage of the backgroundrb server. At first the memory usage increases to a certain extent and then seems to stop increasing, although it seems that I still can access the status of all workers. Which makes me wonder if there is some kind of garbage collection on the status? Since I'm planning to use this for external authentication, it's possible that we create thoudsands of process in a day. After the user is authenticated there is no need to keep the status around for a long time. so I'm just concern about the memory usage or storage for this old worker status. I don't want to have to restart backgroundrb regularly. any thought? -reynard -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071218/fbb9a736/attachment.html From reynard.list at gmail.com Tue Dec 18 15:27:39 2007 From: reynard.list at gmail.com (Reynard) Date: Tue, 18 Dec 2007 15:27:39 -0500 Subject: [Backgroundrb-devel] process queue Message-ID: <26e148920712181227r65e85001r3f9f5a51c462b823@mail.gmail.com> I noticed that backgroundrb does some kind of process queueing now? When I pounded the server to create 1000 workers, there are always only around 30-40 processes running at a time, which is good :) Is there some configuration for adjusting the max process? - reynard -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071218/4356a014/attachment.html From jason.lapier at gmail.com Tue Dec 18 20:28:18 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Tue, 18 Dec 2007 17:28:18 -0800 Subject: [Backgroundrb-devel] thread_pooling sleeping Message-ID: <93b44c1f0712181728l6b65e649wf525729f108c3e4f@mail.gmail.com> I'm trying to run a single worker that could perform a periodic task for a given user. >From a controller, I imagine something like: def start_job MiddleMan.ask_work(:worker => :foo_worker, :worker_method => :perform_task, :data => { :user_id = current_user.id }) end def check_job @status = MiddleMan.ask_status(:worker => :foo_worker)[current_user.id] end My worker is something like: class FooWorker < BackgrounDRb::MetaWorker set_worker_name :foo_worker def create(args=nil) @mutex = Mutex.new @mutex.synchronize do @statuses = {} register_status(@statuses) end end def do_task(some_user_id) thread_pool.defer(some_user_id) do |user_id| user = User.find user_id save_status user_id, :progress, "Starting Task" user.do_some_database_stuff save_status user_id, :progress, "Task Stage 2" user.do_some_other_database_stuff save_status user_id, :progress, "Task Stage 3" save_status user_id, :completed, true end end def save_status(user_id, key, data,) @mutex.synchronize do if @statuses[user_id].nil? @statuses[user_id] = {} @statuses[user_id][:completed] = false end @statuses[user_id][key] = data register_status(@statuses) logger.info "statuses synced for #{user_id}, #{key}, #{data}" end end end Problem is, when I use thread_pool, it gets as far as the first part of the task and then just dies, and the backgroundrb script just outputs "going to sleep for a while" over and over until I kill it. I tried taking out the mutex and the register_status bits and the same problem happens. When I use "Thread.new" in place of the thread_pool line, everything works. However, I get some "Some read error" messages during the "do_task" work. Everything happens like it should - I even ran two sessions concurrently and initiated the task with two different users at the same time. The work gets done; although a bit slowly and with those strange errors. Oddly enough, when I run the jobs from the rails irb console, they go without any error - even if I start a couple jobs for multiple users and spam "MiddleMan.ask_status" while the jobs are still running. (of course, when I start the server using the 'Thread.new' method, I get a "no marshal_dump is defined for class Thread" exception in the backgroundrb_server log. I'm guessing this has something to do with why I'm supposed to be using thread_pool instead of Thread). When I get those read errors, the backgroundrb_debug.log shows "Client disconected" (sic). I don't know if this is normal for rails talking to bgrb or if it's an error. I don't see it on every line, but it seems to coincide with the "some read error" messages in the script. Am I doing something wrong? Anyone have a working example using thread_pool? Also, I've never used Mutex before, so if my usage is off, please let me know. Thanks very much, - Jason L. -- My Rails and Linux Blog: http://offtheline.net From gethemant at gmail.com Wed Dec 19 02:04:42 2007 From: gethemant at gmail.com (hemant) Date: Wed, 19 Dec 2007 12:34:42 +0530 Subject: [Backgroundrb-devel] Backgroundrb crash calling ask_status when worker not running In-Reply-To: <48C76E56FDD0DF44B6585A82E5DD27AEEE843D@iotum-sbs.IotumCorp.local> References: <48C76E56FDD0DF44B6585A82E5DD27AEEE843D@iotum-sbs.IotumCorp.local> Message-ID: On Dec 19, 2007 12:28 AM, Rob Nielsen wrote: > > > > > When I call ask_status on a worker that has not been started I get and > exception of type BackgrounDRb::BdrbConnError, which is fine. I can deal > with that, but I also find that the backgroundrb process dies as well. > > > > The scenario is simple: Call ask_status on a worker that is not auto loaded. > I used a job_key, in case that makes a difference. > > I did test again and again, and didn't get any crash Rob. So, if can send us the worker code and the code using which you are accessing it from rails, we would be glad to help. Also, if possible, try to capture the exception, when this happens, there should be something logged in RAILS_ROOT/log/backgroundrb_debug.log or RAILS_ROOT/backgroundrb_server.log or RAILS_ROOT/backgroundrb.log . If they do not indicate any exception, then run bdrb in foreground mode, using: #backgroundrb.yml :backgroundrb: :log: foreground and start backgroundrb server like this: ./script/backgroundrb It should print any unhandled exception. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Wed Dec 19 03:18:01 2007 From: gethemant at gmail.com (hemant) Date: Wed, 19 Dec 2007 13:48:01 +0530 Subject: [Backgroundrb-devel] where does process status get stored? In-Reply-To: <26e148920712181221m42a2c08by20cf32449aabef51@mail.gmail.com> References: <26e148920712181221m42a2c08by20cf32449aabef51@mail.gmail.com> Message-ID: On Dec 19, 2007 1:51 AM, Reynard wrote: > > I'm just curious where does the process status get stored? Is it in the > backgroundrb process, or some external storage? > The reason I'm asking is because I have been doing stress testing by > creating thousands of workers (with uniq job_key), each does register_status > and exit after it's done. > I have been monitoring the memory usage of the backgroundrb server. At first > the memory usage increases to a certain extent and then seems to stop > increasing, although it seems that I still can access the status of all > workers. > > Which makes me wonder if there is some kind of garbage collection on the > status? > > Since I'm planning to use this for external authentication, it's possible > that we create thoudsands of process in a day. After the user is > authenticated there is no need to keep the status around for a long time. so > I'm just concern about the memory usage or storage for this old worker > status. I don't want to have to restart backgroundrb regularly. > > any thought? > Process status objects are stored in master process, and since they generally do not consume much memory, i do not see any reason for exponential growth in process size. Also, there is no garbage collection apart from default Ruby one. However, if you are worrried about growth of memory because of that in long run, you may try to pass memcache option in configuration file, so as status objects are stored in memcache. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Wed Dec 19 03:19:25 2007 From: gethemant at gmail.com (hemant) Date: Wed, 19 Dec 2007 13:49:25 +0530 Subject: [Backgroundrb-devel] process queue In-Reply-To: <26e148920712181227r65e85001r3f9f5a51c462b823@mail.gmail.com> References: <26e148920712181227r65e85001r3f9f5a51c462b823@mail.gmail.com> Message-ID: On Dec 19, 2007 1:57 AM, Reynard wrote: > > I noticed that backgroundrb does some kind of process queueing now? When I > pounded the server to create 1000 workers, there are always only around > 30-40 processes running at a time, which is good :) > Is there some configuration for adjusting the max process? Wow reynard, Thanks for stress testing bdrb. :) There is no cap on maximum number of processes imposed by backgroundrb, but perhaps we can introduce one, if required. > > - reynard > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Wed Dec 19 03:24:03 2007 From: gethemant at gmail.com (hemant) Date: Wed, 19 Dec 2007 13:54:03 +0530 Subject: [Backgroundrb-devel] thread_pooling sleeping In-Reply-To: <93b44c1f0712181728l6b65e649wf525729f108c3e4f@mail.gmail.com> References: <93b44c1f0712181728l6b65e649wf525729f108c3e4f@mail.gmail.com> Message-ID: On Dec 19, 2007 6:58 AM, Jason LaPier wrote: > I'm trying to run a single worker that could perform a periodic task > for a given user. > > >From a controller, I imagine something like: > def start_job > MiddleMan.ask_work(:worker => :foo_worker, :worker_method => :perform_task, > :data => { :user_id = current_user.id }) > end > > def check_job > @status = MiddleMan.ask_status(:worker => :foo_worker)[current_user.id] > end > > My worker is something like: > class FooWorker < BackgrounDRb::MetaWorker > set_worker_name :foo_worker > def create(args=nil) > @mutex = Mutex.new > @mutex.synchronize do > @statuses = {} > register_status(@statuses) > end > end > > def do_task(some_user_id) > thread_pool.defer(some_user_id) do |user_id| > user = User.find user_id > save_status user_id, :progress, "Starting Task" > user.do_some_database_stuff > save_status user_id, :progress, "Task Stage 2" > user.do_some_other_database_stuff > save_status user_id, :progress, "Task Stage 3" > save_status user_id, :completed, true > end > end > > def save_status(user_id, key, data,) > @mutex.synchronize do > if @statuses[user_id].nil? > @statuses[user_id] = {} > @statuses[user_id][:completed] = false > end > @statuses[user_id][key] = data > register_status(@statuses) > logger.info "statuses synced for #{user_id}, #{key}, #{data}" > end > end > > end > > > Problem is, when I use thread_pool, it gets as far as the first part > of the task and then just dies, and the backgroundrb script just > outputs "going to sleep for a while" over and over until I kill it. I > tried taking out the mutex and the register_status bits and the same > problem happens. When I use "Thread.new" in place of the thread_pool > line, everything works. However, I get some "Some read error" messages > during the "do_task" work. Everything happens like it should - I even > ran two sessions concurrently and initiated the task with two > different users at the same time. The work gets done; although a bit > slowly and with those strange errors. Oddly enough, when I run the > jobs from the rails irb console, they go without any error - even if I > start a couple jobs for multiple users and spam "MiddleMan.ask_status" > while the jobs are still running. (of course, when I start the server > using the 'Thread.new' method, I get a "no marshal_dump is defined for > class Thread" exception in the backgroundrb_server log. I'm guessing > this has something to do with why I'm supposed to be using thread_pool > instead of Thread). > > When I get those read errors, the backgroundrb_debug.log shows "Client > disconected" (sic). I don't know if this is normal for rails talking > to bgrb or if it's an error. I don't see it on every line, but it > seems to coincide with the "some read error" messages in the script. > > Am I doing something wrong? Anyone have a working example using > thread_pool? Also, I've never used Mutex before, so if my usage is > off, please let me know. > > Thanks very much, Hmm, I am having a look at the problem. Do not worry about "some read error" messages, they just indicate that connection that rails was having with backgroundrb server is closed. This happens quite often because whenever you do: MiddleMan.xxx A new connection to backgroundrb server is initiated and older one is closed and at that time you get that message. I put those "puts" for debugging purpose. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Wed Dec 19 04:18:21 2007 From: gethemant at gmail.com (hemant) Date: Wed, 19 Dec 2007 14:48:21 +0530 Subject: [Backgroundrb-devel] thread_pooling sleeping In-Reply-To: References: <93b44c1f0712181728l6b65e649wf525729f108c3e4f@mail.gmail.com> Message-ID: On Dec 19, 2007 1:54 PM, hemant wrote: > > On Dec 19, 2007 6:58 AM, Jason LaPier wrote: > > I'm trying to run a single worker that could perform a periodic task > > for a given user. > > > > >From a controller, I imagine something like: > > def start_job > > MiddleMan.ask_work(:worker => :foo_worker, :worker_method => :perform_task, > > :data => { :user_id = current_user.id }) > > end > > > > def check_job > > @status = MiddleMan.ask_status(:worker => :foo_worker)[current_user.id] > > end > > > > My worker is something like: > > class FooWorker < BackgrounDRb::MetaWorker > > set_worker_name :foo_worker > > def create(args=nil) > > @mutex = Mutex.new > > @mutex.synchronize do > > @statuses = {} > > register_status(@statuses) > > end > > end > > > > def do_task(some_user_id) > > thread_pool.defer(some_user_id) do |user_id| > > user = User.find user_id > > save_status user_id, :progress, "Starting Task" > > user.do_some_database_stuff > > save_status user_id, :progress, "Task Stage 2" > > user.do_some_other_database_stuff > > save_status user_id, :progress, "Task Stage 3" > > save_status user_id, :completed, true > > end > > end > > > > def save_status(user_id, key, data,) > > @mutex.synchronize do > > if @statuses[user_id].nil? > > @statuses[user_id] = {} > > @statuses[user_id][:completed] = false > > end > > @statuses[user_id][key] = data > > register_status(@statuses) > > logger.info "statuses synced for #{user_id}, #{key}, #{data}" > > end > > end > > > > end > > > > > > Problem is, when I use thread_pool, it gets as far as the first part > > of the task and then just dies, and the backgroundrb script just > > outputs "going to sleep for a while" over and over until I kill it. I > > tried taking out the mutex and the register_status bits and the same > > problem happens. When I use "Thread.new" in place of the thread_pool > > line, everything works. However, I get some "Some read error" messages > > during the "do_task" work. Everything happens like it should - I even > > ran two sessions concurrently and initiated the task with two > > different users at the same time. The work gets done; although a bit > > slowly and with those strange errors. Oddly enough, when I run the > > jobs from the rails irb console, they go without any error - even if I > > start a couple jobs for multiple users and spam "MiddleMan.ask_status" > > while the jobs are still running. (of course, when I start the server > > using the 'Thread.new' method, I get a "no marshal_dump is defined for > > class Thread" exception in the backgroundrb_server log. I'm guessing > > this has something to do with why I'm supposed to be using thread_pool > > instead of Thread). > > > > When I get those read errors, the backgroundrb_debug.log shows "Client > > disconected" (sic). I don't know if this is normal for rails talking > > to bgrb or if it's an error. I don't see it on every line, but it > > seems to coincide with the "some read error" messages in the script. > > > > Am I doing something wrong? Anyone have a working example using > > thread_pool? Also, I've never used Mutex before, so if my usage is > > off, please let me know. > > > > Thanks very much, > > Hmm, > > I am having a look at the problem. Do not worry about "some read > error" messages, they just indicate that connection that rails was > having with backgroundrb server is closed. This happens quite often > because whenever you do: > > MiddleMan.xxx > > A new connection to backgroundrb server is initiated and older one is > closed and at that time you get that message. I put those "puts" for > debugging purpose. Hi Jason, Looks like there is an exception being thrown thats not handled in your code. Here is my sample worker code, similar to yours. I also got similar behaviour as you described, but then i set "Thread.abort_on_exception = true" and also ran backgroundrb in foreground mode(see README file for details). And then it turned out, there was an exception being thrown in my code and hence it was not going beyond first stage. Also, you don't need to protect register_status() with mutex, but rather you should protect status variable with a mutex as demonstrated below. class ErrorWorker < BackgrounDRb::MetaWorker set_worker_name :error_worker attr_accessor :worker_status def create(args = nil) @worker_status = { } @status_mutex = Mutex.new register_status(@worker_status) Thread.abort_on_exception = true end def do_task(user_id) thread_pool.defer(user_id) do |user_id| save_status(user_id,:progress,"Started") t_user = World.find_by_id(user_id) t_user.change_message logger.info "message changed" save_status(user_id,:progress,"message_changed") t_user.change_name logger.info "name changed" save_status(user_id,:progress,"name changed") t_user.save end end def save_status(user_id,key,data) @status_mutex.synchronize do @worker_status[user_id] = { :key => key, :data => data} end register_status(@worker_status) logger.info "status synced for user #{user_id} #{key} #{data}" end end To stress test I did this: require File.join(File.dirname(__FILE__) + "/../config/environment") p MiddleMan.ask_status(:worker => :error_worker) (7..200).to_a.each do |world_id| MiddleMan.ask_work(:worker => :error_worker, :worker_method => :do_task, :data => world_id) p MiddleMan.ask_status(:worker => :error_worker) end and it worked flawlessly. See if this helps. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From jason.lapier at gmail.com Wed Dec 19 12:28:02 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Wed, 19 Dec 2007 09:28:02 -0800 Subject: [Backgroundrb-devel] thread_pooling sleeping In-Reply-To: References: <93b44c1f0712181728l6b65e649wf525729f108c3e4f@mail.gmail.com> Message-ID: <93b44c1f0712190928u2dde79ebo32aea8e7bcfb353b@mail.gmail.com> On Dec 19, 2007 1:18 AM, hemant wrote: > > Hi Jason, > > Looks like there is an exception being thrown thats not handled in > your code. Here is my sample worker code, similar to yours. I also got > similar behaviour as you described, but then i set > "Thread.abort_on_exception = true" and also ran backgroundrb in > foreground mode(see README file for details). > > And then it turned out, there was an exception being thrown in my code > and hence it was not going beyond first stage. Also, you don't need to > protect register_status() with mutex, but rather you should protect > status variable with a mutex as demonstrated below. > > > class ErrorWorker < BackgrounDRb::MetaWorker > set_worker_name :error_worker > attr_accessor :worker_status > def create(args = nil) > @worker_status = { } > @status_mutex = Mutex.new > register_status(@worker_status) > Thread.abort_on_exception = true > end > > def do_task(user_id) > thread_pool.defer(user_id) do |user_id| > save_status(user_id,:progress,"Started") > t_user = World.find_by_id(user_id) > t_user.change_message > logger.info "message changed" > save_status(user_id,:progress,"message_changed") > t_user.change_name > logger.info "name changed" > save_status(user_id,:progress,"name changed") > t_user.save > end > end > > def save_status(user_id,key,data) > @status_mutex.synchronize do > @worker_status[user_id] = { :key => key, :data => data} > end > register_status(@worker_status) > logger.info "status synced for user #{user_id} #{key} #{data}" > end > end > > To stress test I did this: > > require File.join(File.dirname(__FILE__) + "/../config/environment") > p MiddleMan.ask_status(:worker => :error_worker) > > (7..200).to_a.each do |world_id| > MiddleMan.ask_work(:worker => :error_worker, :worker_method => > :do_task, :data => world_id) > p MiddleMan.ask_status(:worker => :error_worker) > end > > and it worked flawlessly. > > See if this helps. > Ok, awesome - thanks very much for your help. With the abort_on_exception flag and the foreground logging, I was able to track down the problem. If you try something like: def do_task(some_user_id) logger.info "do_task: some_user_id: #{some_user_id.class}" logger.info "starting task - defer to the thread pool" thread_pool.defer(some_user_id) do |user_id| logger.info "user_id (in pool): #{user_id.class}" # and the rest of it end end You'll see in the log: do_task: some_user_id: Fixnum starting task - defer to the thread pool user_id (in pool): Array So some_user_id is being converted to [user_id] for the block. Calling AR's find method with an array gives you back an array (which was the source of my original exception - trying to act on an Array as if it were a User object). When you use User.find_by_id in your example, AR is taking just the first element in the array and doing a find on that. I suspect that's a bug in thread_pool, or at least if it's not, it's a little confusing syntactically. Now that I know how it works, I've got my code working, so if it's not a bug, no problem :) - Jason L. -- My Rails and Linux Blog: http://offtheline.net From gethemant at gmail.com Wed Dec 19 16:08:15 2007 From: gethemant at gmail.com (hemant) Date: Thu, 20 Dec 2007 02:38:15 +0530 Subject: [Backgroundrb-devel] thread_pooling sleeping In-Reply-To: <93b44c1f0712190928u2dde79ebo32aea8e7bcfb353b@mail.gmail.com> References: <93b44c1f0712181728l6b65e649wf525729f108c3e4f@mail.gmail.com> <93b44c1f0712190928u2dde79ebo32aea8e7bcfb353b@mail.gmail.com> Message-ID: On Dec 19, 2007 10:58 PM, Jason LaPier wrote: > > On Dec 19, 2007 1:18 AM, hemant wrote: > > > > Hi Jason, > > > > Looks like there is an exception being thrown thats not handled in > > your code. Here is my sample worker code, similar to yours. I also got > > similar behaviour as you described, but then i set > > "Thread.abort_on_exception = true" and also ran backgroundrb in > > foreground mode(see README file for details). > > > > And then it turned out, there was an exception being thrown in my code > > and hence it was not going beyond first stage. Also, you don't need to > > protect register_status() with mutex, but rather you should protect > > status variable with a mutex as demonstrated below. > > > > > > class ErrorWorker < BackgrounDRb::MetaWorker > > set_worker_name :error_worker > > attr_accessor :worker_status > > def create(args = nil) > > @worker_status = { } > > @status_mutex = Mutex.new > > register_status(@worker_status) > > Thread.abort_on_exception = true > > end > > > > def do_task(user_id) > > thread_pool.defer(user_id) do |user_id| > > save_status(user_id,:progress,"Started") > > t_user = World.find_by_id(user_id) > > t_user.change_message > > logger.info "message changed" > > save_status(user_id,:progress,"message_changed") > > t_user.change_name > > logger.info "name changed" > > save_status(user_id,:progress,"name changed") > > t_user.save > > end > > end > > > > def save_status(user_id,key,data) > > @status_mutex.synchronize do > > @worker_status[user_id] = { :key => key, :data => data} > > end > > register_status(@worker_status) > > logger.info "status synced for user #{user_id} #{key} #{data}" > > end > > end > > > > To stress test I did this: > > > > require File.join(File.dirname(__FILE__) + "/../config/environment") > > p MiddleMan.ask_status(:worker => :error_worker) > > > > (7..200).to_a.each do |world_id| > > MiddleMan.ask_work(:worker => :error_worker, :worker_method => > > :do_task, :data => world_id) > > p MiddleMan.ask_status(:worker => :error_worker) > > end > > > > and it worked flawlessly. > > > > See if this helps. > > > > Ok, awesome - thanks very much for your help. With the > abort_on_exception flag and the foreground logging, I was able to > track down the problem. If you try something like: > > def do_task(some_user_id) > logger.info "do_task: some_user_id: #{some_user_id.class}" > logger.info "starting task - defer to the thread pool" > > thread_pool.defer(some_user_id) do |user_id| > logger.info "user_id (in pool): #{user_id.class}" > # and the rest of it > end > end > > You'll see in the log: > do_task: some_user_id: Fixnum > starting task - defer to the thread pool > user_id (in pool): Array > > So some_user_id is being converted to [user_id] for the block. Calling > AR's find method with an array gives you back an array (which was the > source of my original exception - trying to act on an Array as if it > were a User object). When you use User.find_by_id in your example, AR > is taking just the first element in the array and doing a find on > that. > > I suspect that's a bug in thread_pool, or at least if it's not, it's a > little confusing syntactically. Now that I know how it works, I've got > my code working, so if it's not a bug, no problem :) > > Glad that you solved the problem. Yes the block is getting passed value as an Array is a bug and its fixed now. Whats more now, if your block throws an exception which you are not handling, it will be handled by bdrb and logged in backgroundrb.log. Sync with latest code for all this goodness. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From jason.lapier at gmail.com Wed Dec 19 16:42:01 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Wed, 19 Dec 2007 13:42:01 -0800 Subject: [Backgroundrb-devel] thread_pooling sleeping In-Reply-To: References: <93b44c1f0712181728l6b65e649wf525729f108c3e4f@mail.gmail.com> <93b44c1f0712190928u2dde79ebo32aea8e7bcfb353b@mail.gmail.com> Message-ID: <93b44c1f0712191342p256df710ke037327140cc828e@mail.gmail.com> On Dec 19, 2007 1:08 PM, hemant wrote: > > Glad that you solved the problem. Yes the block is getting passed > value as an Array is a bug and its fixed now. > Whats more now, if your block throws an exception which you are not > handling, it will be handled by bdrb and logged in backgroundrb.log. > > Sync with latest code for all this goodness. Great! Thanks for fixing this stuff so quickly. I just synced and everything is working great. - Jason L. -- My Rails and Linux Blog: http://offtheline.net From faivrem at gmail.com Wed Dec 19 17:40:26 2007 From: faivrem at gmail.com (Mickael Faivre-Macon) Date: Wed, 19 Dec 2007 23:40:26 +0100 Subject: [Backgroundrb-devel] thread_pooling sleeping In-Reply-To: <93b44c1f0712191342p256df710ke037327140cc828e@mail.gmail.com> References: <93b44c1f0712181728l6b65e649wf525729f108c3e4f@mail.gmail.com> <93b44c1f0712190928u2dde79ebo32aea8e7bcfb353b@mail.gmail.com> <93b44c1f0712191342p256df710ke037327140cc828e@mail.gmail.com> Message-ID: Yes, Hemant is quick ! BackgrounDRd is one of the best piece of soft for Rails so far ! Thanks for your work Hemant ! I have implemented a little chat over XMPP with your help (with threadpool too), thanks ! Mickael. On Dec 19, 2007 10:42 PM, Jason LaPier wrote: > On Dec 19, 2007 1:08 PM, hemant wrote: > > > > Glad that you solved the problem. Yes the block is getting passed > > value as an Array is a bug and its fixed now. > > Whats more now, if your block throws an exception which you are not > > handling, it will be handled by bdrb and logged in backgroundrb.log. > > > > Sync with latest code for all this goodness. > > Great! Thanks for fixing this stuff so quickly. I just synced and > everything is working great. > > - Jason L. From gethemant at gmail.com Thu Dec 20 04:28:42 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 20 Dec 2007 14:58:42 +0530 Subject: [Backgroundrb-devel] wday not honored in cron trigger In-Reply-To: <78D6B090-4B5F-4713-B266-A8EDCEF406C9@holonyx.com> References: <65B8EC13-DDE9-4E10-A356-446CAD04D6FA@holonyx.com> <78D6B090-4B5F-4713-B266-A8EDCEF406C9@holonyx.com> Message-ID: <1198142922.6468.11.camel@shire> On Tue, 2007-12-18 at 11:32 -0700, Adam Lebsack wrote: > I got a patch for this. I think I figured out why wday processing > was omitted. :) > > > It's attached. > Thanks Adam, I reviewed your patch and committed it. > > > Adam Lebsack > Engineering Manager > Ruffdogs Software, a division of Holonyx, Inc. > adam at holonyx.com > 970-232-2050 x2010 > > > > > On Dec 17, 2007, at 9:55 AM, Adam Lebsack wrote: > > > Hey guys, > > > > I was playing with the cron trigger, and I noticed that wday is not > > honored. Looking at the code confirms this. Has anyone else seen > > this? > > > > Thanks > > > > Adam_______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From emarceta at gmail.com Thu Dec 20 10:29:49 2007 From: emarceta at gmail.com (Emil Marceta) Date: Thu, 20 Dec 2007 16:29:49 +0100 Subject: [Backgroundrb-devel] More pleasant / concise API Message-ID: <74b000990712200729h1d2c726dtdcd2c7b4125976be@mail.gmail.com> Hi there, Just a small suggestion : The following apis could be a bit more concise / shorter / pleasant I thought. This is particularly difficult, my brain stumbles on this each time when scanning thorugh the code :) set_no_auto_load(true) Could it be two apis instead? auto_load no_auto_load and the other one : worker_name :important_worker (no set_ prefix) thanks, emil From gethemant at gmail.com Thu Dec 20 10:47:25 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 20 Dec 2007 21:17:25 +0530 Subject: [Backgroundrb-devel] More pleasant / concise API In-Reply-To: <74b000990712200729h1d2c726dtdcd2c7b4125976be@mail.gmail.com> References: <74b000990712200729h1d2c726dtdcd2c7b4125976be@mail.gmail.com> Message-ID: <1198165645.7853.1.camel@shire> On Thu, 2007-12-20 at 16:29 +0100, Emil Marceta wrote: > Hi there, > > Just a small suggestion : > The following apis could be a bit more concise / shorter / pleasant I thought. > > This is particularly difficult, my brain stumbles on this each time > when scanning thorugh the code :) > set_no_auto_load(true) > > Could it be two apis instead? > > auto_load > no_auto_load > > and the other one : > > worker_name :important_worker (no set_ prefix) Yup, set_* prefix can be annoying at times and hence can be removed. Although all these methods are auto generated and hence a bit complicated to change them. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From zach at plugthegap.co.uk Thu Dec 20 11:12:38 2007 From: zach at plugthegap.co.uk (Zachary Powell) Date: Thu, 20 Dec 2007 11:12:38 -0500 Subject: [Backgroundrb-devel] updating to 1.0 In-Reply-To: References: Message-ID: Hey, in 1.0 how do I delete a worker from the inside (in 0.2 it was self.delete)? I need them to delete themselves after they've finished their work. thanks, Zach -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071220/2ff974f4/attachment.html From gethemant at gmail.com Thu Dec 20 11:15:05 2007 From: gethemant at gmail.com (hemant kumar) Date: Thu, 20 Dec 2007 21:45:05 +0530 Subject: [Backgroundrb-devel] updating to 1.0 In-Reply-To: References: Message-ID: <1198167305.7853.3.camel@shire> On Thu, 2007-12-20 at 11:12 -0500, Zachary Powell wrote: > Hey, in 1.0 how do I delete a worker from the inside (in 0.2 it was > self.delete)? I need them to delete themselves after they've finished > their work. Call "exit" on them, when they are done processing. > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From jason.lapier at gmail.com Thu Dec 20 13:40:32 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Thu, 20 Dec 2007 10:40:32 -0800 Subject: [Backgroundrb-devel] mocking a few things in the test helper Message-ID: <93b44c1f0712201040l3f5632c4sd8f04d341eb0b99b@mail.gmail.com> When using the provided bdrb_test_helper, I was getting some undefined methods (register_status, thread_pool, etc) in unit tests. The README says to go ahead and stub out methods that need it, and send the patch to the list, so here it is (attached). Also, the MetaWorker initialize method was written as "initializes", causing the @logger never to be set, causing undefined method errors reported as nil.info and nil.debug (nil being the 'logger' method). - Jason L. -- My Rails and Linux Blog: http://offtheline.net -------------- next part -------------- A non-text attachment was scrubbed... Name: bdrb_test_helper.patch Type: text/x-patch Size: 870 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071220/37db8910/attachment.bin From gethemant at gmail.com Thu Dec 20 14:05:16 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 21 Dec 2007 00:35:16 +0530 Subject: [Backgroundrb-devel] mocking a few things in the test helper In-Reply-To: <93b44c1f0712201040l3f5632c4sd8f04d341eb0b99b@mail.gmail.com> References: <93b44c1f0712201040l3f5632c4sd8f04d341eb0b99b@mail.gmail.com> Message-ID: <1198177516.12014.0.camel@shire> Hi Jason, On Thu, 2007-12-20 at 10:40 -0800, Jason LaPier wrote: > When using the provided bdrb_test_helper, I was getting some undefined > methods (register_status, thread_pool, etc) in unit tests. The README > says to go ahead and stub out methods that need it, and send the patch > to the list, so here it is (attached). Also, the MetaWorker initialize > method was written as "initializes", causing the @logger never to be > set, causing undefined method errors reported as nil.info and > nil.debug (nil being the 'logger' method). > > - Jason L. > Thanks buddy. I would merge them asap. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From kenglish at camberhawaii.org Thu Dec 20 19:40:16 2007 From: kenglish at camberhawaii.org (Kevin W. English) Date: Thu, 20 Dec 2007 14:40:16 -1000 Subject: [Backgroundrb-devel] rake test:plugins gives uninitialized constant RAILS_ROOT Message-ID: <476B0B70.7090107@camberhawaii.org> Hello, Just upgraded to the the latest backgroundRb.I am getting an error when i run the task "rake test:plugin". Any idea why I am getting this? Here is the error message: /usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb" "vendor/plugins/backgroundrb/test/master_worker_test.rb" "vendor/plugins/backgroundrb/test/backgroundrb_test.rb" "vendor/plugins/backgroundrb/test/meta_worker_test.rb" /home/kenglish/workspace/brbtest/vendor/plugins/backgroundrb/lib/backgroundrb.rb:4: uninitialized constant RAILS_ROOT (NameError) from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require' from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require' from ./vendor/plugins/backgroundrb/test/backgroundrb_test.rb:2 from /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb:5:in `load' from /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb:5 from /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb:5:in `each' from /usr/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb:5 rake aborted! -- Aloha, Kevin English Rails Evangelist Software Development and Integration Team Camber Corporation Honolulu, HI Office: 808.237.3539 Fax: 808.237.3540 -- Aloha, Kevin English Rails Evangelist Software Development and Integration Team Camber Corporation Honolulu, HI Office: 808.237.3539 Fax: 808.237.3540 From joevandyk at gmail.com Fri Dec 21 02:15:15 2007 From: joevandyk at gmail.com (Joe Van Dyk) Date: Thu, 20 Dec 2007 23:15:15 -0800 Subject: [Backgroundrb-devel] Exceptions in workers Message-ID: If an exception is thrown in a worker, will the worker process exit? If it does and I don't want it to exit, do I need to rescue all exceptions in every worker method? Joe From joevandyk at gmail.com Fri Dec 21 02:17:12 2007 From: joevandyk at gmail.com (Joe Van Dyk) Date: Thu, 20 Dec 2007 23:17:12 -0800 Subject: [Backgroundrb-devel] Jobs running too slow Message-ID: Say I have a job that runs every minute, but it takes longer than a minute to execute. Will it start a new job every minute? If it does, then how would I make a job that sleeps for a certain amount of time, then restarts? I have a job that could take an hour or two to complete, and I don't want to have multiple copies of that job running at the same time. Joe From gethemant at gmail.com Fri Dec 21 02:43:51 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 21 Dec 2007 13:13:51 +0530 Subject: [Backgroundrb-devel] Exceptions in workers In-Reply-To: References: Message-ID: <1198223031.28189.3.camel@shire> On Thu, 2007-12-20 at 23:15 -0800, Joe Van Dyk wrote: > If an exception is thrown in a worker, will the worker process exit? > If it does and I don't want it to exit, do I need to rescue all > exceptions in every worker method? > Every worker is just a Ruby program forked in a new process, so a unhandled exception WILL cause the worker to exit. From joevandyk at gmail.com Fri Dec 21 02:49:45 2007 From: joevandyk at gmail.com (Joe Van Dyk) Date: Thu, 20 Dec 2007 23:49:45 -0800 Subject: [Backgroundrb-devel] Exceptions in workers In-Reply-To: <1198223031.28189.3.camel@shire> References: <1198223031.28189.3.camel@shire> Message-ID: On Dec 20, 2007 11:43 PM, hemant kumar wrote: > > > On Thu, 2007-12-20 at 23:15 -0800, Joe Van Dyk wrote: > > If an exception is thrown in a worker, will the worker process exit? > > If it does and I don't want it to exit, do I need to rescue all > > exceptions in every worker method? > > > > Every worker is just a Ruby program forked in a new process, so a > unhandled exception WILL cause the worker to exit. When the worker is scheduled to start a new job, a new process will run if the old one stopped, right? From joevandyk at gmail.com Fri Dec 21 04:35:35 2007 From: joevandyk at gmail.com (Joe Van Dyk) Date: Fri, 21 Dec 2007 01:35:35 -0800 Subject: [Backgroundrb-devel] Running backgroundrb on two servers Message-ID: So, if I wanted to have backgroundrb running on two servers, that would just be a matter of creating two different backgroundrb.yml configuration files (one for each server) and starting backgroundrb on both servers, right? From gethemant at gmail.com Fri Dec 21 05:05:50 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 21 Dec 2007 15:35:50 +0530 Subject: [Backgroundrb-devel] Exceptions in workers In-Reply-To: References: <1198223031.28189.3.camel@shire> Message-ID: <1198231550.28189.9.camel@shire> On Thu, 2007-12-20 at 23:49 -0800, Joe Van Dyk wrote: > On Dec 20, 2007 11:43 PM, hemant kumar wrote: > > > > > > On Thu, 2007-12-20 at 23:15 -0800, Joe Van Dyk wrote: > > > If an exception is thrown in a worker, will the worker process exit? > > > If it does and I don't want it to exit, do I need to rescue all > > > exceptions in every worker method? > > > > > > > Every worker is just a Ruby program forked in a new process, so a > > unhandled exception WILL cause the worker to exit. > > When the worker is scheduled to start a new job, a new process will > run if the old one stopped, right? Generally when you are scheduling a worker from backgroundrb.yml file, same worker is reused to handle new jobs. It would be costly to start a new worker for each job. From gethemant at gmail.com Fri Dec 21 05:06:39 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 21 Dec 2007 15:36:39 +0530 Subject: [Backgroundrb-devel] Running backgroundrb on two servers In-Reply-To: References: Message-ID: <1198231599.28189.10.camel@shire> On Fri, 2007-12-21 at 01:35 -0800, Joe Van Dyk wrote: > So, if I wanted to have backgroundrb running on two servers, that > would just be a matter of creating two different backgroundrb.yml > configuration files (one for each server) and starting backgroundrb on > both servers, right? Correct and if you want some kinda synchronization to happen on all nodes, you can use memcache for storing results. From gethemant at gmail.com Fri Dec 21 05:07:04 2007 From: gethemant at gmail.com (hemant kumar) Date: Fri, 21 Dec 2007 15:37:04 +0530 Subject: [Backgroundrb-devel] Jobs running too slow In-Reply-To: References: Message-ID: <1198231624.28189.12.camel@shire> On Thu, 2007-12-20 at 23:17 -0800, Joe Van Dyk wrote: > Say I have a job that runs every minute, but it takes longer than a > minute to execute. Will it start a new job every minute? > > If it does, then how would I make a job that sleeps for a certain > amount of time, then restarts? I have a job that could take an hour > or two to complete, and I don't want to have multiple copies of that > job running at the same time. If your job takes longer than a minute to complete and yet you are scheduling it with an interval of one minute and you don't want it to run concurrently, seems like an mutually exclusive case. :) From joevandyk at gmail.com Fri Dec 21 05:26:07 2007 From: joevandyk at gmail.com (Joe Van Dyk) Date: Fri, 21 Dec 2007 02:26:07 -0800 Subject: [Backgroundrb-devel] Jobs running too slow In-Reply-To: <1198231624.28189.12.camel@shire> References: <1198231624.28189.12.camel@shire> Message-ID: On Dec 21, 2007 2:07 AM, hemant kumar wrote: > On Thu, 2007-12-20 at 23:17 -0800, Joe Van Dyk wrote: > > Say I have a job that runs every minute, but it takes longer than a > > minute to execute. Will it start a new job every minute? > > > > If it does, then how would I make a job that sleeps for a certain > > amount of time, then restarts? I have a job that could take an hour > > or two to complete, and I don't want to have multiple copies of that > > job running at the same time. > > > If your job takes longer than a minute to complete and yet you are > scheduling it with an interval of one minute and you don't want it to > run concurrently, seems like an mutually exclusive case. :) Well, say it normally takes 10 seconds, but then sometimes, due to the internet being really slow, it takes 5 minutes. I don't want to be running multiple jobs in that scenario, I want to wait for the slow job to finish before starting the next one. From jason.lapier at gmail.com Fri Dec 21 11:32:35 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Fri, 21 Dec 2007 08:32:35 -0800 Subject: [Backgroundrb-devel] Jobs running too slow In-Reply-To: References: <1198231624.28189.12.camel@shire> Message-ID: <93b44c1f0712210832y2cd084d9t130c78175610f19e@mail.gmail.com> On Dec 21, 2007 2:26 AM, Joe Van Dyk wrote: > On Dec 21, 2007 2:07 AM, hemant kumar wrote: > > On Thu, 2007-12-20 at 23:17 -0800, Joe Van Dyk wrote: > > > Say I have a job that runs every minute, but it takes longer than a > > > minute to execute. Will it start a new job every minute? > > > > > > If it does, then how would I make a job that sleeps for a certain > > > amount of time, then restarts? I have a job that could take an hour > > > or two to complete, and I don't want to have multiple copies of that > > > job running at the same time. > > > > > > If your job takes longer than a minute to complete and yet you are > > scheduling it with an interval of one minute and you don't want it to > > run concurrently, seems like an mutually exclusive case. :) > > Well, say it normally takes 10 seconds, but then sometimes, due to the > internet being really slow, it takes 5 minutes. I don't want to be > running multiple jobs in that scenario, I want to wait for the slow > job to finish before starting the next one. > Is it a single worker that's always running and periodically kicks off the job? If so, you could set a variable when the job starts - something like: def create @in_job = false end def start_the_job if @in_job return false else @in_job = true # run the job # might be in this part for a while @in_job = false end end - Jason L. -- My Rails and Linux Blog: http://offtheline.net From gobigdave at gmail.com Fri Dec 21 12:14:54 2007 From: gobigdave at gmail.com (Dave Dupre) Date: Fri, 21 Dec 2007 12:14:54 -0500 Subject: [Backgroundrb-devel] Using my models Message-ID: <258cc3f50712210914w65985100h1aa235792f274301@mail.gmail.com> I'm working on upgrading my app to the latest version of backgroundrb. Everything went find until I tried to execute my tasks. Here is my simple worker for testing: class MscWorker < BackgrounDRb::MetaWorker set_worker_name :msc_worker def create(args = nil) # this method is called, when worker is loaded for the first time end # Send a message to everyone def send_message_to_all(args = nil) Message.send_to_everyone(args[:sender], args[:subject], args[:body], false) rescue logger.error "MscWorker - #{err.class}: #{err}" end end Called like so: MiddleMan.ask_work(:worker => :msc_worker, :worker_method => :send_message_to_all, :data => { :sender => sender, :subject => subject, :body => body }) I get lots of errors starting with this: /Users/dave/projects/backgroundrb1.0_msc/vendor/plugins/backgroundrb/server/master_worker.rb:31:in `load': undefined class/module User (ArgumentError) I tried to require the classes in my worker file, and that will make some errors go away, but it looks to want EVERY possible rails class or my model required. Something doesn't seem right here. What do I need to do to use my models? Piston says I'm on revision 293. -- Dave Dupre -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071221/246d647e/attachment.html From jason.lapier at gmail.com Fri Dec 21 12:48:09 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Fri, 21 Dec 2007 09:48:09 -0800 Subject: [Backgroundrb-devel] Using my models In-Reply-To: <258cc3f50712210914w65985100h1aa235792f274301@mail.gmail.com> References: <258cc3f50712210914w65985100h1aa235792f274301@mail.gmail.com> Message-ID: <93b44c1f0712210948h2c64e8baje9a2142b9b390508@mail.gmail.com> On Dec 21, 2007 9:14 AM, Dave Dupre wrote: > I'm working on upgrading my app to the latest version of backgroundrb. > Everything went find until I tried to execute my tasks. > > Here is my simple worker for testing: > > class MscWorker < BackgrounDRb::MetaWorker > set_worker_name :msc_worker > def create(args = nil) > # this method is called, when worker is loaded for the first time > end > # Send a message to everyone > def send_message_to_all(args = nil) > Message.send_to_everyone(args[:sender], args[:subject], args[:body], > false) > rescue > logger.error "MscWorker - #{err.class}: #{err}" > end > end > > Called like so: > MiddleMan.ask_work(:worker => :msc_worker, :worker_method => > :send_message_to_all, :data => { :sender => sender, :subject => subject, > :body => body }) > > I get lots of errors starting with this: > /Users/dave/projects/backgroundrb1.0_msc/vendor/plugins/backgroundrb/server/master_worker.rb:31:in > `load': undefined class/module User (ArgumentError) I'm taking a wild guess here, but I think the problem may be in sending a user object to your worker via MiddleMan. Instead of using "sender", maybe try { :sender_id => sender.id } and in your worker look up the sender (i.e. sender = User.find(sender_id) ). - Jason L. -- My Rails and Linux Blog: http://offtheline.net From gobigdave at gmail.com Fri Dec 21 13:36:40 2007 From: gobigdave at gmail.com (Dave Dupre) Date: Fri, 21 Dec 2007 13:36:40 -0500 Subject: [Backgroundrb-devel] Using my models In-Reply-To: <93b44c1f0712210948h2c64e8baje9a2142b9b390508@mail.gmail.com> References: <258cc3f50712210914w65985100h1aa235792f274301@mail.gmail.com> <93b44c1f0712210948h2c64e8baje9a2142b9b390508@mail.gmail.com> Message-ID: <258cc3f50712211036w7eb7319cgbb3da4a85f13c110@mail.gmail.com> That was it! Now that I think of it, I think I made the same mistake with the old version as well. Thanks! Now I can add the rest of my jobs. Dave On Dec 21, 2007 12:48 PM, Jason LaPier wrote: > On Dec 21, 2007 9:14 AM, Dave Dupre wrote: > > I'm working on upgrading my app to the latest version of backgroundrb. > > Everything went find until I tried to execute my tasks. > > > > Here is my simple worker for testing: > > > > class MscWorker < BackgrounDRb::MetaWorker > > set_worker_name :msc_worker > > def create(args = nil) > > # this method is called, when worker is loaded for the first time > > end > > # Send a message to everyone > > def send_message_to_all(args = nil) > > Message.send_to_everyone(args[:sender], args[:subject], args[:body], > > false) > > rescue > > logger.error "MscWorker - #{err.class}: #{err}" > > end > > end > > > > Called like so: > > MiddleMan.ask_work(:worker => :msc_worker, :worker_method => > > :send_message_to_all, :data => { :sender => sender, :subject => subject, > > :body => body }) > > > > I get lots of errors starting with this: > > > /Users/dave/projects/backgroundrb1.0_msc/vendor/plugins/backgroundrb/server/master_worker.rb:31:in > > `load': undefined class/module User (ArgumentError) > > I'm taking a wild guess here, but I think the problem may be in > sending a user object to your worker via MiddleMan. Instead of using > "sender", maybe try { :sender_id => sender.id } and in your worker > look up the sender (i.e. sender = User.find(sender_id) ). > > - Jason L. > > > > -- > My Rails and Linux Blog: http://offtheline.net > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Dave Dupre (m) 617-921-1684 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071221/95df9562/attachment.html From gobigdave at gmail.com Fri Dec 21 13:48:49 2007 From: gobigdave at gmail.com (Dave Dupre) Date: Fri, 21 Dec 2007 13:48:49 -0500 Subject: [Backgroundrb-devel] Using my models In-Reply-To: <258cc3f50712211036w7eb7319cgbb3da4a85f13c110@mail.gmail.com> References: <258cc3f50712210914w65985100h1aa235792f274301@mail.gmail.com> <93b44c1f0712210948h2c64e8baje9a2142b9b390508@mail.gmail.com> <258cc3f50712211036w7eb7319cgbb3da4a85f13c110@mail.gmail.com> Message-ID: <258cc3f50712211048n6e0c4988p4680d95a1ed63230@mail.gmail.com> Maybe I spoke to soon. Everything works every time, but after the first time I run it, "Some read error" shows up in the command prompt. This is only when called like: script/backrgoundrb There is nothing in the logs that shows an error. Again, the job works every time. Not a big deal since I will not be running it this way in production anyway. Just an FYI... Dave On Dec 21, 2007 1:36 PM, Dave Dupre wrote: > That was it! Now that I think of it, I think I made the same mistake with > the old version as well. > > Thanks! Now I can add the rest of my jobs. > > Dave > > > On Dec 21, 2007 12:48 PM, Jason LaPier wrote: > > > On Dec 21, 2007 9:14 AM, Dave Dupre wrote: > > > I'm working on upgrading my app to the latest version of backgroundrb. > > > > > Everything went find until I tried to execute my tasks. > > > > > > Here is my simple worker for testing: > > > > > > class MscWorker < BackgrounDRb::MetaWorker > > > set_worker_name :msc_worker > > > def create(args = nil) > > > # this method is called, when worker is loaded for the first time > > > end > > > # Send a message to everyone > > > def send_message_to_all(args = nil) > > > Message.send_to_everyone (args[:sender], args[:subject], > > args[:body], > > > false) > > > rescue > > > logger.error "MscWorker - #{err.class}: #{err}" > > > end > > > end > > > > > > Called like so: > > > MiddleMan.ask_work (:worker => :msc_worker, :worker_method => > > > :send_message_to_all, :data => { :sender => sender, :subject => > > subject, > > > :body => body }) > > > > > > I get lots of errors starting with this: > > > > > /Users/dave/projects/backgroundrb1.0_msc/vendor/plugins/backgroundrb/server/master_worker.rb:31:in > > > `load': undefined class/module User (ArgumentError) > > > > I'm taking a wild guess here, but I think the problem may be in > > sending a user object to your worker via MiddleMan. Instead of using > > "sender", maybe try { :sender_id => sender.id } and in your worker > > look up the sender ( i.e. sender = User.find(sender_id) ). > > > > - Jason L. > > > > > > > > -- > > My Rails and Linux Blog: http://offtheline.net > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > -- > Dave Dupre > (m) 617-921-1684 -- Dave Dupre (m) 617-921-1684 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071221/7bf8f785/attachment.html From jason.lapier at gmail.com Fri Dec 21 13:54:37 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Fri, 21 Dec 2007 10:54:37 -0800 Subject: [Backgroundrb-devel] Using my models In-Reply-To: <258cc3f50712211048n6e0c4988p4680d95a1ed63230@mail.gmail.com> References: <258cc3f50712210914w65985100h1aa235792f274301@mail.gmail.com> <93b44c1f0712210948h2c64e8baje9a2142b9b390508@mail.gmail.com> <258cc3f50712211036w7eb7319cgbb3da4a85f13c110@mail.gmail.com> <258cc3f50712211048n6e0c4988p4680d95a1ed63230@mail.gmail.com> Message-ID: <93b44c1f0712211054l36aa080clfd4c7adc67f29755@mail.gmail.com> According to hemant, those "some read errors" are in there for debugging and not actually a fatal error; they just represent opening/closing connection between rails and the worker: http://rubyforge.org/pipermail/backgroundrb-devel/2007-December/001169.html So if it works, you don't have to worry about those errors. - Jason L. On Dec 21, 2007 10:48 AM, Dave Dupre wrote: > Maybe I spoke to soon. Everything works every time, but after the first > time I run it, "Some read error" shows up in the command prompt. This is > only when called like: > > script/backrgoundrb > > There is nothing in the logs that shows an error. Again, the job works > every time. > > Not a big deal since I will not be running it this way in production anyway. > Just an FYI... > > Dave > > > > On Dec 21, 2007 1:36 PM, Dave Dupre < gobigdave at gmail.com> wrote: > > That was it! Now that I think of it, I think I made the same mistake with > the old version as well. > > > > Thanks! Now I can add the rest of my jobs. > > > > Dave > > > > > > > > > > > > On Dec 21, 2007 12:48 PM, Jason LaPier wrote: > > > > > > > > On Dec 21, 2007 9:14 AM, Dave Dupre wrote: > > > > > > > I'm working on upgrading my app to the latest version of backgroundrb. > > > > Everything went find until I tried to execute my tasks. > > > > > > > > Here is my simple worker for testing: > > > > > > > > class MscWorker < BackgrounDRb::MetaWorker > > > > set_worker_name :msc_worker > > > > def create(args = nil) > > > > # this method is called, when worker is loaded for the first time > > > > end > > > > # Send a message to everyone > > > > def send_message_to_all(args = nil) > > > > Message.send_to_everyone (args[:sender], args[:subject], > args[:body], > > > > false) > > > > rescue > > > > logger.error "MscWorker - #{err.class}: #{err}" > > > > end > > > > end > > > > > > > > Called like so: > > > > MiddleMan.ask_work (:worker => :msc_worker, :worker_method => > > > > :send_message_to_all, :data => { :sender => sender, :subject => > subject, > > > > :body => body }) > > > > > > > > I get lots of errors starting with this: > > > > > /Users/dave/projects/backgroundrb1.0_msc/vendor/plugins/backgroundrb/server/master_worker.rb:31:in > > > > `load': undefined class/module User (ArgumentError) > > > > > > > > > I'm taking a wild guess here, but I think the problem may be in > > > sending a user object to your worker via MiddleMan. Instead of using > > > "sender", maybe try { :sender_id => sender.id } and in your worker > > > look up the sender ( i.e. sender = User.find(sender_id) ). > > > > > > - Jason L. > > > > > > > > > > > > -- > > > My Rails and Linux Blog: http://offtheline.net > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > > -- > > Dave Dupre > > (m) 617-921-1684 > > > > -- > Dave Dupre > (m) 617-921-1684 -- My Rails and Linux Blog: http://offtheline.net From gethemant at gmail.com Fri Dec 21 14:39:02 2007 From: gethemant at gmail.com (hemant) Date: Sat, 22 Dec 2007 01:09:02 +0530 Subject: [Backgroundrb-devel] Using my models In-Reply-To: <93b44c1f0712211054l36aa080clfd4c7adc67f29755@mail.gmail.com> References: <258cc3f50712210914w65985100h1aa235792f274301@mail.gmail.com> <93b44c1f0712210948h2c64e8baje9a2142b9b390508@mail.gmail.com> <258cc3f50712211036w7eb7319cgbb3da4a85f13c110@mail.gmail.com> <258cc3f50712211048n6e0c4988p4680d95a1ed63230@mail.gmail.com> <93b44c1f0712211054l36aa080clfd4c7adc67f29755@mail.gmail.com> Message-ID: On Dec 22, 2007 12:24 AM, Jason LaPier wrote: > According to hemant, those "some read errors" are in there for > debugging and not actually a fatal error; they just represent > opening/closing connection between rails and the worker: > http://rubyforge.org/pipermail/backgroundrb-devel/2007-December/001169.html > Yes, don't worry about the log message. It has been removed from trunk anyways. Also, if you sync with trunk you now have the capability to pass model objects directly to BackgrounDRb workers. Its still not advisable, since model objects are generally complex objects and they contain class hierarchies, and marhalling them across wire and constructing them back at other end is tricky business. But you have the capability so use with care, only pass simple models around if you must. There are other important fixes as well, so upgrade now. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From kenglish at camberhawaii.org Fri Dec 21 14:51:27 2007 From: kenglish at camberhawaii.org (Kevin W. English) Date: Fri, 21 Dec 2007 09:51:27 -1000 Subject: [Backgroundrb-devel] what happened to get_worker? In-Reply-To: References: <258cc3f50712210914w65985100h1aa235792f274301@mail.gmail.com> <93b44c1f0712210948h2c64e8baje9a2142b9b390508@mail.gmail.com> <258cc3f50712211036w7eb7319cgbb3da4a85f13c110@mail.gmail.com> <258cc3f50712211048n6e0c4988p4680d95a1ed63230@mail.gmail.com> <93b44c1f0712211054l36aa080clfd4c7adc67f29755@mail.gmail.com> Message-ID: <476C193F.2050807@camberhawaii.org> Hello, I recently upgraded to the new backgroundrb. I noticed that the "get_worker" method is gone. I am trying to create multiple instances of the same worker and retrieve the status of them using the job key: To create the worker, I do: @worker = MiddleMan.new_worker(:worker => :bar_worker, :worker_method => :echo_value, :job_key => @job_key, :data => {:param1 => 'val1',:param2 => 'val2' }) To retrieve the worker, I have tried: @worker = MiddleMan.ask_status(:worker => :bar_worker, :job_key => @job_key) but it does not work. I used to be able to do this with get_worker. Can someone explain how to use this feature with the new version? Kevin English From gethemant at gmail.com Fri Dec 21 15:03:19 2007 From: gethemant at gmail.com (hemant kumar) Date: Sat, 22 Dec 2007 01:33:19 +0530 Subject: [Backgroundrb-devel] what happened to get_worker? In-Reply-To: <476C193F.2050807@camberhawaii.org> References: <258cc3f50712210914w65985100h1aa235792f274301@mail.gmail.com> <93b44c1f0712210948h2c64e8baje9a2142b9b390508@mail.gmail.com> <258cc3f50712211036w7eb7319cgbb3da4a85f13c110@mail.gmail.com> <258cc3f50712211048n6e0c4988p4680d95a1ed63230@mail.gmail.com> <93b44c1f0712211054l36aa080clfd4c7adc67f29755@mail.gmail.com> <476C193F.2050807@camberhawaii.org> Message-ID: <1198267399.28189.41.camel@shire> On Fri, 2007-12-21 at 09:51 -1000, Kevin W. English wrote: > Hello, I recently upgraded to the new backgroundrb. I noticed that the > "get_worker" method is gone. I am trying to create multiple instances of > the same worker and retrieve the status of them using the job key: > > To create the worker, I do: > > @worker = MiddleMan.new_worker(:worker => :bar_worker, > :worker_method => :echo_value, > :job_key => @job_key, > :data => {:param1 => 'val1',:param2 => 'val2' }) > > > To retrieve the worker, I have tried: > > @worker = MiddleMan.ask_status(:worker => :bar_worker, :job_key => > @job_key) > > but it does not work. I used to be able to do this with get_worker. Can > someone explain how to use this feature with the new version? > What happens, you get an error? Ok, You must register status of your worker before able to retrieve it using ask_status. By default, all workers start with a status of nil. So, make sure that in worker code, you are using register_status() to register current status of worker. From kenglish at camberhawaii.org Fri Dec 21 22:47:31 2007 From: kenglish at camberhawaii.org (Kevin W. English) Date: Fri, 21 Dec 2007 17:47:31 -1000 Subject: [Backgroundrb-devel] [Fwd: Re: what happened to get_worker?] Message-ID: <476C88D3.7080809@camberhawaii.org> Thank you. I got it working using the register_status . hemant kumar wrote: > On Fri, 2007-12-21 at 09:51 -1000, Kevin W. English wrote: > >> Hello, I recently upgraded to the new backgroundrb. I noticed that the >> "get_worker" method is gone. I am trying to create multiple instances of >> the same worker and retrieve the status of them using the job key: >> >> To create the worker, I do: >> >> @worker = MiddleMan.new_worker(:worker => :bar_worker, >> :worker_method => :echo_value, >> :job_key => @job_key, >> :data => {:param1 => 'val1',:param2 => 'val2' }) >> >> >> To retrieve the worker, I have tried: >> >> @worker = MiddleMan.ask_status(:worker => :bar_worker, :job_key => >> @job_key) >> >> but it does not work. I used to be able to do this with get_worker. Can >> someone explain how to use this feature with the new version? >> >> > > What happens, you get an error? > Ok, You must register status of your worker before able to retrieve it > using ask_status. By default, all workers start with a status of nil. > > So, make sure that in worker code, you are using register_status() to > register current status of worker. > > > -- Aloha, Kevin English Rails Evangelist Software Development and Integration Team Camber Corporation Honolulu, HI Office: 808.237.3539 Fax: 808.237.3540 -- Aloha, Kevin English Rails Evangelist Software Development and Integration Team Camber Corporation Honolulu, HI Office: 808.237.3539 Fax: 808.237.3540 From gethemant at gmail.com Sat Dec 22 08:30:41 2007 From: gethemant at gmail.com (hemant) Date: Sat, 22 Dec 2007 19:00:41 +0530 Subject: [Backgroundrb-devel] Implementation details for worker Queue Message-ID: Hi All, One can easily code a backgroundrb worker so as it can act as a worker queue, having reached a stable status, perhaps its time we should add this feature. Why? Because if you are already running your application using BackgrounDRb, perhaps you don't want to manage other solutions when capability to do that is right there with you. Interface? This where i need feedback. You can already dynamically start a worker and submit a job to it ( using MiddleMan.new_worker) and it would be done in a new process. But its not a queue, it starts executing your task immediately. Sometimes I doubt if its seriously required, because you can easily queue your tasks within a worker using inbuilt thread pool ( if you don't want to spawn new worker ). What then? Perhaps what we CAN do is, have a specialized worker that can act on a priority queue. This priority queue is nothing but a database table and depending upon the number of tasks in the queue, master process automatically starts new workers or deletes them as need be.Our specialized worker, when picks a task from db, it marks that task as taken and starts processing them. The control to start new workers depending upon size of queue can be left either to master or the "specialized worker" itself. I was thinking on lines of having a ratio as in lets say per 10 queued tasks you want one worker, then you can specify that in configuration file. If the number of tasks goes higher either "specialized worker" or master starts a new instance of "specialized worker". There could be a upper limit on number of parallel workers running and if on one machine this limit is reached backgroundrb can request master to start workers on another machine. How does this sound? Feedbacks? -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From jim.scott at yahoo.com Sat Dec 22 11:10:44 2007 From: jim.scott at yahoo.com (Jim Scott) Date: Sat, 22 Dec 2007 09:10:44 -0700 Subject: [Backgroundrb-devel] Implementation details for worker Queue In-Reply-To: References: Message-ID: <003a01c844b5$35610820$6501a8c0@LIGHTFOOT> Hemant - Strangely enough, you have exactly described how we use backgroundrb. We queue jobs to a table, and BackgroundRb processes them. We have specialized workers for each row type. When we have higher volumes of work, we use Xen virtualization to stand up new utility servers that process queued jobs. We can scale this solution horizontally, much in the same we that we scale application servers. I for one vote YES. Jim Scott Plan Source -----Original Message----- From: backgroundrb-devel-bounces at rubyforge.org [mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of hemant Sent: Saturday, December 22, 2007 6:31 AM To: backgroundrb-devel Subject: [Backgroundrb-devel] Implementation details for worker Queue Hi All, One can easily code a backgroundrb worker so as it can act as a worker queue, having reached a stable status, perhaps its time we should add this feature. Why? Because if you are already running your application using BackgrounDRb, perhaps you don't want to manage other solutions when capability to do that is right there with you. Interface? This where i need feedback. You can already dynamically start a worker and submit a job to it ( using MiddleMan.new_worker) and it would be done in a new process. But its not a queue, it starts executing your task immediately. Sometimes I doubt if its seriously required, because you can easily queue your tasks within a worker using inbuilt thread pool ( if you don't want to spawn new worker ). What then? Perhaps what we CAN do is, have a specialized worker that can act on a priority queue. This priority queue is nothing but a database table and depending upon the number of tasks in the queue, master process automatically starts new workers or deletes them as need be.Our specialized worker, when picks a task from db, it marks that task as taken and starts processing them. The control to start new workers depending upon size of queue can be left either to master or the "specialized worker" itself. I was thinking on lines of having a ratio as in lets say per 10 queued tasks you want one worker, then you can specify that in configuration file. If the number of tasks goes higher either "specialized worker" or master starts a new instance of "specialized worker". There could be a upper limit on number of parallel workers running and if on one machine this limit is reached backgroundrb can request master to start workers on another machine. How does this sound? Feedbacks? -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org _______________________________________________ Backgroundrb-devel mailing list Backgroundrb-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/backgroundrb-devel No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.17.6/1192 - Release Date: 12/21/2007 1:17 PM No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.17.6/1192 - Release Date: 12/21/2007 1:17 PM From faivrem at gmail.com Sat Dec 22 13:26:02 2007 From: faivrem at gmail.com (Mickael Faivre-Macon) Date: Sat, 22 Dec 2007 19:26:02 +0100 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: References: Message-ID: Thanks Hemant, it works. How can I change the default port 11009 ? It is taken on my shared server. I have changed it in the bdrb config file, but how I tell my workers to connect to this new port ? Mickael. On Dec 17, 2007 10:15 AM, hemant wrote: > > On Dec 16, 2007 3:38 AM, Mickael Faivre-Macon wrote: > > Hello ! > > > > 1) I have this error logged by bdrb: > Hi Michale, > > There was a genuine bug with backgroundrb in job_key handling. If you > don't want to use threads and would rather want to have processes, you > can try your code now. > > It should be fixed and working. From gethemant at gmail.com Sat Dec 22 14:48:47 2007 From: gethemant at gmail.com (hemant kumar) Date: Sun, 23 Dec 2007 01:18:47 +0530 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: References: Message-ID: <1198352927.6367.1.camel@shire> Hi On Sat, 2007-12-22 at 19:26 +0100, Mickael Faivre-Macon wrote: > Thanks Hemant, it works. > > How can I change the default port 11009 ? > It is taken on my shared server. > > I have changed it in the bdrb config file, but how I tell my workers > to connect to this new port ? Restarting the backgroundrb should fix it and all the workers should be automatically aware of new port, there is nothing much to do. Let me know, if it doesn't work. Run the server in foreground mode as described in docs for getting debugging information. From joevandyk at gmail.com Sat Dec 22 20:59:13 2007 From: joevandyk at gmail.com (Joe Van Dyk) Date: Sat, 22 Dec 2007 17:59:13 -0800 Subject: [Backgroundrb-devel] Implementation details for worker Queue In-Reply-To: References: Message-ID: On Dec 22, 2007 5:30 AM, hemant wrote: > Hi All, > > One can easily code a backgroundrb worker so as it can act as a worker > queue, having reached a stable status, perhaps its time we should add > this feature. > > Why? > Because if you are already running your application using > BackgrounDRb, perhaps you don't want to manage other solutions when > capability to do that is right there with you. Sort of related question: If I submit multiple jobs to a single worker (i.e. from multiple Rails processes), does the worker do one of those jobs at a time? Or does it two both of them at the same time? Joe From faivrem at gmail.com Sun Dec 23 04:54:02 2007 From: faivrem at gmail.com (Mickael Faivre-Macon) Date: Sun, 23 Dec 2007 10:54:02 +0100 Subject: [Backgroundrb-devel] 1) Error 2) sending request to a specific worker In-Reply-To: <1198352927.6367.1.camel@shire> References: <1198352927.6367.1.camel@shire> Message-ID: Oh yes, I had to restart my Rails app also, of course. Thanks ! On Dec 22, 2007 8:48 PM, hemant kumar wrote: > Hi > On Sat, 2007-12-22 at 19:26 +0100, Mickael Faivre-Macon wrote: > > Thanks Hemant, it works. > > > > How can I change the default port 11009 ? > > It is taken on my shared server. > > > > I have changed it in the bdrb config file, but how I tell my workers > > to connect to this new port ? > > Restarting the backgroundrb should fix it and all the workers should be > automatically aware of new port, there is nothing much to do. > > Let me know, if it doesn't work. Run the server in foreground mode as > described in docs for getting debugging information. > > > -- Mickael. Coding an AI ! http://faivrem.googlepages.com/antbattle From gethemant at gmail.com Sun Dec 23 07:42:55 2007 From: gethemant at gmail.com (hemant) Date: Sun, 23 Dec 2007 18:12:55 +0530 Subject: [Backgroundrb-devel] Updates and Fixes and more of awesome stuff Message-ID: 1. Whenever you ask BackgrounDRb to perform a particular task using: MiddleMan.ask_work(:worker => :rss_worker,:worker_method => :fetch_url, :data => "www.google.com") Your task would be performed then and there. But, if your worker is busy, task won't be queued, except in socket buffer. And your request may get lost. This isn't generally a problem with tasks that is being scheduled through scheduler, but problem happens when you are pusing lots of tasks to worker and you don't want them to run concurrently since that may hog CPU resources. You would rather want tasks to be queued and be processed one after another easily. The simple solution for such tasks is to set thread pool size for worker to 1 and use thread_pool.defer { do_something } class RssWorker < BackgrounDRb::MetaWorker set_worker_name :rss_worker pool_size(1) end Now no matter how many requests you send to the worker, they all will be queued in a thread pool, but since thread pool is of size 1, only one of the task would run at a time. You don't have to worry with thread issues, because only one thread will at a time anyways. 2. I have a method called MiddleMan.worker_info, which can be used like this: >> MiddleMan.worker_info(:worker => :rss_worker) => {:status=>:running, :worker=>:rss_worker, :job_key=>nil} You can pass some job_key also to check whether a worker is running or not. If a worker is not running, :status value will be :stopped. 3. Removed all modifications to core classes. BackgrounDRb no longer messes with any of the core classes. Poor thing has everything in its own namespace. :) It will fix any issues with ActiveSupport and other such libraries. Give it a spin and we can release it as 1.0.1 version. Thanks. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gobigdave at gmail.com Sun Dec 23 09:54:35 2007 From: gobigdave at gmail.com (Dave Dupre) Date: Sun, 23 Dec 2007 09:54:35 -0500 Subject: [Backgroundrb-devel] Minor warnings since 296 Message-ID: <258cc3f50712230654t691f5407ma787d5713548fe5c@mail.gmail.com> Since moving to r296, I get warnings (" warning: already initialized constant...") for many of my class constants. This did not happen before, and I do not get the same warnings when loading my application normally. They are only warnings, so everything still runs, but I thought I would provide the FYI. -- Dave Dupre -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071223/b6d49cd6/attachment.html From gethemant at gmail.com Sun Dec 23 10:39:06 2007 From: gethemant at gmail.com (hemant) Date: Sun, 23 Dec 2007 21:09:06 +0530 Subject: [Backgroundrb-devel] Minor warnings since 296 In-Reply-To: <258cc3f50712230654t691f5407ma787d5713548fe5c@mail.gmail.com> References: <258cc3f50712230654t691f5407ma787d5713548fe5c@mail.gmail.com> Message-ID: On Dec 23, 2007 8:24 PM, Dave Dupre wrote: > Since moving to r296, I get warnings (" warning: already initialized > constant...") for many of my class constants. This did not happen before, > and I do not get the same warnings when loading my application normally. > They are only warnings, so everything still runs, but I thought I would > provide the FYI. > I think that would be because of loading of all the model classes in master process. Thanks for the information. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Sun Dec 23 10:40:31 2007 From: gethemant at gmail.com (hemant) Date: Sun, 23 Dec 2007 21:10:31 +0530 Subject: [Backgroundrb-devel] Multiple copies of same worker In-Reply-To: <4766914D.4030404@bestwaytech.com> References: <76341dbf0712170617i1dfa7000j4aa9bb77e108f481@mail.gmail.com> <47668A47.4070208@bestwaytech.com> <4766914D.4030404@bestwaytech.com> Message-ID: Hi On Dec 17, 2007 8:40 PM, Danila Ulyanov wrote: > Is there any special reason that creating worker does not return a > unique job_key itself upon creation like it used to in the old > backgroundrb? :-) It does return that now. > > Would there be much overhead creating/destroying up to 8 workers versus > having 8 workers running continuously and just reusing them? > > Thanks! > > > hemant wrote: > > Hi > > > >> On Dec 17, 2007 3:40 PM, Danila Ulyanov wrote: > >>> Hello! > >>> > >>> I remember there was a thread sometime last month about multiple copies > >>> of the same worker. Is that still impossible to do? > >>> > >>> What I am trying to accomplish is to limit number of concurrent > >>> connections for a lengthy search. I used to put search requests in a > >>> queue, then as slots freed up, they spawned off a new copy of the > >>> "search worker" which killed itself upon completion. That way no more > >>> than 8 searches were conducted at the same time. > >>> > >>> Will it be possible to do this in the new version or is there an > >>> alternative solution to this problem? > >>> > > > > Yes, you can have multiple copies of same worker running. But there > > are two catches: > > > > 1. You have to specify a unique job_key for each worker, when you are > > making a call to a start worker. > > 2. To start a worker, you will have to use > > MiddleMan.new_worker(:worker => :foo_worker, :job_key => "whoa_man") > > > > To exit a worker, once its done with search, simply call "exit" on them. > > > > > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From jason.lapier at gmail.com Sun Dec 23 23:26:53 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Sun, 23 Dec 2007 20:26:53 -0800 Subject: [Backgroundrb-devel] backgroundrb script crashing on startup Message-ID: <93b44c1f0712232026j3f729e63u38cdc8686bebb403@mail.gmail.com> Everything works fine on my development box (Ubuntu 7.10, Ruby 1.8.6) but on my server (Fedora Core 6, Ruby 1.8.5) when I run script/backgroundrb start, the process fails to start. This is with Rails 2.0.2 frozen in the app, and I have the most recent version of the plugin (r297). Is Ruby 1.8.6 a requirement for backgroundrb now? I can't figure any other reason it would fail... backgroundrb_server.log shows: ==> log/backgroundrb_server.log <== /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:264:in `load_missing_constant': uninitialized constant ActionView::Helpers::BasicObject (NameError) from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:453:in `const_missing' from /var/www/apps/rpg/releases/20071224034514/vendor/rails/actionpack/lib/action_view/helpers/prototype_helper.rb:1068 from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require' from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require' from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:496:in `require' from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:343:in `new_constants_in' from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:496:in `require' from /var/www/apps/rpg/releases/20071224034514/vendor/rails/actionpack/lib/action_view/helpers/javascript_helper.rb:2 ... 44 levels... from /var/www/apps/rpg/releases/20071224034514/vendor/plugins/backgroundrb/server/master_worker.rb:156:in `load_rails_env' from /var/www/apps/rpg/releases/20071224034514/vendor/plugins/backgroundrb/server/master_worker.rb:143:in `initialize' from script/backgroundrb:39:in `new' from script/backgroundrb:39 - Jason L. -- My Rails and Linux Blog: http://offtheline.net From gethemant at gmail.com Sun Dec 23 23:49:48 2007 From: gethemant at gmail.com (hemant) Date: Mon, 24 Dec 2007 10:19:48 +0530 Subject: [Backgroundrb-devel] backgroundrb script crashing on startup In-Reply-To: <93b44c1f0712232026j3f729e63u38cdc8686bebb403@mail.gmail.com> References: <93b44c1f0712232026j3f729e63u38cdc8686bebb403@mail.gmail.com> Message-ID: On Dec 24, 2007 9:56 AM, Jason LaPier wrote: > Everything works fine on my development box (Ubuntu 7.10, Ruby 1.8.6) > but on my server (Fedora Core 6, Ruby 1.8.5) when I run > script/backgroundrb start, the process fails to start. > This is with Rails 2.0.2 frozen in the app, and I have the most recent > version of the plugin (r297). Is Ruby 1.8.6 a requirement for > backgroundrb now? I can't figure any other reason it would fail... No it shouldn't be. > > backgroundrb_server.log shows: > > ==> log/backgroundrb_server.log <== > > /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:264:in > `load_missing_constant': uninitialized constant > ActionView::Helpers::BasicObject (NameError) > > from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:453:in > `const_missing' > > from /var/www/apps/rpg/releases/20071224034514/vendor/rails/actionpack/lib/action_view/helpers/prototype_helper.rb:1068 > > from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in > `gem_original_require' > > from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in > `require' > > from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:496:in > `require' > > from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:343:in > `new_constants_in' > > from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:496:in > `require' > > from /var/www/apps/rpg/releases/20071224034514/vendor/rails/actionpack/lib/action_view/helpers/javascript_helper.rb:2 > > ... 44 levels... > > from /var/www/apps/rpg/releases/20071224034514/vendor/plugins/backgroundrb/server/master_worker.rb:156:in > `load_rails_env' > > from /var/www/apps/rpg/releases/20071224034514/vendor/plugins/backgroundrb/server/master_worker.rb:143:in > `initialize' > > from script/backgroundrb:39:in `new' > > from script/backgroundrb:39 > > > I am looking into this now. Can you please sync with latest code and see if it fixes the issue. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From jason.lapier at gmail.com Mon Dec 24 00:10:29 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Sun, 23 Dec 2007 21:10:29 -0800 Subject: [Backgroundrb-devel] backgroundrb script crashing on startup In-Reply-To: References: <93b44c1f0712232026j3f729e63u38cdc8686bebb403@mail.gmail.com> Message-ID: <93b44c1f0712232110r34eabb32gd7d7a1af49b98b43@mail.gmail.com> On Dec 23, 2007 8:49 PM, hemant wrote: > On Dec 24, 2007 9:56 AM, Jason LaPier wrote: > > Everything works fine on my development box (Ubuntu 7.10, Ruby 1.8.6) > > but on my server (Fedora Core 6, Ruby 1.8.5) when I run > > script/backgroundrb start, the process fails to start. > > This is with Rails 2.0.2 frozen in the app, and I have the most recent > > version of the plugin (r297). Is Ruby 1.8.6 a requirement for > > backgroundrb now? I can't figure any other reason it would fail... > > No it shouldn't be. > > > > > > backgroundrb_server.log shows: > > > > ==> log/backgroundrb_server.log <== > > > > /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:264:in > > `load_missing_constant': uninitialized constant > > ActionView::Helpers::BasicObject (NameError) > > > > from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:453:in > > `const_missing' > > > > from /var/www/apps/rpg/releases/20071224034514/vendor/rails/actionpack/lib/action_view/helpers/prototype_helper.rb:1068 > > > > from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in > > `gem_original_require' > > > > from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in > > `require' > > > > from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:496:in > > `require' > > > > from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:343:in > > `new_constants_in' > > > > from /usr/lib/ruby/site_ruby/1.8/active_support/dependencies.rb:496:in > > `require' > > > > from /var/www/apps/rpg/releases/20071224034514/vendor/rails/actionpack/lib/action_view/helpers/javascript_helper.rb:2 > > > > ... 44 levels... > > > > from /var/www/apps/rpg/releases/20071224034514/vendor/plugins/backgroundrb/server/master_worker.rb:156:in > > `load_rails_env' > > > > from /var/www/apps/rpg/releases/20071224034514/vendor/plugins/backgroundrb/server/master_worker.rb:143:in > > `initialize' > > > > from script/backgroundrb:39:in `new' > > > > from script/backgroundrb:39 > > > > > > > > I am looking into this now. Can you please sync with latest code and > see if it fixes the issue. > That did the trick - working fine now. Thanks again for another quick fix! - Jason L. -- My Rails and Linux Blog: http://offtheline.net From gethemant at gmail.com Mon Dec 24 08:00:22 2007 From: gethemant at gmail.com (hemant) Date: Mon, 24 Dec 2007 18:30:22 +0530 Subject: [Backgroundrb-devel] Minor warnings since 296 In-Reply-To: References: <258cc3f50712230654t691f5407ma787d5713548fe5c@mail.gmail.com> Message-ID: On Dec 23, 2007 9:09 PM, hemant wrote: > > On Dec 23, 2007 8:24 PM, Dave Dupre wrote: > > Since moving to r296, I get warnings (" warning: already initialized > > constant...") for many of my class constants. This did not happen before, > > and I do not get the same warnings when loading my application normally. > > They are only warnings, so everything still runs, but I thought I would > > provide the FYI. > > Hi Dave, Please update with latest code and let me know if you are still getting those warnings. I am unable to reproduce this at my end and hence unable to fix. I think, this is the last annoyance left, till we can release 1.0.1 version. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From jason.lapier at gmail.com Mon Dec 24 20:32:07 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Mon, 24 Dec 2007 17:32:07 -0800 Subject: [Backgroundrb-devel] error: no marshal_dump is defined for class Thread Message-ID: <93b44c1f0712241732i2dbb37a9qa8cbf9570d96a068@mail.gmail.com> I get this error periodically in backgroundrb.log, but only on my production box (the one running ruby 1.8.5) - my dev box doesn't get this error. As far as I can tell, everything is working ok, despite the error (I don't really know how to determine if the jobs are getting threaded successfully, but they are running, that I am sure of). Here's the dump: no marshal_dump is defined for class Thread /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in `dump' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in `dump_object' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:32:in `send_data' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:291:in `send_response' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:224:in `process_request' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:202:in `receive_data' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:54:in `receive_internal_data' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in `call' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in `extract' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:52:in `receive_internal_data' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:48:in `handle_internal_messages' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:158:in `start_reactor' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in `each' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in `start_reactor' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in `loop' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in `start_reactor' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:21:in `start_worker' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in `fork_and_load' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in `load_workers' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in `each' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in `load_workers' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in `run' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in `initialize' script/backgroundrb:41:in `new' script/backgroundrb:41 - Jason L. -- My Rails and Linux Blog: http://offtheline.net From gethemant at gmail.com Tue Dec 25 00:29:16 2007 From: gethemant at gmail.com (hemant kumar) Date: Tue, 25 Dec 2007 10:59:16 +0530 Subject: [Backgroundrb-devel] error: no marshal_dump is defined for class Thread In-Reply-To: <93b44c1f0712241732i2dbb37a9qa8cbf9570d96a068@mail.gmail.com> References: <93b44c1f0712241732i2dbb37a9qa8cbf9570d96a068@mail.gmail.com> Message-ID: <1198560556.6337.25.camel@shire> On Mon, 2007-12-24 at 17:32 -0800, Jason LaPier wrote: > I get this error periodically in backgroundrb.log, but only on my > production box (the one running ruby 1.8.5) - my dev box doesn't get > this error. As far as I can tell, everything is working ok, despite > the error (I don't really know how to determine if the jobs are > getting threaded successfully, but they are running, that I am sure > of). > > > Here's the dump: > > no marshal_dump is defined for class Thread > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > `dump' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > `dump_object' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:32:in > `send_data' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:291:in > `send_response' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:224:in > `process_request' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:202:in > `receive_data' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:54:in > `receive_internal_data' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > `call' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > `extract' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:52:in > `receive_internal_data' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:48:in > `handle_internal_messages' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:158:in > `start_reactor' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > `each' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > `start_reactor' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > `loop' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > `start_reactor' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:21:in > `start_worker' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in > `fork_and_load' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in > `load_workers' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > `each' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > `load_workers' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > `run' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > `initialize' > > script/backgroundrb:41:in `new' > > script/backgroundrb:41 Looks your code is trying to dump threads, somewhere! I will have a look..anyways. And if jobs are running means, there are getting threaded successfully. From joost at yelloyello.com Thu Dec 27 03:28:43 2007 From: joost at yelloyello.com (Joost Hietbrink (YelloYello)) Date: Thu, 27 Dec 2007 09:28:43 +0100 Subject: [Backgroundrb-devel] Lost connection.. In-Reply-To: <47735CBF.8010106@yelloyello.com> References: <47735CBF.8010106@yelloyello.com> Message-ID: <4773623B.5000005@yelloyello.com> Short update: It seems that the workers go into some state in which they do not have a correct Rails environment (or db connection). When they enter this state it goes wrong all the time :( Joost Hietbrink (YelloYello) wrote: > Hi all, > > We're experiencing some problems with Backgroundrb. > > Why is "# master_reactor_instance.result_hash = {}" commented in > packet_master.rb? BackgroundRb only seems to work when it is enabled? > Sometimes we get a "Mysql::Error: Lost connection to MySQL server > during query" error. This in the Backgroundrb job and occurs only half > of the time or something. It seems that the Rails environment isn't > correct?! or something.. > > Let me know if you have any suggestions! Thanks! > > Joost > From gethemant at gmail.com Thu Dec 27 04:11:35 2007 From: gethemant at gmail.com (hemant) Date: Thu, 27 Dec 2007 14:41:35 +0530 Subject: [Backgroundrb-devel] Lost connection.. In-Reply-To: <4773623B.5000005@yelloyello.com> References: <47735CBF.8010106@yelloyello.com> <4773623B.5000005@yelloyello.com> Message-ID: On Dec 27, 2007 1:58 PM, Joost Hietbrink (YelloYello) wrote: > Short update: > It seems that the workers go into some state in which they do not have a > correct Rails environment (or db connection). When they enter this state > it goes wrong all the time :( There doesn't seem to be any problem with problem with RAILS environment in my application. But first make sure you are running C version of mysql gem. > > Joost Hietbrink (YelloYello) wrote: > > Hi all, > > > > We're experiencing some problems with Backgroundrb. > > > > Why is "# master_reactor_instance.result_hash = {}" commented in > > packet_master.rb? BackgroundRb only seems to work when it is enabled? > > Sometimes we get a "Mysql::Error: Lost connection to MySQL server > > during query" error. This in the Backgroundrb job and occurs only half > > of the time or something. It seems that the Rails environment isn't > > correct?! or something.. > > > > Let me know if you have any suggestions! Thanks! > > > > Joost > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Thu Dec 27 04:13:11 2007 From: gethemant at gmail.com (hemant) Date: Thu, 27 Dec 2007 14:43:11 +0530 Subject: [Backgroundrb-devel] error: no marshal_dump is defined for class Thread In-Reply-To: <1198560556.6337.25.camel@shire> References: <93b44c1f0712241732i2dbb37a9qa8cbf9570d96a068@mail.gmail.com> <1198560556.6337.25.camel@shire> Message-ID: On Dec 25, 2007 10:59 AM, hemant kumar wrote: > > > On Mon, 2007-12-24 at 17:32 -0800, Jason LaPier wrote: > > I get this error periodically in backgroundrb.log, but only on my > > production box (the one running ruby 1.8.5) - my dev box doesn't get > > this error. As far as I can tell, everything is working ok, despite > > the error (I don't really know how to determine if the jobs are > > getting threaded successfully, but they are running, that I am sure > > of). > > > > > > Here's the dump: > > > > no marshal_dump is defined for class Thread > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > `dump' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > `dump_object' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:32:in > > `send_data' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:291:in > > `send_response' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:224:in > > `process_request' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:202:in > > `receive_data' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:54:in > > `receive_internal_data' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > `call' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > `extract' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:52:in > > `receive_internal_data' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:48:in > > `handle_internal_messages' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:158:in > > `start_reactor' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > `each' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > `start_reactor' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > `loop' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > `start_reactor' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:21:in > > `start_worker' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in > > `fork_and_load' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in > > `load_workers' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > `each' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > `load_workers' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > > `run' > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > > `initialize' > > > > script/backgroundrb:41:in `new' > > > > script/backgroundrb:41 > > Looks your code is trying to dump threads, somewhere! > > I will have a look..anyways. And if jobs are running means, there are > getting threaded successfully. Jason, did you debug this further? Why you are getting those errors logged? Can you send me worker code, if possible? -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From joost at yelloyello.com Thu Dec 27 04:47:57 2007 From: joost at yelloyello.com (Joost Hietbrink) Date: Thu, 27 Dec 2007 10:47:57 +0100 Subject: [Backgroundrb-devel] Lost connection.. In-Reply-To: References: <47735CBF.8010106@yelloyello.com> <4773623B.5000005@yelloyello.com> Message-ID: <477374CD.4000603@yelloyello.com> I see we haven't got the mysql gem installed on the production server. So that may cause the problem? Is it required for backgroundrb?! Joost hemant wrote: > On Dec 27, 2007 1:58 PM, Joost Hietbrink (YelloYello) > wrote: > >> Short update: >> It seems that the workers go into some state in which they do not have a >> correct Rails environment (or db connection). When they enter this state >> it goes wrong all the time :( >> > > There doesn't seem to be any problem with problem with RAILS > environment in my application. > But first make sure you are running C version of mysql gem. > > >> Joost Hietbrink (YelloYello) wrote: >> >>> Hi all, >>> >>> We're experiencing some problems with Backgroundrb. >>> >>> Why is "# master_reactor_instance.result_hash = {}" commented in >>> packet_master.rb? BackgroundRb only seems to work when it is enabled? >>> Sometimes we get a "Mysql::Error: Lost connection to MySQL server >>> during query" error. This in the Backgroundrb job and occurs only half >>> of the time or something. It seems that the Rails environment isn't >>> correct?! or something.. >>> >>> Let me know if you have any suggestions! Thanks! >>> >>> Joost >>> >>> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071227/2874f051/attachment.html From gethemant at gmail.com Thu Dec 27 05:03:31 2007 From: gethemant at gmail.com (hemant) Date: Thu, 27 Dec 2007 15:33:31 +0530 Subject: [Backgroundrb-devel] Lost connection.. In-Reply-To: <477374CD.4000603@yelloyello.com> References: <47735CBF.8010106@yelloyello.com> <4773623B.5000005@yelloyello.com> <477374CD.4000603@yelloyello.com> Message-ID: On Dec 27, 2007 3:17 PM, Joost Hietbrink wrote: > > I see we haven't got the mysql gem installed on the production server. So > that may cause the problem? Is it required for backgroundrb?! If you are using your models, then surely you need mysql gem. By default rails comes with pure ruby mysql adapter, which is known to have problems with database connectivity. Generally, moving to C based gem solves the problem. > > Joost > > > hemant wrote: > On Dec 27, 2007 1:58 PM, Joost Hietbrink (YelloYello) > wrote: > > > Short update: > It seems that the workers go into some state in which they do not have a > correct Rails environment (or db connection). When they enter this state > it goes wrong all the time :( > > There doesn't seem to be any problem with problem with RAILS > environment in my application. > But first make sure you are running C version of mysql gem. > > > > Joost Hietbrink (YelloYello) wrote: > > > Hi all, > > We're experiencing some problems with Backgroundrb. > > Why is "# master_reactor_instance.result_hash = {}" commented in > packet_master.rb? BackgroundRb only seems to work when it is enabled? > Sometimes we get a "Mysql::Error: Lost connection to MySQL server > during query" error. This in the Backgroundrb job and occurs only half > of the time or something. It seems that the Rails environment isn't > correct?! or something.. > > Let me know if you have any suggestions! Thanks! > > Joost > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From joost at yelloyello.com Thu Dec 27 07:28:04 2007 From: joost at yelloyello.com (Joost Hietbrink) Date: Thu, 27 Dec 2007 13:28:04 +0100 Subject: [Backgroundrb-devel] Lost connection.. In-Reply-To: References: <47735CBF.8010106@yelloyello.com> <4773623B.5000005@yelloyello.com> <477374CD.4000603@yelloyello.com> Message-ID: <47739A54.1000603@yelloyello.com> Thanks! This fixes the problem! I have another error. When using the trunk version of backgroundrb and when I have "require RAILS_HOME + '/config/boot.rb'" set in script/backgroundrb I get errors like: /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/base.rb:1238:in `method_missing': undefined method `attr_readonly' So I removed that line..(like in the 1.0 version).. and everything works perfectly. Greets, Joost hemant wrote: > On Dec 27, 2007 3:17 PM, Joost Hietbrink wrote: > >> I see we haven't got the mysql gem installed on the production server. So >> that may cause the problem? Is it required for backgroundrb?! >> > > If you are using your models, then surely you need mysql gem. By > default rails comes with pure ruby mysql adapter, which is known to > have problems with database connectivity. > > Generally, moving to C based gem solves the problem. > > >> Joost >> >> >> hemant wrote: >> On Dec 27, 2007 1:58 PM, Joost Hietbrink (YelloYello) >> wrote: >> >> >> Short update: >> It seems that the workers go into some state in which they do not have a >> correct Rails environment (or db connection). When they enter this state >> it goes wrong all the time :( >> >> There doesn't seem to be any problem with problem with RAILS >> environment in my application. >> But first make sure you are running C version of mysql gem. >> >> >> >> Joost Hietbrink (YelloYello) wrote: >> >> >> Hi all, >> >> We're experiencing some problems with Backgroundrb. >> >> Why is "# master_reactor_instance.result_hash = {}" commented in >> packet_master.rb? BackgroundRb only seems to work when it is enabled? >> Sometimes we get a "Mysql::Error: Lost connection to MySQL server >> during query" error. This in the Backgroundrb job and occurs only half >> of the time or something. It seems that the Rails environment isn't >> correct?! or something.. >> >> Let me know if you have any suggestions! Thanks! >> >> Joost >> >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> >> >> >> >> >> >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071227/67029934/attachment.html From jason.lapier at gmail.com Thu Dec 27 15:04:08 2007 From: jason.lapier at gmail.com (Jason LaPier) Date: Thu, 27 Dec 2007 12:04:08 -0800 Subject: [Backgroundrb-devel] error: no marshal_dump is defined for class Thread In-Reply-To: References: <93b44c1f0712241732i2dbb37a9qa8cbf9570d96a068@mail.gmail.com> <1198560556.6337.25.camel@shire> Message-ID: <93b44c1f0712271204v99cde9m197aff6c17000f2c@mail.gmail.com> On Dec 27, 2007 1:13 AM, hemant wrote: > > On Dec 25, 2007 10:59 AM, hemant kumar wrote: > > > > > > On Mon, 2007-12-24 at 17:32 -0800, Jason LaPier wrote: > > > I get this error periodically in backgroundrb.log, but only on my > > > production box (the one running ruby 1.8.5) - my dev box doesn't get > > > this error. As far as I can tell, everything is working ok, despite > > > the error (I don't really know how to determine if the jobs are > > > getting threaded successfully, but they are running, that I am sure > > > of). > > > > > > > > > Here's the dump: > > > > > > no marshal_dump is defined for class Thread > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > > `dump' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > > `dump_object' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:32:in > > > `send_data' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:291:in > > > `send_response' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:224:in > > > `process_request' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:202:in > > > `receive_data' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:54:in > > > `receive_internal_data' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > `call' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > `extract' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:52:in > > > `receive_internal_data' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:48:in > > > `handle_internal_messages' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:158:in > > > `start_reactor' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > `each' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > `start_reactor' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > `loop' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > `start_reactor' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:21:in > > > `start_worker' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in > > > `fork_and_load' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in > > > `load_workers' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > `each' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > `load_workers' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > > > `run' > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > > > `initialize' > > > > > > script/backgroundrb:41:in `new' > > > > > > script/backgroundrb:41 > > > > Looks your code is trying to dump threads, somewhere! > > > > I will have a look..anyways. And if jobs are running means, there are > > getting threaded successfully. > > Jason, did you debug this further? Why you are getting those errors > logged? Can you send me worker code, if possible? > Sorry, hemant - I didn't track down the problem. It appeared that mongrel processes were hanging on me, and these seemed to coincide with those errors, but I can't be 100% sure. This was a Fedora Core 6 box and the repositories only have ruby 1.8.5, so in desperation I yanked it out and compiled Ruby 1.8.6 and reinstalled all the gems. The Thread error is gone now - it could be the updated ruby, or it could be that when I reinstalled I got a newer version of mongrel than I had before. I am however still getting occasional hung mongrels. Here are some errors popping up in the logs: --background_server.log-- "production" /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:70:in `add_thread': undefined local variable or method `logger' for # (NameError) from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in `initialize' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in `new' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in `add_thread' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in `initialize' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in `times' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in `initialize' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:169:in `new' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:169:in `worker_init' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:20:in `start_worker' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in `fork_and_load' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in `load_workers' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in `each' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in `load_workers' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in `run' from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in `initialize' from script/backgroundrb:41:in `new' from script/backgroundrb:41 --background_debug.log-- 000000087{ : type: do_work: datai:worker_method:build_battle: worker:battle_builder_worker {:type=>:do_work, :worker=>:battle_builder_worker, :data=>2, :worker_method=>:build_battle} undefined method `send_request' for nil:NilClass /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:43:in `ask_worker' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:87:in `process_work' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:35:in `receive_data' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in `call' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in `extract' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:31:in `receive_data' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:199:in `read_external_socket' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:191:in `handle_external_messages' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:160:in `start_reactor' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in `each' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in `start_reactor' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in `loop' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in `start_reactor' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:20:in `run' /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in `initialize' script/backgroundrb:41:in `new' script/backgroundrb:41 When this happens, backgroundrb seems to be working (I see the requests in the log) but jobs never execute. I have to stop backgroundrb and restart it. From what I can tell, it doesn't hang all the mongrels, just one of the four I have running. The first error, it's obviously trying to log an exception but can't find logger. The second error looks like it can't find the worker (getting Nil instead of the worker). I wasn't checking for the existence of the worker before I call MiddleMan.ask_work because it's supposed to be running at all times (maybe I need to do that anyway for troubleshooting purposes). I'll keep poking around and see if I can figure anything out. The problem is very intermittent - it ran fine for several hours and the job in the worker was called at least 20 times; so I thought the upgrade to ruby 1.8.6 and/or mongrel 1.1.2 had taken care of it, but then I got those errors above today and had to restart backgroundrb. - Jason L. -- My Rails and Linux Blog: http://offtheline.net From gethemant at gmail.com Thu Dec 27 15:14:58 2007 From: gethemant at gmail.com (hemant) Date: Fri, 28 Dec 2007 01:44:58 +0530 Subject: [Backgroundrb-devel] error: no marshal_dump is defined for class Thread In-Reply-To: <93b44c1f0712271204v99cde9m197aff6c17000f2c@mail.gmail.com> References: <93b44c1f0712241732i2dbb37a9qa8cbf9570d96a068@mail.gmail.com> <1198560556.6337.25.camel@shire> <93b44c1f0712271204v99cde9m197aff6c17000f2c@mail.gmail.com> Message-ID: On Dec 28, 2007 1:34 AM, Jason LaPier wrote: > > On Dec 27, 2007 1:13 AM, hemant wrote: > > > > On Dec 25, 2007 10:59 AM, hemant kumar wrote: > > > > > > > > > On Mon, 2007-12-24 at 17:32 -0800, Jason LaPier wrote: > > > > I get this error periodically in backgroundrb.log, but only on my > > > > production box (the one running ruby 1.8.5) - my dev box doesn't get > > > > this error. As far as I can tell, everything is working ok, despite > > > > the error (I don't really know how to determine if the jobs are > > > > getting threaded successfully, but they are running, that I am sure > > > > of). > > > > > > > > > > > > Here's the dump: > > > > > > > > no marshal_dump is defined for class Thread > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > > > `dump' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > > > `dump_object' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:32:in > > > > `send_data' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:291:in > > > > `send_response' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:224:in > > > > `process_request' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:202:in > > > > `receive_data' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:54:in > > > > `receive_internal_data' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > > `call' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > > `extract' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:52:in > > > > `receive_internal_data' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:48:in > > > > `handle_internal_messages' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:158:in > > > > `start_reactor' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > > `each' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > > `start_reactor' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > > `loop' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > > `start_reactor' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:21:in > > > > `start_worker' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in > > > > `fork_and_load' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in > > > > `load_workers' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > > `each' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > > `load_workers' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > > > > `run' > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > > > > `initialize' > > > > > > > > script/backgroundrb:41:in `new' > > > > > > > > script/backgroundrb:41 > > > > > > Looks your code is trying to dump threads, somewhere! > > > > > > I will have a look..anyways. And if jobs are running means, there are > > > getting threaded successfully. > > > > Jason, did you debug this further? Why you are getting those errors > > logged? Can you send me worker code, if possible? > > > > Sorry, hemant - I didn't track down the problem. It appeared that > mongrel processes were hanging on me, and these seemed to coincide > with those errors, but I can't be 100% sure. This was a Fedora Core 6 > box and the repositories only have ruby 1.8.5, so in desperation I > yanked it out and compiled Ruby 1.8.6 and reinstalled all the gems. > The Thread error is gone now - it could be the updated ruby, or it > could be that when I reinstalled I got a newer version of mongrel than > I had before. > > I am however still getting occasional hung mongrels. Here are some > errors popping up in the logs: > > --background_server.log-- > "production" > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:70:in > `add_thread': undefined local variable or method `logger' for > # (NameError) > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in > `initialize' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in > `new' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in > `add_thread' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in > `initialize' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in > `times' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in > `initialize' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:169:in > `new' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:169:in > `worker_init' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:20:in > `start_worker' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in > `fork_and_load' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in > `load_workers' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > `each' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > `load_workers' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > `run' > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > `initialize' > from script/backgroundrb:41:in `new' > from script/backgroundrb:41 > > --background_debug.log-- > 000000087{ : type: > do_work: datai: worker_method: build_battle: > > worker: battle_builder_worker > {:type=>:do_work, :worker=>:battle_builder_worker, :data=>2, > :worker_method=>:build_battle} > undefined method `send_request' for nil:NilClass > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:43:in > `ask_worker' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:87:in > `process_work' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:35:in > `receive_data' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > `call' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > `extract' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:31:in > `receive_data' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:199:in > `read_external_socket' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:191:in > `handle_external_messages' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:160:in > `start_reactor' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > `each' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > `start_reactor' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > `loop' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > `start_reactor' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:20:in > `run' > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > `initialize' > script/backgroundrb:41:in `new' > script/backgroundrb:41 > > When this happens, backgroundrb seems to be working (I see the > requests in the log) but jobs never execute. I have to stop > backgroundrb and restart it. From what I can tell, it doesn't hang all > the mongrels, just one of the four I have running. > > The first error, it's obviously trying to log an exception but can't > find logger. The second error looks like it can't find the worker > (getting Nil instead of the worker). I wasn't checking for the > existence of the worker before I call MiddleMan.ask_work because it's > supposed to be running at all times (maybe I need to do that anyway > for troubleshooting purposes). > > I'll keep poking around and see if I can figure anything out. The > problem is very intermittent - it ran fine for several hours and the > job in the worker was called at least 20 times; so I thought the > upgrade to ruby 1.8.6 and/or mongrel 1.1.2 had taken care of it, but > then I got those errors above today and had to restart backgroundrb. > > This is awesome. Lets fix it. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From walter at katipo.co.nz Thu Dec 27 23:20:20 2007 From: walter at katipo.co.nz (Walter McGinnis) Date: Fri, 28 Dec 2007 17:20:20 +1300 Subject: [Backgroundrb-devel] Arity? Message-ID: Hello, First off, nice improvements hemant! Thanks for all the hard work. I just whipped up a new worker for a scheduled job that sends digest emails. Very smooth. However, moving some of my old backgroundrb into new backgroundrb isn't going quite as smoothly. I have a import worker that takes a number of parameters and then is started dynamically. Here's the a snippet of the worker code: class PastPerfect4ImporterWorker < BackgrounDRb::MetaWorker set_worker_name :past_perfect4_importer_worker set_no_auto_load true # importer has the version of methods that will work in the context # of backgroundrb include Importer def create(args = nil) @successful = false @last_related_topic = nil @last_related_topic_pp4_objectid = nil @import_field_to_extended_field_map = Hash.new @attributions = Hash.new logger.info('PastPerfect4ImporterWorker started') @results = { :do_work_time => Time.now.to_s, :done_with_do_work => false, :records_processed => 0 } register_status(@results) end def do_work(args = nil) begin logger.debug("in worker") ... I get to the create method's logger message, but not the first do_work logger call. Here's the "ask_work" call: MiddleMan.new_worker(:worker => @worker_type, :job_key => :importer) MiddleMan.ask_work(:worker => @worker_type, :job_key => :importer, :worker_method => :do_work, :data => { :zoom_class => @zoom_class, :import_topic_type => @import_topic_type, :import_type => @import_type, :import_dir_path => @import_dir_path, :import_parent_dir_for_image_dirs => @import_parent_dir_for_image_dirs, :params => params, :import_request => import_request, :default_attribution => @default_attribution, :contributing_user => @contributing_user.id }) I know that @worker_type is set correctly because of the log. And in the log I get: ... PastPerfect4ImporterWorker started past_perfect4_importer_worker started Schedules for worker loaded do_work {:zoom_class=>"StillImage", ... :default_attribution=>"Some jerk wrote this.", :import_type=>"past_perfect4"} Arity of method is -1 That seems to be the end of the road for the worker. It just stalls there from what I can tell. What does Arity mean? Any clues? I suspect it's something small I'm missing. Also, I replaced this on line 210 of server/meta_worker.rb: logger.info "#{user_input[:worker_method]} #{user_input[:data]}" with this: logger.info "#{user_input[:worker_method]} #{user_input[:data].inspect}" for nicer debugging formatting. Seems like it might be a nice improvement. Cheers, Walter McGinnis --- Kete Project Lead Katipo Communications, Ltd. http://katipo.co.nz/ http://blog.katipo.co.nz/ http://kete.net.nz/ From walter at katipo.co.nz Fri Dec 28 00:20:31 2007 From: walter at katipo.co.nz (Walter McGinnis) Date: Fri, 28 Dec 2007 18:20:31 +1300 Subject: [Backgroundrb-devel] Arity? In-Reply-To: References: Message-ID: <02041D42-5AA2-41EE-B698-833755C46500@katipo.co.nz> Looks to be an issue with use of logger.debug, replaced that with logger.info and things came right (well mostly). Looking better. Cheers, Walter From gethemant at gmail.com Fri Dec 28 02:09:25 2007 From: gethemant at gmail.com (hemant) Date: Fri, 28 Dec 2007 12:39:25 +0530 Subject: [Backgroundrb-devel] error: no marshal_dump is defined for class Thread In-Reply-To: References: <93b44c1f0712241732i2dbb37a9qa8cbf9570d96a068@mail.gmail.com> <1198560556.6337.25.camel@shire> <93b44c1f0712271204v99cde9m197aff6c17000f2c@mail.gmail.com> Message-ID: On Dec 28, 2007 1:44 AM, hemant wrote: > > On Dec 28, 2007 1:34 AM, Jason LaPier wrote: > > > > On Dec 27, 2007 1:13 AM, hemant wrote: > > > > > > On Dec 25, 2007 10:59 AM, hemant kumar wrote: > > > > > > > > > > > > On Mon, 2007-12-24 at 17:32 -0800, Jason LaPier wrote: > > > > > I get this error periodically in backgroundrb.log, but only on my > > > > > production box (the one running ruby 1.8.5) - my dev box doesn't get > > > > > this error. As far as I can tell, everything is working ok, despite > > > > > the error (I don't really know how to determine if the jobs are > > > > > getting threaded successfully, but they are running, that I am sure > > > > > of). > > > > > > > > > > > > > > > Here's the dump: > > > > > > > > > > no marshal_dump is defined for class Thread > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > > > > `dump' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > > > > `dump_object' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:32:in > > > > > `send_data' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:291:in > > > > > `send_response' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:224:in > > > > > `process_request' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:202:in > > > > > `receive_data' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:54:in > > > > > `receive_internal_data' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > > > `call' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > > > `extract' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:52:in > > > > > `receive_internal_data' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:48:in > > > > > `handle_internal_messages' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:158:in > > > > > `start_reactor' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > > > `each' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > > > `start_reactor' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > > > `loop' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > > > `start_reactor' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:21:in > > > > > `start_worker' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in > > > > > `fork_and_load' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in > > > > > `load_workers' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > > > `each' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > > > `load_workers' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > > > > > `run' > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > > > > > `initialize' > > > > > > > > > > script/backgroundrb:41:in `new' > > > > > > > > > > script/backgroundrb:41 > > > > > > > > Looks your code is trying to dump threads, somewhere! > > > > > > > > I will have a look..anyways. And if jobs are running means, there are > > > > getting threaded successfully. > > > > > > Jason, did you debug this further? Why you are getting those errors > > > logged? Can you send me worker code, if possible? > > > > > > > Sorry, hemant - I didn't track down the problem. It appeared that > > mongrel processes were hanging on me, and these seemed to coincide > > with those errors, but I can't be 100% sure. This was a Fedora Core 6 > > box and the repositories only have ruby 1.8.5, so in desperation I > > yanked it out and compiled Ruby 1.8.6 and reinstalled all the gems. > > The Thread error is gone now - it could be the updated ruby, or it > > could be that when I reinstalled I got a newer version of mongrel than > > I had before. > > > > I am however still getting occasional hung mongrels. Here are some > > errors popping up in the logs: > > > > --background_server.log-- > > "production" > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:70:in > > `add_thread': undefined local variable or method `logger' for > > # (NameError) > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in > > `initialize' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in > > `new' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in > > `add_thread' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in > > `initialize' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in > > `times' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in > > `initialize' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:169:in > > `new' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:169:in > > `worker_init' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:20:in > > `start_worker' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in > > `fork_and_load' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in > > `load_workers' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > `each' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > `load_workers' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > > `run' > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > > `initialize' > > from script/backgroundrb:41:in `new' > > from script/backgroundrb:41 > > > > --background_debug.log-- > > 000000087{ : type: > > do_work: datai: worker_method: build_battle: > > > > worker: battle_builder_worker > > {:type=>:do_work, :worker=>:battle_builder_worker, :data=>2, > > :worker_method=>:build_battle} > > undefined method `send_request' for nil:NilClass > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:43:in > > `ask_worker' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:87:in > > `process_work' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:35:in > > `receive_data' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > `call' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > `extract' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:31:in > > `receive_data' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:199:in > > `read_external_socket' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:191:in > > `handle_external_messages' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:160:in > > `start_reactor' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > `each' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > `start_reactor' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > `loop' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > `start_reactor' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:20:in > > `run' > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > > `initialize' > > script/backgroundrb:41:in `new' > > script/backgroundrb:41 > > > > When this happens, backgroundrb seems to be working (I see the > > requests in the log) but jobs never execute. I have to stop > > backgroundrb and restart it. From what I can tell, it doesn't hang all > > the mongrels, just one of the four I have running. > > > > The first error, it's obviously trying to log an exception but can't > > find logger. The second error looks like it can't find the worker > > (getting Nil instead of the worker). I wasn't checking for the > > existence of the worker before I call MiddleMan.ask_work because it's > > supposed to be running at all times (maybe I need to do that anyway > > for troubleshooting purposes). > > > > I'll keep poking around and see if I can figure anything out. The > > problem is very intermittent - it ran fine for several hours and the > > job in the worker was called at least 20 times; so I thought the > > upgrade to ruby 1.8.6 and/or mongrel 1.1.2 had taken care of it, but > > then I got those errors above today and had to restart backgroundrb. > > > > > > This is awesome. Lets fix it. Can you send me your worker code Jason? -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Fri Dec 28 02:10:41 2007 From: gethemant at gmail.com (hemant) Date: Fri, 28 Dec 2007 12:40:41 +0530 Subject: [Backgroundrb-devel] Arity? In-Reply-To: <02041D42-5AA2-41EE-B698-833755C46500@katipo.co.nz> References: <02041D42-5AA2-41EE-B698-833755C46500@katipo.co.nz> Message-ID: On Dec 28, 2007 10:50 AM, Walter McGinnis wrote: > Looks to be an issue with use of logger.debug, replaced that with > logger.info and things came right (well mostly). Looking better. > Thats cool. File a ticket abour debug thing perhaps, it has already skipped one full bug fixing cycle. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From joost at yelloyello.com Fri Dec 28 02:57:35 2007 From: joost at yelloyello.com (Joost Hietbrink (YelloYello)) Date: Fri, 28 Dec 2007 08:57:35 +0100 Subject: [Backgroundrb-devel] lost connection 2.0 Message-ID: <4774AC6F.3030302@yelloyello.com> I lost my "connection lost" error.. but now I've got a "Mysql::Error: MySQL server has gone away" error.. It comes from: /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/connection_adapters/abstract_adapter.rb:128:in 'log'. I'm not sure if Rails / BackgroundRb uses it .. but i've got the mysql (2.7) gem installed. The problem does not occur in Rails itself.. and also not in BackgroundRb in the beginning (only after a while). Any ideas? From gethemant at gmail.com Fri Dec 28 03:22:09 2007 From: gethemant at gmail.com (hemant) Date: Fri, 28 Dec 2007 13:52:09 +0530 Subject: [Backgroundrb-devel] lost connection 2.0 In-Reply-To: <4774AC6F.3030302@yelloyello.com> References: <4774AC6F.3030302@yelloyello.com> Message-ID: On Dec 28, 2007 1:27 PM, Joost Hietbrink (YelloYello) wrote: > I lost my "connection lost" error.. but now I've got a "Mysql::Error: > MySQL server has gone away" error.. > It comes from: > > /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/connection_adapters/abstract_adapter.rb:128:in 'log'. Complete backtrace please. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From joost at yelloyello.com Fri Dec 28 03:24:32 2007 From: joost at yelloyello.com (Joost Hietbrink (YelloYello)) Date: Fri, 28 Dec 2007 09:24:32 +0100 Subject: [Backgroundrb-devel] lost connection 2.0 In-Reply-To: References: <4774AC6F.3030302@yelloyello.com> Message-ID: <4774B2C0.5040608@yelloyello.com> Here you are :) A ActiveRecord::StatementInvalid occurred in IndexingWorker#refetch_record: Mysql::Error: MySQL server has gone away: SELECT * FROM places WHERE (places.`id` = 616752) /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/connection_adapters/abstract_adapter.rb:128:in `log' ------------------------------- Request: ------------------------------- * URL: N/A * Parameters: [] * Rails root: /var/home/releases/yy/20071227135247 ------------------------------- Session: ------------------------------- * session id: "N/A" * data: "N/A" ------------------------------- Environment: ------------------------------- * Process: 10327 * Server : yy ------------------------------- Backtrace: ------------------------------- /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/connection_adapters/abstract_adapter.rb:128:in `log' /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/connection_adapters/mysql_adapter.rb:243:in `execute' /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/connection_adapters/mysql_adapter.rb:399:in `select' /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all' /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/base.rb:427:in `find_by_sql' /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/base.rb:997:in `find_every' /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/base.rb:1028:in `find_one' /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/base.rb:1014:in `find_from_ids' /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/base.rb:419:in `find' [RAILS_ROOT]/vendor/plugins/backgroundrb_extensions/lib/rails_model_worker.rb:86:in `refetch_record' [RAILS_ROOT]/vendor/plugins/backgroundrb_extensions/lib/rails_model_worker.rb:84:in `each' [RAILS_ROOT]/vendor/plugins/backgroundrb_extensions/lib/rails_model_worker.rb:84:in `refetch_record' [RAILS_ROOT]/vendor/plugins/backgroundrb_extensions/lib/rails_model_worker.rb:59:in `request_work' [RAILS_ROOT]/vendor/plugins/backgroundrb/server/meta_worker.rb:219:in `send' [RAILS_ROOT]/vendor/plugins/backgroundrb/server/meta_worker.rb:219:in `process_request' [RAILS_ROOT]/vendor/plugins/backgroundrb/server/meta_worker.rb:202:in `receive_data' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/worker.rb:54:in `receive_internal_data' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/bin_parser.rb:29:in `call' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/bin_parser.rb:29:in `extract' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/worker.rb:52:in `receive_internal_data' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/worker.rb:48:in `handle_internal_messages' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:158:in `start_reactor' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:156:in `each' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:156:in `start_reactor' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:147:in `loop' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:147:in `start_reactor' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/worker.rb:21:in `start_worker' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in `fork_and_load' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/packet_master.rb:115:in `start_worker' (__FORWARDABLE__):3:in `__send__' (__FORWARDABLE__):3:in `start_worker' [RAILS_ROOT]/vendor/plugins/backgroundrb/server/master_worker.rb:78:in `start_worker_request' [RAILS_ROOT]/vendor/plugins/backgroundrb/server/master_worker.rb:38:in `receive_data' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/bin_parser.rb:29:in `call' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/bin_parser.rb:29:in `extract' [RAILS_ROOT]/vendor/plugins/backgroundrb/server/master_worker.rb:31:in `receive_data' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:199:in `read_external_socket' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:191:in `handle_external_messages' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:160:in `start_reactor' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:156:in `each' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:156:in `start_reactor' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:147:in `loop' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/core.rb:147:in `start_reactor' [RAILS_ROOT]/vendor/plugins/backgroundrb/framework/packet_master.rb:20:in `run' [RAILS_ROOT]/vendor/plugins/backgroundrb/server/master_worker.rb:144:in `initialize' script/backgroundrb:41:in `new' script/backgroundrb:41 hemant wrote: > On Dec 28, 2007 1:27 PM, Joost Hietbrink (YelloYello) > wrote: > >> I lost my "connection lost" error.. but now I've got a "Mysql::Error: >> MySQL server has gone away" error.. >> It comes from: >> >> /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/connection_adapters/abstract_adapter.rb:128:in 'log'. >> > > Complete backtrace please. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071228/1b6e190a/attachment.html From joost at yelloyello.com Fri Dec 28 03:29:21 2007 From: joost at yelloyello.com (Joost Hietbrink (YelloYello)) Date: Fri, 28 Dec 2007 09:29:21 +0100 Subject: [Backgroundrb-devel] lost connection 2.0 In-Reply-To: References: <4774AC6F.3030302@yelloyello.com> Message-ID: <4774B3E1.4010908@yelloyello.com> I read here (http://rubyforge.org/pipermail/backgroundrb-devel/2006-July/000126.html) (on the old backgroundrb) that I should/could set: ActiveRecord::Base.allow_concurrency = true Is this also relevant for the new one? Since it doesn't use threads?! hemant wrote: > On Dec 28, 2007 1:27 PM, Joost Hietbrink (YelloYello) > wrote: > >> I lost my "connection lost" error.. but now I've got a "Mysql::Error: >> MySQL server has gone away" error.. >> It comes from: >> >> /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.6/lib/active_record/connection_adapters/abstract_adapter.rb:128:in 'log'. >> > > Complete backtrace please. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071228/f74d824e/attachment.html From gethemant at gmail.com Fri Dec 28 03:33:28 2007 From: gethemant at gmail.com (hemant) Date: Fri, 28 Dec 2007 14:03:28 +0530 Subject: [Backgroundrb-devel] lost connection 2.0 In-Reply-To: <4774B3E1.4010908@yelloyello.com> References: <4774AC6F.3030302@yelloyello.com> <4774B3E1.4010908@yelloyello.com> Message-ID: On Dec 28, 2007 1:59 PM, Joost Hietbrink (YelloYello) wrote: > > I read here > (http://rubyforge.org/pipermail/backgroundrb-devel/2006-July/000126.html) > (on the old backgroundrb) that I should/could set: > > ActiveRecord::Base.allow_concurrency = true > Hmm, what are those [RAILS_ROOT]/vendor/plugins/backgroundrb_extensions/lib/rails_model_worker.rb things? Looks like exception is being thrown in rails code, rather than worker code, isn't it? -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Fri Dec 28 04:05:31 2007 From: gethemant at gmail.com (hemant) Date: Fri, 28 Dec 2007 14:35:31 +0530 Subject: [Backgroundrb-devel] error: no marshal_dump is defined for class Thread In-Reply-To: References: <93b44c1f0712241732i2dbb37a9qa8cbf9570d96a068@mail.gmail.com> <1198560556.6337.25.camel@shire> <93b44c1f0712271204v99cde9m197aff6c17000f2c@mail.gmail.com> Message-ID: Ok Jason, This problem is fixed now. Sync with latest code. The problem was because, your code inside defer is throwing an exception. BackgrounDRb tries to log that exception, but there wasn't any logger object available in thread pool. Now, in production, if your code throws an exception, it will be logged in backgroundrb.log file. Watch out for exceptions anyways. Thanks. On Dec 28, 2007 12:39 PM, hemant wrote: > > On Dec 28, 2007 1:44 AM, hemant wrote: > > > > On Dec 28, 2007 1:34 AM, Jason LaPier wrote: > > > > > > On Dec 27, 2007 1:13 AM, hemant wrote: > > > > > > > > On Dec 25, 2007 10:59 AM, hemant kumar wrote: > > > > > > > > > > > > > > > On Mon, 2007-12-24 at 17:32 -0800, Jason LaPier wrote: > > > > > > I get this error periodically in backgroundrb.log, but only on my > > > > > > production box (the one running ruby 1.8.5) - my dev box doesn't get > > > > > > this error. As far as I can tell, everything is working ok, despite > > > > > > the error (I don't really know how to determine if the jobs are > > > > > > getting threaded successfully, but they are running, that I am sure > > > > > > of). > > > > > > > > > > > > > > > > > > Here's the dump: > > > > > > > > > > > > no marshal_dump is defined for class Thread > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > > > > > `dump' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:55:in > > > > > > `dump_object' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:32:in > > > > > > `send_data' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:291:in > > > > > > `send_response' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:224:in > > > > > > `process_request' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:202:in > > > > > > `receive_data' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:54:in > > > > > > `receive_internal_data' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > > > > `call' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > > > > `extract' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:52:in > > > > > > `receive_internal_data' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:48:in > > > > > > `handle_internal_messages' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:158:in > > > > > > `start_reactor' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > > > > `each' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > > > > `start_reactor' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > > > > `loop' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > > > > `start_reactor' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:21:in > > > > > > `start_worker' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in > > > > > > `fork_and_load' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in > > > > > > `load_workers' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > > > > `each' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > > > > `load_workers' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > > > > > > `run' > > > > > > > > > > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > > > > > > `initialize' > > > > > > > > > > > > script/backgroundrb:41:in `new' > > > > > > > > > > > > script/backgroundrb:41 > > > > > > > > > > Looks your code is trying to dump threads, somewhere! > > > > > > > > > > I will have a look..anyways. And if jobs are running means, there are > > > > > getting threaded successfully. > > > > > > > > Jason, did you debug this further? Why you are getting those errors > > > > logged? Can you send me worker code, if possible? > > > > > > > > > > Sorry, hemant - I didn't track down the problem. It appeared that > > > mongrel processes were hanging on me, and these seemed to coincide > > > with those errors, but I can't be 100% sure. This was a Fedora Core 6 > > > box and the repositories only have ruby 1.8.5, so in desperation I > > > yanked it out and compiled Ruby 1.8.6 and reinstalled all the gems. > > > The Thread error is gone now - it could be the updated ruby, or it > > > could be that when I reinstalled I got a newer version of mongrel than > > > I had before. > > > > > > I am however still getting occasional hung mongrels. Here are some > > > errors popping up in the logs: > > > > > > --background_server.log-- > > > "production" > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:70:in > > > `add_thread': undefined local variable or method `logger' for > > > # (NameError) > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in > > > `initialize' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in > > > `new' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:62:in > > > `add_thread' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in > > > `initialize' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in > > > `times' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:34:in > > > `initialize' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:169:in > > > `new' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/meta_worker.rb:169:in > > > `worker_init' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/worker.rb:20:in > > > `start_worker' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:134:in > > > `fork_and_load' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:98:in > > > `load_workers' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > `each' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:93:in > > > `load_workers' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:19:in > > > `run' > > > from /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > > > `initialize' > > > from script/backgroundrb:41:in `new' > > > from script/backgroundrb:41 > > > > > > --background_debug.log-- > > > 000000087{ : type: > > > do_work: datai: worker_method: build_battle: > > > > > > worker: battle_builder_worker > > > {:type=>:do_work, :worker=>:battle_builder_worker, :data=>2, > > > :worker_method=>:build_battle} > > > undefined method `send_request' for nil:NilClass > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:43:in > > > `ask_worker' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:87:in > > > `process_work' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:35:in > > > `receive_data' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > `call' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in > > > `extract' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:31:in > > > `receive_data' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:199:in > > > `read_external_socket' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:191:in > > > `handle_external_messages' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:160:in > > > `start_reactor' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > `each' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:156:in > > > `start_reactor' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > `loop' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/core.rb:147:in > > > `start_reactor' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/framework/packet_master.rb:20:in > > > `run' > > > /var/www/apps/rpg/releases/20071224050255/vendor/plugins/backgroundrb/server/master_worker.rb:144:in > > > `initialize' > > > script/backgroundrb:41:in `new' > > > script/backgroundrb:41 > > > > > > When this happens, backgroundrb seems to be working (I see the > > > requests in the log) but jobs never execute. I have to stop > > > backgroundrb and restart it. From what I can tell, it doesn't hang all > > > the mongrels, just one of the four I have running. > > > > > > The first error, it's obviously trying to log an exception but can't > > > find logger. The second error looks like it can't find the worker > > > (getting Nil instead of the worker). I wasn't checking for the > > > existence of the worker before I call MiddleMan.ask_work because it's > > > supposed to be running at all times (maybe I need to do that anyway > > > for troubleshooting purposes). > > > > > > I'll keep poking around and see if I can figure anything out. The > > > problem is very intermittent - it ran fine for several hours and the > > > job in the worker was called at least 20 times; so I thought the > > > upgrade to ruby 1.8.6 and/or mongrel 1.1.2 had taken care of it, but > > > then I got those errors above today and had to restart backgroundrb. > > > > > > > > > > This is awesome. Lets fix it. > > Can you send me your worker code Jason? > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From joost at yelloyello.com Fri Dec 28 06:55:18 2007 From: joost at yelloyello.com (Joost Hietbrink (YelloYello)) Date: Fri, 28 Dec 2007 12:55:18 +0100 Subject: [Backgroundrb-devel] lost connection 2.0 In-Reply-To: References: <4774AC6F.3030302@yelloyello.com> <4774B3E1.4010908@yelloyello.com> Message-ID: <4774E426.4030901@yelloyello.com> The rails_model_worker.rb contains a module that fetches an object from the database. We pass parameters to the worker (like classname & id (eg. Place and 12312)) and fetch the object again in the worker (using this module method). The lost connection occurs when we try to fetch the object (record) again (in the worker..using the included module method). The old BackgroundRb did not support sending ActiveRecord objects to the worker (thats why we refetch it)..does the new version?? We now also installed libmysql-ruby1.8 (see: http://wiki.rubyonrails.org/rails/pages/Mysql%2BConnection%2BProblems) so maybe this fixes the problem..but we have to wait some time to see.. :) Thanks! hemant wrote: > On Dec 28, 2007 1:59 PM, Joost Hietbrink (YelloYello) > wrote: > >> I read here >> (http://rubyforge.org/pipermail/backgroundrb-devel/2006-July/000126.html) >> (on the old backgroundrb) that I should/could set: >> >> ActiveRecord::Base.allow_concurrency = true >> >> > > Hmm, what are those > [RAILS_ROOT]/vendor/plugins/backgroundrb_extensions/lib/rails_model_worker.rb > things? > > Looks like exception is being thrown in rails code, rather than worker > code, isn't it? > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071228/02d45946/attachment.html From gethemant at gmail.com Sat Dec 29 11:09:41 2007 From: gethemant at gmail.com (hemant) Date: Sat, 29 Dec 2007 21:39:41 +0530 Subject: [Backgroundrb-devel] lost connection 2.0 In-Reply-To: <4774E426.4030901@yelloyello.com> References: <4774AC6F.3030302@yelloyello.com> <4774B3E1.4010908@yelloyello.com> <4774E426.4030901@yelloyello.com> Message-ID: Hi Joost, On Dec 28, 2007 5:25 PM, Joost Hietbrink (YelloYello) wrote: > > The rails_model_worker.rb contains a module that fetches an object from the > database. > We pass parameters to the worker (like classname & id (eg. Place and > 12312)) and fetch the object again in the worker (using this module method). > > The lost connection occurs when we try to fetch the object (record) again > (in the worker..using the included module method). > > The old BackgroundRb did not support sending ActiveRecord objects to the > worker (thats why we refetch it)..does the new version?? > > We now also installed libmysql-ruby1.8 (see: > http://wiki.rubyonrails.org/rails/pages/Mysql%2BConnection%2BProblems) so > maybe this fixes the problem..but we have to wait some time to see.. :) > Sorry for late reply. But If you can send me the worker code, I will try to fix it. There seems to be some problems with AR in threaded environment. I just want to make sure, none remain unfixed. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From joost at yelloyello.com Sat Dec 29 15:59:15 2007 From: joost at yelloyello.com (Joost Hietbrink (YelloYello)) Date: Sat, 29 Dec 2007 21:59:15 +0100 Subject: [Backgroundrb-devel] lost connection 2.0 Message-ID: <4776B523.9010207@yelloyello.com> I mailed the code to you directly (to keep the formatting). The problem is still occurring.. I did not yet put the config things in environment.rb (like I said on: http://rubyforge.org/pipermail/backgroundrb-devel/2007-December/001233.html).. so maybe that fixes it? Hope you can help.. hemant wrote: > Hi Joost, > > On Dec 28, 2007 5:25 PM, Joost Hietbrink (YelloYello) > wrote: > >> The rails_model_worker.rb contains a module that fetches an object from the >> database. >> We pass parameters to the worker (like classname & id (eg. Place and >> 12312)) and fetch the object again in the worker (using this module method). >> >> The lost connection occurs when we try to fetch the object (record) again >> (in the worker..using the included module method). >> >> The old BackgroundRb did not support sending ActiveRecord objects to the >> worker (thats why we refetch it)..does the new version?? >> >> We now also installed libmysql-ruby1.8 (see: >> http://wiki.rubyonrails.org/rails/pages/Mysql%2BConnection%2BProblems) so >> maybe this fixes the problem..but we have to wait some time to see.. :) >> >> > > Sorry for late reply. But If you can send me the worker code, I will > try to fix it. > There seems to be some problems with AR in threaded environment. I > just want to make sure, none remain unfixed. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071229/d59e5c5b/attachment.html From gethemant at gmail.com Sat Dec 29 21:25:45 2007 From: gethemant at gmail.com (hemant kumar) Date: Sun, 30 Dec 2007 07:55:45 +0530 Subject: [Backgroundrb-devel] Problem with ActiveRecord Connections Message-ID: <1198981545.13052.4.camel@shire> Hi, ActiveRecord behaves badly in threaded environment unless used with care. I am attaching a version of plugin that fixes the issue. I am testing it, currently. Any of you, who are having MySql Connection issues in workers, can run this plugin and see if it fixes the problem. Also, as noted by Jason, problem won't happen immediately, but will rather take sometime. So give it a whirl and see what happens. -------------- next part -------------- A non-text attachment was scrubbed... Name: thread_fix.tar.gz Type: application/x-compressed-tar Size: 35808 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071230/b797000e/attachment-0001.bin From maku at makuchaku.info Sun Dec 30 06:40:52 2007 From: maku at makuchaku.info (=?UTF-8?Q?=E0=A4=AE=E0=A4=AF=E0=A4=82=E0=A4=95_?= =?UTF-8?Q?=E0=A4=9C=E0=A5=88=E0=A4=A8_(?= =?UTF-8?Q?makuchaku)?=) Date: Sun, 30 Dec 2007 17:10:52 +0530 Subject: [Backgroundrb-devel] A newbie question regarding BackgroundRb Message-ID: <185269960712300340v5337f362j7994a6f7b5ae203b@mail.gmail.com> Hi all, I just read Ezra's article on BackgroundRb and decided to use it one of my projects. I installed the plugin into my rails project using script/plugin and then ran rake backgroundrb:setup to copy the yml scripts. Now my backgroundrb.yml looks like ======= --- port: "22222" timer_sleep: 60 load_rails: true environment: development host: localhost database_yml: config/database.yml acl: deny: all allow: localhost 127.0.0.1 order: deny,allow scheduled_task: :class: :tx_reaper_worker :worker_method: :do_work :trigger_args: :start: <%= Time.now %> :repeat_interval: <%= 1.second % ======= TxReaperWorker is my class which I generated. ======= class TxReaperWorker < BackgrounDRb::Rails def do_work(args) puts "Hello BackgroundRb" end end ======= I started my rails app using rake backgroundrb:start script/server However, I am unable to view the output of "puts" anywhere. I can connect successfully to localhost:22222 but cannot see anything happening. Neither anything shown up on the app's output, nor on the log/backgroundrb.log file. However, the logfile does shows that "Log file created...". Am I missing any step? Doing anything wrong? Thanks in advance :) -- Maku http://www.makuchaku.info/blog http://www.apnabill.com - A startup! From ruby at geoffgarside.co.uk Sun Dec 30 07:22:45 2007 From: ruby at geoffgarside.co.uk (Geoff Garside) Date: Sun, 30 Dec 2007 12:22:45 +0000 Subject: [Backgroundrb-devel] A newbie question regarding BackgroundRb In-Reply-To: <185269960712300340v5337f362j7994a6f7b5ae203b@mail.gmail.com> References: <185269960712300340v5337f362j7994a6f7b5ae203b@mail.gmail.com> Message-ID: Did you follow on from http://backgroundrb.rubyforge.org/ or not? BackgrounDRb has been massively overhauled and improved by Hemant. Ezra's article might be related to 0.20 instead of the 1.0 version we have now. Geoff On 30 Dec 2007, at 11:40, ???? ??? (makuchaku) wrote: > Hi all, > > I just read Ezra's article on BackgroundRb and decided to use it one > of my projects. I installed the plugin into my rails project using > script/plugin and then ran rake backgroundrb:setup to copy the yml > scripts. > > Now my backgroundrb.yml looks like > ======= > --- > port: "22222" > timer_sleep: 60 > load_rails: true > environment: development > host: localhost > database_yml: config/database.yml > acl: > deny: all > allow: localhost 127.0.0.1 > order: deny,allow > > scheduled_task: > :class: :tx_reaper_worker > :worker_method: :do_work > :trigger_args: > :start: <%= Time.now %> > :repeat_interval: <%= 1.second % > ======= > > TxReaperWorker is my class which I generated. > > ======= > class TxReaperWorker < BackgrounDRb::Rails > def do_work(args) > puts "Hello BackgroundRb" > end > end > ======= > > I started my rails app using > rake backgroundrb:start > script/server > > However, I am unable to view the output of "puts" anywhere. I can > connect successfully to localhost:22222 but cannot see anything > happening. Neither anything shown up on the app's output, nor on the > log/backgroundrb.log file. > However, the logfile does shows that "Log file created...". > > Am I missing any step? Doing anything wrong? > > Thanks in advance :) > > -- > Maku > http://www.makuchaku.info/blog > http://www.apnabill.com - A startup! > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From gethemant at gmail.com Sun Dec 30 07:39:33 2007 From: gethemant at gmail.com (hemant kumar) Date: Sun, 30 Dec 2007 18:09:33 +0530 Subject: [Backgroundrb-devel] A newbie question regarding BackgroundRb In-Reply-To: <185269960712300340v5337f362j7994a6f7b5ae203b@mail.gmail.com> References: <185269960712300340v5337f362j7994a6f7b5ae203b@mail.gmail.com> Message-ID: <1199018373.13052.12.camel@shire> On Sun, 2007-12-30 at 17:10 +0530, ???? ??? (makuchaku) wrote: > Hi all, > > I just read Ezra's article on BackgroundRb and decided to use it one > of my projects. I installed the plugin into my rails project using > script/plugin and then ran rake backgroundrb:setup to copy the yml > scripts. > > Now my backgroundrb.yml looks like > ======= > --- > port: "22222" > timer_sleep: 60 > load_rails: true > environment: development > host: localhost > database_yml: config/database.yml > acl: > deny: all > allow: localhost 127.0.0.1 > order: deny,allow > > scheduled_task: > :class: :tx_reaper_worker > :worker_method: :do_work > :trigger_args: > :start: <%= Time.now %> > :repeat_interval: <%= 1.second % > ======= > > TxReaperWorker is my class which I generated. > > ======= > class TxReaperWorker < BackgrounDRb::Rails > def do_work(args) > puts "Hello BackgroundRb" > end > end > ======= > > I started my rails app using > rake backgroundrb:start > script/server > > However, I am unable to view the output of "puts" anywhere. I can > connect successfully to localhost:22222 but cannot see anything > happening. Neither anything shown up on the app's output, nor on the > log/backgroundrb.log file. > However, the logfile does shows that "Log file created...". > > Am I missing any step? Doing anything wrong? > > Thanks in advance :) > You are using VERY old version of plugin, can you please checkout details regarding new version from: http://backgroundrb.rubyforge.org