From ezmobius at gmail.com Sat Jul 1 19:53:09 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sat, 1 Jul 2006 16:53:09 -0700 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb New release. Message-ID: Howdy Folks- I'm happy to announce a new release of BackgrounDRb! I have added quite a few new features and included some nice patches from folks on the list. $ script/plugin install svn://rubyforge.org//var/svn/backgroundrb All of the code now stays within the plugin and the start and stop scripts are now just stubs. This makes it easier to tweak or figure out how it works. The backgroundrb.rb file in the root of the plugin dir is the drb server part of MiddleMan and the pluginroot/lib/ backgroundrb.rb file is the rails client side MiddleMan. One of the new things is much better threading job control. This is mostly invisible to you as a user of the plugin but it makes the whole system a lot more sturdy. It gives your worker objects the chance to clean up after themselves when they get deleted instead of just getting the kill brick dropped on them ;) Thanks to David Lemstra for the job management code! The other big new feature is a built in timer and a time to live argument for workers and caches. This allows you to set how long a worker or a cache survives before it gets reaped. The :ttl param defaults to 30 minutes for worker classes and 10 minutes for cached objects. If you don;t specify the :ttl param then thats how long your stuff will last. If you want longer or shorter ttl's then specify :ttl in *seconds* like this: #worker session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, :args => "Bar", :ttl => 300) # 300 seconds == 5 minutes #caching #fill_cache, expire in 5 minutes @posts = Post.find(:all, :include => :comments) MiddleMan.cache_as(:post_cache, 300, @posts) #OR @posts = MiddleMan.cache_as :post_cache, 300 do Post.find(:all, :include => :comments) end #retrieve_cache @posts = MiddleMan.cache_get :post_cache do Post.find(:all, :include => :comments) end By default the timer_sleep is set to 60 seconds. This means that the timer will wakeup every 60 seconds and delete caches and worker that are past their time to live. You can change this in the config file if you want it to run faster or slower. All your worker classes should now inherit from BackgrounDRb::Rails so they get all the good new thread management features. If you are already using the plugin you should delete the plugin from your rails app and reinstall it. You should also delete the script/backgroundrb directory and the config/backgroundrb.yml file. Then run: # to install the new code. $ rake backgroundrb:setup Your workers should not need to change as long as they inherit from BackgrounDRb::Rails. Please test this and give me feedback if you are using it. I would also like to start collecting worker classes or at least hear what folks are using this plugin for. *Home page: http://backgroundrb.rubyforge.org *svn repo: svn://rubyforge.org//var/svn/backgroundrb *Mailing list: http://rubyforge.org/mailman/listinfo/backgroundrb-devel Thanks -Ezra From david at lemstra.ca Sun Jul 2 00:22:25 2006 From: david at lemstra.ca (David Lemstra) Date: Sun, 02 Jul 2006 00:22:25 -0400 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb New release. In-Reply-To: References: Message-ID: <44A74A01.5060000@lemstra.ca> Nifty stuff Ezra! Haven't had time to test yet, but just want to point out something that I'm worried could cause people some grief, unless I'm missing something. > One of the new things is much better threading job control. This is > mostly invisible to you as a user of the plugin but it makes the > whole system a lot more sturdy. It gives your worker objects the > chance to clean up after themselves when they get deleted instead of > just getting the kill brick dropped on them ;) The assumption is that your worker class has some kind of a loop and calls 'terminate' somewhere in the loop where it is safe to kill (or uses 'check_terminate' to see if we should cleanup and then call 'terminate'). 'terminate' then raises a signal saying it is OK to be killed. The problem: by default, if you don't call 'terminate' anywhere in the worker class, the OK-to-Kill signal is never raised, so 'delete_worker' never follows through and kills the worker, but instead sits there waiting for the OK-to-kill signal. Maybe the worker class should have a variable (off by default) that enables this behaviour? When I sent out the code, I assumed it'd be a reference and people would know that they should use terminate if they went that route. Now I'm afraid people will upgrade but not realize their processes are no longer being killed. I think a simple condition shoved in like: if (@jobs[key].jb_ctrl) then do_the_signal_stuff end Then @jb_ctrl must be enabled in worker class. ie. def delete_worker(key) @mutex.synchronize { if @jobs[key] if @jobs[key].respond_to? :thread if @jobs[key].thread.alive? if @jobs[key].jb_ctrl @jobs[key].thread[:kill] = true; @jobs[key].thread[:safe_to_kill].wait(@mutex) end @jobs[key].thread.kill end end @jobs.delete(key) end @timestamps.delete(key) if @timestamps.has_key?(key) } end > I would also like to start collecting worker classes or at > least hear what folks are using this plugin for. I'm using it to do a hole whack of calculations w/ the database in the background. BackgrounDRb keeps my www server responsive and lets me use periodically_call_remote() to update a progress bar. W/ the job control, www clients can even cancel the calculations. cheers, David Lemstra From gethemant at gmail.com Mon Jul 3 10:04:31 2006 From: gethemant at gmail.com (hemant) Date: Mon, 3 Jul 2006 19:34:31 +0530 Subject: [Backgroundrb-devel] [Rails] [ANN] BackgrounDRb New release. In-Reply-To: References: Message-ID: How about jobs that I would like to run forever? I collect newsfeeds from various sites in background and would like to run them forever!! In fact...i was doing this, but what to do now? So..when a job gets deleted after 30 minutes..does it also deletes the session key? If not...then we won't be able to detect within our rails app, if the client has died!! On 7/2/06, Ezra Zygmuntowicz wrote: > > Howdy Folks- > > I'm happy to announce a new release of BackgrounDRb! I have added > quite a few new features and included some nice patches from folks on > the list. > > $ script/plugin install svn://rubyforge.org//var/svn/backgroundrb > > All of the code now stays within the plugin and the start and stop > scripts are now just stubs. This makes it easier to tweak or figure > out how it works. The backgroundrb.rb file in the root of the plugin > dir is the drb server part of MiddleMan and the pluginroot/lib/ > backgroundrb.rb file is the rails client side MiddleMan. > > One of the new things is much better threading job control. This > is > mostly invisible to you as a user of the plugin but it makes the > whole system a lot more sturdy. It gives your worker objects the > chance to clean up after themselves when they get deleted instead of > just getting the kill brick dropped on them ;) Thanks to David > Lemstra for the job management code! > > The other big new feature is a built in timer and a time to live > argument for workers and caches. This allows you to set how long a > worker or a cache survives before it gets reaped. The :ttl param > defaults to 30 minutes for worker classes and 10 minutes for cached > objects. If you don;t specify the :ttl param then thats how long your > stuff will last. If you want longer or shorter ttl's then > specify :ttl in *seconds* like this: > > #worker > session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, > > :args => "Bar", > > :ttl => 300) # 300 seconds == 5 minutes > > #caching > > #fill_cache, expire in 5 minutes > @posts = Post.find(:all, :include => :comments) > MiddleMan.cache_as(:post_cache, 300, @posts) > #OR > @posts = MiddleMan.cache_as :post_cache, 300 do > Post.find(:all, :include => :comments) > end > > #retrieve_cache > @posts = MiddleMan.cache_get :post_cache do > Post.find(:all, :include => :comments) > end > > By default the timer_sleep is set to 60 seconds. This means that > the > timer will wakeup every 60 seconds and delete caches and worker that > are past their time to live. You can change this in the config file > if you want it to run faster or slower. > > All your worker classes should now inherit from > BackgrounDRb::Rails > so they get all the good new thread management features. If you are > already using the plugin you should delete the plugin from your rails > app and reinstall it. You should also delete the script/backgroundrb > directory and the config/backgroundrb.yml file. Then run: > > # to install the new code. > $ rake backgroundrb:setup > > Your workers should not need to change as long as they inherit from > BackgrounDRb::Rails. Please test this and give me feedback if you are > using it. I would also like to start collecting worker classes or at > least hear what folks are using this plugin for. > > *Home page: http://backgroundrb.rubyforge.org > *svn repo: svn://rubyforge.org//var/svn/backgroundrb > *Mailing list: http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > Thanks > -Ezra > > _______________________________________________ > Rails mailing list > Rails at lists.rubyonrails.org > http://lists.rubyonrails.org/mailman/listinfo/rails > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060703/ef34b1aa/attachment.html From gladwig at gmx.de Mon Jul 3 11:28:53 2006 From: gladwig at gmx.de (=?ISO-8859-1?Q?G=FCnter_Ladwig?=) Date: Mon, 3 Jul 2006 17:28:53 +0200 Subject: [Backgroundrb-devel] Req: Workers as singletons Message-ID: <392631A5-368F-4378-B257-590F7F9839D3@gmx.de> Hi, it would be nice to be able to specify workers as singletons. By this I mean that every call to the new_worker method returns the same instance of said worker. This can be done transparently either by adding a new argument to new_worker (something like :singleton => true) or adding a new method, like I did in my installation of BackgrounDrb: def get_worker_by_class(klass) klasse = klass.to_s.split('_').inject('') { |total,part| total << part.capitalize } @jobs.each do |e| @logger.debug e.inspect if e[1].class.name == klasse return e[0] end end return nil end def new_singleton_worker(opts={}) @mutex.synchronize { worker = get_worker_by_class(opts[:class]) if worker.nil? job_key = opts[:job_key] || gen_key unless self[job_key] self[job_key] = instantiate_worker(opts[:class]).new(opts [:args]) return job_key else raise "job_key: #{job_key} already refers to a running worker" end else return worker end } end First the jobs array is checked if an instance of the requested worker class is already available. If yes, it's returned otherwise a new instance is created using the standard instantiate_worker method. Could you include something like this in the next version? G?nter From gladwig at gmx.de Mon Jul 3 11:32:46 2006 From: gladwig at gmx.de (=?ISO-8859-1?Q?G=FCnter_Ladwig?=) Date: Mon, 3 Jul 2006 17:32:46 +0200 Subject: [Backgroundrb-devel] Fwd: Models and Backgroundrb References: <3AE99A35-9480-4ABF-8F5D-F42C4163A1B6@gmx.de> Message-ID: <80931ADF-E2E4-4B17-8AA3-B8BDDA214C68@gmx.de> Seems this mail did not make it to the list for some reason. It contains my solution for the problem with login_engine without having to load the whole Rails environment. Begin forwarded message: > From: G?nter Ladwig > Date: 22. Juni 2006 23:51:18 MESZ > To: backgroundrb-devel at rubyforge.org > Subject: Re: [Backgroundrb-devel] Models and Backgroundrb > > Hi, > > I used this to get LoginEngine to work (add to the top of your > worker): > > $: << "#{RAILS_ROOT}/vendor/plugins/engines/lib/engines" > $: << "#{RAILS_ROOT}/vendor/plugins/login_engine/lib" > > This adds the login engine directories to the search path Ruby uses > when loading require'd files. > > Worked for me :) > > G?nter > > On 22.06.2006, at 23:31, Charles Brian Quinn wrote: > >> Your model that you're trying find on includes some libraries that >> backgroundrb doesn't know about. >> >> Try including all of rails when loading the script/backgroundrb/ >> start -- was just posted by me and someone else recently on how to >> load ALL of rails instead of just the ActiveRecord connection. >> >> If that solves your problem, you can either, a) keep it and take >> the large memory hit for running backgroundrb with your full RAILS >> env loaded up, or b) find out what libraries (the >> authenticated_system.rb, including LogineEngine, etc.) that your >> model that you're performing the Find on is using, and load those >> up in your backgroundrb process instead (see archive for how to do >> this, too) of the entire environment. >> >> Hope that helps! >> >> On 6/22/06, hemant wrote: >> Hi Erza, >> >> I am using login engine in my code...and i get the error >> "uninitialized constant LoginEngine and uninitialized constant >> LoginEngine", when i try to do a find in the worker code!!! >> >> Why am i getting this error? >> >> >> >> >> -- >> nothing much to talk >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> >> >> >> >> -- >> Charles Brian Quinn >> www.seebq.com >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From ezmobius at gmail.com Mon Jul 3 16:19:42 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 3 Jul 2006 13:19:42 -0700 Subject: [Backgroundrb-devel] Req: Workers as singletons In-Reply-To: <392631A5-368F-4378-B257-590F7F9839D3@gmx.de> References: <392631A5-368F-4378-B257-590F7F9839D3@gmx.de> Message-ID: On Jul 3, 2006, at 8:28 AM, G?nter Ladwig wrote: > Hi, > > it would be nice to be able to specify workers as singletons. By this > I mean that every call to the new_worker method returns the same > instance of said worker. This can be done transparently either by > adding a new argument to new_worker (something like :singleton => > true) or adding a new method, like I did in my installation of > BackgrounDrb: > > def get_worker_by_class(klass) > klasse = klass.to_s.split('_').inject('') { |total,part| total > << part.capitalize } > @jobs.each do |e| > @logger.debug e.inspect > if e[1].class.name == klasse > return e[0] > end > end > return nil > end > > def new_singleton_worker(opts={}) > @mutex.synchronize { > worker = get_worker_by_class(opts[:class]) > if worker.nil? > job_key = opts[:job_key] || gen_key > unless self[job_key] > self[job_key] = instantiate_worker(opts[:class]).new(opts > [:args]) > return job_key > else > raise "job_key: #{job_key} already refers to a running > worker" > end > else > return worker > end > } > end > > First the jobs array is checked if an instance of the requested > worker class is already available. If yes, it's returned otherwise a > new instance is created using the standard instantiate_worker method. > > Could you include something like this in the next version? > > G?nter Hi Gunter- I can add a singleton type worker probably. Can you tell me what your use case is for needing this? I'm curious as to how you would use or what you plan on doing with singleton workers. Are you defining your worker classes as actual singletons using include Singleton? If so then maybe I can make a different type of superclass to inherit your workers from that is a true singleton. I'll have to think about it a bit. I just added new job management stuff to the threading model so I will have to work that in with your singleton idea but I'm sure I can come up with a nice way of doing this for you. Cheers- -Ezra From ezmobius at gmail.com Mon Jul 3 16:34:27 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 3 Jul 2006 13:34:27 -0700 Subject: [Backgroundrb-devel] [Rails] [ANN] BackgrounDRb New release. In-Reply-To: References: Message-ID: <0B22C2CC-7D81-4F2D-BCC4-11285ED6420D@brainspl.at> On Jul 3, 2006, at 7:04 AM, hemant wrote: > How about jobs that I would like to run forever? > > I collect newsfeeds from various sites in background and would like > to run them forever!! > > In fact...i was doing this, but what to do now? > So..when a job gets deleted after 30 minutes..does it also deletes > the session key? If not...then we won't be able to detect within > our rails app, if the client has died!! > > > On 7/2/06, Ezra Zygmuntowicz wrote: > Howdy Folks- > > I'm happy to announce a new release of BackgrounDRb! I have > added > quite a few new features and included some nice patches from folks on > the list. > > $ script/plugin install svn://rubyforge.org//var/svn/backgroundrb > Hemant- Thats a good point. I will add an option to allow for 'immortal' workers. Does this syntax look good to you? session[:job_key] = MiddleMan.new_worker(:class => :long_worker, :args => 'whatever', :ttl => :immortal) I will see what the best way to add this is and get another release out today. Thanks -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060703/aa09003d/attachment-0001.html From gladwig at gmx.de Mon Jul 3 17:45:54 2006 From: gladwig at gmx.de (=?ISO-8859-1?Q?G=FCnter_Ladwig?=) Date: Mon, 3 Jul 2006 23:45:54 +0200 Subject: [Backgroundrb-devel] Req: Workers as singletons In-Reply-To: References: <392631A5-368F-4378-B257-590F7F9839D3@gmx.de> Message-ID: On 03.07.2006, at 22:19, Ezra Zygmuntowicz wrote: > > Hi Gunter- > > I can add a singleton type worker probably. Can you tell me what > your use case is for needing this? I'm curious as to how you would > use or what you plan on doing with singleton workers. Are you > defining your worker classes as actual singletons using include > Singleton? If so then maybe I can make a different type of > superclass to inherit your workers from that is a true singleton. > I'll have to think about it a bit. I just added new job management > stuff to the threading model so I will have to work that in with > your singleton idea but I'm sure I can come up with a nice way of > doing this for you. No, the worker class is not defined as a Singleton. It's just that I need only one instance of the worker. I'm not using the worker for temporary background calculations (which seems to be the common use case), but instead as a background process that periodically updates the database and takes requests from the Rails app. It should not be killed and respawned because it has persistent connections to other servers. G?nter From gladwig at gmx.de Mon Jul 3 17:56:11 2006 From: gladwig at gmx.de (=?ISO-8859-1?Q?G=FCnter_Ladwig?=) Date: Mon, 3 Jul 2006 23:56:11 +0200 Subject: [Backgroundrb-devel] [Rails] [ANN] BackgrounDRb New release. In-Reply-To: <0B22C2CC-7D81-4F2D-BCC4-11285ED6420D@brainspl.at> References: <0B22C2CC-7D81-4F2D-BCC4-11285ED6420D@brainspl.at> Message-ID: On 03.07.2006, at 22:34, Ezra Zygmuntowicz wrote: > > Hemant- > > Thats a good point. I will add an option to allow for 'immortal' > workers. Does this syntax look good to you? > > session[:job_key] = MiddleMan.new_worker(:class => :long_worker, > > :args => 'whatever', > > :ttl => :immortal) > > I will see what the best way to add this is and get another > release out today. Wouldn't it be better to just assume 'immortality' if there is no :ttl argument? Seems more logical (to me, anyway) and doesn't change the semantics to the first release :) G?nter P.S.: I replied to this message before, using a different account and the message now seems to wait for moderator approval... just delete it ;) From guenter at ladwig.info Mon Jul 3 17:48:53 2006 From: guenter at ladwig.info (=?ISO-8859-1?Q?G=FCnter_Ladwig?=) Date: Mon, 3 Jul 2006 23:48:53 +0200 Subject: [Backgroundrb-devel] [Rails] [ANN] BackgrounDRb New release. In-Reply-To: <0B22C2CC-7D81-4F2D-BCC4-11285ED6420D@brainspl.at> References: <0B22C2CC-7D81-4F2D-BCC4-11285ED6420D@brainspl.at> Message-ID: <9F57D2A4-4838-44B2-AEC7-0B8C5823D19A@ladwig.info> On 03.07.2006, at 22:34, Ezra Zygmuntowicz wrote: > Hemant- > > Thats a good point. I will add an option to allow for 'immortal' > workers. Does this syntax look good to you? > > session[:job_key] = MiddleMan.new_worker(:class => :long_worker, > > :args => 'whatever', > > :ttl => :immortal) > Wouldn't it be better to just assume 'immortality' if there is no :ttl argument? Ie. the default is that the worker is never killed? Seems more logical somehow :) To me, anyway. G?nter From ezmobius at gmail.com Mon Jul 3 18:28:24 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 3 Jul 2006 15:28:24 -0700 Subject: [Backgroundrb-devel] Req: Workers as singletons In-Reply-To: References: <392631A5-368F-4378-B257-590F7F9839D3@gmx.de> Message-ID: <0769CC9A-9805-43B4-9853-47004E94FC67@brainspl.at> On Jul 3, 2006, at 2:45 PM, G?nter Ladwig wrote: > > On 03.07.2006, at 22:19, Ezra Zygmuntowicz wrote: > >> >> Hi Gunter- >> >> I can add a singleton type worker probably. Can you tell me what >> your use case is for needing this? I'm curious as to how you would >> use or what you plan on doing with singleton workers. Are you >> defining your worker classes as actual singletons using include >> Singleton? If so then maybe I can make a different type of >> superclass to inherit your workers from that is a true singleton. >> I'll have to think about it a bit. I just added new job management >> stuff to the threading model so I will have to work that in with >> your singleton idea but I'm sure I can come up with a nice way of >> doing this for you. > > No, the worker class is not defined as a Singleton. It's just that I > need only one instance of the worker. I'm not using the worker for > temporary background calculations (which seems to be the common use > case), but instead as a background process that periodically updates > the database and takes requests from the Rails app. It should not be > killed and respawned because it has persistent connections to other > servers. > > G?nter Gunter- OK I see what you want to do. I will come up with a way to do that for you. I also just changed things so that if you don't specify a :ttl param when you do new_worker then your worker will become :immortal. The only way to delete an immortal worker is by calling gc! or using delete_worker directly. Cheers- -Ezra From gladwig at gmx.de Mon Jul 3 18:38:55 2006 From: gladwig at gmx.de (=?ISO-8859-1?Q?G=FCnter_Ladwig?=) Date: Tue, 4 Jul 2006 00:38:55 +0200 Subject: [Backgroundrb-devel] Req: Workers as singletons In-Reply-To: <0769CC9A-9805-43B4-9853-47004E94FC67@brainspl.at> References: <392631A5-368F-4378-B257-590F7F9839D3@gmx.de> <0769CC9A-9805-43B4-9853-47004E94FC67@brainspl.at> Message-ID: On 04.07.2006, at 00:28, Ezra Zygmuntowicz wrote: > > Gunter- > > OK I see what you want to do. I will come up with a way to do that > for you. I also just changed things so that if you don't specify > a :ttl param when you do new_worker then your worker will > become :immortal. The only way to delete an immortal worker is by > calling gc! or using delete_worker directly. Thanks! I just thought of another "bug" which caused me some headaches: Apparently, on Macs (I'm using an iBook) the DRb server needs to be started like this: DRb.start_service('druby://localhost:0') instead of just DRb.start_service Don't ask me why, I found the solution to this problem here: http://rubyforge.org/pipermail/bdrg-members/2006-May/000043.html G?nter From gethemant at gmail.com Mon Jul 3 20:36:46 2006 From: gethemant at gmail.com (hemant) Date: Tue, 4 Jul 2006 06:06:46 +0530 Subject: [Backgroundrb-devel] [Rails] [ANN] BackgrounDRb New release. In-Reply-To: <0B22C2CC-7D81-4F2D-BCC4-11285ED6420D@brainspl.at> References: <0B22C2CC-7D81-4F2D-BCC4-11285ED6420D@brainspl.at> Message-ID: Yeah looks fine. BTW, you didn't tell me abt, what happens to the session object, when after 30 minutes that thread gets deleted? On 7/4/06, Ezra Zygmuntowicz wrote: > > > On Jul 3, 2006, at 7:04 AM, hemant wrote: > > How about jobs that I would like to run forever? > > I collect newsfeeds from various sites in background and would like to run > them forever!! > > In fact...i was doing this, but what to do now? > So..when a job gets deleted after 30 minutes..does it also deletes the > session key? If not...then we won't be able to detect within our rails app, > if the client has died!! > > > On 7/2/06, Ezra Zygmuntowicz wrote: > > > > Howdy Folks- > > > > I'm happy to announce a new release of BackgrounDRb! I have > > added > > quite a few new features and included some nice patches from folks on > > the list. > > > > $ script/plugin install svn://rubyforge.org//var/svn/backgroundrb > > > > > Hemant- > > Thats a good point. I will add an option to allow for 'immortal' workers. > Does this syntax look good to you? > > session[:job_key] = MiddleMan.new_worker(:class => :long_worker, > > :args => 'whatever', > > :ttl => :immortal) > > I will see what the best way to add this is and get another release out > today. > > Thanks > -Ezra > > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060703/d6ccd3d5/attachment.html From ezmobius at gmail.com Mon Jul 3 22:10:21 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 3 Jul 2006 19:10:21 -0700 Subject: [Backgroundrb-devel] [Rails] [ANN] BackgrounDRb New release. In-Reply-To: References: <0B22C2CC-7D81-4F2D-BCC4-11285ED6420D@brainspl.at> Message-ID: <614E5B6A-0AF7-45B7-B344-4343DB755F0D@gmail.com> On Jul 3, 2006, at 5:36 PM, hemant wrote: > Yeah looks fine. > > BTW, you didn't tell me abt, what happens to the session object, > when after 30 minutes that thread gets deleted? > > Hemant- Ok after thinking about things this is what it in the current svn. I think you are right to wonder about what to do with a stale session. So I changed things to behave like this: When you do: session[:job_key] = MiddleMan.new_worker(:class => :long_worker, :args => 'whatever') It will create an immortal worker by default if you don't include the :ttl param. But you can include the :ttl if you want to be specific. #So this will create an immortal worker too: session[:job_key] = MiddleMan.new_worker(:class => :long_worker, :args => 'whatever', :ttl => :immortal) So now you have to specify the :ttl param if you want your worker to be killed after a certain amount of time. session[:job_key] = MiddleMan.new_worker(:class => :long_worker, :args => 'whatever', :ttl => 900) # 15 minutes When you create immortal workers you are responsible for deleting them when you are done. When you do use caching you still get a default :ttl of 10 minutes. And you should use the block form of cache_get so that if the cache has expired, it will get filled again and behave like a cache hit. So to answer your question about what to do with the session if your worker dies is something like this: unless MiddleMan[session[:jobkey]].nil? MiddleMan[session[:jobkey]].some_method end So I recommend either using immortal workers and deleting them with delete_worker or being careful about checking that your job is alive before calling methods on it. Cheers- -Ezra From gethemant at gmail.com Tue Jul 4 00:42:02 2006 From: gethemant at gmail.com (hemant) Date: Tue, 4 Jul 2006 10:12:02 +0530 Subject: [Backgroundrb-devel] [Rails] [ANN] BackgrounDRb New release. In-Reply-To: <614E5B6A-0AF7-45B7-B344-4343DB755F0D@gmail.com> References: <0B22C2CC-7D81-4F2D-BCC4-11285ED6420D@brainspl.at> <614E5B6A-0AF7-45B7-B344-4343DB755F0D@gmail.com> Message-ID: Thanks Erza, Just updated the SVN.... Yeah...i had the idea that, i must either reset the rails session after half an hour, or check for availability of the Middleman class object!! Anyway..thanks for the immortal worker threads, they are a required feature i guess. On 7/4/06, Ezra Zygmuntowicz wrote: > > > On Jul 3, 2006, at 5:36 PM, hemant wrote: > > > Yeah looks fine. > > > > BTW, you didn't tell me abt, what happens to the session object, > > when after 30 minutes that thread gets deleted? > > > > > > Hemant- > > Ok after thinking about things this is what it in the current svn. > I > think you are right to wonder about what to do with a stale session. > So I changed things to behave like this: > > When you do: > session[:job_key] = MiddleMan.new_worker(:class => :long_worker, > > :args => 'whatever') > > > It will create an immortal worker by default if you don't include > the :ttl param. But you can include the :ttl if you want to be specific. > > #So this will create an immortal worker too: > session[:job_key] = MiddleMan.new_worker(:class => :long_worker, > > :args => 'whatever', > > :ttl => :immortal) > > So now you have to specify the :ttl param if you want your worker to > be killed after a certain amount of time. > > session[:job_key] = MiddleMan.new_worker(:class => :long_worker, > > :args => 'whatever', > > :ttl => 900) # 15 minutes > > When you create immortal workers you are responsible for deleting > them when you are done. When you do use caching you still get a > default :ttl of 10 minutes. And you should use the block form of > cache_get so that if the cache has expired, it will get filled again > and behave like a cache hit. > > So to answer your question about what to do with the session if your > worker dies is something like this: > > unless MiddleMan[session[:jobkey]].nil? > MiddleMan[session[:jobkey]].some_method > end > > So I recommend either using immortal workers and deleting them with > delete_worker or being careful about checking that your job is alive > before calling methods on it. > > > Cheers- > -Ezra > > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060704/299fbfbc/attachment-0001.html From hutch at recursive.ca Tue Jul 4 09:51:47 2006 From: hutch at recursive.ca (Bob Hutchison) Date: Tue, 4 Jul 2006 09:51:47 -0400 Subject: [Backgroundrb-devel] Req: Workers as singletons In-Reply-To: References: <392631A5-368F-4378-B257-590F7F9839D3@gmx.de> <0769CC9A-9805-43B4-9853-47004E94FC67@brainspl.at> Message-ID: <71430502-8700-42A8-853C-06DE62937AB8@recursive.ca> Hi Ezra, On Jul 3, 2006, at 6:38 PM, G?nter Ladwig wrote: > > On 04.07.2006, at 00:28, Ezra Zygmuntowicz wrote: > >> >> Gunter- >> >> OK I see what you want to do. I will come up with a way to do that >> for you. I also just changed things so that if you don't specify >> a :ttl param when you do new_worker then your worker will >> become :immortal. The only way to delete an immortal worker is by >> calling gc! or using delete_worker directly. > > Thanks! I have a similar situation to G?nter, so... Thanks from me too. Cheers, Bob > > G?nter > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel ---- Bob Hutchison -- blogs at Recursive Design Inc. -- Raconteur -- xampl for Ruby -- From david at fallside.com Wed Jul 5 23:32:17 2006 From: david at fallside.com (David Fallside) Date: Wed, 05 Jul 2006 20:32:17 -0700 Subject: [Backgroundrb-devel] Req: Workers as singletons In-Reply-To: References: <392631A5-368F-4378-B257-590F7F9839D3@gmx.de> Message-ID: <44AC8441.5080502@fallside.com> I am looking to use Rails to manage an application. More specifically, I am looking to start/stop and change parameters of my application through a webpage built in Rails, and I am looking to backgroundrb workers to be the application classes. So, +1 for Gunter's use case! Thanks, David G?nter Ladwig wrote: >On 03.07.2006, at 22:19, Ezra Zygmuntowicz wrote: > > > >>Hi Gunter- >> >> I can add a singleton type worker probably. Can you tell me what >>your use case is for needing this? I'm curious as to how you would >>use or what you plan on doing with singleton workers. Are you >>defining your worker classes as actual singletons using include >>Singleton? If so then maybe I can make a different type of >>superclass to inherit your workers from that is a true singleton. >>I'll have to think about it a bit. I just added new job management >>stuff to the threading model so I will have to work that in with >>your singleton idea but I'm sure I can come up with a nice way of >>doing this for you. >> >> > >No, the worker class is not defined as a Singleton. It's just that I >need only one instance of the worker. I'm not using the worker for >temporary background calculations (which seems to be the common use >case), but instead as a background process that periodically updates >the database and takes requests from the Rails app. It should not be >killed and respawned because it has persistent connections to other >servers. > >G?nter > >_______________________________________________ >Backgroundrb-devel mailing list >Backgroundrb-devel at rubyforge.org >http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > From gethemant at gmail.com Thu Jul 6 02:08:59 2006 From: gethemant at gmail.com (hemant) Date: Thu, 6 Jul 2006 11:38:59 +0530 Subject: [Backgroundrb-devel] memory uses Message-ID: Hi All, I have this small doubt. Does backgroundrb have anything to do with dispatch.fcgi? some dispatch.fcgi are taking memory as much as 186 MB .... this is monstrous man... Also, if i put load rails environment with backgroundrb.yml file to be false.What will happen? i just loses access to db through ActiveRecord right? I can still use normal mysql gem to access db and modify the tables..correct? I should be able to create Middleman objects from Rails as usual? -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060706/0be3fca1/attachment.html From gethemant at gmail.com Thu Jul 6 02:22:19 2006 From: gethemant at gmail.com (hemant) Date: Thu, 6 Jul 2006 11:52:19 +0530 Subject: [Backgroundrb-devel] memory uses In-Reply-To: References: Message-ID: Also....there seems to be some problem with mysql connections.... I am getting "too many connections error - mysql", this could be either because of open connections to mysql - are not being closed. So the question...normally from rails when using dispatch.fcgi, only a limited number of connections are being opened to the database(people infact tell me, only one is opened). So, the big question is, Erza, how you are handling connections to the DB? I mean..when i use ActiveRecord? So, is that each thread will create its own connection? Again..the question..arises does backgroundrb has any business with dispatch.fcgi? On 7/6/06, hemant wrote: > > Hi All, > > I have this small doubt. Does backgroundrb have anything to do with > dispatch.fcgi? > > some dispatch.fcgi are taking memory as much as 186 MB .... > this is monstrous man... > > Also, if i put load rails environment with backgroundrb.yml file to be > false.What will happen? i just loses access to db through ActiveRecord > right? > I can still use normal mysql gem to access db and modify the > tables..correct? > > I should be able to create Middleman objects from Rails as usual? > > > > -- > nothing much to talk > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060706/e221bdd5/attachment.html From ezmobius at gmail.com Thu Jul 6 02:42:00 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 5 Jul 2006 23:42:00 -0700 Subject: [Backgroundrb-devel] memory uses In-Reply-To: References: Message-ID: Hemant- No backgroundrb has nothing to do with dispatch.fcgi. Thats all rails in there. And BackgrounDRb only makes one connection to the db on startup. I will investigate to see if i can reproduce your error but it sounds like there may be an issue in your rails app. I do not think that each thread gets its own connection to the db since AR only conects once on startup. As for setting load_rails to false you are right, you will not be able to use your AR models. But if you require the mysql gem you will be able to use that like you want. Sounds like you have a memory leak in your rails app if its taking up 186Mb. Are you running in development mode or production? -Ezra On Jul 5, 2006, at 11:22 PM, hemant wrote: > Also....there seems to be some problem with mysql connections.... > > I am getting "too many connections error - mysql", this could be > either because of open connections to mysql - are not being closed. > > So the question...normally from rails when using dispatch.fcgi, > only a limited number of connections are being opened to the > database(people infact tell me, only one is opened). So, the big > question is, Erza, how you are handling connections to the DB? > > I mean..when i use ActiveRecord? So, is that each thread will > create its own connection? Again..the question..arises does > backgroundrb has any business with dispatch.fcgi? > > > On 7/6/06, hemant wrote: > Hi All, > > I have this small doubt. Does backgroundrb have anything to do with > dispatch.fcgi? > > some dispatch.fcgi are taking memory as much as 186 MB .... > this is monstrous man... > > Also, if i put load rails environment with backgroundrb.yml file to > be false.What will happen? i just loses access to db through > ActiveRecord right? > I can still use normal mysql gem to access db and modify the > tables..correct? > > I should be able to create Middleman objects from Rails as usual? > > > > -- > nothing much to talk > > > > -- > nothing much to talk > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060706/e007123d/attachment.html From gethemant at gmail.com Thu Jul 6 03:20:47 2006 From: gethemant at gmail.com (hemant) Date: Thu, 6 Jul 2006 12:50:47 +0530 Subject: [Backgroundrb-devel] memory uses In-Reply-To: References: Message-ID: Its running in development mode....and may be yes. But i am not any using any railr/ruby plugin/modules that can cause memory leak. But it is one of those big bang apps...i mean its a little complicated and lots of wierd stuff thrown in. but after i stopped backgroundrb the error that i was getting stopped coming.(So no more too many connections error, after stopping backgroundrb). On 7/6/06, Ezra Zygmuntowicz wrote: > > Hemant- > No backgroundrb has nothing to do with dispatch.fcgi. Thats all rails in > there. And BackgrounDRb only makes one connection to the db on startup. I > will investigate to see if i can reproduce your error but it sounds like > there may be an issue in your rails app. I do not think that each thread > gets its own connection to the db since AR only conects once on startup. > > As for setting load_rails to false you are right, you will not be able to > use your AR models. But if you require the mysql gem you will be able to use > that like you want. > > Sounds like you have a memory leak in your rails app if its taking up > 186Mb. Are you running in development mode or production? > > -Ezra > > On Jul 5, 2006, at 11:22 PM, hemant wrote: > > Also....there seems to be some problem with mysql connections.... > > I am getting "too many connections error - mysql", this could be either > because of open connections to mysql - are not being closed. > > So the question...normally from rails when using dispatch.fcgi, only a > limited number of connections are being opened to the database(people infact > tell me, only one is opened). So, the big question is, Erza, how you are > handling connections to the DB? > > I mean..when i use ActiveRecord? So, is that each thread will create its > own connection? Again..the question..arises does backgroundrb has any > business with dispatch.fcgi? > > > On 7/6/06, hemant wrote: > > > > Hi All, > > > > I have this small doubt. Does backgroundrb have anything to do with > > dispatch.fcgi? > > > > some dispatch.fcgi are taking memory as much as 186 MB .... > > this is monstrous man... > > > > Also, if i put load rails environment with backgroundrb.yml file to be > > false.What will happen? i just loses access to db through ActiveRecord > > right? > > I can still use normal mysql gem to access db and modify the > > tables..correct? > > > > I should be able to create Middleman objects from Rails as usual? > > > > > > > > -- > > nothing much to talk > > > > > > -- > nothing much to talk > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060706/d769922e/attachment-0001.html From ezmobius at gmail.com Thu Jul 6 03:35:25 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 6 Jul 2006 00:35:25 -0700 Subject: [Backgroundrb-devel] memory uses In-Reply-To: References: Message-ID: <1702CE82-1504-4341-8C1D-5F7E9354DF6D@gmail.com> How many worker threads are you running? Are you using the latest code? Maybe you can try this.. In the start script comment out this line: ActiveRecord::Base.allow_concurrency = true Comment that out and try again and let me know if you still get too many connections. Thanks -Ezra On Jul 6, 2006, at 12:20 AM, hemant wrote: > Its running in development mode....and may be yes. > > But i am not any using any railr/ruby plugin/modules that can cause > memory leak. > But it is one of those big bang apps...i mean its a little > complicated and lots of wierd stuff thrown in. > > but after i stopped backgroundrb the error that i was getting > stopped coming.(So no more too many connections error, after > stopping backgroundrb). > > > > On 7/6/06, Ezra Zygmuntowicz wrote: > Hemant- > > No backgroundrb has nothing to do with dispatch.fcgi. Thats all > rails in there. And BackgrounDRb only makes one connection to the > db on startup. I will investigate to see if i can reproduce your > error but it sounds like there may be an issue in your rails app. I > do not think that each thread gets its own connection to the db > since AR only conects once on startup. > > As for setting load_rails to false you are right, you will not be > able to use your AR models. But if you require the mysql gem you > will be able to use that like you want. > > Sounds like you have a memory leak in your rails app if its taking > up 186Mb. Are you running in development mode or production? > > -Ezra > > On Jul 5, 2006, at 11:22 PM, hemant wrote: > > Also....there seems to be some problem with mysql connections.... > > I am getting "too many connections error - mysql", this could be > either because of open connections to mysql - are not being closed. > > So the question...normally from rails when using dispatch.fcgi, > only a limited number of connections are being opened to the > database(people infact tell me, only one is opened). So, the big > question is, Erza, how you are handling connections to the DB? > > I mean..when i use ActiveRecord? So, is that each thread will > create its own connection? Again..the question..arises does > backgroundrb has any business with dispatch.fcgi? > > > On 7/6/06, hemant wrote: > Hi All, > > I have this small doubt. Does backgroundrb have anything to do with > dispatch.fcgi? > > some dispatch.fcgi are taking memory as much as 186 MB .... > this is monstrous man... > > Also, if i put load rails environment with backgroundrb.yml file to > be false.What will happen? i just loses access to db through > ActiveRecord right? > I can still use normal mysql gem to access db and modify the > tables..correct? > > I should be able to create Middleman objects from Rails as usual? > > > > -- > nothing much to talk > > > > -- > nothing much to talk > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > -- > nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060706/f79036e3/attachment.html From gethemant at gmail.com Thu Jul 6 04:29:57 2006 From: gethemant at gmail.com (hemant) Date: Thu, 6 Jul 2006 13:59:57 +0530 Subject: [Backgroundrb-devel] memory uses In-Reply-To: <1702CE82-1504-4341-8C1D-5F7E9354DF6D@gmail.com> References: <1702CE82-1504-4341-8C1D-5F7E9354DF6D@gmail.com> Message-ID: Yes..i am running the latest version... and i will try to analyze the errors a little further and get back. On 7/6/06, Ezra Zygmuntowicz wrote: > > > > How many worker threads are you running? Are you using the latest code? > Maybe you can try this.. In the start script comment out this line: > > ActiveRecord::Base.allow_concurrency = true > > Comment that out and try again and let me know if you still get too many > connections. > > Thanks > -Ezra > > On Jul 6, 2006, at 12:20 AM, hemant wrote: > > Its running in development mode....and may be yes. > > But i am not any using any railr/ruby plugin/modules that can cause memory > leak. > But it is one of those big bang apps...i mean its a little complicated and > lots of wierd stuff thrown in. > > but after i stopped backgroundrb the error that i was getting stopped > coming.(So no more too many connections error, after stopping backgroundrb). > > > > On 7/6/06, Ezra Zygmuntowicz wrote: > > > > Hemant- > > No backgroundrb has nothing to do with dispatch.fcgi. Thats all rails in > > there. And BackgrounDRb only makes one connection to the db on startup. I > > will investigate to see if i can reproduce your error but it sounds like > > there may be an issue in your rails app. I do not think that each thread > > gets its own connection to the db since AR only conects once on startup. > > > > As for setting load_rails to false you are right, you will not be able > > to use your AR models. But if you require the mysql gem you will be able to > > use that like you want. > > > > Sounds like you have a memory leak in your rails app if its taking up > > 186Mb. Are you running in development mode or production? > > > > -Ezra > > > > On Jul 5, 2006, at 11:22 PM, hemant wrote: > > > > Also....there seems to be some problem with mysql connections.... > > > > I am getting "too many connections error - mysql", this could be either > > because of open connections to mysql - are not being closed. > > > > So the question...normally from rails when using dispatch.fcgi, only a > > limited number of connections are being opened to the database(people infact > > tell me, only one is opened). So, the big question is, Erza, how you are > > handling connections to the DB? > > > > I mean..when i use ActiveRecord? So, is that each thread will create its > > own connection? Again..the question..arises does backgroundrb has any > > business with dispatch.fcgi? > > > > > > On 7/6/06, hemant wrote: > > > > > > Hi All, > > > > > > I have this small doubt. Does backgroundrb have anything to do with > > > dispatch.fcgi? > > > > > > some dispatch.fcgi are taking memory as much as 186 MB .... > > > this is monstrous man... > > > > > > Also, if i put load rails environment with backgroundrb.yml file to be > > > false.What will happen? i just loses access to db through ActiveRecord > > > right? > > > I can still use normal mysql gem to access db and modify the > > > tables..correct? > > > > > > I should be able to create Middleman objects from Rails as usual? > > > > > > > > > > > > -- > > > nothing much to talk > > > > > > > > > > > -- > > nothing much to talk > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > -- > nothing much to talk > > > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060706/93591b6e/attachment.html From mklein at bpl.org Thu Jul 6 19:32:43 2006 From: mklein at bpl.org (Klein, Michael) Date: Thu, 6 Jul 2006 19:32:43 -0400 Subject: [Backgroundrb-devel] Worker Expiration/Timeout Strategies References: Message-ID: An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060706/3d7c9f14/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: backgroundrb_expire_type.patch Type: application/octet-stream Size: 973 bytes Desc: backgroundrb_expire_type.patch Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060706/3d7c9f14/attachment.obj From wmacfarl at gmail.com Fri Jul 7 09:22:12 2006 From: wmacfarl at gmail.com (William Macfarlane) Date: Fri, 7 Jul 2006 09:22:12 -0400 Subject: [Backgroundrb-devel] Fwd: installing backgroundrb In-Reply-To: <4514535e0607070620n1b9823e0r7b3b98bf22baac0a@mail.gmail.com> References: <4514535e0607070620n1b9823e0r7b3b98bf22baac0a@mail.gmail.com> Message-ID: <4514535e0607070622u33e1d1e0p60ff533f61e03375@mail.gmail.com> I'm trying to use BackgrounDRb to do image processing on large uploaded zip files of images, but $script/plugin install svn://rubyforge.org//var/svn/backgroundrb gives me this: /usr/lib/ruby/1.8/open-uri.rb:88:in `initialize': No such file or directory - svn://rubyforge.org//var/svn/backgroundrb (Errno::ENOENT) I can install the old version with $script/plugin install http://opensvn.csie.org/ezra/rails/backgroundrb but $rake backgroundrb:setup gives me this: rake aborted! undefined method `namespace' for # ./Rakefile:10 I'm running ruby 1.8.4 and rails 1.1 on an up to date gentoo installation. thanks, Will wmacfarl at gmail.com From ezmobius at gmail.com Fri Jul 7 13:33:42 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 7 Jul 2006 10:33:42 -0700 Subject: [Backgroundrb-devel] Fwd: installing backgroundrb In-Reply-To: <4514535e0607070622u33e1d1e0p60ff533f61e03375@mail.gmail.com> References: <4514535e0607070620n1b9823e0r7b3b98bf22baac0a@mail.gmail.com> <4514535e0607070622u33e1d1e0p60ff533f61e03375@mail.gmail.com> Message-ID: <6B89BA14-BD4D-4698-A88D-387F95312882@gmail.com> On Jul 7, 2006, at 6:22 AM, William Macfarlane wrote: > I'm trying to use BackgrounDRb to do image processing on large > uploaded zip files of images, but > > $script/plugin install svn://rubyforge.org//var/svn/backgroundrb > > gives me this: > > /usr/lib/ruby/1.8/open-uri.rb:88:in `initialize': No such file or > directory - svn://rubyforge.org//var/svn/backgroundrb (Errno::ENOENT) > > I can install the old version with > > $script/plugin install http://opensvn.csie.org/ezra/rails/backgroundrb > > but > > $rake backgroundrb:setup > > gives me this: > > rake aborted! > undefined method `namespace' for # > ./Rakefile:10 > > I'm running ruby 1.8.4 and rails 1.1 on an up to date gentoo > installation. > > thanks, > Will > wmacfarl at gmail.com Will- There must have been a problem at rubyforge. But it is ok now. I just tried installing it in a test rails app using the following command and it worked fine. Can you try it again please? The version at opensvn is hopelessly old and out of date. script/plugin install svn://rubyforge.org//var/svn/backgroundrb Cheers- -Ezra From Maximilian.Schoefmann at stud.ifi.lmu.de Fri Jul 7 13:52:32 2006 From: Maximilian.Schoefmann at stud.ifi.lmu.de (=?ISO-8859-15?Q?Maximilian_Sch=F6fmann?=) Date: Fri, 07 Jul 2006 19:52:32 +0200 Subject: [Backgroundrb-devel] Problems with ActiveRecord in workers Message-ID: <44AE9F60.1040500@stud.ifi.lmu.de> Hi *, are there some special settings except of "load_rails: true" to make ActiveRecord models accessible within my workers? I've just installed the latest version of this great plugin, but I keep getting undefined method [] for # (line 105 in backgroundrb.rb) when one of the arguments for the worker is an ActiveRecord object. I assume that the MiddleMan somehow doesn't see my models. I have already tried to put model :blabla, :foo, :bar in application.rb, but this didn't help. Any ideas? Regards, Max From ezmobius at gmail.com Fri Jul 7 14:00:26 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 7 Jul 2006 11:00:26 -0700 Subject: [Backgroundrb-devel] Problems with ActiveRecord in workers In-Reply-To: <44AE9F60.1040500@stud.ifi.lmu.de> References: <44AE9F60.1040500@stud.ifi.lmu.de> Message-ID: On Jul 7, 2006, at 10:52 AM, Maximilian Sch?fmann wrote: > Hi *, > > are there some special settings except of "load_rails: true" to make > ActiveRecord models accessible within my workers? > I've just installed the latest version of this great plugin, but I > keep > getting > > undefined method [] for # (line 105 in > backgroundrb.rb) > > when one of the arguments for the worker is an ActiveRecord object. > I assume that the MiddleMan somehow doesn't see my models. I have > already tried to put > > model :blabla, :foo, :bar > > in application.rb, but this didn't help. > > Any ideas? > > Regards, > Max Hey Max- If you need to pass an active record object to one of your workers as an argument then you need to add one line of code to each model that you intend on using this way. So if the model you want to send from rails to the drb server is Post then you need to add this line: class Post < ActiveRecord::base include DRbUndumped #the rewst of your code end If you add that to the models you need to send to the drb server it will work. Cheers- -Ezra From schoefma at cip.ifi.lmu.de Fri Jul 7 14:34:10 2006 From: schoefma at cip.ifi.lmu.de (=?ISO-8859-1?Q?Maximilian_Sch=F6fmann?=) Date: Fri, 07 Jul 2006 20:34:10 +0200 Subject: [Backgroundrb-devel] Problems with ActiveRecord in workers In-Reply-To: References: <44AE9F60.1040500@stud.ifi.lmu.de> Message-ID: <44AEA922.2090304@cip.ifi.lmu.de> Hi Ezra, thank a lot for the fast answer! I think this plugin is one of the most useful around. > If you need to pass an active record object to one of your workers > as an argument then you need to add one line of code to each model > that you intend on using this way. So if the model you want to send > from rails to the drb server is Post then you need to add this line: > > class Post < ActiveRecord::base > > include DRbUndumped If my understanding of DRb is correct, this will stop me from storing these objects in a session... As a workaround, I will try to store them with BackgroundDRb instead of the session. Thanks again, Max From ezmobius at gmail.com Fri Jul 7 14:54:25 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 7 Jul 2006 11:54:25 -0700 Subject: [Backgroundrb-devel] Problems with ActiveRecord in workers In-Reply-To: <44AEA922.2090304@cip.ifi.lmu.de> References: <44AE9F60.1040500@stud.ifi.lmu.de> <44AEA922.2090304@cip.ifi.lmu.de> Message-ID: On Jul 7, 2006, at 11:34 AM, Maximilian Sch?fmann wrote: > Hi Ezra, > > thank a lot for the fast answer! > I think this plugin is one of the most useful around. >> If you need to pass an active record object to one of your >> workers >> as an argument then you need to add one line of code to each model >> that you intend on using this way. So if the model you want to send >> from rails to the drb server is Post then you need to add this line: >> >> class Post < ActiveRecord::base >> >> include DRbUndumped > > If my understanding of DRb is correct, this will stop me from storing > these objects in a session... > As a workaround, I will try to store them with BackgroundDRb > instead of > the session. > > Thanks again, > > Max Max- I'm not so sure it would keep you from storing models in the session. You should experiment before you give up on that. BUt I don't like storing whole models in the session. I avoid it by just storing the id of a model in the session and using a before fiulter to repopulate the model from its id on each requests. Believe it or not this is usually faster then unmarshaling an object out of the session. Also what are you doing that requires you to send full AR objects to the drb server as args? Couldn't you just as easily send the id of the record or a name and then do a find in the drb server instead of sending a full AR object as an arg? Thanks for the kind words. I'm glad you find the plugin useful. I use it all the time now ;) Cheers- -Ezra From bjohnson at contuitive.com Fri Jul 7 16:11:17 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Fri, 7 Jul 2006 15:11:17 -0500 Subject: [Backgroundrb-devel] Problems installing, please help Message-ID: <09C2F5E9-E1AA-470E-A45D-1A6E18D8E5A8@contuitive.com> I am in my rails project directory and I ran the following command: script/plugin install svn://rubyforge.org//var/svn/backgroundrb and I get this: sh: svn: command not found Any idea how I can install backgroundrb or what the problem is? Any help is greatly appreciated. Thank You, Ben Johnson E: bjohnson at contuitive.com O: 800-341-6826 M: 817-229-4258 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060707/fbadd3a2/attachment.html From ezmobius at gmail.com Fri Jul 7 16:22:06 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 7 Jul 2006 13:22:06 -0700 Subject: [Backgroundrb-devel] Problems installing, please help In-Reply-To: <09C2F5E9-E1AA-470E-A45D-1A6E18D8E5A8@contuitive.com> References: <09C2F5E9-E1AA-470E-A45D-1A6E18D8E5A8@contuitive.com> Message-ID: <988ABD92-AB20-49D2-84BF-B1ED9E610A2C@gmail.com> On Jul 7, 2006, at 1:11 PM, Ben Johnson wrote: > I am in my rails project directory and I ran the following command: > > script/plugin install svn://rubyforge.org//var/svn/backgroundrb > > and I get this: > > sh: svn: command not found > > Any idea how I can install backgroundrb or what the problem is? > > Any help is greatly appreciated. > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com > O: 800-341-6826 > M: 817-229-4258 Ben - It looks like you don't have subversion installed on your computer? You can either install subversion which I highly recommend or I am also attaching a zip of the plugin so you can get it running now. Cheers- -Ezra ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060707/3d03b421/attachment-0002.html -------------- next part -------------- A non-text attachment was scrubbed... Name: backgroundrb.zip Type: application/zip Size: 105269 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060707/3d03b421/attachment-0001.zip -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060707/3d03b421/attachment-0003.html From gladwig at gmx.de Fri Jul 7 16:25:14 2006 From: gladwig at gmx.de (=?ISO-8859-1?Q?G=FCnter_Ladwig?=) Date: Fri, 7 Jul 2006 22:25:14 +0200 Subject: [Backgroundrb-devel] Problems installing, please help In-Reply-To: <09C2F5E9-E1AA-470E-A45D-1A6E18D8E5A8@contuitive.com> References: <09C2F5E9-E1AA-470E-A45D-1A6E18D8E5A8@contuitive.com> Message-ID: <9A1C877D-F774-4FF0-BF48-90F82B18168C@gmx.de> On 07.07.2006, at 22:11, Ben Johnson wrote: > I am in my rails project directory and I ran the following command: > > script/plugin install svn://rubyforge.org//var/svn/backgroundrb > > and I get this: > > sh: svn: command not found > > Any idea how I can install backgroundrb or what the problem is? > > Any help is greatly appreciated. You do not have Subversion installed: http://subversion.tigris.org Or use the package manager of your distribution (if you're using Linux). G?nter From bjohnson at contuitive.com Fri Jul 7 16:38:57 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Fri, 7 Jul 2006 15:38:57 -0500 Subject: [Backgroundrb-devel] uninitialized constant MiddleMan Message-ID: <4ADE051C-BE3C-4B93-90C5-02E1DA46558A@contuitive.com> I just updated backgroundrb to the latest version, and now I get this: uninitialized constant MiddleMan /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ active_support/dependencies.rb:123:in `const_missing' /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ active_support/dependencies.rb:131:in `const_missing' /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ active_support/dependencies.rb:133:in `const_missing' #{RAILS_ROOT}/app/models/client_dispatch.rb:3:in `get_worker' #{RAILS_ROOT}/app/models/event.rb:93:in `save_all_from_url!' #{RAILS_ROOT}/app/controllers/events_controller.rb:16:in `create' /usr/local/bin/mongrel_rails:1 This error occured while loading the following files: middle_man.rb Supposedly I dont have middle_man.rb any ideas? Thanks for your help. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060707/66b9a868/attachment.html From ezmobius at gmail.com Fri Jul 7 16:47:44 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 7 Jul 2006 13:47:44 -0700 Subject: [Backgroundrb-devel] uninitialized constant MiddleMan In-Reply-To: <4ADE051C-BE3C-4B93-90C5-02E1DA46558A@contuitive.com> References: <4ADE051C-BE3C-4B93-90C5-02E1DA46558A@contuitive.com> Message-ID: <84846F26-C944-4313-9E60-B29D207186A5@brainspl.at> On Jul 7, 2006, at 1:38 PM, Ben Johnson wrote: > I just updated backgroundrb to the latest version, and now I get this: > > uninitialized constant MiddleMan > > > /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ > active_support/dependencies.rb:123:in `const_missing' > /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ > active_support/dependencies.rb:131:in `const_missing' > /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ > active_support/dependencies.rb:133:in `const_missing' > #{RAILS_ROOT}/app/models/client_dispatch.rb:3:in `get_worker' > #{RAILS_ROOT}/app/models/event.rb:93:in `save_all_from_url!' > #{RAILS_ROOT}/app/controllers/events_controller.rb:16:in `create' > /usr/local/bin/mongrel_rails:1 > > > This error occured while loading the following files: > middle_man.rb > > > Supposedly I dont have middle_man.rb any ideas? > > Thanks for your help. > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com Ben- That is strange because there is no file called middle_man.rb. It looks like the plugin is not getting loaded for you on the rails side of things. Did you install the plugin in vendor/plugins/backgroundrb? If so then you should be good to go. I have never seen that error before so you will have to check and make sure you have installed it correctly. Also maybe include some snippets of code from your controllers where you call middleman so I can see what you are doing. -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060707/ba9b4967/attachment.html From ml at e4net.com Fri Jul 7 16:52:43 2006 From: ml at e4net.com (Jim Morris) Date: Fri, 07 Jul 2006 13:52:43 -0700 Subject: [Backgroundrb-devel] changing workers needs restart? Message-ID: <44AEC99B.6070206@e4net.com> Hi, Great tool BTW! I have noticed that changing the workers (ie editing them) requires restarting the backrounddb server. Is there a way to improve this for development environments? much like the way rails reloads in development mode without having to stop and restart webbrick? Thanks From ezmobius at gmail.com Fri Jul 7 17:03:09 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 7 Jul 2006 14:03:09 -0700 Subject: [Backgroundrb-devel] changing workers needs restart? In-Reply-To: <44AEC99B.6070206@e4net.com> References: <44AEC99B.6070206@e4net.com> Message-ID: <4A995359-9CC8-43D0-BAFC-6FE066B43FD3@gmail.com> On Jul 7, 2006, at 1:52 PM, Jim Morris wrote: > Hi, > > Great tool BTW! > > I have noticed that changing the workers (ie editing them) requires > restarting the backrounddb > server. Is there a way to improve this for development > environments? much like the way rails reloads > in development mode without having to stop and restart webbrick? > > Thanks Hey Kim- That my friend is a very good question ;-) I haven't come up with a way yet to make this happen. I will look into it this weekend though when I add a few patches and enhance a few features. I plan on adding breakpoint support so you can breakpoint and inspect your workers for debugging. I will see if I can come up with a way to get worker classes reloaded without a restart of the server. Although the way things work I think there will still have to be a command you send to MiddleMan to get it to reload your worker classes. I can't see a way to have it autoreload your workers without a bunch of hackery. Perhaps it could look like this: $ script/console >> MiddleMan.reload_workers! I'll see what I can come up with as this seems like a good thing to support. Cheers- -Ezra From schoefma at cip.ifi.lmu.de Sat Jul 8 04:33:24 2006 From: schoefma at cip.ifi.lmu.de (=?ISO-8859-1?Q?Maximilian_Sch=F6fmann?=) Date: Sat, 8 Jul 2006 10:33:24 +0200 Subject: [Backgroundrb-devel] Problems with ActiveRecord in workers In-Reply-To: References: <44AE9F60.1040500@stud.ifi.lmu.de> <44AEA922.2090304@cip.ifi.lmu.de> Message-ID: Hi Ezra, storing them in the session indeed doesn't work (I tried just PStore, but I think others will use Marshall, too). Marshalling them using YAML would maybe work. > Believe it or not this is usually faster then unmarshaling an > object out of the session. > I didn't consider that until now, because i thought fetching them from the db would be slower than unmarshalling. This would of course be an option for me. I will give it a try. Thanks for your suggestions Cheers, Max From ml at e4net.com Sat Jul 8 16:05:54 2006 From: ml at e4net.com (Jim Morris) Date: Sat, 08 Jul 2006 13:05:54 -0700 Subject: [Backgroundrb-devel] access to lib? Message-ID: <44B01022.9040105@e4net.com> Do I need to do anything special to access classes in RAILS_ROOT/lib from a worker? Maybe: require File.dirname(__FILE__) + '../my_lib.rb' or does it do that automatically like a controller does? Also if I write a unit test for a worker that is accessing AR, is there anything special needed to be done to access fixtures etc to test it? Thanks -- Jim Morris, http://blog.wolfman.com From bjohnson at contuitive.com Sat Jul 8 16:46:11 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Sat, 8 Jul 2006 15:46:11 -0500 Subject: [Backgroundrb-devel] Question about killing threads Message-ID: <4C1ADFA8-B237-436A-974C-0F9426CEA597@contuitive.com> If I do the following: threads = [] threads[0] << Thread.new do while true # some code end end # Will this kill off the first thread? threads[0] << Thread.new do while true # some more code end end Will the second assignment kill off the first thread? Or will that first thread keep going and never stop unless i explicitly kill it using the Thread.kill method? Thanks for your help. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060708/4a79d792/attachment.html From ml at e4net.com Sun Jul 9 01:59:11 2006 From: ml at e4net.com (Jim Morris) Date: Sat, 08 Jul 2006 22:59:11 -0700 Subject: [Backgroundrb-devel] where do uncaught exceptions printouts go? Message-ID: <44B09B2F.9000606@e4net.com> Hi, I have a worker, which should have thrown an uncaught exception and dumped the stack in the do_work method, however I saw no dump or indication of any error in any of the logs. This makes it very hard to debug, where does stdout and or stderr go? How canb I know if it gets some kind of error? (in this case not being able to find a class). Thanks -- Jim Morris, http://blog.wolfman.com From ml at e4net.com Sun Jul 9 02:19:44 2006 From: ml at e4net.com (Jim Morris) Date: Sat, 08 Jul 2006 23:19:44 -0700 Subject: [Backgroundrb-devel] access to lib? In-Reply-To: <44B01022.9040105@e4net.com> References: <44B01022.9040105@e4net.com> Message-ID: <44B0A000.80803@e4net.com> To answer my own question... Yes you do need to require any library you use with... require File.dirname(__FILE__) + '/../testlib.rb' The problem is if you don't, the task silently fails and you have no idea why, as the expected error is lost somewhere (see my other posting). Jim Morris wrote: > Do I need to do anything special to access classes in RAILS_ROOT/lib from a worker? > > Maybe: > require File.dirname(__FILE__) + '../my_lib.rb' > > or does it do that automatically like a controller does? > > Also if I write a unit test for a worker that is accessing AR, is there anything special needed to > be done to access fixtures etc to test it? > > Thanks > -- Jim Morris, http://blog.wolfman.com From gethemant at gmail.com Sun Jul 9 02:57:46 2006 From: gethemant at gmail.com (hemant) Date: Sun, 9 Jul 2006 12:27:46 +0530 Subject: [Backgroundrb-devel] where do uncaught exceptions printouts go? In-Reply-To: <44B09B2F.9000606@e4net.com> References: <44B09B2F.9000606@e4net.com> Message-ID: That is one problem with Backgroundrb with, unhandled exceptions are getting lost in wilderland. But ofcourse, whatever Erza would say will be more authorative. On 7/9/06, Jim Morris wrote: > > Hi, > > I have a worker, which should have thrown an uncaught exception and dumped > the stack in the do_work > method, however I saw no dump or indication of any error in any of the > logs. > > This makes it very hard to debug, where does stdout and or stderr go? How > canb I know if it gets > some kind of error? (in this case not being able to find a class). > > Thanks > > -- > Jim Morris, http://blog.wolfman.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060709/1acc7704/attachment.html From mklein at bpl.org Sun Jul 9 12:41:13 2006 From: mklein at bpl.org (Klein, Michael) Date: Sun, 9 Jul 2006 12:41:13 -0400 Subject: [Backgroundrb-devel] where do uncaught exceptions printouts go? References: <44B09B2F.9000606@e4net.com> Message-ID: An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060709/ea92b667/attachment.html From ezmobius at gmail.com Sun Jul 9 21:44:06 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sun, 9 Jul 2006 18:44:06 -0700 Subject: [Backgroundrb-devel] where do uncaught exceptions printouts go? In-Reply-To: References: <44B09B2F.9000606@e4net.com> Message-ID: <8719482C-2D6A-4807-A176-52E1561E2AD4@brainspl.at> Ok, I just committed an update to the plugin. I fixed exception handling so if your workers throw an error the full error message, class of error and complete backtrace will get printed to the log file. Sorry about the hassle finding this bug. It happened when I added the job canceling and handling code. I forgot to put the begin rescue end block inside of the Thread.new block. Its all fixed now. I also added Michael Klien's accessed based timestamps patch. It doesn't change the external interface to how you use things but it changes the way timestamps are stored. Now timestamps are stored as a hash of attributes instead of an array. This allows workers expiry to act like sessions in that the time to live will be updated evey time a worker is accessed. So it will update the timestamp when it gets accessed and push the time to live forward again. This allows the worker to expire based on time to live since the worker was last accessed. So this worker will expire 300 seconds after the last time it was accessed. session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, :args => "Bar", :ttl => 300, :expire_type => :accessed ) I also added a CONTRIBUTORS file to the plugin. Please have a look and remind me if I missed anyone who had a patch or enhancement applied. I am in the process of adding breakpoint support to the plugin so you can set breakpoints and inspect your workers and other state. Along with this will be a small stats system so you can get stats about all your running workers and timestamps and what the drb server is working on. Should have this stuff done this week some time. Cheers- -Ezra PS. Thanks everyone for using and abusing this to help ferret out any bugs and missing features. This is a fun plugin to work on and with and I think it will only get better. From aela at loc.gov Mon Jul 10 16:39:31 2006 From: aela at loc.gov (Aravind Elango) Date: Mon, 10 Jul 2006 16:39:31 -0400 Subject: [Backgroundrb-devel] 'uninitialized constant' error Message-ID: Hi All, I am fairly new to Ruby and backgroundRB. I am trying to run a simple example in Windows to get my feet wet with backgroundRB and am running into an 'uninitialized constant' error when I invoke the controller on my rails app. In 'MyTest' Controller ------------------- def longrun session[:job_key] = MiddleMan.new_worker(:class => :tail_worker, :args => {:baz => 'hello!', :qux => 'another arg!'}) end In cookbook\lib\workers\tail_worker.rb ----------------------------------------------- class TailWorker < BackgrounDRb::Rails def do_work(args) puts "Printing from inside TailWorker::do_work()" end end When I issue the request to http://127.0.0.1:3000/My_Test/longrun I get the following error: NameError in My testController#longrun uninitialized constant TailWorker I'd appreciate any light thrown on what I might have done to cause this error and about how to rectify it. Thanks, Aravind Elango From ezmobius at gmail.com Mon Jul 10 17:21:18 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 10 Jul 2006 14:21:18 -0700 Subject: [Backgroundrb-devel] 'uninitialized constant' error In-Reply-To: References: Message-ID: On Jul 10, 2006, at 1:39 PM, Aravind Elango wrote: > Hi All, > > I am fairly new to Ruby and backgroundRB. I am trying to run a > simple example in Windows to get my feet wet with backgroundRB and > am running into an 'uninitialized constant' error when I invoke the > controller on my rails app. > > In 'MyTest' Controller > ------------------- > def longrun > session[:job_key] = MiddleMan.new_worker(:class > => :tail_worker, :args => {:baz => 'hello!', :qux => 'another arg!'}) > end > > In cookbook\lib\workers\tail_worker.rb > ----------------------------------------------- > class TailWorker < BackgrounDRb::Rails > def do_work(args) > puts "Printing from inside TailWorker::do_work()" > end > end > > > When I issue the request to > http://127.0.0.1:3000/My_Test/longrun > I get the following error: > NameError in My testController#longrun > uninitialized constant TailWorker > > > I'd appreciate any light thrown on what I might have done to cause > this error and about how to rectify it. > > Thanks, > Aravind Elango Hi Aravind and welcome ;) Your code looks like it should work fine. Did you make sure to restart the backgroundrb server after you added the tails_worker.rb file to lib/workers? That could cause this error. Cheers- -Ezra From aela at loc.gov Mon Jul 10 20:01:22 2006 From: aela at loc.gov (Aravind Elango) Date: Mon, 10 Jul 2006 20:01:22 -0400 Subject: [Backgroundrb-devel] 'uninitialized constant' error Message-ID: Gotcha. Restarting the server worked. Thanks Ezra :-) -Aravind >>> Ezra Zygmuntowicz 07/10/06 5:21 PM >>> On Jul 10, 2006, at 1:39 PM, Aravind Elango wrote: > Hi All, > > I am fairly new to Ruby and backgroundRB. I am trying to run a > simple example in Windows to get my feet wet with backgroundRB and > am running into an 'uninitialized constant' error when I invoke the > controller on my rails app. > > In 'MyTest' Controller > ------------------- > def longrun > session[:job_key] = MiddleMan.new_worker(:class > => :tail_worker, :args => {:baz => 'hello!', :qux => 'another arg!'}) > end > > In cookbook\lib\workers\tail_worker.rb > ----------------------------------------------- > class TailWorker < BackgrounDRb::Rails > def do_work(args) > puts "Printing from inside TailWorker::do_work()" > end > end > > > When I issue the request to > http://127.0.0.1:3000/My_Test/longrun > I get the following error: > NameError in My testController#longrun > uninitialized constant TailWorker > > > I'd appreciate any light thrown on what I might have done to cause > this error and about how to rectify it. > > Thanks, > Aravind Elango Hi Aravind and welcome ;) Your code looks like it should work fine. Did you make sure to restart the backgroundrb server after you added the tails_worker.rb file to lib/workers? That could cause this error. Cheers- -Ezra From bjohnson at contuitive.com Wed Jul 12 01:37:03 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Wed, 12 Jul 2006 00:37:03 -0500 Subject: [Backgroundrb-devel] When to use Mutex::synchronize? Message-ID: <0C53C458-84E3-4ADB-9F04-FE9375212DEF@contuitive.com> I have a simple question when to the synchronize method in the Mutex class. Now that backgroundrb has allow_concurrency = true there is no need to synchronize database calls in threads. The question I have is lets say I have a simple method in my worker as follows: def some_method SomeModel.find_all each do |obj| obj.some_count += 1 obj.save! end end It accesses the database, but is not in a thread. Since this is a single process running in the background, if that method gets called simultaneously will the database lose connection? Will I get unexpected results? For instance, if 500 people access www.somedomain.com/controller/method which calls that method in the backgroundrb process will it screw everything up? Basically do I need to wrap that in a synchronize method? Like this: def some_method @mutex_obj.snchronize do SomeModel.find_all each do |obj| obj.some_count += 1 obj.save! end end end Thanks a lot for your help. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060712/7097d3ed/attachment-0001.html From bjohnson at contuitive.com Wed Jul 12 01:42:46 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Wed, 12 Jul 2006 00:42:46 -0500 Subject: [Backgroundrb-devel] Initialize limited? Message-ID: <3D890E3E-6F66-4366-8E25-2FC796F41C8B@contuitive.com> Can I not do things with the models in my initialize method? I have this: def initialize(args) super args i = 0 Event.find_all do |event| @lock = i i += 1 end end i is null and there are events in the database. Any ideas how I can use my models in the initialize method? Thank You, Ben Johnson E: bjohnson at contuitive.com O: 800-341-6826 M: 817-229-4258 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060712/4dc5fd55/attachment.html From ezmobius at gmail.com Wed Jul 12 11:37:52 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 12 Jul 2006 08:37:52 -0700 Subject: [Backgroundrb-devel] When to use Mutex::synchronize? In-Reply-To: <0C53C458-84E3-4ADB-9F04-FE9375212DEF@contuitive.com> References: <0C53C458-84E3-4ADB-9F04-FE9375212DEF@contuitive.com> Message-ID: <508BF7A9-46D1-4B04-9B87-323057F3377D@brainspl.at> On Jul 11, 2006, at 10:37 PM, Ben Johnson wrote: > I have a simple question when to the synchronize method in the > Mutex class. > > Now that backgroundrb has allow_concurrency = true there is no need > to synchronize database calls in threads. > > The question I have is lets say I have a simple method in my worker > as follows: > > def some_method > SomeModel.find_all each do |obj| > obj.some_count += 1 > obj.save! > end > end > > It accesses the database, but is not in a thread. Since this is a > single process running in the background, if that method gets > called simultaneously will the database lose connection? Will I get > unexpected results? For instance, if 500 people access > www.somedomain.com/controller/method which calls that method in the > backgroundrb process will it screw everything up? > > Basically do I need to wrap that in a synchronize method? Like this: > > def some_method > @mutex_obj.snchronize do > SomeModel.find_all each do |obj| > obj.some_count += 1 > obj.save! > end > end > end > > Thanks a lot for your help. > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com > Ben- It all depends on what you want to happen. If you only want one call to that method at a time to be running then yes you will want to synchronize access to that method. You will need to create your own mutex object. Also I will answer your other question at the same time here as it looks like they are related. You asked: "Can I not do things with the models in my initialize method? I have this: def initialize(args) super args i = 0 Event.find_all do |event| @lock = i i += 1 end end " Yes you can use your db models in initialize but you must do it slightly differently then you have done here. This is also where you would create a mutex for your worker to use to guard certain actions. class BenWorker < BackgrounDRb::Rails def initialize(args) @mymutex = Mutex.new Event.find_all do |event| @lock = i i += 1 end super end def some_method @mymutex.snchronize do SomeModel.find_all each do |obj| obj.some_count += 1 obj.save! end end end end So you must do whatever it is you need to do in initialize before you call super. ANd call super without any args. Because when you call super it spins off the thread to start running do_work inside of. Now since your some_method method is making changes to the db then yes you should synchronize access to it or else you will get undefined results. If the method was just checking for progress or doing something that just reads state and doesn't change it then you can probably get away without a mutex. Just think about what you are doing carefully. The more mutex locking you build the slower and more complex things get. Multi threading is hard stuff ;) Cheers- -Ezra > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060712/ea430607/attachment.html From ml at e4net.com Wed Jul 12 18:36:40 2006 From: ml at e4net.com (Jim Morris) Date: Wed, 12 Jul 2006 15:36:40 -0700 Subject: [Backgroundrb-devel] where do uncaught exceptions printouts go? In-Reply-To: <8719482C-2D6A-4807-A176-52E1561E2AD4@brainspl.at> References: <44B09B2F.9000606@e4net.com> <8719482C-2D6A-4807-A176-52E1561E2AD4@brainspl.at> Message-ID: <44B57978.4060504@e4net.com> Was this update checked into rubyforge, because script/plugin update says the plugin doesn't exist? Ezra Zygmuntowicz wrote: > Ok, I just committed an update to the plugin. I fixed exception > handling so if your workers throw an error the full error message, > class of error and complete backtrace will get printed to the log > file. Sorry about the hassle finding this bug. It happened when I > added the job canceling and handling code. I forgot to put the begin > rescue end block inside of the Thread.new block. Its all fixed now. > > I also added Michael Klien's accessed based timestamps patch. It > doesn't change the external interface to how you use things but it > changes the way timestamps are stored. Now timestamps are stored as a > hash of attributes instead of an array. This allows workers expiry to > act like sessions in that the time to live will be updated evey time > a worker is accessed. So it will update the timestamp when it gets > accessed and push the time to live forward again. This allows the > worker to expire based on time to live since the worker was last > accessed. So this worker will expire 300 seconds after the last time > it was accessed. > > session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, > > :args => "Bar", > > :ttl => 300, > > :expire_type => :accessed ) > > > I also added a CONTRIBUTORS file to the plugin. Please have a look > and remind me if I missed anyone who had a patch or enhancement applied. > > I am in the process of adding breakpoint support to the plugin so > you can set breakpoints and inspect your workers and other state. > Along with this will be a small stats system so you can get stats > about all your running workers and timestamps and what the drb server > is working on. Should have this stuff done this week some time. > > > Cheers- > -Ezra > > PS. Thanks everyone for using and abusing this to help ferret out any > bugs and missing features. This is a fun plugin to work on and with > and I think it will only get better. > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Jim Morris, http://blog.wolfman.com From ml at e4net.com Wed Jul 12 18:50:31 2006 From: ml at e4net.com (Jim Morris) Date: Wed, 12 Jul 2006 15:50:31 -0700 Subject: [Backgroundrb-devel] where do uncaught exceptions printouts go? In-Reply-To: <91750C14-DB70-4A48-BDAD-BDED1920CED6@gmail.com> References: <44B09B2F.9000606@e4net.com> <8719482C-2D6A-4807-A176-52E1561E2AD4@brainspl.at> <44B57978.4060504@e4net.com> <91750C14-DB70-4A48-BDAD-BDED1920CED6@gmail.com> Message-ID: <44B57CB7.7060601@e4net.com> Ok, actually another way which works is... > script/plugin install --force svn://rubyforge.org//var/svn/backgroundrb This re-installed and I now get the exception dump in the log, thanks. Ezra Zygmuntowicz wrote: > Jim- > > I don't think the script/plugin will find this plugin because of the > way it works. You will have to either cd into > vendor/plugins/backgroundrb and do an svn up or do a fresh svn co of the > plugin and put it in vendor/plugins. > > Cheers- > -Ezra > > > On Jul 12, 2006, at 3:36 PM, Jim Morris wrote: > >> Was this update checked into rubyforge, because script/plugin update >> says the plugin doesn't exist? >> >> Ezra Zygmuntowicz wrote: >>> Ok, I just committed an update to the plugin. I fixed exception >>> handling so if your workers throw an error the full error message, >>> class of error and complete backtrace will get printed to the log >>> file. Sorry about the hassle finding this bug. It happened when I >>> added the job canceling and handling code. I forgot to put the begin >>> rescue end block inside of the Thread.new block. Its all fixed now. >>> >>> I also added Michael Klien's accessed based timestamps patch. It >>> doesn't change the external interface to how you use things but it >>> changes the way timestamps are stored. Now timestamps are stored as a >>> hash of attributes instead of an array. This allows workers expiry to >>> act like sessions in that the time to live will be updated evey time >>> a worker is accessed. So it will update the timestamp when it gets >>> accessed and push the time to live forward again. This allows the >>> worker to expire based on time to live since the worker was last >>> accessed. So this worker will expire 300 seconds after the last time >>> it was accessed. >>> >>> session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, >>> >>> :args => "Bar", >>> >>> :ttl => 300, >>> >>> :expire_type => :accessed ) >>> >>> >>> I also added a CONTRIBUTORS file to the plugin. Please have a look >>> and remind me if I missed anyone who had a patch or enhancement applied. >>> >>> I am in the process of adding breakpoint support to the plugin so >>> you can set breakpoints and inspect your workers and other state. >>> Along with this will be a small stats system so you can get stats >>> about all your running workers and timestamps and what the drb server >>> is working on. Should have this stuff done this week some time. >>> >>> >>> Cheers- >>> -Ezra >>> >>> PS. Thanks everyone for using and abusing this to help ferret out any >>> bugs and missing features. This is a fun plugin to work on and with >>> and I think it will only get better. >>> _______________________________________________ >>> Backgroundrb-devel mailing list >>> Backgroundrb-devel at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >> >> --Jim Morris, http://blog.wolfman.com >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> > > -- Jim Morris, http://blog.wolfman.com From ezmobius at gmail.com Thu Jul 13 00:23:22 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 12 Jul 2006 21:23:22 -0700 Subject: [Backgroundrb-devel] Update Message-ID: Folks- I just commited another revision of the plugin. Nothing huge in this update but some extra features. Acl lists are now configurable in the config file. So you will want to blow away and recreate your config file again. Thanks to Georg Friedrich for this. And I applied Gunter Ladwig's patch for singleton worker classes. These are workers that only have one instance of the class in the whole drb server. Even if you call new_worker more then once with the same :class and it is a singleton worker, it will just return the one worker instance. Use like this: MiddleMan.new_worker(:class => :foo_worker, :args => "Bar", :job_key => :singleton_worker, :singleton => true) Cheers- -Ezra From jacob.patton at gmail.com Thu Jul 13 20:22:27 2006 From: jacob.patton at gmail.com (Jacob Patton) Date: Thu, 13 Jul 2006 20:22:27 -0400 Subject: [Backgroundrb-devel] Using BackgrounDRb to replace nested loops? Message-ID: In response to my Rails Weenie post today[1], someone mentioned that I try BackgrounDRb to help take the strain off my application. 1: http://rails.techno-weenie.net/question/2006/7/13/nested-loops- best-way-to-send-messages-to-blog-subscribers I've just tried to install and use BackgrounDRb with my app, but I'm afraid that I'm not too versed in running processes like these, and I can't get the caching to work--I get this error: private method `cache' called for #:BackgrounDRb::MiddleMan Would someone help me by looking over my Rails Weenie question, below, and recommend to me whether BackgrounDRb will help me with my problem? (Thanks for the great site, as always, Ezra, and it was good meeting you after the Workshop for Good Event, too!) From the Rails Weenie post: I run a members-only blog where subscribers can opt to receive messages in given categories via email in real-time or daily/weekly digests. What?s the best way to deliver the digest messages without pushing my app into an ugly, processor-hungry loop? Each message belongs to a given category, and each subscriber can subscribe to 0-n categories to receive updates. Right now the daily emails are sent as soon as a message is posted: 1. # within the create action...after the message is saved: 2. 3. # get subscribers to this category 4. @subscribers = @msg.category.subscribers 5. 6. # loop through the subscribers, delivering the message to each one 7. for subscriber in @subscribers 8. MsgNotifier.deliver_update(@msg) 9. end This causes a slight delay upon posting--the message has to be sent to all the subscribers for this category--but it works. (I'm not terribly concerned about the delay in this part, for now, but if anyone has a suggestion on how to improve things, I'd love to hear it.) The real problem arises, however, when I'm trying to compile the messages for daily and weekly digests: 1. # get all real-time subscribers 2. @subscribers = User.find_all_by_freq('real-time') 3. 4. # loop through subscribers 5. for subscriber in @subscribers 6. 7. @text = nil 8. 9. # get categories for subscriber 10. @categories = subscriber.categories 11. 12. # loop through categories 13. for category in @categories 14. 15. # get messages posted in that category in the past day 16. @msgs = category.messages(:conditions => "[created_at > ?], 1.day.ago") 17. 18. # loop through messages 19. for msg in @msgs 20. 21. # concatenate the messages for that category 22. @text += msg.body 23. end 24. end 25. end 26. 27. # deliver subscriber's update containing all of the past days messages 28. MsgNotifier.deliver_daily(@subscriber, @text) 29. 30. # unset @text 31. @text = nil 32. end As you can see, this is really, really ugly. Is there a better way to gather together the content for the digest messages? I'm open to any suggestions--this code is very, very slow right now... = Best, Jacob Patton From ezmobius at gmail.com Thu Jul 13 23:11:36 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 13 Jul 2006 20:11:36 -0700 Subject: [Backgroundrb-devel] Using BackgrounDRb to replace nested loops? In-Reply-To: References: Message-ID: <47A09899-1738-4965-A867-51B5290D2D24@brainspl.at> Hi- Answers inline: On Jul 13, 2006, at 5:22 PM, Jacob Patton wrote: > In response to my Rails Weenie post today[1], someone mentioned that > I try BackgrounDRb to help take the strain off my application. > > 1: http://rails.techno-weenie.net/question/2006/7/13/nested-loops- > best-way-to-send-messages-to-blog-subscribers > > I've just tried to install and use BackgrounDRb with my app, but I'm > afraid that I'm not too versed in running processes like these, and I > can't get the caching to work--I get this error: private method > `cache' called for # 0x254f9b0>:BackgrounDRb::MiddleMan I apologize for this. A recent change introduced this bug. Please svn up or co the plugin again and try again. Caching will work as advertised now. But I don't think caching is the answer to your problem. I think you would be best served by creating a worker class that did the compilation of messages and sent the emails in the backgroundrb server. I also think that every 5 or 10 minutes is real time enough for most things like this. So you might be better served by building and sending out the email digests at intervals and not with every time a post gets updated. So since you need to use ActionMailer, which in turn needs ActionView, you will want to pull most of rails into your drb server so you can make all this happen. So add this line to your script/ backgroundrb/start script at the top after the boot.rb require line: require File.dirname(__FILE__) + "/../../config/environment.rb" Then build a worker class that does the processing and sending of emails in the do_work method. class EmailWorker < BackgrounDRb::Rails def do_work(args) # loop over emails and build digest # loop over subscribers and send email # end end You will need a cron job to fire off this workers method every 5 or 10 minutes like this: #!/usr/bin/env ruby require "drb" DRb.start_service MiddleMan = DRbObject.new(nil, "druby://localhost:22222") # setting :ttl to 600 seconds/10 minutes. Make sure you set this for long enough time for your worker to finish the emails # but you need to set this so the workers will get garbage collected after they are done. MiddleMan.new_worker(:class => :email_worker, :ttl => 600) See if this gets you started and ask if you get stuck. Cheers- -Ezra > > Best, > > Jacob Patton > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From jacob.patton at gmail.com Mon Jul 17 10:09:42 2006 From: jacob.patton at gmail.com (Jacob Patton) Date: Mon, 17 Jul 2006 10:09:42 -0400 Subject: [Backgroundrb-devel] Using BackgrounDRb to replace nested loops? In-Reply-To: <47A09899-1738-4965-A867-51B5290D2D24@brainspl.at> References: <47A09899-1738-4965-A867-51B5290D2D24@brainspl.at> Message-ID: <7E00635C-48A3-490E-A147-41AC9219754D@gmail.com> Ezra thanks for your help and advice. I'll try out the steps you suggest, and I'll report back to the list. Best, Jacob On Jul 13, 2006, at 11:11 PM, Ezra Zygmuntowicz wrote: > Hi- > > Answers inline: > > On Jul 13, 2006, at 5:22 PM, Jacob Patton wrote: > >> In response to my Rails Weenie post today[1], someone mentioned that >> I try BackgrounDRb to help take the strain off my application. >> >> 1: http://rails.techno-weenie.net/question/2006/7/13/nested-loops- >> best-way-to-send-messages-to-blog-subscribers >> >> I've just tried to install and use BackgrounDRb with my app, but I'm >> afraid that I'm not too versed in running processes like these, and I >> can't get the caching to work--I get this error: private method >> `cache' called for #> 0x254f9b0>:BackgrounDRb::MiddleMan > > I apologize for this. A recent change introduced this bug. Please > svn up or co the plugin again and try again. Caching will work as > advertised now. But I don't think caching is the answer to your > problem. > > I think you would be best served by creating a worker class that > did the compilation of messages and sent the emails in the > backgroundrb server. I also think that every 5 or 10 minutes is > real time enough for most things like this. So you might be better > served by building and sending out the email digests at intervals > and not with every time a post gets updated. > > So since you need to use ActionMailer, which in turn needs > ActionView, you will want to pull most of rails into your drb > server so you can make all this happen. So add this line to your > script/backgroundrb/start script at the top after the boot.rb > require line: > > require File.dirname(__FILE__) + "/../../config/environment.rb" > > Then build a worker class that does the processing and sending of > emails in the do_work method. > > class EmailWorker < BackgrounDRb::Rails > > def do_work(args) > # loop over emails and build digest > # loop over subscribers and send email > # > end > > end > > You will need a cron job to fire off this workers method every 5 or > 10 minutes like this: > > #!/usr/bin/env ruby > require "drb" > DRb.start_service > MiddleMan = DRbObject.new(nil, "druby://localhost:22222") > # setting :ttl to 600 seconds/10 minutes. Make sure you set this > for long enough time for your worker to finish the emails > # but you need to set this so the workers will get garbage > collected after they are done. > MiddleMan.new_worker(:class => :email_worker, :ttl => 600) > > > > See if this gets you started and ask if you get stuck. > > Cheers- > -Ezra > > >> >> Best, >> >> Jacob Patton >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> > From bjohnson at contuitive.com Mon Jul 17 18:49:44 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Mon, 17 Jul 2006 17:49:44 -0500 Subject: [Backgroundrb-devel] Very strange after_save problem. Please help. Message-ID: <3DE3EBDF-CE4B-4027-BC0A-460A3B3B5926@contuitive.com> I have a very strange problem here. I do not get this. So any help is greatly appreciated. Basically I have a model that calls a method in the background process in the "after_save" method. Let's call the model products. So what happens is this: 1. I create a new product. 2. Everything works and the product is saved with id 13. 3. A method is called in my background process passing it the NEW product id in the after_save method. 4. That method in the background process says "Can't find Product with id=13". 5. An exception is then passed back to my after_save method and the product rolls back like adding it never happened. Because when an exception is raised in after_save it rolls back. So I figured maybe it wasn't connecting to the database or something. I adding a product in the db manually and added the following in my backgroundrb method: p = Product.find_all raise p.inspect I try to add another product and it shows me that product I just added into the database manually but not the one that was just created through the product model. So the backgrond process is connecting to the database. So in my after_save method i did the following to make sure it was getting into the database: p = Product.find id raise p.inspect I tried to add a product again and it raised an exception with the product information that I JUST added. Why is backgroundrb not seeing new additions to the database when being called from after_save? It's like backgroundrb doesn't acknowledge it's existence until after_save is finished. Anyways, if this is the case, is there a work around so that I can invoke a backgroundrb method on a freshly saved product? Thanks a lot for your help. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060717/fb4082e8/attachment.html From ezmobius at gmail.com Mon Jul 17 19:14:50 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 17 Jul 2006 16:14:50 -0700 Subject: [Backgroundrb-devel] Very strange after_save problem. Please help. In-Reply-To: <3DE3EBDF-CE4B-4027-BC0A-460A3B3B5926@contuitive.com> References: <3DE3EBDF-CE4B-4027-BC0A-460A3B3B5926@contuitive.com> Message-ID: On Jul 17, 2006, at 3:49 PM, Ben Johnson wrote: > I have a very strange problem here. I do not get this. So any help > is greatly appreciated. > > Basically I have a model that calls a method in the background > process in the "after_save" method. Let's call the model products. > So what happens is this: > > 1. I create a new product. > 2. Everything works and the product is saved with id 13. > 3. A method is called in my background process passing it the NEW > product id in the after_save method. > 4. That method in the background process says "Can't find Product > with id=13". > 5. An exception is then passed back to my after_save method and the > product rolls back like adding it never happened. Because when an > exception is raised in after_save it rolls back. > > So I figured maybe it wasn't connecting to the database or > something. I adding a product in the db manually and added the > following in my backgroundrb method: > > p = Product.find_all > raise p.inspect > > I try to add another product and it shows me that product I just > added into the database manually but not the one that was just > created through the product model. So the backgrond process is > connecting to the database. > > So in my after_save method i did the following to make sure it was > getting into the database: > > p = Product.find id > raise p.inspect > > I tried to add a product again and it raised an exception with the > product information that I JUST added. > > Why is backgroundrb not seeing new additions to the database when > being called from after_save? It's like backgroundrb doesn't > acknowledge it's existence until after_save is finished. > > Anyways, if this is the case, is there a work around so that I can > invoke a backgroundrb method on a freshly saved product? > > Thanks a lot for your help. > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com Hey Ben- I'm not 100% certain but what I think is happening is that since the after_save is run inside the transaction for that save, the new item is not commited to the db because the after_save is failing. Now i think whats happening is that since the save has not been commited in rails yet, it isn't actually in the db yet so your background worker cannot see it yet. I think that instead of trying to do this in after_save that maybe you should put the call to your background worker in an AR Observer that runs the background task *after* the save has been commited. Basically I think you are short circuiting yourself becazsue the record is not saved to the db until the transaction is commited but your drb task cannot find that record until after it is commited. Make any sense? -Ezra > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060717/f3266cb0/attachment-0001.html From ezmobius at gmail.com Tue Jul 18 11:29:58 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 18 Jul 2006 08:29:58 -0700 Subject: [Backgroundrb-devel] Two backgroundrb questions In-Reply-To: References: Message-ID: <20394852-E552-4E5C-8A01-3F5A20AE8BB8@gmail.com> On Jul 18, 2006, at 6:24 AM, Matt Mower wrote: > Hi Ezra, > > I'm converting some code I have which basically ran Enumerable#map > asynchronously using a thread-per-value to use backgroundrb > (recommended by zedas). That code works but I have a scaling problem > and needed to introduce throttling. > > However now that I've looked at it I don't see any sign of throttling > in backgroundrb. It looks like there is a 1:1 relationship between > workers and threads. Perhaps zed misunderstood what I was looking > for... > > Also I don't see how: > > MiddleMan.get_worker(session[:job_key]).progress > > or > > MiddleMan.get_worker(session[:job_key]).results > > described in the documentation work. Where in the worker are you > controlling the value of progress or results? Nothing in the > definition of Backgroundrb::Rails seems to support this. Am I missing > some magic? > > The only docs I seem to be able to find are: > > http://backgroundrb.rubyforge.org/ > > Sorry if I'm being dense and/or inobservant. > > Cheers, > > Matt Hey Matt- Yeah you are right, there is a one worker per thread mapping in backgroundrb. So I am not clear exactly what you need to achieve with it. Can you give me a code sample or an outline of what you need to accomplish? As far as progress or results goes, you are left to take care of that yourself if you want to use progress bars. Here is an example worker that uses progress and result: class Worker < BackgrounDRb::Rails attr_reader :result attr_reader :progress def do_work(args) @progress = 0 @result = args[:text] while @progress < 100 sleep rand / 2 # do some work here and store it in @result @progress += 1 end end def progress @progress end end The do_work() method is called within the thread for that worker so it can do its processing loop without blocking the call within rails. I'm not sure if it is the best match for trying to spawn hundreds of threads. But if you need to do something like that then it is possible. Maybe if you can explain what you want to accomplish I can help you find the right course of action. It could be that backgroundrb is not the solution you need and maybe you need to make a custom drb server that is lightweight and just handles your passed blocks. Cheers- -Ezra From ezmobius at gmail.com Tue Jul 18 11:40:34 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 18 Jul 2006 08:40:34 -0700 Subject: [Backgroundrb-devel] Very strange after_save problem. Please help. In-Reply-To: References: <3DE3EBDF-CE4B-4027-BC0A-460A3B3B5926@contuitive.com> Message-ID: <0EC697C4-0988-47BE-B8CB-429317229D4A@gmail.com> On Jul 18, 2006, at 6:54 AM, Christian Romney wrote: > Hi all, > I'm looking for a little deployment guidance. I have a webfarm with > two app servers running rails that need to make use of BackgrounDRb. > I presume I should have a single machine running backgroundrb, > since otherwise we would run the chance of a client switching > servers and polling for a non-existent job. > Assuming I need to change the backgroundrb.yml file on the two app > servers to point to an instance running on some third machine, I'm > a little lost on what each config should look like. > (The config on the app servers vs. the config on the drb server). > Any guidance is much appreciated. Alternatively if there's an > effective way to "cluster" backgroundrb instances and maintain a > global, shared keyspace (thinking of Mogilefs, for instance) I'd > welcome suggestions as well. > > Thanks for releasing such a great tool! > Christian Romney > Hey Christian- Yeah you will want to run the drb server one either one of the two app servers or a third box depending on your setup. THe way things work is that backgroundrb is tied to your rails app so if you run it on a third box you will still need to check out a copy of your app onto that third box. You just wont run the rails app part of it and only the drb server. In the config file you will need to set your IP addresses and your acl lists correctly. Here is a config file. You will want to change probably the port number, the IP address and the acl list. Lets say you will run the drb server on 192.168.0.3 and your two app servers are 192.168.0.1 & 192.168.0.2. --- port: "22222" timer_sleep: 60 load_rails: true environment: development host: 192.168.0.3 database_yml: config/database.yml acl: deny: all allow: localhost 127.0.0.1 192.168.0.1 192.168.0.2 order: deny,allow You will need to make sure that the backgroundrb.yml config file is the same in all three places so they all point to the same drb server. As far as clustered drb servers, that is something I do plan on tackling but probably not for another month or so. I plan on setting up a distributed worker pool using Rinda and tuplespace to have a pool of backgroundrb processes available at any one time. -Ezra From jacob.patton at gmail.com Tue Jul 18 14:24:58 2006 From: jacob.patton at gmail.com (Jacob Patton) Date: Tue, 18 Jul 2006 14:24:58 -0400 Subject: [Backgroundrb-devel] Using BackgrounDRb to replace nested loops? In-Reply-To: <47A09899-1738-4965-A867-51B5290D2D24@brainspl.at> References: <47A09899-1738-4965-A867-51B5290D2D24@brainspl.at> Message-ID: <24755DE2-D8E0-4480-8011-03BBD16287F1@gmail.com> Hi Ezra, Thanks for your advice regarding using BackgrounDRb to handle long- running emailing processes for my blog application. I'm having a little trouble getting BackgrounDRb to work--I'm just using it to post a message to the logs for now, to make sure I can get the code running. My questions are inline: > [You said:] I think you would be best served by creating a worker > class that did the compilation of messages and sent the emails in > the backgroundrb server. I also think that every 5 or 10 minutes is > real time enough for most things like this. So you might be better > served by building and sending out the email digests at intervals > and not with every time a post gets updated. > > So since you need to use ActionMailer, which in turn needs > ActionView, you will want to pull most of rails into your drb > server so you can make all this happen. So add this line to your > script/backgroundrb/start script at the top after the boot.rb > require line: > > require File.dirname(__FILE__) + "/../../config/environment.rb" I've done this... > Then build a worker class that does the processing and sending of > emails in the do_work method. > > class EmailWorker < BackgrounDRb::Rails > > def do_work(args) > # loop over emails and build digest > # loop over subscribers and send email > # > end > > end I did this, too, but my file looks like: # in lib/workers/checker_worker.rb class CheckerWorker < BackgrounDRb::Rails def do_work(args) logger.debug "BackgrounDRb run at #{Time.now}" end end > You will need a cron job to fire off this workers method every 5 or > 10 minutes like this: > > #!/usr/bin/env ruby > require "drb" > DRb.start_service > MiddleMan = DRbObject.new(nil, "druby://localhost:22222") Here is the file I'm using # in lib/checker.rb #!/usr/bin/env ruby require "drb" DRb.start_service MiddleMan = DRbObject.new(nil, "druby://localhost:22222") MiddleMan.new_worker(:class => :checker_worker, :ttl => 600) I think I've set up everything correctly, and that by running checker.rb by typing ruby checker.rb while within the lib directory, the BackgrounDRb object should fire, and the debug message should be posted to the log. When I run the command, though, I get this error message: (druby://localhost:22222) /usr/local/lib/ruby/gems/1.8/gems/ activesupport-1.3.1/ lib/active_support/dependencies.rb:100:in `const_missing': uninitialized constan t CheckerWorker (NameError) ... Can you, Ezra, or anyone else help me discover where I've gone wrong? Thanks very much for your help, Jacob Patton From hughes.james at gmail.com Tue Jul 18 14:51:27 2006 From: hughes.james at gmail.com (James Hughes) Date: Tue, 18 Jul 2006 11:51:27 -0700 Subject: [Backgroundrb-devel] backgroundrb and autotest Message-ID: <765a2c230607181151n144900d1ge0bf1ed44b798741@mail.gmail.com> Hi, First, thanks for the plugin. Sorry if this is not strictly a backgroundrb issue but I thought others on this list might have come across this problem. I'm trying to run the test that comes via the worker generator. The test works fine when running 'rake test_functional', but when the tests are run by autotest I get "uninitialized constant BackgrounDRb" (full stack trace below). I notice that BackgrounDRb::Rails is defined in "script/lib", is this a non-standard location that autotest somehow bypasses? I'm totally clueless as to how Rails loads plugins, so if anyone else has seen this issue, help appreciated. The stack trace: /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:123:in `const_missing': uninitialized constant BackgrounDRb (NameError) from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:131:in `const_missing' from /home/jhughes/work/project-tracker-rails/trunk/config/../lib/workers/task_checker_worker.rb:5 from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require' from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:147:in `require' from ./test/unit/task_checker_worker_test.rb:2 from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:140:in `load' from -e:1 from -e:1 /usr/lib/ruby/gems/1.8/gems/ZenTest-3.2.0/lib/unit_diff.rb:196:in `unit_diff': undefined method `first' for nil:NilClass (NoMethodError) from /usr/lib/ruby/gems/1.8/gems/ZenTest-3.2.0/lib/unit_diff.rb:84:in `unit_diff' from /usr/lib/ruby/gems/1.8/gems/ZenTest-3.2.0/bin/unit_diff:38 from /usr/bin/unit_diff:18 -- James Hughes Web application developer Vancouver, BC "Developing a coherent political analysis is in many respects contingent upon an ability to connect one context to another, a process not dissimilar to playing the kid's game of dot-to-dot." - Ward Churchill, from '"A Government of Laws"?' From ezmobius at gmail.com Tue Jul 18 15:07:27 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 18 Jul 2006 12:07:27 -0700 Subject: [Backgroundrb-devel] backgroundrb and autotest In-Reply-To: <765a2c230607181151n144900d1ge0bf1ed44b798741@mail.gmail.com> References: <765a2c230607181151n144900d1ge0bf1ed44b798741@mail.gmail.com> Message-ID: <64A09144-148A-4027-B0C7-84847AFCE9BD@gmail.com> On Jul 18, 2006, at 11:51 AM, James Hughes wrote: > Hi, > > First, thanks for the plugin. Sorry if this is not strictly a > backgroundrb issue but I thought others on this list might have come > across this problem. > > I'm trying to run the test that comes via the worker generator. The > test works fine when running 'rake test_functional', but when the > tests are run by autotest I get "uninitialized constant BackgrounDRb" > (full stack trace below). I notice that BackgrounDRb::Rails is defined > in "script/lib", is this a non-standard location that autotest somehow > bypasses? I'm totally clueless as to how Rails loads plugins, so if > anyone else has seen this issue, help appreciated. > ' Hey James- Yeah having the main file in script/lib was a mistake. I have changed that in more recent version s to leave all code exzcept for stubs for the start and stop scripts within the plugin. So I recommend that you update to the newest version of the plugin and start from there. I'm not sure how autotest will deal with testing workers. It is a bit tough to test the workers in there real life state and I haven't used autotest with this plugin yet. So try it with the latest version and let me know if that works for you. You will need to remove the entire script/backgroundrb folder and the config/backgroundrb.yml files from your app and run rake backgroundrb:setup again once you install the new version. -Ezra > The stack trace: > > /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/ > dependencies.rb:123:in > `const_missing': uninitialized constant BackgrounDRb (NameError) > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ > active_support/dependencies.rb:131:in > `const_missing' > from /home/jhughes/work/project-tracker-rails/trunk/config/../lib/ > workers/task_checker_worker.rb:5 > from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in > `require' > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ > active_support/dependencies.rb:147:in > `require' > from ./test/unit/task_checker_worker_test.rb:2 > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ > active_support/dependencies.rb:140:in > `load' > from -e:1 > from -e:1 > /usr/lib/ruby/gems/1.8/gems/ZenTest-3.2.0/lib/unit_diff.rb:196:in > `unit_diff': undefined method `first' for nil:NilClass (NoMethodError) > from /usr/lib/ruby/gems/1.8/gems/ZenTest-3.2.0/lib/unit_diff.rb:84:in > `unit_diff' > from /usr/lib/ruby/gems/1.8/gems/ZenTest-3.2.0/bin/unit_diff:38 > from /usr/bin/unit_diff:18 > > -- > James Hughes > Web application developer > Vancouver, BC > > "Developing a coherent political analysis is in many respects > contingent upon an ability to connect one context to another, a > process not dissimilar to playing the kid's game of dot-to-dot." > - Ward Churchill, from '"A Government of Laws"?' > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From ezmobius at gmail.com Tue Jul 18 15:14:59 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 18 Jul 2006 12:14:59 -0700 Subject: [Backgroundrb-devel] Using BackgrounDRb to replace nested loops? In-Reply-To: <24755DE2-D8E0-4480-8011-03BBD16287F1@gmail.com> References: <47A09899-1738-4965-A867-51B5290D2D24@brainspl.at> <24755DE2-D8E0-4480-8011-03BBD16287F1@gmail.com> Message-ID: <51029631-28DB-4926-ABD6-134C19D6BDDC@gmail.com> On Jul 18, 2006, at 11:24 AM, Jacob Patton wrote: > Hi Ezra, > > Thanks for your advice regarding using BackgrounDRb to handle long- > running emailing processes for my blog application. > > I'm having a little trouble getting BackgrounDRb to work--I'm just > using it to post a message to the logs for now, to make sure I can > get the code running. > > My questions are inline: > >> [You said:] I think you would be best served by creating a worker >> class that did the compilation of messages and sent the emails in >> the backgroundrb server. I also think that every 5 or 10 minutes >> is real time enough for most things like this. So you might be >> better served by building and sending out the email digests at >> intervals and not with every time a post gets updated. >> >> So since you need to use ActionMailer, which in turn needs >> ActionView, you will want to pull most of rails into your drb >> server so you can make all this happen. So add this line to your >> script/backgroundrb/start script at the top after the boot.rb >> require line: >> >> require File.dirname(__FILE__) + "/../../config/environment.rb" > > I've done this... > >> Then build a worker class that does the processing and sending of >> emails in the do_work method. >> >> class EmailWorker < BackgrounDRb::Rails >> >> def do_work(args) >> # loop over emails and build digest >> # loop over subscribers and send email >> # >> end >> >> end > > I did this, too, but my file looks like: > > # in lib/workers/checker_worker.rb > > class CheckerWorker < BackgrounDRb::Rails > > def do_work(args) > logger.debug "BackgrounDRb run at #{Time.now}" needs to be @logger > end > > end > >> You will need a cron job to fire off this workers method every 5 >> or 10 minutes like this: >> >> #!/usr/bin/env ruby >> require "drb" >> DRb.start_service >> MiddleMan = DRbObject.new(nil, "druby://localhost:22222") > > Here is the file I'm using > > # in lib/checker.rb > > #!/usr/bin/env ruby > require "drb" > DRb.start_service > MiddleMan = DRbObject.new(nil, "druby://localhost:22222") > MiddleMan.new_worker(:class => :checker_worker, :ttl => 600) > > I think I've set up everything correctly, and that by running > checker.rb by typing ruby checker.rb while within the lib > directory, the BackgrounDRb object should fire, and the debug > message should be posted to the log. > > When I run the command, though, I get this error message: > > (druby://localhost:22222) /usr/local/lib/ruby/gems/1.8/gems/ > activesupport-1.3.1/ > lib/active_support/dependencies.rb:100:in `const_missing': > uninitialized constan > t CheckerWorker (NameError) > ... Hrmm.. I wonder if this is caused when you require the config/ environment.rb file? Can you test that for me? comment out the require File.dirname(__FILE__) + "/../../config/environment.rb" line and stop/start the drb server. Then try this again and see if you get errors. Also you need to realize that Backgroundrb requires all your worker classes in lib/workers when it starts up. So if you didn't restart the drb server after you put the checker_worker.rb file in lib/ workers then that would cause this exact problem. So first restart your drb server. If that doesnt fix it then try what I stated above about the environemtn.rb require. Then report back if you still have errors. THanks -Ezra > > Can you, Ezra, or anyone else help me discover where I've gone wrong? > > Thanks very much for your help, > > Jacob Patton From chrisjroos at gmail.com Wed Jul 19 05:53:16 2006 From: chrisjroos at gmail.com (Chris Roos) Date: Wed, 19 Jul 2006 10:53:16 +0100 Subject: [Backgroundrb-devel] A couple of problems Message-ID: <3a5e51050607190253o4709857djae750e883e3bfb0f@mail.gmail.com> Hi Ezra, Thanks for the great work on BackgrounDRb. I have come across a couple of problems. 1. I wanted to define a simple class in the worker file to wrap up and pass back some data to my controller. It seems that if I create the class either inside or outside of my worker class (in the same file) it gets wrapper by a DRb::DRbUknown object and therefore cannot be accessed from my controller. class MyWorker < BackgrounDRb::Rails class MyResult attr_reader :result def initialize(result) @result = result end end def do_work() end def results MyResult.new(100) end end -- my_controller.rb -- def results results = MiddleMan.get_worker(session[:job_key]).results p results.result #=> undefined method `result' for # end I got around this by defining the class within the results method and returning like that. Really just wondering if this is by design or a bug? 2. When trying to pass a constant (class name in this instance) in the args hash I receive the following error. #=> undefined method `[]' for # (druby://localhost:22222) /rails_app/vendor/plugins/backgroundrb/backgroundrb.rb:105:in `new_worker' (druby://localhost:22222) /rails_app/vendor/plugins/backgroundrb/backgroundrb.rb:104:in `synchronize' (druby://localhost:22222) /rails_app/vendor/plugins/backgroundrb/backgroundrb.rb:104:in `new_worker' app/controllers/feed_controller.rb:36:in `method_calling_drb_worker' Again, I wonder if this is by design or a bug? Cheers, Chris From tim.anglade at gmail.com Wed Jul 19 09:49:46 2006 From: tim.anglade at gmail.com (Tim Anglade) Date: Wed, 19 Jul 2006 15:49:46 +0200 Subject: [Backgroundrb-devel] BackgrounDRb not working properly with files ? Message-ID: <60d0f3ab0607190649x38b9b0fbpc67db8d5e37cc733@mail.gmail.com> Hey there, I'm kinda new to BackgrounDRb but I'm facing a big problem which I can't seem to wrap my head around. The short version of it is : code that works when tested inside RoR's console doesn't work when executed by a BackgrounDRb worker. So I've tried a lot of things, including checking that the "load_rails" was true in the config, and I'm starting to wonder if this is not some kind of problem between BackgrounDRb and files or BackgrounDRb and file_column (which I'm using here). Any idea or help is greatly appreciated. My models are basically : - Clip has_many records - Record belongs_to Clip with :path as the file_column field. What I'm trying to do here is import a batch of files from a directory that's passed as the args to the worker and add them as a new records to new clips. Again, this code is working perfectly from inside the console, but not when I activate this worker through the application (the worker does start, show the progress bar and completes the action, the clips and records are created but the files are not attached correctly to the records. My hinch is that the worker cannot understand that :path is a file_column and that the file object (fl) is not passed correctly, but I could be completely off-target. Here's my worker's do_work : def do_work(args) # This method is called in it's own new thread when you # call new worker. args is set to :args #We define the variables used to monitor the progress #@progress is used by the controller for the progress bar, #file_number is used to calculate the progress @progress = 0 file_number = 0 #Checking if the arg is indeed a directory if File.directory?(args) path = args #we get number_of_files to calculate the progress number_of_files = Dir.entries(path).size #Let's get into that dir and look inside Dir.open(path) do |dir| dir.each do |entry| #We ignore ".", ".." (included in ruby dir listings) #we also ignore sub-directories and hidden files unless ["..", "."].include?(entry) or File.directory?(entry) or entry.match('^\.') puts "treating file : "+entry #First we create a new clip with the filename cl = Clip.new(:title => entry) #We get a File object out of the path fl = File.open(path+entry) #We create a new record. path is the file_column defined in the model. rec = cl.records.new(:path => fl) #... and we save everything. rec.save cl.save end file_number += 1 @progress = file_number * 100 / number_of_files end end else puts "Gimme a directory !" end end Any help greatly apreciated. Tim From ezmobius at gmail.com Wed Jul 19 11:25:06 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 19 Jul 2006 08:25:06 -0700 Subject: [Backgroundrb-devel] A couple of problems In-Reply-To: <3a5e51050607190253o4709857djae750e883e3bfb0f@mail.gmail.com> References: <3a5e51050607190253o4709857djae750e883e3bfb0f@mail.gmail.com> Message-ID: <94AB1251-6DBE-4C43-8CCE-E214F8B14B48@gmail.com> On Jul 19, 2006, at 2:53 AM, Chris Roos wrote: > Hi Ezra, > > Thanks for the great work on BackgrounDRb. I have come across a > couple of problems. > > 1. I wanted to define a simple class in the worker file to wrap up and > pass back some data to my controller. It seems that if I create the > class either inside or outside of my worker class (in the same file) > it gets wrapper by a DRb::DRbUknown object and therefore cannot be > accessed from my controller. > > class MyWorker < BackgrounDRb::Rails > > class MyResult > attr_reader :result > def initialize(result) > @result = result > end > end > > def do_work() > end > def results > MyResult.new(100) > end > > end > > -- my_controller.rb -- > > def results > results = MiddleMan.get_worker(session[:job_key]).results > p results.result > #=> undefined method `result' for # > end > > I got around this by defining the class within the results method and > returning like that. Really just wondering if this is by design or a > bug? > > 2. When trying to pass a constant (class name in this instance) in the > args hash I receive the following error. > > #=> undefined method `[]' for # > > (druby://localhost:22222) > /rails_app/vendor/plugins/backgroundrb/backgroundrb.rb:105:in > `new_worker' > (druby://localhost:22222) > /rails_app/vendor/plugins/backgroundrb/backgroundrb.rb:104:in > `synchronize' > (druby://localhost:22222) > /rails_app/vendor/plugins/backgroundrb/backgroundrb.rb:104:in > `new_worker' > app/controllers/feed_controller.rb:36:in `method_calling_drb_worker' > > Again, I wonder if this is by design or a bug? > > Cheers, > > Chris Hey Chris- Both of these issues stem from the same problem. When you see DRbUnknown errors like that it usually means that there is an issue sending that particular object over the wire. The answer to this is almost always that you need to include DRbUndumped in your classes that are causing these errors. And I would move your little class outside of the worker class but in the same file. SO make it like this: class MyResult include DRbUndumped attr_reader :result def initialize(result) @result = result end end class MyWorker < BackgrounDRb::Rails def do_work() end def results MyResult.new(100) end end Same thing goes for if you are trying to pass a classname or an AR object in the args hash. You will have to include DRbUndumped in any classes you want to sedn back and forth. This tells drb that this class is undumpable. This means it will keep the class on its normal side of the network and just pass methods and results back and forth> This happens sometimes when Drb has problems sending the entire class across the wire without losing some functionality. -Ezra From ezmobius at gmail.com Wed Jul 19 11:28:52 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 19 Jul 2006 08:28:52 -0700 Subject: [Backgroundrb-devel] BackgrounDRb not working properly with files ? In-Reply-To: <60d0f3ab0607190649x38b9b0fbpc67db8d5e37cc733@mail.gmail.com> References: <60d0f3ab0607190649x38b9b0fbpc67db8d5e37cc733@mail.gmail.com> Message-ID: <26C5DCE6-F48F-4146-BEE7-EABC5502A317@gmail.com> Hi! On Jul 19, 2006, at 6:49 AM, Tim Anglade wrote: > Hey there, > > I'm kinda new to BackgrounDRb but I'm facing a big problem which I > can't seem to wrap my head around. > > Here's my worker's do_work : > > def do_work(args) > # This method is called in it's own new thread when you > # call new worker. args is set to :args > > #We define the variables used to monitor the progress > #@progress is used by the controller for the progress bar, > #file_number is used to calculate the progress > @progress = 0 > file_number = 0 > > #Checking if the arg is indeed a directory > if File.directory?(args) > > path = args > > #we get number_of_files to calculate the progress > number_of_files = Dir.entries(path).size > > #Let's get into that dir and look inside > Dir.open(path) do |dir| > dir.each do |entry| > #We ignore ".", ".." (included in ruby dir listings) > #we also ignore sub-directories and hidden files > unless ["..", "."].include?(entry) or > File.directory?(entry) or entry.match('^\.') > puts "treating file : "+entry > > #First we create a new clip with the filename > cl = Clip.new(:title => entry) > > #We get a File object out of the path > fl = File.open(path+entry) > > #We create a new record. path is the file_column defined > in the model. > rec = cl.records.new(:path => fl) > > #... and we save everything. > rec.save > cl.save > end > file_number += 1 > @progress = file_number * 100 / number_of_files > end > end > > else > puts "Gimme a directory !" > end > end > > > Any help greatly apreciated. > > Tim Hey Tim- The first thing I would think of is that you are not passing an absolute path to this worker. Can you verify? What is the exact path you pass in to kick this worker off? There should not be an issue with using any of the code you have in a worker, it should work fine. This line: > if File.directory?(args) Makes me think that the rest of your code is just getting skipped because the directory may not exist? Can you show me the call you use in rails to kick this off? Also I would put some @logger.debug statements inside your worker there so you can see which parts of the code actually get run when you kick it off. Thanks -Ezra From tim.anglade at gmail.com Wed Jul 19 11:43:54 2006 From: tim.anglade at gmail.com (Tim Anglade) Date: Wed, 19 Jul 2006 17:43:54 +0200 Subject: [Backgroundrb-devel] BackgrounDRb not working properly with files ? In-Reply-To: <26C5DCE6-F48F-4146-BEE7-EABC5502A317@gmail.com> References: <60d0f3ab0607190649x38b9b0fbpc67db8d5e37cc733@mail.gmail.com> <26C5DCE6-F48F-4146-BEE7-EABC5502A317@gmail.com> Message-ID: <60d0f3ab0607190843l1a626f26jb6b602b8250c13a5@mail.gmail.com> Hey there, thanks for the quick response ! On 7/19/06, Ezra Zygmuntowicz wrote: > > Hey Tim- > > The first thing I would think of is that you are not passing an > absolute path to this worker. Can you verify? What is the exact path > you pass in to kick this worker off? There should not be an issue > with using any of the code you have in a worker, it should work fine. > This line: > > > if File.directory?(args) > > > Makes me think that the rest of your code is just getting skipped > because the directory may not exist? Can you show me the call you use > in rails to kick this off? Also I would put some @logger.debug > statements inside your worker there so you can see which parts of the > code actually get run when you kick it off. > > Thanks > -Ezra > Thanks for the input, alas the code is not getting skipped, as I can see debugging lines (I've got some 'puts' thrown around), and the new clips and records are being created. Also, there is no error thrown anywhere, and the problem obviously comes from this line: rec = cl.records.new(:path => fl) which obviously does not work as intended _and_ still does not throw an error... At first I thought (logically) that the error was on file_column's side, but at the same time, this line does work as intended in the console, so the origin of the problem does seem to come from BackgrounDRb. Is there any reason why files would not be passed correctly from a worker to file_column ? I'm thinking this is some kind of linking error between the backgroundrb daemon and the rails models. BackgrounDRb might very well think that :path is a standard field and throw fl at him, while if use the same line within Rails, it will trigger file_column functions to assign the field correctly. Thanks again and Cheers, Tim From tim.anglade at gmail.com Wed Jul 19 11:57:58 2006 From: tim.anglade at gmail.com (Tim Anglade) Date: Wed, 19 Jul 2006 17:57:58 +0200 Subject: [Backgroundrb-devel] BackgrounDRb not working properly with files ? In-Reply-To: <6B02271D-28B4-450A-BBF1-9FD1933F639C@gmail.com> References: <60d0f3ab0607190649x38b9b0fbpc67db8d5e37cc733@mail.gmail.com> <26C5DCE6-F48F-4146-BEE7-EABC5502A317@gmail.com> <60d0f3ab0607190843l1a626f26jb6b602b8250c13a5@mail.gmail.com> <6B02271D-28B4-450A-BBF1-9FD1933F639C@gmail.com> Message-ID: <60d0f3ab0607190857m502ed780n7aa5f62cc5e845f4@mail.gmail.com> Yup, that did the trick. Nice one, I obviously wouldn't have been able to catch that by myself... I'll tune it to load only file_column. Many, many thanks ! On 7/19/06, Ezra Zygmuntowicz wrote: > Hi~ > > On Jul 19, 2006, at 8:43 AM, Tim Anglade wrote: > > > Hey there, thanks for the quick response ! > > > > On 7/19/06, Ezra Zygmuntowicz wrote: > > > >> > >> Hey Tim- > >> > >> The first thing I would think of is that you are not > >> passing an > >> absolute path to this worker. Can you verify? What is the exact path > >> you pass in to kick this worker off? There should not be an issue > >> with using any of the code you have in a worker, it should work fine. > >> This line: > >> > >> > if File.directory?(args) > >> > >> > >> Makes me think that the rest of your code is just getting skipped > >> because the directory may not exist? Can you show me the call you use > >> in rails to kick this off? Also I would put some @logger.debug > >> statements inside your worker there so you can see which parts of the > >> code actually get run when you kick it off. > >> > >> Thanks > >> -Ezra > >> > > Thanks for the input, alas the code is not getting skipped, as I can > > see debugging lines (I've got some 'puts' thrown around), and the new > > clips and records are being created. > > > > Also, there is no error thrown anywhere, and the problem obviously > > comes from this line: > > rec = cl.records.new(:path => fl) > > > > which obviously does not work as intended _and_ still does not throw > > an error... At first I thought (logically) that the error was on > > file_column's side, but at the same time, this line does work as > > intended in the console, so the origin of the problem does seem to > > come from BackgrounDRb. > > Is there any reason why files would not be passed correctly from a > > worker to file_column ? > > > > I'm thinking this is some kind of linking error between the > > backgroundrb daemon and the rails models. BackgrounDRb might very well > > think that :path is a standard field and throw fl at him, while if use > > the same line within Rails, it will trigger file_column functions to > > assign the field correctly. > > > > > > Thanks again and Cheers, > > Tim > > > Tim- > > Ok maybe there is something else going on. BackgrounDRb by defalt > even with the load_rails set to true doers not actually load all of > rails. It just loads the database connection via boot.rb and all your > model files. So what I think is happening is that the file_column > plugin is not being loaded : / So lets try this... Open up the script/ > backgroundrb/start script and add the follwoing line right after the > require boot.rb line: > > require File.dirname(__FILE__) + "/../../config/environment.rb" > > > That wilkl make sure all of rails is loaded including your plugins. > Now you need to realize that this will also make the backgroundrb > process take up about 22Mb of ram since it is loading all of rails > now. This isn't too big of a problem its just a tad inefficient. If > this causes memory problems for you then you might consider writing a > few lines to require in just the file column plugin instead of all of > rails. But I think if you do this then you will get your worker to > function correctly. > > > Thanks- > -Ezra > > From hughes.james at gmail.com Wed Jul 19 13:41:26 2006 From: hughes.james at gmail.com (James Hughes) Date: Wed, 19 Jul 2006 10:41:26 -0700 Subject: [Backgroundrb-devel] backgroundrb and autotest In-Reply-To: <64A09144-148A-4027-B0C7-84847AFCE9BD@gmail.com> References: <765a2c230607181151n144900d1ge0bf1ed44b798741@mail.gmail.com> <64A09144-148A-4027-B0C7-84847AFCE9BD@gmail.com> Message-ID: <765a2c230607191041v4fdbc428nd1ac5e9540224b3c@mail.gmail.com> (Ezra, sorry for the repeat, I didn't hit reply-all the first time so this didn't get to the list) On 7/18/06, Ezra Zygmuntowicz wrote: > > On Jul 18, 2006, at 11:51 AM, James Hughes wrote: > > > Hi, > > I'm trying to run the test that comes via the worker generator. The > > test works fine when running 'rake test_functional', but when the Scratch that test_functional above. The generated test is a *unit* test, so no wonder the functional tests were running without error. ;] Anyway, the upshot is that this is not an autotest issue, but a testing issue in general. [goes away and hacks a bit] Ok, found the problem; in the generated test, these two lines need to be reversed so that the backgroundrb lib is loaded before the worker class: require "#{RAILS_ROOT}/lib/workers/task_checker_worker" require "#{RAILS_ROOT}/vendor/plugin/backgroundrb/backgroundrb.rb" Also, it should be /vendor/plugins/, not /vendor/plugin. So, lines 2 and 3 of your test end up like the following: require "#{RAILS_ROOT}/vendor/plugins/backgroundrb/backgroundrb.rb" require "#{RAILS_ROOT}/lib/workers/task_checker_worker" jh -- James Hughes Web application developer Vancouver, BC "Developing a coherent political analysis is in many respects contingent upon an ability to connect one context to another, a process not dissimilar to playing the kid's game of dot-to-dot." - Ward Churchill, from '"A Government of Laws"?' From ezmobius at gmail.com Wed Jul 19 13:51:13 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 19 Jul 2006 10:51:13 -0700 Subject: [Backgroundrb-devel] backgroundrb and autotest In-Reply-To: <765a2c230607191041v4fdbc428nd1ac5e9540224b3c@mail.gmail.com> References: <765a2c230607181151n144900d1ge0bf1ed44b798741@mail.gmail.com> <64A09144-148A-4027-B0C7-84847AFCE9BD@gmail.com> <765a2c230607191041v4fdbc428nd1ac5e9540224b3c@mail.gmail.com> Message-ID: <82D5CE38-265C-4388-953B-7D6A7DB4E5F1@brainspl.at> Hi~ On Jul 19, 2006, at 10:41 AM, James Hughes wrote: > (Ezra, sorry for the repeat, I didn't hit reply-all the first time so > this didn't get to the list) > On 7/18/06, Ezra Zygmuntowicz wrote: >> >> On Jul 18, 2006, at 11:51 AM, James Hughes wrote: >> >>> Hi, >>> I'm trying to run the test that comes via the worker generator. The >>> test works fine when running 'rake test_functional', but when the > > Scratch that test_functional above. The generated test is a *unit* > test, so no wonder the functional tests were running without error. ;] > > Anyway, the upshot is that this is not an autotest issue, but a > testing issue in general. > > [goes away and hacks a bit] > > Ok, found the problem; in the generated test, these two lines need to > be reversed so that the backgroundrb lib is loaded before the worker > class: > > require "#{RAILS_ROOT}/lib/workers/task_checker_worker" > require "#{RAILS_ROOT}/vendor/plugin/backgroundrb/backgroundrb.rb" > > Also, it should be /vendor/plugins/, not /vendor/plugin. > > So, lines 2 and 3 of your test end up like the following: > > require "#{RAILS_ROOT}/vendor/plugins/backgroundrb/backgroundrb.rb" > require "#{RAILS_ROOT}/lib/workers/task_checker_worker" > > > jh > > -- > James Hughes Hey James- Thanks for that. I just commited this change to the generated test case. So it should be fixed now. Good catch. Thanks -Ezra From bjohnson at contuitive.com Wed Jul 19 17:36:57 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Wed, 19 Jul 2006 16:36:57 -0500 Subject: [Backgroundrb-devel] Passing objects to drb, does it keep the existing db connection? Message-ID: <02134CAF-8460-4F5F-98AC-5DC1A6EBC141@contuitive.com> I have a simple question. Let's say I do this in one of my models: after_save drb_conn.some_method(self) end When that object gets over to the background process is it the exact same as?.... Model.find some_id #in the background process As far as my background process is concerned passing the object is not any different than passing the id and using the find method? Because when I try to inspect the object that has been passed over I don't get the typical .inspect output that you get from object of models. Thanks for your help. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060719/cb2d4bfb/attachment-0001.html From ezmobius at gmail.com Wed Jul 19 17:41:50 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 19 Jul 2006 14:41:50 -0700 Subject: [Backgroundrb-devel] Passing objects to drb, does it keep the existing db connection? In-Reply-To: <02134CAF-8460-4F5F-98AC-5DC1A6EBC141@contuitive.com> References: <02134CAF-8460-4F5F-98AC-5DC1A6EBC141@contuitive.com> Message-ID: <41911CDF-B412-4044-A59F-B7D0438FE615@gmail.com> On Jul 19, 2006, at 2:36 PM, Ben Johnson wrote: > I have a simple question. Let's say I do this in one of my models: > > after_save > drb_conn.some_method(self) > end > > When that object gets over to the background process is it the > exact same as?.... > > Model.find some_id #in the background process > > As far as my background process is concerned passing the object is > not any different than passing the id and using the find method? > Because when I try to inspect the object that has been passed over > I don't get the typical .inspect output that you get from object of > models. > > Thanks for your help. > > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com Ben- The inspect output will be different because the object is now DRbUndumped. THis means that the object actually stays on the rails side of things and the drb server just sends method calls to it. You should read up a bit on drb and DRbUndumped so you can understand how these things work. -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060719/8c7f6142/attachment.html From hughes.james at gmail.com Wed Jul 19 18:24:40 2006 From: hughes.james at gmail.com (James Hughes) Date: Wed, 19 Jul 2006 15:24:40 -0700 Subject: [Backgroundrb-devel] Testing worker classes Message-ID: <765a2c230607191524y2dd3b341lfeef7d108548edf9@mail.gmail.com> Hi, Does it make sense to test a worker class by calling Middleman.new_worker? Or should I just be testing the functionality in the do_work method, with, say, a mock object. This seems like a really ugly DRY violation: you'd need to keep the code in the mock and the real worker in sync. That's the philosophical half of my question. In fact, until someone tells me why I shouldn't, I am trying to call new_worker from a test method in the test class generated by the worker generator. I'm getting NoMethodError: undefined method `[]' for # I note in another thread that this is due to DRbundumped not being included in the class that is causing the error. I took this to mean that I need "include DRbundumped" in my test class, but that didn't work. I'm not sure where else I should include it. I know that eventually I'll need to put it in the non-test class that calls new_worker, but I'm not there yet. Is testing workers just a non-starter? Here's the test method: def test_task_is_triggered job_key = MiddleMan.new_worker :class => :task_checker_worker, :args => tasks(:should_be_triggered) @job_keys << job_key # ...assertions go here... end thanks, jh -- James Hughes Web application developer Vancouver, BC "Developing a coherent political analysis is in many respects contingent upon an ability to connect one context to another, a process not dissimilar to playing the kid's game of dot-to-dot." - Ward Churchill, from '"A Government of Laws"?' From ezmobius at gmail.com Wed Jul 19 18:38:26 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 19 Jul 2006 15:38:26 -0700 Subject: [Backgroundrb-devel] Testing worker classes In-Reply-To: <765a2c230607191524y2dd3b341lfeef7d108548edf9@mail.gmail.com> References: <765a2c230607191524y2dd3b341lfeef7d108548edf9@mail.gmail.com> Message-ID: <851A6BF8-F63C-4A98-A58D-834D6149F51F@gmail.com> On Jul 19, 2006, at 3:24 PM, James Hughes wrote: > Hi, > > Does it make sense to test a worker class by calling > Middleman.new_worker? Or should I just be testing the functionality in > the do_work method, with, say, a mock object. This seems like a really > ugly DRY violation: you'd need to keep the code in the mock and the > real worker in sync. > > That's the philosophical half of my question. In fact, until someone > tells me why I shouldn't, I am trying to call new_worker from a test > method in the test class generated by the worker generator. I'm > getting > > NoMethodError: undefined method `[]' for # > > I note in another thread that this is due to DRbundumped not being > included in the class that is causing the error. I took this to mean > that I need "include DRbundumped" in my test class, but that didn't > work. I'm not sure where else I should include it. I know that > eventually I'll need to put it in the non-test class that calls > new_worker, but I'm not there yet. Is testing workers just a > non-starter? > > Here's the test method: > > def test_task_is_triggered > job_key = MiddleMan.new_worker :class => :task_checker_worker, > :args => tasks(:should_be_triggered) > @job_keys << job_key > # ...assertions go here... > end > > > thanks, > jh > > -- > James Hughes Hey James- Yeah I have some code for a better test harness that I need to get working correctly and release. The best way I have come up with is to create an additional config file for test mode that connects to the test db. Then you have a rake wrapper for your tests that starts the drb server in test mode when your tests fire up. Because there needs to be a good way to test things from end to end through the MiddleMan. I am open for suggestions on a better test harness. I am just very busy this week so I won't have time until at least the weekend to get something new released. Thanks- -Ezra P.S. I would love a patch for a better test harness if any one comes up with one ;) From robby at 1066propaganda.com Wed Jul 19 22:10:26 2006 From: robby at 1066propaganda.com (robby at 1066propaganda.com) Date: Wed, 19 Jul 2006 19:10:26 -0700 (PDT) Subject: [Backgroundrb-devel] Mysql Server has gone away? Message-ID: <2673.24.60.208.243.1153361426.squirrel@webmail.1066propaganda.com> First off, amazing plugin! Being a rails noob, it helped me do some stuff I would've thought way over my head. Which brings me to my problem. :) I'm using backgroundrb to handle some processes that take so long that I don't want them tied to a page refresh (several xml-rpc calls and saving data in text files on the server). I got it working with backgroundrb fairly quickly and got the AJAX status stuff working (was REALLY pleasantly surprised with how easy that part was :). So everything is working great, until a couple hours later when everything stops working. At first I couldn't figure it out. things would be working great, and then not, no obvious error messages or anything. I was able to track down an exception getting thrown: Mysql::Error: MySQL server has gone away: I searched through the list and through the rails site for any reference to this being an issue. All I could find is some reference to this being a problem in older versions of rails. Is anyone aware of a fix? I'm using the latest version of BackgroundRB and rails v. 1.1, and I'm currently hosted on textdrive, if any of that is helpful. If there's any code that would be helpful to post I can do that too, I'm just not sure what would be relevant. Thanks! From gethemant at gmail.com Thu Jul 20 03:47:37 2006 From: gethemant at gmail.com (hemant) Date: Thu, 20 Jul 2006 13:17:37 +0530 Subject: [Backgroundrb-devel] Mysql Server has gone away? In-Reply-To: <2673.24.60.208.243.1153361426.squirrel@webmail.1066propaganda.com> References: <2673.24.60.208.243.1153361426.squirrel@webmail.1066propaganda.com> Message-ID: I had similar problem, removing concurrent line from ./script/backgroundrb/start may help... On 7/20/06, robby at 1066propaganda.com wrote: > > First off, amazing plugin! Being a rails noob, it helped me do some stuff > I would've thought way over my head. Which brings me to my problem. :) > > I'm using backgroundrb to handle some processes that take so long that I > don't want them tied to a page refresh (several xml-rpc calls and saving > data in text files on the server). I got it working with backgroundrb > fairly quickly and got the AJAX status stuff working (was REALLY > pleasantly surprised with how easy that part was :). > > So everything is working great, until a couple hours later when everything > stops working. At first I couldn't figure it out. things would be working > great, and then not, no obvious error messages or anything. I was able to > track down an exception getting thrown: > > Mysql::Error: MySQL server has gone away: > > I searched through the list and through the rails site for any reference > to this being an issue. All I could find is some reference to this being a > problem in older versions of rails. > > Is anyone aware of a fix? I'm using the latest version of BackgroundRB and > rails v. 1.1, and I'm currently hosted on textdrive, if any of that is > helpful. > > If there's any code that would be helpful to post I can do that too, I'm > just not sure what would be relevant. > > Thanks! > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060720/77d9d750/attachment.html From ezmobius at gmail.com Thu Jul 20 11:13:21 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 20 Jul 2006 08:13:21 -0700 Subject: [Backgroundrb-devel] Mysql Server has gone away? In-Reply-To: <2673.24.60.208.243.1153361426.squirrel@webmail.1066propaganda.com> References: <2673.24.60.208.243.1153361426.squirrel@webmail.1066propaganda.com> Message-ID: <8EAC8FB0-C0C8-4361-A5A3-F784EDBC67C5@gmail.com> On Jul 19, 2006, at 7:10 PM, robby at 1066propaganda.com wrote: > First off, amazing plugin! Being a rails noob, it helped me do some > stuff > I would've thought way over my head. Which brings me to my problem. :) > > I'm using backgroundrb to handle some processes that take so long > that I > don't want them tied to a page refresh (several xml-rpc calls and > saving > data in text files on the server). I got it working with backgroundrb > fairly quickly and got the AJAX status stuff working (was REALLY > pleasantly surprised with how easy that part was :). > > So everything is working great, until a couple hours later when > everything > stops working. At first I couldn't figure it out. things would be > working > great, and then not, no obvious error messages or anything. I was > able to > track down an exception getting thrown: > > Mysql::Error: MySQL server has gone away: > > I searched through the list and through the rails site for any > reference > to this being an issue. All I could find is some reference to this > being a > problem in older versions of rails. > > Is anyone aware of a fix? I'm using the latest version of > BackgroundRB and > rails v. 1.1, and I'm currently hosted on textdrive, if any of that is > helpful. > > If there's any code that would be helpful to post I can do that > too, I'm > just not sure what would be relevant. > > Thanks! Hey Robby- Glad you're getting some use out of the plugin ;) ActiveRecord does have some issues with threading and concurrency that can sometimes cause that error you are seeing. Open up your script/backgroundrb/ start file and comment out the following line: ActiveRecord::Base.allow_concurrency = true Make sure you do that up in your main app script folder and not in the plugin. Well you can do it in the plugin as well to keep it consistent. Then you need to restart the drb server. Give this a shot and see if it gets rid of your problem. Also I am curious about using this plugin on textdrive. Did they give you an extra port number to run it on? If you run into problems with running the drb server on a port then let me know and I can show you how to use a unix domain socket file instead so you don't take up an extra port. Cheers- -Ezra From info at siebert-wd.de Thu Jul 20 15:47:04 2006 From: info at siebert-wd.de (Michael Siebert) Date: Thu, 20 Jul 2006 21:47:04 +0200 Subject: [Backgroundrb-devel] delete workers after use Message-ID: <32b567370607201247l69906815t303ada07ca27a974@mail.gmail.com> Hello folks, does backgroundrb delete workers who exited do_work()? i experienced it doesnt (and that freaked me because i didnt know that for several hours). is there a way to let the middleman or so delete workers who did their job imediately after they're done? if not, i coud think of that as a nice improvement. greetz -- Michael Siebert www.stellar-legends.de - Weltraum-Browsergame im Alpha-Stadium From bjohnson at contuitive.com Fri Jul 21 21:47:58 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Fri, 21 Jul 2006 20:47:58 -0500 Subject: [Backgroundrb-devel] Problem with drbundumped? Message-ID: <8C0AFC58-6C7C-4505-AB23-4D9A608611FD@contuitive.com> I'm not sure if this is a problem or not, but it definitely henders the way one would use the passing of objects to the drb server. Isn't the point of passing objects to the drb server to act like its passing to just another local method? When I pass an object to the drb server a lot of the useful methods are gone. Such as id() or class () or inspect(), etc. I spent a good 8 hours figuring out what the hell was going on with my worker class when I realized the id of a model object was 543543534, something completely different than that in the database. Not only that, I do different class type checking to perform different actions depending on the class type of the object. I'm not sure its possible, but I think it would seem more intuitive to simply make the class(), inspect(), etc methods all make a call back to the original object to get the information. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060721/ce8071d8/attachment-0001.html From robby at 1066propaganda.com Sat Jul 22 18:53:52 2006 From: robby at 1066propaganda.com (Robby Macdonell) Date: Sat, 22 Jul 2006 18:53:52 -0400 Subject: [Backgroundrb-devel] Mysql Server has gone away? In-Reply-To: <8EAC8FB0-C0C8-4361-A5A3-F784EDBC67C5@gmail.com> References: <2673.24.60.208.243.1153361426.squirrel@webmail.1066propaganda.com> <8EAC8FB0-C0C8-4361-A5A3-F784EDBC67C5@gmail.com> Message-ID: Looks like setting allow_concurrency to true did the trick. I'm having a few other server issues so it took me a while to verify, but yeah, all seems to be working now. Thanks so much for your quick response! >> Also I am curious about using this plugin on textdrive. Did they give you an extra port number to run it on? If you run into problems with running the drb server on a port then let me know and I can show you how to use a unix domain socket file instead so you don't take up an extra port. Well, I thought there might be an issue with TextDrive because they have a note about long running processes in there terms of use statement. But I emailed them about it specifically, and the response I got was basically "as long as you don't put too much strain on anything, we don't really mind." So I just started it up as normal, with rake backgroundrb:start and everything worked like it would on any other server (aside from the database timeouts). I didn't have to ask for a port number or anything. The whole system still seems to be acting a little bit squirrelly though, so I'm not sure I'm totally in the clear yet. I just started getting a message about the nfs server not responding, and I hadn't seen that before. But that just seems to be causing a lag, not an out- and-out breakage. Thanks again for being so helpful, I'm new to rails and so far the community has seemed really amazing. ----------------------------------------------------------- Robby Macdonell robby at 1066propaganda.com On Jul 20, 2006, at 11:13 AM, Ezra Zygmuntowicz wrote: On Jul 19, 2006, at 7:10 PM, robby at 1066propaganda.com wrote: > First off, amazing plugin! Being a rails noob, it helped me do some > stuff > I would've thought way over my head. Which brings me to my problem. :) > > I'm using backgroundrb to handle some processes that take so long > that I > don't want them tied to a page refresh (several xml-rpc calls and > saving > data in text files on the server). I got it working with backgroundrb > fairly quickly and got the AJAX status stuff working (was REALLY > pleasantly surprised with how easy that part was :). > > So everything is working great, until a couple hours later when > everything > stops working. At first I couldn't figure it out. things would be > working > great, and then not, no obvious error messages or anything. I was > able to > track down an exception getting thrown: > > Mysql::Error: MySQL server has gone away: > > I searched through the list and through the rails site for any > reference > to this being an issue. All I could find is some reference to this > being a > problem in older versions of rails. > > Is anyone aware of a fix? I'm using the latest version of > BackgroundRB and > rails v. 1.1, and I'm currently hosted on textdrive, if any of that is > helpful. > > If there's any code that would be helpful to post I can do that > too, I'm > just not sure what would be relevant. > > Thanks! Hey Robby- Glad you're getting some use out of the plugin ;) ActiveRecord does have some issues with threading and concurrency that can sometimes cause that error you are seeing. Open up your script/backgroundrb/ start file and comment out the following line: ActiveRecord::Base.allow_concurrency = true Make sure you do that up in your main app script folder and not in the plugin. Well you can do it in the plugin as well to keep it consistent. Then you need to restart the drb server. Give this a shot and see if it gets rid of your problem. Also I am curious about using this plugin on textdrive. Did they give you an extra port number to run it on? If you run into problems with running the drb server on a port then let me know and I can show you how to use a unix domain socket file instead so you don't take up an extra port. Cheers- -Ezra From ezmobius at gmail.com Sun Jul 23 15:10:50 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sun, 23 Jul 2006 12:10:50 -0700 Subject: [Backgroundrb-devel] Problem with drbundumped? In-Reply-To: <8C0AFC58-6C7C-4505-AB23-4D9A608611FD@contuitive.com> References: <8C0AFC58-6C7C-4505-AB23-4D9A608611FD@contuitive.com> Message-ID: <7BF9EB01-2122-4313-A7EC-521E35EF58D1@gmail.com> On Jul 21, 2006, at 6:47 PM, Ben Johnson wrote: > I'm not sure if this is a problem or not, but it definitely henders > the way one would use the passing of objects to the drb server. > Isn't the point of passing objects to the drb server to act like > its passing to just another local method? When I pass an object to > the drb server a lot of the useful methods are gone. Such as id() > or class() or inspect(), etc. I spent a good 8 hours figuring out > what the hell was going on with my worker class when I realized the > id of a model object was 543543534, something completely different > than that in the database. Not only that, I do different class type > checking to perform different actions depending on the class type > of the object. > > I'm not sure its possible, but I think it would seem more intuitive > to simply make the class(), inspect(), etc methods all make a call > back to the original object to get the information. > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com Hey Ben- There are definitely issues you need to be aware of with drb. Any object that cannot be marshaled will not be able to be used via drb without include DRbUndumped. So sometimes you can run into issues like you are having. This is a limitation of drb itself and not this plugin. Passing full objects back and forth can work very well but sometimes you lose certain methods like you seem to be running into. So I would see if you can rethink the way you are using the setup and see if you can come up with a way to do what you are trying to do without sending the full AR objects over to the drb server. You will have to go diving in the drb source and tutorials to see if you can accomplish what you want. the DRb code base is not huge so its not too hard to read. inspect, class and friends are giving you back DRbUnkown objects when you call these methods because of the way drb works, I don't think there is a direct way around this. Please do lt me know if you figure out a way around it ;) I try not to pass full AR objects across to drb myself and just pass the id's or some other identifier that you can use to find the records you need in your worker class. Cheers- -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060723/bf9aba93/attachment.html From info at siebert-wd.de Mon Jul 24 10:36:57 2006 From: info at siebert-wd.de (Michael Siebert) Date: Mon, 24 Jul 2006 16:36:57 +0200 Subject: [Backgroundrb-devel] require file from /lib in a worker Message-ID: <32b567370607240736m74db586cv7241b5a0859ca496@mail.gmail.com> Hello, i need to require a file located in /lib from a worker class, but when i try to start the backgroundrb server, i get something like d:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:21:in `require__': no such file to load -- ./lib/office_thumnail_generator.rb (MissingSourceFile) i tried it with and without lib, with RAILS_ROOT, nothing worked, altough the paths look good for me, but ruby wont find the file any suggestions? greets -- Michael Siebert www.stellar-legends.de - Weltraum-Browsergame im Alpha-Stadium -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060724/aa568557/attachment.html From rosskarchner at gmail.com Mon Jul 24 11:17:14 2006 From: rosskarchner at gmail.com (Ross M Karchner) Date: Mon, 24 Jul 2006 11:17:14 -0400 Subject: [Backgroundrb-devel] Worker-be-gone Message-ID: <9cd84480607240817g1a244ca4xa46c596026ede6ab@mail.gmail.com> Hello, Thank you so very much for Backgroundrb. Good Stuff. A general question: Do: - calling 'teminate' within a worker - calling delete_worker through MiddleMan - letting the :ttl expire - calling MiddleMan.gc! (with an appropriate timestamp) All accomplish the same thing? And here's a bonus feature request: It'd be cool if you could specify how many instances of a particular worker class can be active at the same time-- anything over that gets queued. I'm open to any suggestions for implementing that on my own, but it would make a nifty feature for the plugin. -Ross From ezmobius at gmail.com Mon Jul 24 12:31:58 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 24 Jul 2006 09:31:58 -0700 Subject: [Backgroundrb-devel] Worker-be-gone In-Reply-To: <9cd84480607240817g1a244ca4xa46c596026ede6ab@mail.gmail.com> References: <9cd84480607240817g1a244ca4xa46c596026ede6ab@mail.gmail.com> Message-ID: On Jul 24, 2006, at 8:17 AM, Ross M Karchner wrote: > Hello, > > Thank you so very much for Backgroundrb. Good Stuff. > > A general question: > > Do: > - calling 'teminate' within a worker > - calling delete_worker through MiddleMan > - letting the :ttl expire > - calling MiddleMan.gc! (with an appropriate timestamp) > > All accomplish the same thing? > > And here's a bonus feature request: It'd be cool if you could specify > how many instances of a particular worker class can be active at the > same time-- anything over that gets queued. > > I'm open to any suggestions for implementing that on my own, but it > would make a nifty feature for the plugin. > > -Ross Ross- Yeah those all do mostly the same thing with a few differences. the terminate method is what you should have your worker call when it is done processing its job. THis will still leave the worker in the jobs hash but it wont be running anymore. THen you can either use :tt;, delete_worker or gc! top delete the worker. NExt release will remove the worker from the jobs hash as well as stop its thread when you call terminate. The next release of the plugin will address a bunch of these issues. I will be cleaning up the code base and refactoring to use rinda ring where possible. This will mean some API changes but not much. Your worker classes will be essentially the same. But this will allow for you to have workers run in there own drb server. So you can have multiple drb servers but they all have dns of sorts through rinda ring finger. The limit on the number of allowed workers is probably a good idea. BUt I am pretty busy these days so a patch for that would probably be applied but I might not get around to doing it myself for a while yet. Cheers- -Ezra From ezmobius at gmail.com Mon Jul 24 12:34:31 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 24 Jul 2006 09:34:31 -0700 Subject: [Backgroundrb-devel] require file from /lib in a worker In-Reply-To: <32b567370607240736m74db586cv7241b5a0859ca496@mail.gmail.com> References: <32b567370607240736m74db586cv7241b5a0859ca496@mail.gmail.com> Message-ID: <599A66BB-A682-4F01-8B7F-55479CEAADA9@gmail.com> On Jul 24, 2006, at 7:36 AM, Michael Siebert wrote: > lib/office_thumnail_generator.rb Michael- Try it like this, in your script/backgroundrb/start file add this line : require "#{RAILS_ROOT}/lib/office_thumnail_generator.rb" Make sure you make this change to the start script up in the main rails app and not the plugin. That should work fine I do the same thing. Let me know if you still have issues Cheers- -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060724/eee05aaa/attachment.html From dhoefler at gmail.com Mon Jul 24 16:22:00 2006 From: dhoefler at gmail.com (Dave Hoefler) Date: Mon, 24 Jul 2006 15:22:00 -0500 Subject: [Backgroundrb-devel] Mongrel + BackgrounDRb + File Column = Upload Progress Bar? Message-ID: <7d983dfe0607241322q3811bdf8h6490553f69f28522@mail.gmail.com> Hi there, I've been digging around trying to find some information (mostly examples) on how one would handle a progress bar upload using BackgrounDRb and File Column. I did read http://backgroundrb.rubyforge.org/ and Ezra's blog of course, but I still have a few questions. I made a simple upload form (as a test) that is submitted to the "upload_song" action. def upload_song @song = Song.new(params[:song]) @song.member_id = session[:member_id] @song.save redirect_to :action => 'edit_songs' end The question is, do I create the session[:job_key] within this method like so: ?? def upload_song # I know my :args is probably wrong. What is the correct argument for args in this case? session[:job_key] = MiddleMan.new_worker(:class => :upload_worker, :args => "Uploading song...") @song = Song.new(params[:song]) @song.member_id = session[:member_id] @song.save redirect_to :action => 'edit_songs' end In my view, within the form parameters I have: <%= periodically_call_remote(:url => {:action => 'get_progress'}, :frequency => 1) %> which calls the below method: def get_progress if request.xhr? progress_percent = MiddleMan.get_worker(session[:job_key]).progress render :update do |page| page.call('progressPercent', 'progressbar', progress_percent) page.redirect_to ( :action => 'done') if progress_percent >= 100 end else redirect_to :action => 'index' end end def done MiddleMan.delete_worker(session[:job_key]) redirect_to :action => 'edit_songs' end Just for clarity sake, here's my upload_worker class: class UploadWorker < BackgrounDRb::Rails attr_reader :progress def do_work(args) @progress = 0 start_working end def start_working Thread.new do while @progress < 100 sleep rand / 2 a = [1,3,5,7] @progress += a[rand(a.length-1)] if @progress > 100 @progress = 100 end end end end def progress puts "Rails is fetching progress: #{@progress}" Integer(@progress) end end Are there any good tutorials out there that walk through the implementation of backgroundrb (specifically covering file uploads)? Am I close to getting this thing to work? Thank you, Dave Hoefler -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060724/e8a870c4/attachment-0001.html From chris at etechdata.com.au Tue Jul 25 02:25:00 2006 From: chris at etechdata.com.au (Chris H) Date: Tue, 25 Jul 2006 16:25:00 +1000 Subject: [Backgroundrb-devel] MY worker won't stop working In-Reply-To: <765a2c230607191041v4fdbc428nd1ac5e9540224b3c@mail.gmail.com> References: <765a2c230607181151n144900d1ge0bf1ed44b798741@mail.gmail.com> <64A09144-148A-4027-B0C7-84847AFCE9BD@gmail.com> <765a2c230607191041v4fdbc428nd1ac5e9540224b3c@mail.gmail.com> Message-ID: <44C5B93C.60505@etechdata.com.au> Hi Ezra, love backgroundrb! Awesome plugin! I have a worker that processes some data files and loads this data in mysql. It runs fine and processes all the files. However if I try to call delete_worker from my controller while it is processing , it says it has stopped the worker but I can still see the process in top and also see the rows still going into mysql. My do_work looks like this: def do_work(args) # This method is called in it's own new thread when you # call new worker. args is set to :args if args @process_cdr = ProcessCdr.new(args) else @process_cdr = ProcessCdr.new end @process_cdr.process end I have start and stop actions in my controller: def start if @params[:cdr_file].to_s.length > 0 MiddleMan[:process_watcher] ||MiddleMan.new_worker(:class => :watch_cdr_process_worker, :args => @params[:cdr_file].to_s, :job_key => :process_watcher) else MiddleMan[:process_watcher] || MiddleMan.new_worker(:class => :watch_cdr_process_worker, :job_key => :process_watcher) end redirect_to :action => 'status' end def stop MiddleMan.delete_worker(:process_watcher) if MiddleMan[:process_watcher] redirect_to :action => 'status' end I have some ajax code monitoring the status of my worker and when I choose stop it responds that my worker has indeed stopped. I also tried adding a method stop_work to my worker which called self.terminate but this didn't seem to work either. I'm confused as to what I should be doing to get my worker to stop. Thanks in advance, Chris. From ezmobius at gmail.com Tue Jul 25 11:59:18 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 25 Jul 2006 08:59:18 -0700 Subject: [Backgroundrb-devel] MY worker won't stop working In-Reply-To: <44C5B93C.60505@etechdata.com.au> References: <765a2c230607181151n144900d1ge0bf1ed44b798741@mail.gmail.com> <64A09144-148A-4027-B0C7-84847AFCE9BD@gmail.com> <765a2c230607191041v4fdbc428nd1ac5e9540224b3c@mail.gmail.com> <44C5B93C.60505@etechdata.com.au> Message-ID: <7CF7FEE5-0424-4369-B56F-04FB787DDEC8@gmail.com> Hi~ On Jul 24, 2006, at 11:25 PM, Chris H wrote: > Hi Ezra, > > love backgroundrb! Awesome plugin! Thanks! > > I have a worker that processes some data files and loads this data > in mysql. > It runs fine and processes all the files. > > However if I try to call delete_worker from my controller while it is > processing , > it says it has stopped the worker but I can still see the process > in top and also see the rows still going into mysql. > > > My do_work looks like this: > > def do_work(args) > # This method is called in it's own new thread when you > # call new worker. args is set to :args > if args > @process_cdr = ProcessCdr.new(args) > else > @process_cdr = ProcessCdr.new > end > @process_cdr.process > > end > > I have start and stop actions in my controller: > > def start > > if @params[:cdr_file].to_s.length > 0 > > MiddleMan[:process_watcher] ||MiddleMan.new_worker(:class => > :watch_cdr_process_worker, :args => @params > [:cdr_file].to_s, :job_key => > :process_watcher) > > else > > MiddleMan[:process_watcher] || MiddleMan.new_worker(:class => > :watch_cdr_process_worker, :job_key => :process_watcher) > > end > > redirect_to :action => 'status' > > end > > def stop > > MiddleMan.delete_worker(:process_watcher) if MiddleMan > [:process_watcher] Middleman[:process_watcher].terminate && MiddleMan.delete_worker (:process_watcher) > redirect_to :action => 'status' > > end > > I have some ajax code monitoring the status of my worker and when I > choose stop it responds that my worker has indeed stopped. > > I also tried adding a method stop_work to my worker which called > self.terminate but this didn't seem to work either. > > I'm confused as to what I should be doing to get my worker to stop. > > Thanks in advance, > Chris. Chris- When you say you can see the process in top still what do you mean? Killing one worker will not stop the whole backgroundrb process so it will still show in top. The way terminate works is that you call it and it will signal the thread handle that it is ok to delete this worker. THen delete_worker will actually delete it. The next release will have a method you can use in your worker to kill it from within itself when its finished. See if using terminate this way works for you and let me know. Thanks -Ezra From mthanawala at gmail.com Tue Jul 25 12:08:31 2006 From: mthanawala at gmail.com (Mayank Thanawala) Date: Tue, 25 Jul 2006 09:08:31 -0700 Subject: [Backgroundrb-devel] Production deployment? Message-ID: Hi all, BackgrounDrb was exactly what I needed for a project I'm working on at work. Everyone's really impressed, and now it's time to put the system into production. My question is, how do I avoid having to manually run the startup and shutdown scripts (or the rake tasks) for backgroundrb? I tried putting a system "rake backgroundrb:start" in my environment.rb, and it worked, but (a) it won't handle the shutdown, so if I restart the web server, I'll have two instances of backgroundrb running, which can't be good; and (b) in production, there are multiple dispatch.fcgi instances (using Apache/FastCGI), so doesn't that mean environment.rb will get called twice, once again leading me to multiple backgroundrb instances? Any input on the preferred way to do this would be most helpful. Thanks again Ezra for a wonderful resource! Mayank From ezmobius at gmail.com Tue Jul 25 12:19:47 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 25 Jul 2006 09:19:47 -0700 Subject: [Backgroundrb-devel] Production deployment? In-Reply-To: References: Message-ID: <86956532-925E-47AE-8273-A41AFCF0BAFA@gmail.com> On Jul 25, 2006, at 9:08 AM, Mayank Thanawala wrote: > Hi all, > > BackgrounDrb was exactly what I needed for a project I'm working on at > work. Everyone's really impressed, and now it's time to put the > system into production. > > My question is, how do I avoid having to manually run the startup and > shutdown scripts (or the rake tasks) for backgroundrb? > > I tried putting a system "rake backgroundrb:start" in my > environment.rb, and it worked, but (a) it won't handle the shutdown, > so if I restart the web server, I'll have two instances of > backgroundrb running, which can't be good; and (b) in production, > there are multiple dispatch.fcgi instances (using Apache/FastCGI), so > doesn't that mean environment.rb will get called twice, once again > leading me to multiple backgroundrb instances? > > Any input on the preferred way to do this would be most helpful. > > Thanks again Ezra for a wonderful resource! > > Mayank Hi Mayank- One way I have done this is to make a wrapper script for starting and stopping my rails app. I usually use mongrel and mongrel_cluster for my apps now. So I make a script that checks to see if bgdrb is already running and either restart or start it and then use system to start the mongrel cluster. Then I do the same thing for a restart or shutdown script. This way I create a single point of entry and exit from my app that I can use to start and stop the drb server. I am working hard on a new version of the plugin that addresses a bunch of issues that have shaken out now that a lot of people are using this plugin. IT will address this issue as well as others and make deployment with bgdrb even easier. Cheers- -Ezra From info at siebert-wd.de Tue Jul 25 16:39:58 2006 From: info at siebert-wd.de (Michael Siebert) Date: Tue, 25 Jul 2006 22:39:58 +0200 Subject: [Backgroundrb-devel] bgdrb hangup Message-ID: <32b567370607251339m543c99f0j166df9adbcf89e46@mail.gmail.com> Hi, i had a little problem a few minutes ago. somehow the backgroundrb process got mad and consumed all the cpu power available, i think it got an endless loop. i dont know why, no exceptions were thrown. could this be an issue of bgdrb or ruby or rails (as i load the rails environment) -- Michael Siebert www.stellar-legends.de - Weltraum-Browsergame im Alpha-Stadium -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060725/848d6f72/attachment.html From ezmobius at gmail.com Tue Jul 25 18:46:44 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 25 Jul 2006 15:46:44 -0700 Subject: [Backgroundrb-devel] bgdrb hangup In-Reply-To: <32b567370607251339m543c99f0j166df9adbcf89e46@mail.gmail.com> References: <32b567370607251339m543c99f0j166df9adbcf89e46@mail.gmail.com> Message-ID: <8520CF13-DF20-487C-9B04-A1950E668C56@gmail.com> On Jul 25, 2006, at 1:39 PM, Michael Siebert wrote: > Hi, > i had a little problem a few minutes ago. somehow the backgroundrb > process got mad and consumed all the cpu power available, i think > it got an endless loop. i dont know why, no exceptions were thrown. > could this be an issue of bgdrb or ruby or rails (as i load the > rails environment) > > -- > Michael Siebert > > www.stellar-legends.de - Weltraum-Browsergame im Alpha-Stadium Sounds like you had a infinite recursion going on or something. I haven't had that happen yet at all. But of course I am not loading all of rails into my drb server> I just load active record and the models and libs I need. I am still not sure how good it is to load all of rails into the drb server. It is convenient though but a bit wasteful imho. Cheers- -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060725/077114d4/attachment.html From hughes.james at gmail.com Tue Jul 25 19:49:42 2006 From: hughes.james at gmail.com (James Hughes) Date: Tue, 25 Jul 2006 16:49:42 -0700 Subject: [Backgroundrb-devel] can't dump: session wierdness Message-ID: <765a2c230607251649g1fe6eabasdd3a03365d4c869f@mail.gmail.com> Hi, I have "include DRbUndumped" in one of my model classes. Whenever my app accesses the session I get the following: can't dump /usr/lib/ruby/1.8/drb/drb.rb:395:in `_dump' /usr/lib/ruby/1.8/pstore.rb:349:in `dump' /usr/lib/ruby/1.8/pstore.rb:327:in `transaction' /usr/lib/ruby/1.8/cgi/session/pstore.rb:90:in `update' /usr/lib/ruby/1.8/cgi/session/pstore.rb:97:in `close' /usr/lib/ruby/1.8/cgi/session.rb:330:in `close' Any thoughts on what is happening here? jh -- James Hughes Web application developer Vancouver, BC "Developing a coherent political analysis is in many respects contingent upon an ability to connect one context to another, a process not dissimilar to playing the kid's game of dot-to-dot." - Ward Churchill, from '"A Government of Laws"?' From ezmobius at gmail.com Tue Jul 25 19:59:17 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 25 Jul 2006 16:59:17 -0700 Subject: [Backgroundrb-devel] can't dump: session wierdness In-Reply-To: <765a2c230607251649g1fe6eabasdd3a03365d4c869f@mail.gmail.com> References: <765a2c230607251649g1fe6eabasdd3a03365d4c869f@mail.gmail.com> Message-ID: On Jul 25, 2006, at 4:49 PM, James Hughes wrote: > Hi, > > I have "include DRbUndumped" in one of my model classes. Whenever my > app accesses the session I get the following: > > can't dump > /usr/lib/ruby/1.8/drb/drb.rb:395:in `_dump' > /usr/lib/ruby/1.8/pstore.rb:349:in `dump' > /usr/lib/ruby/1.8/pstore.rb:327:in `transaction' > /usr/lib/ruby/1.8/cgi/session/pstore.rb:90:in `update' > /usr/lib/ruby/1.8/cgi/session/pstore.rb:97:in `close' > /usr/lib/ruby/1.8/cgi/session.rb:330:in `close' > > > Any thoughts on what is happening here? > > jh > > -- > James Hughes > Web application developer > Vancouver, BC Hey James- What are you trying to store in the session? Are you storing the entire model object or just the id? I've seen that error when you try to store something in the session that can't be marshaled. -Ezra From chris at etechdata.com.au Sun Jul 30 20:11:04 2006 From: chris at etechdata.com.au (Chris H) Date: Mon, 31 Jul 2006 10:11:04 +1000 Subject: [Backgroundrb-devel] MY worker won't stop working In-Reply-To: <7CF7FEE5-0424-4369-B56F-04FB787DDEC8@gmail.com> References: <765a2c230607181151n144900d1ge0bf1ed44b798741@mail.gmail.com> <64A09144-148A-4027-B0C7-84847AFCE9BD@gmail.com> <765a2c230607191041v4fdbc428nd1ac5e9540224b3c@mail.gmail.com> <44C5B93C.60505@etechdata.com.au> <7CF7FEE5-0424-4369-B56F-04FB787DDEC8@gmail.com> Message-ID: <44CD4A98.9000507@etechdata.com.au> Hi Ezra, thanks for the reply. There's a ruby process that appears in top when I fire off the do_work method. It uses around 30-50% cpu and disappears once all processing has completed. When I try to stop processing using delete_worker I was expecting this ruby process to stop, but understand/expect that backgroundrb is still running. I tried MiddleMan[:process_watcher].terminate && MiddleMan.delete_worker (:process_watcher) however this seems to do less then MiddleMan.delete_worker (:process_watcher) on its own. As Michael Siebert wrote, will I need to have do_work return when I want my process to stop? Thanks in advance, Chris. > Hi~ > > On Jul 24, 2006, at 11:25 PM, Chris H wrote: > >> Hi Ezra, >> >> love backgroundrb! Awesome plugin! > > > Thanks! > >> >> I have a worker that processes some data files and loads this data >> in mysql. >> It runs fine and processes all the files. >> >> However if I try to call delete_worker from my controller while it is >> processing , >> it says it has stopped the worker but I can still see the process >> in top and also see the rows still going into mysql. >> >> >> My do_work looks like this: >> >> def do_work(args) >> # This method is called in it's own new thread when you >> # call new worker. args is set to :args >> if args >> @process_cdr = ProcessCdr.new(args) >> else >> @process_cdr = ProcessCdr.new >> end >> @process_cdr.process >> >> end >> >> I have start and stop actions in my controller: >> >> def start >> >> if @params[:cdr_file].to_s.length > 0 >> >> MiddleMan[:process_watcher] ||MiddleMan.new_worker(:class => >> :watch_cdr_process_worker, :args => @params [:cdr_file].to_s, >> :job_key => >> :process_watcher) >> >> else >> >> MiddleMan[:process_watcher] || MiddleMan.new_worker(:class => >> :watch_cdr_process_worker, :job_key => :process_watcher) >> >> end >> >> redirect_to :action => 'status' >> >> end >> >> def stop >> >> MiddleMan.delete_worker(:process_watcher) if MiddleMan >> [:process_watcher] > > > Middleman[:process_watcher].terminate && MiddleMan.delete_worker > (:process_watcher) > > >> redirect_to :action => 'status' >> >> end >> >> I have some ajax code monitoring the status of my worker and when I >> choose stop it responds that my worker has indeed stopped. >> >> I also tried adding a method stop_work to my worker which called >> self.terminate but this didn't seem to work either. >> >> I'm confused as to what I should be doing to get my worker to stop. >> >> Thanks in advance, >> Chris. > > > Chris- > > When you say you can see the process in top still what do you > mean? Killing one worker will not stop the whole backgroundrb process > so it will still show in top. The way terminate works is that you > call it and it will signal the thread handle that it is ok to delete > this worker. THen delete_worker will actually delete it. The next > release will have a method you can use in your worker to kill it from > within itself when its finished. > > See if using terminate this way works for you and let me know. > > Thanks > -Ezra > > > > > From bjohnson at contuitive.com Sun Jul 30 23:49:40 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Sun, 30 Jul 2006 22:49:40 -0500 Subject: [Backgroundrb-devel] MySQL too many connections? Message-ID: <5C9B8BF7-2B0E-47C9-8A31-CCA99A678E45@contuitive.com> Does any have any idea why I would get: Mysql::Error: Too many connections I am creating new threads in my background process. I know allow_concurrency is set to true. Does this create a new database connection when a new thread is created? Because what I'm doing is similar to this: @threads[model.id] = Thread.new do end I also kill off these threads and over write some of the @threads keys. Does it clean up the connections for me or do I need to do this manually? Lastly, could this be because 2 threads are trying to access the database at the same time? Thanks for your help. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060730/56c4499f/attachment.html From bjohnson at contuitive.com Mon Jul 31 04:08:08 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Mon, 31 Jul 2006 03:08:08 -0500 Subject: [Backgroundrb-devel] BackgroundRB time is not consistent with rails. Message-ID: I added the following in my rails environment: ENV['TZ'] = 'UTC' Background rb does not have the same time as my rails app. In fact it is a few hours behind. Any idea why this is? Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/efbf2816/attachment.html From ezmobius at gmail.com Mon Jul 31 10:44:02 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 31 Jul 2006 07:44:02 -0700 Subject: [Backgroundrb-devel] MY worker won't stop working In-Reply-To: <44CD4A98.9000507@etechdata.com.au> References: <765a2c230607181151n144900d1ge0bf1ed44b798741@mail.gmail.com> <64A09144-148A-4027-B0C7-84847AFCE9BD@gmail.com> <765a2c230607191041v4fdbc428nd1ac5e9540224b3c@mail.gmail.com> <44C5B93C.60505@etechdata.com.au> <7CF7FEE5-0424-4369-B56F-04FB787DDEC8@gmail.com> <44CD4A98.9000507@etechdata.com.au> Message-ID: <57BA470F-5D6F-4D86-8463-518E254785BC@brainspl.at> On Jul 30, 2006, at 5:11 PM, Chris H wrote: > Hi Ezra, > > thanks for the reply. > > There's a ruby process that appears in top when I fire off the do_work > method. > It uses around 30-50% cpu and disappears once all processing has > completed. > > When I try to stop processing using delete_worker I was expecting this > ruby process to stop, > but understand/expect that backgroundrb is still running. > > I tried > MiddleMan[:process_watcher].terminate && MiddleMan.delete_worker > (:process_watcher) > however this seems to do less then MiddleMan.delete_worker > (:process_watcher) on its own. > > As Michael Siebert wrote, will I need to have do_work return when I > want > my process to stop? > > Thanks in advance, > Chris. Hey Chris- By itself Backgroundrb won't spawn any extra processes except for itself. Are you doing a system call or using `` to call an external program? Or are you using fork at all? Maybe if you paste your worker code and how you are calling it here I can help you track down the issue. -Ezra From ezmobius at gmail.com Mon Jul 31 10:50:59 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 31 Jul 2006 07:50:59 -0700 Subject: [Backgroundrb-devel] MySQL too many connections? In-Reply-To: <5C9B8BF7-2B0E-47C9-8A31-CCA99A678E45@contuitive.com> References: <5C9B8BF7-2B0E-47C9-8A31-CCA99A678E45@contuitive.com> Message-ID: On Jul 30, 2006, at 8:49 PM, Ben Johnson wrote: > Does any have any idea why I would get: > > Mysql::Error: Too many connections > > I am creating new threads in my background process. I know > allow_concurrency is set to true. Does this create a new database > connection when a new thread is created? Because what I'm doing is > similar to this: > > @threads[model.id] = Thread.new do > > end > > I also kill off these threads and over write some of the @threads > keys. Does it clean up the connections for me or do I need to do > this manually? > > Lastly, could this be because 2 threads are trying to access the > database at the same time? > > Thanks for your help. > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com Hey Ben- I'm not sure if it is a good idea to use threads within a worker class. Since the workers are each run in their own threads and you are spawning more threads within those threads, I am not sure how that will behave. It's not clear from the ActiveRecord docs what allow_concurrency really does. It is quite possible that you may be opening new db connections for each thread but I don't think that is the case. Can you monitor mysql to see how many connection handles are open when you are running your workers? Also if you are using threads in your worker you may need to join these threads to make sure they clean up after themselves. So in your worker class after the work is done and before you delete the worker it would probably be a good idea to do @threads[model.id].join BgDRb already makes heavy use of threads and mutexes so I don't think its a good idea to create more threads in your worker classes. You would be better off instantiating a new worker for each task instead of multiple threads within one worker. Of course if you can make it work then more power to you. I enjoy seeing this plugin pushed as far as it can go ;) -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/238e1891/attachment.html From bjohnson at contuitive.com Mon Jul 31 15:00:08 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Mon, 31 Jul 2006 14:00:08 -0500 Subject: [Backgroundrb-devel] Creating workers on server startup? Message-ID: <02DBD389-BBBC-49CD-A387-B5602BAD1156@contuitive.com> I have workers that need to be created when I run rake backgroundrb:start. Where do I put this code? Thanks for your help. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/d201c052/attachment.html From jedgecombe at carolina.rr.com Mon Jul 31 15:19:46 2006 From: jedgecombe at carolina.rr.com (Jason Edgecombe) Date: Mon, 31 Jul 2006 15:19:46 -0400 Subject: [Backgroundrb-devel] Starting backgroundrb from rails and restarting with rails Message-ID: <44CE57D2.9080300@carolina.rr.com> Hi, I have my rails sites tricked out with capistrano, and backgroundrb, so I can easily use the ant tasks, but I would like to be able to start and stop backgroundrb from within rails. I have a few reasons for this: 1. Using fastcgi, backgroundrb would start under the apache user and the same mod_security context as apache, instead of my developer account which has many more privileges. 2. I'd like rails to be able to start backgroundrb if it isn't running. 3. I'd also like rails to be to restart backgroundrb if rails itself has been restarted. 4. I'm lazy. ;) I'm thinking of having rails store the time that rails started in backgroundrb and restart backgroundrb if the time if the rails time is later than the backgroundrb time. Any pointers? Thanks, Jason Edgecombe From bjohnson at contuitive.com Mon Jul 31 16:06:43 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Mon, 31 Jul 2006 15:06:43 -0500 Subject: [Backgroundrb-devel] delete_worker doesn't kill the thread? Message-ID: <92C2499E-2C89-45EC-B50F-3A1AB91C8EA0@contuitive.com> I made a worker that run an infinite loop and does the following: while true @logger << "logging #{Time.now}" sleep 2 end In my secong console I did: tail log/backgroundrb.log -f This is so I could see the line getting added to the log every 2 seconds. Then in my other console I started script/console and created a new worker. Then killed it with MiddleMan.delete_worker(job_key), where job_key was the job key for the worker. After I did that the other console that was monitoring my log continued to add a line every 2 seconds, it didn't stop. Am I missing something here or shouldn't deleting a worker stop the loop and kill the thread? Thanks for your help. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/6d22b016/attachment.html From info at siebert-wd.de Mon Jul 31 16:17:44 2006 From: info at siebert-wd.de (Michael Siebert) Date: Mon, 31 Jul 2006 22:17:44 +0200 Subject: [Backgroundrb-devel] [PATCH-SUGGESTION] autostarting workers Message-ID: <32b567370607311317y6e6d60f7s57ed482b5659f0bc@mail.gmail.com> Hi there, as i stated before in my talk with Ben Johnson, I hacked a little bit and here it is - autostarting jobs. I think someone (eh... Ezra) should review it and check to see if i made any mistakes. Also i dont have the possibility to test it on other-than-Windows systems, but since the calling process works just the same, it should do well... Now the neccesary evil, the documentation... If you want a worker to start when BackgrounDRb start, you just have to add a few lines to your backgroundrb.yml file: [.. your config...] autostart: job_key: class: foo_worker args: bar args can be everything YAML allows. Since the backgroundrb.yml file is parsed by ERb _before_ YAML, conditional autostart is possible, just like in your rhtml templates. But keep in mind that the Rails environment is loaded _after_ the config file is parsed, so you won't have access to your Models! Don't forget: Have fun! -- Michael Siebert www.stellar-legends.de - Weltraum-Browsergame im Alpha-Stadium -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/57020de6/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: autostart.patch Type: application/octet-stream Size: 7066 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/57020de6/attachment-0001.obj From bjohnson at contuitive.com Mon Jul 31 16:23:30 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Mon, 31 Jul 2006 15:23:30 -0500 Subject: [Backgroundrb-devel] Please help, if @jobs[key].respond_to? :thread is returning false Message-ID: <74FD1D23-584F-4A16-9A56-0A6E839EE637@contuitive.com> I noticed in the BackgroundRB class in the delete_worker method there is a line: if @jobs[key].respond_to? :thread For some reason this is returning false for me, it gets down to this line and returns false, not killing the thread. Any idea why this is returning false? Thanks for your help. Thank You, Ben Johnson E: bjohnson at contuitive.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/3537ad90/attachment.html From me at seebq.com Mon Jul 31 19:02:21 2006 From: me at seebq.com (Charles Brian Quinn) Date: Mon, 31 Jul 2006 19:02:21 -0400 Subject: [Backgroundrb-devel] Creating workers on server startup? In-Reply-To: <02DBD389-BBBC-49CD-A387-B5602BAD1156@contuitive.com> References: <02DBD389-BBBC-49CD-A387-B5602BAD1156@contuitive.com> Message-ID: <3a2de0cd0607311602t5da93e41r96f447074a962b5f@mail.gmail.com> We currently do something like this -- we use backgroundrb for a long-running task that starts and continues running -- however we don't do it when backgroundrb starts, but rather instead execute a rake task after backgroundrb starts that continues to process. So if you were to make changes (or redeploy/restart using capistrano), you'd do: # rake backgroundrb:stop ... wait ... or sleep # rake backgroundrb:start # rake start_your_task where start_your_task is defined in your Rakefile to simply start the Middleman. We mention this in a blog posting: http://cleanair.highgroove.com/articles/2006/06/23/running-background-jobs This process has been running for a looong time now, and all is well: # ps auxww | grep backgroundrb highgroo 8002 0.2 10.7 29764 25840 ? S Jul19 45:17 backgroundrb Did i mention we love backgrounDRb? cheers On 7/31/06, Ben Johnson wrote: > > I have workers that need to be created when I run rake backgroundrb:start. > Where do I put this code? > > Thanks for your help. > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Charles Brian Quinn www.seebq.com From me at seebq.com Mon Jul 31 19:03:24 2006 From: me at seebq.com (Charles Brian Quinn) Date: Mon, 31 Jul 2006 19:03:24 -0400 Subject: [Backgroundrb-devel] Starting backgroundrb from rails and restarting with rails In-Reply-To: <44CE57D2.9080300@carolina.rr.com> References: <44CE57D2.9080300@carolina.rr.com> Message-ID: <3a2de0cd0607311603r29820992t66cd25ed91b5948f@mail.gmail.com> While I'm not addressing all your issues -- we start backgroundrb using a rake task and leave it running, so we start it up with our rails app, see my previous message to the list. To start under a user, can't you just chown apache:apache the script/backgroundrb file itself? If you're capistranoing, I'd put in the after update code or after deploy a task to start it, something like this: desc "Tasks to run after a deploy" task :after_deploy do run("cd #{current_path} && rake backgroundrb:stop") run("cd #{current_path} && rake backgroundrb:start") end And if you anticpate contention problems (i.e. it starting before it's done stopping), you could put it in a script file with a sleep 5 command in b/t to make sure stop stops it.... p.s. Ezra, maybe we can add backgroundrb:restart and status tasks? hope that helps On 7/31/06, Jason Edgecombe wrote: > Hi, > > I have my rails sites tricked out with capistrano, and backgroundrb, so > I can easily use the ant tasks, but I would like to be able to start and > stop backgroundrb from within rails. > > I have a few reasons for this: > 1. Using fastcgi, backgroundrb would start under the apache user and the > same mod_security context as apache, instead of my developer account > which has many more privileges. > 2. I'd like rails to be able to start backgroundrb if it isn't running. > 3. I'd also like rails to be to restart backgroundrb if rails itself has > been restarted. > 4. I'm lazy. ;) > > I'm thinking of having rails store the time that rails started in > backgroundrb and restart backgroundrb if the time if the rails time is > later than the backgroundrb time. > > Any pointers? > > Thanks, > Jason Edgecombe > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Charles Brian Quinn www.seebq.com From me at seebq.com Mon Jul 31 19:07:38 2006 From: me at seebq.com (Charles Brian Quinn) Date: Mon, 31 Jul 2006 19:07:38 -0400 Subject: [Backgroundrb-devel] delete_worker doesn't kill the thread? In-Reply-To: <92C2499E-2C89-45EC-B50F-3A1AB91C8EA0@contuitive.com> References: <92C2499E-2C89-45EC-B50F-3A1AB91C8EA0@contuitive.com> Message-ID: <3a2de0cd0607311607r7f092f6ct20b0cb614b81905@mail.gmail.com> Couldn't you do something like this: # in your initalize worker put: @running = true in your main work method: while @running @logger << "logging #{Time.now}" sleep 2 end add a method: def stop @running = false end Now before deleting the worker, just call stop. I am curious about the behavior you're seeing, though. On 7/31/06, Ben Johnson wrote: > > I made a worker that run an infinite loop and does the following: > > while true > @logger << "logging #{Time.now}" > sleep 2 > end > > In my secong console I did: > > tail log/backgroundrb.log -f > > This is so I could see the line getting added to the log every 2 seconds. > > Then in my other console I started script/console and created a new worker. > Then killed it with MiddleMan.delete_worker(job_key), where job_key was the > job key for the worker. > > After I did that the other console that was monitoring my log continued to > add a line every 2 seconds, it didn't stop. > > Am I missing something here or shouldn't deleting a worker stop the loop and > kill the thread? > > Thanks for your help. > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Charles Brian Quinn www.seebq.com From ezmobius at gmail.com Mon Jul 31 19:18:02 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 31 Jul 2006 16:18:02 -0700 Subject: [Backgroundrb-devel] [PATCH-SUGGESTION] autostarting workers In-Reply-To: <32b567370607311317y6e6d60f7s57ed482b5659f0bc@mail.gmail.com> References: <32b567370607311317y6e6d60f7s57ed482b5659f0bc@mail.gmail.com> Message-ID: <285BC610-5499-4532-BE74-60E616D4C26F@gmail.com> On Jul 31, 2006, at 1:17 PM, Michael Siebert wrote: > Looks good Michael. When I get a free moment I will give it a look over and commit it. Thanks -Ezra From chris at etechdata.com.au Mon Jul 31 19:30:26 2006 From: chris at etechdata.com.au (Chris H) Date: Tue, 01 Aug 2006 09:30:26 +1000 Subject: [Backgroundrb-devel] MY worker won't stop working Message-ID: <44CE9292.8050300@etechdata.com.au> > On Jul 30, 2006, at 5:11 PM, Chris H wrote: > >> Hi Ezra, >> >> thanks for the reply. >> >> There's a ruby process that appears in top when I fire off the do_work >> method. >> It uses around 30-50% cpu and disappears once all processing has >> completed. >> >> When I try to stop processing using delete_worker I was expecting this >> ruby process to stop, >> but understand/expect that backgroundrb is still running. >> >> I tried >> MiddleMan[:process_watcher].terminate && MiddleMan.delete_worker >> (:process_watcher) >> however this seems to do less then MiddleMan.delete_worker >> (:process_watcher) on its own. >> >> As Michael Siebert wrote, will I need to have do_work return when I >> want >> my process to stop? >> >> Thanks in advance, >> Chris. > > > > > Hey Chris- > > By itself Backgroundrb won't spawn any extra processes except for > itself. Are you doing a system call or using `` to call an external > program? Or are you using fork at all? Maybe if you paste your worker > code and how you are calling it here I can help you track down the > issue. > > -Ezra > Hi Ezra, Sorry I've been a bit misleading. When viewing top I had the processes ordered by cpu % usage. So when I ran my worker it appeared at the top and then disappeared off the list once it was finished. I'm not using a system call or any external program in my worker. It's main function is to read data from a file into a database. I've been successful in getting the worker to stop now by adding a stop_work method to my worker which sets @running to false in the process_cdr class and causes the code in do_work to return. My woker method stop_work calls processs_cdr.stop_process: def stop_work @process_cdr.stop_process end def do_work(args) # This method is called in it's own new thread when you # call new worker. args is set to :args if args @process_cdr = ProcessCdr.new(args) else @process_cdr = ProcessCdr.new end @process_cdr.process end When @running is set to false the loop breaks and process() returns. This is the main code from the process_cdr class: def stop_process @running = false end def process @running = true @time_start = Time.new.to_f @files_processed = 0 phone_service_accounts = PhoneServiceAccount.find(:all) phone_service_accounts.each do |myPsa| process_class_name = myPsa.process_class @current_account = myPsa.name if @file.strip.length == 0 @phone_service_cdr = PhoneServiceCdr.find(:all, :conditions => "processed = 'n'") else @phone_service_cdr = PhoneServiceCdr.find(:all, :conditions => "processed = 'n' AND name='#{@file}'") end @total_files = @phone_service_cdr.size.to_i @phone_service_cdr.each do |myPsc| @current_file = myPsc.name @process_class = eval("#{process_class_name}.new('#{CDR_DIRECTORY}#{myPsc.name}')") finished_processing_cdr = false while(@running && !finished_processing_cdr) @process_class.process_records do |record| PhoneServiceRecord.new do |phone_service_record| phone_service_number = PhoneServiceNumber.find(:first, :conditions => "number = '#{record[:charged_party]}'") customer_id = 0 customer_id = phone_service_number.customer_id if phone_service_number != nil phone_service_record.date_time = record[:date_time] phone_service_record.duration = record[:duration] phone_service_record.originating_number = record[:originating_number] phone_service_record.terminating_number = record[:terminating_number] phone_service_record.charged_party = record[:charged_party] phone_service_record.currency = record[:currency] phone_service_record.wholesale_cost = record[:wholesale_cost] phone_service_record.customer_cost = record[:customer_cost] phone_service_record.distance = record[:distance] phone_service_record.is_local = record[:is_local] phone_service_record.type_of_call = record[:type_of_call] phone_service_record.rate_code = record[:rate_code] phone_service_record.batchname = record[:batchname] phone_service_record.customer_id = customer_id phone_service_record.phone_service_account_id = myPsa.id phone_service_record.save end end finished_processing_cdr = true @files_processed += 1 myPsc.processed = 'y' #myPsc.save end return false if !@running end end @running = false end I'm not to sure if this is the correct way to go about stopping the worker. How does this look to you? Thanks in advance, Chris. From ezmobius at gmail.com Mon Jul 31 20:04:42 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 31 Jul 2006 17:04:42 -0700 Subject: [Backgroundrb-devel] Please help, if @jobs[key].respond_to? :thread is returning false In-Reply-To: <74FD1D23-584F-4A16-9A56-0A6E839EE637@contuitive.com> References: <74FD1D23-584F-4A16-9A56-0A6E839EE637@contuitive.com> Message-ID: On Jul 31, 2006, at 1:23 PM, Ben Johnson wrote: > I noticed in the BackgroundRB class in the delete_worker method > there is a line: > > if @jobs[key].respond_to? :thread > > For some reason this is returning false for me, it gets down to > this line and returns false, not killing the thread. Any idea why > this is returning false? > > Thanks for your help. > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com Hey Ben- It is hard to say without seeing your worker class. But I will look into it and see what is happening. That would certainly explain why your delete_worker calls are not actually deleting workers. Its time for a bit of refactoring in the code base to go over all the thread handling and add some more corner test cases. Hopefully I will be able to do this this week and push a new release later this week. Cheers- -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/cc8f6313/attachment.html From bjohnson at contuitive.com Mon Jul 31 23:05:39 2006 From: bjohnson at contuitive.com (Ben Johnson) Date: Mon, 31 Jul 2006 22:05:39 -0500 Subject: [Backgroundrb-devel] Creating workers from workers? Message-ID: <43FA15C4-906F-4211-9F4C-C2215D658C1E@contuitive.com> Since MiddleMan is an uninitialized constant in a worker how do you create new workers inside a worker? Thank You, Ben Johnson E: bjohnson at contuitive.com O: 800-341-6826 M: 817-229-4258 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/75a77fa6/attachment-0001.html From ezmobius at gmail.com Mon Jul 31 23:18:18 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 31 Jul 2006 20:18:18 -0700 Subject: [Backgroundrb-devel] Creating workers from workers? In-Reply-To: <43FA15C4-906F-4211-9F4C-C2215D658C1E@contuitive.com> References: <43FA15C4-906F-4211-9F4C-C2215D658C1E@contuitive.com> Message-ID: <2C5595C3-1457-42D4-B289-909B7E8B6A73@gmail.com> On Jul 31, 2006, at 8:05 PM, Ben Johnson wrote: > Since MiddleMan is an uninitialized constant in a worker how do you > create new workers inside a worker? > > Thank You, > Ben Johnson > E: bjohnson at contuitive.com > O: 800-341-6826 > M: 817-229-4258 Ben- You would have to use the full module path to the MiddleMan class. Like this: ::BackgrounDRb::MiddleMan.new_worker .... -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060731/d27036d9/attachment.html