From gladwig at gmx.de Sun Jun 4 15:28:59 2006 From: gladwig at gmx.de (=?ISO-8859-1?Q?G=FCnter_Ladwig?=) Date: Sun, 4 Jun 2006 21:28:59 +0200 Subject: [Backgroundrb-devel] Database access for workers Message-ID: Hi, nice work on BackgrounDrb! I have a question, though: What would be the best way to enable database access for a worker, preferably through ActiveRecord models. Background: I'm trying to write a worker that periodically gets data from an external source, writes it to the database and is then displayed by Rails. Anybody done something like this? Thanks in advance! ciao, guenter From chriscodes at gmail.com Mon Jun 5 17:07:42 2006 From: chriscodes at gmail.com (Chris Martin) Date: Mon, 5 Jun 2006 14:07:42 -0700 Subject: [Backgroundrb-devel] Database access for workers Message-ID: <93334f200606051407x4b7219e0me4666342639269ab@mail.gmail.com> Funny. I subscribed to ask a question, but checked the archives first (even though there weren't any postings the other day), and there it was http://rubyforge.org/pipermail/backgroundrb-devel/2006-June/000000.html I have, more or less, the same question.. Is caching an AR object the best way to pass data to a worker class? Or is there an easy way to access data from within a worker? I've tried getting data from within there and although no error is generated, my loop that starts with the data, never starts, so I'm assuming a worker can't access rails' models by default. Nice work on this Ezra! -- Chris Martin Web Developer Open Source & Web Standards Advocate http://www.chriscodes.com/ From cremes.devlist at mac.com Tue Jun 6 18:50:12 2006 From: cremes.devlist at mac.com (cremes.devlist at mac.com) Date: Tue, 6 Jun 2006 17:50:12 -0500 Subject: [Backgroundrb-devel] no view involved... Message-ID: <840436BF-440F-4C8E-8934-D26341960186@mac.com> The canonical example of using BackgrounDRb involves updating a progress bar on some user's web page. Therefore, the long running background process has some affiliation with a view. How can backgroundrb be used for long running processes that are NOT associated with a controller/view? Is this even possible? I understand the nature of Rails is such that it does "just in time" work, but I'm hoping to twist it to my own nefarious purposes. Am I even making sense? Anyone have any ideas on how to do long running processes that are unattached to any specific controller/view instance? cr From cremes.devlist at mac.com Tue Jun 6 20:06:30 2006 From: cremes.devlist at mac.com (Chuck Remes) Date: Tue, 6 Jun 2006 19:06:30 -0500 Subject: [Backgroundrb-devel] no view involved... In-Reply-To: References: <840436BF-440F-4C8E-8934-D26341960186@mac.com> Message-ID: On Jun 6, 2006, at 6:10 PM, Ezra Zygmuntowicz wrote: > > On Jun 6, 2006, at 3:50 PM, cremes.devlist at mac.com wrote: > >> The canonical example of using BackgrounDRb involves updating a >> progress bar on some user's web page. Therefore, the long running >> background process has some affiliation with a view. >> >> How can backgroundrb be used for long running processes that are NOT >> associated with a controller/view? Is this even possible? I >> understand the nature of Rails is such that it does "just in time" >> work, but I'm hoping to twist it to my own nefarious purposes. >> >> Am I even making sense? Anyone have any ideas on how to do long >> running processes that are unattached to any specific controller/view >> instance? > Hey- > > You absolutely do not have to use progress bars or any sort of > view updating. If you just want to kick off some long running tasks > then just use MiddleMan.new_worker to create a worker and let it > go. You can reconnect any time later by just keeping the return > value of the new_worker call in a session or somewhere else. > > But I don't think I totally understand what you are trying to do. > Can you explain your problem in more detail? If you want long tasks > to run but they don't get kicked off by rails then there might not > be any need for backgroundrb. You could just run a ruby daemon. If > you tell me what you are trying to accomplish I can help you out. (Moving this back to the list.) Sure, I'll provide more detail. For a project I'm working on I've been developing a web service for uploading large files. POSTing large files (large as in hundreds of MB or several GB) is pointless, and doing it on top of SOAP even moreso. So I think I have a workaround that will fit my environment. The web service receives a request (say #AddFile) that contains various bits of info like filename, filesize, md5 checksum, some metadata, and a URI for fetching the file. The web service (running as the Rails app) will sanity check the submitted information, verify sufficient space exists for the file, generate a unique file id (uuid) and kick-off a background process to actually go fetch the file. It returns the uuid to the caller and gets ready to receive the next request. I guess you could say its a web service front-end to a storage subsystem. The long running process now needs to go out and fetch the file. It could take minutes or hours to transfer the file and I certainly don't want anyone hung up waiting for that to finish. So the background process is essentially decoupled from the Rails app once the fetch request gets handed off to it. I suppose I could add another web service call for someone to check on "bytes transferred" or something which the MM could deliver, but it isn't a necessary service. Let me know if you need more detail or some clarification. cr From ezmobius at gmail.com Tue Jun 6 20:40:33 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 6 Jun 2006 17:40:33 -0700 Subject: [Backgroundrb-devel] no view involved... In-Reply-To: References: <840436BF-440F-4C8E-8934-D26341960186@mac.com> Message-ID: <112CE913-67D3-4F01-B758-3F143E9FB656@brainspl.at> On Jun 6, 2006, at 5:06 PM, Chuck Remes wrote: > > On Jun 6, 2006, at 6:10 PM, Ezra Zygmuntowicz wrote: > >> >> On Jun 6, 2006, at 3:50 PM, cremes.devlist at mac.com wrote: >> >>> The canonical example of using BackgrounDRb involves updating a >>> progress bar on some user's web page. Therefore, the long running >>> background process has some affiliation with a view. >>> >>> How can backgroundrb be used for long running processes that are NOT >>> associated with a controller/view? Is this even possible? I >>> understand the nature of Rails is such that it does "just in time" >>> work, but I'm hoping to twist it to my own nefarious purposes. >>> >>> Am I even making sense? Anyone have any ideas on how to do long >>> running processes that are unattached to any specific controller/ >>> view >>> instance? > >> Hey- >> >> You absolutely do not have to use progress bars or any sort of >> view updating. If you just want to kick off some long running tasks >> then just use MiddleMan.new_worker to create a worker and let it >> go. You can reconnect any time later by just keeping the return >> value of the new_worker call in a session or somewhere else. >> >> But I don't think I totally understand what you are trying to do. >> Can you explain your problem in more detail? If you want long tasks >> to run but they don't get kicked off by rails then there might not >> be any need for backgroundrb. You could just run a ruby daemon. If >> you tell me what you are trying to accomplish I can help you out. > > (Moving this back to the list.) > > Sure, I'll provide more detail. > > For a project I'm working on I've been developing a web service for > uploading large files. POSTing large files (large as in hundreds of > MB or several GB) is pointless, and doing it on top of SOAP even > moreso. So I think I have a workaround that will fit my environment. > > The web service receives a request (say #AddFile) that contains > various bits of info like filename, filesize, md5 checksum, some > metadata, and a URI for fetching the file. The web service (running > as the Rails app) will sanity check the submitted information, verify > sufficient space exists for the file, generate a unique file id > (uuid) and kick-off a background process to actually go fetch the > file. It returns the uuid to the caller and gets ready to receive the > next request. I guess you could say its a web service front-end to a > storage subsystem. > > The long running process now needs to go out and fetch the file. It > could take minutes or hours to transfer the file and I certainly > don't want anyone hung up waiting for that to finish. > > So the background process is essentially decoupled from the Rails app > once the fetch request gets handed off to it. I suppose I could add > another web service call for someone to check on "bytes transferred" > or something which the MM could deliver, but it isn't a necessary > service. > > Let me know if you need more detail or some clarification. > > cr > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > Yeah that sounds like a perfect use case for backgroundrb. You can just kick off the backend worker from a normal rails action after you have validated all the stuff and the worker can stay alive and working as long as you want to leave it in there. You could have a worker something like this: class FileFetcher def initialize(options) @options = options @done = false start_file_fetch end def start_file_fetch Thread.new do # parse the details out of the options hash and start your file download # once your script has the complete file, set @done to true end end def done? @done end end This is of course a simplification. But you could do something like this and just kick off the job and let it go. Then you could have a cron job that runs every hour or so and iterates over the running FileFetcher workers and id done? is true, delete them from the pool of workers. I will soon be adding a time to live variable that you can give to new_worker so that when the job is done working and it is older then the ttl, it will be automatically deleted. -Ezra From ezmobius at gmail.com Tue Jun 6 21:06:46 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 6 Jun 2006 18:06:46 -0700 Subject: [Backgroundrb-devel] no view involved... In-Reply-To: <280A6A67-395A-4067-914D-815E7E63CDB4@mac.com> References: <840436BF-440F-4C8E-8934-D26341960186@mac.com> <112CE913-67D3-4F01-B758-3F143E9FB656@brainspl.at> <280A6A67-395A-4067-914D-815E7E63CDB4@mac.com> Message-ID: <94F90218-196F-4574-AA30-0874A9509089@gmail.com> On Jun 6, 2006, at 5:56 PM, cremes.devlist at mac.com wrote: > > On Jun 6, 2006, at 7:40 PM, Ezra Zygmuntowicz wrote: > >> Yeah that sounds like a perfect use case for backgroundrb. You can >> just kick off the backend worker from a normal rails action after you >> have validated all the stuff and the worker can stay alive and >> working as long as you want to leave it in there. You could have a >> worker something like this: >> > > Sounds good! > > Since a lot of backgroundrb relies on DRb, is it safe to assume I > could push these workers out to other DRb instances running on > other machines? Or is there a dependency on MiddleMan that would > prevent that? > > (Thanks for being so responsive!) > > cr No dependency that I can forsee, you can keep pushing things out to more drb servers on other computers. In fact the next release will allow your worker classes to have 'slaves' a slave is another drb server run in a forked process that will only stay alive as long as your worker stays alive. So your worker class could potentially start a bunch of new processes to do heavy lifting and this slave feature will make it very easy. Details soon. -Ezra From chaodoze at yahoo.com Tue Jun 6 21:08:52 2006 From: chaodoze at yahoo.com (Chih-Chao Lam) Date: Tue, 6 Jun 2006 18:08:52 -0700 Subject: [Backgroundrb-devel] Ruby threads? Message-ID: <786ddd160606061808t4d9623c1uabc28f9f542c191@mail.gmail.com> First off, Ezra, thank you for putting out BackgrounDrb. Looks like a very useful framework. I have a newbie question about ruby threads. Are there any general things to avoid doing when using ruby threads and rails? Also specifically, will using BackgrounDrb avoid the socket problem described in this link (extracted from Rails wiki): http://clipclip.org/chao/clips/detail/6216/howtosworkerthreads-in-ruby-on-rails I'm hoping it will because, presumably, BackgrounDrb runs in a separate process from the actual rails web app? Thanks, chao lam -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060606/4a46a260/attachment.htm From cremes.devlist at mac.com Tue Jun 6 21:32:07 2006 From: cremes.devlist at mac.com (Chuck Remes) Date: Tue, 6 Jun 2006 20:32:07 -0500 Subject: [Backgroundrb-devel] no view involved... In-Reply-To: <94F90218-196F-4574-AA30-0874A9509089@gmail.com> References: <840436BF-440F-4C8E-8934-D26341960186@mac.com> <112CE913-67D3-4F01-B758-3F143E9FB656@brainspl.at> <280A6A67-395A-4067-914D-815E7E63CDB4@mac.com> <94F90218-196F-4574-AA30-0874A9509089@gmail.com> Message-ID: On Jun 6, 2006, at 8:06 PM, Ezra Zygmuntowicz wrote: > > No dependency that I can forsee, you can keep pushing things out > to more drb servers on other computers. In fact the next release > will allow your worker classes to have 'slaves' a slave is another > drb server run in a forked process that will only stay alive as > long as your worker stays alive. So your worker class could > potentially start a bunch of new processes to do heavy lifting and > this slave feature will make it very easy. Details soon. > > -Ezra Ah, the slave gem! I've been futzing with that lately too... You're doing great work. Better yet, you're doing work that I would be doing if it wasn't for you. Er, or something like that! :-) cr From ezmobius at gmail.com Tue Jun 6 21:41:50 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 6 Jun 2006 18:41:50 -0700 Subject: [Backgroundrb-devel] Ruby threads? In-Reply-To: <786ddd160606061808t4d9623c1uabc28f9f542c191@mail.gmail.com> References: <786ddd160606061808t4d9623c1uabc28f9f542c191@mail.gmail.com> Message-ID: <3E8E9E58-8547-4153-81BD-7C8CAD4C09A8@brainspl.at> On Jun 6, 2006, at 6:08 PM, Chih-Chao Lam wrote: > First off, Ezra, thank you for putting out BackgrounDrb. Looks like > a very useful framework. > > I have a newbie question about ruby threads. Are there any general > things to avoid doing when using ruby threads and rails? > > Also specifically, will using BackgrounDrb avoid the socket problem > described in this link (extracted from Rails wiki): > http://clipclip.org/chao/clips/detail/6216/howtosworkerthreads-in- > ruby-on-rails > > I'm hoping it will because, presumably, BackgrounDrb runs in a > separate process from the actual rails web app? > > Thanks, > chao lam > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel You're welcome, thanks for the kind words! And you are correct, your worker classes run in a separate ruby interpreter in its own separate process. But currently all your workers run in threads in the same process so you have to be mindful. I think you should be fine doing what you want with sockets. And if you end up running into issues, I am working on an addition that will allow your workers to spawn their own slave child processes. So stay tuned. And post back here if you run into any issues. Cheers- -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060606/cf967077/attachment-0001.htm From mark at martianrover.com Mon Jun 12 15:32:15 2006 From: mark at martianrover.com (P.Mark Anderson) Date: Mon, 12 Jun 2006 12:32:15 -0700 Subject: [Backgroundrb-devel] Database access for workers Message-ID: <1E69924D-8AC3-4A11-99AA-069CFB2E0082@martianrover.com> > http://rubyforge.org/pipermail/backgroundrb-devel/2006-June/ > 000000.html > What would be the best way to enable database access for a worker, > preferably through ActiveRecord models. Background: I'm trying to > write a worker that periodically gets data from an external source, > writes it to the database and is then displayed by Rails. Nobody seems to be answering this problem, so how can we figure out how to access an ActiveRecord model class from within a worker? pmark From gladwig at gmx.de Mon Jun 12 17:06:22 2006 From: gladwig at gmx.de (=?ISO-8859-1?Q?G=FCnter_Ladwig?=) Date: Mon, 12 Jun 2006 23:06:22 +0200 Subject: [Backgroundrb-devel] Database access for workers In-Reply-To: <1E69924D-8AC3-4A11-99AA-069CFB2E0082@martianrover.com> References: <1E69924D-8AC3-4A11-99AA-069CFB2E0082@martianrover.com> Message-ID: <4F18BC9C-C5E7-4F9D-A683-45984115FEF8@gmx.de> On 12.06.2006, at 21:32, P.Mark Anderson wrote: > >> http://rubyforge.org/pipermail/backgroundrb-devel/2006-June/ >> 000000.html >> What would be the best way to enable database access for a worker, >> preferably through ActiveRecord models. Background: I'm trying to >> write a worker that periodically gets data from an external source, >> writes it to the database and is then displayed by Rails. > > > Nobody seems to be answering this problem, so how can we figure out > how to access an ActiveRecord model class from within a worker? Ezra did reply, although apparently he did not send his response the list. Here it is: [....snip...] Hello G?nter- Thanks for the kind words. It's pretty easy to get access to ActiveRecord in a worker class. Here is a simple stab at a worker that can do this: require 'rubygems' require 'active_record' # Set up connection ActiveRecord::Base.establish_connection( :adapter => 'mysql', :username => 'root', :host => 'localhost', :password => 'SECRET', :database => 'foo_development' ) # require your model classes from your rails app. Don't require observers or notifiers. Dir["#{RAILS_ROOT}/app/models/**/*.rb"].each do |file| require file unless file =~ /_observer\.rb|_notifier\.rb/ end class WorkerWithAR include DRbUndumped def initialize(args) @args = args start_working end def start_working Thread.new do # do something with your model classes here. foo = MyModel.find(args[0]) end end end Something like that will work for what you are trying to do. BUt maybe you need to explain your problem a little bit more. You want this worker to do things at times intervals? Or on command from your rails app? If it is timed intervals and you are just writing to the database then you may not need backgroundrb. You could get away with a ruby script and cron. If you let me know a little more detail about what it is you are trying to accomplish, maybe I can suggest some other design patterns for you. Cheers- -EZra [...snip...] Works for me :) You can modify it to load the database configuration from config/database.yml (although you have to change the environment manually, I'm afraid): require 'yaml' @rails_config = YAML::load(File.open("#{RAILS_ROOT}/config/ database.yml")) ActiveRecord::Base.establish_connection( :adapter => @rails_config['development']['adapter'], :username => @rails_config['development']['username'], :host => @rails_config['development']['host'], :password => @rails_config['development']['password'], :database => @rails_config['development']['database'] ) ciao, G?nter From ezmobius at gmail.com Mon Jun 12 18:08:56 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 12 Jun 2006 15:08:56 -0700 Subject: [Backgroundrb-devel] [ANN] New release! Message-ID: Hey Folks- I have just pushed a new release of BackgrounDRb to rubyforge. It has some nice new features to allow for usage of your ActiveRecord Models within your worker classes. I also added a config file for setting hosts and ports and if you want to load rails or not. Please have a look and let me know if anyone runs into any issues. If you already have an older version installed then you need to delete the script/backgroundrb dir from your rails app and reinstall the plugin. You can leave your lib/workers dir alone and it won't get clobbered. Once you have run rake backgroundrb:setup with the new plugin you need to look into config/backlgroundrb.yml for your settings. The defaults are fine but you need to be aware of them. You can set whether ActiveRecord connects to your development or production databases in the conf file. It will use your existing database.ymkl to make the connection. This is a new improved way to create your worker classes. You just need to inherit from BackgrounDRb::Rails and define a do_work(args) method. This method will automatically get called in its own thread when you create a new worker from rails. So when you say: MiddleMan.new_worker :class => :foo_worker, :args => "Hello!" The "Hello" argument gets sent to your do_work method. # Put your code that runs your task inside the do_work method # it will be run automatically in a thread. You have access to # all of your rails models if you set load_rails to true in the # config file. You also get @logger inside of this class by default. class FooWorker < BackgrounDRb::Rails def do_work(args) # This method is called in it's own new thread when you # call new worker. args is set to :args. @logger.debug Post.find(:all).to_yaml end end So you no longer have to deal with THreads yourself. Just inherit from the right class and this is taken care of for you. Cheers- -Ezra From mark at martianrover.com Tue Jun 13 03:38:29 2006 From: mark at martianrover.com (P.Mark Anderson) Date: Tue, 13 Jun 2006 00:38:29 -0700 Subject: [Backgroundrb-devel] [ANN] New release! In-Reply-To: References: Message-ID: > Please have a look and let me know if anyone runs into any issues. Ezra, This is fantastic! However, it's not working for me. I think it's because my app's custom config (in config/environments/stuff.rb) does not appear to be loaded along with the rest of Rails. Could that be? uninitialized constant StuffModule Thanks for your awesome work, pmark From ezmobius at gmail.com Tue Jun 13 11:48:08 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 13 Jun 2006 08:48:08 -0700 Subject: [Backgroundrb-devel] [ANN] New release! In-Reply-To: References: Message-ID: <2ACBD8EC-8A0E-49B5-80DB-AFA0B9EACEA9@brainspl.at> On Jun 13, 2006, at 12:38 AM, P.Mark Anderson wrote: > >> Please have a look and let me know if anyone runs into any issues. > > Ezra, > This is fantastic! > However, it's not working for me. I think it's because my app's > custom config (in config/environments/stuff.rb) does not appear to be > loaded along with the rest of Rails. Could that be? > > uninitialized constant StuffModule > > Thanks for your awesome work, > pmark > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > Hey PMark- Yeah that's what it is. When backgroundrb starts up and the load_rails config is set to true, all I do is require boot.rb and then set up the db connection. If you have other classes that don't get loaded when just boot.rb is required then you will need to explicitly require them. BackgrounDRb doesn't need all of rails loaded. THat would make it consume way more memory then it needs to run. I am open for suggestions though on how to better handle this. Maybe I can create a config file option where you can list the classes you need to require. For now you should just open script/backgroundrb/start and add a require for the file you need. USe rails root to qualify its path. So say you have RAILS_ROOT/lib/stuff_module.rb that you need to require. Look for this line and add your require right after it. # Require all worker classes in lib/workers/ Dir["#{RAILS_ROOT}/lib/workers/*"].each{ |worker| require worker } require "#{RAILS_ROOT}/lib/stuff_module" Let me know if you run into any more problems. Cheers- -Ezra From mark at martianrover.com Wed Jun 14 20:15:30 2006 From: mark at martianrover.com (P.Mark Anderson) Date: Wed, 14 Jun 2006 17:15:30 -0700 Subject: [Backgroundrb-devel] [ANN] New release! and Database access for workers In-Reply-To: <2ACBD8EC-8A0E-49B5-80DB-AFA0B9EACEA9@brainspl.at> References: <2ACBD8EC-8A0E-49B5-80DB-AFA0B9EACEA9@brainspl.at> Message-ID: <6EAE755C-6791-4EAE-BE9C-65C09C1F1736@martianrover.com> > > BackgrounDRb doesn't need all of rails loaded. THat would make it > consume way more memory then it needs Hi Ezra - I got it to work for me. At first I changed the start script and config to include only the libs I needed, but then I needed a plugin (acts_as_taggable), so I included that too, but then the 'logger' instance used by my models wasn't defined, so I decided to simply include the whole Rails environment and now all is well. Is it the most efficient use of memory? No. But oh well. I need it. To include the whole Rails environment I made this change in script/ backgroundrb/start: if CONFIG['load_rails'] require "#{RAILS_ROOT}/config/environment" end Alternatively, you can pick and choose environment config libs and plugins by adding this to your config/backgroundrb.yml: libs: environments: pocket_config some_other_config plugins: acts_as_taggable some_other_plugin And then change script/backgroundrb/start: # Require all worker classes in lib/workers/ Dir["#{RAILS_ROOT}/lib/workers/*"].each{ |worker| require worker } # Load other libraries unless CONFIG['libs'].nil? env_libs = CONFIG['libs']['environments'] plugin_libs = CONFIG['libs']['plugins'] env_libs.each {|lib| require "#{RAILS_ROOT}/config/environments/# {lib}"} unless env_libs.nil? plugin_libs.each {|plugin| require "#{RAILS_ROOT}/vendor/plugins/# {plugin}/init"} unless plugin_libs.nil? end HTH. I might name my firstborn Ezra now. pmark From ezmobius at gmail.com Wed Jun 14 22:21:45 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 14 Jun 2006 19:21:45 -0700 Subject: [Backgroundrb-devel] [ANN] New release! and Database access for workers In-Reply-To: <6EAE755C-6791-4EAE-BE9C-65C09C1F1736@martianrover.com> References: <2ACBD8EC-8A0E-49B5-80DB-AFA0B9EACEA9@brainspl.at> <6EAE755C-6791-4EAE-BE9C-65C09C1F1736@martianrover.com> Message-ID: <0B38AD5D-9649-4344-AACE-2A43E7FA12AD@gmail.com> Hi- >> >> BackgrounDRb doesn't need all of rails loaded. THat would make it >> consume way more memory then it needs > > > Hi Ezra - I got it to work for me. At first I changed the start > script and config to include only the libs I needed, but then I > needed a plugin (acts_as_taggable), so I included that too, but then > the 'logger' instance used by my models wasn't defined, so I decided > to simply include the whole Rails environment and now all is well. > Is it the most efficient use of memory? No. But oh well. I need it. > > To include the whole Rails environment I made this change in script/ > backgroundrb/start: > > if CONFIG['load_rails'] > require "#{RAILS_ROOT}/config/environment" > end > > > Alternatively, you can pick and choose environment config libs and > plugins by adding this to your config/backgroundrb.yml: > > libs: > environments: > pocket_config > some_other_config > plugins: > acts_as_taggable > some_other_plugin > > > And then change script/backgroundrb/start: > > # Require all worker classes in lib/workers/ > Dir["#{RAILS_ROOT}/lib/workers/*"].each{ |worker| require worker } > > # Load other libraries > unless CONFIG['libs'].nil? > env_libs = CONFIG['libs']['environments'] > plugin_libs = CONFIG['libs']['plugins'] > env_libs.each {|lib| require "#{RAILS_ROOT}/config/environments/# > {lib}"} unless env_libs.nil? > plugin_libs.each {|plugin| require "#{RAILS_ROOT}/vendor/plugins/# > {plugin}/init"} unless plugin_libs.nil? > end > > > HTH. > I might name my firstborn Ezra now. > pmark Cool man I'm glad its working good for you. And thanks for the additions. Now that we have a real config file stuff like this is easy to add. I'll check this in when I get a chance. Can you do me a favor? Start up your drb server that requires all your rails stuff and use top to see how much mem it takes up. If you can just copy paste some top output that would be cool. I'm curious to see how much ram it takes up with everything loaded. When I run it without ActiveRecord or anything other then itself and a few workers it only takes up 2-3Mb of ram. ActiveRecord adds a bit to this bringing it up to 8Mb or so. Cheers- -Ezra From mark at martianrover.com Thu Jun 15 01:27:32 2006 From: mark at martianrover.com (P.Mark Anderson) Date: Wed, 14 Jun 2006 22:27:32 -0700 Subject: [Backgroundrb-devel] [ANN] New release! and Database access for workers In-Reply-To: <0B38AD5D-9649-4344-AACE-2A43E7FA12AD@gmail.com> References: <2ACBD8EC-8A0E-49B5-80DB-AFA0B9EACEA9@brainspl.at> <6EAE755C-6791-4EAE-BE9C-65C09C1F1736@martianrover.com> <0B38AD5D-9649-4344-AACE-2A43E7FA12AD@gmail.com> Message-ID: <770D08B7-8055-49C8-A63F-BFD023F2D0EA@martianrover.com> >>> >>> BackgrounDRb doesn't need all of rails loaded. THat would make it >>> consume way more memory then it needs >> >> >> ...I decided to simply include the whole Rails environment... > > Can you do me a favor? Start up your drb server that requires all > your rails stuff and use top to see how much mem it takes up. If > you can just copy paste some top output that would be cool. I'm > curious to see how much ram it takes up with everything loaded. > When I run it without ActiveRecord or anything other then itself > and a few workers it only takes up 2-3Mb of ram. ActiveRecord adds > a bit to this bringing it up to 8Mb or so. > 602M free goes to 581M free, so it consumes ~20 megabytes. Running BackgrounDRb this way is equivalent to running the console. Too bad, since the only thing missing after loading my AR models and extra libs is the @logger instance (I think). Mark From dorian.taylor.lists at gmail.com Thu Jun 15 08:42:57 2006 From: dorian.taylor.lists at gmail.com (dorian taylor) Date: Thu, 15 Jun 2006 05:42:57 -0700 Subject: [Backgroundrb-devel] Cannot invoke ActiveRecord subclass accessors in DRb worker Message-ID: Hi there, I'm trying to create a worker to do some asynchronous web client work and update the data model with its results. It seems, however, that every call I make to an ActiveRecord subclass causes the whole thing to block. At least I think it's blocked. Right now I'm just using debug log output. Is there a decent way to debug/test workers I can try? Is there any insight as to why this might be happening (thread safety issues, etc)? Cheers Dorian From ezmobius at gmail.com Thu Jun 15 10:55:10 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 15 Jun 2006 07:55:10 -0700 Subject: [Backgroundrb-devel] Cannot invoke ActiveRecord subclass accessors in DRb worker In-Reply-To: References: Message-ID: <0D2200F0-2630-4987-8332-26DF9E360ED8@brainspl.at> On Jun 15, 2006, at 5:42 AM, dorian taylor wrote: > Hi there, > > I'm trying to create a worker to do some asynchronous web client work > and update the data model with its results. It seems, however, that > every call I make to an ActiveRecord subclass causes the whole thing > to block. At least I think it's blocked. Right now I'm just using > debug log output. > > Is there a decent way to debug/test workers I can try? > > Is there any insight as to why this might be happening (thread safety > issues, etc)? > > Cheers > > Dorian > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > Dorian- Can you show me an example of your worker? You can shave it down to just the problem area. I will try to reproduce the problem and get back to you with an answer. But yeah threads can be hard to debug. But each worker does run in it's own thread and I have tested workers that use activerecord models by starting a bunch of them at once and they did not block and performed as they should. But I am interested to find any problems and fix them so please give a concrete example. Thanks -Ezra From dorian.taylor.lists at gmail.com Thu Jun 15 16:04:25 2006 From: dorian.taylor.lists at gmail.com (dorian taylor) Date: Thu, 15 Jun 2006 13:04:25 -0700 Subject: [Backgroundrb-devel] Cannot invoke ActiveRecord subclass accessors in DRb worker In-Reply-To: References: <0D2200F0-2630-4987-8332-26DF9E360ED8@brainspl.at> Message-ID: argh, forgot to add recipient. ---------- Forwarded message ---------- From: dorian taylor Date: Jun 15, 2006 1:02 PM Subject: Re: [Backgroundrb-devel] Cannot invoke ActiveRecord subclass accessors in DRb worker To: Ezra Zygmuntowicz On 6/15/06, Ezra Zygmuntowicz wrote: > Dorian- > > Can you show me an example of your worker? You can shave it down to > just the problem area. I will try to reproduce the problem and get > back to you with an answer. Sure, it's as simple as this: class FoobarWorker include DRbUndumped def initialize(options={}) @progress = 0 @logger = BACKGROUNDRB_LOGGER start_working end def start_working(args={}) Thread.new do @logger.debug "Trying to invoke stuff" person = Person.find(2) @logger.debug "First name is #{person.firstname}" end end end Or the new version: class FoobarWorker2 < BackgrounDRb::Rails def do_work (args) @logger.debug("Attempting to get ActiveRecord instance") person = Person.find(2) @logger.debug("First name is #{person.firstname}") end end In both cases, execution stops after (during?) the attempt to find() the record. I set the ulimit and no core is being dumped. The Rails is from trunk, and the database is MySQL 5.0 on a Debian/testing system. > But yeah threads can be hard to debug. But each worker does run in > it's own thread and I have tested workers that use activerecord > models by starting a bunch of them at once and they did not block and > performed as they should. But I am interested to find any problems > and fix them so please give a concrete example. I'm sure you did test workers with ActiveRecord as I imagine it would be a pretty common thing to want to do. Perhaps it's something with the underlying software then? I'm not sure. The ActiveRecord instances work fine if I invoke them directly within a Thread in script/console. Any insight would be great. Cheers Dorian From ezmobius at gmail.com Thu Jun 15 16:57:13 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 15 Jun 2006 13:57:13 -0700 Subject: [Backgroundrb-devel] Cannot invoke ActiveRecord subclass accessors in DRb worker In-Reply-To: References: <0D2200F0-2630-4987-8332-26DF9E360ED8@brainspl.at> Message-ID: . > > I'm sure you did test workers with ActiveRecord as I imagine it would > be a pretty common thing to want to do. > > Perhaps it's something with the underlying software then? I'm not > sure. The ActiveRecord instances work fine if I invoke them directly > within a Thread in script/console. > > Any insight would be great. > > Cheers > > Dorian > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > Hey Dorian- I'm not sure what could be the problem. I just made a test worker that creates, reads and writes ActiveRecords in one worker in a loop and ran 10 of them at once and it worked for me. Here is the worker: class TestWorker < BackgrounDRb::Rails def do_work(args) 10.times do |i| post = Post.create :title => "New: #{args[:id]}:#{i}" @logger.debug "#{args[:id]} : #{self.object_id}: " + (post.title) post = Post.find(args[:id]) post.title = "New title from worker #{args[:id]}: time: #{i}" post.save @logger.debug post.title end end end THen I called them like this: def index MiddleMan.new_worker :class => :test_worker, :args => {:id => 1} MiddleMan.new_worker :class => :test_worker, :args => {:id => 2} MiddleMan.new_worker :class => :test_worker, :args => {:id => 3} MiddleMan.new_worker :class => :test_worker, :args => {:id => 4} MiddleMan.new_worker :class => :test_worker, :args => {:id => 5} MiddleMan.new_worker :class => :test_worker, :args => {:id => 6} MiddleMan.new_worker :class => :test_worker, :args => {:id => 7} MiddleMan.new_worker :class => :test_worker, :args => {:id => 8} MiddleMan.new_worker :class => :test_worker, :args => {:id => 9} MiddleMan.new_worker :class => :test_worker, :args => {:id => 10} end And here is a little except from the log so you can see that it is working: 1 : 10643990: New: 1:0 New title from worker 1: time: 0 2 : 10624220: New: 2:0 New title from worker 2: time: 0 1 : 10643990: New: 1:1 3 : 10599690: New: 3:0 2 : 10624220: New: 2:1 New title from worker 1: time: 1 4 : 10574310: New: 4:0 New title from worker 3: time: 0 New title from worker 2: time: 1 1 : 10643990: New: 1:2 5 : 10549280: New: 5:0 New title from worker 4: time: 0 3 : 10599690: New: 3:1 2 : 10624220: New: 2:2 New title from worker 1: time: 2 6 : 11499220: New: 6:0 New title from worker 5: time: 0 4 : 10574310: New: 4:1 7 : 11472730: New: 7:0 New title from worker 3: time: 1 New title from worker 2: time: 2 1 : 10643990: New: 1:3 8 : 11435250: New: 8:0 New title from worker 6: time: 0 5 : 10549280: New: 5:1 New title from worker 4: time: 1 9 : 11396760: New: 9:0 New title from worker 7: time: 0 3 : 10599690: New: 3:2 I'm not sure what the problem could be in your end. If you want I can send you a copy of the sample app I have where this works but I think it should be working for you. You have the latest version of the plugin? And have you tried stoping and restarting the drb server and the webserver? I'd like to help you get this working so let me kn ow if you can't get around it. Cheers- -Ezra From dorian.taylor.lists at gmail.com Fri Jun 16 05:20:49 2006 From: dorian.taylor.lists at gmail.com (dorian taylor) Date: Fri, 16 Jun 2006 02:20:49 -0700 Subject: [Backgroundrb-devel] Cannot invoke ActiveRecord subclass accessors in DRb worker In-Reply-To: References: <0D2200F0-2630-4987-8332-26DF9E360ED8@brainspl.at> Message-ID: The problem was that we were using ERB in the config/database.yml which wasn't getting evaluated and therefore the database wouldn't connect. This trivial patch will fix that. Dorian -------------- next part -------------- A non-text attachment was scrubbed... Name: backgroundrb.diff Type: application/octet-stream Size: 1633 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060616/f8795ed9/attachment.obj From ezmobius at gmail.com Fri Jun 16 10:46:48 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 16 Jun 2006 07:46:48 -0700 Subject: [Backgroundrb-devel] Cannot invoke ActiveRecord subclass accessors in DRb worker In-Reply-To: References: <0D2200F0-2630-4987-8332-26DF9E360ED8@brainspl.at> Message-ID: <57791759-0F09-4F87-94E7-18CFA4BF41D0@gmail.com> On Jun 16, 2006, at 2:20 AM, dorian taylor wrote: > The problem was that we were using ERB in the config/database.yml > which wasn't getting evaluated and therefore the database wouldn't > connect. This trivial patch will fix that. > > Dorian > > _______________________________________________ Ahh, thanks Dorian. I hadn't even thought of that since I don't tend to use erb in my yml but I will add the patch as soon as i get a chance. Thanks! -Ezra From douglass.turner at gmail.com Fri Jun 16 10:59:47 2006 From: douglass.turner at gmail.com (Douglass Turner) Date: Fri, 16 Jun 2006 10:59:47 -0400 Subject: [Backgroundrb-devel] Need to call C++ processing beast from backgroudrb Message-ID: <9d0b89570606160759n49554ed2j2b43856880108998@mail.gmail.com> Hello, Bravo Ezra for Backgroundrb! What an elegant piece of engineering. I envision many folks like myself looking at our CPU hogging executables and finally seeing a clean path to the Web. I was wondering if any folks have a good strategy for calling out to a C++ (or any other compiled) executable from a worker. Is the only option to call system() from a worker? That feels a bit unsatisfying. How for example is the Ruby binding of ImageMagick (RMagick) done and is that perhaps a strategy for a cleaner more Ruby-ish solution. For context, I intend to call an image manipulation engine written in C++ (yacc + lex + pixel wrangler) from a worker which receives and uploaded photo from a user. My plan is to mail the processed photo back to said user using ActionMailer. Anyway, happy days Ezra. Regards, Doug Turner skype: dduuggllaa -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060616/0f3e9567/attachment.htm From ezmobius at gmail.com Fri Jun 16 11:34:28 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 16 Jun 2006 08:34:28 -0700 Subject: [Backgroundrb-devel] Need to call C++ processing beast from backgroudrb In-Reply-To: <9d0b89570606160759n49554ed2j2b43856880108998@mail.gmail.com> References: <9d0b89570606160759n49554ed2j2b43856880108998@mail.gmail.com> Message-ID: <1219ABA5-88A9-4856-9427-DFBBD17966BA@gmail.com> On Jun 16, 2006, at 7:59 AM, Douglass Turner wrote: > Hello, > > Bravo Ezra for Backgroundrb! What an elegant piece of engineering. > I envision many folks like myself looking at our CPU hogging > executables and finally seeing a clean path to the Web. > > I was wondering if any folks have a good strategy for calling out > to a C++ (or any other compiled) executable from a worker. Is the > only option to call system() from a worker? That feels a bit > unsatisfying. How for example is the Ruby binding of ImageMagick > (RMagick) done and is that perhaps a strategy for a cleaner more > Ruby-ish solution. > > For context, I intend to call an image manipulation engine written > in C++ (yacc + lex + pixel wrangler) from a worker which receives > and uploaded photo from a user. My plan is to mail the processed > photo back to said user using ActionMailer. > > Anyway, happy days Ezra. > > Regards, > Doug Turner > skype: dduuggllaa > _______________________________________________ > Backgroundrb-devel mailing list Welcome Doug! Thanks for the kind words. THe quick and easy way to call your c++ binary is with system. OR even better IO.popen. IO.popen allows you to open the binary and connect to stdin and stdout so you can read and write to the binary. But both of these are hacks compared to the better way to do it. Ruby is extremely easy ot write C or C++ extensions for, its one of the best features of the language. So I would suggest that you write a ruby/C++ extension for your imaging module and use that. There are a few ways to go about it. One way is swig, another way is just straight ruby/C++ extension and a third way is via DL. Google around and you will find much info on all three of these. I'm not familiar with your binary and I am not much of a C coder so I don't think I will be too much help but here's a nice link to get you started: http://www.onlamp.com/pub/a/onlamp/2004/11/18/extending_ruby.html Cheers- -Ezra From douglass.turner at gmail.com Fri Jun 16 13:09:43 2006 From: douglass.turner at gmail.com (Douglass Turner) Date: Fri, 16 Jun 2006 13:09:43 -0400 Subject: [Backgroundrb-devel] Need to call C++ processing beast from backgroudrb In-Reply-To: <1219ABA5-88A9-4856-9427-DFBBD17966BA@gmail.com> References: <9d0b89570606160759n49554ed2j2b43856880108998@mail.gmail.com> <1219ABA5-88A9-4856-9427-DFBBD17966BA@gmail.com> Message-ID: <9d0b89570606161009i6836f470xbf8917d913ade526@mail.gmail.com> Thanks. Got some reading to do. Cheers, Doug On 6/16/06, Ezra Zygmuntowicz wrote: > > > Welcome Doug! > > Thanks for the kind words. THe quick and easy way to call your c++ > binary is with system. OR even better IO.popen. IO.popen allows you > to open the binary and connect to stdin and stdout so you can read > and write to the binary. But both of these are hacks compared to the > better way to do it. Ruby is extremely easy ot write C or C++ > extensions for, its one of the best features of the language. So I > would suggest that you write a ruby/C++ extension for your imaging > module and use that. There are a few ways to go about it. One way is > swig, another way is just straight ruby/C++ extension and a third way > is via DL. Google around and you will find much info on all three of > these. I'm not familiar with your binary and I am not much of a C > coder so I don't think I will be too much help but here's a nice link > to get you started: > > http://www.onlamp.com/pub/a/onlamp/2004/11/18/extending_ruby.html > > Cheers- > -Ezra > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060616/4c84609e/attachment.htm From me at seebq.com Fri Jun 16 13:50:55 2006 From: me at seebq.com (Charles Brian Quinn) Date: Fri, 16 Jun 2006 13:50:55 -0400 Subject: [Backgroundrb-devel] running rake after install fails (fixed) Message-ID: <3a2de0cd0606161050x150efad7o353d2d3924da4f8f@mail.gmail.com> Hi all (and Ezra), I couldn't get my rake tasks to work after installing the plugin -- until I added a line at top of to test_helper.rb: require File.expand_path(File.dirname(__FILE__) + "/../script/backgroundrb/lib/backgroundrb.rb") Is this in the docs? -- Charles Brian Quinn www.seebq.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060616/75ac207e/attachment-0001.htm From dorian.taylor.lists at gmail.com Fri Jun 16 20:09:47 2006 From: dorian.taylor.lists at gmail.com (dorian taylor) Date: Fri, 16 Jun 2006 17:09:47 -0700 Subject: [Backgroundrb-devel] switch lines 7 and 8 in script/backgroundrb/start Message-ID: Getting a problem with RAILS_ROOT being undefined, calling config/boot.rb first fixes it. Cheers Dorian From dorian.taylor.lists at gmail.com Wed Jun 21 17:18:13 2006 From: dorian.taylor.lists at gmail.com (dorian taylor) Date: Wed, 21 Jun 2006 14:18:13 -0700 Subject: [Backgroundrb-devel] one more suggestion for backgroundrb Message-ID: this: Index: script/backgroundrb/lib/backgroundrb.rb =================================================================== --- script/backgroundrb/lib/backgroundrb.rb (revision 16) +++ script/backgroundrb/lib/backgroundrb.rb (working copy) @@ -29,7 +29,11 @@ # the log via @logger def initialize(args) @logger = BACKGROUNDRB_LOGGER - Thread.new { do_work(args) } + begin + Thread.new { do_work(args) } + rescue Exception => e + @logger.error e.inspect + end end end From ezmobius at gmail.com Wed Jun 21 17:26:05 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 21 Jun 2006 14:26:05 -0700 Subject: [Backgroundrb-devel] one more suggestion for backgroundrb In-Reply-To: References: Message-ID: <97EAF42E-4223-4FA3-B75F-80B437BF05EE@brainspl.at> On Jun 21, 2006, at 2:18 PM, dorian taylor wrote: > this: > > Index: script/backgroundrb/lib/backgroundrb.rb > =================================================================== > --- script/backgroundrb/lib/backgroundrb.rb (revision 16) > +++ script/backgroundrb/lib/backgroundrb.rb (working copy) > @@ -29,7 +29,11 @@ > # the log via @logger > def initialize(args) > @logger = BACKGROUNDRB_LOGGER > - Thread.new { do_work(args) } > + begin > + Thread.new { do_work(args) } > + rescue Exception => e > + @logger.error e.inspect > + end > end > > end > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > Dorian- Commited.. Thanks. I am all tied up doing railsconf stuff this weekend but I will be doing a little hack fest on backgroundrb while I'm there to get it into a little better shape. I will be moving all code into the plugin and only using stubs in script/backgroundrb. this will make things easier to work with and test. I appreciate your patches and feedback. Any feature requests for my hack fest? Cheers- -Ezra From me at seebq.com Wed Jun 21 18:12:29 2006 From: me at seebq.com (Charles Brian Quinn) Date: Wed, 21 Jun 2006 18:12:29 -0400 Subject: [Backgroundrb-devel] one more suggestion for backgroundrb In-Reply-To: <97EAF42E-4223-4FA3-B75F-80B437BF05EE@brainspl.at> References: <97EAF42E-4223-4FA3-B75F-80B437BF05EE@brainspl.at> Message-ID: <3a2de0cd0606211512g50dbe170xcdf71be3d07e4a4a@mail.gmail.com> Can we name that logger something different since it goes to the log/backgroundrb.log file instead of one of the dev/test/prod logs? Just a suggestion.... Great fix, though, awesome. There was another post that came by in June with code to fully load the entire Rails environment -- versus loading up just the ActiveRecord connection: -if CONFIG['load_rails'] - ActiveRecord::Base.establish_connection(YAML.load(ERB.new(IO.read ("#{RAILS_ROOT}/#{CONFIG['databse_yml']}")).result)[CONFIG['environment']]) -end +if CONFIG['load_rails'] + require "#{RAILS_ROOT}/config/environment" +end It was silently failing for me when loading up models that included required classes or utilized includes, such as a model that has: require 'net-ssh' or something like that. It was mentioned that this does use a lot of memory -- it's true, perhaps this could be another option? load_full_rails: true or document the use of loading up required modules too? One more request -- does the rake backgroundrb:start task read the config file for the environment -- I still have to use: RAILS_ENV="production" rake backgroundrb:start to get it to load in production mode. Though, you may fix this in your move to stub the scripts. Great work Ezra, we're having fun with your backgroundrb, and have a blog article coming out on it soon. Thanks! On 6/21/06, Ezra Zygmuntowicz wrote: > > > On Jun 21, 2006, at 2:18 PM, dorian taylor wrote: > > > this: > > > > Index: script/backgroundrb/lib/backgroundrb.rb > > =================================================================== > > --- script/backgroundrb/lib/backgroundrb.rb (revision 16) > > +++ script/backgroundrb/lib/backgroundrb.rb (working copy) > > @@ -29,7 +29,11 @@ > > # the log via @logger > > def initialize(args) > > @logger = BACKGROUNDRB_LOGGER > > - Thread.new { do_work(args) } > > + begin > > + Thread.new { do_work(args) } > > + rescue Exception => e > > + @logger.error e.inspect > > + end > > end > > > > end > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > Dorian- > > Commited.. Thanks. I am all tied up doing railsconf stuff this > weekend but I will be doing a little hack fest on backgroundrb while > I'm there to get it into a little better shape. I will be moving all > code into the plugin and only using stubs in script/backgroundrb. > this will make things easier to work with and test. I appreciate your > patches and feedback. > > Any feature requests for my hack fest? > > Cheers- > -Ezra > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Charles Brian Quinn www.seebq.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060621/9bc040f9/attachment.html From ezmobius at gmail.com Wed Jun 21 19:09:47 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 21 Jun 2006 16:09:47 -0700 Subject: [Backgroundrb-devel] one more suggestion for backgroundrb In-Reply-To: <3a2de0cd0606211512g50dbe170xcdf71be3d07e4a4a@mail.gmail.com> References: <97EAF42E-4223-4FA3-B75F-80B437BF05EE@brainspl.at> <3a2de0cd0606211512g50dbe170xcdf71be3d07e4a4a@mail.gmail.com> Message-ID: <41D3A239-50D6-4E50-824B-9DD4516FF82F@gmail.com> On Jun 21, 2006, at 3:12 PM, Charles Brian Quinn wrote: > Can we name that logger something different since it goes to the > log/backgroundrb.log file instead of one of the dev/test/prod > logs? Just a suggestion.... Great fix, though, awesome. > > There was another post that came by in June with code to fully load > the entire Rails environment -- versus loading up just the > ActiveRecord connection: > > -if CONFIG['load_rails'] > - ActiveRecord::Base.establish_connection(YAML.load(ERB.new(IO.read > ("#{RAILS_ROOT}/#{CONFIG['databse_yml']}")).result)[CONFIG > ['environment']]) > -end > > +if CONFIG['load_rails'] > + require "#{RAILS_ROOT}/config/environment" > +end > > It was silently failing for me when loading up models that included > required classes or utilized includes, such as a model that has: > require 'net-ssh' or something like that. It was mentioned that > this does use a lot of memory -- it's true, perhaps this could be > another option? load_full_rails: true or document the use of > loading up required modules too? > > One more request -- does the rake backgroundrb:start task read the > config file for the environment -- I still have to use: > RAILS_ENV="production" rake backgroundrb:start to get it to load in > production mode. Though, you may fix this in your move to stub the > scripts. > > Great work Ezra, we're having fun with your backgroundrb, and have > a blog article coming out on it soon. Thanks! > > > Any feature requests for my hack fest? > > Cheers- > -Ezra > > -- > Charles Brian Quinn > www.seebq.com Charles- So how would you like the logger to behave? Do you mean having it write into the rails dev and prod log files? Or just an option to rename it to something else? I will flesh out the config file and its related parser to make it much more flexible and tunable when I refactor things. Also will be adding a windows service ability. -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060621/81dcf61e/attachment.html From me at seebq.com Wed Jun 21 20:20:05 2006 From: me at seebq.com (Charles Brian Quinn) Date: Wed, 21 Jun 2006 20:20:05 -0400 Subject: [Backgroundrb-devel] one more suggestion for backgroundrb In-Reply-To: <41D3A239-50D6-4E50-824B-9DD4516FF82F@gmail.com> References: <97EAF42E-4223-4FA3-B75F-80B437BF05EE@brainspl.at> <3a2de0cd0606211512g50dbe170xcdf71be3d07e4a4a@mail.gmail.com> <41D3A239-50D6-4E50-824B-9DD4516FF82F@gmail.com> Message-ID: <3a2de0cd0606211720k345b39afnacee3481d11399a5@mail.gmail.com> On 6/21/06, Ezra Zygmuntowicz wrote: > Charles- > > So how would you like the logger to behave? Do you mean having it write > into the rails dev and prod log files? Or just an option to rename it to > something else? I will flesh out the config file and its related parser to > make it much more flexible and tunable when I refactor things. Also will be > adding a windows service ability. > Logging to the rails log files would be cool -- if it's possible. I imagine if you loaded the rails environment you could just go ahead and use logger.info, though I didn't try it. All great work, those were just off the top of my head for your hackfest. Let us know when we can help test. -- Charles Brian Quinn www.seebq.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060621/3424d376/attachment.html From gethemant at gmail.com Thu Jun 22 17:15:24 2006 From: gethemant at gmail.com (hemant) Date: Fri, 23 Jun 2006 02:45:24 +0530 Subject: [Backgroundrb-devel] Models and Backgroundrb Message-ID: Hi Erza, I am using login engine in my code...and i get the error "uninitialized constant LoginEngine and uninitialized constant LoginEngine", when i try to do a find in the worker code!!! Why am i getting this error? -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060622/5550f7bf/attachment-0001.html From me at seebq.com Thu Jun 22 17:31:53 2006 From: me at seebq.com (Charles Brian Quinn) Date: Thu, 22 Jun 2006 17:31:53 -0400 Subject: [Backgroundrb-devel] Models and Backgroundrb In-Reply-To: References: Message-ID: <3a2de0cd0606221431x3acc5074g121c7d3710242329@mail.gmail.com> Your model that you're trying find on includes some libraries that backgroundrb doesn't know about. Try including all of rails when loading the script/backgroundrb/start -- was just posted by me and someone else recently on how to load ALL of rails instead of just the ActiveRecord connection. If that solves your problem, you can either, a) keep it and take the large memory hit for running backgroundrb with your full RAILS env loaded up, or b) find out what libraries (the authenticated_system.rb, including LogineEngine, etc.) that your model that you're performing the Find on is using, and load those up in your backgroundrb process instead (see archive for how to do this, too) of the entire environment. Hope that helps! On 6/22/06, hemant wrote: > > Hi Erza, > > I am using login engine in my code...and i get the error "uninitialized > constant LoginEngine and uninitialized constant LoginEngine", when i try to > do a find in the worker code!!! > > Why am i getting this error? > > > > > -- > nothing much to talk > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Charles Brian Quinn www.seebq.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060622/d888d2e5/attachment.html From david at lemstra.ca Wed Jun 28 12:18:37 2006 From: david at lemstra.ca (David Lemstra) Date: Wed, 28 Jun 2006 12:18:37 -0400 Subject: [Backgroundrb-devel] How to add some process control... Message-ID: <44A2ABDD.3080907@lemstra.ca> Hi, First off, thanks Ezra for showing the way with BackgrounDrb. It's a real godsend. I wanted to add a bit of job control to the system so that I could cancel DRB processes and so on. I thought I'd share how I did it in case anyone should find it useful or have feedback. I wanted to be able to put checkpoints into the drb job so that I could let the job do whatever needs to be done (cleanup etc) and then terminate it gracefully, so that complicates things a little bit. The code is pretty simple, but I'm just learning DRB, so it took me a while to get it right. In MiddleMan::new_worker(), the job is first created, then started. This is because the instance functions I added to BackgrounDrb::Rails don't seem to be available when kicked off from BackgrounDRb::Rails.initialize() def new_worker(opts={}) @mutex.synchronize { job_key = opts[:job_key] || gen_key unless self[job_key] self[job_key] = instantiate_worker(opts[:class]).new(job_key,opts[:args]) self[job_key].start_process return job_key else raise ::BackgrounDRbDuplicateKeyError end } end In MiddleMan::delete_worker(), I've added steps where, if the job is still alive, activate the kill signal to the job, and wait for it to clean itself up and activate the :safe_to_kill signal. Then kill it. def delete_worker(key) @mutex.synchronize { if @jobs[key] if (@jobs[key].thread.alive?) @jobs[key].thread[:kill] = true; @jobs[key].thread[:safe_to_kill].wait(@mutex) @jobs[key].thread.kill end @jobs.delete(key) end @timestamps.delete(key) if @timestamps.has_key?(key) } end Inside BackgrounDrb:Rails: you can use terminate() to establish the checkpoints/killpoints inside your job code, or check_terminate/terminate? to check, run cleanup code, and then raise the :safe_to_kill signal. def initialize(job_key, args={}) @logger = BACKGROUNDRB_LOGGER @thread = nil; @job_key = job_key @args = args end def start_process begin @thread = Thread.new do Thread.current[:safe_to_kill] = ConditionVariable.new Thread.current[:kill] = false do_work(@args) end rescue Exception => e @logger.error e.inspect end end def check_terminate raise "Somehow this worker doesn't have a registered thread" if thread.nil? return @thread[:kill] end def terminate? terminate if check_terminate end def terminate if check_terminate Thread.critical = true @logger.info "Canceling job #{@job_key}" @thread[:safe_to_kill].signal @thread.stop end return end Hope someone finds it useful, David Lemstra From ezmobius at gmail.com Wed Jun 28 12:32:59 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 28 Jun 2006 09:32:59 -0700 Subject: [Backgroundrb-devel] How to add some process control... In-Reply-To: <44A2ABDD.3080907@lemstra.ca> References: <44A2ABDD.3080907@lemstra.ca> Message-ID: On Jun 28, 2006, at 9:18 AM, David Lemstra wrote: > Hi, > First off, thanks Ezra for showing the way with BackgrounDrb. It's a > real godsend. > > I wanted to add a bit of job control to the system so that I could > cancel DRB processes and so on. I thought I'd share how I did it in > case > anyone should find it useful or have feedback. > > > Hope someone finds it useful, > David Lemstra > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > Hey David- Thanks for the patch. It does look useful. I will add it to the next release once I get a few other features implemented. One thing I really need is a thread that sleeps and wakes up to call gc! on items based on a time to live variable. THis way you could set a cache and tell it how long to live before it gets reaped. This will fit right in with what you just submitted so that workers can clenup after themselves when told to terminate. Thanks- -Ezra From douglass.turner at gmail.com Thu Jun 29 22:45:33 2006 From: douglass.turner at gmail.com (Douglass Turner) Date: Thu, 29 Jun 2006 22:45:33 -0400 Subject: [Backgroundrb-devel] Need to send email from a worker Message-ID: <9d0b89570606291945r7c21c361tb25e36da2aebe729@mail.gmail.com> Hello, Can someone suggest the simplest way to send email from a worker? At completion, I need to send the results of it's work to a recipient. Thanks, Doug Turner skype: dduuggllaa -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060629/94d9fb64/attachment.html From gethemant at gmail.com Thu Jun 29 23:47:44 2006 From: gethemant at gmail.com (hemant) Date: Fri, 30 Jun 2006 09:17:44 +0530 Subject: [Backgroundrb-devel] Need to send email from a worker In-Reply-To: <9d0b89570606291945r7c21c361tb25e36da2aebe729@mail.gmail.com> References: <9d0b89570606291945r7c21c361tb25e36da2aebe729@mail.gmail.com> Message-ID: I guess the simplest way would be to forget about ActionMailer cock and bull...and use plain SMTP class of ruby.It takes any way..only 3 to 4 lines of coding to send mails using ruby. On 6/30/06, Douglass Turner wrote: > > Hello, > > Can someone suggest the simplest way to send email from a worker? At > completion, I need to send the results of it's work to a recipient. > > Thanks, > Doug Turner > skype: dduuggllaa > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060629/42f8f830/attachment.html From ezmobius at gmail.com Thu Jun 29 23:51:02 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 29 Jun 2006 20:51:02 -0700 Subject: [Backgroundrb-devel] Fwd: Need to send email from a worker References: <41B4A17A-D96E-4615-95AF-CF1DE1E79D76@brainspl.at> Message-ID: Begin forwarded message: > From: Ezra Zygmuntowicz > Date: June 29, 2006 8:18:10 PM PDT > To: Douglass Turner > Subject: Re: [Backgroundrb-devel] Need to send email from a worker > > > On Jun 29, 2006, at 7:45 PM, Douglass Turner wrote: > >> Hello, >> >> Can someone suggest the simplest way to send email from a worker? >> At completion, I need to send the results of it's work to a >> recipient. >> >> Thanks, >> Doug Turner >> skype: dduuggllaa >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > The simplest way to do that is probably to use net/smtp. Or you > could use your actionmailer class in the worker but that requires > you to load pretty much all of rails in your drb server which is > not very resource friendly. If its just a simple text email then I > would just use net/smtp like so: > > msgstr = < From: Your Name > To: Destination Address > Subject: test message > Date: Sat, 23 Jun 2001 16:26:43 +0900 > Message-Id: > > This is a test message. > END_OF_MESSAGE > > require 'net/smtp' > Net::SMTP.start('your.smtp.server', 25) do |smtp| > smtp.send_message msgstr, > 'your at mail.address', > 'his_addess at example.com' > end > > > Cheers- > -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060629/d3956f1f/attachment-0001.html From lars at pinds.com Fri Jun 30 01:24:50 2006 From: lars at pinds.com (Lars Pind) Date: Fri, 30 Jun 2006 07:24:50 +0200 Subject: [Backgroundrb-devel] Need to send email from a worker In-Reply-To: <9d0b89570606291945r7c21c361tb25e36da2aebe729@mail.gmail.com> References: <9d0b89570606291945r7c21c361tb25e36da2aebe729@mail.gmail.com> Message-ID: <4074E15D-55F8-43FA-9FA8-43F287370FFE@pinds.com> I'm doing this by just loading the entire rails environment from the background process and using ActionMailer. It increases the overhead of the background process significantly, but since there's just one of them, that's OK for me. That way, I was able to take something that already worked in serial, and make it parallel. /Lars On Jun 30, 2006, at 4:45 AM, Douglass Turner wrote: > Hello, > > Can someone suggest the simplest way to send email from a worker? > At completion, I need to send the results of it's work to a recipient. > > Thanks, > Doug Turner > skype: dduuggllaa > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From douglass.turner at gmail.com Fri Jun 30 09:37:33 2006 From: douglass.turner at gmail.com (Douglass Turner) Date: Fri, 30 Jun 2006 09:37:33 -0400 Subject: [Backgroundrb-devel] Rather ominous resource consumption figures Message-ID: <9d0b89570606300637w4b849cebk813dbd0bd0976fac@mail.gmail.com> Hello, Yesterday I posted about how to email from a worker. I decided to be a bit piggy and just inhale all of rails. Sorted. I am noticing some rather ominous behavior though. The setup: My worker takes an uploaded image as a param and passes it (via system call) to a C++ image processing engine. Take a look at these memory consumption figures for successive runs uploading the identical image each time: All figures in megabytes 343 C++ image processing idle 365 max memory consumption during 3rd C++ image processing run 318 C++ image processing idle 340 max memory consumption during 2nd C++ image processing run 290 C++ image processing idle 310 max memory consumption during 1st C++ image processing run 256 fire up rails + lighttpd 225 fire up backgroundrb 207 base (Linux box idle) As Sly Stone used to say ".. I want to take you hiya ..". Any thoughts on how to tame this beast? Cheers, Doug Turner skype: dduuggllaa -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060630/fc2acaf7/attachment.html From douglass.turner at gmail.com Fri Jun 30 10:05:40 2006 From: douglass.turner at gmail.com (Douglass Turner) Date: Fri, 30 Jun 2006 10:05:40 -0400 Subject: [Backgroundrb-devel] Rather ominous resource consumption figures In-Reply-To: <9d0b89570606300637w4b849cebk813dbd0bd0976fac@mail.gmail.com> References: <9d0b89570606300637w4b849cebk813dbd0bd0976fac@mail.gmail.com> Message-ID: <9d0b89570606300705q733af2acv468c95d421920c4@mail.gmail.com> Ah, GC is your friend. I use RMagick to prep my image before processing by my C++ engine. It was not releasing mems between successive calls to the worker. I now call GC.start after I nolonger need the image object created by RMagick. However, t still does appear that I have incurred a memory hit from the initial call to my worker that is never released even when the worker dies. -Doug On 6/30/06, Douglass Turner wrote: > > Hello, > > Yesterday I posted about how to email from a worker. I decided to be a bit > piggy and just inhale all of rails. Sorted. I am noticing some rather > ominous behavior though. > > The setup: My worker takes an uploaded image as a param and passes it (via > system call) to a C++ image processing engine. Take a look at these memory > consumption figures for successive runs uploading the identical image each > time: > > All figures in megabytes > > 343 C++ image processing idle > 365 max memory consumption during 3rd C++ image processing run > > 318 C++ image processing idle > 340 max memory consumption during 2nd C++ image processing run > > 290 C++ image processing idle > 310 max memory consumption during 1st C++ image processing run > > 256 fire up rails + lighttpd > 225 fire up backgroundrb > > 207 base (Linux box idle) > > > As Sly Stone used to say ".. I want to take you hiya ..". Any thoughts on > how to tame this beast? > > Cheers, > Doug Turner > skype: dduuggllaa > -- Doug Turner mobile: 781 775 3708 skype: dduuggllaa -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060630/6e4f9308/attachment.html From ezmobius at gmail.com Fri Jun 30 10:48:11 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 30 Jun 2006 07:48:11 -0700 Subject: [Backgroundrb-devel] Rather ominous resource consumption figures In-Reply-To: <9d0b89570606300705q733af2acv468c95d421920c4@mail.gmail.com> References: <9d0b89570606300637w4b849cebk813dbd0bd0976fac@mail.gmail.com> <9d0b89570606300705q733af2acv468c95d421920c4@mail.gmail.com> Message-ID: <4CA3CF19-1500-4E65-BB77-AEE4A6455083@brainspl.at> Hey On Jun 30, 2006, at 7:05 AM, Douglass Turner wrote: > Ah, > > GC is your friend. I use RMagick to prep my image before processing > by my C++ engine. It was not releasing mems between successive > calls to the worker. I now call GC.start after I nolonger need the > image object created by RMagick. > > However, t still does appear that I have incurred a memory hit > from the initial call to my worker that is never released even when > the worker dies. > > -Doug > > > On 6/30/06, Douglass Turner wrote: > Hello, > > Yesterday I posted about how to email from a worker. I decided to > be a bit piggy and just inhale all of rails. Sorted. I am noticing > some rather ominous behavior though. > > The setup: My worker takes an uploaded image as a param and passes > it (via system call) to a C++ image processing engine. Take a look > at these memory consumption figures for successive runs uploading > the identical image each time: > > All figures in megabytes > > 343 C++ image processing idle > 365 max memory consumption during 3rd C++ image processing run > > 318 C++ image processing idle > 340 max memory consumption during 2nd C++ image processing run > > 290 C++ image processing idle > 310 max memory consumption during 1st C++ image processing run > > 256 fire up rails + lighttpd > 225 fire up backgroundrb > > 207 base (Linux box idle) > > > As Sly Stone used to say ".. I want to take you hiya ..". Any > thoughts on how to tame this beast? > > Cheers, > Doug Turner > skype: dduuggllaa > In the current implementation you are responsible for calling MiddleMan.gc! with a timestamp. This will delete all workers older then the timestamp you pass to gc!. So this could acount for a perceived memory leak. So you either need to set up a cron script to expire things older then a certain age. Or you need to use MiddleMan.delete_worker(key). The reason for this is that the workers are held in the @jobs hash. So even after the worker is done working, it is not allowed to be GC'ed by ruby since the jobs hash still holds a ref to it. The next release will add a time to live param and a timer built in for expiring jobs and caches on a timed basis. For now you need to delete_worker when your done or use a cron script like this: Here is a ruby script you could run from cron that will delete all workers and cached items older then 30 minutes. #!/usr/bin/env ruby require "drb" DRb.start_service MiddleMan = DRbObject.new(nil, "druby://localhost:22222") MiddleMan.gc!(Time.now - 60*30) Let me know if this helps you or if you still have problems. Then we can figure out what the issue is and correct it for you. Cheers- -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060630/9e744c77/attachment.html