From ittay.dror at gmail.com Thu Sep 4 08:39:19 2008 From: ittay.dror at gmail.com (Ittay Dror) Date: Thu, 04 Sep 2008 15:39:19 +0300 Subject: [Rake-devel] make like variables? Message-ID: <48BFD6F7.6080602@gmail.com> Hi, Is there a way to get make-like behavior for variables? Meaning, that when passed in the command line, all settings in the rakefile are ignored. Thanks, Ittay -- -- Ittay Dror From jim.weirich at gmail.com Fri Sep 5 20:19:12 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Fri, 5 Sep 2008 19:19:12 -0500 Subject: [Rake-devel] make like variables? In-Reply-To: <48BFD6F7.6080602@gmail.com> References: <48BFD6F7.6080602@gmail.com> Message-ID: <2EB2CFB0-9C23-48AA-BBB3-22FE51CCF00B@gmail.com> On Sep 4, 2008, at 7:39 AM, Ittay Dror wrote: > Is there a way to get make-like behavior for variables? Meaning, > that when passed in the command line, all settings in the rakefile > are ignored. Can you give an example of what you want? -- -- Jim Weirich -- jim.weirich at gmail.com From jim.weirich at gmail.com Sat Sep 6 01:25:46 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Sat, 6 Sep 2008 00:25:46 -0500 Subject: [Rake-devel] Plans for 0.8.2 release In-Reply-To: References: Message-ID: On Aug 31, 2008, at 7:56 PM, Jim Weirich wrote: > The code base for 0.8.2 release is mostly feature complete. I'm > going to review the code for ruby 1.9 compatibility before the final > release. However, if you want to try the latest beta for ruby 1.8, > you can pull the beta gem from GitHub. The latest version is > 0.8.1.9 at the moment. An 0.8.1.10 beta is on the github server at the moment. I would really appreciate it if a windows user could grab that version and verify that the system dir stuff works on a windows box. If I can get someone to verify that, I'll cut a new release. -- -- Jim Weirich -- jim.weirich at gmail.com From ittay.dror at gmail.com Sat Sep 6 03:21:22 2008 From: ittay.dror at gmail.com (Ittay Dror) Date: Sat, 06 Sep 2008 10:21:22 +0300 Subject: [Rake-devel] make like variables? In-Reply-To: <2EB2CFB0-9C23-48AA-BBB3-22FE51CCF00B@gmail.com> References: <48BFD6F7.6080602@gmail.com> <2EB2CFB0-9C23-48AA-BBB3-22FE51CCF00B@gmail.com> Message-ID: <48C22F72.7090601@gmail.com> in make, if in my Makefile I have FOO = 1 FOO += 2 test: echo $(FOO) then running: > make 1 2 > make FOO=3 3 That is, if passed in the command line, any assignments to FOO are ignored. In Makefile, this can be changed by using the 'override' method. It is not that important, just an idiom that people coming from Make expect. Obviously, Rakefile being Ruby this sort of behavior can be done. Ittay Jim Weirich wrote: > > On Sep 4, 2008, at 7:39 AM, Ittay Dror wrote: > >> Is there a way to get make-like behavior for variables? Meaning, that >> when passed in the command line, all settings in the rakefile are >> ignored. > > Can you give an example of what you want? > -- -- Ittay Dror From jim.weirich at gmail.com Sat Sep 6 11:39:14 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Sat, 6 Sep 2008 10:39:14 -0500 Subject: [Rake-devel] make like variables? In-Reply-To: <48C22F72.7090601@gmail.com> References: <48BFD6F7.6080602@gmail.com> <2EB2CFB0-9C23-48AA-BBB3-22FE51CCF00B@gmail.com> <48C22F72.7090601@gmail.com> Message-ID: <941C77FE-8C38-4106-9833-7B1529B04E33@gmail.com> On Sep 6, 2008, at 2:21 AM, Ittay Dror wrote: > in make, if in my Makefile I have > FOO = 1 > FOO += 2 > test: > echo $(FOO) > > then running: > > make > 1 2 > > make FOO=3 > 3 > > That is, if passed in the command line, any assignments to FOO are > ignored. In Makefile, this can be changed by using the 'override' > method. It is not that important, just an idiom that people coming > from Make expect. Obviously, Rakefile being Ruby this sort of > behavior can be done. I think if I had to do this, I would use an idiom like: ENV['FOO'] ||= 1 I really don't have control over how ruby variables are assigned. I'm not sure how to do it in rake for any arbitrary variable. -- -- Jim Weirich -- jim.weirich at gmail.com From hgs at dmu.ac.uk Sat Sep 6 12:02:54 2008 From: hgs at dmu.ac.uk (Hugh Sasse) Date: Sat, 6 Sep 2008 17:02:54 +0100 (BST) Subject: [Rake-devel] Plans for 0.8.2 release In-Reply-To: References: Message-ID: On Sat, 6 Sep 2008, Jim Weirich wrote: > On Aug 31, 2008, at 7:56 PM, Jim Weirich wrote: > > The code base for 0.8.2 release is mostly feature complete. I'm going to > > review the code for ruby 1.9 compatibility before the final release. > > However, if you want to try the latest beta for ruby 1.8, you can pull the > > beta gem from GitHub. The latest version is 0.8.1.9 at the moment. > > > An 0.8.1.10 beta is on the github server at the moment. I would really > appreciate it if a windows user could grab that version and verify that the > system dir stuff works on a windows box. If I can get someone to verify that, > I'll cut a new release. I don't mind running a test for you, but I'm unfamiliar with github, so URLs of zip|7z|tar files would be good. Have you got specific tests in mind that you want done (I've not been following closely of late)? When you say windows, what ruby do you want? The One Click installer still seems to be on 1.8.6 and hasn't experienced the security updates of late, AFAICT. If I had a working MS compiler on here I'd offer to build a new one, but I don't. I do have cygwin on here and a more recent ruby as a result. I probably can't test on Solaris any more, but if I find I can I'll let you know. Hugh From ittay.dror at gmail.com Sat Sep 6 13:18:47 2008 From: ittay.dror at gmail.com (Ittay Dror) Date: Sat, 06 Sep 2008 20:18:47 +0300 Subject: [Rake-devel] make like variables? In-Reply-To: <941C77FE-8C38-4106-9833-7B1529B04E33@gmail.com> References: <48BFD6F7.6080602@gmail.com> <2EB2CFB0-9C23-48AA-BBB3-22FE51CCF00B@gmail.com> <48C22F72.7090601@gmail.com> <941C77FE-8C38-4106-9833-7B1529B04E33@gmail.com> Message-ID: <48C2BB77.60304@gmail.com> Jim Weirich wrote: > > On Sep 6, 2008, at 2:21 AM, Ittay Dror wrote: > >> in make, if in my Makefile I have >> FOO = 1 >> FOO += 2 >> test: >> echo $(FOO) >> >> then running: >> > make >> 1 2 >> > make FOO=3 >> 3 >> >> That is, if passed in the command line, any assignments to FOO are >> ignored. In Makefile, this can be changed by using the 'override' >> method. It is not that important, just an idiom that people coming >> from Make expect. Obviously, Rakefile being Ruby this sort of >> behavior can be done. > > > I think if I had to do this, I would use an idiom like: > > ENV['FOO'] ||= 1 yes, but then ENV['FOO'] += 1 will increase FOO by one, which is not the Make behavior. > > I really don't have control over how ruby variables are assigned. I'm > not sure how to do it in rake for any arbitrary variable. > I didn't mean to do it for arbitrary variables, instead, accessing variables through an object that controls the behavior. I don't think this in itself (mimicking the exact behavior of Make) is worth the hassle, but providing a nicer interface to dealing with command line variables might be, for example, if the object controlling variables also knows how to handle 'FOO+=a' and 'FOO-=b' in the command line, it could be a nice feature, mainly for C/C++ builds. If you like, since I need such behavior anyway, I can post it when it is ready. Ittay -- -- Ittay Dror From ittay.dror at gmail.com Sun Sep 7 01:20:45 2008 From: ittay.dror at gmail.com (Ittay Dror) Date: Sun, 07 Sep 2008 08:20:45 +0300 Subject: [Rake-devel] make like variables? In-Reply-To: <48C2BB77.60304@gmail.com> References: <48BFD6F7.6080602@gmail.com> <2EB2CFB0-9C23-48AA-BBB3-22FE51CCF00B@gmail.com> <48C22F72.7090601@gmail.com> <941C77FE-8C38-4106-9833-7B1529B04E33@gmail.com> <48C2BB77.60304@gmail.com> Message-ID: <48C364AD.1010803@gmail.com> Ittay Dror wrote: > > > Jim Weirich wrote: >> >> On Sep 6, 2008, at 2:21 AM, Ittay Dror wrote: >> >>> in make, if in my Makefile I have >>> FOO = 1 >>> FOO += 2 >>> test: >>> echo $(FOO) >>> >>> then running: >>> > make >>> 1 2 >>> > make FOO=3 >>> 3 >>> >>> That is, if passed in the command line, any assignments to FOO are >>> ignored. In Makefile, this can be changed by using the 'override' >>> method. It is not that important, just an idiom that people coming >>> from Make expect. Obviously, Rakefile being Ruby this sort of >>> behavior can be done. >> >> >> I think if I had to do this, I would use an idiom like: >> >> ENV['FOO'] ||= 1 > yes, but then ENV['FOO'] += 1 will increase FOO by one, which is not > the Make behavior. >> >> I really don't have control over how ruby variables are assigned. >> I'm not sure how to do it in rake for any arbitrary variable. >> > I didn't mean to do it for arbitrary variables, instead, accessing > variables through an object that controls the behavior. > > I don't think this in itself (mimicking the exact behavior of Make) is > worth the hassle, but providing a nicer interface to dealing with > command line variables might be, for example, if the object > controlling variables also knows how to handle 'FOO+=a' and 'FOO-=b' > in the command line, it could be a nice feature, mainly for C/C++ builds. unfortunately, this cannot be done currently, since rake uses the Regexp /^(\w+)=(.*)$/ to match command line variables, which doesn't apply to things like F+++=a or FOO-=b. Instead, they are interpreted as task names. This is also the case for 'foo.bar=val' (dotted variables are common in java) In general, I think it is better to use the regexp /^(.+)=(.*)$/, restricting task names to not use '=' and allowing more characters in variable names. Ittay > > If you like, since I need such behavior anyway, I can post it when it > is ready. > > Ittay > -- -- Ittay Dror From ittay.dror at gmail.com Wed Sep 10 03:09:46 2008 From: ittay.dror at gmail.com (Ittay Dror) Date: Wed, 10 Sep 2008 10:09:46 +0300 Subject: [Rake-devel] core dumps due to deep task dependencies Message-ID: <48C772BA.5080203@gmail.com> Hi, I'm getting core dumps due to stack exhaustion as a result of deep task dependencies and the recursion they create. Finding out what chain of dependencies caused the dump is hard since ruby crashes (so no nice stack trace), and due to the nature of tasks, once files have been created in the previous run the recursion stops (and deleting everything to recreate means waiting a long time for the crash to happen). Also, even after finding the reason for the crash, it is hard for me to redesign my flow so that the dependencies are not deep. Any tips? Thank you, Ittay -- -- Ittay Dror From jim.weirich at gmail.com Wed Sep 10 11:45:18 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Wed, 10 Sep 2008 11:45:18 -0400 Subject: [Rake-devel] Rake 0.8.2 released Message-ID: <2272AAA4-B634-4FF9-8E88-78C57BDDB543@gmail.com> I release 0.8.2 last night. A big thanks to everyone who contributed. -- -- Jim Weirich -- jim.weirich at gmail.com From jim.weirich at gmail.com Thu Sep 11 00:44:46 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Thu, 11 Sep 2008 00:44:46 -0400 Subject: [Rake-devel] Parallel tasks with Rake In-Reply-To: <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> References: <996afcbc0804131259k1962024du3dd08c81bb8e1c80@mail.gmail.com> <996afcbc0806231758t588c8d55l8b9bcfac3e73129f@mail.gmail.com> <996afcbc0806250643k27e2241bv8a3b62aca3cc6fd1@mail.gmail.com> <1B6AB87C-04F6-418F-A3CC-1EF045C61C87@gmail.com> <996afcbc0806262232l7d2d35d6h1d6643bb6f9df07f@mail.gmail.com> <719FA606-2B94-4BE9-8FBD-D925CD687D4F@gmail.com> <996afcbc0807031638m308291fei363a7e42ff4b0873@mail.gmail.com> <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> Message-ID: On Sep 4, 2008, at 2:33 AM, James M. Lawrence wrote: Hi James, Saw your announcement on ruby-talk and want to say good job on getting drake out. Now that 0.8.2 is released, I've taken some time to look at some of what you've done. It looks impressive. BTW, I'm copying this email to the Rake Development list. I think it would be great to get this conversation on the list for everyone to participate. You can sign up at http://rubyforge.org/mailman/listinfo/rake-devel I've pulled the changes into a branch of rake and have begun playing with it. I've not delved into the deeps of CompTree yet, but the changes to rake look pretty straight forward. And of course, I have some questions: (1) Are you using Ruby threads or processes for the parallelism? (2) We should think about the sematics of the the command "rake -j2 a b" Are "a" and "b" executed in parallel or sequentially. It looks like the code base goes with sequentially, and I think this is the right choice. But it may be worth a discussion. (3) I see a lot of the files are marked "GENERATED -- DO NOT EDIT". Generated from what? Will I be able to regenerate them if they need changing? Would it be better to just use CompTree as a gem? (4) As far as I can tell, when running with num_threads > 1, you invoke all the tasks and gather the task arguments. Then you pass the task dependency graph off to the CompTree code to execute the code in parallel. So all the code execution actually happens after ALL the invokes are done on the code, rather than intermingled in standard rake. Is my understanding correct. (if so, very interesting ... I'm thinking that if it wasn't for the need for the task arguments, you could skip the invoke step and pass the dependency graph immediately to your CompTree package, yes?) (5) I see there is a synchronization lock in the invoke method. Since this part of the code is executed by a single task (the main task), I'm not sure I see the need for a lock. Am I missing something? (6) Have you tried running any of this under Ruby 1.9? That's all for now. Again, thanks for the work you put into this. I'll probably have more questions later. -- -- Jim Weirich -- jim.weirich at gmail.com From ittay.dror at gmail.com Thu Sep 11 00:55:19 2008 From: ittay.dror at gmail.com (Ittay Dror) Date: Thu, 11 Sep 2008 07:55:19 +0300 Subject: [Rake-devel] Parallel tasks with Rake In-Reply-To: References: <996afcbc0804131259k1962024du3dd08c81bb8e1c80@mail.gmail.com> <996afcbc0806231758t588c8d55l8b9bcfac3e73129f@mail.gmail.com> <996afcbc0806250643k27e2241bv8a3b62aca3cc6fd1@mail.gmail.com> <1B6AB87C-04F6-418F-A3CC-1EF045C61C87@gmail.com> <996afcbc0806262232l7d2d35d6h1d6643bb6f9df07f@mail.gmail.com> <719FA606-2B94-4BE9-8FBD-D925CD687D4F@gmail.com> <996afcbc0807031638m308291fei363a7e42ff4b0873@mail.gmail.com> <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> Message-ID: <48C8A4B7.3010406@gmail.com> Jim Weirich wrote: > On Sep 4, 2008, at 2:33 AM, James M. Lawrence wrote: > > Hi James, > > Saw your announcement on ruby-talk and want to say good job on getting > drake out. Now that 0.8.2 is released, I've taken some time to look > at some of what you've done. It looks impressive. > > BTW, I'm copying this email to the Rake Development list. I think it > would be great to get this conversation on the list for everyone to > participate. You can sign up at > http://rubyforge.org/mailman/listinfo/rake-devel > > I've pulled the changes into a branch of rake and have begun playing > with it. I've not delved into the deeps of CompTree yet, but the > changes to rake look pretty straight forward. > > And of course, I have some questions: > > (1) Are you using Ruby threads or processes for the parallelism? > > (2) We should think about the sematics of the the command "rake -j2 a > b" Are "a" and "b" executed in parallel or sequentially. It looks > like the code base goes with sequentially, and I think this is the > right choice. But it may be worth a discussion. For me it is more intuitive to expect a and b to execute in parallel (unless one depends on the other) > > (3) I see a lot of the files are marked "GENERATED -- DO NOT EDIT". > Generated from what? Will I be able to regenerate them if they need > changing? Would it be better to just use CompTree as a gem? > > (4) As far as I can tell, when running with num_threads > 1, you > invoke all the tasks and gather the task arguments. Then you pass the > task dependency graph off to the CompTree code to execute the code in > parallel. So all the code execution actually happens after ALL the > invokes are done on the code, rather than intermingled in standard > rake. Is my understanding correct. (if so, very interesting ... I'm > thinking that if it wasn't for the need for the task arguments, you > could skip the invoke step and pass the dependency graph immediately > to your CompTree package, yes?) > > (5) I see there is a synchronization lock in the invoke method. Since > this part of the code is executed by a single task (the main task), > I'm not sure I see the need for a lock. Am I missing something? Why the need to do this inside 'invoke'? Isn't it enough to use this code on the top level tasks only? (invoking them along the way) Ittay > > (6) Have you tried running any of this under Ruby 1.9? > > That's all for now. Again, thanks for the work you put into this. > I'll probably have more questions later. > -- -- Ittay Dror From jim.weirich at gmail.com Thu Sep 11 01:57:24 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Thu, 11 Sep 2008 01:57:24 -0400 Subject: [Rake-devel] Parallel tasks with Rake In-Reply-To: <48C8A4B7.3010406@gmail.com> References: <996afcbc0804131259k1962024du3dd08c81bb8e1c80@mail.gmail.com> <996afcbc0806231758t588c8d55l8b9bcfac3e73129f@mail.gmail.com> <996afcbc0806250643k27e2241bv8a3b62aca3cc6fd1@mail.gmail.com> <1B6AB87C-04F6-418F-A3CC-1EF045C61C87@gmail.com> <996afcbc0806262232l7d2d35d6h1d6643bb6f9df07f@mail.gmail.com> <719FA606-2B94-4BE9-8FBD-D925CD687D4F@gmail.com> <996afcbc0807031638m308291fei363a7e42ff4b0873@mail.gmail.com> <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> <48C8A4B7.3010406@gmail.com> Message-ID: On Sep 11, 2008, at 12:55 AM, Ittay Dror wrote: > For me it is more intuitive to expect a and b to execute in parallel > (unless one depends on the other) If one depends on the other, there is no need to express both on the command line. I often do things like: rake rdoc publish to rebuild the rdoc html and then publish it to the web. I've seen rails taks that do things like: rake test_env migrate Where test_env sets up the environment for the following task. This is why I think sequential is the proper choice. -- -- Jim Weirich -- jim.weirich at gmail.com From quixoticsycophant at gmail.com Thu Sep 11 03:59:08 2008 From: quixoticsycophant at gmail.com (James M. Lawrence) Date: Thu, 11 Sep 2008 03:59:08 -0400 Subject: [Rake-devel] Parallel tasks with Rake In-Reply-To: References: <996afcbc0804131259k1962024du3dd08c81bb8e1c80@mail.gmail.com> <996afcbc0806231758t588c8d55l8b9bcfac3e73129f@mail.gmail.com> <996afcbc0806250643k27e2241bv8a3b62aca3cc6fd1@mail.gmail.com> <1B6AB87C-04F6-418F-A3CC-1EF045C61C87@gmail.com> <996afcbc0806262232l7d2d35d6h1d6643bb6f9df07f@mail.gmail.com> <719FA606-2B94-4BE9-8FBD-D925CD687D4F@gmail.com> <996afcbc0807031638m308291fei363a7e42ff4b0873@mail.gmail.com> <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> Message-ID: <996afcbc0809110059p38f03cb6nf75a4248d217d54@mail.gmail.com> On Thu, Sep 11, 2008 at 12:44 AM, Jim Weirich wrote: > On Sep 4, 2008, at 2:33 AM, James M. Lawrence wrote: > > Hi James, > > Saw your announcement on ruby-talk and want to say good job on getting drake > out. Now that 0.8.2 is released, I've taken some time to look at some of > what you've done. It looks impressive. Thanks. > (1) Are you using Ruby threads or processes for the parallelism? Threads. CompTree (http://comptree.rubyforge.org) has an option to fork nodes, but I haven't enabled it for Drake. Since I expect -j to be commonly used for compiling, forking would be redundant anyway. Yet it's ready to go with the options [-k, --fork] in rake.rb, just commented out. Especially for the first release I didn't see a compelling need to go there yet. A single option -j felt nicer. > (2) We should think about the sematics of the the command "rake -j2 a b" > Are "a" and "b" executed in parallel or sequentially. It looks like the > code base goes with sequentially, and I think this is the right choice. But > it may be worth a discussion. Yes, I intentionally decided not to put a and b under the same parent node. On the command line we especially think sequentially. > (3) I see a lot of the files are marked "GENERATED -- DO NOT EDIT". > Generated from what? Will I be able to regenerate them if they need > changing? Would it be better to just use CompTree as a gem? The master files are under contrib/comp_tree. rake pull_contrib gets the latest comp_tree from github then repackages it under the Rake::CompTree in lib/rake/comp_tree. I was never happy with this, btw. I wanted to Rake to contain a fork of CompTree while at the same time not conflicting with a regular installation of CompTree. I experimented with dynamically evaling it into Rake::CompTree, but my method failed to act transitively to the sub-sub module Quix (utilities). The other reason was that I thought having external dependencies might give you pause about merging into the mainline. But if you are OK with the dependency, we could use the gem. Also, at the time I thought the CompTree API might change enough to cause hassles being an external dependency. But since versioning seems to work OK in rubygems, it's kind of a moot point. And CompTree may be stable enough after all. > (4) As far as I can tell, when running with num_threads > 1, you invoke all > the tasks and gather the task arguments. Then you pass the task dependency > graph off to the CompTree code to execute the code in parallel. So all the > code execution actually happens after ALL the invokes are done on the code, > rather than intermingled in standard rake. Is my understanding correct. > (if so, very interesting ... I'm thinking that if it wasn't for the need > for the task arguments, you could skip the invoke step and pass the > dependency graph immediately to your CompTree package, yes?) Yes, excellent detective work. I wrested with getting the task arguments right on my own and finally gave up. Actually I still don't understand them -- they seemed to be context-dependent. It was amazing -- one unit test would fail and the other would succeed. I make a change, and now it's swapped! The former succeeds and the latter fails. > (5) I see there is a synchronization lock in the invoke method. Since this > part of the code is executed by a single task (the main task), I'm not sure > I see the need for a lock. Am I missing something? Calling invoke inside invoke seemed to be a problem, both practically and theoretically. Practically, the computation tree has already been built, but now someone wants to build a new one with possibly overlapping nodes. On the theoretical side, I said in the readme: Parallelizing code means surrendering control over the micro-management of its execution. Manually invoking tasks inside other tasks is rather contrary to this notion, throwing a monkey wrench into the system. It seems the parallelizer cannot make good decisions if the user is allowed to rearrange the furniture on a whim. But I leave the door open here -- I haven't fully considered invoke inside invoke. > (6) Have you tried running any of this under Ruby 1.9? The CompTree unit tests will core dump 1.9. But those unit tests pound extremely hard on the system, running many threads with many forks on large trees. It crashes cygwin too. On darwin I have to catch EAGAIN signals, but the tests all succeed. I designed it this way, of course, to see if I could shake out any race conditions or whatever multi-threaded problems that might exist. CompTree might be OK in 1.9 for a sunday drive. I haven't tried it with small Rakefiles cases yet. > That's all for now. Again, thanks for the work you put into this. I'll > probably have more questions later. And thanks for Rake. It is a pleasure to be involved. James M. Lawrence From quixoticsycophant at gmail.com Thu Sep 11 14:49:18 2008 From: quixoticsycophant at gmail.com (James M. Lawrence) Date: Thu, 11 Sep 2008 14:49:18 -0400 Subject: [Rake-devel] Parallel tasks with Rake In-Reply-To: References: <996afcbc0804131259k1962024du3dd08c81bb8e1c80@mail.gmail.com> <996afcbc0806231758t588c8d55l8b9bcfac3e73129f@mail.gmail.com> <996afcbc0806250643k27e2241bv8a3b62aca3cc6fd1@mail.gmail.com> <1B6AB87C-04F6-418F-A3CC-1EF045C61C87@gmail.com> <996afcbc0806262232l7d2d35d6h1d6643bb6f9df07f@mail.gmail.com> <719FA606-2B94-4BE9-8FBD-D925CD687D4F@gmail.com> <996afcbc0807031638m308291fei363a7e42ff4b0873@mail.gmail.com> <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> Message-ID: <996afcbc0809111149w193d5e23i52a42300673c7e32@mail.gmail.com> Though it would be a privilege to see my code merged into Rake, of course a case would have to be made for it. What are the problems it solves? I am not a good person to make the case, as I have no experience with non-trivial use of 'multitask'. But fortunately others here do. After all, Drake may be a totally misguided project. That would be OK with me, as my primary interest was in CompTree. The implementation of Drake is trivial, given CompTree. Here are Ittay's points from a previous thread, > 1. If some top level tasks run in parallel, and each of them > recursively runs other tasks, and one of the bottom tasks fail, it > is impossible to stop the other tasks, short of a very ugly abort of > all threads. > 2. Tasks that run in parallel can't tell when another task's execution > has failed. They may read a wrong timestamp from the failed task. CompTree uses Thread.abort_on_exception = true. If something goes wrong, why shouldn't we abort all threads? I don't yet see the issue here. That said, CompTree does give us an option. A Rake task is translated into a computation node which discards its result. But we don't have to discard the result. We could wrap the task inside begin/rescue, returning the Exception instance as the result. CompTree 'computes' the exception. # top-level result = driver.compute(root_node, :num_threads => n) if result and result.is_a? Exception raise result end > 3. Threads are created per prerequisite task, rather than a fixed number > (based initially on the number of cores/cpus), which causes thrashing CompTree uses a fixed number of threads. > 4. Even if a thread pool will be utilized, dependency information is > still hard to take into account. Imagine a task has 2 prerequisites, > where one depends on the other. Adding the tasks into a thread pool, > they may be invoked in two different threads, but one waits on the > other so the thread is not utilized. Maybe add a "distance" method > which calculates how far one task if from the other in dependency > (adding nil if not dependent), so when adding tasks to a queue, they > are added to the queue where the current tasks have the minimal > distance (nil being infinite). In CompTree, whenever a node finishes its computation, the tree is scanned for nodes waiting to be computed. Available nodes are handed out to the available threads. Those threads which didn't get a node are put to sleep. No soup for you, and go to bed. Repeat. Thus CompTree operates at "max capacity" at all times. Given N threads, if at any time N computations are not running, it is because the graph topology demands it (children node results are not available). In short, I believe it does what you want. As you can see, I can't help myself from using graph terminology, as my contact with Rake has been rather superficial. A quick rosetta stone: node <--> task child <--> prerequisite parent <--> ?? what do you call this ?? compute <--> invoke function <--> @actions.each { |act| act.call } function.call <--> execute result <--> N/A node.name <--> task.name.to_sym It is not clear whether these issues would be solved better within Rake itself, or whether CompTree should be used to solve them. What are the other issues, and how would you solve them? James M. Lawrence From quixoticsycophant at gmail.com Thu Sep 11 14:54:33 2008 From: quixoticsycophant at gmail.com (James M. Lawrence) Date: Thu, 11 Sep 2008 14:54:33 -0400 Subject: [Rake-devel] Parallel tasks with Rake In-Reply-To: <996afcbc0809111149w193d5e23i52a42300673c7e32@mail.gmail.com> References: <996afcbc0804131259k1962024du3dd08c81bb8e1c80@mail.gmail.com> <996afcbc0806231758t588c8d55l8b9bcfac3e73129f@mail.gmail.com> <996afcbc0806250643k27e2241bv8a3b62aca3cc6fd1@mail.gmail.com> <1B6AB87C-04F6-418F-A3CC-1EF045C61C87@gmail.com> <996afcbc0806262232l7d2d35d6h1d6643bb6f9df07f@mail.gmail.com> <719FA606-2B94-4BE9-8FBD-D925CD687D4F@gmail.com> <996afcbc0807031638m308291fei363a7e42ff4b0873@mail.gmail.com> <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> <996afcbc0809111149w193d5e23i52a42300673c7e32@mail.gmail.com> Message-ID: <996afcbc0809111154t3ec071b1jb2c7f6e2640d43fb@mail.gmail.com> > node <--> task > child <--> prerequisite > parent <--> ?? what do you call this ?? > compute <--> invoke > function <--> @actions.each { |act| act.call } > function.call <--> execute > result <--> N/A > node.name <--> task.name.to_sym I forgot to mention: I just noticed that I could have mapped a node to each of @actions, which would further parallelize Rake while confusing users who were depending their sequential execution. James M. Lawrence From ittay.dror at gmail.com Thu Sep 11 15:33:55 2008 From: ittay.dror at gmail.com (Ittay Dror) Date: Thu, 11 Sep 2008 22:33:55 +0300 Subject: [Rake-devel] Parallel tasks with Rake In-Reply-To: <996afcbc0809111149w193d5e23i52a42300673c7e32@mail.gmail.com> References: <996afcbc0804131259k1962024du3dd08c81bb8e1c80@mail.gmail.com> <996afcbc0806231758t588c8d55l8b9bcfac3e73129f@mail.gmail.com> <996afcbc0806250643k27e2241bv8a3b62aca3cc6fd1@mail.gmail.com> <1B6AB87C-04F6-418F-A3CC-1EF045C61C87@gmail.com> <996afcbc0806262232l7d2d35d6h1d6643bb6f9df07f@mail.gmail.com> <719FA606-2B94-4BE9-8FBD-D925CD687D4F@gmail.com> <996afcbc0807031638m308291fei363a7e42ff4b0873@mail.gmail.com> <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> <996afcbc0809111149w193d5e23i52a42300673c7e32@mail.gmail.com> Message-ID: <48C972A3.7000703@gmail.com> James M. Lawrence wrote: > Though it would be a privilege to see my code merged into Rake, of > course a case would have to be made for it. What are the problems it > solves? I am not a good person to make the case, as I have no > experience with non-trivial use of 'multitask'. But fortunately > others here do. > 1. when compiling c/c++ code, usually a compiler is launched per source file. each runs separately from the other, so they can be parallelized. as c/c++ compilations are usually slower than java (for example), and since multi-core servers are common these days, this is an important improvement. 2. multi-module builds can benefit in the same way, so a project with 2 java modules can run their build in parallel. 3. it makes rake more compatible with make which helps non-ruby people (c/c++ peolple) choose it. 4. multitask does not do a good job for two reasons: a. it cannot be controlled from outside (no equivalent to -j), so on 1/2/4 cpus you get the same amount of threads (depending on the size of prerequisites) b. it runs thread for each prerequisites. so it can thrash the system, running for example 30 threads on 2 cpus (again, the main use case here being c/c++ with 30 source files) 5. alternatives that give better control, but are still implementations of task are wrong since you eventually loose control of the number of threads. i have a JobTask implementation that can be configured to tell it how many jobs to run. now imagine i use it to compile modules in parallel, and i have 2 top modules each with 2 sub modules, i end up with 3 job tasks (top and 2 sub) each running 2 threads, so 6 threads, which again brings us to thrashing the system. it is hard to overall control the existence of only x threads. (currently i use my job task for only the "leaf" prerequisites). 6. both multitask and jobtask suffer from the fact that the threads are independent. usually, if there's a compilation error, i want to stop the build and signal the error to the user. but when there are several independent threads, it is hard to know them all (think several multi-task/job-task) so as to stop them (cleanly) when one fails - btw, this is possible with drake, right? so if one thread fails the other is signaled to quit execution when the current execution block finishes (cleanly) 7. current rake relies on recursion to execute tasks. for a large build this may create a deep recursion stack that exhausts ruby's execution stack (for 20 deep execution, the ruby stack was ~800 which caused segfaults in linux even when the stack size was unlimited) Ittay > After all, Drake may be a totally misguided project. That would be OK > with me, as my primary interest was in CompTree. The implementation > of Drake is trivial, given CompTree. > > Here are Ittay's points from a previous thread, > > >> 1. If some top level tasks run in parallel, and each of them >> recursively runs other tasks, and one of the bottom tasks fail, it >> is impossible to stop the other tasks, short of a very ugly abort of >> all threads. >> 2. Tasks that run in parallel can't tell when another task's execution >> has failed. They may read a wrong timestamp from the failed task. >> > > CompTree uses Thread.abort_on_exception = true. If something goes > wrong, why shouldn't we abort all threads? I don't yet see the issue > here. > > That said, CompTree does give us an option. A Rake task is translated > into a computation node which discards its result. But we don't have > to discard the result. We could wrap the task inside begin/rescue, > returning the Exception instance as the result. CompTree 'computes' > the exception. > > # top-level > result = driver.compute(root_node, :num_threads => n) > if result and result.is_a? Exception > raise result > end > > >> 3. Threads are created per prerequisite task, rather than a fixed number >> (based initially on the number of cores/cpus), which causes thrashing >> > > CompTree uses a fixed number of threads. > > >> 4. Even if a thread pool will be utilized, dependency information is >> still hard to take into account. Imagine a task has 2 prerequisites, >> where one depends on the other. Adding the tasks into a thread pool, >> they may be invoked in two different threads, but one waits on the >> other so the thread is not utilized. Maybe add a "distance" method >> which calculates how far one task if from the other in dependency >> (adding nil if not dependent), so when adding tasks to a queue, they >> are added to the queue where the current tasks have the minimal >> distance (nil being infinite). >> > > In CompTree, whenever a node finishes its computation, the tree is > scanned for nodes waiting to be computed. Available nodes are handed > out to the available threads. Those threads which didn't get a node > are put to sleep. No soup for you, and go to bed. Repeat. > > Thus CompTree operates at "max capacity" at all times. Given N > threads, if at any time N computations are not running, it is because > the graph topology demands it (children node results are not > available). In short, I believe it does what you want. > > As you can see, I can't help myself from using graph terminology, as > my contact with Rake has been rather superficial. A quick rosetta > stone: > > node <--> task > child <--> prerequisite > parent <--> ?? what do you call this ?? > compute <--> invoke > function <--> @actions.each { |act| act.call } > function.call <--> execute > result <--> N/A > node.name <--> task.name.to_sym > > It is not clear whether these issues would be solved better within > Rake itself, or whether CompTree should be used to solve them. What > are the other issues, and how would you solve them? > > James M. Lawrence > _______________________________________________ > Rake-devel mailing list > Rake-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/rake-devel > -- -- Ittay Dror -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at pragprog.com Mon Sep 15 14:30:09 2008 From: dave at pragprog.com (Dave Thomas) Date: Mon, 15 Sep 2008 12:30:09 -0600 Subject: [Rake-devel] The 'ruby' help on windows Message-ID: One of our authors has Ruby under Windows installed in c:\Program Files \ruby In our rake tasks, we have ruby %{"#{BIBEXTRACT}" "#{t.source}" >"#{t.name}"}, :verbose => true When they run this, they see the tracing: c:/Program Files/ruby/bin/ruby "C:/Development/workspace/EBAND/PPStuff/ util/bin/bibextract.rb" "book.xml" >"book.cited-xml" 'c:/Program' is not recognized as an internal or external command, The space in the 'Program Files' directory name is being taken as a separator, and so the command doesn't execute. Is there a simple workaround for this, or do I need o tell the author to reinstall Ruby somewhere with no spaces in the path? Cheers Dave From luislavena at gmail.com Mon Sep 15 14:37:34 2008 From: luislavena at gmail.com (Luis Lavena) Date: Mon, 15 Sep 2008 15:37:34 -0300 Subject: [Rake-devel] The 'ruby' help on windows In-Reply-To: References: Message-ID: <71166b3b0809151137w12c7f473u6a92072a1dd196cc@mail.gmail.com> On Mon, Sep 15, 2008 at 3:30 PM, Dave Thomas wrote: > One of our authors has Ruby under Windows installed in c:\Program Files\ruby > > In our rake tasks, we have > > ruby %{"#{BIBEXTRACT}" "#{t.source}" >"#{t.name}"}, :verbose => true > > When they run this, they see the tracing: > > c:/Program Files/ruby/bin/ruby > "C:/Development/workspace/EBAND/PPStuff/util/bin/bibextract.rb" "book.xml" >>"book.cited-xml" > 'c:/Program' is not recognized as an internal or external command, > > The space in the 'Program Files' directory name is being taken as a > separator, and so the command doesn't execute. > > Is there a simple workaround for this, or do I need o tell the author to > reinstall Ruby somewhere with no spaces in the path? > Hello Dave, A similar issue to this one in RubyGems was reported to Ruby-core and FileUtils was fixed. Since rake is extending FileUtils and adding the RUBY constant, it needs to be updated. in the similar way. This is the solution I've implemented in RubyGems, based on Nobu's commit to the ruby svn: # escape string in case path to ruby executable contain spaces. @ruby.sub!(/.*\s.*/m, '"\&"') Regards, -- Luis Lavena AREA 17 - Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so. Douglas Adams From jim.weirich at gmail.com Mon Sep 15 14:44:18 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Mon, 15 Sep 2008 14:44:18 -0400 Subject: [Rake-devel] The 'ruby' help on windows In-Reply-To: <71166b3b0809151137w12c7f473u6a92072a1dd196cc@mail.gmail.com> References: <71166b3b0809151137w12c7f473u6a92072a1dd196cc@mail.gmail.com> Message-ID: <5D5A12FC-23A9-4EED-8355-95E87213CAD5@gmail.com> On Sep 15, 2008, at 2:37 PM, Luis Lavena wrote: > This is the solution I've implemented in RubyGems, based on Nobu's > commit to the ruby svn: > > # escape string in case path to ruby executable contain spaces. > @ruby.sub!(/.*\s.*/m, '"\&"') I created a bug report for this: http://onestepback.org/redmine/issues/show/20 -- -- Jim Weirich -- jim.weirich at gmail.com From quixoticsycophant at gmail.com Mon Sep 15 22:19:58 2008 From: quixoticsycophant at gmail.com (James M. Lawrence) Date: Mon, 15 Sep 2008 22:19:58 -0400 Subject: [Rake-devel] Parallel tasks with Rake In-Reply-To: <48C972A3.7000703@gmail.com> References: <996afcbc0804131259k1962024du3dd08c81bb8e1c80@mail.gmail.com> <996afcbc0806250643k27e2241bv8a3b62aca3cc6fd1@mail.gmail.com> <1B6AB87C-04F6-418F-A3CC-1EF045C61C87@gmail.com> <996afcbc0806262232l7d2d35d6h1d6643bb6f9df07f@mail.gmail.com> <719FA606-2B94-4BE9-8FBD-D925CD687D4F@gmail.com> <996afcbc0807031638m308291fei363a7e42ff4b0873@mail.gmail.com> <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> <996afcbc0809111149w193d5e23i52a42300673c7e32@mail.gmail.com> <48C972A3.7000703@gmail.com> Message-ID: <996afcbc0809151919w629a3b19v3428e9c909a7cf1a@mail.gmail.com> A few updates -- * The crashes I described previously with CompTree + 1.9 were all from the forking tests. Drb + threads + fork is a very delicate matter, and there are likely some caveats for 1.9 and cygwin. Fortunately this has nothing to do with Drake, as I don't use CompTree's fork option. So Drake runs well under 1.9. * I removed the generated rb files. It's better to deal with an occasional merge conflict from the CompTree --> Rake::CompTree rename. For reasons given below, I think my initial inclination for using a Rake-ified branch of CompTree was right after all (as opposed to the gem). * Fixed two curious bugs missed by the Rake unit tests: (a) I did not notice @prerequisites contained namespace-unqualifed names. (a) FileTask#needed? only looks at the immediate children. In Drake this caused grandparents and above to not be executed (since all tasks are gathered with a dry run). If 'needed?' is officially supported for end-users, this might be considered a bug in Rake. Otherwise there is no problem, of course, since Task#invoke propagates 'needed?' via recursion. * Added a --rand[=SEED] flag to randomize prerequisites, a debugging tool for reaching j-safety. SEED can be any string (String#hash is the real seed). Ittay: > - btw, this is possible with drake, right? so if one thread fails > the other is signaled to quit execution when the current execution block > finishes (cleanly) It is possible, but CompTree doesn't work that way by default. Since in CompTree the purpose is to get an answer, there would be no reason to finish any node's computation once an error is detected. This is a fundamental difference between the goals of Rake (create side-effects) and CompTree (compute a value), a good justification for keeping the Rake::CompTree branch. I think I would prefer not having to support two antithetical "modes" at once (I already don't like TaskNode in there). Plus I get to remove drb in the Rake version, which adds a measurable load time due to socket et al. James M. Lawrence From jim.weirich at gmail.com Tue Sep 16 09:33:12 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Tue, 16 Sep 2008 09:33:12 -0400 Subject: [Rake-devel] Parallel tasks with Rake In-Reply-To: <996afcbc0809151919w629a3b19v3428e9c909a7cf1a@mail.gmail.com> References: <996afcbc0804131259k1962024du3dd08c81bb8e1c80@mail.gmail.com> <996afcbc0806250643k27e2241bv8a3b62aca3cc6fd1@mail.gmail.com> <1B6AB87C-04F6-418F-A3CC-1EF045C61C87@gmail.com> <996afcbc0806262232l7d2d35d6h1d6643bb6f9df07f@mail.gmail.com> <719FA606-2B94-4BE9-8FBD-D925CD687D4F@gmail.com> <996afcbc0807031638m308291fei363a7e42ff4b0873@mail.gmail.com> <996afcbc0809032333g456e1f87yf75b44b962ada63e@mail.gmail.com> <996afcbc0809111149w193d5e23i52a42300673c7e32@mail.gmail.com> <48C972A3.7000703@gmail.com> <996afcbc0809151919w629a3b19v3428e9c909a7cf1a@mail.gmail.com> Message-ID: <526192A1-0FEB-4B31-8F92-75E47108FA36@gmail.com> On Sep 15, 2008, at 10:19 PM, James M. Lawrence wrote: > A few updates -- Thanks for the updates. It may be a while before I can respond in depth. Power is still out in major portions of Cincinnati so I'm relying on my laptop battery and EVDO card at the moment. -- -- Jim Weirich -- jim.weirich at gmail.com From lacton at users.sourceforge.net Fri Sep 19 15:46:39 2008 From: lacton at users.sourceforge.net (lacton) Date: Fri, 19 Sep 2008 21:46:39 +0200 Subject: [Rake-devel] have_rakefile() in 0.8.2 Message-ID: At http://rake.rubyforge.org/classes/Rake/Application.html#M000081 I read "have_rakefile() True if one of the files in RAKEFILES is in the current directory. If a match is found, it is copied into @rakefile." Yet, the attached source does not mention the @rakefile attribute. # File lib/rake.rb, line 2072 def have_rakefile @rakefiles.each do |fn| if File.exist?(fn) || fn == '' return fn end end return nil end Is the method broken or is it the documentation? Regards. Lacton From smparkes at smparkes.net Mon Sep 22 19:27:48 2008 From: smparkes at smparkes.net (Steven Parkes) Date: Mon, 22 Sep 2008 16:27:48 -0700 Subject: [Rake-devel] :needs example Message-ID: The rakefile.rdoc says that when passing prereqs to a task with params you have to use a :needs key, but the example doesn't seem to actually have it. I believe task :name, [:first_name, :last_name] => [:pre_name] do |t, args| should be task :name, [:first_name, :last_name], :needs => [:pre_name] do |t, args| ? From jim.weirich at gmail.com Mon Sep 22 21:36:05 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Mon, 22 Sep 2008 21:36:05 -0400 Subject: [Rake-devel] :needs example In-Reply-To: References: Message-ID: On Sep 22, 2008, at 7:27 PM, Steven Parkes wrote: > The rakefile.rdoc says that when passing prereqs to a task with > params you > have to use a :needs key, but the example doesn't seem to actually > have it. > > I believe > task :name, [:first_name, :last_name] => [:pre_name] do |t, args| > should be > task :name, [:first_name, :last_name], :needs => [:pre_name] do | > t, args| Rake was updated to not need the :needs parameter. I must have missed a spot in the documentation. Sorry. -- -- Jim Weirich -- jim.weirich at gmail.com From smparkes at smparkes.net Mon Sep 22 21:58:28 2008 From: smparkes at smparkes.net (Steven Parkes) Date: Mon, 22 Sep 2008 18:58:28 -0700 Subject: [Rake-devel] :needs example In-Reply-To: References: Message-ID: > -----Original Message----- > From: Jim Weirich [mailto:jim.weirich at gmail.com] > Sent: Monday, September 22, 2008 6:36 PM > Rake was updated to not need the :needs parameter. I must > have missed > a spot in the documentation. Sorry. Ah. In that case, it does look like the arguments have to be an array, even if there's a single entry? I had tried it without :needs but I didn't need two params, so I just did task :name, :first_name => :pre_name do |t, args| which fails. The docs only talk about an array, so I should have tried that, but I didn't. [Did this used to be true? I used to have (god knows why ...) Task { :name => :prereq }, :arg do |t,args| which used to work but no longer does ... but perhaps never should have?] Entirely optional, but it wouldn't hurt to mention explicitly that the args has to be an array, even if there's only one element, given that being able to pass single objects unwrapped seems to be a common idiom in Ruby? And, if you're in the docs, The next to argumements are the names ... should be The next two arguments are the names ... no? From jim.weirich at gmail.com Mon Sep 22 23:32:26 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Mon, 22 Sep 2008 23:32:26 -0400 Subject: [Rake-devel] :needs example In-Reply-To: References: Message-ID: <38CA53FE-CFFF-457C-B88E-0C1BE61E6B71@gmail.com> On Sep 22, 2008, at 9:58 PM, Steven Parkes wrote: > Ah. In that case, it does look like the arguments have to be an > array, even > if there's a single entry? Yes. > I had tried it without :needs but I didn't need > two params, so I just did > task :name, :first_name => :pre_name do |t, args| > which fails. > > The docs only talk about an array, so I should have tried that, but I > didn't. > > [Did this used to be true? I used to have (god knows why ...) > Task { :name => :prereq }, :arg do |t,args| > which used to work but no longer does ... but perhaps never should > have?] That was never supported syntax. I'm kinda surprised it worked. > Entirely optional, but it wouldn't hurt to mention explicitly that > the args > has to be an array, even if there's only one element, given that > being able > to pass single objects unwrapped seems to be a common idiom in Ruby? So noted. > And, if you're in the docs, > The next to argumements are the names ... > should be > The next two arguments are the names ... > no? Thanks. -- -- Jim Weirich -- jim.weirich at gmail.com From jay.turpin at intel.com Thu Sep 25 23:08:43 2008 From: jay.turpin at intel.com (Turpin, Jay) Date: Thu, 25 Sep 2008 20:08:43 -0700 Subject: [Rake-devel] Rake::Win32.rake_system Message-ID: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> All, Once we upgraded from rake 0.8.1 to 0.8.3, our builds began to break. I traced the issue down to the tasks which were executing sh commands that had embedded newline characters in them. For instance, we will often construct small SQL queries using Ruby and then execute using a command-line SQL tool: sh "sqlcmd -E -S localhost -Q \"SET NOCOUNT ON EXEC sp_who" This syntax used to work fine, but now there has been a change to use Rake::Win32.rake_system() instead of using the system() # Run a command line on windows. def rake_system(*cmd) if cmd.size == 1 system("call #{cmd}") else system(*cmd) end end I'm not sure why this change was made, but if we can modify it to use this syntax instead, it resolves the issue for us: def rake_system(*cmd) cmd.each do |c| system(c) end end Thanks Regards, Jay Turpin "Wit is educated insolence." - Aristotle -------------- next part -------------- An HTML attachment was scrubbed... URL: From luislavena at gmail.com Thu Sep 25 23:56:09 2008 From: luislavena at gmail.com (Luis Lavena) Date: Fri, 26 Sep 2008 00:56:09 -0300 Subject: [Rake-devel] Rake::Win32.rake_system In-Reply-To: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> References: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> Message-ID: <71166b3b0809252056r5a4d48b0hd3ccf97c1827b755@mail.gmail.com> On Fri, Sep 26, 2008 at 12:08 AM, Turpin, Jay wrote: > All, > > Once we upgraded from rake 0.8.1 to 0.8.3, our builds began to break. I > traced the issue down to the tasks which were executing sh commands that had > embedded newline characters in them. > > ... > > > This syntax used to work fine, but now there has been a change to use > Rake::Win32.rake_system() instead of using the system() > > > > # Run a command line on windows. > > def rake_system(*cmd) > > if cmd.size == 1 > > system("call #{cmd}") > > else > > system(*cmd) > > end > > end > > Will be good to know why this change was introduced. I think this commit introduced the change: http://github.com/jimweirich/rake/commit/9981a2fb9bede211b0f7d00265e7c0e2aa9bc8c9 But I cannot understand the issue behind the shell expansion, since after all system is indeed generating a new, child cmd.exe which inherits parent environment. But maybe is not that the problem, I'm missing something? -- Luis Lavena AREA 17 - Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so. Douglas Adams From jim.weirich at gmail.com Fri Sep 26 00:18:15 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Fri, 26 Sep 2008 00:18:15 -0400 Subject: [Rake-devel] Rake::Win32.rake_system In-Reply-To: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> References: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> Message-ID: <929010EB-0A91-4157-AAC5-AF09560C4880@gmail.com> On Sep 25, 2008, at 11:08 PM, Turpin, Jay wrote: > All, > > Once we upgraded from rake 0.8.1 to 0.8.3, our builds began to > break. I traced the issue down to the tasks which were executing sh > commands that had embedded newline characters in them. > > For instance, we will often construct small SQL queries using Ruby > and then execute using a command-line SQL tool: > > sh "sqlcmd -E -S localhost -Q \"SET NOCOUNT ON > EXEC sp_who" > > This syntax used to work fine, but now there has been a change to > use Rake::Win32.rake_system() instead of using the system() > > # Run a command line on windows. > def rake_system(*cmd) > if cmd.size == 1 > system("call #{cmd}") > else > system(*cmd) > end > end > > I'm not sure why this change was made, but if we can modify it to > use this syntax instead, it resolves the issue for us: The change was made because system commands were not properly interpolating environment variable values. The change was to enable the proper semantics when running on windows. > def rake_system(*cmd) > cmd.each do |c| > system(c) > end > end Does this version pass the existing unit tests? -- -- Jim Weirich -- jim.weirich at gmail.com From jim.weirich at gmail.com Fri Sep 26 00:27:26 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Fri, 26 Sep 2008 00:27:26 -0400 Subject: [Rake-devel] Rake::Win32.rake_system In-Reply-To: <71166b3b0809252056r5a4d48b0hd3ccf97c1827b755@mail.gmail.com> References: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> <71166b3b0809252056r5a4d48b0hd3ccf97c1827b755@mail.gmail.com> Message-ID: <3BE70AD8-B336-4F62-A96A-DB29B4E2B61E@gmail.com> On Sep 25, 2008, at 11:56 PM, Luis Lavena wrote: > Will be good to know why this change was introduced. sh was not expanding shell environment variables (on windows) without this change. If there is a better way to do this, let me know (or better yet, provide a patch). I have very limited ability to test on windows, so if someone wishes to run with this, I would be very grateful. -- -- Jim Weirich -- jim.weirich at gmail.com From jay.turpin at intel.com Fri Sep 26 12:18:02 2008 From: jay.turpin at intel.com (Turpin, Jay) Date: Fri, 26 Sep 2008 09:18:02 -0700 Subject: [Rake-devel] Rake::Win32.rake_system In-Reply-To: <3BE70AD8-B336-4F62-A96A-DB29B4E2B61E@gmail.com> References: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> <71166b3b0809252056r5a4d48b0hd3ccf97c1827b755@mail.gmail.com> <3BE70AD8-B336-4F62-A96A-DB29B4E2B61E@gmail.com> Message-ID: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C261@azsmsx502.amr.corp.intel.com> OK - now I understand why this was done. However, the CALL command is choking on any commands sent to sh that contains a newline character. It's not pretty, but this fix appears to resolve the problem (at least all of the tests pass): # Run a command line on windows. def rake_system(*cmd) perform_variable_expansion cmd[0] system(*cmd) end def perform_variable_expansion(cmd) ENV.each do |k, v| cmd.sub!(/\%#{k}\%/i, v) end end Regards, Jay Turpin "There are four things that hold back human progress. Ignorance, stupidity, committees and accountants." - Charles J.C. Lyall From dsisnero at gmail.com Fri Sep 26 12:35:37 2008 From: dsisnero at gmail.com (Dominic Sisneros) Date: Fri, 26 Sep 2008 10:35:37 -0600 Subject: [Rake-devel] Rake::Win32.rake_system In-Reply-To: <3BE70AD8-B336-4F62-A96A-DB29B4E2B61E@gmail.com> References: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> <71166b3b0809252056r5a4d48b0hd3ccf97c1827b755@mail.gmail.com> <3BE70AD8-B336-4F62-A96A-DB29B4E2B61E@gmail.com> Message-ID: I have a windows box to test on. Which test is exercising this behavior so that I can try some things? sh was not expanding shell environment variables (on windows) without this > change. > > If there is a better way to do this, let me know (or better yet, provide a > patch). I have very limited ability to test on windows, so if someone > wishes to run with this, I would be very grateful. > > -- > -- Jim Weirich > -- jim.weirich at gmail.com > > _______________________________________________ > Rake-devel mailing list > Rake-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/rake-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsisnero at gmail.com Fri Sep 26 12:37:24 2008 From: dsisnero at gmail.com (Dominic Sisneros) Date: Fri, 26 Sep 2008 10:37:24 -0600 Subject: [Rake-devel] win32_system_dir Message-ID: I defined a HOME env variable under my user dir. I would like this the Win32.win32_system_dir to use this ENV var if defined. def test_win32_system_dir_uses_env_HOME_if_defined in_environment('HOME' => 'C:/home') do assert_equal "C:/home", Win32.win32_system_dir end end -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay.turpin at intel.com Fri Sep 26 12:39:07 2008 From: jay.turpin at intel.com (Turpin, Jay) Date: Fri, 26 Sep 2008 09:39:07 -0700 Subject: [Rake-devel] Rake::Win32.rake_system In-Reply-To: References: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> <71166b3b0809252056r5a4d48b0hd3ccf97c1827b755@mail.gmail.com> <3BE70AD8-B336-4F62-A96A-DB29B4E2B61E@gmail.com> Message-ID: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C2B5@azsmsx502.amr.corp.intel.com> test\test_fileutils.rb - take a look at the test_sh* methods - especially, test_sh_multiple_arguments Regards, Jay Turpin Phone: 480-206-0981 "As long as you're going to be thinking anyway, THINK BIG." - Donald Trump From: rake-devel-bounces at rubyforge.org [mailto:rake-devel-bounces at rubyforge.org] On Behalf Of Dominic Sisneros Sent: Friday, September 26, 2008 9:36 AM To: Rake Development and Discussion Subject: Re: [Rake-devel] Rake::Win32.rake_system I have a windows box to test on. Which test is exercising this behavior so that I can try some things? sh was not expanding shell environment variables (on windows) without this change. If there is a better way to do this, let me know (or better yet, provide a patch). I have very limited ability to test on windows, so if someone wishes to run with this, I would be very grateful. -- -- Jim Weirich -- jim.weirich at gmail.com _______________________________________________ Rake-devel mailing list Rake-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/rake-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From luislavena at gmail.com Fri Sep 26 12:46:51 2008 From: luislavena at gmail.com (Luis Lavena) Date: Fri, 26 Sep 2008 13:46:51 -0300 Subject: [Rake-devel] win32_system_dir In-Reply-To: References: Message-ID: <71166b3b0809260946i4565780dp27c585cd4c2b952c@mail.gmail.com> On Fri, Sep 26, 2008 at 1:37 PM, Dominic Sisneros wrote: > I defined a HOME env variable under my user dir. I would like this the > Win32.win32_system_dir to use this ENV var if defined. > > > def test_win32_system_dir_uses_env_HOME_if_defined > in_environment('HOME' => 'C:/home') do > assert_equal "C:/home", Win32.win32_system_dir > end > end > The problem is that current Rake is giving the following order: APPDATA HOMEDRIVE + HOMEPATH USERPROFILE >From my point of view and experience, to it should try: HOME HOMEDRIVE + HOMEPATH APPDATA USERPROFILE Giving less priority to APPDATA will make users with mounted networked drives assigned as home keep their settings outside the terminal they are using and in their "home" folder. -- Luis Lavena AREA 17 - Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so. Douglas Adams From dsisnero at gmail.com Fri Sep 26 13:20:02 2008 From: dsisnero at gmail.com (Dominic Sisneros) Date: Fri, 26 Sep 2008 11:20:02 -0600 Subject: [Rake-devel] win32_system_dir In-Reply-To: <71166b3b0809260946i4565780dp27c585cd4c2b952c@mail.gmail.com> References: <71166b3b0809260946i4565780dp27c585cd4c2b952c@mail.gmail.com> Message-ID: +1 > > The problem is that current Rake is giving the following order: > > APPDATA > HOMEDRIVE + HOMEPATH > USERPROFILE > > >From my point of view and experience, to it should try: > > HOME > HOMEDRIVE + HOMEPATH > APPDATA > USERPROFILE > > Giving less priority to APPDATA will make users with mounted networked > drives assigned as home keep their settings outside the terminal they > are using and in their "home" folder. > > -- > Luis Lavena > AREA 17 > - > Human beings, who are almost unique in having the ability to learn from > the experience of others, are also remarkable for their apparent > disinclination to do so. > Douglas Adams > _______________________________________________ > Rake-devel mailing list > Rake-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/rake-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.weirich at gmail.com Fri Sep 26 20:44:21 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Fri, 26 Sep 2008 20:44:21 -0400 Subject: [Rake-devel] Rake::Win32.rake_system In-Reply-To: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C261@azsmsx502.amr.corp.intel.com> References: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> <71166b3b0809252056r5a4d48b0hd3ccf97c1827b755@mail.gmail.com> <3BE70AD8-B336-4F62-A96A-DB29B4E2B61E@gmail.com> <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C261@azsmsx502.amr.corp.intel.com> Message-ID: On Sep 26, 2008, at 12:18 PM, Turpin, Jay wrote: > OK - now I understand why this was done. However, the CALL command > is choking on any commands sent to sh that contains a newline > character. It's not pretty, but this fix appears to resolve the > problem (at least all of the tests pass): > > # Run a command line on windows. > def rake_system(*cmd) > perform_variable_expansion cmd[0] > system(*cmd) > end > > def perform_variable_expansion(cmd) > ENV.each do |k, v| > cmd.sub!(/\%#{k}\%/i, v) > end > end Perform the variable expansion by hand? Ok by me. I'm thinking I like this for the expansion: def perform_variable_expansion(cmd) s.gsub!(/%([A-Za-z_]+)%/) { |k| ENV[$1] || k } end -- -- Jim Weirich -- jim.weirich at gmail.com From jim.weirich at gmail.com Fri Sep 26 20:47:41 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Fri, 26 Sep 2008 20:47:41 -0400 Subject: [Rake-devel] win32_system_dir In-Reply-To: <71166b3b0809260946i4565780dp27c585cd4c2b952c@mail.gmail.com> References: <71166b3b0809260946i4565780dp27c585cd4c2b952c@mail.gmail.com> Message-ID: <7BA71DBF-8634-4F98-89EB-046FD963C524@gmail.com> On Sep 26, 2008, at 12:46 PM, Luis Lavena wrote: > On Fri, Sep 26, 2008 at 1:37 PM, Dominic Sisneros > wrote: >> I defined a HOME env variable under my user dir. I would like this >> the >> Win32.win32_system_dir to use this ENV var if defined. >> >> >> def test_win32_system_dir_uses_env_HOME_if_defined >> in_environment('HOME' => 'C:/home') do >> assert_equal "C:/home", Win32.win32_system_dir >> end >> end >> > > The problem is that current Rake is giving the following order: > > APPDATA > HOMEDRIVE + HOMEPATH > USERPROFILE > >> From my point of view and experience, to it should try: > > HOME > HOMEDRIVE + HOMEPATH > APPDATA > USERPROFILE APPDATA was what was suggested earlier on this mailing list (unless my memory is failing me). I have no strong feelings on the priority. However, I would like to get consensus here. -- -- Jim Weirich -- jim.weirich at gmail.com From jim.weirich at gmail.com Fri Sep 26 20:51:44 2008 From: jim.weirich at gmail.com (Jim Weirich) Date: Fri, 26 Sep 2008 20:51:44 -0400 Subject: [Rake-devel] win32_system_dir In-Reply-To: <7BA71DBF-8634-4F98-89EB-046FD963C524@gmail.com> References: <71166b3b0809260946i4565780dp27c585cd4c2b952c@mail.gmail.com> <7BA71DBF-8634-4F98-89EB-046FD963C524@gmail.com> Message-ID: On Sep 26, 2008, at 8:47 PM, Jim Weirich wrote: > APPDATA was what was suggested earlier on this mailing list (unless > my memory is failing me). I see my memory was failing me. In the ~/.rake thread there were several suggestions. Some with a hierarchy, some with just APPDATA. I misremembered that thread. -- -- Jim Weirich -- jim.weirich at gmail.com From luislavena at gmail.com Sat Sep 27 10:27:44 2008 From: luislavena at gmail.com (Luis Lavena) Date: Sat, 27 Sep 2008 12:27:44 -0200 Subject: [Rake-devel] win32_system_dir In-Reply-To: References: <71166b3b0809260946i4565780dp27c585cd4c2b952c@mail.gmail.com> <7BA71DBF-8634-4F98-89EB-046FD963C524@gmail.com> Message-ID: <71166b3b0809270727i4c61ae2bl45896d504abf9c95@mail.gmail.com> On Fri, Sep 26, 2008 at 10:51 PM, Jim Weirich wrote: > > On Sep 26, 2008, at 8:47 PM, Jim Weirich wrote: > >> APPDATA was what was suggested earlier on this mailing list (unless my >> memory is failing me). > > I see my memory was failing me. In the ~/.rake thread there were several > suggestions. Some with a hierarchy, some with just APPDATA. I > misremembered that thread. > That sometimes happens :-) The thing is that under some environments, APPDATA and USERPROFILE always maps to the local copy of the profile and not the server located one. Under those environments, mapping user home to a drive letter also happens, and you end having HOMEDRIVE and HOMEPATH defined. When the user logout of the terminal, sometimes the roaming profile only sync back to the server when terminal is shutting down, so you end with not up to date info in other terminals (that's the purpose of the mapped user home, you store info directly there instead). In Ruby 1.8, expansion of ~ required you to have defined HOME environment variable. In 1.9 however, that changed to use HOMEDRIVE + HOMEPATH if HOME is not defined, and the trying the other combinations (USERPROFILE and special Personal folder, which applies only to Vista). Thank you for your time invested in improve thsi situation for Windows users Jim :-) Regards, -- Luis Lavena AREA 17 - Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so. Douglas Adams From jay.turpin at intel.com Mon Sep 29 12:00:22 2008 From: jay.turpin at intel.com (Turpin, Jay) Date: Mon, 29 Sep 2008 09:00:22 -0700 Subject: [Rake-devel] Rake::Win32.rake_system In-Reply-To: References: <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C025@azsmsx502.amr.corp.intel.com> <71166b3b0809252056r5a4d48b0hd3ccf97c1827b755@mail.gmail.com> <3BE70AD8-B336-4F62-A96A-DB29B4E2B61E@gmail.com> <02EA1FD4E5205C48BE881A7ACE2F7C8C1214C261@azsmsx502.amr.corp.intel.com> Message-ID: <02EA1FD4E5205C48BE881A7ACE2F7C8C121F1AF2@azsmsx502.amr.corp.intel.com> Jim, That's much cleaner - I fixed a minor typo in the example you sent and tested on my machine. Seems to work fine. Any idea when we'll see a new gem with the fix in it? def perform_variable_expansion(cmd) cmd.gsub!(/%([A-Za-z_]+)%/) { |k| ENV[$1] || k } end Thanks for the help! Regards, Jay Turpin "May the road rise up to meet you. May the wind be always at your back. The rain falls soft upon your fields. And until we meet again, may God hold you in the palm of His hand." - Irish blessing -----Original Message----- From: rake-devel-bounces at rubyforge.org [mailto:rake-devel-bounces at rubyforge.org] On Behalf Of Jim Weirich Sent: Friday, September 26, 2008 5:44 PM To: Rake Development and Discussion Subject: Re: [Rake-devel] Rake::Win32.rake_system On Sep 26, 2008, at 12:18 PM, Turpin, Jay wrote: > OK - now I understand why this was done. However, the CALL command > is choking on any commands sent to sh that contains a newline > character. It's not pretty, but this fix appears to resolve the > problem (at least all of the tests pass): > > # Run a command line on windows. > def rake_system(*cmd) > perform_variable_expansion cmd[0] > system(*cmd) > end > > def perform_variable_expansion(cmd) > ENV.each do |k, v| > cmd.sub!(/\%#{k}\%/i, v) > end > end Perform the variable expansion by hand? Ok by me. I'm thinking I like this for the expansion: def perform_variable_expansion(cmd) s.gsub!(/%([A-Za-z_]+)%/) { |k| ENV[$1] || k } end -- -- Jim Weirich -- jim.weirich at gmail.com _______________________________________________ Rake-devel mailing list Rake-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/rake-devel