From win32utils-devel at rubyforge.org Wed Dec 1 14:39:33 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Wed Dec 1 14:38:43 2004 Subject: [Win32utils-devel] streams in fork In-Reply-To: <8FE83020B9E1A248A182A9B0A7B76E7358B0BA@itomae2km07.AD.QINTRA.COM> References: <8FE83020B9E1A248A182A9B0A7B76E7358B0BA@itomae2km07.AD.QINTRA.COM> Message-ID: <8d961d9004120111392a4e0a6b@mail.gmail.com> On Wed, 24 Nov 2004 10:52:24 -0600, win32utils-devel@rubyforge.org wrote: > > -----Original Message----- > > > > From: win32utils-devel-bounces@rubyforge.org > > [mailto:win32utils-devel-bounces@rubyforge.org] On Behalf Of > > win32utils-devel@rubyforge.org > > Sent: Tuesday, November 23, 2004 9:01 PM > > To: win32utils-devel@rubyforge.org > > Subject: [Win32utils-devel] streams in fork > > > > > > hi all, > > > > is there a way to use fork with redirected streams? i mean in > > a similar way that ruby's standard open3 does it. > > > > i'd like to be able to read the stdout/stdin/stderr streams > > from the forked ruby process in the parent process. > > > > cheers, > > aslak > > Do you mean so that you could create pipes using IO.popen for example? > Not at the moment, but it's something I would like to have implemented > at some point. > > If that's not what you meant, can you provide an example of what you > wish to accomplish? > Here is what I need to accomplish: I am writing a Ruby app (DamageControl - http://damagecontrol.codehaus.org/) that needs to execute on both native windows Ruby (i.e. no cygwin) and Linux/Unix Ruby - using the same ruby source code. DamageControl regulary spawns child processes and reads their stdout and stderr streams. I need to know the pid for these processes so they can be killed in the event of a app shutdown or if the process takes longer to execute than some timeout value that I specify. I also need to be able to wait for the process to return. To summarise I need to: o spawn a child process o get its stdin, stdout and stderr streams o get its pid o wait for it to complete o kill it if i want o set environment variables only available to the child I tried with your fork implementation, but discovered it wouldn't give me the streams. Then I tried with the open3 implementation, which doesn't give me the pid. I have submitted a patch for open3 (http://tinyurl.com/5b9ja) that turns it into an open4, which gives the pid. -So all my requirements seem to be satisfied except for the last one (env vars). If you think there are other ways to accomplish what I need it would be great to know. I also hope you have some feedback on my 2 patches. (It turns out I probably won't use fork unless support is added to it). Cheers, Aslak > Dan > > _______________________________________________ > win32utils-devel mailing list > win32utils-devel@rubyforge.org > http://rubyforge.org/mailman/listinfo/win32utils-devel > From win32utils-devel at rubyforge.org Wed Dec 1 14:47:42 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Wed Dec 1 14:46:52 2004 Subject: [Win32utils-devel] typo Message-ID: <8d961d90041201114757beb305@mail.gmail.com> In my previous mail the last line was supposed to be: "It turns out I probably won't use fork unless support FOR STREAMS is added to it." aslak From win32utils-devel at rubyforge.org Wed Dec 1 14:58:18 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Wed Dec 1 14:57:30 2004 Subject: [Win32utils-devel] typo Message-ID: <8FE83020B9E1A248A182A9B0A7B76E7358B0D8@itomae2km07.AD.QINTRA.COM> > -----Original Message----- > From: win32utils-devel-bounces@rubyforge.org > [mailto:win32utils-devel-bounces@rubyforge.org] On Behalf Of > win32utils-devel@rubyforge.org > Sent: Wednesday, December 01, 2004 12:48 PM > To: win32utils-devel@rubyforge.org > Subject: [Win32utils-devel] typo > > > In my previous mail the last line was supposed to be: > > "It turns out I probably won't use fork unless support FOR > STREAMS is added to it." > > aslak Ah, ok. As for open4, it was originally included in Park's win32_popen package (which became win32-open3). I didn't include open4 (or open2) initially because I just wanted to get something out the door quickly with a unified API. Now that we've had an actual request for it, I'll merge it back into win32-open3 asap (along with open2). I'll also compare your patch to Park's current implementation and see if there are any advantages to either. Or perhaps they're virtually identical. Look for a release this week. Now, about passing a separate environment to the child - you know, I've never done that and if you were to press me (press, press!) I would have to admit that I don't know how to do that, evn in *nix, because I've never needed to. I guess I'm a bad programmer. In Windows, the 7th argument to CreateProcess() allows you to specify an environment, or NULL to use the current environment. Currently, process_fork() just passes NULL. Should we allow an optional hash argument to specify the environment for forked children? Thoughts anyone? Regards, Dan From win32utils-devel at rubyforge.org Wed Dec 1 15:45:48 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Wed Dec 1 15:45:01 2004 Subject: [Win32utils-devel] The Hidden NT API Message-ID: <8FE83020B9E1A248A182A9B0A7B76E7358B0DA@itomae2km07.AD.QINTRA.COM> Here's a couple interesting links I snagged from OS News: http://www.sysinternals.com/ntw2k/info/ntdll.shtml http://undocumented.ntinternals.net/ Dan From win32utils-devel at rubyforge.org Thu Dec 2 01:10:48 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Thu Dec 2 01:10:11 2004 Subject: [Win32utils-devel] Updated win32-process in CVS Message-ID: Hi all, Aslak's Patch #1137 suggestion inspired me this evening, so I made some modifications to win32-process. Instead of passing back HANDLE's, I decided to pass back standard integers, and use OpenProcess() to get HANDLE's as needed on the fly based on the pid numbers. This probably has a minor performance hit, but I think it's more unix-like on the frontend. It also solves bug #712, and I no longer get a segfault with the sample code (though that may also be because I'm using a newer version of the Ruby Installer for Windows). Please give it a whirl and/or take a look at the source code and let me know what you think. Dan PS - Aslak, this should address Patch #1137 (return int instead of handle), though I haven't yet taken a look at Patch 1087 ($LOAD_PATH). From win32utils-devel at rubyforge.org Fri Dec 10 09:29:10 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Fri Dec 10 09:28:03 2004 Subject: [Win32utils-devel] win32-process 0.3.1 is out Message-ID: <8FE83020B9E1A248A182A9B0A7B76E7358B110@itomae2km07.AD.QINTRA.COM> Hi all, Just wanted to let you know that I released 0.3.1 last night. This addresses Bug #712 and incorporates Patches #1087 and #1137 (thanks Aslak). I'll try to get an open4 implementation out this weekend, though I noticed that Aslak's patch is different than what Park originally had, so I'll have to decide which I prefer, or if they can be blended somehow. Regards, Dan From win32utils-devel at rubyforge.org Fri Dec 10 11:08:25 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Fri Dec 10 11:07:17 2004 Subject: [Win32utils-devel] Modification for mkmf Message-ID: <8FE83020B9E1A248A182A9B0A7B76E7358B112@itomae2km07.AD.QINTRA.COM> Hi all, I came across this page today: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winprog /winprog/using_the_windows_headers.asp Based on that, I think we need a patch for mkmf when on Windows that sets _WIN32_WINNT automatically based on platform. This is the basic code I've come up with: require "mkmf" require "win32ole" require "socket" # We need to get the exact version number and set _WIN32_WINNT accordingly host = Socket.gethostname cs = "winmgmts:{impersonationLevel=impersonate,(security)}//#{host}/root/cimv 2" wmi = WIN32OLE.connect(cs) major = nil minor = nil wmi.InstancesOf("Win32_OperatingSystem").each{ |ole| major, minor = ole.Version.split(".").map{ |e| e.to_i } break } win32_winnt = nil # The value for Windows NT 4 and Windows 95 are the same. # The value for Windows 2000 and Windows ME are the same. case major when 5 case minor when 2 # 2003 win32_winnt = "0x0502" when 1 # XP win32_winnt = "0x0501" else # 2000 win32_winnt = "0x0500" end when 4 case minor when 0 # NT or 95 win32_winnt = "0x0400" when 90 # ME win32_winnt = "0x0500" when 10 # 98 win32_winnt = "0x0410" end end # So long as we're not on Windows 3.51 or earlier, this should be defined if win32_winnt $CPPFLAGS += " -D_WIN32_WINNT=#{win32_winnt}" end # END I realize there are possible scenarios where this won't quite work (see the example they discuss near the bottom of the link I provided). In such cases, it would be up to the extension writer to set $CPPFLAGS accordingly. Optionally, the WMI + OLE code could be replaced with WIN32API + GetVersionEx(). What do you think? Dan From win32utils-devel at rubyforge.org Fri Dec 10 12:08:51 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Fri Dec 10 12:07:43 2004 Subject: [Win32utils-devel] Modification for mkmf Message-ID: <8FE83020B9E1A248A182A9B0A7B76E7358B113@itomae2km07.AD.QINTRA.COM> Here's a Win32API snippet for getting major and minor: require "Win32API" GetVersionEx = Win32API.new('kernel32','GetVersionEx','P','I') swCSDVersion = "\0" * 128 OSVERSIONINFO = [148,0,0,0,0,swCSDVersion].pack("LLLLLa128") GetVersionEx.call(OSVERSIONINFO) info = OSVERSIONINFO.unpack("LLLLLa128") major, minor = info[1,2] Regards, Dan From win32utils-devel at rubyforge.org Fri Dec 10 15:12:43 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Fri Dec 10 15:12:01 2004 Subject: [Win32utils-devel] win32-process 0.3.1 is out In-Reply-To: <8FE83020B9E1A248A182A9B0A7B76E7358B110@itomae2km07.AD.QINTRA.COM> References: <8FE83020B9E1A248A182A9B0A7B76E7358B110@itomae2km07.AD.QINTRA.COM> Message-ID: <8d961d90041210121267964195@mail.gmail.com> Thanks Dan, much appreciated. I have some other patches coming up soon :-) Here is what I'd like to do (expressed as a unit test since I am a TDD zealot). This relies on the soon-to-be popen4. require 'test/unit' require 'timeout' require "win32/open3" require 'win32/process' class Process2Test < Test::Unit::TestCase def test_can_timeout_native_function timeout(2) do pid = nil begin Open4.popen4("notepad") do |stdin, stdout, stderr, pid| Process.waitpid2(pid) end fail("Should have timed out") rescue Timeout::Error => expected Process.kill(-9, pid) if pid end end end end The problem is - the timeout never happens and the test program blocks until notepad is closed manually. Any idea how to implement this so that the timeout can happen? I am assuming the problem is related to the blocking nature of WaitForSingleObject, and that we can't interrupt it. Do we have to introduce some threads and mutexes and and such from the win32 api to deal with this? Cheers, Aslak On Fri, 10 Dec 2004 08:29:10 -0600, win32utils-devel@rubyforge.org wrote: > Hi all, > > Just wanted to let you know that I released 0.3.1 last night. This > addresses Bug #712 and incorporates Patches #1087 and #1137 (thanks > Aslak). > > I'll try to get an open4 implementation out this weekend, though I > noticed that Aslak's patch is different than what Park originally had, > so I'll have to decide which I prefer, or if they can be blended > somehow. > > Regards, > > Dan > > _______________________________________________ > win32utils-devel mailing list > win32utils-devel@rubyforge.org > http://rubyforge.org/mailman/listinfo/win32utils-devel > From win32utils-devel at rubyforge.org Fri Dec 10 15:37:35 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Fri Dec 10 15:36:29 2004 Subject: [Win32utils-devel] win32-process 0.3.1 is out Message-ID: <8FE83020B9E1A248A182A9B0A7B76E7358B115@itomae2km07.AD.QINTRA.COM> Hi Aslak, > -----Original Message----- > From: win32utils-devel-bounces@rubyforge.org > [mailto:win32utils-devel-bounces@rubyforge.org] On Behalf Of > win32utils-devel@rubyforge.org > Sent: Friday, December 10, 2004 1:13 PM > To: win32utils-devel@rubyforge.org > Subject: Re: [Win32utils-devel] win32-process 0.3.1 is out > > > Thanks Dan, much appreciated. > > I have some other patches coming up soon :-) Here is what I'd > like to do (expressed as a unit test since I am a TDD > zealot). This relies on the soon-to-be popen4. > > require 'test/unit' > require 'timeout' > require "win32/open3" > require 'win32/process' > > class Process2Test < Test::Unit::TestCase > def test_can_timeout_native_function > timeout(2) do > pid = nil > begin > Open4.popen4("notepad") do |stdin, stdout, stderr, pid| > Process.waitpid2(pid) > end > fail("Should have timed out") > rescue Timeout::Error => expected > Process.kill(-9, pid) if pid > end > end > end > end > > The problem is - the timeout never happens and the test > program blocks until notepad is closed manually. > > Any idea how to implement this so that the timeout can > happen? I am assuming the problem is related to the blocking > nature of WaitForSingleObject, and that we can't interrupt > it. Do we have to introduce some threads and mutexes and and > such from the win32 api to deal with this? > > Cheers, > Aslak As is, I don't think that code can work on Win32. The only answer I can think of would be to allow a timeout value as the second argument to the waitpid and waitpid2 methods. So, instead of having INFINITE hard coded as the second argument to WaitForSingleObject, for example, you could do Process.waitpid2(pid,2), and the 2 (x1000, since the 2nd arg is milliseconds) would be passed as the 2nd argument to the WaitForSingleObject() function internally. That seems a reasonable compromise to me, though I worry about breaking too much with the way things work on *nix. Thoughts? - Dan From win32utils-devel at rubyforge.org Fri Dec 10 15:42:25 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Fri Dec 10 15:41:18 2004 Subject: [Win32utils-devel] win32-process 0.3.1 is out Message-ID: <8FE83020B9E1A248A182A9B0A7B76E7358B117@itomae2km07.AD.QINTRA.COM> Also, if anyone has any ideas as to how WUNTRACED or WNOHANG should work on Win32 (if at all), I'm all ears. :) - Dan From win32utils-devel at rubyforge.org Fri Dec 10 15:54:56 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Fri Dec 10 15:54:27 2004 Subject: [Win32utils-devel] win32-process 0.3.1 is out In-Reply-To: <8FE83020B9E1A248A182A9B0A7B76E7358B115@itomae2km07.AD.QINTRA.COM> References: <8FE83020B9E1A248A182A9B0A7B76E7358B115@itomae2km07.AD.QINTRA.COM> Message-ID: <8d961d9004121012548b30f30@mail.gmail.com> On Fri, 10 Dec 2004 14:37:35 -0600, win32utils-devel@rubyforge.org wrote: > Hi Aslak, > > > > > -----Original Message----- > > From: win32utils-devel-bounces@rubyforge.org > > [mailto:win32utils-devel-bounces@rubyforge.org] On Behalf Of > > win32utils-devel@rubyforge.org > > Sent: Friday, December 10, 2004 1:13 PM > > To: win32utils-devel@rubyforge.org > > Subject: Re: [Win32utils-devel] win32-process 0.3.1 is out > > > > > > Thanks Dan, much appreciated. > > > > I have some other patches coming up soon :-) Here is what I'd > > like to do (expressed as a unit test since I am a TDD > > zealot). This relies on the soon-to-be popen4. > > > > require 'test/unit' > > require 'timeout' > > require "win32/open3" > > require 'win32/process' > > > > class Process2Test < Test::Unit::TestCase > > def test_can_timeout_native_function > > timeout(2) do > > pid = nil > > begin > > Open4.popen4("notepad") do |stdin, stdout, stderr, pid| > > Process.waitpid2(pid) > > end > > fail("Should have timed out") > > rescue Timeout::Error => expected > > Process.kill(-9, pid) if pid > > end > > end > > end > > end > > > > The problem is - the timeout never happens and the test > > program blocks until notepad is closed manually. > > > > Any idea how to implement this so that the timeout can > > happen? I am assuming the problem is related to the blocking > > nature of WaitForSingleObject, and that we can't interrupt > > it. Do we have to introduce some threads and mutexes and and > > such from the win32 api to deal with this? > > > > Cheers, > > Aslak > > As is, I don't think that code can work on Win32. The only answer I can > think of would be to allow a timeout value as the second argument to the > waitpid and waitpid2 methods. So, instead of having INFINITE hard coded > as the second argument to WaitForSingleObject, for example, you could do > Process.waitpid2(pid,2), and the 2 (x1000, since the 2nd arg is > milliseconds) would be passed as the 2nd argument to the > WaitForSingleObject() function internally. > > That seems a reasonable compromise to me, though I worry about breaking > too much with the way things work on *nix. Thoughts? > The standard Ruby Process.waitpid2 method already takes a 2nd argument, but this is a flag and not a timeout value. I think it would be a bad idea to introduce incompatible semantics for the sake of portability of ruby code beween windows and *nix. I am pretty unfamiliar with the Win32 API, but I would assume it's possible to use threads and mutexes in the C code. Could the C code launch a new thread and do the WaitForSingleObject there? Then the process_waitpid2 could block until a condition variable is released - either as the result of WaitForSingleObject returning, or as the result of a C-intercepted Ruby exception (the timeout exception). I don't know if this is possible though - what do you think - you're the Ruby-Win32-C specialist ;-) Aslak > - Dan > > > > > _______________________________________________ > win32utils-devel mailing list > win32utils-devel@rubyforge.org > http://rubyforge.org/mailman/listinfo/win32utils-devel > From win32utils-devel at rubyforge.org Fri Dec 10 16:05:41 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Fri Dec 10 16:04:33 2004 Subject: [Win32utils-devel] win32-process 0.3.1 is out Message-ID: <8FE83020B9E1A248A182A9B0A7B76E7358B118@itomae2km07.AD.QINTRA.COM> > > That seems a reasonable compromise to me, though I worry about > > breaking too much with the way things work on *nix. Thoughts? > > > > The standard Ruby Process.waitpid2 method already takes a 2nd > argument, but this is a flag and not a timeout value. I think > it would be a bad idea to introduce incompatible semantics > for the sake of portability of ruby code beween windows and *nix. Heh - the 2nd argument is unused internally at the moment. I put it there in case we ever came up with definitions for WNOHANG and WUNTRACED (or our own special constants). Perhaps a 3rd argument? > I am pretty unfamiliar with the Win32 API, but I would assume > it's possible to use threads and mutexes in the C code. > > Could the C code launch a new thread and do the > WaitForSingleObject there? Then the process_waitpid2 could > block until a condition variable is released - either as the > result of WaitForSingleObject returning, or as the result of > a C-intercepted Ruby exception (the timeout exception). I > don't know if this is possible though - what do you think - > you're the Ruby-Win32-C specialist ;-) > > Aslak Now there's an idea. I'll have to look into it. In the meantime if you or anyone else listening has any ideas as to what the actual C code should be (Park?), I'd be happy to hear them. :) - Dan From win32utils-devel at rubyforge.org Mon Dec 13 00:43:14 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Mon Dec 13 00:42:50 2004 Subject: [Win32utils-devel] win32-thread experiment Message-ID: Hi all, I've checked in some semi-working code into CVS for win32-thread. I don't really expect this to ever work properly, given that the Ruby interpreter itself isn't thread safe, but I thought it was fun to tinker. Anyway, with that in mind, give the following a shot: require "win32/thread" t = Win32::Thread.new{ puts "hello" sleep 2 } puts "Done" sleep 1 t.join That should work. But, if more than one thread gets involved you'll likely see errors. Feel free to tinker. :) - Dan From win32utils-devel at rubyforge.org Fri Dec 17 01:59:20 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Fri Dec 17 01:57:57 2004 Subject: [Win32utils-devel] win32-thread experiment In-Reply-To: References: Message-ID: Hi Dan, On Fri, 17 Dec 2004 00:22:19 +0900, Daniel Berger wrote: > I've tried tinkering with rb_gc_disable() and rb_gc_start() in various > places in the code but I haven't had any luck. I'm still getting > segfaults. > > Feel free to join the win32utils-devel mailing list. I'd be happy to > hear any ideas you have. :) Actually, I have joined the win32utils-devel mailing list. I gave you a little help with converting FILE* to fileno or to HANDLE back in November :) On the threading, what if you call rb_gc_disable(), then create some win32 threads and see if you still get segfaults? (Don't ever call rb_gc_start). Of course, if the threads are doing much with Ruby objects they'll be creating garbage, so at some point the process will run out of memory due to the lack of GC. This is just an experiment to see if threads can run happily if GC is off. If so, we can try to figure out where to go from here. Wayne From win32utils-devel at rubyforge.org Sat Dec 18 10:05:07 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Sat Dec 18 10:03:40 2004 Subject: [Win32utils-devel] Removing COMSPEC from win32-open3 Message-ID: <20041218150507.65209.qmail@web50303.mail.yahoo.com> Hi all, Aslak brought up a good point with regards to returning the pid for a popen4() method. As it stands now, it returns the pid of the shell rather than the pid of the process itself. However, my attempts to remove the COMSPEC stuff result in CreateProcess() failing with an error code of 2, i.e. "the system cannot find the file specified" - not too helpful. This is when I tried to run the "ver" command. (A short while later) Hmm...it looks like some of these command line apps are actually dll files, not .exe files (as in the case of 'ver') that get loaded by cmd.exe. Ugh. (Thinking out loud) Hmm...if you run a command like "ver", then the pid of the shell would seem to be the proper value. But if you run a command like "notepad", then it probably isn't what you want. (More thinking out loud) Try CreateProcess twice. The first time, call it without COMSPEC. If that fails, then try it with COMSPEC. Only if the second call fails would it finally be considered an error. That still leaves the SW_HIDE vs. SW_SHOW issue, though. (Possible solution) Create an Open3.show_window= method that accepts a boolean value. It would be the onus of the programmer to remember to set this to true if they want to create a GUI window of some sort using popen3 or popen4. Ok, I'm done thinking for a little while. :) Comments? Dan __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From win32utils-devel at rubyforge.org Sat Dec 18 13:33:38 2004 From: win32utils-devel at rubyforge.org (win32utils-devel@rubyforge.org) Date: Sat Dec 18 13:32:15 2004 Subject: [Win32utils-devel] Minor correction to last post Message-ID: <20041218183338.80491.qmail@web50310.mail.yahoo.com> I was mistaken about some of these commands being dll's. They are builtins for cmd.exe. Anyway, I think the "try twice" for CreateProcess() approach is still the best option. Dan __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com