From Daniel.Berger at qwest.com Wed Nov 1 14:49:20 2006 From: Daniel.Berger at qwest.com (Berger, Daniel) Date: Wed, 1 Nov 2006 13:49:20 -0600 Subject: [Win32utils-devel] fatal flaw in popen4 on windows? [WAS] Re: Nonblocking IO read (fwd) In-Reply-To: Message-ID: <39AA6550E5AA554AB1456707D6E5563D0259E8D5@QTOMAE2K3M01.AD.QINTRA.COM> Hi Ara, I'll take a look. In the meantime, I've cc'd the win32utils-devel mailing list to see if anyone has any ideas/suggestions. Regards, Dan > -----Original Message----- > From: ara.t.howard at noaa.gov [mailto:ara.t.howard at noaa.gov] > Sent: Wednesday, November 01, 2006 12:36 PM > To: Berger, Daniel > Subject: fatal flaw in popen4 on windows? [WAS] Re: > Nonblocking IO read (fwd) > > > > hi dan- > > forgot to cc you on this... hopefully you'll have some suggestions? > > regards. > > -a > -- > my religion is very simple. my religion is kindness. -- the > dalai lama > > ---------- Forwarded message ---------- > Date: Wed, 1 Nov 2006 12:19:12 -0700 (MST) > From: ara.t.howard at noaa.gov > To: ruby-talk ML > Subject: fatal flaw in popen4 on windows? [WAS] Re: > Nonblocking IO read > > > On Thu, 2 Nov 2006, Robert Klemme wrote: > > > Tom Pollard wrote: > >> Anyway, you would only need nonblocking IO if you wanted > to read bits > >> of the > >> stderr stream before the command exited, but that doesn't > sound like what > >> you're want. > > > > Actually this is not correct: if there is a lot written to > stderr then > > you need to read that concurrently. If you do not do that then the > > process will block on some stderr write operation that fills up the > > pipe and you get a deadlock because your code waits for process > > termination. > > not only is that true but, afaik, it's why popen4 cannot even > work on windows! this program will eventually hang on either > windows or unix > > > harp:~ > cat a.rb > require 'rubygems' > require 'popen4' > > n = (ARGV.shift || 4242).to_i > ruby = ARGV.shift || 'ruby' > > system "ruby -e 42" or abort "ruby not in your path!" > > STDOUT.sync = STDERR.sync = true > > program = <<-program > #{ n }.times do > t = Time.now.to_f > STDOUT.puts t > STDERR.puts t > end > program > > > POpen4.popen4(ruby) do |stdout, stderr, stdin, pid| > STDOUT.puts pid > > stdin.puts program > stdin.close > > Thread.new{ stdout.each{|line| STDOUT.puts line} } > # > # uncomment and it won't hang!!! > # > #Thread.new{ stderr.each{|line| STDERR.puts line} } > end > > puts 'done' > > > > however, on windows it will always hang - even if the line > above is uncommented. this is because if one popen4s a > process it's __essential__, as robert correctly points out, > to continually consume any stdout or stderr produced - > otherwise the program will eventually get stuck in EPIPE and > you'll be waiting for this stuck program. > > and here's the rub: you cannot reliably consume both stdout > and stderr under windows using threads or, afaik, nonblocking > io. perhaps the new nonblock_* methods could help with this? > it'd no doubt be a major reworking... > > *** i'm really hoping someone will chime in here and prove me > wrong *** > > for reference i'm including the code for my open4 lib's spawn > method: which illustrates the logical concept of what must be > done to avoid a subprocess blocked writing to it's parent's pipes... > > > # > # for complete code see > # http://codeforpeople.com/lib/ruby/open4/open4-0.9.1/lib/open4.rb > # http://rubyforge.org/frs/?group_id=1024&release_id=7556 > # > > def spawn arg, *argv #--{{{ > argv.unshift(arg) > opts = ((argv.size > 1 and Hash === argv.last) ? argv.pop : {}) > argv.flatten! > cmd = argv.join(' ') > > getopt = getopts opts > > ignore_exit_failure = getopt[ 'ignore_exit_failure', > getopt['quiet', false] > ] > ignore_exec_failure = getopt[ 'ignore_exec_failure', > !getopt['raise', true] > ] > exitstatus = getopt[ %w( exitstatus exit_status status ) ] > stdin = getopt[ %w( stdin in i 0 ) << 0 ] > stdout = getopt[ %w( stdout out o 1 ) << 1 ] > stderr = getopt[ %w( stderr err e 2 ) << 2 ] > pid = getopt[ 'pid' ] > timeout = getopt[ %w( timeout spawn_timeout ) ] > stdin_timeout = getopt[ %w( stdin_timeout ) ] > stdout_timeout = getopt[ %w( stdout_timeout io_timeout ) ] > stderr_timeout = getopt[ %w( stderr_timeout ) ] > status = getopt[ %w( status ) ] > cwd = getopt[ %w( cwd dir ), Dir.pwd ] > > exitstatus = > case exitstatus > when TrueClass, FalseClass > ignore_exit_failure = true if exitstatus > [0] > else > [*(exitstatus || 0)].map{|i| Integer i} > end > > stdin ||= '' if stdin_timeout > stdout ||= '' if stdout_timeout > stderr ||= '' if stderr_timeout > > started = false > > status = > begin > Dir.chdir(cwd) do > Timeout::timeout(timeout) do > popen4(*argv) do |c, i, o, e| > started = true > > %w( replace pid= << push update ).each do |msg| > break(pid.send(msg, c)) if pid.respond_to? msg > end > > # > # this is the critical bit!!! > # > > te = ThreadEnsemble.new c > > te.add_thread(i, stdin) do |i, stdin| > relay stdin, i, stdin_timeout > i.close rescue nil > end > > te.add_thread(o, stdout) do |o, stdout| > relay o, stdout, stdout_timeout > end > > te.add_thread(e, stderr) do |o, stderr| > relay e, stderr, stderr_timeout > end > > te.run > end > end > end > rescue > raise unless(not started and ignore_exec_failure) > end > > raise SpawnError.new(cmd, status) unless > (ignore_exit_failure or (status.nil? and > ignore_exec_failure) or > exitstatus.include?(status.exitstatus)) > > status > #--}}} > end > > > kind regards. > > -a > -- > my religion is very simple. my religion is kindness. -- the > dalai lama > > -a > -- > my religion is very simple. my religion is kindness. -- the > dalai lama > This communication is the property of Qwest and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments. From noreply at rubyforge.org Fri Nov 17 05:17:30 2006 From: noreply at rubyforge.org (noreply at rubyforge.org) Date: Fri, 17 Nov 2006 05:17:30 -0500 (EST) Subject: [Win32utils-devel] [ win32utils-Bugs-6722 ] win32-changenotify fails on Ruby 1.8.4 Message-ID: <20061117101730.EBCA95240E89@rubyforge.org> Bugs item #6722, was opened at 2006-11-17 05:17 You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=411&aid=6722&group_id=85 Category: None Group: None Status: Open Resolution: None Priority: 3 Submitted By: Nobody (None) Assigned to: Nobody (None) Summary: win32-changenotify fails on Ruby 1.8.4 Initial Comment: throws an exception when loading, roughly NameError: Invalid name "recursive?" (probably complaining about the ivar named "@recursive?") Can be fixed by replacing "recursive?" with "recursive" in changenotify.c ---------------------------------------------------------------------- You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=411&aid=6722&group_id=85 From djberg96 at yahoo.com Mon Nov 20 23:55:23 2006 From: djberg96 at yahoo.com (Daniel Berger) Date: Mon, 20 Nov 2006 20:55:23 -0800 (PST) Subject: [Win32utils-devel] Fw: re. win32-process Message-ID: <20061121045523.73248.qmail@web50311.mail.yahoo.com> Hi all, Any ideas for the question below? I know how to do this in theory - make the 'inherit' flag true, and set the 'stdout' and 'stderr' startf_flags hash options to something in the startup_info hash, but I wasn't sure how to do this in practice. It would be nice if the answer could be something like this: require 'win32/process' app_name = 'c:\ruby\bin\ruby "' + Dir.pwd + '/test.rb"' my_out = $stdout.clone my_err = $stderr.clone my_out.reopen('NUL') my_err.reopen('NUL') Process.create( :app_name => app_name, :inherit => true, :creation_flags => Process::DETACHED_PROCESS, :startup_info => { :startf_flags => Process::STARTF_USESTDHANDLES, :stdout => my_out, :stderr => my_err } ) I think it would be neat if you could pass IO objects (or perhaps a fileno) to :stdout, :stderr, and :stdin directly. Any ideas on how to do that? Thanks, Dan ----- Forwarded Message ---- From: Kirill Miazine To: Daniel J. Berger Sent: Monday, November 20, 2006 12:35:40 PM Subject: re. win32-process Hello Daniel,- I'm one of those poor people who sit behind a corporate firewall that for some reason blocks access to useful ports. But I'm lucky enought to have access to port 22, so I can fire up ssh and set up port forwarding. I used to do so with PuTTY and it worked nice. But having that terminal window hanging around is annoying, so I tried to find a better workaround. I installed Ruby and win32-process. I'm new to Ruby. I'm also new to Windows, so I've no idea about how process managemenent, stdin/stdout/stderr etc happen in this environment (and where is the documentation?!?). (I've been programming Perl on *nix for several years.) I find out that I use the flag Process::DETACHED_PROCESS to avoid the process window. But how do I suppress command output? I want something like "2>&1 > /dev/null". I saw stdin, stdout and stderr options in startup_info, but I wasn't able to find a solution to the ======================================= require 'win32/process' class <<$stdout def write(stuff) return end end class <<$stderr def write(stuff) return end end cmd = 'c:\Program Files\plink.exe -batch -N -load proxy' p = Process.create(:app_name => cmd, :creation_flags => Process::DETACHED_PROCESS) Process.waitpid p.process_i ======================================= (PLink is a command-line ssh client, very handy.) I'm grateful for any advice! Thanks in advance! - Kirill ____________________________________________________________________________________ Sponsored Link Online or Campus degree Associate's, Bachelor's, or Master's in less than one year.www.findtherightschool.com From phasis at gmail.com Tue Nov 21 03:16:40 2006 From: phasis at gmail.com (Heesob Park) Date: Tue, 21 Nov 2006 17:16:40 +0900 Subject: [Win32utils-devel] Fw: re. win32-process In-Reply-To: <20061121045523.73248.qmail@web50311.mail.yahoo.com> References: <20061121045523.73248.qmail@web50311.mail.yahoo.com> Message-ID: Hi, 2006/11/21, Daniel Berger : > > Hi all, > > Any ideas for the question below? I know how to do this in theory - make > the 'inherit' flag true, and set the 'stdout' and 'stderr' startf_flags hash > options to something in the startup_info hash, but I wasn't sure how to do > this in practice. > > It would be nice if the answer could be something like this: > > require 'win32/process' > > app_name = 'c:\ruby\bin\ruby "' + Dir.pwd + '/test.rb"' > my_out = $stdout.clone > my_err = $stderr.clone > my_out.reopen('NUL') > my_err.reopen('NUL') > > Process.create( > :app_name => app_name, > :inherit => true, > :creation_flags => Process::DETACHED_PROCESS, > :startup_info => { > :startf_flags => Process::STARTF_USESTDHANDLES, > :stdout => my_out, > :stderr => my_err > } > ) > > I think it would be neat if you could pass IO objects (or perhaps a > fileno) to :stdout, :stderr, and :stdin directly. Any ideas on how to do > that? > > Thanks, > > Dan Use _get_osfhandle function like this: :stdout => _get_osfhandle(my_out.to_i), :stderr => _get_osfhandle(my_err.to_i), Regards, Park Heesob -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/win32utils-devel/attachments/20061121/fb330ea0/attachment.html From johan.nilsson at esrange.ssc.se Tue Nov 21 02:44:19 2006 From: johan.nilsson at esrange.ssc.se (Johan Nilsson) Date: Tue, 21 Nov 2006 08:44:19 +0100 Subject: [Win32utils-devel] Fw: re. win32-process In-Reply-To: <20061121045523.73248.qmail@web50311.mail.yahoo.com> Message-ID: <6D2C2D1D3D1E8040AEFA12C90600868201105038@rbe-mail1.ssc.internal> > -----Original Message----- > From: win32utils-devel-bounces at rubyforge.org > [mailto:win32utils-devel-bounces at rubyforge.org] On Behalf Of > Daniel Berger > Sent: den 21 november 2006 05:55 > To: win32utils-devel at rubyforge.org > Subject: [Win32utils-devel] Fw: re. win32-process > > Hi all, > > Any ideas for the question below? I know how to do this in > theory - make the 'inherit' flag true, and set the 'stdout' > and 'stderr' startf_flags hash options to something in the > startup_info hash, but I wasn't sure how to do this in practice. > > It would be nice if the answer could be something like this: > > require 'win32/process' > > app_name = 'c:\ruby\bin\ruby "' + Dir.pwd + '/test.rb"' > my_out = $stdout.clone > my_err = $stderr.clone > my_out.reopen('NUL') > my_err.reopen('NUL') > > Process.create( > :app_name => app_name, > :inherit => true, > :creation_flags => Process::DETACHED_PROCESS, > :startup_info => { > :startf_flags => Process::STARTF_USESTDHANDLES, > :stdout => my_out, > :stderr => my_err > } > ) > > I think it would be neat if you could pass IO objects (or > perhaps a fileno) to :stdout, :stderr, and :stdin directly. > Any ideas on how to do that? If you implement natively using VC, and have access to either the underlying FILE* or file descriptor, take a look at: _fileno _get_osfhandle If you are doing it in pure Ruby, I don't know. As for the original question, perhaps something like using Process.create to launch cmd.exe with the args: '/c "c:/Program Files/plink.exe -batch -N -load proxy > 2>&1 > NUL"' could work (haven't tried it). / Johan From Daniel.Berger at qwest.com Tue Nov 21 11:07:46 2006 From: Daniel.Berger at qwest.com (Daniel Berger) Date: Tue, 21 Nov 2006 09:07:46 -0700 Subject: [Win32utils-devel] Fw: re. win32-process In-Reply-To: References: <20061121045523.73248.qmail@web50311.mail.yahoo.com> Message-ID: <45632452.7030007@qwest.com> Heesob Park wrote: > Hi, > > 2006/11/21, Daniel Berger >: > > Hi all, > > Any ideas for the question below? I know how to do this in theory > - make the 'inherit' flag true, and set the 'stdout' and 'stderr' > startf_flags hash options to something in the startup_info hash, > but I wasn't sure how to do this in practice. > > It would be nice if the answer could be something like this: > > require 'win32/process' > > app_name = 'c:\ruby\bin\ruby "' + Dir.pwd + '/test.rb"' > my_out = $stdout.clone > my_err = $stderr.clone > my_out.reopen('NUL') > my_err.reopen('NUL') > > Process.create( > :app_name => app_name, > :inherit => true, > :creation_flags => Process::DETACHED_PROCESS, > :startup_info => { > :startf_flags => Process::STARTF_USESTDHANDLES, > :stdout => my_out, > :stderr => my_err > } > ) > > I think it would be neat if you could pass IO objects (or perhaps > a fileno) to :stdout, :stderr, and :stdin directly. Any ideas on > how to do that? > > Thanks, > > Dan > > > Use _get_osfhandle function like this: > > :stdout => _get_osfhandle(my_out.to_i), > :stderr => _get_osfhandle(my_err.to_i), > > Regards, > > Park Heesob Thanks, I couldn't remember the name of the function last night - too tired. ;) I've added get_osfhandle to the Windows::Handle module (in windows-pr) in CVS. Then I added this to the Process.create method: # Automatically handle stdin, stdout and stderr as either IO objects # or file descriptors. This won't work for StringIO, however. ['stdin', 'stdout', 'stderr'].each{ |io| if si_hash[io] if si_hash[io].respond_to?(:fileno) handle = get_osfhandle(si_hash[io].fileno) else handle = get_osfhandle(si_hash[io]) end if handle == INVALID_HANDLE_VALUE raise ProcessError, get_last_error end si_hash[io] = handle end } That seems to work fairly well. The only quirk I noticed is that it doesn't seem to honor the append flag for the file descriptor: # test.rb 5.times{ |i| puts "I: #{i}" } # proc_test.rb require 'win32/process' app = 'c:\ruby\bin\ruby "' + Dir.pwd + '/test.rb"' fh = File.open('test.txt', 'a') Process.create( :app_name => app, :inherit => true, :creation_flags => Process::DETACHED_PROCESS, :startup_info => { :startf_flags => Process::STARTF_USESTDHANDLES, :stdout => fh } ) fh.close Instead of appending to the test.txt file, it appears to simply overwrite it each time. I can't remember if we discussed this in the past or not. If there's no way to deal with it, then it's not a huge deal - I just need to document it. What do you think? Regards, Dan PS - Any way we could support StringIO as well? From noreply at rubyforge.org Wed Nov 22 01:22:34 2006 From: noreply at rubyforge.org (noreply at rubyforge.org) Date: Wed, 22 Nov 2006 01:22:34 -0500 (EST) Subject: [Win32utils-devel] [ win32utils-Bugs-6722 ] win32-changenotify fails on Ruby 1.8.4 Message-ID: <20061122062235.346B55240ABD@rubyforge.org> Bugs item #6722, was opened at 2006-11-17 03:17 You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=411&aid=6722&group_id=85 >Category: win32-changenotify Group: None Status: Open Resolution: None Priority: 3 Submitted By: Nobody (None) >Assigned to: Daniel Berger (djberg96) Summary: win32-changenotify fails on Ruby 1.8.4 Initial Comment: throws an exception when loading, roughly NameError: Invalid name "recursive?" (probably complaining about the ivar named "@recursive?") Can be fixed by replacing "recursive?" with "recursive" in changenotify.c ---------------------------------------------------------------------- >Comment By: Daniel Berger (djberg96) Date: 2006-11-21 23:22 Message: Hi, Looks like a bug caused by some changes in the Ruby internals that have been made over the last couple of releases. I'll have a fix out this weekend. Thanks for the report. - Dan ---------------------------------------------------------------------- You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=411&aid=6722&group_id=85 From noreply at rubyforge.org Wed Nov 22 01:33:18 2006 From: noreply at rubyforge.org (noreply at rubyforge.org) Date: Wed, 22 Nov 2006 01:33:18 -0500 (EST) Subject: [Win32utils-devel] [ win32utils-Bugs-6722 ] win32-changenotify fails on Ruby 1.8.4 Message-ID: <20061122063318.F1D0C5240A41@rubyforge.org> Bugs item #6722, was opened at 2006-11-17 03:17 You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=411&aid=6722&group_id=85 Category: win32-changenotify >Group: Code >Status: Closed >Resolution: Accepted Priority: 3 Submitted By: Nobody (None) Assigned to: Daniel Berger (djberg96) Summary: win32-changenotify fails on Ruby 1.8.4 Initial Comment: throws an exception when loading, roughly NameError: Invalid name "recursive?" (probably complaining about the ivar named "@recursive?") Can be fixed by replacing "recursive?" with "recursive" in changenotify.c ---------------------------------------------------------------------- >Comment By: Daniel Berger (djberg96) Date: 2006-11-21 23:33 Message: Alright, fixed in CVS. The updated code will be in the next release. - Dan ---------------------------------------------------------------------- Comment By: Daniel Berger (djberg96) Date: 2006-11-21 23:22 Message: Hi, Looks like a bug caused by some changes in the Ruby internals that have been made over the last couple of releases. I'll have a fix out this weekend. Thanks for the report. - Dan ---------------------------------------------------------------------- You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=411&aid=6722&group_id=85 From noreply at rubyforge.org Thu Nov 23 20:38:24 2006 From: noreply at rubyforge.org (noreply at rubyforge.org) Date: Thu, 23 Nov 2006 20:38:24 -0500 (EST) Subject: [Win32utils-devel] [ win32utils-Feature Requests-1446 ] Add a " filesystem" interface Message-ID: <20061124013824.C1E24524135C@rubyforge.org> Feature Requests item #1446, was opened at 2005-02-04 20:14 You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=414&aid=1446&group_id=85 Category: None Group: None Status: Open >Resolution: Accepted Priority: 3 Submitted By: Daniel Berger (djberg96) >Assigned to: Daniel Berger (djberg96) >Summary: Add a "filesystem" interface Initial Comment: Create a package that allows users to get filesystem information (diskspace, etc). If possible, model in after Mike Hall's filesystem package and API. See http://raa.ruby-lang.org/project/filesystem/ for more information. ---------------------------------------------------------------------- >Comment By: Daniel Berger (djberg96) Date: 2006-11-23 18:38 Message: This is partially complete. See the 'sys-filesystem' package, available as part of the sysutils project. - Dan ---------------------------------------------------------------------- You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=414&aid=1446&group_id=85 From noreply at rubyforge.org Thu Nov 23 20:38:47 2006 From: noreply at rubyforge.org (noreply at rubyforge.org) Date: Thu, 23 Nov 2006 20:38:47 -0500 (EST) Subject: [Win32utils-devel] [ win32utils-Feature Requests-1446 ] Add a & quot; filesystem& quot; interface Message-ID: <20061124013847.47D445241365@rubyforge.org> Feature Requests item #1446, was opened at 2005-02-04 20:14 You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=414&aid=1446&group_id=85 Category: None Group: None Status: Open Resolution: Accepted Priority: 3 Submitted By: Daniel Berger (djberg96) Assigned to: Daniel Berger (djberg96) >Summary: Add a &quot;filesystem&quot; interface Initial Comment: Create a package that allows users to get filesystem information (diskspace, etc). If possible, model in after Mike Hall's filesystem package and API. See http://raa.ruby-lang.org/project/filesystem/ for more information. ---------------------------------------------------------------------- Comment By: Daniel Berger (djberg96) Date: 2006-11-23 18:38 Message: This is partially complete. See the 'sys-filesystem' package, available as part of the sysutils project. - Dan ---------------------------------------------------------------------- You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=414&aid=1446&group_id=85