From dlutter at redhat.com Wed May 2 21:56:05 2007 From: dlutter at redhat.com (David Lutterkort) Date: Wed, 02 May 2007 18:56:05 -0700 Subject: [Rubygems-developers] Packaging gems as RPM's for Fedora Message-ID: <1178157365.4962.18.camel@galia.watzmann.net> Hi all, I've been in the (much delayed) process of writing up packaging guidelines [1] for Fedora, and they have been discussed with the Fedora Packaging Committee, which needs to approve them before we can create RPM packages from gems. There's a few things I'd like to get input from people who understand gems way better than I do. Apologies in advance for a somewhat lengthy post. The goal of this is to wrap Ruby gems as easily as possible in RPM's so that Fedora users can install them using their RPM-based tools (like yum), have them show up in the RPM database etc. At the same time, I want to avoid breaking any expectations Ruby developers have how gems work, e.g. 'gem list --local' should still work, RPM-packaged gems should be fully recognized by gem, so that if I install a gem 'foo' from an RPM, I can still use gem to install a gem 'bar' that depends on 'foo' and have the RPM-installed one satsify that dependency. Here then is a random list of issues I'd like to get some guidance on: Multilib ======== Ruby Gems doesn't have any provisions for multilib systems, i.e., systems that can run both 32-bit and 64-bit code. Multilib in general is a thorny issue, and the goal is to enable users to have 32-bit and 64-bit versions of libraries installed simultaneously. In the guidelines draft I wrote that arch specific code (e.g., DSO's) need to be moved from the gemdir into Config::CONFIG["sitearchdir"], the dirs set up by the ruby RPM for this purpose. Would it be possible for gems to support that natively, e.g., by sticking dirs like i386-linux and x86_64-linux into the lib/ dirs of a gem and installing arch specific content there ? Lack of LSB compliance ====================== Debian states [2] that Gems aren't LSB compliant (and that is unfortunately echoed on the rubygems website, though I couldn't find the reference just now) I think it's a bit of an overstatement; the only LSB violation with Gems that I can think of is that executables might wind up in /usr/lib, since Gems might contain commands in a bin/ directory. Really only a problem if files in those bin/ directories are actually arch-specific and not just ruby scripts. Is there any way to remedy that, e.g. by putting such commands into /usr/bin ? For real LSB purists, there are probably some other things they won't like about Gems, e.g. that documentation goes into the gem dir instead of /usr/share/doc, though I think that can be discussed away; fixing it would require a lot more configurability in Gems, and I doubt it's worth the effort. Gem and non-Gem versions of the same library ============================================ Since using a gem in your code requires that you do something like require 'rubygems' require 'mygem' there's a chance that people need the same library installed as a Gem and as a non-Gem. For Fedora, we'd very much like to avoid that there are two different packages with almost the same bits that install into two different locations to support this. I'm experimenting with putting symlinks to things in the gem dir into the standard ruby library path - I was wondering if there are any plans to support that with gems more directly. Generating RPM specfiles ======================== Since Gems carry almost all the information that is needed to build an RPM, I've hacked up a little script to generate a RPM specfile from a ruby gem [3]; this is only meant to help people write the initial RPM spec, after that they have to maintain it manually. I was wondering what you think of adding such a tool to rubygems directly (I think the PHP pear people do that, and it's generally liked a lot) For examples of what RPM specfiles look like for Rails, have a look at [4] David [1] http://fedoraproject.org/wiki/PackagingDrafts/RubyGems [2] http://pkg-ruby-extras.alioth.debian.org/rubygems.html [3] http://people.redhat.com/dlutter/gem2spec.html [4] http://people.redhat.com/dlutter/yum/spec/ From stephen at touset.org Wed May 2 23:14:23 2007 From: stephen at touset.org (Stephen Touset) Date: Wed, 02 May 2007 23:14:23 -0400 Subject: [Rubygems-developers] Packaging gems as RPM's for Fedora In-Reply-To: <1178157365.4962.18.camel@galia.watzmann.net> References: <1178157365.4962.18.camel@galia.watzmann.net> Message-ID: <4639538F.9070704@touset.org> David Lutterkort wrote: > Since Gems carry almost all the information that is needed to build an > RPM, I've hacked up a little script to generate a RPM specfile from a > ruby gem [3]; this is only meant to help people write the initial RPM > spec, after that they have to maintain it manually. I was wondering what > you think of adding such a tool to rubygems directly (I think the PHP > pear people do that, and it's generally liked a lot) I would be extremely interested in potentially merging this script with pallet, a multi-packaging library I'm writing that works well with Rakefiles. Right now I only support building Debian packages and Ruby gems from the same source, but I'm hoping to add RPM support in the near future (< 1 month) for a talk I'm doing at the local RUG. This would be a huge start. -- Stephen Touset From chad at chadfowler.com Thu May 3 15:33:06 2007 From: chad at chadfowler.com (Chad Fowler) Date: Thu, 3 May 2007 13:33:06 -0600 Subject: [Rubygems-developers] Do you guys have Tattle Reports wishlist? In-Reply-To: References: Message-ID: On 4/30/07, Gregory Brown wrote: > Hi Chad, Eric, Jim, et. al > > It's been a while since I said I'd work on this at MWRC, but with > Ruport 1.0 coming up around the corner, it's about time to get some > quality examples of what it can do.The tattle data is a nice, fairly > simple data set to work with, so it should be a good task to target. > > I've got a database dump, and I'm sure we can think up some reports, > but I'd be interested in producing whatever is most important to the > RubyGems developers. > > So if you have a wish list, let me know, and we'll see what we can do > with it in the next couple weeks. > > -greg > > PS: Chad, I've CCed Mike, who is in charge of our Rails integration, I > think it'll be easy to integrate reports right into the tattle rails > app, but Mike would be the one to go to on that. Howdy Greg. The big thing I think would help is to have some reports that show less granular views of platform. Maybe some kind of drill-down where you can see platform at the OS level (Windows, Linux, Mac) and then drill to see the details. Also, ruby version by platform and rubygems version by platform would be cool. Thanks, Chad From ryand-ruby at zenspider.com Fri May 4 03:50:57 2007 From: ryand-ruby at zenspider.com (Ryan Davis) Date: Fri, 4 May 2007 00:50:57 -0700 Subject: [Rubygems-developers] Reviewing the Tattle Data (was RubyGems plaform thread) In-Reply-To: <463263BB.4070103@sun.com> References: <462D6820.6070104@weirichhouse.org> <10937DF3-10C0-4500-8983-97CFD2F22D04@segment7.net> <462EB947.3090604@weirichhouse.org> <462FAB75.1010101@sun.com> <7D423F6D-15EE-42F8-8255-0891BEB70FE7@segment7.net> <4630EF7F.10201@sun.com> <67CA79BE-3BEC-4CA4-A30F-81D334761875@segment7.net> <463263BB.4070103@sun.com> Message-ID: <01EE4909-B89E-4EA8-A200-C5D4B361B390@zenspider.com> On Apr 27, 2007, at 13:57 , Charles Oliver Nutter wrote: > Eric Hodel wrote: >> If you find tattle is reporting information you don't want public >> after you've published it, you should have checked out what it was >> doing first and not run tattle. >> >> (While the first release of tattle didn't have the report option, it >> was still simple enough to visit the tattle report page or inspect >> the source to discover what was being reported.) > > If you didn't want a security hole in your system you should have read > all the code first. > > Sound a little silly? Actually, no. That is part of the appeal of open source in the first place, remember? Also... it is called TATTLE. If you're sensitive about your information, maybe applying a bit of critical thinking towards the name of the tool in the first place would be prudent. From gregory.t.brown at gmail.com Fri May 4 10:31:15 2007 From: gregory.t.brown at gmail.com (Gregory Brown) Date: Fri, 4 May 2007 10:31:15 -0400 Subject: [Rubygems-developers] Do you guys have Tattle Reports wishlist? In-Reply-To: References: Message-ID: On 5/3/07, Chad Fowler wrote: > The big thing I think would help is to have some reports that show > less granular views of platform. Maybe some kind of drill-down where > you can see platform at the OS level (Windows, Linux, Mac) and then > drill to see the details. Also, ruby version by platform and rubygems > version by platform would be cool. Cool. We're having an example building day on May 12th, and we'll try to see if we can get some of this stuff together for you. Anyone here is welcome to stop by #ruport that day and see what we're up to and make suggestions. -greg From gmiller at marketcetera.com Fri May 4 18:27:01 2007 From: gmiller at marketcetera.com (Graham Miller) Date: Fri, 4 May 2007 15:27:01 -0700 Subject: [Rubygems-developers] Building native gems for deployment Message-ID: <96c48fe10705041527g6b4e7d11m3a033eacaaad2b79@mail.gmail.com> Hey Patrick, I just got around to giving this a try. It's exactly what I needed! Thanks so much! graham On 4/18/07, rubygems-developers-request at rubyforge.org < rubygems-developers-request at rubyforge.org> wrote: > > > Message: 1 > Date: Wed, 18 Apr 2007 00:14:01 -0400 > From: "Patrick Hurley" > Subject: Re: [Rubygems-developers] Building native gems for deployment > To: rubygems-developers at rubyforge.org > Message-ID: > <554ac39c0704172114x14452e90pe5116a60f7d361b4 at mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 4/13/07, Graham Miller wrote: > > So I guess my question would be: what is the best way to install a gem > onto > > a machine without having to do the native compilation step? > > I had a similar problem, with very little testing (so far), try this > script, if you have any problems let me know. > > pth > > --------------------------------------------- > #!/usr/bin/env ruby -w > > require "rbconfig" > require "rubygems" > require "pp" > require "tmpdir" > require "find" > require "fileutils" > > Gem.manage_gems > unless gem = ARGV.first > puts "You must provide the name of a gem on the command line." > exit 1 > end > > gi = Gem::Installer.new(gem) > format = Gem::Format.from_file_by_path(gem) > > dir = File.join(Dir.tmpdir, "gembuilder") > FileUtils.rm_r(dir) rescue nil > > puts "Unpacking gem" > gi.unpack(dir) > > gi.build_extensions(dir, format.spec) > files = [] > Find.find(dir) do |fname| > next if fname == dir > files << fname.sub(Regexp.quote(dir+"/"),'') > end > > spec = format.spec > spec.extensions = [] > spec.files += (files - format.spec.files) > spec.platform = Config::CONFIG['arch'].sub(/[\.0-9]*$/, '') > > puts "Building gem in #{dir}" > start_dir = Dir.pwd > Dir.chdir(dir) do > gb = Gem::Builder.new(spec) > gb.build > FileUtils.mv Dir.glob("*.gem"), start_dir > end > > puts "Cleaning up #{dir}" > FileUtils.rm_rf(dir) > > -- Marketcetera Trading Platform download.run.trade. www.marketcetera.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/rubygems-developers/attachments/20070504/2ee69d8f/attachment-0001.html From phurley at gmail.com Fri May 4 23:47:03 2007 From: phurley at gmail.com (Patrick Hurley) Date: Fri, 4 May 2007 23:47:03 -0400 Subject: [Rubygems-developers] Building native gems for deployment In-Reply-To: <96c48fe10705041527g6b4e7d11m3a033eacaaad2b79@mail.gmail.com> References: <96c48fe10705041527g6b4e7d11m3a033eacaaad2b79@mail.gmail.com> Message-ID: <554ac39c0705042047o4881e2x872835d32b6b14fb@mail.gmail.com> On 5/4/07, Graham Miller wrote: > Hey Patrick, > I just got around to giving this a try. It's exactly what I needed! Thanks so much! > Glad to hear it. This has been invaluable to me as well -- this is admittedly an "enterprisy" sort of need. Is there any interest in having this feature integrated into the gem command? Or otherwise cleaned up for inclusion into the gem package? I would be happy to do so if there was a reasonable chance it would be integrated into the standard package. In case the context was lost -- I am referring to a script/command that will take an existing source gem with a binary extension and build a platform specific binary gem. This is most useful in building a platform specific binary gem (i.e. for a specific Linux distribution) to be installed on a production machine that does not have a build chain. pth From chad at chadfowler.com Sun May 6 03:49:50 2007 From: chad at chadfowler.com (Chad Fowler) Date: Sun, 6 May 2007 08:49:50 +0100 Subject: [Rubygems-developers] Building native gems for deployment In-Reply-To: <554ac39c0705042047o4881e2x872835d32b6b14fb@mail.gmail.com> References: <96c48fe10705041527g6b4e7d11m3a033eacaaad2b79@mail.gmail.com> <554ac39c0705042047o4881e2x872835d32b6b14fb@mail.gmail.com> Message-ID: On 5/5/07, Patrick Hurley wrote: > On 5/4/07, Graham Miller wrote: > > Hey Patrick, > > I just got around to giving this a try. It's exactly what I needed! Thanks so much! > > > > Glad to hear it. This has been invaluable to me as well -- this is > admittedly an "enterprisy" sort of need. Is there any interest in > having this feature integrated into the gem command? Or otherwise > cleaned up for inclusion into the gem package? > > I would be happy to do so if there was a reasonable chance it would be > integrated into the standard package. > > In case the context was lost -- I am referring to a script/command > that will take an existing source gem with a binary extension and > build a platform specific binary gem. This is most useful in building > a platform specific binary gem (i.e. for a specific Linux > distribution) to be installed on a production machine that does not > have a build chain. > Hi Patrick. I think this is cool as well. Perhaps you could package it up as a separate gem and release via RubyForge? Chad From charles.nutter at sun.com Sun May 6 19:34:27 2007 From: charles.nutter at sun.com (Charles Oliver Nutter) Date: Sun, 06 May 2007 16:34:27 -0700 Subject: [Rubygems-developers] Reviewing the Tattle Data (was RubyGems plaform thread) In-Reply-To: <01EE4909-B89E-4EA8-A200-C5D4B361B390@zenspider.com> References: <462D6820.6070104@weirichhouse.org> <10937DF3-10C0-4500-8983-97CFD2F22D04@segment7.net> <462EB947.3090604@weirichhouse.org> <462FAB75.1010101@sun.com> <7D423F6D-15EE-42F8-8255-0891BEB70FE7@segment7.net> <4630EF7F.10201@sun.com> <67CA79BE-3BEC-4CA4-A30F-81D334761875@segment7.net> <463263BB.4070103@sun.com> <01EE4909-B89E-4EA8-A200-C5D4B361B390@zenspider.com> Message-ID: <463E6603.6060704@sun.com> Ryan Davis wrote: > On Apr 27, 2007, at 13:57 , Charles Oliver Nutter wrote: > >> Eric Hodel wrote: >>> If you find tattle is reporting information you don't want public >>> after you've published it, you should have checked out what it was >>> doing first and not run tattle. >>> >>> (While the first release of tattle didn't have the report option, it >>> was still simple enough to visit the tattle report page or inspect >>> the source to discover what was being reported.) >> If you didn't want a security hole in your system you should have read >> all the code first. >> >> Sound a little silly? > > Actually, no. That is part of the appeal of open source in the first > place, remember? The appeal of open source is that you *can* look at the source, not that you *have to*. > > Also... it is called TATTLE. If you're sensitive about your > information, maybe applying a bit of critical thinking towards the > name of the tool in the first place would be prudent. I'm not the only one that felt this way, and the information has been removed from the site. So I think it's a moot point to labor this point anymore. - Charlie From thewoolleyman at gmail.com Mon May 7 15:01:49 2007 From: thewoolleyman at gmail.com (Chad Woolley) Date: Mon, 7 May 2007 12:01:49 -0700 Subject: [Rubygems-developers] Error when remotely listing ActiveRecord-JDBC with RubyGems 0.9.2 In-Reply-To: References: Message-ID: Just providing closure on this. I got no response, here or on the jruby list, so I monkey-patched rubygems to do an exact name match for 'gem list'. That fixed my problem, since all I care about is making it work when called programmatically from my app. I still think it's kind of an issue if "gem list activerecord --remote" throws an exception, but oh well :) -- Chad W. On 4/30/07, Chad Woolley wrote: > Hi, > > I think this is actually a somewhat serious problem - anyone who tries > to do a remote gem list on 'activerecord' with RubGems 0.9.2+ will get > an exception. > > I've got a thread going on the Jruby and jruby-extra lists to have > them look into it, but I'd like to fix the related problem in > RubyGems. Even if Jruby fixes this particular gem, it could still > theoretically happen with another gem in the future - my upcoming > rails-but-even-l337er gem, for example. (that was a joke) > > The real problem that I see is the wildcard matching of the remote gem > list command, which cannot currently be overridden. If it could, then > I could just request an exact match like /^activerecord$/, and not > even pick up the ActiveRecord-jdbc gem. > > So, I could hack this in by making a change to the > Gem::Commands::ListCommand#execute method, and trying to make it > "detect" whether some regexp was passed as the argument. It currently > matches /^#{string}/, which is the wildcard matching at the root of > the problem. > > However, fixing this proved to be tricky to do without breaking any of > the existing behavior. Should the gem list search term be the > forward-slash-delimited regexp style? If so, how to detect that? > Eval'ing and checking the type could cause a syntax error if the eval > rails. Passing it to Regexp.compile to see if it compiles would > require stripping the slash delimiters (and any other trailing flags). > > There could also be a separate command-line flag to the 'gem list' > command to specify and exact name match instead of a wildcard - > perhaps '--exact-match'? > > Alternately, we could fix whatever it was that allowed this invalid > gem to be uploaded and indexed on the rubygems mirrors, but I have no > idea where to start on that one. I looked at the ActiveRecord-JDBC > gem source, and I see nothing suspicious - it's just a standard Hoe > config. Not even any mention of YAML::Private anywhere. > > So, to summarize, I'd be glad to help fix this, but I'd appreciate > some direction from the devs so I don't waste time going down the > wrong path. > > Thanks, > Chad > > On 4/28/07, Chad Woolley wrote: > > Hi, > > > > I get the following error when listing ActiveRecord-JDBC remotely with > > RubyGems 0.9.2. It happens on OSX and Linux, but not windows. It > > doesn't happen on RubyGems 0.9.0. There also looks like there's a > > related bug which is open: > > > > http://rubyforge.org/tracker/index.php?func=detail&aid=8359&group_id=126&atid=575 > > > > Here's the error. It also occurs when using plain 'activerecord' as > > the search string, but I think this is because ActiveRecord-JDBC is > > matching the substring, because you can see that activerecord-mimer > > does not get an error: > > > > /> gem --version > > 0.9.2 > > /> gem list ActiveRecord-JDBC --remote --debug --backtrace > > Exception `Errno::ENOENT' at > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/config_file.rb:51 - No such > > file or directory - /Users/pivotal/.gemrc > > > > *** REMOTE GEMS *** > > Exception `NoMethodError' at > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/version.rb:136 - private > > method `scan' called for # > > ERROR: While executing gem ... (NoMethodError) > > private method `scan' called for # > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/version.rb:136:in `to_ints' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/version.rb:147:in `<=>' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/version.rb:198 > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/version.rb:292:in `satisfy?' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/version.rb:283:in `satisfied_by?' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `all?' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/version.rb:283:in `satisfied_by?' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:210:in `search' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:208:in `search' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/source_info_cache.rb:109:in `search' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `map' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/source_info_cache.rb:108:in `search' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/source_info_cache.rb:49:in `search' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/gem_commands.rb:806:in `execute' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/gem_commands.rb:886:in `execute' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/command.rb:70:in `invoke' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/cmd_manager.rb:120:in `process_args' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/cmd_manager.rb:91:in `run' > > /usr/local/lib/ruby/site_ruby/1.8/rubygems/gem_runner.rb:30:in `run' > > /usr/local/bin/gem:23 > > /> gem list activerecord-mimer --remote > > > > *** REMOTE GEMS *** > > > > > > activerecord-mimer (0.0.4, 0.0.3, 0.0.2, 0.0.1) > > Mimer support for ActiveRecord. > > /> > > > From phurley at gmail.com Mon May 7 16:20:52 2007 From: phurley at gmail.com (Patrick Hurley) Date: Mon, 7 May 2007 16:20:52 -0400 Subject: [Rubygems-developers] Building native gems for deployment In-Reply-To: References: <96c48fe10705041527g6b4e7d11m3a033eacaaad2b79@mail.gmail.com> <554ac39c0705042047o4881e2x872835d32b6b14fb@mail.gmail.com> Message-ID: <554ac39c0705071320h439ee811ra67feb97e26c3ca3@mail.gmail.com> On 5/6/07, Chad Fowler wrote: > Hi Patrick. I think this is cool as well. Perhaps you could package > it up as a separate gem and release via RubyForge? Will do in the next few days. Thanks as always for all the great tools pth From jim.weirich at gmail.com Mon May 7 16:44:30 2007 From: jim.weirich at gmail.com (Jim Weirich) Date: Mon, 7 May 2007 16:44:30 -0400 Subject: [Rubygems-developers] Error when remotely listing ActiveRecord-JDBC with RubyGems 0.9.2 In-Reply-To: References: Message-ID: On 5/7/07, Chad Woolley wrote: > Just providing closure on this. I got no response, here or on the > jruby list, so I monkey-patched rubygems to do an exact name match for > 'gem list'. That fixed my problem, since all I care about is making > it work when called programmatically from my app. > > I still think it's kind of an issue if "gem list activerecord > --remote" throws an exception, but oh well :) Chad, Sorry for the late response ... things have finally slowed down a bit here. I've tried this on the platforms I have available (MacOS, Ubuntu) and don't see the same problem. By the error trace it seems that the program is aborting in the middle of the version comparison software (to_ints converts "1.2.3" to [1, 2, 3]). Since the trace complains about a YAML object, I'm suspecting that the version data is somehow bogus. The YAML data comes from the local source index cache, so maybe your source index cache is corrupted. Try this: (1) Save your current source index cache somewhere. (2) Delete current source index cache. (3) Rerun gem list --remote (it will automatically regenerate the cache). If the problem goes away, then we can blame the cache. At that point, I would like to see the file, so send me the file you saved in step 1 above). If the problem persists, then it is something else ... and i'll have to think harder. -- Jim Weirich -- -- Jim Weirich jim at weirichhouse.org http://onestepback.org ----------------------------------------------------------------- "Beware of bugs in the above code; I have only proved it correct, not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas) From thewoolleyman at gmail.com Mon May 7 20:40:19 2007 From: thewoolleyman at gmail.com (Chad Woolley) Date: Mon, 7 May 2007 17:40:19 -0700 Subject: [Rubygems-developers] Error when remotely listing ActiveRecord-JDBC with RubyGems 0.9.2 In-Reply-To: References: Message-ID: On 5/7/07, Jim Weirich wrote: > On 5/7/07, Chad Woolley wrote: > > Just providing closure on this. I got no response, here or on the > > jruby list, so I monkey-patched rubygems to do an exact name match for > > 'gem list'. That fixed my problem, since all I care about is making > > it work when called programmatically from my app. > > > > I still think it's kind of an issue if "gem list activerecord > > --remote" throws an exception, but oh well :) > > Chad, > > Sorry for the late response ... things have finally slowed down a bit here. > > I've tried this on the platforms I have available (MacOS, Ubuntu) and > don't see the same problem. > > By the error trace it seems that the program is aborting in the middle > of the version comparison software (to_ints converts "1.2.3" to [1, 2, > 3]). Since the trace complains about a YAML object, I'm suspecting > that the version data is somehow bogus. The YAML data comes from the > local source index cache, so maybe your source index cache is > corrupted. Try this: > > (1) Save your current source index cache somewhere. > (2) Delete current source index cache. > (3) Rerun gem list --remote (it will automatically regenerate the cache). > > If the problem goes away, then we can blame the cache. At that point, > I would like to see the file, so send me the file you saved in step 1 > above). > > If the problem persists, then it is something else ... and i'll have > to think harder. > > -- Jim Weirich Hi Jim, I know I did those steps on at least one of the machines (blow away source index), and the problem persisted. I'll try again though, and send you the failing file. Thanks, Chad From thewoolleyman at gmail.com Mon May 7 22:16:16 2007 From: thewoolleyman at gmail.com (Chad Woolley) Date: Mon, 7 May 2007 19:16:16 -0700 Subject: [Rubygems-developers] Error when remotely listing ActiveRecord-JDBC with RubyGems 0.9.2 In-Reply-To: References: Message-ID: On 5/7/07, Jim Weirich wrote: > Chad, > > Sorry for the late response ... things have finally slowed down a bit here. > > I've tried this on the platforms I have available (MacOS, Ubuntu) and > don't see the same problem. > > By the error trace it seems that the program is aborting in the middle > of the version comparison software (to_ints converts "1.2.3" to [1, 2, > 3]). Since the trace complains about a YAML object, I'm suspecting > that the version data is somehow bogus. The YAML data comes from the > local source index cache, so maybe your source index cache is > corrupted. Try this: > > (1) Save your current source index cache somewhere. > (2) Delete current source index cache. > (3) Rerun gem list --remote (it will automatically regenerate the cache). > > If the problem goes away, then we can blame the cache. At that point, > I would like to see the file, so send me the file you saved in step 1 > above). > > If the problem persists, then it is something else ... and i'll have > to think harder. > > -- Jim Weirich OK, I just reproduced this on my mac and two linux boxes, all rubygems 0.9.2. It did not occur on linux with rubygems 0.8.11. One linux is ruby 1.8.4, other linux and mac are 1.8.5. Strangely, it was NOT occurring on the two linux boxes until I deleted my source_cache. Then, it started happening again after I deleted it. So, I can recreate the problem without any source_cache at all (list --remote doesn't create it), on three different systems. chadmac:/ woolley$ rm /usr/local/lib/ruby/gems/1.8/source_cache rm: /usr/local/lib/ruby/gems/1.8/source_cache: No such file or directory chadmac:/ woolley$ gem list activerecord --remote *** REMOTE GEMS *** ERROR: While executing gem ... (NoMethodError) private method `scan' called for # I have the source_cache from my mac, where it was occurring even before I deleted the file, but is that still relevant, since I can recreate it without any source_cache at all? Thanks, Chad From jim at weirichhouse.org Mon May 7 22:58:55 2007 From: jim at weirichhouse.org (Jim Weirich) Date: Mon, 07 May 2007 22:58:55 -0400 Subject: [Rubygems-developers] Error when remotely listing ActiveRecord-JDBC with RubyGems 0.9.2 In-Reply-To: References: Message-ID: <463FE76F.6040806@weirichhouse.org> Chad Woolley wrote: > chadmac:/ woolley$ rm /usr/local/lib/ruby/gems/1.8/source_cache > rm: /usr/local/lib/ruby/gems/1.8/source_cache: No such file or directory > chadmac:/ woolley$ gem list activerecord --remote > > *** REMOTE GEMS *** > ERROR: While executing gem ... (NoMethodError) > private method `scan' called for # > > I have the source_cache from my mac, where it was occurring even > before I deleted the file, but is that still relevant, since I can > recreate it without any source_cache at all? Thanks Chad. Hmmmm ... you blow away the system cache in /usr/local, but what about the user cache, probably stored in $HOME/.gem/source_cache? RubyGems will prefer to use the /usr/local version, but if it is protected and you are not running as root, then it will use the user writable version version in $HOME/.gem. The fact that it didn't say it was doing a bulk download indicates that the user cache was still present. Try again and blow both caches away. Sorry, I should have been more explicit. Note to self ... perhaps the gem command needs a clear cache subcommand? -- -- Jim Weirich jim at weirichhouse.org http://onestepback.org -- In theory, practice and theory are the same. -- In practice, they are different. From thewoolleyman at gmail.com Mon May 7 23:50:31 2007 From: thewoolleyman at gmail.com (Chad Woolley) Date: Mon, 7 May 2007 20:50:31 -0700 Subject: [Rubygems-developers] Error when remotely listing ActiveRecord-JDBC with RubyGems 0.9.2 In-Reply-To: <463FE76F.6040806@weirichhouse.org> References: <463FE76F.6040806@weirichhouse.org> Message-ID: On 5/7/07, Jim Weirich wrote: > > Try again and blow both caches away. Jim, Thanks for looking at this. Deleting ~/.gem/source_cache did make the problem go away. It comes back when I do a remote list against a non-mirror gem server I set up for my company. Indeed, even a list of all gems on this server causes the same error, so I think the ActiveRecord-JDBC thing is a red herring - it just happens to be the first one in alphabetical order. The problem is probably with the yaml index on our server, which doesn't surprise me, because I am generating it with a hacked index_gem_repository command. Another co-worker reported this same problem with his personal machine, in a thread on April 11th, titled "index_gem_repository problem". Basically, the command fails on some gems: http://rubyforge.org/pipermail/rubygems-developers/2007-March/002632.html I get the same errors Steve described in that thread, and I'm even running against a different server. I'm hitting rsync://rubyforge.rubyuser.de/gems (which I was told about in a private email before that webpage went up), and Steve is hitting the one on the webpage (rubyforge.lauschmusik.de/gems/). The instructions for this are from here: http://rubyforge.org/docman/view.php/5/231/mirror_setup.html So, I think the root cause is something with mirroring gems via rsync for a non-public server, which seems to cause problems with index_gem_repository, forcing us to hack it and result in a seemingly buggy yaml file. I will send you the address of our private server's yaml file via personal mail, so you can check it out. I'm still very willing to help debug this issue, because we want to have a working private gem mirror. Thanks, Chad From gregory.t.brown at gmail.com Fri May 11 11:15:37 2007 From: gregory.t.brown at gmail.com (Gregory Brown) Date: Fri, 11 May 2007 11:15:37 -0400 Subject: [Rubygems-developers] Is the Tattle Rails app on RubyForge up to date? Message-ID: Mike and I will be working on some Ruport based reports against the tattle data, and might even work on integrating them into the tattle rails app. Is the code on RubyForge up to date? If so, we'll patch against that. -greg From chad at chadfowler.com Fri May 11 11:52:00 2007 From: chad at chadfowler.com (Chad Fowler) Date: Fri, 11 May 2007 16:52:00 +0100 Subject: [Rubygems-developers] Is the Tattle Rails app on RubyForge up to date? In-Reply-To: References: Message-ID: On 5/11/07, Gregory Brown wrote: > Mike and I will be working on some Ruport based reports against the > tattle data, and might even work on integrating them into the tattle > rails app. > > Is the code on RubyForge up to date? > > If so, we'll patch against that. Yep. I deploy Tattle from subversion, so no chance of getting things out of sync. Chad From trevor at corevx.com Thu May 17 16:51:18 2007 From: trevor at corevx.com (Trevor Wennblom) Date: Thu, 17 May 2007 15:51:18 -0500 Subject: [Rubygems-developers] gem_server - clean urls Message-ID: <67534D8C-F53E-455D-BD78-AE637EB36A88@corevx.com> I'm wondering if there's a way to use gem_server with nice looking URLs. Right now I have in my Apache config: ProxyRequests Off ProxyPass /gemlist http://foo.com:8808/ ProxyPassReverse /gemlist http://foo.com:8808/ Which works great except the generated links assume they're at the top-level. For instance: a href="/doc_root/camping-1.5/rdoc/index.html" Whereas I'd need: a href="/gemlist/doc_root/camping-1.5/rdoc/index.html" Any clever ideas? Trevor From ryand-ruby at zenspider.com Tue May 22 05:21:54 2007 From: ryand-ruby at zenspider.com (Ryan Davis) Date: Tue, 22 May 2007 02:21:54 -0700 Subject: [Rubygems-developers] backwards compatibility lost? Message-ID: <6BECAE6E-E8EA-4149-BC19-D8F1A0D9B0BF@zenspider.com> What is the current equivalent of: begin require 'rubygems' require 'rubygems/loadpath_manager' Gem::LoadPathManager.build_paths paths.push(*Gem::LoadPathManager.paths) rescue LoadError # do nothing end This worked under 0.9.0.9 (read: not too long ago) and the changelogs don't seem to give any relevant hints. Trunk seems a lot more opaque to me than the previous design (took me all of 5 minutes to come up with the above). Should we at least provide some deprecation warnings and some level of backwards compatibility for a while? From ryand-ruby at zenspider.com Tue May 22 05:44:22 2007 From: ryand-ruby at zenspider.com (Ryan Davis) Date: Tue, 22 May 2007 02:44:22 -0700 Subject: [Rubygems-developers] backwards compatibility lost? In-Reply-To: <6BECAE6E-E8EA-4149-BC19-D8F1A0D9B0BF@zenspider.com> References: <6BECAE6E-E8EA-4149-BC19-D8F1A0D9B0BF@zenspider.com> Message-ID: On May 22, 2007, at 02:21 , Ryan Davis wrote: > What is the current equivalent of: > > begin > require 'rubygems' > require 'rubygems/loadpath_manager' > Gem::LoadPathManager.build_paths > paths.push(*Gem::LoadPathManager.paths) > rescue LoadError > # do nothing > end This seems to be the correct way to go: begin require 'rubygems' paths.push(*Gem.latest_load_paths) rescue LoadError => e # do nothing end Maybe I was playing too deep? From drbrain at segment7.net Wed May 23 04:09:24 2007 From: drbrain at segment7.net (Eric Hodel) Date: Wed, 23 May 2007 01:09:24 -0700 Subject: [Rubygems-developers] SourceInfoCache is self-repairing Message-ID: <40029193-E940-4AE1-B6C8-746943E4069F@segment7.net> I checked in code to let SourceInfoCache repair itself if it encounters a bad cache file. I think the root cause is gem repositories containing old index scripts. If the cache is irreparable (I haven't seen any in the wild) then it gets nuked and forcibly refetched (which I've marked with a hack, since I'm too tired to puzzle out the dense tangle of code in SIC at this late hour). From jim at weirichhouse.org Wed May 23 11:55:37 2007 From: jim at weirichhouse.org (Jim Weirich) Date: Wed, 23 May 2007 11:55:37 -0400 Subject: [Rubygems-developers] Ready for 0.9.4 release? Message-ID: <465463F9.6070508@weirichhouse.org> Along with Eric's self-healing SIC changes, I have: * Added a "gem sources --clear-all" command to delete the source file caches (removing them manually is too error prone ... hopefully with Eric's fixes, this will not be needed so much). * Fixed a warning in the tests. * Refactored SourceInfoCache to report the cache file names without loading (and syncing) the source index cache data. With these changes, I think we are ready for a 0.9.4 release. I'll do that this afternoon if there are no objections. -- -- Jim Weirich jim at weirichhouse.org http://onestepback.org -- In theory, practice and theory are the same. -- In practice, they are different. From jim at weirichhouse.org Wed May 23 17:02:44 2007 From: jim at weirichhouse.org (Jim Weirich) Date: Wed, 23 May 2007 17:02:44 -0400 Subject: [Rubygems-developers] Ready for 0.9.4 release? In-Reply-To: <465463F9.6070508@weirichhouse.org> References: <465463F9.6070508@weirichhouse.org> Message-ID: <4654ABF4.3050307@weirichhouse.org> Jim Weirich wrote: > Along with Eric's self-healing SIC changes, I have: > > * Added a "gem sources --clear-all" command to delete the source > file caches (removing them manually is too error prone ... > hopefully with Eric's fixes, this will not be needed so much). > > * Fixed a warning in the tests. > > * Refactored SourceInfoCache to report the cache file names without > loading (and syncing) the source index cache data. > > With these changes, I think we are ready for a 0.9.4 release. I'll > do that this afternoon if there are no objections. > Its done. -- -- Jim Weirich From drnicwilliams at gmail.com Thu May 24 13:02:59 2007 From: drnicwilliams at gmail.com (Nic Williams) Date: Thu, 24 May 2007 19:02:59 +0200 Subject: [Rubygems-developers] Ready for 0.9.4 release? In-Reply-To: <4654ABF4.3050307@weirichhouse.org> References: <465463F9.6070508@weirichhouse.org> <4654ABF4.3050307@weirichhouse.org> Message-ID: <44b555bb0705241002i40b6fe0ake3f391ff19d5b1ee@mail.gmail.com> When did Gem::UnpackCommand get deprecated from RubyGems? What has it been replaced with? In which version did it disappear? Cheers Nic On 5/23/07, Jim Weirich wrote: > Jim Weirich wrote: > > Along with Eric's self-healing SIC changes, I have: > > > > * Added a "gem sources --clear-all" command to delete the source > > file caches (removing them manually is too error prone ... > > hopefully with Eric's fixes, this will not be needed so much). > > > > * Fixed a warning in the tests. > > > > * Refactored SourceInfoCache to report the cache file names without > > loading (and syncing) the source index cache data. > > > > With these changes, I think we are ready for a 0.9.4 release. I'll > > do that this afternoon if there are no objections. > > > > Its done. > > -- > -- Jim Weirich > _______________________________________________ > Rubygems-developers mailing list > Rubygems-developers at rubyforge.org > http://rubyforge.org/mailman/listinfo/rubygems-developers > -- Dr Nic Williams http://myconfplan.com - plan the next conference you go to http://www.drnicwilliams.com - Ruby/Rails/Javascript/Web2.0 skype: nicwilliams (p) +61 7 3102 3237 (Finds me anywhere in the world, via Skype) (m) +4673 681 5093 (Swedish mobile) (f) +61 7 3305 7572 (sends fax to my email) Bj?rnsonsgatan 153, 16 844 Bromma, Sweden From Daniel.Berger at qwest.com Thu May 24 16:49:47 2007 From: Daniel.Berger at qwest.com (Daniel Berger) Date: Thu, 24 May 2007 14:49:47 -0600 Subject: [Rubygems-developers] gem update --system, heavy ram and cpu consumption Message-ID: <4655FA6B.7050609@qwest.com> Ruby 1.8.6 Solaris 10 Rubygems 0.9.3 (upgrading to 0.9.4) I recently ran "sudo gem update --system", and this was the result. It started at about 45mb and ended up here. This seems rather extreme to me: PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND 12691 root 2 10 0 175M 161M run 1:31 94.11% ruby On my poor little Sunblade 150 with 512mb RAM this was painful. Any way to improve this? Thanks, Dan From drbrain at segment7.net Thu May 24 19:19:44 2007 From: drbrain at segment7.net (Eric Hodel) Date: Thu, 24 May 2007 16:19:44 -0700 Subject: [Rubygems-developers] gem update --system, heavy ram and cpu consumption In-Reply-To: <4655FA6B.7050609@qwest.com> References: <4655FA6B.7050609@qwest.com> Message-ID: <1996CD69-1EEC-4EA7-B0B6-C43AA3E2DF7C@segment7.net> On May 24, 2007, at 13:49, Daniel Berger wrote: > Ruby 1.8.6 > Solaris 10 > Rubygems 0.9.3 (upgrading to 0.9.4) > > I recently ran "sudo gem update --system", and this was the result. It > started at about 45mb and ended up here. This seems rather extreme > to me: > > PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND > 12691 root 2 10 0 175M 161M run 1:31 94.11% ruby > > On my poor little Sunblade 150 with 512mb RAM this was painful. (Build ruby's ri index on a Soekris net4801, takes about a day of wall-clock time.) > Any way to improve this? This seems to come from downloading and extracting a bulk index. I see similar large growth from: gem sources -c gem list -r nosuchgem Jim and I were talking about trying incremental upload all the time, rather than bulk at RailsConf... From jim.weirich at gmail.com Fri May 25 10:42:13 2007 From: jim.weirich at gmail.com (Jim Weirich) Date: Fri, 25 May 2007 10:42:13 -0400 Subject: [Rubygems-developers] Ready for 0.9.4 release? In-Reply-To: <44b555bb0705241002i40b6fe0ake3f391ff19d5b1ee@mail.gmail.com> References: <465463F9.6070508@weirichhouse.org> <4654ABF4.3050307@weirichhouse.org> <44b555bb0705241002i40b6fe0ake3f391ff19d5b1ee@mail.gmail.com> Message-ID: On 5/24/07, Nic Williams wrote: > When did Gem::UnpackCommand get deprecated from RubyGems? What has it > been replaced with? In which version did it disappear? It seems to be working for me: $ gem --version 0.9.4 $ gem unpack rake Unpacked gem: 'rake-0.7.3.1' -- -- Jim Weirich jim at weirichhouse.org http://onestepback.org ----------------------------------------------------------------- "Beware of bugs in the above code; I have only proved it correct, not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas) From hgs at dmu.ac.uk Fri May 25 10:51:34 2007 From: hgs at dmu.ac.uk (Hugh Sasse) Date: Fri, 25 May 2007 15:51:34 +0100 (WEST) Subject: [Rubygems-developers] gem update --system, heavy ram and cpu consumption In-Reply-To: <1996CD69-1EEC-4EA7-B0B6-C43AA3E2DF7C@segment7.net> References: <4655FA6B.7050609@qwest.com> <1996CD69-1EEC-4EA7-B0B6-C43AA3E2DF7C@segment7.net> Message-ID: On Thu, 24 May 2007, Eric Hodel wrote: > On May 24, 2007, at 13:49, Daniel Berger wrote: > > > Ruby 1.8.6 > > Solaris 10 > > Rubygems 0.9.3 (upgrading to 0.9.4) > > > > I recently ran "sudo gem update --system", and this was the result. It > > started at about 45mb and ended up here. This seems rather extreme > > to me: > > > > PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND > > 12691 root 2 10 0 175M 161M run 1:31 94.11% ruby > > > > On my poor little Sunblade 150 with 512mb RAM this was painful. > > (Build ruby's ri index on a Soekris net4801, takes about a day of > wall-clock time.) It failed on one of our alpha servers. > > > Any way to improve this? > > This seems to come from downloading and extracting a bulk index. I > see similar large growth from: There is a Etag for this document . What about using VCDIFF to shrink the size? RFC3284. Ruby's http module doesn't support that out of the box. To get that into http see RFC3229. I can see evidence for discussion of RFC3229, but nothing from Apache saying they finally decided to support it. One non-apache source says 2.2 does this. Hugh From drbrain at segment7.net Fri May 25 18:51:20 2007 From: drbrain at segment7.net (Eric Hodel) Date: Fri, 25 May 2007 15:51:20 -0700 Subject: [Rubygems-developers] gem update --system, heavy ram and cpu consumption In-Reply-To: References: <4655FA6B.7050609@qwest.com> <1996CD69-1EEC-4EA7-B0B6-C43AA3E2DF7C@segment7.net> Message-ID: <6413CA2A-1854-4569-9A49-2DE1EC31EF84@segment7.net> On May 25, 2007, at 07:51, Hugh Sasse wrote: > On Thu, 24 May 2007, Eric Hodel wrote: >> On May 24, 2007, at 13:49, Daniel Berger wrote: >>> Any way to improve this? >> >> This seems to come from downloading and extracting a bulk index. I >> see similar large growth from: > > There is a Etag for this document . What about using VCDIFF to > shrink the size? RFC3284. Ruby's http module doesn't support that > out of the box. To get that into http see RFC3229. > > I can see evidence for discussion of RFC3229, but nothing from > Apache saying they finally decided to support it. One non-apache > source says 2.2 does this. Rather than writing all that code, we could remove one (maybe two) lines for the solution Jim and I discussed (that was snipped). From Daniel.Berger at qwest.com Sun May 27 11:35:54 2007 From: Daniel.Berger at qwest.com (Berger, Daniel) Date: Sun, 27 May 2007 10:35:54 -0500 Subject: [Rubygems-developers] Odd gem error on gem update --system Message-ID: <7524A45A1A5B264FA4809E2156496CFBE72C30@ITOMAE2KM01.AD.QINTRA.COM> Saw this at the end of a "gem update --system" on OS X 10.4.9 with Ruby 1.8.6: hook /usr/local/lib/ruby/gems/1.8/gems/rubygems-update-0.9.4/./post-install.r b failed: undefined method `file_name' for # Try 'ruby setup.rb --help' for detailed usage. RubyGems system software updated daniel-bergers-computer:~/Documents/workspace/ruby_test djberge$ gem -v 0.9.4 Everything seems to be ok, though. I saw a couple mentions of this error on Google, but no definitive explanation. Should I be worried? Regards, Dan This communication is the property of Qwest and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments. From ryand-ruby at zenspider.com Sun May 27 20:18:15 2007 From: ryand-ruby at zenspider.com (Ryan Davis) Date: Sun, 27 May 2007 17:18:15 -0700 Subject: [Rubygems-developers] gem update --system, heavy ram and cpu consumption In-Reply-To: <1996CD69-1EEC-4EA7-B0B6-C43AA3E2DF7C@segment7.net> References: <4655FA6B.7050609@qwest.com> <1996CD69-1EEC-4EA7-B0B6-C43AA3E2DF7C@segment7.net> Message-ID: <0B05280D-85B7-4332-85ED-AB4AB9C23DED@zenspider.com> On May 24, 2007, at 16:19 , Eric Hodel wrote: > Jim and I were talking about trying incremental upload all the time, > rather than bulk at RailsConf... How crazy would it be to incorporate librsync into ruby and/or rubygems? From jim.weirich at gmail.com Sun May 27 20:23:24 2007 From: jim.weirich at gmail.com (Jim Weirich) Date: Sun, 27 May 2007 20:23:24 -0400 Subject: [Rubygems-developers] gem update --system, heavy ram and cpu consumption In-Reply-To: <0B05280D-85B7-4332-85ED-AB4AB9C23DED@zenspider.com> References: <4655FA6B.7050609@qwest.com> <1996CD69-1EEC-4EA7-B0B6-C43AA3E2DF7C@segment7.net> <0B05280D-85B7-4332-85ED-AB4AB9C23DED@zenspider.com> Message-ID: On 5/27/07, Ryan Davis wrote: > > On May 24, 2007, at 16:19 , Eric Hodel wrote: > > > Jim and I were talking about trying incremental upload all the time, > > rather than bulk at RailsConf... > > How crazy would it be to incorporate librsync into ruby and/or rubygems? I don't know how crazy. Is it commonly available (on all platforms)? -- -- Jim Weirich jim at weirichhouse.org http://onestepback.org ----------------------------------------------------------------- "Beware of bugs in the above code; I have only proved it correct, not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas) From djberg96 at gmail.com Sun May 27 20:34:53 2007 From: djberg96 at gmail.com (Daniel Berger) Date: Sun, 27 May 2007 18:34:53 -0600 Subject: [Rubygems-developers] gem update --system, heavy ram and cpu consumption In-Reply-To: References: <4655FA6B.7050609@qwest.com> <1996CD69-1EEC-4EA7-B0B6-C43AA3E2DF7C@segment7.net> <0B05280D-85B7-4332-85ED-AB4AB9C23DED@zenspider.com> Message-ID: <465A23AD.4090601@gmail.com> Jim Weirich wrote: > On 5/27/07, Ryan Davis wrote: >> On May 24, 2007, at 16:19 , Eric Hodel wrote: >> >>> Jim and I were talking about trying incremental upload all the time, >>> rather than bulk at RailsConf... >> How crazy would it be to incorporate librsync into ruby and/or rubygems? > > I don't know how crazy. Is it commonly available (on all platforms)? It's not on Windows afaik. I'd have to check my Solaris box. I thought there was a pure Ruby implementation of rsync out there, but all I could find was this: http://eigenclass.org/hiki/cheap+rsync I saw rysnc-manager on rubyforge, but I think it's just a shell wrapper. A pure Ruby solution would be ideal if we can pull it off. Regards, Dan From self at mattmower.com Tue May 29 07:04:57 2007 From: self at mattmower.com (Matt Mower) Date: Tue, 29 May 2007 12:04:57 +0100 Subject: [Rubygems-developers] Unhappy memory usage on bulk update Message-ID: Hi. I'm having a problem with bulk updating the gem index on one of my VPS such that it doesn't seem likely to complete (it didn't finish in a little over 8hrs). The ruby process eats all available memory and most, although not all, of the swap. CPU usage appeared minimal although the load average crept up. There is nothing else on the system competing for memory or cpu while this was going on. I narrowed the problem down to SourceIndex#convert_specs: YAML.load(reduce_specs(yaml_spec)) and tried directly using:: curl -o yaml http://gems.rubyforge.org/yaml irb require 'yaml' YAML.load( File.open( 'yaml' ) ) This uses a lot of CPU until free ram is exhausted (with ruby between 70-75mb), then CPU use drops to almost nothing while the process acretes swap space. The VPS has 96mb ram + 128mb of swap. I was a bit surprised to see that the yaml file is 10mb given how routinely gem seems to bulk updating. Has this file been recently poisoned with something huge? Or has it just grown this big? I had someone else, using a VPS with 128mb of RAM, try updating and he reported similar problems. I'd welcome any suggestions on how to get bulk-update working again. Regards, Matt -- Matt Mower :: http://matt.blogs.it/ From darix at web.de Tue May 29 07:22:41 2007 From: darix at web.de (Marcus Rueckert) Date: Tue, 29 May 2007 13:22:41 +0200 Subject: [Rubygems-developers] Unhappy memory usage on bulk update In-Reply-To: References: Message-ID: <20070529112241.GM3490@pixel.global-banlist.de> On 2007-05-29 12:04:57 +0100, Matt Mower wrote: > I'm having a problem with bulk updating the gem index on one of my VPS > such that it doesn't seem likely to complete (it didn't finish in a > little over 8hrs). > > The ruby process eats all available memory and most, although not all, > of the swap. CPU usage appeared minimal although the load average > crept up. There is nothing else on the system competing for memory or > cpu while this was going on. mostlikely it needed the 8hrs because it was constantly swapping. > curl -o yaml http://gems.rubyforge.org/yaml > irb > require 'yaml' > YAML.load( File.open( 'yaml' ) ) > > This uses a lot of CPU until free ram is exhausted (with ruby between > 70-75mb), then CPU use drops to almost nothing while the process > acretes swap space. The VPS has 96mb ram + 128mb of swap. i tried it on my box and it went up to 220 mb (the peak usage i could see in top). just curious: what ruby version do you have? darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org From self at mattmower.com Tue May 29 07:54:45 2007 From: self at mattmower.com (Matt Mower) Date: Tue, 29 May 2007 12:54:45 +0100 Subject: [Rubygems-developers] Unhappy memory usage on bulk update In-Reply-To: <20070529112241.GM3490@pixel.global-banlist.de> References: <20070529112241.GM3490@pixel.global-banlist.de> Message-ID: Hi Marcus, On 29/05/07, Marcus Rueckert wrote: > > The ruby process eats all available memory and most, although not all, > > of the swap. CPU usage appeared minimal although the load average > > crept up. There is nothing else on the system competing for memory or > > cpu while this was going on. > > mostlikely it needed the 8hrs because it was constantly swapping. > Yes. I guess I was puzzled that it didn't just exhaust it all and die. > > curl -o yaml http://gems.rubyforge.org/yaml > > irb > > require 'yaml' > > YAML.load( File.open( 'yaml' ) ) > > > > This uses a lot of CPU until free ram is exhausted (with ruby between > > 70-75mb), then CPU use drops to almost nothing while the process > > acretes swap space. The VPS has 96mb ram + 128mb of swap. > > i tried it on my box and it went up to 220 mb (the peak usage i could > see in top). > I think it got to somewhere north 500mb when I tried it on my MBP (with 2gb of RAM). > just curious: what ruby version do you have? > root at host:/var/apps/squib/current# uname -a Linux host.mattmower.com 2.6.16-xen #8 SMP Mon Jun 26 03:20:22 PDT 2006 i686 GNU/Linux root at host:/var/apps/squib/current# ruby -v ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-linux] root at host:/var/apps/squib/current# gem -v 0.9.3 Regards, Matt -- Matt Mower :: http://matt.blogs.it/ From darix at web.de Tue May 29 14:52:27 2007 From: darix at web.de (Marcus Rueckert) Date: Tue, 29 May 2007 20:52:27 +0200 Subject: [Rubygems-developers] Unhappy memory usage on bulk update In-Reply-To: <20070529112241.GM3490@pixel.global-banlist.de> References: <20070529112241.GM3490@pixel.global-banlist.de> Message-ID: <20070529185227.GN3490@pixel.global-banlist.de> hi, i did a small profile run here: $ ruby -v ruby 1.8.5 (2006-12-25 patchlevel 12) [x86_64-linux] $ cat test2.rb require 'yaml' YAML.load(File.open('yaml')) $ ruby -r profile test2.rb > test2.rb.txt (output attached) seems the yaml parser doesnt like the file size. on the one hand side i wonder why you still track e.g. actionmailer 0.7 (i saw it when i glanced over the file), but on the other hand if gem will grow, the problem would just return. hope this helps darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org -------------- next part -------------- % cumulative self self total time seconds seconds calls ms/call ms/call name 22.04 23.12 23.12 1 23120.00 104830.00 YAML::Syck::Parser#load 21.94 46.14 23.02 41677 0.55 0.86 YAML::Syck::Resolver#transfer 18.55 65.60 19.46 597020 0.03 0.14 #.node_import 7.03 72.97 7.37 255079 0.03 0.03 Hash#default 3.00 76.12 3.15 4089 0.77 2.38 Rational#reduce 2.76 79.02 2.90 6815 0.43 0.67 Rational#initialize 2.72 81.87 2.85 1363 2.09 3.23 Date#jd_to_civil 2.50 84.49 2.62 4089 0.64 0.80 Integer#gcd 2.34 86.95 2.46 41677 0.06 0.06 Object#yaml_tag_subclasses? 2.21 89.27 2.32 41677 0.06 0.06 Module#yaml_tag_read_class 1.08 90.40 1.13 16356 0.07 0.07 Fixnum#< 1.08 91.53 1.13 2726 0.41 3.58 Rational#/ 0.99 92.57 1.04 25897 0.04 0.04 Kernel.kind_of? 0.95 93.57 1.00 1363 0.73 1.70 Date#civil_to_jd 0.85 94.46 0.89 4089 0.22 2.66 Object#Rational 0.82 95.32 0.86 18001 0.05 0.43 Fixnum#- 0.81 96.17 0.85 1363 0.62 5.58 Date#valid_civil? 0.77 96.98 0.81 6815 0.12 0.86 Rational#new! 0.75 97.77 0.79 41677 0.02 0.02 YAML::Object#yaml_initialize 0.67 98.47 0.70 2222 0.32 0.32 IO#read 0.59 99.09 0.62 13630 0.05 0.05 Float#floor 0.55 99.67 0.58 5452 0.11 0.22 Fixnum#/ 0.55 100.25 0.58 5452 0.11 0.11 Float#coerce 0.49 100.76 0.51 6818 0.07 0.74 Class#new 0.47 101.25 0.49 10904 0.04 0.04 Fixnum#> 0.44 101.71 0.46 1363 0.34 12.66 Date#jd_to_ajd 0.41 102.14 0.43 6890 0.06 0.06 Fixnum#* 0.33 102.49 0.35 1363 0.26 18.55 Date#new 0.32 102.83 0.34 1363 0.25 2.93 Integer#to_r 0.31 103.15 0.32 2726 0.12 0.15 Date#os? 0.27 103.43 0.28 13937 0.02 0.02 Fixnum#+ 0.26 103.70 0.27 1363 0.20 3.61 Rational#- 0.23 103.94 0.24 9541 0.03 0.03 Fixnum#== 0.14 104.09 0.15 5211 0.03 0.03 Time#utc 0.13 104.23 0.14 1363 0.10 1.39 Rational#coerce 0.10 104.33 0.10 6815 0.01 0.01 Fixnum#% 0.08 104.41 0.08 8178 0.01 0.01 Fixnum#div 0.08 104.49 0.08 5211 0.02 0.02 Time#at 0.07 104.56 0.07 10422 0.01 0.01 Time#to_i 0.07 104.63 0.07 8178 0.01 0.01 Fixnum#abs 0.07 104.70 0.07 1363 0.05 0.05 Date#initialize 0.05 104.75 0.05 2726 0.02 0.02 Module#=== 0.04 104.79 0.04 16 2.50 13.75 Kernel.require 0.03 104.82 0.03 5452 0.01 0.01 Float#* 0.02 104.84 0.02 1363 0.01 0.07 Class#new0 0.02 104.86 0.02 8178 0.00 0.00 Float#/ 0.01 104.87 0.01 25 0.40 0.40 Module#class_eval 0.01 104.88 0.01 26 0.38 0.38 Symbol#to_s 0.01 104.89 0.01 401 0.02 0.02 Module#method_added 0.01 104.90 0.01 28 0.36 0.36 Module#module_eval 0.01 104.91 0.01 2726 0.00 0.00 Fixnum#<= 0.00 104.91 0.00 22 0.00 0.00 Module#alias_method 0.00 104.91 0.00 2 0.00 0.00 Hash#keys 0.00 104.91 0.00 25 0.00 0.00 Module#attr_writer 0.00 104.91 0.00 25 0.00 0.00 Hash#has_key? 0.00 104.91 0.00 25 0.00 0.40 Module#yaml_as 0.00 104.91 0.00 7 0.00 1.43 Date#once 0.00 104.91 0.00 25 0.00 0.00 YAML.tag_class 0.00 104.91 0.00 2 0.00 0.00 Array#join 0.00 104.91 0.00 1 0.00 104830.00 YAML.load 0.00 104.91 0.00 1 0.00 0.00 YAML.resolver 0.00 104.91 0.00 1 0.00 0.00 IO#open 0.00 104.91 0.00 5 0.00 0.00 Module#attr_accessor 0.00 104.91 0.00 1 0.00 0.00 File#initialize 0.00 104.91 0.00 2 0.00 0.00 YAML::Syck::Resolver#initialize 0.00 104.91 0.00 1 0.00 0.00 YAML.parser 0.00 104.91 0.00 2 0.00 0.00 Module#public 0.00 104.91 0.00 1363 0.00 0.00 Array#== 0.00 104.91 0.00 2 0.00 0.00 Module#append_features 0.00 104.91 0.00 5 0.00 0.00 Module#private_class_method 0.00 104.91 0.00 2 0.00 0.00 Module#method_undefined 0.00 104.91 0.00 91 0.00 0.00 Fixnum#to_s 0.00 104.91 0.00 2 0.00 0.00 Module#include 0.00 104.91 0.00 1 0.00 0.00 YAML::Syck::Resolver#use_types_at 0.00 104.91 0.00 27 0.00 0.00 Class#inherited 0.00 104.91 0.00 23 0.00 0.00 Module#private 0.00 104.91 0.00 1 0.00 0.00 Module#undef_method 0.00 104.91 0.00 91 0.00 0.00 Symbol#to_i 0.00 104.91 0.00 1 0.00 0.00 IO#binmode 0.00 104.91 0.00 140 0.00 0.00 Kernel.singleton_method_added 0.00 104.91 0.00 1 0.00 0.00 YAML::Syck::Parser#set_resolver 0.00 104.91 0.00 1 0.00 0.00 Kernel.singleton_method_undefined 0.00 104.91 0.00 25 0.00 0.00 String#dump 0.00 104.91 0.00 2 0.00 0.00 Module#attr 0.00 104.91 0.00 1 0.00 0.00 YAML::Syck::Parser#initialize 0.00 104.91 0.00 1 0.00 0.00 Kernel.hash 0.00 104.91 0.00 10 0.00 2.00 Array#each 0.00 104.91 0.00 25 0.00 0.00 Hash#[]= 0.00 104.91 0.00 2 0.00 0.00 Module#included 0.00 104.91 0.00 2 0.00 0.00 Array#+ 0.00 104.91 0.00 1 0.00 104910.00 #toplevel From self at mattmower.com Tue May 29 18:18:14 2007 From: self at mattmower.com (Matt Mower) Date: Tue, 29 May 2007 23:18:14 +0100 Subject: [Rubygems-developers] Unhappy memory usage on bulk update In-Reply-To: <20070529185227.GN3490@pixel.global-banlist.de> References: <20070529112241.GM3490@pixel.global-banlist.de> <20070529185227.GN3490@pixel.global-banlist.de> Message-ID: On 29/05/07, Marcus Rueckert wrote: > seems the yaml parser doesnt like the file size. > > on the one hand side i wonder why you still track e.g. actionmailer 0.7 > (i saw it when i glanced over the file), but on the other hand if gem > will grow, the problem would just return. > I've been able to resolve my own problem by manually searching for and then downloading the .gem files I need from their respective RubyForge projects. Then attempting to install from the file, discovering new dependencies... rinse.. lather.. repeat. Painful, yes, but less painful that watching gem install do nothing for 8 hours. Regards, Matt -- Matt Mower :: http://matt.blogs.it/ From self at mattmower.com Wed May 30 06:28:02 2007 From: self at mattmower.com (Matt Mower) Date: Wed, 30 May 2007 11:28:02 +0100 Subject: [Rubygems-developers] Bulk updates Message-ID: Hi folks. I did a quick trawl back through the last few months of archives and couldn't find any discussions on changes to the current 'bulk update' approach of downloading a parsing a single, ever growing, YAML file. Has there been discussion of alternative approaches? Anything decided on? Anything definitely to be avoided? Any work planned or in progress? Respectfully. Matt -- Matt Mower :: http://matt.blogs.it/ From drbrain at segment7.net Wed May 30 10:33:36 2007 From: drbrain at segment7.net (Eric Hodel) Date: Wed, 30 May 2007 07:33:36 -0700 Subject: [Rubygems-developers] Bulk updates In-Reply-To: References: Message-ID: <4A9E73EE-3E55-42E5-B245-88B72BF01FDC@segment7.net> On May 30, 2007, at 03:28, Matt Mower wrote: > I did a quick trawl back through the last few months of archives and > couldn't find any discussions on changes to the current 'bulk update' > approach of downloading a parsing a single, ever growing, YAML file. > > Has there been discussion of alternative approaches? Anything decided > on? Anything definitely to be avoided? Any work planned or in > progress? Read back two threads. From self at mattmower.com Wed May 30 11:04:15 2007 From: self at mattmower.com (Matt Mower) Date: Wed, 30 May 2007 16:04:15 +0100 Subject: [Rubygems-developers] Bulk updates In-Reply-To: <4A9E73EE-3E55-42E5-B245-88B72BF01FDC@segment7.net> References: <4A9E73EE-3E55-42E5-B245-88B72BF01FDC@segment7.net> Message-ID: On 30/05/07, Eric Hodel wrote: > On May 30, 2007, at 03:28, Matt Mower wrote: > > I did a quick trawl back through the last few months of archives and > > couldn't find any discussions on changes to the current 'bulk update' > > approach of downloading a parsing a single, ever growing, YAML file. > > > > Has there been discussion of alternative approaches? Anything decided > > on? Anything definitely to be avoided? Any work planned or in > > progress? > > Read back two threads. When I read that thread I took it to mean that people were looking at ways to avoid transferring the entire YAML file but I made an assumption that the data would still be held in a single in-memory data structure (at a glance the same structure loaded from the YAML file but using Marshal instead). Possibly the problem is only related to trying to build such a big structure using YAML but the problems I (and a couple of others) have seen appear to be related to the memory usage of the gem source index. I've got little experience of the Rubygems codebase at this point. Have I misunderstood the thrust of that thread, or how Rubygems operates? M -- Matt Mower :: http://matt.blogs.it/ From drbrain at segment7.net Wed May 30 11:48:47 2007 From: drbrain at segment7.net (Eric Hodel) Date: Wed, 30 May 2007 08:48:47 -0700 Subject: [Rubygems-developers] Bulk updates In-Reply-To: References: <4A9E73EE-3E55-42E5-B245-88B72BF01FDC@segment7.net> Message-ID: <3538E783-B1EF-4087-9810-A13D43F53DAA@segment7.net> On May 30, 2007, at 08:04, Matt Mower wrote: > On 30/05/07, Eric Hodel wrote: >> On May 30, 2007, at 03:28, Matt Mower wrote: >>> I did a quick trawl back through the last few months of archives and >>> couldn't find any discussions on changes to the current 'bulk >>> update' >>> approach of downloading a parsing a single, ever growing, YAML file. >>> >>> Has there been discussion of alternative approaches? Anything >>> decided >>> on? Anything definitely to be avoided? Any work planned or in >>> progress? >> >> Read back two threads. > > When I read that thread I took it to mean that people were looking at > ways to avoid transferring the entire YAML file but I made an > assumption that the data would still be held in a single in-memory > data structure (at a glance the same structure loaded from the YAML > file but using Marshal instead). > > Possibly the problem is only related to trying to build such a big > structure using YAML but the problems I (and a couple of others) have > seen appear to be related to the memory usage of the gem source index. > > I've got little experience of the Rubygems codebase at this point. > Have I misunderstood the thrust of that thread, or how Rubygems > operates? Working with the source cache creates a giant object tree. Loading the YAML file creates two very large Strings and a giant object tree. Loading a handful of updates creates a handful or two of very small object trees and Strings. While its hard to capture the latter, it does take much less memory than the former. The problem is being worked on right now. From ryand-ruby at zenspider.com Wed May 30 15:30:44 2007 From: ryand-ruby at zenspider.com (Ryan Davis) Date: Wed, 30 May 2007 15:30:44 -0400 Subject: [Rubygems-developers] Bulk updates In-Reply-To: <3538E783-B1EF-4087-9810-A13D43F53DAA@segment7.net> References: <4A9E73EE-3E55-42E5-B245-88B72BF01FDC@segment7.net> <3538E783-B1EF-4087-9810-A13D43F53DAA@segment7.net> Message-ID: <37C4EA5E-1C1F-48DF-BE66-2E696005651B@zenspider.com> On May 30, 2007, at 11:48 , Eric Hodel wrote: > Working with the source cache creates a giant object tree. > > Loading the YAML file creates two very large Strings and a giant > object tree. > > Loading a handful of updates creates a handful or two of very small > object trees and Strings. > > While its hard to capture the latter, it does take much less memory > than the former. > > The problem is being worked on right now. I just committed a change that seems to cut memory use by about 40ish %. There was a phase that was munging the output to deal with an old yaml bug and was creating quite a bit of memory. Removing it dropped my memory consumption from 112meg to 73meg. We could drop a tad more if we nuked the part of the cache for the source we were currently bulk loading, but it only equated to about 10 meg in my experiments. From jim at weirichhouse.org Thu May 31 14:30:36 2007 From: jim at weirichhouse.org (Jim Weirich) Date: Thu, 31 May 2007 14:30:36 -0400 Subject: [Rubygems-developers] Erroneously Named Gems Message-ID: <465F144C.8000607@weirichhouse.org> Dear Sirs: You are receiving this email because you are listed as an author to one of the following RubyGems packages: * crypt-isaac_0.9.1.gem * evdb-current.gem * evdbapi-current.gem * fireruby_0.4.0_powerpc_darwin.gem * fireruby_0.4.1_powerpc_darwin.gem These gems are named erroneously. (e.g.. a "_" rather than a "-" before the version, or a non-numeric version number). Because of this, RubyGems is not finding these gems and they are not available for downloading via the gem command. Would it be possible to re-release these files with a proper name? Thank you. I appreciate your valuable time if fixing this matter. Finally, I am curious how these files were generated. The "gem build" (or the Rakefile equivalent) should be properly forming these names. Do we have a bug in the build command, or are you using a non-standard method of constructing gems. Is there anyway we can make this process less error-prone. I've copied the RubyGems developers mailing list on this email. Please send any responses there. Thank you for your time. -- -- Jim Weirich jim at weirichhouse.org http://onestepback.org -- In theory, practice and theory are the same. -- In practice, they are different. From paul at eventful.com Thu May 31 19:54:15 2007 From: paul at eventful.com (Paul Knight) Date: Thu, 31 May 2007 16:54:15 -0700 Subject: [Rubygems-developers] Erroneously Named Gems References: <7143249B-EE20-4DB9-A7F6-AC16C8B1CD1A@eventful.com> Message-ID: <463482C3-C388-47FC-8A27-9AE2AC6A49D6@eventful.com> On May 31, 2007, at 11:30 AM, Jim Weirich wrote: > Dear Sirs: > > You are receiving this email because you are listed as an author to > one of the following RubyGems packages: > > * evdb-current.gem > * evdbapi-current.gem > [...] Yep, those two used to be mine. > Because of this, RubyGems is not finding these gems and they are > not available for downloading via the gem command. > > Would it be possible to re-release these files with a proper name? > Thank you. This library has been deprecated for some time, and the new series "eventfulapi" follows the standard gem naming conventions. In fact, this particular gem file should no longer be available, and I thought I removed both the evdbapi package and its associated files on RubyForge. At least, they do not appear on my project admin pages. > Finally, I am curious how these files were generated. The "gem > build" (or the Rakefile equivalent) should be properly forming > these names. Do we have a bug in the build command, or are you > using a non-standard method of constructing gems. Is there anyway > we can make this process less error-prone. Like the one built by Kirk Haines, this gem was built a long time ago. It was published with the gem build command, but to complement our libraries in other languages, on our own distribution site the current version was offered under the alias name "-current". I offered both the numbered version and the alias without thinking. Sorry about that. Hope this didn't cause any trouble. Please let me know if there's anything I need to do. ------ Paul Knight Eventful, Inc. paul at eventful.com Life is short... make it Eventful!