From wjrogers at gmail.com  Fri Jan 11 21:14:56 2008
From: wjrogers at gmail.com (Will Rogers)
Date: Fri, 11 Jan 2008 21:14:56 -0500
Subject: [Rubyinstaller-devel] Building Ruby + extensions with modern Visual
	C++
Message-ID: <323a37200801111814m4cc8f433u1508da3d69fca8db@mail.gmail.com>

Hi,

There doesn't seem to be any discussion on this list right now, so let
me kick something off. I contacted Curt and Luis earlier this week to
ask what their plans were with regard to migrating the OCI to a new
compiler. Luis seems to be working on a bootstrapping approach using
mingw, which is fine, but I still think that the "best" thing to do
would be to build with the Microsoft compiler on Windows, it being the
native solution and all. Also, according to what I've read, linking
with the newer versions of the MS C runtime has some advantages in
terms of security and stability compared to the old msvcrt.dll that
mingw (and vc6) links to.

I started trying to build everything necessary for the OCI using
VC2005. I quickly realized that finding, installing, updating, and
configuring VC2005 Express is a gigantic pain in the ass. It does not
work with the Platform SDK out of the box, which means you have to
edit some config files by hand to build win32 applications. This is
far from ideal.

Getting set up with VC2008 Express, however, is easy. You just have to
download one file from http://www.microsoft.com/express/ and install
it. Everything works out of the box. I also see some advantage to
using the latest version (it will probably be available for longer, at
the least).

So, on to the actual compiling.

I started with Ruby 1.8.6-p111 and zlib. Both build flawlessly with
vc2008. Easy. I continued with openssl, building with NASM 2.00 +
vc2008. I had to copy the nasm.exe binary to nasmw.exe for the openssl
makefile to work, but then it also built fine.

Then I hit readline. Readline is a mess. There isn't really an
up-to-date, well-maintained windows port, as far as I can tell. I
think there needs to be a reliable port of this library to really
depend on the Microsoft compiler. I'm willing to try to learn how to
port it, but I don't really have the existing expertise to do it
quickly. If anyone has some ideas or pointers here, I'm all ears.

What other dependencies does the OCI have? iconv, tcl?

I want to be clear that I'm not challenging Luis's work on the
mingw-based bootstrapping setup. I think it's valuable to work on both
in parallel, to learn more about what's possible. I will have some
time this weekend to work on getting further with the vc2008 builds.


A good night to all,

- Will

From curt.hibbs at gmail.com  Sat Jan 12 01:55:56 2008
From: curt.hibbs at gmail.com (Curt Hibbs)
Date: Sat, 12 Jan 2008 00:55:56 -0600
Subject: [Rubyinstaller-devel] Building Ruby + extensions with modern
	Visual C++
In-Reply-To: <323a37200801111814m4cc8f433u1508da3d69fca8db@mail.gmail.com>
References: <323a37200801111814m4cc8f433u1508da3d69fca8db@mail.gmail.com>
Message-ID: <31d15f490801112255h689a958lf4673b5084694074@mail.gmail.com>

Remember that you also have to build all of the extensions and RubyGems
because the binaries produced with VC2008 use a different runtime dll than
VC6 and you can't mix them without causing problems.

Curt

On Jan 11, 2008 8:14 PM, Will Rogers <wjrogers at gmail.com> wrote:

> Hi,
>
> There doesn't seem to be any discussion on this list right now, so let
> me kick something off. I contacted Curt and Luis earlier this week to
> ask what their plans were with regard to migrating the OCI to a new
> compiler. Luis seems to be working on a bootstrapping approach using
> mingw, which is fine, but I still think that the "best" thing to do
> would be to build with the Microsoft compiler on Windows, it being the
> native solution and all. Also, according to what I've read, linking
> with the newer versions of the MS C runtime has some advantages in
> terms of security and stability compared to the old msvcrt.dll that
> mingw (and vc6) links to.
>
> I started trying to build everything necessary for the OCI using
> VC2005. I quickly realized that finding, installing, updating, and
> configuring VC2005 Express is a gigantic pain in the ass. It does not
> work with the Platform SDK out of the box, which means you have to
> edit some config files by hand to build win32 applications. This is
> far from ideal.
>
> Getting set up with VC2008 Express, however, is easy. You just have to
> download one file from http://www.microsoft.com/express/ and install
> it. Everything works out of the box. I also see some advantage to
> using the latest version (it will probably be available for longer, at
> the least).
>
> So, on to the actual compiling.
>
> I started with Ruby 1.8.6-p111 and zlib. Both build flawlessly with
> vc2008. Easy. I continued with openssl, building with NASM 2.00 +
> vc2008. I had to copy the nasm.exe binary to nasmw.exe for the openssl
> makefile to work, but then it also built fine.
>
> Then I hit readline. Readline is a mess. There isn't really an
> up-to-date, well-maintained windows port, as far as I can tell. I
> think there needs to be a reliable port of this library to really
> depend on the Microsoft compiler. I'm willing to try to learn how to
> port it, but I don't really have the existing expertise to do it
> quickly. If anyone has some ideas or pointers here, I'm all ears.
>
> What other dependencies does the OCI have? iconv, tcl?
>
> I want to be clear that I'm not challenging Luis's work on the
> mingw-based bootstrapping setup. I think it's valuable to work on both
> in parallel, to learn more about what's possible. I will have some
> time this weekend to work on getting further with the vc2008 builds.
>
>
> A good night to all,
>
> - Will
> _______________________________________________
> Rubyinstaller-devel mailing list
> Rubyinstaller-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rubyinstaller-devel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/rubyinstaller-devel/attachments/20080112/6ef033ea/attachment.html 

From luislavena at gmail.com  Sat Jan 12 07:50:40 2008
From: luislavena at gmail.com (Luis Lavena)
Date: Sat, 12 Jan 2008 10:50:40 -0200
Subject: [Rubyinstaller-devel] Building Ruby + extensions with modern
	Visual C++
In-Reply-To: <31d15f490801112255h689a958lf4673b5084694074@mail.gmail.com>
References: <323a37200801111814m4cc8f433u1508da3d69fca8db@mail.gmail.com>
	<31d15f490801112255h689a958lf4673b5084694074@mail.gmail.com>
Message-ID: <71166b3b0801120450r730801e7s8d919568607559cb@mail.gmail.com>

On Jan 12, 2008 4:55 AM, Curt Hibbs <curt.hibbs at gmail.com> wrote:
> Remember that you also have to build all of the extensions and RubyGems
> because the binaries produced with VC2008 use a different runtime dll than
> VC6 and you can't mix them without causing problems.
>

That will not be a problem, since the changes I introduced in RubyGems
(before they released version 1.0.x)  make a strong difference between
VC6 and VC8/VC9.

So gems pre-built for VC6 can co-exist with ones for VC8 (on
rubyforge), but you can't install VC8 gems on top of VC6 ruby (or vice
versa).

-- 
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

From luislavena at gmail.com  Sat Jan 12 08:22:10 2008
From: luislavena at gmail.com (Luis Lavena)
Date: Sat, 12 Jan 2008 11:22:10 -0200
Subject: [Rubyinstaller-devel] Building Ruby + extensions with modern
	Visual C++
In-Reply-To: <323a37200801111814m4cc8f433u1508da3d69fca8db@mail.gmail.com>
References: <323a37200801111814m4cc8f433u1508da3d69fca8db@mail.gmail.com>
Message-ID: <71166b3b0801120522u25c94608qe23953b7d563f37f@mail.gmail.com>

On Jan 12, 2008 12:14 AM, Will Rogers <wjrogers at gmail.com> wrote:
> Hi,
>
> There doesn't seem to be any discussion on this list right now, so let
> me kick something off. I contacted Curt and Luis earlier this week to
> ask what their plans were with regard to migrating the OCI to a new
> compiler. Luis seems to be working on a bootstrapping approach using
> mingw, which is fine, but I still think that the "best" thing to do
> would be to build with the Microsoft compiler on Windows, it being the
> native solution and all. Also, according to what I've read, linking
> with the newer versions of the MS C runtime has some advantages in
> terms of security and stability compared to the old msvcrt.dll that
> mingw (and vc6) links to.

I used to think that "all native tools" was better than MinGW path,
security and performance related, but if you take a closer look at the
build process of ruby, you will find the _CRT_SECURE_NO_DEPRECATE
being defined for every file being compiled.

That define is used to avoid errors and warning of code being using
the unsafe CRT functions found in MSVCRT.dll

They suggest you switch to the safest version, which requires another
set of parameters to the functions that are signature incompatible
with other CRT implementations.

Also I'll like to point that, if you take a closer look to the
MSVCRT.dll store in windows\system32, you will find is not version 6.0
(which was buggy) but is 7.0.2600.2180

So, Windows XP SP2 is using version 7.0! and ruby was linked with CRT 6.0!

Hehehe, the function signatures of both dlls match, so there is no
problem. And 7.0 CRT is safer than 6.0...

Anyway...

> I started trying to build everything necessary for the OCI using
> VC2005. I quickly realized that finding, installing, updating, and
> configuring VC2005 Express is a gigantic pain in the ass. It does not
> work with the Platform SDK out of the box, which means you have to
> edit some config files by hand to build win32 applications. This is
> far from ideal.

I've used the Updated Windows SDK for Windows Vista (1GB, free
download), which was VC8 and is the same as VS2005. It shipped with
SDK so no need to change things to get it working.

Also you get the properly configured console to run all the tools from there.

> Getting set up with VC2008 Express, however, is easy. You just have to
> download one file from http://www.microsoft.com/express/ and install
> it. Everything works out of the box. I also see some advantage to
> using the latest version (it will probably be available for longer, at
> the least).

Yes, VC2008 Sounds interesting, I'll need to give it a whirl...

The idea with the bootstrap process is that we can create, with the
same set of tasks, the interpreter and the dependencies to have it
working, compiler-agnostic.

So far, I've only implemented MinGW+MSYS, but the idea is we can add
some recipes or maybe a "rake/compiletask" that can determine the
available compiler and build the dependencies and Ruby.

As I've commented to James Tucker, first we must get a complete stable
environment, then we can take over the world :)

> So, on to the actual compiling.
>
> I started with Ruby 1.8.6-p111 and zlib. Both build flawlessly with
> vc2008. Easy. I continued with openssl, building with NASM 2.00 +
> vc2008. I had to copy the nasm.exe binary to nasmw.exe for the openssl
> makefile to work, but then it also built fine.
>

That change should be pushed upstream (to openssl guys) to allow them
be compatible with newer versions of the compiler).

> Then I hit readline. Readline is a mess. There isn't really an
> up-to-date, well-maintained windows port, as far as I can tell. I
> think there needs to be a reliable port of this library to really
> depend on the Microsoft compiler. I'm willing to try to learn how to
> port it, but I don't really have the existing expertise to do it
> quickly. If anyone has some ideas or pointers here, I'm all ears.
>

Oh, now you know my pain. Not event the MinGW port is up to date, I've
tried 4.3, 5.0 and 5.2 that I don't remember where I found to my VC8
experiment.

At that time (May 2007), I bundled everything in the same SVN repo, so
after adding the dependencies the thing was big (150MB svn repo).

I need to undust that form my backup and switch these changes to
bazaar, which helped me keep track of upstream changes and provide
better set of patches to the maintainers of some libraries.

> What other dependencies does the OCI have? iconv, tcl?

iconv, tk and tcl. The last two I think will not worth the effort...

This paste contains the list of the extensions that are bundled with
official (VC6, garbagecollect) build of Ruby 1.8.6:

http://rafb.net/p/Wyx6mG28.html

And this was my status about them:

http://rafb.net/p/1GUz5L68.html

If you take a closer look, tk and tkutils didn't get marked as OK...
Every time I tried couldn't get them working, every time I failed.

But it seems lot of folks rely on Tk to do some GUIs, so... maybe we
can keep them "enabled".

> I want to be clear that I'm not challenging Luis's work on the
> mingw-based bootstrapping setup. I think it's valuable to work on both
> in parallel, to learn more about what's possible. I will have some
> time this weekend to work on getting further with the vc2008 builds.

The job to get all the dependencies working with VC8 a bit stressing,
mostly due lack of insterest from upstream.

Anyway, what do you think if we put the mingw+msys recipes on hold and
implement some for VC8? it seems it's worth we give it a try but not
in the manual way, but somehow automated.

Maybe we can came up with a CompilerUtils that can #compile #make,
#link or #configure disregard of it's GCC or VC in the background.

The idea of build the dependencies from scratch is good too, since
some of the binaries are outdated.

In any case, we could have a series of patches for these tools until
upstream maintainers get them included.

A good thing about Bazaar is that we can branch and later merge back
without too much problems :-D

I'll be hanging on ruby-lang later today, but couldn't promise I'll
have Express 208 installed, since I hate these tools modify my file
associations ;-)

> A good night to all,

Night to you too,

-- 
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

From rogerpack2005 at gmail.com  Sat Jan 12 10:20:54 2008
From: rogerpack2005 at gmail.com (Roger Pack)
Date: Sat, 12 Jan 2008 08:20:54 -0700
Subject: [Rubyinstaller-devel] Rubyinstaller-devel Digest, Vol 15,
	Issue 1
In-Reply-To: <mailman.218.1200104100.8416.rubyinstaller-devel@rubyforge.org>
References: <mailman.218.1200104100.8416.rubyinstaller-devel@rubyforge.org>
Message-ID: <966599840801120720i775191d2yb0322c4520b3cf8@mail.gmail.com>

> There doesn't seem to be any discussion on this list right now, so let
> me kick something off. I contacted Curt and Luis earlier this week to
> ask what their plans were with regard to migrating the OCI to a new
> compiler. Luis seems to be working on a bootstrapping approach using
> mingw, which is fine, but I still think that the "best" thing to do
> would be to build with the Microsoft compiler on Windows, it being the
> native solution and all. Also, according to what I've read, linking
> with the newer versions of the MS C runtime has some advantages in
> terms of security and stability compared to the old msvcrt.dll that
> mingw (and vc6) links to.

I think half the reason of using mingw is to avoid the pain of
configuring VC8.  That being said VC8 supports 64-bit 'a little
better' than mingw, so an awkward trade-off.
Mingw can be built to link against (according to Luis) different
versions of the msvcrtXXX.dll, so that shoudn't be a strike against
it.  I'd say overall, since Ruby is SO entrenched in the Linux world,
it is easier to just use mingw.  That being said, I've never gotten
readline to compile on mingw (Luis has [?]), but I mean mingw also has
a setup fee.  Sigh.  Good luck!
-Roger
Note please CC me responses, as I seldom read from the list.
--
Wherefore if ye shall press forward, feasting upon the word of Christ,
and endure to the end, behold thus saith the Father: ye shall have
eternal life. - 2 Ne 31:20

From luislavena at gmail.com  Sat Jan 12 12:48:09 2008
From: luislavena at gmail.com (Luis Lavena)
Date: Sat, 12 Jan 2008 15:48:09 -0200
Subject: [Rubyinstaller-devel] Rubyinstaller-devel Digest, Vol 15,
	Issue 1
In-Reply-To: <966599840801120720i775191d2yb0322c4520b3cf8@mail.gmail.com>
References: <mailman.218.1200104100.8416.rubyinstaller-devel@rubyforge.org>
	<966599840801120720i775191d2yb0322c4520b3cf8@mail.gmail.com>
Message-ID: <71166b3b0801120948h189ca888uf4ff46753cba3e95@mail.gmail.com>

On Jan 12, 2008 1:20 PM, Roger Pack <rogerpack2005 at gmail.com> wrote:
> > There doesn't seem to be any discussion on this list right now, so let
> > me kick something off. I contacted Curt and Luis earlier this week to
> > ask what their plans were with regard to migrating the OCI to a new
> > compiler. Luis seems to be working on a bootstrapping approach using
> > mingw, which is fine, but I still think that the "best" thing to do
> > would be to build with the Microsoft compiler on Windows, it being the
> > native solution and all. Also, according to what I've read, linking
> > with the newer versions of the MS C runtime has some advantages in
> > terms of security and stability compared to the old msvcrt.dll that
> > mingw (and vc6) links to.
>
> I think half the reason of using mingw is to avoid the pain of
> configuring VC8.  That being said VC8 supports 64-bit 'a little
> better' than mingw, so an awkward trade-off.

Well, the thing with 64bits is worse than that.

Ruby isn't optimized for 64bits, as discussed previously on ruby-core
and my own conclusion.
Every object under 64bits ruby will use almost twice (!!!) amount of
memory than 32bits version. The performance issue regarding 64bits OS
on top of 64 bit processor do not worth the pain of having 400MB
Ruby+Rails processes.

The worse part of x64 ruby is that not only we have to compile ruby
for 64bits, but all the dependencies, and the thing is that some of
them don't even compile in 32bits! (I'm talking of course about GNU
libraries and others).

> Mingw can be built to link against (according to Luis) different
> versions of the msvcrtXXX.dll, so that shoudn't be a strike against
> it.

The targetting of msvcr90 was discussed in cygwin development list [1]
by Paul Moore (python-dev) and was later announced in python-dev list
[2]

There is another way to target "at runtime" which version of msvcrt
will be used, but is not official part of MinGW. In any case, the
Python guys don't worry about that since they have distutils
package... I wish we have that too...

> I'd say overall, since Ruby is SO entrenched in the Linux world,
> it is easier to just use mingw.  That being said, I've never gotten
> readline to compile on mingw (Luis has [?]), but I mean mingw also has
> a setup fee.  Sigh.  Good luck!

I got it working, but but couldn't reproduce it, just failed... some
GNU tools are just a pain get them right... too many patches
distributed across the world that you need to apply before compile...
and of course, the RTFM attitude...

> -Roger
> Note please CC me responses, as I seldom read from the list.

Hey Roger, this is a low-traffic list, subscribe! :-D

[1] http://thread.gmane.org/gmane.comp.gnu.mingw.user/24686
[2] http://mail.python.org/pipermail/python-dev/2007-November/075473.html

-- 
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

From luislavena at gmail.com  Thu Jan 17 06:54:06 2008
From: luislavena at gmail.com (Luis Lavena)
Date: Thu, 17 Jan 2008 09:54:06 -0200
Subject: [Rubyinstaller-devel] Some thoughts about VC++ 2008
Message-ID: <71166b3b0801170354x22817d1bn94f375c59f407268@mail.gmail.com>

Hello List,

Discussed previously with Will Rogers about using VC++ 2008 (VC9,
Express Edition), I decided to download the ISO and give it a whirl.

Leaving all the "build every extension with VC9" outside of the
discussion for a moment, lets  compare different builds of ruby in the
same system.

These results are based on a simple, not optimized sudoku solver that
I've used previously to compare Ruby builds and started a "my ruby is
faster than yours" discussion in ruby-talk a few months back, anyway:

Sudoku: http://pastie.caboo.se/92995
Benchmarks: http://pastie.caboo.se/pastes/95210

This time, I'm at home computer (Dual Core E2160, 1GB RAM, Windows XP
SP2), so I did the standard bench for VC6 build and took it as base:

Note: all the versions of ruby were build with --with-winsock2
--disable-install-doc configuration switches, no other optimization or
tweaking.

ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]
mean elapsed time (5 runs): 16.550 sec.

ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32_90]
mean elapsed time (5 runs): 12.272 sec.

Comparing both VC6 vs VC9, we have a 25.84% performance boost, not bad.

But, I decided to try the other platforms available, MinGW and VC8
too, since I have the Windows Vista SDK Update also installed:

ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32_80]
mean elapsed time (5 runs): 12.281 sec.

ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mingw32]
mean elapsed time (5 runs): 11.306 sec.

Mmm, something funny, it seems VC8 do not differ from VC9 generated
code, and it loss against MinGW (and we are talking of GCC 3.4.5
(mingw special build))...

Not GCC version 4... what the heck, let's try it...

So, I put my helmet and downloaded the experimental GCC 4.2.1 (dw2):
http://sourceforge.net/project/shownotes.php?release_id=532060&group_id=2435

gcc version: 4.2.1-dw2 (mingw32-2)

ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mingw32]
mean elapsed time (5 runs): 11.0 sec.

That's 33.54% performance boost compared with standard VC6 Ruby... and
VC9 loss this battle, again.

So I wonder if VC8/9 is something good in the long run.

Please, now add to the equation all the troubles tweaking the
dependencies makefiles to make them compatible with MSVCR80/90 and
that we must manually build all them *just for ruby*, and not using
the standard ones, diverging from upstream maintainers.

Still thinking worth the effort?

Some users will argue that "with VC8/9 we can have 64bits builds"...
and that is true, except that we will also need to fix the
dependencies to properly build for 64bits.

There is also MinGW 64bits, just in case:
http://sourceforge.net/projects/mingw-w64/

Still need to undust my copy of Windows XP SP2 x86_64 (or x64 Edition)
to test it.

Thoughts?

-- 
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams