[Wtr-general] Order of execution for ruby tests with test/unit

[Paul Rogers]

test::unit does a sort on the method names. so test_a comes before test_b

I cant remember what happens with test_a and test_A
-------------- next part --------------
Hi George, it's been my experience that the unit tests run in what appears
to be an alphanumerically-ascending order.  It's been a while since I've
looked at it, but I think this is the general pattern.  (This is all off the
top of my head, but I'm sure I can easily confirm it.  I still have the
scripts lying around somewhere that gave me this info.)  This might be
academic to some but it was enlightening to me so I thought I'd share it
since you asked.

(BTW, please forgive me for not using correct Test case/suite/whatever
terminology here.. I don't have my Ruby/Watir references handy as I type
this.)

Let's say that you have a master script that calls 3 other scripts that have
the test classes in them.. something like:
---
# Setup
require 'watir'
require 'test/unit'

# Set up common methods, variables, whatever

# My Tests
require Startup.rb
require HomePage.rb
require ShutDown.rb
---

The imaginary contents of ' Startup.rb' could be something like:
---
class Test_Startup < Test::Unit::TestCase
  ie = Watir::IE...
  puts 'This is the start'

  def test_navigation
    ie.goto(..)
    ie.blah.blah
    assert_equal(yadda, yadda)
  end

  def test_login
    ie.text_field().set()
    ....
  end

  puts 'This is the end.'

end
---

And so on.  You can assume that a similar pattern exists for the other two
scripts.  Here's where it gets fun.

My initial, uneducated, guess was that these tests would run in the
following order:
1) Startup.rb
   a) test_navigation
   b) test_login
   c) ...
2) HomePage.rb
   a) ..
   b) ...
3) ShutDown.rb
   a) ..
   b) ...

Ya, well, reality was something a bit different.  That is, I found out that
these tests are supposed to run in a 'random' order.

I didn't believe for one second that Ruby was really going to run these
tests randomly so I tried to figure out what the pattern actually was.
Here's what I found out.

First of all, the code is put into two 'pools': (1) Stuff not in a 'test'
definition (i.e. *not* a test), and (2) stuff that's in a test definition (
i.e. the tests).

-> All the non-test stuff is executed first, in the order in which they
appear.

So, all the lines in the Master script are run first, as you would imagine,
followed by the lines it finds in the test scripts that aren't in 'test'
definitions either.  That means that the following lines will be executed
*before* any tests are run, even though they're within test classes:
---
  ie = Watir::IE...
  puts 'This is the start'
  puts 'This is the end.'
---

Next, all the Test Classes are put into alphanumeric order.  So regardless
of how I called the scripts above, they will be run in the following order:
---
class Test_HomePage < Test::Unit::TestCase
class Test_ShutDown < Test::Unit::TestCase
class Test_Startup < Test::Unit::TestCase
---

Lastly, all of the test cases are sorted in an ascending alphanumeric
order.  So, given the two tests in my 'Startup' script above, they will
actually run in this order:
  def test_login
  def test_navigation
...and so on..

To get around this, I've been using a naming convention for my test classes
that include numeric labels.  Something like this:
---
class Test_01_Startup < Test::Unit::TestCase
class Test_02_HomePage < Test::Unit::TestCase
class Test_03_ShutDown < Test::Unit::TestCase
---

-> Which assures me that they will be run by Ruby in that order.

Then I also use numeric labels similarly with the test case names too.  That
way, I know that they run in the correct order.  So using my example from
above, you would probably see something like this if you browse through my
scripts:
---
class Test_01_Startup < Test::Unit::TestCase
  def test_0101_navigation
  def test_0102_login
class Test_02_HomePage < Test::Unit::TestCase
  def test_0201_check_userdata
  def test_0202_check_reports
class Test_03_ShutDown < Test::Unit::TestCase
   ...
---

This way, I don't have to worry about which order the tests are run because
I *know* that they will run from top to bottom, in the order that I specify
the tests to run.

I was quite happy to discover that pattern.  I don't recall where I got the
initial tip, but I'm pretty sure that someone or something mentioned that I
should consider ascending labels.

Works for me.  I would be happy to have someone confirm this pattern for
me.  As I said, it was a while ago that I figured this out (probably 4-6
weeks ago now), and I am just winging this off the top of my head, but I'm
pretty sure I got the details correct.

Hope this helps.  Have a great day!

Cheers.  Paul.


On 04/05/06, George Flaherty <George.Flaherty at sas.com > wrote:
>
> Sorry to repeat here, but did someone post a way to execute ruby unit
> tests in a "predetermined order" vs.the default "random" order?
>
> Thanks
> -george
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/wtr-general/attachments/20060504/5d60b40e/attachment.html 
-------------- next part --------------
_______________________________________________
Wtr-general mailing list
Wtr-general at rubyforge.org
http://rubyforge.org/mailman/listinfo/wtr-general