From bret at pettichord.com Fri May 14 00:12:21 2004 From: bret at pettichord.com (Bret Pettichord) Date: Thu, 13 May 2004 23:12:21 -0500 Subject: [Wtr-general] Fwd: tutorial description: scripting web tests Message-ID: <5.1.0.14.2.20040513215726.047dd7d8@127.0.0.1> Today was the deadline for the tutorial description that will appear in the XP/AU proceedings. Our submission is below. It had to be written in the past tense, so what it really describes is what happened this week when i taught the class in Portland and Seattle. I will be teaching it again next week in Orlando. After that, i'll be able to focus more fully on the next revision of the class. 1. Using blocks to specify rules that identified forms and elements worked rather well, certainly better than the dedidicated functions we'd used last October. But learning about when to identify using actions vs. values vs. names takes time -- time that we won't have in the half-day version in Calgary. So we are going to have to insert unique names for all the forms and elements. We'll also have to add named spans for text we want to verify. 2. The network is unnecessary. Having everyone test a single server is pretty pointless because it starts crashing pretty fast. On monday, when this started happening, i told them that they could run their own local server, and five minutes later every one of them was doing this. I didn't even set up the network on tuesday and won't be using it in orlando next week. I'm not sure if the server is less stable that previously. It might be, perhaps due to platform (windows vs. mac) or the current version of Ruby (1.8.1-12). 3. A couple students were getting semi-reproducible seg faults in Ruby when they were running harnessed tests. These were client side. I'm going to see if i can track this down. In one case, the student was using 1.8.1-13. 4. As the description below implies, we pretty much got most of the students to put together a suite of tests by the end of the day. It's rather satisfying to get this far and will be a challenge to get as far in only a half-day. The biggest area of streamlining has got to come from using named elements. This also means we can use more Chris's original IEC code and concept. A lot of students wasn't better logging. I know that Paul has this and this really needs to get folded into the code we use for the class. 5. Even with named objects, the hardest part of the class is probably getting students to associate the names of things with the actual objects. After seeing Chris's Motunes demo, i'm thinking that it might be possible to directly annotate objects by modifying the page via the DOM. This is something i want to try out. It could be major cool thing that accellerates the visibility of the testing hooks we use. 6. My policy has been that i need a TA for more than 12 students. On Monday, i had 17 students, but the TA didn't work out. That was a bear for me. I was constantly scrambling and didn't really have time to reflect on what was happening. I had a TA (Dan) on Tuesday with about the same number of students. Things went much much better. 7. In each class, i had one or two people who really ran with what we have. We may have more of these at XP/AU, because of the kind of conference it is. If so, we may want to have one of us run a fast-track in some corner. 8. The current docs are too disorganized and unspecific for a few people in each class. They get discouraged and expect better doc. I plan to keep working on this. I think we actually addressed this somewhat better in early classes. In other words, Brian addressed it better. This is important to ensure success with people who are not quick learners or experience programmers. I think they can be reached. 9. I've given up on trying to make people pair. Everyone wants to use their own machine. In one case, two students came with one computer between them and that was ok (and we should continue to encourage this). XP/AU may give us a better context to encourage this, but i really think that we should be happy to let the students decide what to do on this point. 10. I need to drop all uses of variables from the materials for Monday. (And later as well). The problem is that i had examples that stored references to forms and elements in variables, but these references would become stale when new pages were loaded. Yet, they would act and behave in large part like real references. This lead to endless confusion that was very difficult to explain. 11. The class was a big hit. In both classes, most of the students came up to me and shook my hand and thanked me for the class. I suspect that half of what they get comes not from my agenda, but simply from the fact they are given an authentic and rich environment. 12. In several cases, students found bugs in the test tool. One was a checkbox bug that Dan reported last week (it's fixed in CVS) in IEC. One student in each class found this, and i showed each of them how to fix it. They got a lot from this, from being shown the code and how things hooked together, so i'm glad i didn't worry about updating the IEC version for the class at the last minute. As the class proceeds, i try to explain to students how they can change the tool to work they want it to and in a few cases, we even do this. This is another big benefit of the class. I don't see that it's necessary to have the perfect tool. Rather, students get more from realizing that they can change it. 13. I have a stack of bug reports that students wrote during the class. I'll have to sort through them and will send basically minor timeclock bugs to Brian. Encouraging them to log bugs on index cards really seems to diffuse potential dissatisfaction. And there may be some good stuff in their too. Bret >Date: Thu, 13 May 2004 21:41:40 -0500 >To: xpau2004 at agilestl.com >From: Bret Pettichord >Subject: tutorial description: scripting web tests > >Brian, > >Here is the description for the XP/AU proceedings. I just taught versions >of this class on Monday and Tuesday, which allows me to be pretty sure >that they claims i make here will in fact be true. > >Bret > > > >Scripting Web Tests > >Bret Pettichord, Pettichord Consulting LLC, bret at pettichord.com >Brian Marick, Testing Foundations, marick at testing.com >Paul Rogers, Wireless Matrix, paul.rogers at shaw.ca >Jonathan Kohl, Westjet Airlines, jkohl at telusplanet.net > >Students in this tutorial learned how to write automated customer >tests for a web-based application. They used an open-source tool kit >to create tests that drive a web browser. Most completed a suite of >multiple tests by the conclusion of the tutorial. This hands-on >tutorial used open-source software installed on student-provided >laptops. Students learned to write these tests using the Ruby >scripting language and the Web Testing with Ruby toolkit, available at >http://rubyforge.org/projects/wtr/. A timeclock application was used >as the target of the tests. > >The tool kit consists of a library to make it convenient to access the >COM interface to Internet Explorer, including it's document object >mode. Similar methods could be used by any language the provides >access to COM (which is to say: most languages). Students also made >extensive use of the interactive Ruby interface, which facilitates >learning and exploration. (Similar interactive features are also >part of the Python and Tcl languages.) > >The hands-on learning experience allowed students to focus on the >issues of greatest personal interest. Many appreciated being able to >continue to experiment with and review the class exercises after the >completion of the tutorial. The tutorial instructors are contributors >to the tool kit and have used it in testing commercial software >developed by agile teams. > >______________________________________ > Bret Pettichord, Software Tester > Consultant - www.pettichord.com > Author - www.testinglessons.com > Blogger - www.io.com/~wazmo/blog > > Scripting for Testers Tutorial > Portland, Seattle, Orlando, Austin, Calgary > www.pettichord.com/training.html ______________________________________ Bret Pettichord, Software Tester Consultant - www.pettichord.com Author - www.testinglessons.com Blogger - www.io.com/~wazmo/blog Scripting for Testers Tutorial Portland, Seattle, Orlando, Austin, Calgary www.pettichord.com/training.html From marick at testing.com Fri May 14 10:05:51 2004 From: marick at testing.com (Brian Marick) Date: Fri, 14 May 2004 09:05:51 -0500 Subject: [Wtr-general] Re: tutorial description: scripting web tests In-Reply-To: <5.1.0.14.2.20040513215726.047dd7d8@127.0.0.1> References: <5.1.0.14.2.20040513215726.047dd7d8@127.0.0.1> Message-ID: I'm glad to hear this is working out well. I'll add id tags to the timeclock pages. On May 13, 2004, at 11:12 PM, Bret Pettichord wrote: > Today was the deadline for the tutorial description that will appear > in the XP/AU proceedings. Our submission is below. It had to be > written in the past tense, so what it really describes is what > happened this week when i taught the class in Portland and Seattle. I > will be teaching it again next week in Orlando. After that, i'll be > able to focus more fully on the next revision of the class. > > 1. Using blocks to specify rules that identified forms and elements > worked rather well, certainly better than the dedidicated functions > we'd used last October. But learning about when to identify using > actions vs. values vs. names takes time -- time that we won't have in > the half-day version in Calgary. So we are going to have to insert > unique names for all the forms and elements. We'll also have to add > named spans for text we want to verify. > > 2. The network is unnecessary. Having everyone test a single server is > pretty pointless because it starts crashing pretty fast. On monday, > when this started happening, i told them that they could run their own > local server, and five minutes later every one of them was doing this. > I didn't even set up the network on tuesday and won't be using it in > orlando next week. I'm not sure if the server is less stable that > previously. It might be, perhaps due to platform (windows vs. mac) or > the current version of Ruby (1.8.1-12). > > 3. A couple students were getting semi-reproducible seg faults in Ruby > when they were running harnessed tests. These were client side. I'm > going to see if i can track this down. In one case, the student was > using 1.8.1-13. > > 4. As the description below implies, we pretty much got most of the > students to put together a suite of tests by the end of the day. It's > rather satisfying to get this far and will be a challenge to get as > far in only a half-day. The biggest area of streamlining has got to > come from using named elements. This also means we can use more > Chris's original IEC code and concept. A lot of students wasn't better > logging. I know that Paul has this and this really needs to get folded > into the code we use for the class. > > 5. Even with named objects, the hardest part of the class is probably > getting students to associate the names of things with the actual > objects. After seeing Chris's Motunes demo, i'm thinking that it might > be possible to directly annotate objects by modifying the page via the > DOM. This is something i want to try out. It could be major cool thing > that accellerates the visibility of the testing hooks we use. > > 6. My policy has been that i need a TA for more than 12 students. On > Monday, i had 17 students, but the TA didn't work out. That was a bear > for me. I was constantly scrambling and didn't really have time to > reflect on what was happening. I had a TA (Dan) on Tuesday with about > the same number of students. Things went much much better. > > 7. In each class, i had one or two people who really ran with what we > have. We may have more of these at XP/AU, because of the kind of > conference it is. If so, we may want to have one of us run a > fast-track in some corner. > > 8. The current docs are too disorganized and unspecific for a few > people in each class. They get discouraged and expect better doc. I > plan to keep working on this. I think we actually addressed this > somewhat better in early classes. In other words, Brian addressed it > better. This is important to ensure success with people who are not > quick learners or experience programmers. I think they can be reached. > > 9. I've given up on trying to make people pair. Everyone wants to use > their own machine. In one case, two students came with one computer > between them and that was ok (and we should continue to encourage > this). XP/AU may give us a better context to encourage this, but i > really think that we should be happy to let the students decide what > to do on this point. > > 10. I need to drop all uses of variables from the materials for > Monday. (And later as well). The problem is that i had examples that > stored references to forms and elements in variables, but these > references would become stale when new pages were loaded. Yet, they > would act and behave in large part like real references. This lead to > endless confusion that was very difficult to explain. > > 11. The class was a big hit. In both classes, most of the students > came up to me and shook my hand and thanked me for the class. I > suspect that half of what they get comes not from my agenda, but > simply from the fact they are given an authentic and rich environment. > > 12. In several cases, students found bugs in the test tool. One was a > checkbox bug that Dan reported last week (it's fixed in CVS) in IEC. > One student in each class found this, and i showed each of them how to > fix it. They got a lot from this, from being shown the code and how > things hooked together, so i'm glad i didn't worry about updating the > IEC version for the class at the last minute. As the class proceeds, i > try to explain to students how they can change the tool to work they > want it to and in a few cases, we even do this. This is another big > benefit of the class. I don't see that it's necessary to have the > perfect tool. Rather, students get more from realizing that they can > change it. > > 13. I have a stack of bug reports that students wrote during the > class. I'll have to sort through them and will send basically minor > timeclock bugs to Brian. Encouraging them to log bugs on index cards > really seems to diffuse potential dissatisfaction. And there may be > some good stuff in their too. > > Bret > >> Date: Thu, 13 May 2004 21:41:40 -0500 >> To: xpau2004 at agilestl.com >> From: Bret Pettichord >> Subject: tutorial description: scripting web tests >> >> Brian, >> >> Here is the description for the XP/AU proceedings. I just taught >> versions of this class on Monday and Tuesday, which allows me to be >> pretty sure that they claims i make here will in fact be true. >> >> Bret >> >> >> >> Scripting Web Tests >> >> Bret Pettichord, Pettichord Consulting LLC, bret at pettichord.com >> Brian Marick, Testing Foundations, marick at testing.com >> Paul Rogers, Wireless Matrix, paul.rogers at shaw.ca >> Jonathan Kohl, Westjet Airlines, jkohl at telusplanet.net >> >> Students in this tutorial learned how to write automated customer >> tests for a web-based application. They used an open-source tool kit >> to create tests that drive a web browser. Most completed a suite of >> multiple tests by the conclusion of the tutorial. This hands-on >> tutorial used open-source software installed on student-provided >> laptops. Students learned to write these tests using the Ruby >> scripting language and the Web Testing with Ruby toolkit, available at >> http://rubyforge.org/projects/wtr/. A timeclock application was used >> as the target of the tests. >> >> The tool kit consists of a library to make it convenient to access the >> COM interface to Internet Explorer, including it's document object >> mode. Similar methods could be used by any language the provides >> access to COM (which is to say: most languages). Students also made >> extensive use of the interactive Ruby interface, which facilitates >> learning and exploration. (Similar interactive features are also >> part of the Python and Tcl languages.) >> >> The hands-on learning experience allowed students to focus on the >> issues of greatest personal interest. Many appreciated being able to >> continue to experiment with and review the class exercises after the >> completion of the tutorial. The tutorial instructors are contributors >> to the tool kit and have used it in testing commercial software >> developed by agile teams. >> >> ______________________________________ >> Bret Pettichord, Software Tester >> Consultant - www.pettichord.com >> Author - www.testinglessons.com >> Blogger - www.io.com/~wazmo/blog >> >> Scripting for Testers Tutorial >> Portland, Seattle, Orlando, Austin, Calgary >> www.pettichord.com/training.html > > ______________________________________ > Bret Pettichord, Software Tester > Consultant - www.pettichord.com > Author - www.testinglessons.com > Blogger - www.io.com/~wazmo/blog > > Scripting for Testers Tutorial > Portland, Seattle, Orlando, Austin, Calgary > www.pettichord.com/training.html > > > ----- Brian Marick Consulting, training, and contracting Mostly on agile methods with a testing slant www.testing.com, www.testing.com/cgi-bin/blog From marick at testing.com Mon May 17 11:46:49 2004 From: marick at testing.com (Brian Marick) Date: Mon, 17 May 2004 10:46:49 -0500 Subject: [Wtr-general] Re: tutorial description: scripting web tests In-Reply-To: <5.1.0.14.2.20040513215726.047dd7d8@127.0.0.1> References: <5.1.0.14.2.20040513215726.047dd7d8@127.0.0.1> Message-ID: <68B6B3EA-A819-11D8-9234-0003939BC6F4@testing.com> On May 13, 2004, at 11:12 PM, Bret Pettichord wrote: > A lot of students wasn't better logging. I know that Paul has this and > this really needs to get folded into the code we use for the class. ?? > 10. I need to drop all uses of variables from the materials for > Monday. (And later as well). The problem is that i had examples that > stored references to forms and elements in variables, but these > references would become stale when new pages were loaded. Yet, they > would act and behave in large part like real references. This lead to > endless confusion that was very difficult to explain. I wonder if there's a way to make variables hold something like "WeakRefs" The references would be invalidated by the next page load, so that dereferencing would fail in some gracefully explanatory way. It might even be cool if some of them (like a variable that points to the current page) could auto-update, so never get stale. ----- Brian Marick Consulting, training, and contracting Mostly on agile methods with a testing slant www.testing.com, www.testing.com/cgi-bin/blog From bret at pettichord.com Tue May 18 15:02:16 2004 From: bret at pettichord.com (Bret Pettichord) Date: Tue, 18 May 2004 14:02:16 -0500 Subject: [Wtr-general] Re: tutorial description: scripting web tests In-Reply-To: <68B6B3EA-A819-11D8-9234-0003939BC6F4@testing.com> References: <5.1.0.14.2.20040513215726.047dd7d8@127.0.0.1> <5.1.0.14.2.20040513215726.047dd7d8@127.0.0.1> Message-ID: <5.1.0.14.2.20040518134942.0328bba8@127.0.0.1> I taught the class again on Monday in Orlando, with Andy. Dropping variables, separating test methods (that return page contents) from assertions, and cleaning up some confusing language resulted in having the class take an hour less time than last week, covering the same material with possibly a somewhat slower group of students. So that is good. I strongly believe that closing with teaching students to create a suite for the various tests they've written makes a strong close for the tutorial. At 10:46 AM 5/17/2004, Brian Marick wrote: >On May 13, 2004, at 11:12 PM, Bret Pettichord wrote: > >>A lot of students wasn't better logging. I know that Paul has this and >>this really needs to get folded into the code we use for the class. > >?? I meant to say they want better logging. More of the same was expressed on Monday. We need to make a record of the basic calls made throuh IEC so that students know how the script progressed. Paul has logging in his version of IEC. It would also be useful for some students if we could point them to a logging module that was hooked up: many have a strong interest in such things. A longer version of the class might include a module on this topic. >>10. I need to drop all uses of variables from the materials for Monday. >>(And later as well). The problem is that i had examples that stored >>references to forms and elements in variables, but these references would >>become stale when new pages were loaded. Yet, they would act and behave >>in large part like real references. This lead to endless confusion that >>was very difficult to explain. > >I wonder if there's a way to make variables hold something like "WeakRefs" > >The references would be invalidated by the next page load, so that >dereferencing would fail in some gracefully explanatory way. It might >even be cool if some of them (like a variable that points to the current >page) could auto-update, so never get stale. I think the autoupdating kind would match expectations based on commercial tools. I think i'd implement them so that they wouldn't actually dereference until the methods on the objects were called. It would be good if we could put a hook into page loading -- but i don't know if this is possible. For example, this might be helpful to log this information. Or this might trigger an automatical call to iec.wait. >----- >Brian Marick >Consulting, training, and contracting >Mostly on agile methods with a testing slant >www.testing.com, www.testing.com/cgi-bin/blog ______________________________________ Bret Pettichord, Software Tester Consultant - www.pettichord.com Author - www.testinglessons.com Blogger - www.io.com/~wazmo/blog Scripting for Testers Tutorial Portland, Seattle, Orlando, Austin, Calgary www.pettichord.com/training.html