From aaron at tenderlovemaking.com Thu Mar 5 12:54:43 2009 From: aaron at tenderlovemaking.com (Aaron Patterson) Date: Thu, 5 Mar 2009 09:54:43 -0800 Subject: [Mechanize-users] [ANN] mechanize 0.9.2 Released Message-ID: <20090305175443.GA5166@Jordan2.local> mechanize version 0.9.2 has been released! * * The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history. Changes: ### 0.9.2 / 2009/03/05 * New Features: * Mechanize#submit and Form#submit take arbitrary headers(thanks penguincoder) * Bug Fixes: * Fixed a bug with bad cookie parsing * Form::RadioButton#click unchecks other buttons (RF #24159) * Fixed problems with Iconv (RF #24190, RF #24192, RF #24043) * POST parameters should be CGI escaped * Made Content-Type match case insensitive (Thanks Kelly Reynolds) * Non-string form parameters work * * -- Aaron Patterson http://tenderlovemaking.com/ From harmaarts at gmail.com Tue Mar 10 15:12:31 2009 From: harmaarts at gmail.com (Harm Aarts) Date: Tue, 10 Mar 2009 20:12:31 +0100 Subject: [Mechanize-users] weird problem with cookies Message-ID: Hi, I wrote a script to login to some page. This works fine on my local machine, but fails on my server. Both run Mechanize 0.9.0 and Nokogiri 1.2.1. Turning debugging on I get for the succeeding run this pastie: http://pastie.org/413049. For the failing run I get this one: http://pastie.org/413052. Note the absence of the cookie request-header in the failing run (from the succeeding run): D, [2009-03-10T19:22:51.409080 #99291] DEBUG -- : request-header: cookie => orangeSessionID=SID%3D178CF332CDB2DAAB051AB16E7A675073227EE45A09DD8AF280A0BA2D64D03E2782EF51FFB3E0756D44AEC76F18668B182179A1A2F06C6C9D4B976C4A322EF6CF%26SID1%3DBA218F59EEEF8F541F9464732E16A148 How is this possible? Both save the cookie created in the previous request: D, [2009-03-10T19:22:51.389433 #99291] DEBUG -- : saved cookie: orangeSessionID=SID%3D178CF332CDB2DAAB051AB16E7A675073227EE45A09DD8AF280A0BA2D64D03E2782EF51FFB3E0756D44AEC76F18668B182179A1A2F06C6C9D4B976C4A322EF6CF%26SID1%3DBA218F59EEEF8F541F9464732E16A148 Where does Mechanize save it's cookies? Maybe it is a permissions issue? And how does it determine when to send the cookie header? I am at a loss and any help would be appreciated. With kind regards, Harm -------------- next part -------------- An HTML attachment was scrubbed... URL: From harmaarts at gmail.com Wed Mar 11 07:03:58 2009 From: harmaarts at gmail.com (Harm Aarts) Date: Wed, 11 Mar 2009 12:03:58 +0100 Subject: [Mechanize-users] weird problem with cookies Message-ID: In the good tradition of the internet I'll answer my own question. After a couple of hours digging through the Mechanize source I found out that the cookie set initially was deleted again because my server time was an hour ahead. This particular cookie had an expiry time less than an hour causing it to be deleted. I fixed the problem by adjusting the server time. However I believe this might be a problem when scraping in different time zones... Anyway make sure the time on your server is correct. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at malditainternet.com Wed Mar 11 10:18:36 2009 From: martin at malditainternet.com (Martin Sarsale) Date: Wed, 11 Mar 2009 11:18:36 -0300 Subject: [Mechanize-users] upload big files via HTTP POST Message-ID: Guys, I'm trying to upload some big files (350mb) via HTTP POST using mechanize and I'm getting this: /usr/lib/ruby/1.8/mechanize/form.rb:102:in `+': failed to allocate memory (NoMemoryError) from /usr/lib/ruby/1.8/mechanize/form.rb:102:in `request_data' from /usr/lib/ruby/1.8/mechanize.rb:255:in `post_form' from /usr/lib/ruby/1.8/mechanize.rb:201:in `submit' any hints? thanks in advance -- Martin Sarsale - martin at malditainternet.com From gsmoraes2 at gmail.com Wed Mar 11 11:08:40 2009 From: gsmoraes2 at gmail.com (gmoraes) Date: Wed, 11 Mar 2009 12:08:40 -0300 Subject: [Mechanize-users] problem scrapping ATnT site (Matt White) Message-ID: Try to use firebug to assist you finding these changes. I never used AT&T website, but you may need to login and find the download url using firebug as I did: http://zenmachine.wordpress.com/2007/11/11/scraping-with-firebug-and-wwwmechanize/ regards, gm On Tue, Mar 10, 2009 at 4:12 PM, wrote: > Send Mechanize-users mailing list submissions to > mechanize-users at rubyforge.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://rubyforge.org/mailman/listinfo/mechanize-users > or, via email, send a message with subject or body 'help' to > mechanize-users-request at rubyforge.org > > You can reach the person managing the list at > mechanize-users-owner at rubyforge.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Mechanize-users digest..." > > > Today's Topics: > > 1. Re: problem scrapping ATnT site (Matt White) > 2. need guidance on following links to download files (Reid Thompson) > 3. Re: problem scrapping ATnT site (subhransu behera) > 4. Mechanize, history and memory (barsalou) > 5. [ANN] mechanize 0.9.2 Released (Aaron Patterson) > 6. weird problem with cookies (Harm Aarts) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 24 Feb 2009 07:18:09 -0800 (PST) > From: Matt White > Subject: Re: [Mechanize-users] problem scrapping ATnT site > To: Ruby Mechanize Users List > Message-ID: <190289.59284.qm at web53309.mail.re2.yahoo.com> > Content-Type: text/plain; charset="us-ascii" > > One thing to be aware of is that Mechanize doesn't interpret Javascript. If > the page changes dynamically as you select things on the page, Mechanize > will not recognize these changes. If this is the problem you are having, you > will have to have the script do whatever the Javascript is doing to get > everything right. > > Matt White > > > > > ________________________________ > From: subhransu behera > To: mechanize-users at rubyforge.org > Sent: Tuesday, February 24, 2009 1:32:08 AM > Subject: [Mechanize-users] problem scrapping ATnT site > > Hi, > > I am trying to download the past call details from ATnT site > in csv format. > > It requires to select the bill period and click on a radio button. > Then clicking on "Submit" link downloads the call summary for > that period. > > I tried to do it in mechanize in the following way, but it download > the src of the page in stead of downloading the actual CSV file. > > # get the download page > > page_download = agent.get " > https://www.wireless.att.com/view/billPayDownloadDetail.doview?execdownloadPage=true > " > > # get the form for bill_period and select a bill period > > bill_period_form = page_download.forms[2] > bill_period_form.field.options[2].select > > # click on the csv radio button > > download_format_form = page_download.forms[3] > download_format_form.radiobuttons[1].click > > # click on the submit link that downloads the csv file. > > download_file = agent.click download_page.search("a")[41] > download_file.save_as(".csv") > > The problem I am facing in the above code is: > > + Doesn't do anything special after selecting a particular bill period from > the select options. > + Download the page source in stead of downloading the actual csv file. > > Can you suggest something? Am I missing something here? > > Thanks, > Shubh > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://rubyforge.org/pipermail/mechanize-users/attachments/20090224/6e39be00/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Tue, 24 Feb 2009 10:19:22 -0500 > From: Reid Thompson > Subject: [Mechanize-users] need guidance on following links to > download files > To: mechanize-users at rubyforge.org > Message-ID: <1235488762.32688.25.camel at raker> > Content-Type: text/plain > > The script below is a mod of one i found via google. I'm trying to > figure out what i'm missing in order to download the files associated > with the links. > > > > require 'mechanize' > > agent = WWW::Mechanize.new > pagent = WWW::Mechanize.new > agent.get("http://www.daytrotter.com/songs?offset=60/") > links = agent.page.search('a') > hrefs = links.map { |m| m['href'] }.select { |u| u =~ /\.mp3.link$/ } # > just links ending in mfile > #puts hrefs > #FileUtils.mkdir_p('daytrotter') # keep it neat > hrefs.each { |mfile| > if mfile.match(/^\/download/) then next end > #puts mfile > filename = "#{mfile.split('/')[-1]}" > filename.gsub!('.link','') > > puts "Saving #{mfile} as #{filename}" > > agent.get(mfile).save_as(filename) > } > > This results in output of the following format: > Saving > http://daytrotter.com/file_download/76/TwoGallants_DaytrotterSession_2.mp3.linkas TwoGallants_DaytrotterSession_2.mp3 > > I can't seem to get the final result to resolve to the actual file... > I'd appreciate any pointers. > > Thanks, > reid > > > > ------------------------------ > > Message: 3 > Date: Wed, 25 Feb 2009 00:53:36 +0530 > From: subhransu behera > Subject: Re: [Mechanize-users] problem scrapping ATnT site > To: Ruby Mechanize Users List > Message-ID: > <8f00add50902241123r403fc219ua5f30a9110b6e615 at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Matt, > > I did exactly what you suggested. And now it works as expected. > Thanks a ton buddy! > > Regards, > Shubh > > On Tue, Feb 24, 2009 at 8:48 PM, Matt White >wrote: > > > One thing to be aware of is that Mechanize doesn't interpret Javascript. > If > > the page changes dynamically as you select things on the page, Mechanize > > will not recognize these changes. If this is the problem you are having, > you > > will have to have the script do whatever the Javascript is doing to get > > everything right. > > > > Matt White > > > > ------------------------------ > > *From:* subhransu behera > > *To:* mechanize-users at rubyforge.org > > *Sent:* Tuesday, February 24, 2009 1:32:08 AM > > *Subject:* [Mechanize-users] problem scrapping ATnT site > > > > Hi, > > > > I am trying to download the past call details from ATnT site > > in csv format. > > > > It requires to select the bill period and click on a radio button. > > Then clicking on "Submit" link downloads the call summary for > > that period. > > > > I tried to do it in mechanize in the following way, but it download > > the src of the page in stead of downloading the actual CSV file. > > > > # get the download page > > > > page_download = agent.get " > > > https://www.wireless.att.com/view/billPayDownloadDetail.doview?execdownloadPage=true > > " > > > > # get the form for bill_period and select a bill period > > > > bill_period_form = page_download.forms[2] > > bill_period_form.field.options[2].select > > > > # click on the csv radio button > > > > download_format_form = page_download.forms[3] > > download_format_form.radiobuttons[1].click > > > > # click on the submit link that downloads the csv file. > > > > download_file = agent.click download_page.search("a")[41] > > download_file.save_as(".csv") > > > > The problem I am facing in the above code is: > > > > + Doesn't do anything special after selecting a particular bill period > from > > the select options. > > + Download the page source in stead of downloading the actual csv file. > > > > Can you suggest something? Am I missing something here? > > > > Thanks, > > Shubh > > > > > > _______________________________________________ > > Mechanize-users mailing list > > Mechanize-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mechanize-users > > > > > > -- > Innovator, Pune - India > Phone : (+91)-98605-59976 > Blog : http://sbehera.livejournal.com/ > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://rubyforge.org/pipermail/mechanize-users/attachments/20090225/fdbd4090/attachment-0001.html > > > > ------------------------------ > > Message: 4 > Date: Tue, 24 Feb 2009 13:24:53 -0900 > From: barsalou > Subject: [Mechanize-users] Mechanize, history and memory > To: Ruby Mechanize Users List > Message-ID: <20090224132453.4yf9cr4so4ckw84g at lcgalaska.com> > Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; > format="flowed" > > I recently wrote a script to read a web page over and over. I ran > into an issue where the script would stop for seemingly an unknown > reason. > > Turns out "browser history" was continually growing. > > The answer of course is to set agent.max_history to some lower number, > in my case one. > > Have you ever considered implementing a warning or changing the > default to max_history to something that won't eat up memory? > > Maybe a note in GUIDE.txt? > > I haven't tested 0.9.1 yet, so you may have changed the default...but > the docs for 0.9.1 don't seem to be very specific about that. > > I'll provide a patch, but wanted to see which way you'd want to go. > > Mike B. > > ---------------------------------------------------------------- > This message was sent using IMP, the Internet Messaging Program. > > > > ------------------------------ > > Message: 5 > Date: Thu, 5 Mar 2009 09:54:43 -0800 > From: Aaron Patterson > Subject: [Mechanize-users] [ANN] mechanize 0.9.2 Released > To: Seattle Ruby Brigade! , > ruby-talk at ruby-lang.org, mechanize-users at rubyforge.org > Message-ID: <20090305175443.GA5166 at Jordan2.local> > Content-Type: text/plain; charset=us-ascii > > mechanize version 0.9.2 has been released! > > * > * > > The Mechanize library is used for automating interaction with websites. > Mechanize automatically stores and sends cookies, follows redirects, > can follow links, and submit forms. Form fields can be populated and > submitted. Mechanize also keeps track of the sites that you have visited > as > a history. > > Changes: > > ### 0.9.2 / 2009/03/05 > > * New Features: > * Mechanize#submit and Form#submit take arbitrary headers(thanks > penguincoder) > > * Bug Fixes: > * Fixed a bug with bad cookie parsing > * Form::RadioButton#click unchecks other buttons (RF #24159) > * Fixed problems with Iconv (RF #24190, RF #24192, RF #24043) > * POST parameters should be CGI escaped > * Made Content-Type match case insensitive (Thanks Kelly Reynolds) > * Non-string form parameters work > > * > * > > -- > Aaron Patterson > http://tenderlovemaking.com/ > > > ------------------------------ > > Message: 6 > Date: Tue, 10 Mar 2009 20:12:31 +0100 > From: Harm Aarts > Subject: [Mechanize-users] weird problem with cookies > To: mechanize-users at rubyforge.org > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > Hi, > > I wrote a script to login to some page. This works fine on my local > machine, > but fails on my server. Both run Mechanize 0.9.0 and Nokogiri 1.2.1. > Turning debugging on I get for the succeeding run this pastie: > http://pastie.org/413049. For the failing run I get this one: > http://pastie.org/413052. Note the absence of the cookie request-header in > the failing run (from the succeeding run): > D, [2009-03-10T19:22:51.409080 #99291] DEBUG -- : request-header: cookie => > > orangeSessionID=SID%3D178CF332CDB2DAAB051AB16E7A675073227EE45A09DD8AF280A0BA2D64D03E2782EF51FFB3E0756D44AEC76F18668B182179A1A2F06C6C9D4B976C4A322EF6CF%26SID1%3DBA218F59EEEF8F541F9464732E16A148 > > How is this possible? Both save the cookie created in the previous request: > D, [2009-03-10T19:22:51.389433 #99291] DEBUG -- : saved cookie: > > orangeSessionID=SID%3D178CF332CDB2DAAB051AB16E7A675073227EE45A09DD8AF280A0BA2D64D03E2782EF51FFB3E0756D44AEC76F18668B182179A1A2F06C6C9D4B976C4A322EF6CF%26SID1%3DBA218F59EEEF8F541F9464732E16A148 > > Where does Mechanize save it's cookies? Maybe it is a permissions issue? > And > how does it determine when to send the cookie header? > I am at a loss and any help would be appreciated. > > With kind regards, > Harm > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://rubyforge.org/pipermail/mechanize-users/attachments/20090310/5f5dcc34/attachment.html > > > > ------------------------------ > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > > End of Mechanize-users Digest, Vol 24, Issue 1 > ********************************************** > -- More cowbell, please ! -------------- next part -------------- An HTML attachment was scrubbed... URL: From therapshow at gmail.com Tue Mar 17 06:42:27 2009 From: therapshow at gmail.com (Frank Gjildberg) Date: Tue, 17 Mar 2009 11:42:27 +0100 Subject: [Mechanize-users] Get a list of users, and for each user, visit a URL Message-ID: <656c2abe0903170342me713d00x73d315992a620bf4@mail.gmail.com> Hey, I got this smalltime social network here with about 200 users. In the beginning we didn't have the "make all users friends of the admin" function that we do now, so I have to manually add these 200 users as friends of the admin. My friend coded this thing in PHP and I ain't got much insight into its underpinnings (but I know that it's a mess, and that the design is horrible, but anyway), I figured this'd be a great way to learn Mechanize. Basically I need to create a script that: 1) Goes to http://members.clebration.com 2) Gets a list of all the users on that page, each user looking like http://my.clebration.com/some_user 3) Do the same for http://members.clebration.com/?page=2 and all the way up to infinite 4) Then, for each user, simply go to http://my.clebration.com/some_user/friendship/add_request 5) The system should respond with a blank page saying "Friend request to some_user is sent" But where do I start? What tutorials do I need to read? Are there any sample scripts? Regards, Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From therapshow at gmail.com Tue Mar 17 08:58:13 2009 From: therapshow at gmail.com (Frank Gjildberg) Date: Tue, 17 Mar 2009 13:58:13 +0100 Subject: [Mechanize-users] Get a list of users, and for each user, visit a URL Message-ID: <656c2abe0903170558n39d5c8bbve538339792e9b88@mail.gmail.com> Hey, I got this smalltime social network here with about 200 users. In the beginning we didn't have the "make all users friends of the admin" function that we do now, so I have to manually add these 200 users as friends of the admin. My friend coded this thing in PHP and I ain't got much insight into its underpinnings (but it's a mess), so I figured this'd be a great way to learn Mechanize. Basically I need to create a script that: 1) Goes to http://members.clebration.com 2) Gets a list of all the users on that page, each user looking like http://my.clebration.com/some_user 3) Do the same for http://members.clebration.com/?page=2 and all the way up to infinite 4) Then, for each user, simply go to http://my.clebration.com/some_user/friendship/add_request 5) The system should respond with a blank page saying "Friend request to some_user is sent" But where do I start? What tutorials do I need to read? Regards, Frank From therapshow at gmail.com Tue Mar 17 09:02:20 2009 From: therapshow at gmail.com (Frank Gjildberg) Date: Tue, 17 Mar 2009 14:02:20 +0100 Subject: [Mechanize-users] Sorry for sending the same mail twice Message-ID: <656c2abe0903170602j3ae18dadxf4ccd99d17eb3b8b@mail.gmail.com> I didn't know the first one got there. Thanks. Regards, Frank From whitethunder922 at yahoo.com Tue Mar 17 11:33:10 2009 From: whitethunder922 at yahoo.com (Matt White) Date: Tue, 17 Mar 2009 08:33:10 -0700 (PDT) Subject: [Mechanize-users] Get a list of users, and for each user, visit a URL In-Reply-To: <656c2abe0903170342me713d00x73d315992a620bf4@mail.gmail.com> References: <656c2abe0903170342me713d00x73d315992a620bf4@mail.gmail.com> Message-ID: <421674.61707.qm@web53306.mail.re2.yahoo.com> Here's a good place to start: http://letmegooglethatforyou.com/?q=ruby+mechanize+tutorial ________________________________ From: Frank Gjildberg To: mechanize-users at rubyforge.org Sent: Tuesday, March 17, 2009 4:42:27 AM Subject: [Mechanize-users] Get a list of users, and for each user, visit a URL Hey, I got this smalltime social network here with about 200 users. In the beginning we didn't have the "make all users friends of the admin" function that we do now, so I have to manually add these 200 users as friends of the admin. My friend coded this thing in PHP and I ain't got much insight into its underpinnings (but I know that it's a mess, and that the design is horrible, but anyway), I figured this'd be a great way to learn Mechanize. Basically I need to create a script that: 1) Goes to http://members.clebration.com 2) Gets a list of all the users on that page, each user looking like http://my.clebration.com/some_user 3) Do the same for http://members.clebration.com/?page=2 and all the way up to infinite 4) Then, for each user, simply go to http://my.clebration.com/some_user/friendship/add_request 5) The system should respond with a blank page saying "Friend request to some_user is sent" But where do I start? What tutorials do I need to read? Are there any sample scripts? Regards, Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From therapshow at gmail.com Wed Mar 18 04:39:00 2009 From: therapshow at gmail.com (Frank Gjildberg) Date: Wed, 18 Mar 2009 09:39:00 +0100 Subject: [Mechanize-users] Get a list of users, and for each user, visit a URL In-Reply-To: <421674.61707.qm@web53306.mail.re2.yahoo.com> References: <656c2abe0903170342me713d00x73d315992a620bf4@mail.gmail.com> <421674.61707.qm@web53306.mail.re2.yahoo.com> Message-ID: <656c2abe0903180139x443b4952j6709787c75ec2e9d@mail.gmail.com> 2009/3/17 Matt White : > Here's a good place to start: > > http://letmegooglethatforyou.com/?q=ruby+mechanize+tutorial > Right. Well, thanks a lot, I'll report back ASAP. From ross at roscommonhq.com Tue Mar 24 19:23:37 2009 From: ross at roscommonhq.com (Ross Cameron) Date: Wed, 25 Mar 2009 10:23:37 +1100 Subject: [Mechanize-users] Capturing the result of submits Message-ID: <49C96B79.7060103@roscommonhq.com> Hi I apologize up front if this is a dumb question because I guess Ajax and thus Javascript is involved. Is there any way to capture the result of a submit if the current page is modified as result of the submit? For example, a couple of input fields, a submit and the result turns up in a modified
and which it looks like Mechanize doesn't get. I hope I haven't answered my own question! Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.schaffer at gmail.com Tue Mar 24 21:17:22 2009 From: mat.schaffer at gmail.com (Mat Schaffer) Date: Tue, 24 Mar 2009 21:17:22 -0400 Subject: [Mechanize-users] Capturing the result of submits In-Reply-To: <49C96B79.7060103@roscommonhq.com> References: <49C96B79.7060103@roscommonhq.com> Message-ID: If the page doesn't refresh then javascript is involved. Of course, that's not to say you couldn't parse the javascript response in ruby and get the information you're looking for. I've done it a lot with good results. I actually scripted most of the major webmail systems with mechanize a few years back and AOL's webmail was the only javascript nut I couldn't crack. -Mat On Mar 24, 2009, at 7:23 PM, Ross Cameron wrote: > Hi > > I apologize up front if this is a dumb question because I guess Ajax > and thus Javascript is involved. > > Is there any way to capture the result of a submit if the current > page is modified as result of the submit? > > For example, a couple of input fields, a submit and the result turns > up in a modified
and which it looks like Mechanize doesn't get. > > I hope I haven't answered my own question! > > Regards > > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users From ross at roscommonhq.com Tue Mar 24 23:29:37 2009 From: ross at roscommonhq.com (Ross Cameron) Date: Wed, 25 Mar 2009 14:29:37 +1100 Subject: [Mechanize-users] Capturing the result of submits In-Reply-To: References: <49C96B79.7060103@roscommonhq.com> Message-ID: <49C9A521.7050609@roscommonhq.com> Hi Matt Many thanks. I sort of went and solved it in the case of a form GET method by scripting the full path for the form action. This wasn't too difficult because the action url can be discovered by inspection. POST is somewhat more difficult but I assume there are ways of finding out what is passed and setting those. But what would be nicer, if you wouldn't mind, is pointing me in the right direction to get at the JavaScript response - not sure how to do that. That would nail it. Regards Ross Mat Schaffer wrote: > If the page doesn't refresh then javascript is involved. Of course, > that's not to say you couldn't parse the javascript response in ruby > and get the information you're looking for. I've done it a lot with > good results. I actually scripted most of the major webmail systems > with mechanize a few years back and AOL's webmail was the only > javascript nut I couldn't crack. > -Mat > > On Mar 24, 2009, at 7:23 PM, Ross Cameron wrote: > >> Hi >> >> I apologize up front if this is a dumb question because I guess Ajax >> and thus Javascript is involved. >> >> Is there any way to capture the result of a submit if the current >> page is modified as result of the submit? >> >> For example, a couple of input fields, a submit and the result turns >> up in a modified
and which it looks like Mechanize doesn't get. >> >> I hope I haven't answered my own question! >> >> Regards >> >> >> >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users -- ------------------------------------------------------------------------ Ross Cameron | Director Roscommon Pty Ltd | ABN 85 099 499 840 p: +61 2 9016 4133 | m: +61 4 3312 9087 | f: +61 2 9420 4525 | w: www.roscommonhq.com | AIM: rossppc Roscommon uses the five sentences email reply policy. Please consider our environment before printing this email. NOTE: This email and any attachments may be confidential. If received in error, please delete the email. Because emails and attachments may be interfered with, may contain computer viruses or other defects and may not be successfully replicated on other systems, you must be cautious. Roscommon cannot guarantee that what you receive is what we sent. If you have any doubts about the authenticity of an email from Roscommon, please contact us immediately. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikemondragon at gmail.com Tue Mar 24 23:35:16 2009 From: mikemondragon at gmail.com (Mike Mondragon) Date: Tue, 24 Mar 2009 20:35:16 -0700 Subject: [Mechanize-users] Capturing the result of submits In-Reply-To: References: <49C96B79.7060103@roscommonhq.com> Message-ID: <967d3b9a0903242035t1c4313f0pe7552c264b7246ba@mail.gmail.com> On Tue, Mar 24, 2009 at 6:17 PM, Mat Schaffer wrote: > If the page doesn't refresh then javascript is involved. Of course, that's > not to say you couldn't parse the javascript response in ruby and get the > information you're looking for. I've done it a lot with good results. I > actually scripted most of the major webmail systems with mechanize a few > years back and AOL's webmail was the only javascript nut I couldn't crack. I think a lot of people came up against the problem with scraping AOL webmail. They had an edgecase for URL formatting that Mechanize was handling a bit differently than a real web browser. Here's the duck punch on WWW::Mechanize::to_absolute_uri that can be used to scrape on AOL webmail properly. http://github.com/contentfree/blackbook/blob/ca9d90ff1be576bdbb42a1c6b81940d81840ed9d/lib/blackbook/importer/page_scraper.rb Mike > -Mat > > On Mar 24, 2009, at 7:23 PM, Ross Cameron wrote: > >> Hi >> >> I apologize up front if this is a dumb question because I guess Ajax and >> thus Javascript is involved. >> >> Is there any way to capture the result of a submit if the current page is >> modified as result of the submit? >> >> For example, a couple of input fields, a submit and the result turns up in >> a modified
and which it looks like Mechanize doesn't get. >> >> I hope I haven't answered my own question! >> >> Regards >> >> >> >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > From ross at roscommonhq.com Tue Mar 24 23:53:11 2009 From: ross at roscommonhq.com (Ross Cameron) Date: Wed, 25 Mar 2009 14:53:11 +1100 Subject: [Mechanize-users] Capturing the result of submits In-Reply-To: <967d3b9a0903242035t1c4313f0pe7552c264b7246ba@mail.gmail.com> References: <49C96B79.7060103@roscommonhq.com> Message-ID: <49C9AAA7.8080003@roscommonhq.com> Mike Most helpful. And a very elegant solution to the mechanize uri problem. Regards Ross Mike Mondragon wrote: > On Tue, Mar 24, 2009 at 6:17 PM, Mat Schaffer wrote: > >> If the page doesn't refresh then javascript is involved. Of course, that's >> not to say you couldn't parse the javascript response in ruby and get the >> information you're looking for. I've done it a lot with good results. I >> actually scripted most of the major webmail systems with mechanize a few >> years back and AOL's webmail was the only javascript nut I couldn't crack. >> > > I think a lot of people came up against the problem with scraping AOL > webmail. They had an edgecase for URL formatting that Mechanize was > handling a bit differently than a real web browser. Here's the duck > punch on WWW::Mechanize::to_absolute_uri that can be used to scrape on > AOL webmail properly. > > http://github.com/contentfree/blackbook/blob/ca9d90ff1be576bdbb42a1c6b81940d81840ed9d/lib/blackbook/importer/page_scraper.rb > > Mike > > >> -Mat >> >> On Mar 24, 2009, at 7:23 PM, Ross Cameron wrote: >> >> >>> Hi >>> >>> I apologize up front if this is a dumb question because I guess Ajax and >>> thus Javascript is involved. >>> >>> Is there any way to capture the result of a submit if the current page is >>> modified as result of the submit? >>> >>> For example, a couple of input fields, a submit and the result turns up in >>> a modified
and which it looks like Mechanize doesn't get. >>> >>> I hope I haven't answered my own question! >>> >>> Regards >>> >>> >>> >>> _______________________________________________ >>> Mechanize-users mailing list >>> Mechanize-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/mechanize-users >>> >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users >> >> > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > -- ------------------------------------------------------------------------ Ross Cameron | Director Roscommon Pty Ltd | ABN 85 099 499 840 p: +61 2 9016 4133 | m: +61 4 3312 9087 | f: +61 2 9420 4525 | w: www.roscommonhq.com | AIM: rossppc Roscommon uses the five sentences email reply policy. Please consider our environment before printing this email. NOTE: This email and any attachments may be confidential. If received in error, please delete the email. Because emails and attachments may be interfered with, may contain computer viruses or other defects and may not be successfully replicated on other systems, you must be cautious. Roscommon cannot guarantee that what you receive is what we sent. If you have any doubts about the authenticity of an email from Roscommon, please contact us immediately. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.schaffer at gmail.com Wed Mar 25 07:58:21 2009 From: mat.schaffer at gmail.com (Mat Schaffer) Date: Wed, 25 Mar 2009 07:58:21 -0400 Subject: [Mechanize-users] Capturing the result of submits In-Reply-To: <967d3b9a0903242035t1c4313f0pe7552c264b7246ba@mail.gmail.com> References: <49C96B79.7060103@roscommonhq.com> <967d3b9a0903242035t1c4313f0pe7552c264b7246ba@mail.gmail.com> Message-ID: On Mar 24, 2009, at 11:35 PM, Mike Mondragon wrote: > I think a lot of people came up against the problem with scraping AOL > webmail. They had an edgecase for URL formatting that Mechanize was > handling a bit differently than a real web browser. Here's the duck > punch on WWW::Mechanize::to_absolute_uri that can be used to scrape on > AOL webmail properly. > > http://github.com/contentfree/blackbook/blob/ca9d90ff1be576bdbb42a1c6b81940d81840ed9d/lib/blackbook/importer/page_scraper.rb > > Mike ha! Nice one, man. Sadly the project I was doing it for is long gone, but thanks for this lovely gem. I'll sure be bookmarking this for later! -Mat From mat.schaffer at gmail.com Wed Mar 25 08:03:03 2009 From: mat.schaffer at gmail.com (Mat Schaffer) Date: Wed, 25 Mar 2009 08:03:03 -0400 Subject: [Mechanize-users] Capturing the result of submits In-Reply-To: <49C9A521.7050609@roscommonhq.com> References: <49C96B79.7060103@roscommonhq.com> <49C9A521.7050609@roscommonhq.com> Message-ID: On Mar 24, 2009, at 11:29 PM, Ross Cameron wrote: > Hi Matt > > Many thanks. I sort of went and solved it in the case of a form GET > method by scripting the full path for the form action. This wasn't > too difficult because the action url can be discovered by > inspection. POST is somewhat more difficult but I assume there are > ways of finding out what is passed and setting those. > > But what would be nicer, if you wouldn't mind, is pointing me in the > right direction to get at the JavaScript response - not sure how to > do that. That would nail it. I often use Charles in these situations (http:// www.charlesproxy.com/). There are other options too like TamperData or Fiddler for windows, but charles feels a bit more organized/reliable and usually the 30 minute time limit is enough to get simple jobs done. Once you've figured out the right request, the response can be obtained from #body in mechanize like usual. -Mat -------------- next part -------------- An HTML attachment was scrubbed... URL: From eatme444 at hotmail.com Thu Mar 26 18:30:53 2009 From: eatme444 at hotmail.com (Anthony F) Date: Thu, 26 Mar 2009 15:30:53 -0700 Subject: [Mechanize-users] Can't get this site to open Message-ID: The site is: http://www.bcbid.gov.bc.ca It's a weird, complicated piece of crap full of frames and cookies and all sorts of god-awful javascript navigation. However, before I even get into that stuff I can't even get the site to open in Mechanize. Can anyone else get this working, or is it just me? _________________________________________________________________ Experience all of the new features, and Reconnect with your life. http://go.microsoft.com/?linkid=9650730 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.schaffer at gmail.com Thu Mar 26 23:23:15 2009 From: mat.schaffer at gmail.com (Mat Schaffer) Date: Thu, 26 Mar 2009 23:23:15 -0400 Subject: [Mechanize-users] Can't get this site to open In-Reply-To: References: Message-ID: <2D8FA06C-17DD-4424-B854-B455813268F6@gmail.com> Loads for me, but it's also got a javascript redirect in there. You'll have to do that yourself with something like agent.click(page.links.first) >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body => "\r\n\r\n\r\nRe-directing to BC Bid...\r\n\r\n\r\nIf this page does not automatically re-direct you to BC Bid®,
\r\nplease click here.\r\n\r\n" On Mar 26, 2009, at 6:30 PM, Anthony F wrote: > > The site is: http://www.bcbid.gov.bc.ca > > It's a weird, complicated piece of crap full of frames and cookies > and all sorts of god-awful javascript navigation. However, before I > even get into that stuff I can't even get the site to open in > Mechanize. Can anyone else get this working, or is it just me? > > Make your Messenger window look the way you want. Express Yourself! > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From eatme444 at hotmail.com Fri Mar 27 04:02:49 2009 From: eatme444 at hotmail.com (Anthony F) Date: Fri, 27 Mar 2009 01:02:49 -0700 Subject: [Mechanize-users] Can't get this site to open Message-ID: Interesting. I tried it with mechanize 0.8.5 and it seemed to work fine. With 0.9.2 it opens the page, but doesn't seem to parse it properly (ie. frames => nil, link => nil, etc). What version are you using? Mat Schaffer wrote: Loads for me, but it's also got a javascript redirect in there. You'll have to do that yourself with something like agent.click(page.links.first) >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body Re-directing to BC Bid... => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome'">\r\nIf this page does not automatically re-direct you to BC Bid?, \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here.\r\n\r\n" On Mar 26, 2009, at 6:30 PM, Anthony F wrote: The site is: http://www.bcbid.gov.bc.ca It's a weird, complicated piece of crap full of frames and cookies and all sorts of god-awful javascript navigation. However, before I even get into that stuff I can't even get the site to open in Mechanize. Can anyone else get this working, or is it just me? Make your Messenger window look the way you want. Express Yourself! _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _________________________________________________________________ Reunite with the people closest to you, chat face to face with Messenger. http://go.microsoft.com/?linkid=9650736 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.schaffer at gmail.com Fri Mar 27 11:32:26 2009 From: mat.schaffer at gmail.com (Mat Schaffer) Date: Fri, 27 Mar 2009 11:32:26 -0400 Subject: [Mechanize-users] Can't get this site to open In-Reply-To: References: Message-ID: <4E8A088D-3F6A-4FD6-BB5D-78A9BE7CE2F7@gmail.com> My previous was 0.9.0, but it works for me with 0.9.2 as well: >> require 'mechanize' => true >> WWW::Mechanize::VERSION => "0.9.2" >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body => "\r\n\r\n\r\nRe-directing to BC Bid...\r\n\r\n\r\nIf this page does not automatically re-direct you to BC Bid®,
\r\nplease click here.\r\n\r\n" I don't see any frames here. Do you maybe have a transparent web proxy where you are? What does your response look like? You might want to check using curl too. -Mat On Mar 27, 2009, at 4:02 AM, Anthony F wrote: > Interesting. I tried it with mechanize 0.8.5 and it seemed to work > fine. With 0.9.2 it opens the page, but doesn't seem to parse it > properly (ie. frames => nil, link => nil, etc). What version are > you using? > > Mat Schaffer wrote: >> >> Loads for me, but it's also got a javascript redirect in there. >> You'll have to do that yourself with something like >> agent.click(page.links.first) >> >> >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body >> => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/ >> welcome'">\r\nIf this page does not automatically re-direct you to >> BC Bid?, >> \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here. >> \r\n\r\n" >> >> >> On Mar 26, 2009, at 6:30 PM, Anthony F wrote: >> >>> >>> The site is: http://www.bcbid.gov.bc.ca >>> >>> It's a weird, complicated piece of crap full of frames and cookies >>> and all sorts of god-awful javascript navigation. However, before >>> I even get into that stuff I can't even get the site to open in >>> Mechanize. Can anyone else get this working, or is it just me? >>> >>> Make your Messenger window look the way you want. Express >>> Yourself! _______________________________________________ >>> Mechanize-users mailing list >>> Mechanize-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/mechanize-users >> >> >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users > > > Messenger has tons of new features that make chatting more fun. > Click here to learn more. > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From eatme444 at hotmail.com Fri Mar 27 12:01:27 2009 From: eatme444 at hotmail.com (Anthony F) Date: Fri, 27 Mar 2009 09:01:27 -0700 Subject: [Mechanize-users] Can't get this site to open Message-ID: I should be more clear. Your code also works for me as is in 0.9.2. However, if I do this: irb(main):001:0> require 'mechanize' => true irb(main):002:0> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca') => #} {meta} {title nil} {iframes} {frames} {links} {forms}> The problem I'm having is title => nil, links => [], etc. I can't actually do anything with the page other than get the body. And when I say frames is empty I mean that when I try to parse the redirected page (http://www.bcbid.gov.bc.ca/open.dll/welcome) it comes up empty as well even though I can do a page.body successfully. I had it working with 0.8.5 last night, but now that doesn't work anymore either. I'm baffled. Mat Schaffer wrote: My previous was 0.9.0, but it works for me with 0.9.2 as well: >> require 'mechanize' => true >> WWW::Mechanize::VERSION => "0.9.2" >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body Re-directing to BC Bid... => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome'">\r\nIf this page does not automatically re-direct you to BC Bid?, \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here.\r\n\r\n" I don't see any frames here. Do you maybe have a transparent web proxy where you are? What does your response look like? You might want to check using curl too. -Mat On Mar 27, 2009, at 4:02 AM, Anthony F wrote: Interesting. I tried it with mechanize 0.8.5 and it seemed to work fine. With 0.9.2 it opens the page, but doesn't seem to parse it properly (ie. frames => nil, link => nil, etc). What version are you using? Mat Schaffer wrote: Loads for me, but it's also got a javascript redirect in there. You'll have to do that yourself with something like agent.click(page.links.first) >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome'">\r\nIf this page does not automatically re-direct you to BC Bid?, \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here.\r\n\r\n" On Mar 26, 2009, at 6:30 PM, Anthony F wrote: The site is: http://www.bcbid.gov.bc.ca It's a weird, complicated piece of crap full of frames and cookies and all sorts of god-awful javascript navigation. However, before I even get into that stuff I can't even get the site to open in Mechanize. Can anyone else get this working, or is it just me? Make your Messenger window look the way you want. Express Yourself! _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users Messenger has tons of new features that make chatting more fun. Click here to learn more. _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _________________________________________________________________ Chat with the whole group, and bring everyone together. http://go.microsoft.com/?linkid=9650735 -------------- next part -------------- An HTML attachment was scrubbed... URL: From eatme444 at hotmail.com Fri Mar 27 12:07:42 2009 From: eatme444 at hotmail.com (Anthony F) Date: Fri, 27 Mar 2009 09:07:42 -0700 Subject: [Mechanize-users] Can't get this site to open Message-ID: Errr... I take that back. I can still get it to work in 0.8.5, but not 0.9.2. If I understand correctly the parser changed between those versions? That's probably the issue... Mat Schaffer wrote: My previous was 0.9.0, but it works for me with 0.9.2 as well: >> require 'mechanize' => true >> WWW::Mechanize::VERSION => "0.9.2" >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body Re-directing to BC Bid... => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome'">\r\nIf this page does not automatically re-direct you to BC Bid?, \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here.\r\n\r\n" I don't see any frames here. Do you maybe have a transparent web proxy where you are? What does your response look like? You might want to check using curl too. -Mat On Mar 27, 2009, at 4:02 AM, Anthony F wrote: Interesting. I tried it with mechanize 0.8.5 and it seemed to work fine. With 0.9.2 it opens the page, but doesn't seem to parse it properly (ie. frames => nil, link => nil, etc). What version are you using? Mat Schaffer wrote: Loads for me, but it's also got a javascript redirect in there. You'll have to do that yourself with something like agent.click(page.links.first) >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome'">\r\nIf this page does not automatically re-direct you to BC Bid?, \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here.\r\n\r\n" On Mar 26, 2009, at 6:30 PM, Anthony F wrote: The site is: http://www.bcbid.gov.bc.ca It's a weird, complicated piece of crap full of frames and cookies and all sorts of god-awful javascript navigation. However, before I even get into that stuff I can't even get the site to open in Mechanize. Can anyone else get this working, or is it just me? Make your Messenger window look the way you want. Express Yourself! _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users Messenger has tons of new features that make chatting more fun. Click here to learn more. _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _________________________________________________________________ Share photos with friends on Windows Live Messenger http://go.microsoft.com/?linkid=9650734 -------------- next part -------------- An HTML attachment was scrubbed... URL: From eatme444 at hotmail.com Fri Mar 27 12:29:16 2009 From: eatme444 at hotmail.com (Anthony F) Date: Fri, 27 Mar 2009 09:29:16 -0700 Subject: [Mechanize-users] Can't get this site to open Message-ID: YES!!! That was it. When I switch the parser to Hpricot all is well. Thanks for the help, Mat! Now I'm off to scrape this god-awful website... A F wrote: Errr... I take that back. I can still get it to work in 0.8.5, but not 0.9.2. If I understand correctly the parser changed between those versions? That's probably the issue... Mat Schaffer wrote: My previous was 0.9.0, but it works for me with 0.9.2 as well: >> require 'mechanize' => true >> WWW::Mechanize::VERSION => "0.9.2" >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body Re-directing to BC Bid... => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome'">\r\nIf this page does not automatically re-direct you to BC Bid?, \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here.\r\n\r\n" I don't see any frames here. Do you maybe have a transparent web proxy where you are? What does your response look like? You might want to check using curl too. -Mat On Mar 27, 2009, at 4:02 AM, Anthony F wrote: Interesting. I tried it with mechanize 0.8.5 and it seemed to work fine. With 0.9.2 it opens the page, but doesn't seem to parse it properly (ie. frames => nil, link => nil, etc). What version are you using? Mat Schaffer wrote: Loads for me, but it's also got a javascript redirect in there. You'll have to do that yourself with something like agent.click(page.links.first) >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome'">\r\nIf this page does not automatically re-direct you to BC Bid?, \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here.\r\n\r\n" On Mar 26, 2009, at 6:30 PM, Anthony F wrote: The site is: http://www.bcbid.gov.bc.ca It's a weird, complicated piece of crap full of frames and cookies and all sorts of god-awful javascript navigation. However, before I even get into that stuff I can't even get the site to open in Mechanize. Can anyone else get this working, or is it just me? Make your Messenger window look the way you want. Express Yourself! _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users Messenger has tons of new features that make chatting more fun. Click here to learn more. _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _________________________________________________________________ Reunite with the people closest to you, chat face to face with Messenger. http://go.microsoft.com/?linkid=9650736 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.schaffer at gmail.com Fri Mar 27 14:12:14 2009 From: mat.schaffer at gmail.com (Mat Schaffer) Date: Fri, 27 Mar 2009 14:12:14 -0400 Subject: [Mechanize-users] Can't get this site to open In-Reply-To: References: Message-ID: Cool. I dunno if Aaron's on this or not, but it might be good to figure out why nokogiri can't parse that page. Here's a file captured with: File.open('response.html', 'w') { |f| f.print WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body } I may play with it myself this weekend, but if maybe Aaron will beat me to it. Thanks for finding the bug Anthony! -Mat On Mar 27, 2009, at 12:29 PM, Anthony F wrote: > YES!!! That was it. When I switch the parser to Hpricot all is > well. Thanks for the help, Mat! > > Now I'm off to scrape this god-awful website... > > A F wrote: >> >> Errr... I take that back. I can still get it to work in 0.8.5, but >> not 0.9.2. If I understand correctly the parser changed between >> those versions? That's probably the issue... >> >> Mat Schaffer wrote: >>> >>> My previous was 0.9.0, but it works for me with 0.9.2 as well: >>> >>> >> require 'mechanize' >>> => true >>> >> WWW::Mechanize::VERSION >>> => "0.9.2" >>> >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body >>> => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/ >>> welcome'">\r\nIf this page does not automatically re-direct you to >>> BC Bid?, >>> \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here. >>> \r\n\r\n" >>> >>> I don't see any frames here. Do you maybe have a transparent web >>> proxy where you are? What does your response look like? You might >>> want to check using curl too. >>> -Mat >>> >>> On Mar 27, 2009, at 4:02 AM, Anthony F wrote: >>> >>>> Interesting. I tried it with mechanize 0.8.5 and it seemed to >>>> work fine. With 0.9.2 it opens the page, but doesn't seem to >>>> parse it properly (ie. frames => nil, link => nil, etc). What >>>> version are you using? >>>> >>>> Mat Schaffer wrote: >>>>> >>>>> Loads for me, but it's also got a javascript redirect in there. >>>>> You'll have to do that yourself with something like >>>>> agent.click(page.links.first) >>>>> >>>>> >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body >>>>> => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome' >>>>> ">\r\nIf this page does not automatically re-direct you to BC >>>>> Bid?, >>>>> \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click >>>>> here.\r\n\r\n" >>>>> >>>>> >>>>> On Mar 26, 2009, at 6:30 PM, Anthony F wrote: >>>>> >>>>>> >>>>>> The site is: http://www.bcbid.gov.bc.ca >>>>>> >>>>>> It's a weird, complicated piece of crap full of frames and >>>>>> cookies and all sorts of god-awful javascript navigation. >>>>>> However, before I even get into that stuff I can't even get the >>>>>> site to open in Mechanize. Can anyone else get this working, or >>>>>> is it just me? >>>>>> >>>>>> Make your Messenger window look the way you want. Express >>>>>> Yourself! _______________________________________________ >>>>>> Mechanize-users mailing list >>>>>> Mechanize-users at rubyforge.org >>>>>> http://rubyforge.org/mailman/listinfo/mechanize-users >>>>> >>>>> >>>>> _______________________________________________ >>>>> Mechanize-users mailing list >>>>> Mechanize-users at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/mechanize-users >>>> >>>> >>>> Messenger has tons of new features that make chatting more fun. >>>> Click here to learn more. >>>> _______________________________________________ >>>> Mechanize-users mailing list >>>> Mechanize-users at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/mechanize-users >>> >>> >>> _______________________________________________ >>> Mechanize-users mailing list >>> Mechanize-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/mechanize-users >> > > > Messenger has tons of new features that make chatting more fun. > Click here to learn more. > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.schaffer at gmail.com Fri Mar 27 14:31:25 2009 From: mat.schaffer at gmail.com (Mat Schaffer) Date: Fri, 27 Mar 2009 14:31:25 -0400 Subject: [Mechanize-users] Can't get this site to open In-Reply-To: References: Message-ID: <7693C5EE-65FB-47F2-A7BD-28EADE3A03DB@gmail.com> Just noticed this: Looks like there's a UTF-8 copyright symbol or something that might be throwing things off. Especially because the server doesn't appear to mark it as UTF-8 in the headers. -Mat On Mar 27, 2009, at 2:12 PM, Mat Schaffer wrote: > Cool. I dunno if Aaron's on this or not, but it might be good to > figure out why nokogiri can't parse that page. > > Here's a file captured with: File.open('response.html', 'w') { |f| > f.print WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body } > > I may play with it myself this weekend, but if maybe Aaron will beat > me to it. > > Thanks for finding the bug Anthony! > -Mat > > > > > > On Mar 27, 2009, at 12:29 PM, Anthony F wrote: > >> YES!!! That was it. When I switch the parser to Hpricot all is >> well. Thanks for the help, Mat! >> >> Now I'm off to scrape this god-awful website... >> >> A F wrote: >>> >>> Errr... I take that back. I can still get it to work in 0.8.5, >>> but not 0.9.2. If I understand correctly the parser changed >>> between those versions? That's probably the issue... >>> >>> Mat Schaffer wrote: >>>> >>>> My previous was 0.9.0, but it works for me with 0.9.2 as well: >>>> >>>> >> require 'mechanize' >>>> => true >>>> >> WWW::Mechanize::VERSION >>>> => "0.9.2" >>>> >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body >>>> => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome' >>>> ">\r\nIf this page does not automatically re-direct you to BC >>>> Bid?, >>>> \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click >>>> here.\r\n\r\n" >>>> >>>> I don't see any frames here. Do you maybe have a transparent web >>>> proxy where you are? What does your response look like? You might >>>> want to check using curl too. >>>> -Mat >>>> >>>> On Mar 27, 2009, at 4:02 AM, Anthony F wrote: >>>> >>>>> Interesting. I tried it with mechanize 0.8.5 and it seemed to >>>>> work fine. With 0.9.2 it opens the page, but doesn't seem to >>>>> parse it properly (ie. frames => nil, link => nil, etc). What >>>>> version are you using? >>>>> >>>>> Mat Schaffer wrote: >>>>>> >>>>>> Loads for me, but it's also got a javascript redirect in there. >>>>>> You'll have to do that yourself with something like >>>>>> agent.click(page.links.first) >>>>>> >>>>>> >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body >>>>>> => "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome' >>>>>> ">\r\nIf this page does not automatically re-direct you to BC >>>>>> Bid?, >>>>>> \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click >>>>>> here.\r\n\r\n" >>>>>> >>>>>> >>>>>> On Mar 26, 2009, at 6:30 PM, Anthony F wrote: >>>>>> >>>>>>> >>>>>>> The site is: http://www.bcbid.gov.bc.ca >>>>>>> >>>>>>> It's a weird, complicated piece of crap full of frames and >>>>>>> cookies and all sorts of god-awful javascript navigation. >>>>>>> However, before I even get into that stuff I can't even get >>>>>>> the site to open in Mechanize. Can anyone else get this >>>>>>> working, or is it just me? >>>>>>> >>>>>>> Make your Messenger window look the way you want. Express >>>>>>> Yourself! _______________________________________________ >>>>>>> Mechanize-users mailing list >>>>>>> Mechanize-users at rubyforge.org >>>>>>> http://rubyforge.org/mailman/listinfo/mechanize-users >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Mechanize-users mailing list >>>>>> Mechanize-users at rubyforge.org >>>>>> http://rubyforge.org/mailman/listinfo/mechanize-users >>>>> >>>>> >>>>> Messenger has tons of new features that make chatting more fun. >>>>> Click here to learn more. >>>>> _______________________________________________ >>>>> Mechanize-users mailing list >>>>> Mechanize-users at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/mechanize-users >>>> >>>> >>>> _______________________________________________ >>>> Mechanize-users mailing list >>>> Mechanize-users at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/mechanize-users >>> >> >> >> Messenger has tons of new features that make chatting more fun. >> Click here to learn more. >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mr.danielaquino at gmail.com Mon Mar 30 20:05:01 2009 From: mr.danielaquino at gmail.com (Daniel Aquino) Date: Mon, 30 Mar 2009 19:05:01 -0500 Subject: [Mechanize-users] HTTP Headers Only Message-ID: <66f0f93e0903301705n1e585a68t9e7635178e889b20@mail.gmail.com> Is there a way to request only the http headers? I have a bot that connects to sites and spits out the html title but for binary files I like it to just read the http headers to get the file size etc... and not read in the entire binary! Thanks! From mat.schaffer at gmail.com Mon Mar 30 20:58:37 2009 From: mat.schaffer at gmail.com (Mat Schaffer) Date: Mon, 30 Mar 2009 20:58:37 -0400 Subject: [Mechanize-users] HTTP Headers Only In-Reply-To: <66f0f93e0903301705n1e585a68t9e7635178e889b20@mail.gmail.com> References: <66f0f93e0903301705n1e585a68t9e7635178e889b20@mail.gmail.com> Message-ID: Seems like WWW::Mechanize#head would work: http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000183 -Mat On Mar 30, 2009, at 8:05 PM, Daniel Aquino wrote: > Is there a way to request only the http headers? > > I have a bot that connects to sites and spits out the html title but > for binary files I like it to just read the http headers to get the > file size etc... and not read in the entire binary! > > Thanks! > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users From aaron.patterson at gmail.com Mon Mar 30 23:18:37 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Mon, 30 Mar 2009 20:18:37 -0700 Subject: [Mechanize-users] HTTP Headers Only In-Reply-To: References: <66f0f93e0903301705n1e585a68t9e7635178e889b20@mail.gmail.com> Message-ID: <6959e1680903302018n7c418751wd176cf524bdaca9@mail.gmail.com> On Mon, Mar 30, 2009 at 5:58 PM, Mat Schaffer wrote: > Seems like WWW::Mechanize#head would work: > > http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000183 Yes. A head request sounds appropriate. -- Aaron Patterson http://tenderlovemaking.com/ From mr.danielaquino at gmail.com Tue Mar 31 01:17:03 2009 From: mr.danielaquino at gmail.com (Daniel Aquino) Date: Tue, 31 Mar 2009 01:17:03 -0400 Subject: [Mechanize-users] HTTP Headers Only In-Reply-To: <6959e1680903302018n7c418751wd176cf524bdaca9@mail.gmail.com> References: <66f0f93e0903301705n1e585a68t9e7635178e889b20@mail.gmail.com> <6959e1680903302018n7c418751wd176cf524bdaca9@mail.gmail.com> Message-ID: <66f0f93e0903302217g29f2cfe8r72c245c6a0e711f1@mail.gmail.com> Yea I really searched around for this and couldn't figure out how to do it... Thanks so much... Also I think I remember reading something that the http server has to support a head request. is this true? On Mon, Mar 30, 2009 at 11:18 PM, Aaron Patterson wrote: > On Mon, Mar 30, 2009 at 5:58 PM, Mat Schaffer wrote: >> Seems like WWW::Mechanize#head would work: >> >> http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000183 > > Yes. ?A head request sounds appropriate. > > -- > Aaron Patterson > http://tenderlovemaking.com/ > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > From mr.danielaquino at gmail.com Tue Mar 31 01:41:17 2009 From: mr.danielaquino at gmail.com (Daniel Aquino) Date: Tue, 31 Mar 2009 01:41:17 -0400 Subject: [Mechanize-users] HTTP Headers Only In-Reply-To: <66f0f93e0903302217g29f2cfe8r72c245c6a0e711f1@mail.gmail.com> References: <66f0f93e0903301705n1e585a68t9e7635178e889b20@mail.gmail.com> <6959e1680903302018n7c418751wd176cf524bdaca9@mail.gmail.com> <66f0f93e0903302217g29f2cfe8r72c245c6a0e711f1@mail.gmail.com> Message-ID: <66f0f93e0903302241m27ea26aavcc2d7c3d38615912@mail.gmail.com> Is there anyway to limit the amount of data to read from any link? Perhaps use a filter to detect the tag and abort connection? Or set a timeout on how long data should be read from the link? I'm sure a malicious person could easily still feed in a massively large file and cause the daemon to stick around reading it all... And the only thing I'm interested in is the <title/> Thanks! On Tue, Mar 31, 2009 at 1:17 AM, Daniel Aquino <mr.danielaquino at gmail.com> wrote: > Yea I really searched around for this and couldn't figure out how to do it... > > Thanks so much... > > Also I think I remember reading something that the http server has to > support a head request. > > is this true? > > On Mon, Mar 30, 2009 at 11:18 PM, Aaron Patterson > <aaron.patterson at gmail.com> wrote: >> On Mon, Mar 30, 2009 at 5:58 PM, Mat Schaffer <mat.schaffer at gmail.com> wrote: >>> Seems like WWW::Mechanize#head would work: >>> >>> http://mechanize.rubyforge.org/mechanize/WWW/Mechanize.html#M000183 >> >> Yes. ?A head request sounds appropriate. >> >> -- >> Aaron Patterson >> http://tenderlovemaking.com/ >> _______________________________________________ >> Mechanize-users mailing list >> Mechanize-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mechanize-users >> > From aaron.patterson at gmail.com Tue Mar 31 01:50:03 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Mon, 30 Mar 2009 22:50:03 -0700 Subject: [Mechanize-users] HTTP Headers Only In-Reply-To: <66f0f93e0903302217g29f2cfe8r72c245c6a0e711f1@mail.gmail.com> References: <66f0f93e0903301705n1e585a68t9e7635178e889b20@mail.gmail.com> <E82FFA43-8588-4324-AEDF-F64793682E41@gmail.com> <6959e1680903302018n7c418751wd176cf524bdaca9@mail.gmail.com> <66f0f93e0903302217g29f2cfe8r72c245c6a0e711f1@mail.gmail.com> Message-ID: <6959e1680903302250l5778b344p214ae0ff6b35a4f1@mail.gmail.com> On Mon, Mar 30, 2009 at 10:17 PM, Daniel Aquino <mr.danielaquino at gmail.com> wrote: > Yea I really searched around for this and couldn't figure out how to do it... > > Thanks so much... > > Also I think I remember reading something that the http server has to > support a head request. > > is this true? Yes, but most do. I don't think I've run in to one that doesn't. -- Aaron Patterson http://tenderlovemaking.com/ From mr.danielaquino at gmail.com Tue Mar 31 05:47:39 2009 From: mr.danielaquino at gmail.com (Daniel Aquino) Date: Tue, 31 Mar 2009 05:47:39 -0400 Subject: [Mechanize-users] HTTP Headers Only In-Reply-To: <6959e1680903302250l5778b344p214ae0ff6b35a4f1@mail.gmail.com> References: <66f0f93e0903301705n1e585a68t9e7635178e889b20@mail.gmail.com> <E82FFA43-8588-4324-AEDF-F64793682E41@gmail.com> <6959e1680903302018n7c418751wd176cf524bdaca9@mail.gmail.com> <66f0f93e0903302217g29f2cfe8r72c245c6a0e711f1@mail.gmail.com> <6959e1680903302250l5778b344p214ae0ff6b35a4f1@mail.gmail.com> Message-ID: <66f0f93e0903310247s63b50c0aod31e8763bfe05364@mail.gmail.com> if I call agent.head and then call agent.get I end up with only head... On Tue, Mar 31, 2009 at 1:50 AM, Aaron Patterson <aaron.patterson at gmail.com> wrote: > On Mon, Mar 30, 2009 at 10:17 PM, Daniel Aquino > <mr.danielaquino at gmail.com> wrote: >> Yea I really searched around for this and couldn't figure out how to do it... >> >> Thanks so much... >> >> Also I think I remember reading something that the http server has to >> support a head request. >> >> is this true? > > Yes, but most do. ?I don't think I've run in to one that doesn't. > > -- > Aaron Patterson > http://tenderlovemaking.com/ > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users > From eatme444 at hotmail.com Tue Mar 31 13:42:17 2009 From: eatme444 at hotmail.com (Anthony F) Date: Tue, 31 Mar 2009 10:42:17 -0700 Subject: [Mechanize-users] Can't get this site to open In-Reply-To: <7693C5EE-65FB-47F2-A7BD-28EADE3A03DB@gmail.com> References: <COL113-W26819A6C3A80101CB0132B838E0@phx.gbl> <B776C57A-3DA9-4F4D-AEC7-C712FF7FEEA1@gmail.com> <7693C5EE-65FB-47F2-A7BD-28EADE3A03DB@gmail.com> Message-ID: <COL113-W47A6BFF12F00276D96BCAF838A0@phx.gbl> Just another update... If I switch the html parser to Nokogiri instead of Nokogiri::HTML it seems to work as well. It turns out I need Nokogiri's extra XPath goodness to deal with this rat's nest, so that's a good thing. From: mat.schaffer at gmail.com To: mat.schaffer at gmail.com Date: Fri, 27 Mar 2009 14:31:25 -0400 CC: mechanize-users at rubyforge.org Subject: Re: [Mechanize-users] Can't get this site to open Just noticed this: <!-- Copyright ? 2001, 2002 OGMA Consulting Corp. --> Looks like there's a UTF-8 copyright symbol or something that might be throwing things off. Especially because the server doesn't appear to mark it as UTF-8 in the headers. -Mat On Mar 27, 2009, at 2:12 PM, Mat Schaffer wrote:Cool. I dunno if Aaron's on this or not, but it might be good to figure out why nokogiri can't parse that page. Here's a file captured with: File.open('response.html', 'w') { |f| f.print WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body } I may play with it myself this weekend, but if maybe Aaron will beat me to it. Thanks for finding the bug Anthony!-Mat <response.html> On Mar 27, 2009, at 12:29 PM, Anthony F wrote:YES!!! That was it. When I switch the parser to Hpricot all is well. Thanks for the help, Mat! Now I'm off to scrape this god-awful website... A F wrote:Errr... I take that back. I can still get it to work in 0.8.5, but not 0.9.2. If I understand correctly the parser changed between those versions? That's probably the issue... Mat Schaffer wrote:My previous was 0.9.0, but it works for me with 0.9.2 as well: >> require 'mechanize'=> true>> WWW::Mechanize::VERSION=> "0.9.2">> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body=> "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome'">\r\nIf this page does not automatically re-direct you to BC Bid?, \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here.\r\n\r\n" I don't see any frames here. Do you maybe have a transparent web proxy where you are? What does your response look like? You might want to check using curl too.-Mat On Mar 27, 2009, at 4:02 AM, Anthony F wrote: Interesting. I tried it with mechanize 0.8.5 and it seemed to work fine. With 0.9.2 it opens the page, but doesn't seem to parse it properly (ie. frames => nil, link => nil, etc). What version are you using? Mat Schaffer wrote:Loads for me, but it's also got a javascript redirect in there. You'll have to do that yourself with something like agent.click(page.links.first) >> WWW::Mechanize.new.get('http://www.bcbid.gov.bc.ca').body=> "\r\n\r\n\r\n\r\n\r\nhttp://www.bcbid.gov.bc.ca/open.dll/welcome'">\r\nIf this page does not automatically re-direct you to BC Bid?, \r\nplease http://www.bcbid.gov.bc.ca/open.dll/welcome">click here.\r\n\r\n" On Mar 26, 2009, at 6:30 PM, Anthony F wrote: The site is: http://www.bcbid.gov.bc.ca It's a weird, complicated piece of crap full of frames and cookies and all sorts of god-awful javascript navigation. However, before I even get into that stuff I can't even get the site to open in Mechanize. Can anyone else get this working, or is it just me? Make your Messenger window look the way you want. Express Yourself! _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users Messenger has tons of new features that make chatting more fun. Click here to learn more. _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users Messenger has tons of new features that make chatting more fun. Click here to learn more. _______________________________________________ Mechanize-users mailing list Mechanize-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mechanize-users _________________________________________________________________ Chat with the whole group, and bring everyone together. http://go.microsoft.com/?linkid=9650735 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20090331/0df3aaa4/attachment.html>