From aaron at tenderlovemaking.com Tue Oct 2 12:52:50 2007 From: aaron at tenderlovemaking.com (Aaron Patterson) Date: Tue, 2 Oct 2007 09:52:50 -0700 Subject: [Mechanize-users] Problems with Forms In-Reply-To: <46FF6D86.9030905@wannawork.de> References: <46FF6D86.9030905@wannawork.de> Message-ID: <20071002165250.GA18704@mac-mini.lan> Hi Bodo, On Sun, Sep 30, 2007 at 11:33:58AM +0200, Bodo Tasche wrote: > Hi, > > I am using mechanize for a while now. Works great. But at the moment I > have a small problem using it. > > The problematic file is attached. In that PHP-File you will see a > formular, but mechanize doesn't recognize it. Could you be more specific? I'm not sure what you mean exactly. > > Could somebody check this? Do you have a sample script to reproduce the error? -- Aaron Patterson http://tenderlovemaking.com/ From jrust at lexus.elabor.com Tue Oct 2 13:03:45 2007 From: jrust at lexus.elabor.com (Rust, Jon) Date: Tue, 02 Oct 2007 10:03:45 -0700 Subject: [Mechanize-users] Client-side certs Message-ID: <470279F1.90102@lexus.elabor.com> Our web-based application is changing from form-based logins only to client side certificates (plus form-based). I can't find any reference to loading a client cert. Is it possible with Mechanize? Any plans to support it? Hoping, jon From bodo at wannawork.de Tue Oct 2 17:36:44 2007 From: bodo at wannawork.de (Bodo Tasche) Date: Tue, 2 Oct 2007 23:36:44 +0200 Subject: [Mechanize-users] Problems with Forms In-Reply-To: <20071002165250.GA18704@mac-mini.lan> References: <46FF6D86.9030905@wannawork.de> <20071002165250.GA18704@mac-mini.lan> Message-ID: <2093A31B-7B04-460F-AF72-BF3CA13ADEDC@wannawork.de> Hi, Am 02.10.2007 um 18:52 schrieb Aaron Patterson: > Hi Bodo, > > On Sun, Sep 30, 2007 at 11:33:58AM +0200, Bodo Tasche wrote: >> Hi, >> >> I am using mechanize for a while now. Works great. But at the >> moment I >> have a small problem using it. >> >> The problematic file is attached. In that PHP-File you will see a >> formular, but mechanize doesn't recognize it. > > Could you be more specific? I'm not sure what you mean exactly. If I load that HTML-File using page=agent.get('URLTOFILE') the form-array is empty, page.forms.size returns 0 >> >> Could somebody check this? > > Do you have a sample script to reproduce the error? page=agent.get('URLTOFILE') puts page.forms.size Regards, Bodo --- http://www.wannawork.de http://www.tvbrowser.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20071002/18d3c8a0/attachment-0001.html From sebastienmaurette at yahoo.fr Thu Oct 4 07:40:47 2007 From: sebastienmaurette at yahoo.fr (=?ISO-8859-1?Q?S=E9bastien_Maurette?=) Date: Thu, 04 Oct 2007 12:40:47 +0100 Subject: [Mechanize-users] newbie question with login form Message-ID: <4704D13F.7020504@yahoo.fr> hi, i'm just starting to work with this incredible tool... but i got a first problem with the login process i'm logging on my app like this : ------ @agent = WWW::Mechanize.new { |a| a.log = Logger.new("mech.log") } @agent.user_agent_alias = 'Mac Safari' @page = @agent.get("http://myappAdress/") @form = @page.forms.first @form['form_login'] = 'myLog' @form['form_password'] = 'myPass' @page = @agent.submit(@form, @form.buttons.first) put @page.body ---- Identification process is done , i can see it in my application log, but the page displayed is still the login form. here is the mechanize log . ----- Net::HTTP::Get: / request-header: accept-language => en-us,en;q0.5 request-header: connection => keep-alive request-header: accept => */* request-header: accept-encoding => gzip,identity request-header: user-agent => Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/418 (KHTML, like Gecko) Safari/417.9.3 request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7 request-header: keep-alive => 300 Read 519 bytes Read 935 bytes Read 1959 bytes Read 2375 bytes Read 3399 bytes Read 3803 bytes response-header: connection => close response-header: p3p => CP='OTI DSP COR NID STP UNI OTPa OUR' response-header: content-type => text/html; charset=utf-8 response-header: x-powered-by => PHP/5.1.6 response-header: date => Thu, 04 Oct 2007 10:34:40 GMT response-header: server => Apache/2.2.4 (Fedora) response-header: content-length => 3803 response-header: set-cookie => pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D; expires=Sun, 01-Oct-2017 10:34:40 GMT, pmv_ck_session=YToyOntzOjU6ImxvZ2luIjtzOjk6ImFub255bW91cyI7czo4OiJwYXNzd29yZCI7YjowO30%3D; expires=Thu, 04-Oct-2007 11:34:40 GMT saved cookie: pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D saved cookie: pmv_ck_session=YToyOntzOjU6ImxvZ2luIjtzOjk6ImFub255bW91cyI7czo4OiJwYXNzd29yZCI7YjowO30%3D status: 200 query: "form_login=myLog&form_password=myPass&form_url=http%3A%2F%2FmyAppUrl%2F&submit=Login+%3F" Net::HTTP::Post: /index.php?mod=login using cookie: pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D request-header: accept-language => en-us,en;q0.5 request-header: connection => keep-alive request-header: accept => */* request-header: accept-encoding => gzip,identity request-header: user-agent => Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/418 (KHTML, like Gecko) Safari/417.9.3 request-header: content-type => application/x-www-form-urlencoded request-header: cookie => pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D request-header: referer => http://myAppUrl/ request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7 request-header: content-length => 101 request-header: keep-alive => 300 Read 0 bytes response-header: connection => close response-header: p3p => CP='OTI DSP COR NID STP UNI OTPa OUR' response-header: content-type => text/html; charset=UTF-8 response-header: x-powered-by => PHP/5.1.6 response-header: date => Thu, 04 Oct 2007 10:34:41 GMT response-header: server => Apache/2.2.4 (Fedora) response-header: content-length => 0 response-header: set-cookie => pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D; expires=Sun, 01-Oct-2017 10:34:41 GMT, pmv_ck_session=YToyOntzOjU6ImxvZ2luIjtzOjM6InNlYiI7czo4OiJwYXNzd29yZCI7czozMjoiMGNhYzVlYWIxNWZlOWY2YTFlZmQzOGI3YzEwM2NkZDciO30%3D; expires=Thu, 04-Oct-2007 11:34:41 GMT response-header: location => myAppUrl saved cookie: pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D saved cookie: pmv_ck_session=YToyOntzOjU6ImxvZ2luIjtzOjM6InNlYiI7czo4OiJwYXNzd29yZCI7czozMjoiMGNhYzVlYWIxNWZlOWY2YTFlZmQzOGI3YzEwM2NkZDciO30%3D status: 302 follow redirect to: myAppUrl Net::HTTP::Get: / using cookie: pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D request-header: accept-language => en-us,en;q0.5 request-header: connection => keep-alive request-header: accept => */* request-header: accept-encoding => gzip,identity request-header: user-agent => Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/418 (KHTML, like Gecko) Safari/417.9.3 request-header: cookie => pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D request-header: referer => myAppUrl request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7 request-header: keep-alive => 300 Read 519 bytes Read 935 bytes Read 1959 bytes Read 2375 bytes Read 3399 bytes Read 3803 bytes response-header: connection => close response-header: p3p => CP='OTI DSP COR NID STP UNI OTPa OUR' response-header: content-type => text/html; charset=utf-8 response-header: x-powered-by => PHP/5.1.6 response-header: date => Thu, 04 Oct 2007 10:34:41 GMT response-header: server => Apache/2.2.4 (Fedora) response-header: content-length => 3803 response-header: set-cookie => pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D; expires=Sun, 01-Oct-2017 10:34:41 GMT, pmv_ck_session=YToyOntzOjU6ImxvZ2luIjtzOjk6ImFub255bW91cyI7czo4OiJwYXNzd29yZCI7YjowO30%3D; expires=Thu, 04-Oct-2007 11:34:41 GMT saved cookie: pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D saved cookie: pmv_ck_session=YToyOntzOjU6ImxvZ2luIjtzOjk6ImFub255bW91cyI7czo4OiJwYXNzd29yZCI7YjowO30%3D status: 200 ----- this application seems to use redirection and probably there is a problem with the cookies session ? what's wrong with my code ? thanks in advance zeb From aaron at tenderlovemaking.com Thu Oct 4 11:19:33 2007 From: aaron at tenderlovemaking.com (Aaron Patterson) Date: Thu, 4 Oct 2007 08:19:33 -0700 Subject: [Mechanize-users] newbie question with login form In-Reply-To: <4704D13F.7020504@yahoo.fr> References: <4704D13F.7020504@yahoo.fr> Message-ID: <20071004151933.GA23621@mac-mini.lan> Hey Zeb! On Thu, Oct 04, 2007 at 12:40:47PM +0100, S?bastien Maurette wrote: > hi, > > i'm just starting to work with this incredible tool... > > but i got a first problem with the login process > > i'm logging on my app like this : > > ------ > > @agent = WWW::Mechanize.new { |a| a.log = Logger.new("mech.log") } > > @agent.user_agent_alias = 'Mac Safari' > @page = @agent.get("http://myappAdress/") > > @form = @page.forms.first > > @form['form_login'] = 'myLog' > @form['form_password'] = 'myPass' > > @page = @agent.submit(@form, @form.buttons.first) > > put @page.body > > ---- > > Identification process is done , i can see it in my application log, but > the page displayed is still the login form. > > here is the mechanize log . > [snip] > > this application seems to use redirection and probably there is a > problem with the cookies session ? > what's wrong with my code ? Your code looks fine! There could be a couple issues that I can think of right of the bat. First, make sure that you are submitting all of the correct data. The way that I typically verify this is by using the livehttpheaders plugin for Firefox (firebug may work as well), and comparing what live httpheaders says to what my mechanize log says. http://livehttpheaders.mozdev.org/ Second, is there javascript on the page that modifies some of the input fields? If so, you'll need to reproduce its functionality. I hope that helps! -- Aaron Patterson http://tenderlovemaking.com/ From sebastienmaurette at yahoo.fr Thu Oct 4 12:53:22 2007 From: sebastienmaurette at yahoo.fr (=?ISO-8859-1?Q?S=E9bastien_Maurette?=) Date: Thu, 04 Oct 2007 17:53:22 +0100 Subject: [Mechanize-users] newbie question with login form In-Reply-To: <20071004151933.GA23621@mac-mini.lan> References: <4704D13F.7020504@yahoo.fr> <20071004151933.GA23621@mac-mini.lan> Message-ID: <47051A82.2000207@yahoo.fr> hi aaron, thanks for your help ! i checked my httpheaders and didn't find any problem... but just a question, i can see a ' pmv_ck_session ' and a 'pmv_ck_view_conf' cookies in response-header: set-cookie. and in request_header i just have one of both, request-header: cookie => pmv_ck_view_conf=YToxOntzOjQ6ImxhbmciO3M6MTI6ImZyLXV0Zi04LnBocCI7fQ%3D%3D maybe application is waitting for this cookie to authenticate me ? how to add the ' pmv_ck_session ' session cookie on each request ? Aaron Patterson a ?crit : > Hey Zeb! > > On Thu, Oct 04, 2007 at 12:40:47PM +0100, S?bastien Maurette wrote: > >> hi, >> >> i'm just starting to work with this incredible tool... >> >> but i got a first problem with the login process >> >> i'm logging on my app like this : >> >> ------ >> >> @agent = WWW::Mechanize.new { |a| a.log = Logger.new("mech.log") } >> >> @agent.user_agent_alias = 'Mac Safari' >> @page = @agent.get("http://myappAdress/") >> >> @form = @page.forms.first >> >> @form['form_login'] = 'myLog' >> @form['form_password'] = 'myPass' >> >> @page = @agent.submit(@form, @form.buttons.first) >> >> put @page.body >> >> ---- >> >> Identification process is done , i can see it in my application log, but >> the page displayed is still the login form. >> >> here is the mechanize log . >> >> > [snip] > >> this application seems to use redirection and probably there is a >> problem with the cookies session ? >> what's wrong with my code ? >> > > Your code looks fine! There could be a couple issues that I can think > of right of the bat. First, make sure that you are submitting all of > the correct data. The way that I typically verify this is by using the > livehttpheaders plugin for Firefox (firebug may work as well), and > comparing what live httpheaders says to what my mechanize log says. > > http://livehttpheaders.mozdev.org/ > > Second, is there javascript on the page that modifies some of the input > fields? If so, you'll need to reproduce its functionality. > > I hope that helps! > > From rubyforge at i-cnct.com Fri Oct 5 15:05:03 2007 From: rubyforge at i-cnct.com (GS Mechanize) Date: Fri, 5 Oct 2007 13:05:03 -0600 (MDT) Subject: [Mechanize-users] basic_auth problem since 0.6.9 In-Reply-To: References: Message-ID: <20071005122538.M25849@sintay.com> I have a site that I don't think "returns" a basic_auth request, but is able to use basic_auth. In the past on 0.6.8, I could use the following code: require 'rubygems' # gem 'mechanize', '=0.6.8' require 'mechanize' agent = WWW::Mechanize.new agent.basic_auth("username", "password") site_response = agent.get("https://link.to.site/") Since 0.6.9, this doesn't work any longer. I "think" prior to 0.6.9 Mechanize would just try basic_auth if I used the basic_auth method, but now I think if I read prior posts by Aaron correctly, Mechanize is making an initial call out to the web page I want to connect to to try and find what authentication the web server supports. In my case (and I don't know if this might be common, or just totally wrong, but this is how the site does it), the web server doesn't respond with basic_auth, so I don't think mechanize tries using it. Or maybe it is something else entirely. Connecting with 0.6.10 to "other" sites that give back the basic_auth response seems to works fine, so works except in this one case. If everything I've said it true, I wonder if there is a way to force Mechanize to just try using basic auth if I use the basic auth method to set username password, regardless of what the server says it supports. I just checkout out the source and will try to just hack something, but I'm such a new programmer, it'll likely be a kludge. Anyhow, thought I'd report it. Here is a 0.6.10 -r 445 logger log: irb(main):011:0> site_request = agent.get("https://link.to.site") I, [2007-10-05T12:46:44.841861 #4958] INFO -- : Net::HTTP::Get: / D, [2007-10-05T12:46:45.213989 #4958] DEBUG -- : request-header: accept-language => en-us,en;q0.5 D, [2007-10-05T12:46:45.214105 #4958] DEBUG -- : request-header: connection => keep-alive D, [2007-10-05T12:46:45.214169 #4958] DEBUG -- : request-header: accept => */* D, [2007-10-05T12:46:45.214227 #4958] DEBUG -- : request-header: accept-encoding => gzip,identity D, [2007-10-05T12:46:45.214285 #4958] DEBUG -- : request-header: user-agent => WWW-Mechanize/0.6.10 (http://rubyforge.org/projects/mechanize/) D, [2007-10-05T12:46:45.214344 #4958] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7 D, [2007-10-05T12:46:45.214402 #4958] DEBUG -- : request-header: keep-alive => 300 D, [2007-10-05T12:46:45.348892 #4958] DEBUG -- : Read 432 bytes D, [2007-10-05T12:46:45.349286 #4958] DEBUG -- : Read 1456 bytes D, [2007-10-05T12:46:45.349583 #4958] DEBUG -- : Read 1510 bytes D, [2007-10-05T12:46:45.349661 #4958] DEBUG -- : response-header: vary => negotiate,BackhandSSL D, [2007-10-05T12:46:45.349726 #4958] DEBUG -- : response-header: connection => close D, [2007-10-05T12:46:45.349784 #4958] DEBUG -- : response-header: bobo-exception-file => /path/to/CORE2.py D, [2007-10-05T12:46:45.349845 #4958] DEBUG -- : response-header: content-location => index.html.asis, index.html.asis D, [2007-10-05T12:46:45.349903 #4958] DEBUG -- : response-header: content-type => text/html; charset=UTF-8 D, [2007-10-05T12:46:45.349962 #4958] DEBUG -- : response-header: x-powered-by => Zope (www.zope.org), Python (www.python.org) D, [2007-10-05T12:46:45.350021 #4958] DEBUG -- : response-header: date => Fri, 05 Oct 2007 18:28:43 GMT D, [2007-10-05T12:46:45.350078 #4958] DEBUG -- : response-header: bobo-exception-value => bobo exception D, [2007-10-05T12:46:45.350136 #4958] DEBUG -- : response-header: bobo-exception-type => LoginRequired D, [2007-10-05T12:46:45.350196 #4958] DEBUG -- : response-header: bobo-exception-line => 958 D, [2007-10-05T12:46:45.350253 #4958] DEBUG -- : response-header: server => Apache/1.3.26 (Unix) mod_backhand/1.2.2 mod_fastcgi/2.2.12 mod_ssl/2.8.10 OpenSSL/0.9.6e D, [2007-10-05T12:46:45.350311 #4958] DEBUG -- : response-header: content-length => 1510 D, [2007-10-05T12:46:45.350369 #4958] DEBUG -- : response-header: tcn => choice, choice I, [2007-10-05T12:46:45.362057 #4958] INFO -- : status: 500 WWW::Mechanize::ResponseCodeError: 500 => Net::HTTPInternalServerError from /usr/local/lib/ruby/gems/1.8/gems/mechanize-0.6.10/lib/mechanize.rb:185:in `get' from (irb):11 Thanks, Gary From rubyforge at i-cnct.com Fri Oct 5 16:25:06 2007 From: rubyforge at i-cnct.com (GS Mechanize) Date: Fri, 5 Oct 2007 14:25:06 -0600 (MDT) Subject: [Mechanize-users] basic_auth problem since 0.6.9 In-Reply-To: References: Message-ID: <20071005142356.A25849@sintay.com> > If everything I've said it true, I wonder if there is a way to force > Mechanize to just try using basic auth if I use the basic auth method to > set username password, regardless of what the server says it supports. I > just checkout out the source and will try to just hack something, Yeah, this is a kludge, but as a test, it does work: In the basic_auth method: def basic_auth(user, password) --> @auth_basic = 1 # set a marker that I really do want basic auth auth(user, password) end Then, in the fetch_page method: def fetch_page(...,...) There is a series of checks for the attempt to connect to the site the first time. If the rez_klass returns an HTTPUnauthorized it sets the @auth_hash. --------------- return page if res_klass <= Net::HTTPSuccess if res_klass == Net::HTTPNotModified elsif res_klass <= Net::HTTPRedirection elsif res_klass <= Net::HTTPUnauthorized raise ResponseCodeError.new(page) unless @user || @password raise ResponseCodeError.new(page) if @auth_hash.has_key?(uri.host) if response['www-authenticate'] =~ /Digest/i @auth_hash[uri.host] = :digest @digest = response['www-authenticate'] else @auth_hash[uri.host] = :basic end ----------------- In my case, the server doesn't correctly respond with HTTPUnauthorized, but if I manually force the auth_hash to be basic, I can connect again: if @auth_basic @auth_hash[uri.host] = :basic end This is a kludge, but it does work. From gary at sintay.com Fri Oct 5 15:56:33 2007 From: gary at sintay.com (Gary Sintay) Date: Fri, 5 Oct 2007 13:56:33 -0600 (MDT) Subject: [Mechanize-users] basic_auth problem since 0.6.9 In-Reply-To: <20071005122538.M25849@sintay.com> References: <20071005122538.M25849@sintay.com> Message-ID: <20071005133925.E25849@sintay.com> > If everything I've said it true, I wonder if there is a way to force > Mechanize to just try using basic auth if I use the basic auth method to set > username password, regardless of what the server says it supports. I just > checkout out the source and will try to just hack something, Yeah, this is a kludge, but as a test, it does work: def basic_auth(user, password) --> @auth_basic = 1 # set a variable that I really do want basic auth auth(user, password) end In def fetch_page(...,...) There is a series of checks for the result class in this method: --------------- return page if res_klass <= Net::HTTPSuccess if res_klass == Net::HTTPNotModified elsif res_klass <= Net::HTTPRedirection elsif res_klass <= Net::HTTPUnauthorized raise ResponseCodeError.new(page) unless @user || @password raise ResponseCodeError.new(page) if @auth_hash.has_key?(uri.host) if response['www-authenticate'] =~ /Digest/i @auth_hash[uri.host] = :digest @digest = response['www-authenticate'] else @auth_hash[uri.host] = :basic end ----------------- If I manually force the auth_hash to be basic, I can connect: if @auth_basic == 1 @auth_hash[uri.host] = :basic end This is a kludge, but it does work. From gary at sintay.com Fri Oct 5 16:19:58 2007 From: gary at sintay.com (Gary Sintay) Date: Fri, 5 Oct 2007 14:19:58 -0600 (MDT) Subject: [Mechanize-users] basic_auth problem since 0.6.9 In-Reply-To: References: Message-ID: <20071005141139.S25849@sintay.com> > If everything I've said it true, I wonder if there is a way to force > Mechanize to just try using basic auth if I use the basic auth method to > set username password, regardless of what the server says it supports. > I just checkout out the source and will try to just hack something, Yeah, this is a kludge, but as a test, it does work: In the basic_auth method: def basic_auth(user, password) --> @auth_basic = 1 # set a marker that I really do want basic auth auth(user, password) end Then, in the fetch_page method: def fetch_page(...,...) There is a series of checks for the attempt to connect to the site the first time. If the rez_klass returns an HTTPUnauthorized it sets the @auth_hash. --------------- return page if res_klass <= Net::HTTPSuccess if res_klass == Net::HTTPNotModified elsif res_klass <= Net::HTTPRedirection elsif res_klass <= Net::HTTPUnauthorized raise ResponseCodeError.new(page) unless @user || @password raise ResponseCodeError.new(page) if @auth_hash.has_key?(uri.host) if response['www-authenticate'] =~ /Digest/i @auth_hash[uri.host] = :digest @digest = response['www-authenticate'] else @auth_hash[uri.host] = :basic end ----------------- In my case, the server doesn't correctly respond with HTTPUnauthorized, but if I manually force the auth_hash to be basic, I can connect again: if @auth_basic @auth_hash[uri.host] = :basic end This is a kludge, but it does work. From maku at makuchaku.info Sun Oct 7 00:28:24 2007 From: maku at makuchaku.info (=?UTF-8?Q?=E0=A4=AE=E0=A4=AF=E0=A4=82=E0=A4=95_?= =?UTF-8?Q?=E0=A4=9C=E0=A5=88=E0=A4=A8_(?= =?UTF-8?Q?makuchaku)?=) Date: Sun, 7 Oct 2007 09:58:24 +0530 Subject: [Mechanize-users] How to store a Mechanize object in the database? Message-ID: <185269960710062128y2e250055rd42c9f9835ad78b0@mail.gmail.com> Hi, I am trying to save a Mechanize object in database (using a Rails Model). But the save operation throws a TypeError Considering that "agent" is an instance of a Rails Model and "user" is defined as a "text" type in the Model. irb(main):039:0> agent.user = WWW::Mechanize.new #WWW::Mechanize::Page}, defaultWWW::Mechanize::File, read_timeoutnil, keep_alive_time300, ca_filenil, watch_for_setnil, proxy_passnil, auth_hash{}, certnil, history[], open_timeoutnil, connection_cache{}, cookie_jar#, proxy_usernil, digestnil, passwordnil, redirect_oktrue irb(main):040:0> agent.save TypeError: can't dump anonymous class Class from /usr/lib/ruby/1.8/yaml/rubytypes.rb:6:in `to_yaml' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `node_export' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `add' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `to_yaml' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:17:in `each' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:17:in `to_yaml' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:16:in `map' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:16:in `to_yaml' from /usr/lib/ruby/1.8/yaml.rb:387:in `call' from /usr/lib/ruby/1.8/yaml.rb:387:in `emit' from /usr/lib/ruby/1.8/yaml.rb:387:in `quick_emit' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:15:in `to_yaml' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `node_export' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `add' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `to_yaml' from /usr/lib/ruby/1.8/yaml/rubytypes.rb:17:in `each' ... 14 levels... from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:2029:in `attributes_with_quotes' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:2128:in `quoted_column_names' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1813:in `create_without_callbacks' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.rb:254:in `create_without_timestamps' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/timestamp.rb:39:in `create' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1789:in `create_or_update_without_callbacks' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.rb:242:in `create_or_update' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1545:in `save_without_validation' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/validations.rb:752:in `save_without_transactions' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:129:in `save' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in `transaction' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:95:in `transaction' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:121:in `transaction' from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:129:in `save' from (irb):40 from (null):0irb(main):041:0> Further, I tried creating a simple WWW::Mechanize instance and did a "to_yaml" on the object - saw a similar error. How can I tackle this? Any suggestions would be welcome :) Thanks! Maku -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20071007/756507e3/attachment.html From maku at makuchaku.info Sun Oct 7 21:24:23 2007 From: maku at makuchaku.info (=?UTF-8?Q?=E0=A4=AE=E0=A4=AF=E0=A4=82=E0=A4=95_?= =?UTF-8?Q?=E0=A4=9C=E0=A5=88=E0=A4=A8_(?= =?UTF-8?Q?makuchaku)?=) Date: Mon, 8 Oct 2007 06:54:23 +0530 Subject: [Mechanize-users] How to store a Mechanize object in the database? In-Reply-To: <185269960710062128y2e250055rd42c9f9835ad78b0@mail.gmail.com> References: <185269960710062128y2e250055rd42c9f9835ad78b0@mail.gmail.com> Message-ID: <185269960710071824g292c033cl97c931114f4cac46@mail.gmail.com> A pseudo solution of sorts... http://www.makuchaku.info/blog/how-to-effectively-serialize-a-wwwmechanize-object :) Maku On 10/7/07, ???? ??? (makuchaku) wrote: > > Hi, > > I am trying to save a Mechanize object in database (using a Rails Model). > But the save operation throws a TypeError > > Considering that "agent" is an instance of a Rails Model and "user" is > defined as a "text" type in the Model. > > irb(main):039:0> agent.user = WWW::Mechanize.new > # @user_agent="WWW-Mechanize/0.6.10 (http://rubyforge.org/projects/mechanize/)", > @log=nil, @proxy_addr=nil, @keep_alive=true, @user=nil, @pass=nil, > @conditional_requests=true, @proxy_port=nil, > @pluggable_parser=# @parsers={"text/html"=>WWW::Mechanize::Page}, defaultWWW::Mechanize::File, > read_timeoutnil, keep_alive_time300, ca_filenil, watch_for_setnil, > proxy_passnil, auth_hash{}, certnil, history[], open_timeoutnil, > connection_cache{}, cookie_jar# @jar={}>, proxy_usernil, digestnil, passwordnil, redirect_oktrue > > irb(main):040:0> agent.save > TypeError: can't dump anonymous class Class > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:6:in `to_yaml' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `node_export' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `add' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `to_yaml' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:17:in `each' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:17:in `to_yaml' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:16:in `map' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:16:in `to_yaml' > from /usr/lib/ruby/1.8/yaml.rb:387:in `call' > from /usr/lib/ruby/1.8/yaml.rb:387:in `emit' > from /usr/lib/ruby/1.8/yaml.rb:387:in `quick_emit' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:15:in `to_yaml' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `node_export' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `add' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:18:in `to_yaml' > from /usr/lib/ruby/1.8/yaml/rubytypes.rb:17:in `each' > ... 14 levels... > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:2029:in > `attributes_with_quotes' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:2128:in > `quoted_column_names' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1813:in > `create_without_callbacks' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.rb:254:in > `create_without_timestamps' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/timestamp.rb:39:in > `create' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1789:in > `create_or_update_without_callbacks' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.rb:242:in > `create_or_update' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1545:in > `save_without_validation' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/validations.rb:752:in > `save_without_transactions' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:129:in > `save' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in > `transaction' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:95:in > `transaction' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:121:in > `transaction' > from /var/lib/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:129:in > `save' > from (irb):40 > from (null):0irb(main):041:0> > > > Further, I tried creating a simple WWW::Mechanize instance and did a > "to_yaml" on the object - saw a similar error. > > How can I tackle this? Any suggestions would be welcome :) > > Thanks! > Maku > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20071008/204d11cd/attachment.html From mikemondragon at gmail.com Wed Oct 10 18:01:55 2007 From: mikemondragon at gmail.com (Mike Mondragon) Date: Wed, 10 Oct 2007 15:01:55 -0700 Subject: [Mechanize-users] Scraping AOL Webmail to login and fetch contacts? Message-ID: <967d3b9a0710101501v6b6a1c16qf4a2d5ee82e323b3@mail.gmail.com> I'm helping with a gem that is going to published under the contentfree project on rubyforge (http://rubyforge.org/projects/contentfree/). The gem is called "blackbook" and basically it will go and fetch your contacts from the major webmail providers. So far Gmail, Yahoo!, and MSN have been completed. We are trying to finish up with fetching contacts from AOL Webmail. However its a bit more difficult because of the javascript-like validation AOL has built into their sign-in service. The only resource I've found that talks about the correct strategy to sign-in to AOL via a scraping tool is here: http://apsquared.net/blog/2007/04/30/scraping-aol-webmail-for-contacts/ However we've not been able to recreate their experience with mechanize. Any suggestions or experience would be appreciated. Blackbook will be released onto rubyforge once we've completed AOL Webmail integration. Thanks Mike -- Mike Mondragon Work> http://sas.quat.ch/ Blog> http://blog.mondragon.cc/ Small URLs> http://hurl.it/ From bodo at wannawork.de Sat Oct 13 08:09:40 2007 From: bodo at wannawork.de (Bodo Tasche) Date: Sat, 13 Oct 2007 14:09:40 +0200 Subject: [Mechanize-users] Problems with Forms In-Reply-To: <2093A31B-7B04-460F-AF72-BF3CA13ADEDC@wannawork.de> References: <46FF6D86.9030905@wannawork.de> <20071002165250.GA18704@mac-mini.lan> <2093A31B-7B04-460F-AF72-BF3CA13ADEDC@wannawork.de> Message-ID: <6CEA9023-E68F-4E46-A380-5D87A3261580@wannawork.de> Hi, can anybody confirm my problem? Regards, Bodo Am 02.10.2007 um 23:36 schrieb Bodo Tasche: > Hi, > > Am 02.10.2007 um 18:52 schrieb Aaron Patterson: > >> Hi Bodo, >> >> On Sun, Sep 30, 2007 at 11:33:58AM +0200, Bodo Tasche wrote: >>> Hi, >>> >>> I am using mechanize for a while now. Works great. But at the >>> moment I >>> have a small problem using it. >>> >>> The problematic file is attached. In that PHP-File you will see a >>> formular, but mechanize doesn't recognize it. >> >> Could you be more specific? I'm not sure what you mean exactly. > > If I load that HTML-File using > > page=agent.get('URLTOFILE') > > the form-array is empty, page.forms.size returns 0 > >>> >>> Could somebody check this? >> >> Do you have a sample script to reproduce the error? > > page=agent.get('URLTOFILE') > puts page.forms.size > > Regards, > Bodo > > --- > http://www.wannawork.de > http://www.tvbrowser.org > > > _______________________________________________ > Mechanize-users mailing list > Mechanize-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mechanize-users --- http://www.wannawork.de http://www.tvbrowser.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mechanize-users/attachments/20071013/9d997cf1/attachment.html From mmacnair at gmail.com Fri Oct 26 19:54:04 2007 From: mmacnair at gmail.com (M Macnair) Date: Sat, 27 Oct 2007 00:54:04 +0100 Subject: [Mechanize-users] Post problems Message-ID: Hello, I'm having trouble using mechanise to post a form. I took a wireshark capture of the form being submitted by firefox and by mechanise. In both there is an HTTP Post and then a 'continuation of non-http traffic' packet. The obvious difference is that the firefox continuation packet has some http metadata (I don't know the proper terminology) where the mechanise packet has none. The firefox continuation packet: Content-Type: application/x-www-form-urlencoded\r\n Content-Length: 46 \r\n Wireshark then interprets what follows as 'Line-based text data: application/x-www-form-urlencoded", and this is the post data (field1=x&field2=y) The mechanize continuation packet does not contain this extra metadata, it merely contains the post data (field1=x&field2=y). Is this a problem? I'm not familiar enough with http to know. If it's NOT a problem, any ideas where the post could be going wrong? The page I get back is just as though I had requested it without posting any data at all. Thanks, Mike From mmacnair at gmail.com Fri Oct 26 19:54:04 2007 From: mmacnair at gmail.com (M Macnair) Date: Sat, 27 Oct 2007 00:54:04 +0100 Subject: [Mechanize-users] Post problems Message-ID: Hello, I'm having trouble using mechanise to post a form. I took a wireshark capture of the form being submitted by firefox and by mechanise. In both there is an HTTP Post and then a 'continuation of non-http traffic' packet. The obvious difference is that the firefox continuation packet has some http metadata (I don't know the proper terminology) where the mechanise packet has none. The firefox continuation packet: Content-Type: application/x-www-form-urlencoded\r\n Content-Length: 46 \r\n Wireshark then interprets what follows as 'Line-based text data: application/x-www-form-urlencoded", and this is the post data (field1=x&field2=y) The mechanize continuation packet does not contain this extra metadata, it merely contains the post data (field1=x&field2=y). Is this a problem? I'm not familiar enough with http to know. If it's NOT a problem, any ideas where the post could be going wrong? The page I get back is just as though I had requested it without posting any data at all. Thanks, Mike From aaron at tenderlovemaking.com Mon Oct 29 11:24:42 2007 From: aaron at tenderlovemaking.com (Aaron Patterson) Date: Mon, 29 Oct 2007 08:24:42 -0700 Subject: [Mechanize-users] Post problems In-Reply-To: References: Message-ID: <20071029152442.GA9799@mac-mini.lan> On Sat, Oct 27, 2007 at 12:54:04AM +0100, M Macnair wrote: > Hello, > > I'm having trouble using mechanise to post a form. I took a wireshark > capture of the form being submitted by firefox and by mechanise. In > both there is an HTTP Post and then a 'continuation of non-http > traffic' packet. The obvious difference is that the firefox > continuation packet has some http metadata (I don't know the proper > terminology) where the mechanise packet has none. > > The firefox continuation packet: > Content-Type: application/x-www-form-urlencoded\r\n > Content-Length: 46 > \r\n > Wireshark then interprets what follows as 'Line-based text data: > application/x-www-form-urlencoded", and this is the post data > (field1=x&field2=y) I'm not sure what a continuation packet is... Can you try examining the request/response with firebug or with livehttpheaders? > > The mechanize continuation packet does not contain this extra > metadata, it merely contains the post data (field1=x&field2=y). > > Is this a problem? I'm not familiar enough with http to know. If > it's NOT a problem, any ideas where the post could be going wrong? > The page I get back is just as though I had requested it without > posting any data at all. I don't think it should be a problem. Typically what I do to debug is to add a logger to mechanize, then compare mechanize output with livehttpheaders output and see where mechanize and firefox differ. Then get them to be the same. If you can provide a sample script that doesn't work, I can probably help more. -- Aaron Patterson http://tenderlovemaking.com/ From mmacnair at gmail.com Mon Oct 29 16:29:14 2007 From: mmacnair at gmail.com (M Macnair) Date: Mon, 29 Oct 2007 20:29:14 +0000 Subject: [Mechanize-users] Post problems In-Reply-To: <20071029152442.GA9799@mac-mini.lan> References: <20071029152442.GA9799@mac-mini.lan> Message-ID: On 10/29/07, Aaron Patterson wrote: > On Sat, Oct 27, 2007 at 12:54:04AM +0100, M Macnair wrote: > > Hello, > > > > I'm having trouble using mechanise to post a form. I took a wireshark > > capture of the form being submitted by firefox and by mechanise. In > > both there is an HTTP Post and then a 'continuation of non-http > > traffic' packet. The obvious difference is that the firefox > > continuation packet has some http metadata (I don't know the proper > > terminology) where the mechanise packet has none. > > > > The firefox continuation packet: > > Content-Type: application/x-www-form-urlencoded\r\n > > Content-Length: 46 > > \r\n > > Wireshark then interprets what follows as 'Line-based text data: > > application/x-www-form-urlencoded", and this is the post data > > (field1=x&field2=y) > > I'm not sure what a continuation packet is... Can you try examining the > request/response with firebug or with livehttpheaders? > A continuation packet is where it the same http message, but stretched across multiple tcp packets. It turns out this wasn't the problem though, both ways of sending it are perfectly valid. > > > > The mechanize continuation packet does not contain this extra > > metadata, it merely contains the post data (field1=x&field2=y). > > > > Is this a problem? I'm not familiar enough with http to know. If > > it's NOT a problem, any ideas where the post could be going wrong? > > The page I get back is just as though I had requested it without > > posting any data at all. > > I don't think it should be a problem. Typically what I do to debug is > to add a logger to mechanize, then compare mechanize output with > livehttpheaders output and see where mechanize and firefox differ. Then > get them to be the same. Useful advice, thanks. > > If you can provide a sample script that doesn't work, I can probably > help more. I got it working thanks, it was a cookie issue. I don't know whether anyone else would find it useful, but here is how I set an initial session cookie, which I didn't find particularly obvious from the documentation: class MyMech < WWW::Mechanize def initialize(session_id) super() Cookie::parse(ROOT_URI,'_session_id=' + session_id + '; path=/') { |c| cookie_jar.add(ROOT_URI,c)} end end