From toplek at polok.pl  Mon Jan  2 23:54:39 2012
From: toplek at polok.pl (Piotr Polok)
Date: Tue, 03 Jan 2012 05:54:39 +0100
Subject: [Mechanize-users] how to click button on the form with no names
Message-ID: <74ed6d3c351c3d8a831b86a68bd4423a@polok.pl>

HI,

I need to click second button on below form and download a file, could 
you please give a advice how to do that :)

HTML source:

             <form method="POST" target="_self">
                  <p style="margin-bottom: 
5px;margin-left:45px;margin-top:-40px;"><input id="radio_5" 
checked="checked" name="format" value="1" 
type="radio"style="vertical-align:middle;"><label for="radio_5"> liniowy 
</label></p></p>
                  <p style="margin-bottom: 5px;margin-left: 45px"><input 
id="radio_6" name="format" value="2" type="radio" 
style="vertical-align:middle;"><label for="radio_6"> xml</label></p>
                  <p style="margin-bottom: 5px;margin-left: 45px"><input 
id="radio_7"  name="format" value="3" 
type="radio"style="vertical-align:middle;"><label for="radio_7"> 
MT942</label></p>
                  <p style="margin-bottom: 5px;margin-left: 45px"><input 
id="radio_8" name="format" value="4" type="radio" 
style="vertical-align:middle;"><label for="radio_8"> 
MT942/Multicash</label></p>
             </td>
          </tr>
          <tr>
              <td colspan="2"><hr width="98%" color="#acacac" 
noshade="noshade" size="1" style="vertical-align: bottom"/></td></tr>
	  </tr>
           <tr>
               <td class="wyciagi_w_td1">
                 <button type="submit" class="buty" 
onclick="document.getElementById('cod').value='1';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px" >Zapisz 
parametry</button>
               </td>
               <td style="text-align: left;">
                 <input id="cod" name="cod" value="" type="hidden">
                 <button type="submit" class="buty" 
style="margin-left:20px" 
onclick="document.getElementById('cod').value='2';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px" >Eksportuj dane do pliku 
</button>

                 <a class="nw" 
href='javascript:openWindow("start.php?page=pdf&pdf=wirtualne&szukaj=wy_oddo&data_new1=2011-12-03&data_new2=2012-01-03&sort_typ=data&rows=100&rach=1&idnrb=5&rach_wir=1&rach_wirtualne=&types=oddo")' 
style="margin-left:30px"><img src="gfx/przyciski/pdf.gif" 
style="vertical-align:middle;margin-right:5px;height:28px">Wydruk 
.pdf</a>&nbsp;
                 <a class="nw" 
href='javascript:openWindow("start.php?page=wydruk&typdruk=wirtualne&szukaj=wy_oddo&data_new1=2011-12-03&data_new2=2012-01-03&sort_typ=data&rows=100&rach=1&idnrb=5&rach_wir=1&rach_wirtualne=&types=oddo")' 
style="margin-left:30px"><img src="gfx/przyciski/drukarka.png" 
style="vertical-align:middle;margin-right:5px;">Przygotuj wydruk</a>
              </td>
	  </tr>
	</table>
     </FORM>


-- 
br
Piotr Polok

From drbrain at segment7.net  Tue Jan  3 17:28:33 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Tue, 3 Jan 2012 14:28:33 -0800
Subject: [Mechanize-users] how to click button on the form with no names
In-Reply-To: <74ed6d3c351c3d8a831b86a68bd4423a@polok.pl>
References: <74ed6d3c351c3d8a831b86a68bd4423a@polok.pl>
Message-ID: <927ECFE1-67C9-4AB0-AA29-23F5F9188C96@segment7.net>

On Jan 2, 2012, at 8:54 PM, Piotr Polok wrote:
> I need to click second button on below form and download a file, could you please give a advice how to do that :)

Try:

form.field_with :text => 'Eksportuj dane do pliku '

http://mechanize.rubyforge.org/Mechanize/Form.html#method-i-field_with-28criteria-29

Or:

form.buttons[1]

From drbrain at segment7.net  Tue Jan  3 18:08:03 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Tue, 3 Jan 2012 15:08:03 -0800
Subject: [Mechanize-users] deobfuscating javascript?
In-Reply-To: <CAD5-QuPctqdgv-fHvr4ZxHhtsQUCA_OCgT79Q7vV9dyXDUMNQQ@mail.gmail.com>
References: <CAD5-QuPctqdgv-fHvr4ZxHhtsQUCA_OCgT79Q7vV9dyXDUMNQQ@mail.gmail.com>
Message-ID: <6CF9C5E4-CC3D-4754-8C65-8F5DE0B7F472@segment7.net>

On Dec 29, 2011, at 7:33 AM, shig odani wrote:
> Not sure if this post is quite appropriate for this list but wasn't sure where else to ask. I'm using mechanize to visit some pages that have obfuscated javascript and I'm wondering if there's some ruby or mechanize way to deobfuscate it or otherwise interact with the html elements which are being obfuscated by javascript.
> 
> For more background, please see http://www.labnol.org/software/deobfuscate-javascript/19815/

Without a Ruby JS interpreter with a DOM you won't be able to run javascript from the web like a browser.  This is an ambitious project and nobody has yet completed it.

In the mean time you can download the javascript the page links to and pass it through jsbeautifier.org (using mechanize) to have something you can read to figure out what to make mechanize do.


From toplek at polok.pl  Wed Jan  4 01:00:56 2012
From: toplek at polok.pl (Piotr Polok)
Date: Wed, 04 Jan 2012 07:00:56 +0100
Subject: [Mechanize-users] how to click button on the form with no names
In-Reply-To: <927ECFE1-67C9-4AB0-AA29-23F5F9188C96@segment7.net>
References: <74ed6d3c351c3d8a831b86a68bd4423a@polok.pl>
	<927ECFE1-67C9-4AB0-AA29-23F5F9188C96@segment7.net>
Message-ID: <c9251ce2373549dada5f9131469a7533@polok.pl>

On 03.01.2012 23:28, Eric Hodel wrote:
> On Jan 2, 2012, at 8:54 PM, Piotr Polok wrote:
>> I need to click second button on below form and download a file, 
>> could you please give a advice how to do that :)
>
> Try:
>
> form.field_with :text => 'Eksportuj dane do pliku '
>
> 
> http://mechanize.rubyforge.org/Mechanize/Form.html#method-i-field_with-28criteria-29
>
> Or:
>
> form.buttons[1]

HI,

Thank you for the reply!

-- 
br
Piotr Polok

From toplek at polok.pl  Wed Jan  4 01:33:16 2012
From: toplek at polok.pl (Piotr Polok)
Date: Wed, 04 Jan 2012 07:33:16 +0100
Subject: [Mechanize-users] How to download file after button click with no
	URL
Message-ID: <b519286444bf5487c94b9863b86c8206@polok.pl>

HI,

I'm trying to download file with no success using below code:

  $mech->get( $search_result );
  $mech->form_number( 4 );

  $mech->current_form()->click("eksport_do_pliku");
  print $mech->res();

This is part of HTML source:
"
<button type="submit" name="eksport_do_pliku" class="buty" 
style="margin-left:20px" 
onclick="document.getElementById('cod').value='2';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px">Eksportuj dane do pliku 
</button>
"

The result of "$mech->res();" is "HTTP::Response=HASH(0x8a49d10)"

Please help :)

-- 
best regards
Piotr Polok

From drbrain at segment7.net  Wed Jan  4 02:35:08 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Tue, 3 Jan 2012 23:35:08 -0800
Subject: [Mechanize-users] How to download file after button click with
	no URL
In-Reply-To: <b519286444bf5487c94b9863b86c8206@polok.pl>
References: <b519286444bf5487c94b9863b86c8206@polok.pl>
Message-ID: <250B57F1-8226-48DE-B78C-A260285C79FD@segment7.net>

On Jan 3, 2012, at 10:33 PM, Piotr Polok wrote:
> $mech->get( $search_result );
> $mech->form_number( 4 );
> 
> $mech->current_form()->click("eksport_do_pliku");
> print $mech->res();

Assuming you've required mechanize, how does this work?  Are you using perl mechanize?  If so, you've got the wrong list as this does not parse on ruby:

$ ruby -v -
ruby 1.8.7 (2010-01-10 patchlevel 249) [universal-darwin11.0]
$mech->get( $search_result );
$mech->form_number( 4 );

$mech->current_form()->click("eksport_do_pliku");
print $mech->res();
^D
-:1: syntax error, unexpected '>'
$mech->get( $search_result );
       ^
-:2: syntax error, unexpected '>'
$mech->form_number( 4 );
       ^
-:4: syntax error, unexpected '>'
$mech->current_form()->click("eksport_do_pliku");
       ^
-:4: syntax error, unexpected '>'
$mech->current_form()->click("eksport_do_pliku");
                       ^
-:5: syntax error, unexpected '>'
print $mech->res();
             ^


From toplek at polok.pl  Wed Jan  4 02:44:33 2012
From: toplek at polok.pl (Piotr Polok)
Date: Wed, 04 Jan 2012 08:44:33 +0100
Subject: [Mechanize-users] How to download file after button click with
	no URL
In-Reply-To: <250B57F1-8226-48DE-B78C-A260285C79FD@segment7.net>
References: <b519286444bf5487c94b9863b86c8206@polok.pl>
	<250B57F1-8226-48DE-B78C-A260285C79FD@segment7.net>
Message-ID: <b812986bb7efabe05a2a34fa56e80710@polok.pl>

On 04.01.2012 08:35, Eric Hodel wrote:
> On Jan 3, 2012, at 10:33 PM, Piotr Polok wrote:
>> $mech->get( $search_result );
>> $mech->form_number( 4 );
>>
>> $mech->current_form()->click("eksport_do_pliku");
>> print $mech->res();
>
> Assuming you've required mechanize, how does this work?  Are you
> using perl mechanize?  If so, you've got the wrong list as this does
> not parse on ruby:

HI,

You are right, sorry!

-- 
br
Piotr Polok

From alex at blackkettle.org  Wed Jan  4 04:38:55 2012
From: alex at blackkettle.org (Alex Young)
Date: Wed, 04 Jan 2012 09:38:55 +0000
Subject: [Mechanize-users] deobfuscating javascript?
In-Reply-To: <CAD5-QuPctqdgv-fHvr4ZxHhtsQUCA_OCgT79Q7vV9dyXDUMNQQ@mail.gmail.com>
References: <CAD5-QuPctqdgv-fHvr4ZxHhtsQUCA_OCgT79Q7vV9dyXDUMNQQ@mail.gmail.com>
Message-ID: <4F041E2F.4050607@blackkettle.org>

On 29/12/11 15:33, shig odani wrote:
> Not sure if this post is quite appropriate for this list but wasn't sure
> where else to ask. I'm using mechanize to visit some pages that have
> obfuscated javascript and I'm wondering if there's some ruby or
> mechanize way to deobfuscate it or otherwise interact with the html
> elements which are being obfuscated by javascript.
>
> For more background, please see
> http://www.labnol.org/software/deobfuscate-javascript/19815/

Take a look at Culerity.

-- 
Alex

From miwarin at gmail.com  Sat Jan  7 10:36:38 2012
From: miwarin at gmail.com (=?ISO-2022-JP?B?GyRCOzBOWD84GyhCKCBNaXdhIFN1c3VtdSAp?=)
Date: Sun, 8 Jan 2012 00:36:38 +0900
Subject: [Mechanize-users] 500 error in tdiary using mechanize
Message-ID: <CAM_ND6rBaNnxYHtBRYsEh0HvtfUCXN2gfT3yoQLi8Js3kvyUkA@mail.gmail.com>

hi all

i use tdiary( The web diary system http://sourceforge.net/projects/tdiary/ ).

i am trying to update the diary with the mechanize.

i run the code (see below), mechanize returns
Net::HTTPInternalServerError (Mechanize::ResponseCodeError)

Googled Treats, that it's a server-side error.

(e.g. but perl... WWW::Mechanize::FAQ - search.cpan.org My Mech
program doesn't work, but it works in the browser.
http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod#My_Mech_program_doesn%27t_work,_but_it_works_in_the_browser.
)

However, there are no errors on the server side.

Do people know the cause?

In addition, web browser( Firefox ) can safely update my diary.

[Environment]
% ruby --version
ruby 1.8.7 (2011-02-18 patchlevel 334) [i386-netbsdelf]

% gem list

*** LOCAL GEMS ***

domain_name (0.5.1)
mechanize (2.1)
net-http-digest_auth (1.2)
net-http-persistent (2.3.3)
nokogiri (1.5.0)
ntlm-http (0.1.1)
tmail (1.2.7.1)
unf (0.0.4)
unf_ext (0.0.4)
webrobots (0.0.12)


 * NetBSD 5.99.58
 * tdiary 3.1.0


[code]
#!/usr/pkg/bin/ruby -Ku

require 'rubygems'
require 'mechanize'
require 'logger'
require 'pp'

def main
  uri = "http://localhost/~rin/diary/update.rb"
  user = "test"
  pass = "test"
  referer = "http://www.area51.gr.jp/~rin/diary/update.rb"

  text = "!test\nhello tdiary!\n"

  agent = Mechanize.new
  agent.log = Logger.new($stdout)
#  agent.log.level = Logger::INFO
  agent.auth(user, pass)                 <= i use digest authentication
  agent.get(uri, nil, referer)
  agent.page.form_with(:action => "update.rb") { |form|
    form['body'] = text
    form.submit(form.button_with(:name => "append"))           <=
seems to be an error here.
  }

end

main


[mechanize log]
% ./tdwrite.rb
I, [2012-01-08T00:25:20.594807 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595147 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.595197 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.595264 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.595308 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.595352 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595395 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.595434 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.598261 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.598313 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.598359 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.598397 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.598433 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.598469 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.598504 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.598538 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.598573 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=100
D, [2012-01-08T00:25:20.598794 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.598965 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.599088 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.605070 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605298 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.605371 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.605417 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.605462 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.605499 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
nc=00000000, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="c304545256e7e4d0a03c3e2d6ae4ec80"
D, [2012-01-08T00:25:20.605545 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605581 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.605616 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.755405 #9165]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2012-01-08T00:25:20.755802 #9165] DEBUG -- : response-header:
content-type => text/html; charset=UTF-8
D, [2012-01-08T00:25:20.755878 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.755925 #9165] DEBUG -- : response-header:
authentication-info => rspauth="af4628a938bc6d2b04bf73ba60fb4a59",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000000, qop=auth
D, [2012-01-08T00:25:20.755962 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.755998 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.756032 #9165] DEBUG -- : response-header:
content-length => 7160
D, [2012-01-08T00:25:20.756067 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.756102 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=99
D, [2012-01-08T00:25:20.756137 #9165] DEBUG -- : response-header: vary
=> User-Agent,Accept-Encoding
D, [2012-01-08T00:25:20.756250 #9165] DEBUG -- : Read 7160 bytes (7160 total)
D, [2012-01-08T00:25:20.756403 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.772181 #9165]  INFO -- : form encoding: ISO-8859-1
D, [2012-01-08T00:25:20.774358 #9165] DEBUG -- : query:
"old=20120108&year=2012&month=1&day=8&title=&body=%21test%0Ahello+tdiary%21%0A&plugin_ping_send=true&api=0&plugin_tags_filter=on&plugin_tb_url=&plugin_tb_section=p01&plugin_tb_excerpt=&append=+%E8%BF%BD%E8%A8%98+"
I, [2012-01-08T00:25:20.775397 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.775641 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.775879 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.776086 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.776280 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.776507 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.776697 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.776922 #9165] DEBUG -- : request-header:
referer => http://diary:diary at localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.777128 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.777381 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.778814 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.779127 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.779357 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.779569 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.779818 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.780037 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.780231 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.780439 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.780628 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=98
D, [2012-01-08T00:25:20.780847 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.781100 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.781460 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.787175 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.787408 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.787652 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.787889 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.788136 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.788393 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.788710 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
nc=00000001, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="a903c6e42ed5896cd1da05516fc5b93d"
D, [2012-01-08T00:25:20.789087 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.789302 #9165] DEBUG -- : request-header:
referer => http://diary:diary at localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.789520 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.789711 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.842091 #9165]  INFO -- : status:
Net::HTTPInternalServerError 1.1 500 Internal Server Error
D, [2012-01-08T00:25:20.842477 #9165] DEBUG -- : response-header:
content-type => text/html
D, [2012-01-08T00:25:20.842718 #9165] DEBUG -- : response-header:
connection => close
D, [2012-01-08T00:25:20.842961 #9165] DEBUG -- : response-header:
authentication-info => rspauth="f92b2000ce402faad6c351d19abdc07c",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000001, qop=auth
D, [2012-01-08T00:25:20.843216 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.843419 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.843644 #9165] DEBUG -- : response-header:
content-length => 416
D, [2012-01-08T00:25:20.843865 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.844076 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.844377 #9165] DEBUG -- : Read 416 bytes (416 total)
D, [2012-01-08T00:25:20.844808 #9165] DEBUG -- : gzip body
/usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:303:in
`fetch': 500 => Net::HTTPInternalServerError
(Mechanize::ResponseCodeError)
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:655:in
`response_authenticate'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:300:in
`fetch'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:1036:in
`post_form'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:426:in
`submit'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/form.rb:178:in
`submit'
        from ./tdwrite.rb:24:in `main'
        from (eval):23:in `form_with'
        from ./tdwrite.rb:22:in `main'
        from ./tdwrite.rb:29


[apache web server access_log]

127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "GET /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "GET
/~rin/diary/update.rb HTTP/1.1" 200 7160
127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "POST /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "POST
/~rin/diary/update.rb HTTP/1.1" 500 416


[apache web server error_log]

(In particular there are no errors)

-- 
miwa

From gerryster at gmail.com  Sun Jan  8 07:55:18 2012
From: gerryster at gmail.com (Ryan Gerry)
Date: Sun, 8 Jan 2012 06:55:18 -0600
Subject: [Mechanize-users] 500 error in tdiary using mechanize
In-Reply-To: <CAM_ND6rBaNnxYHtBRYsEh0HvtfUCXN2gfT3yoQLi8Js3kvyUkA@mail.gmail.com>
References: <CAM_ND6rBaNnxYHtBRYsEh0HvtfUCXN2gfT3yoQLi8Js3kvyUkA@mail.gmail.com>
Message-ID: <CAJmQBMJfOhUGpfHf6ys_0dumcx0iPqD9cofmnJNb31y4=5Y=7A@mail.gmail.com>

I am not sure about the root cause, but have you tried changing the user
agent string to Firefox?  If that does not work, use the net panel in
Firefox to carefully compare the headers to Mechanize to see if there is
something different/missing.

Ryan

On Sat, Jan 7, 2012 at 9:36 AM, ???( Miwa Susumu ) <miwarin at gmail.com>wrote:

> hi all
>
> i use tdiary( The web diary system http://sourceforge.net/projects/tdiary/).
>
> i am trying to update the diary with the mechanize.
>
> i run the code (see below), mechanize returns
> Net::HTTPInternalServerError (Mechanize::ResponseCodeError)
>
> Googled Treats, that it's a server-side error.
>
> (e.g. but perl... WWW::Mechanize::FAQ - search.cpan.org My Mech
> program doesn't work, but it works in the browser.
>
> http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod#My_Mech_program_doesn%27t_work,_but_it_works_in_the_browser
> .
> )
>
> However, there are no errors on the server side.
>
> Do people know the cause?
>
> In addition, web browser( Firefox ) can safely update my diary.
>
> [Environment]
> % ruby --version
> ruby 1.8.7 (2011-02-18 patchlevel 334) [i386-netbsdelf]
>
> % gem list
>
> *** LOCAL GEMS ***
>
> domain_name (0.5.1)
> mechanize (2.1)
> net-http-digest_auth (1.2)
> net-http-persistent (2.3.3)
> nokogiri (1.5.0)
> ntlm-http (0.1.1)
> tmail (1.2.7.1)
> unf (0.0.4)
> unf_ext (0.0.4)
> webrobots (0.0.12)
>
>
>  * NetBSD 5.99.58
>  * tdiary 3.1.0
>
>
> [code]
> #!/usr/pkg/bin/ruby -Ku
>
> require 'rubygems'
> require 'mechanize'
> require 'logger'
> require 'pp'
>
> def main
>  uri = "http://localhost/~rin/diary/update.rb"
>  user = "test"
>  pass = "test"
>  referer = "http://www.area51.gr.jp/~rin/diary/update.rb"
>
>  text = "!test\nhello tdiary!\n"
>
>  agent = Mechanize.new
>  agent.log = Logger.new($stdout)
> #  agent.log.level = Logger::INFO
>  agent.auth(user, pass)                 <= i use digest authentication
>  agent.get(uri, nil, referer)
>  agent.page.form_with(:action => "update.rb") { |form|
>    form['body'] = text
>    form.submit(form.button_with(:name => "append"))           <=
> seems to be an error here.
>  }
>
> end
>
> main
>
>
> [mechanize log]
> % ./tdwrite.rb
> I, [2012-01-08T00:25:20.594807 #9165]  INFO -- : Net::HTTP::Get:
> /~rin/diary/update.rb
> D, [2012-01-08T00:25:20.595147 #9165] DEBUG -- : request-header:
> accept-language => en-us,en;q=0.5
> D, [2012-01-08T00:25:20.595197 #9165] DEBUG -- : request-header: accept =>
> */*
> D, [2012-01-08T00:25:20.595264 #9165] DEBUG -- : request-header:
> user-agent => Mechanize/2.1 Ruby/1.8.7p334
> (http://github.com/tenderlove/mechanize/)
> D, [2012-01-08T00:25:20.595308 #9165] DEBUG -- : request-header:
> accept-encoding => gzip,deflate,identity
> D, [2012-01-08T00:25:20.595352 #9165] DEBUG -- : request-header:
> referer => http://www.area51.gr.jp/~rin/diary/update.rb
> D, [2012-01-08T00:25:20.595395 #9165] DEBUG -- : request-header: host
> => localhost
> D, [2012-01-08T00:25:20.595434 #9165] DEBUG -- : request-header:
> accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
> I, [2012-01-08T00:25:20.598261 #9165]  INFO -- : status:
> Net::HTTPUnauthorized 1.1 401 Authorization Required
> D, [2012-01-08T00:25:20.598313 #9165] DEBUG -- : response-header:
> content-type => text/html; charset=iso-8859-1
> D, [2012-01-08T00:25:20.598359 #9165] DEBUG -- : response-header:
> connection => Keep-Alive
> D, [2012-01-08T00:25:20.598397 #9165] DEBUG -- : response-header:
> www-authenticate => Digest realm="tDiary",
> nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
> algorithm=MD5, qop="auth"
> D, [2012-01-08T00:25:20.598433 #9165] DEBUG -- : response-header:
> server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
> D, [2012-01-08T00:25:20.598469 #9165] DEBUG -- : response-header: date
> => Sat, 07 Jan 2012 15:25:20 GMT
> D, [2012-01-08T00:25:20.598504 #9165] DEBUG -- : response-header:
> content-length => 290
> D, [2012-01-08T00:25:20.598538 #9165] DEBUG -- : response-header:
> content-encoding => gzip
> D, [2012-01-08T00:25:20.598573 #9165] DEBUG -- : response-header:
> keep-alive => timeout=5, max=100
> D, [2012-01-08T00:25:20.598794 #9165] DEBUG -- : response-header: vary
> => Accept-Encoding
> D, [2012-01-08T00:25:20.598965 #9165] DEBUG -- : Read 290 bytes (290 total)
> D, [2012-01-08T00:25:20.599088 #9165] DEBUG -- : gzip body
> I, [2012-01-08T00:25:20.605070 #9165]  INFO -- : Net::HTTP::Get:
> /~rin/diary/update.rb
> D, [2012-01-08T00:25:20.605298 #9165] DEBUG -- : request-header:
> accept-language => en-us,en;q=0.5
> D, [2012-01-08T00:25:20.605371 #9165] DEBUG -- : request-header: accept =>
> */*
> D, [2012-01-08T00:25:20.605417 #9165] DEBUG -- : request-header:
> user-agent => Mechanize/2.1 Ruby/1.8.7p334
> (http://github.com/tenderlove/mechanize/)
> D, [2012-01-08T00:25:20.605462 #9165] DEBUG -- : request-header:
> accept-encoding => gzip,deflate,identity
> D, [2012-01-08T00:25:20.605499 #9165] DEBUG -- : request-header:
> authorization => Digest username="diary", realm="tDiary", qop=auth,
> uri="/~rin/diary/update.rb",
> nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
> nc=00000000, cnonce="a6a1370113ae4db8792628a3e87927e8",
> response="c304545256e7e4d0a03c3e2d6ae4ec80"
> D, [2012-01-08T00:25:20.605545 #9165] DEBUG -- : request-header:
> referer => http://www.area51.gr.jp/~rin/diary/update.rb
> D, [2012-01-08T00:25:20.605581 #9165] DEBUG -- : request-header: host
> => localhost
> D, [2012-01-08T00:25:20.605616 #9165] DEBUG -- : request-header:
> accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
> I, [2012-01-08T00:25:20.755405 #9165]  INFO -- : status: Net::HTTPOK 1.1
> 200 OK
> D, [2012-01-08T00:25:20.755802 #9165] DEBUG -- : response-header:
> content-type => text/html; charset=UTF-8
> D, [2012-01-08T00:25:20.755878 #9165] DEBUG -- : response-header:
> connection => Keep-Alive
> D, [2012-01-08T00:25:20.755925 #9165] DEBUG -- : response-header:
> authentication-info => rspauth="af4628a938bc6d2b04bf73ba60fb4a59",
> cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000000, qop=auth
> D, [2012-01-08T00:25:20.755962 #9165] DEBUG -- : response-header:
> server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
> D, [2012-01-08T00:25:20.755998 #9165] DEBUG -- : response-header: date
> => Sat, 07 Jan 2012 15:25:20 GMT
> D, [2012-01-08T00:25:20.756032 #9165] DEBUG -- : response-header:
> content-length => 7160
> D, [2012-01-08T00:25:20.756067 #9165] DEBUG -- : response-header:
> content-encoding => gzip
> D, [2012-01-08T00:25:20.756102 #9165] DEBUG -- : response-header:
> keep-alive => timeout=5, max=99
> D, [2012-01-08T00:25:20.756137 #9165] DEBUG -- : response-header: vary
> => User-Agent,Accept-Encoding
> D, [2012-01-08T00:25:20.756250 #9165] DEBUG -- : Read 7160 bytes (7160
> total)
> D, [2012-01-08T00:25:20.756403 #9165] DEBUG -- : gzip body
> I, [2012-01-08T00:25:20.772181 #9165]  INFO -- : form encoding: ISO-8859-1
> D, [2012-01-08T00:25:20.774358 #9165] DEBUG -- : query:
>
> "old=20120108&year=2012&month=1&day=8&title=&body=%21test%0Ahello+tdiary%21%0A&plugin_ping_send=true&api=0&plugin_tags_filter=on&plugin_tb_url=&plugin_tb_section=p01&plugin_tb_excerpt=&append=+%E8%BF%BD%E8%A8%98+"
> I, [2012-01-08T00:25:20.775397 #9165]  INFO -- : Net::HTTP::Post:
> /~rin/diary/update.rb
> D, [2012-01-08T00:25:20.775641 #9165] DEBUG -- : request-header:
> accept-language => en-us,en;q=0.5
> D, [2012-01-08T00:25:20.775879 #9165] DEBUG -- : request-header: accept =>
> */*
> D, [2012-01-08T00:25:20.776086 #9165] DEBUG -- : request-header:
> content-type => application/x-www-form-urlencoded
> D, [2012-01-08T00:25:20.776280 #9165] DEBUG -- : request-header:
> user-agent => Mechanize/2.1 Ruby/1.8.7p334
> (http://github.com/tenderlove/mechanize/)
> D, [2012-01-08T00:25:20.776507 #9165] DEBUG -- : request-header:
> accept-encoding => gzip,deflate,identity
> D, [2012-01-08T00:25:20.776697 #9165] DEBUG -- : request-header:
> content-length => 211
> D, [2012-01-08T00:25:20.776922 #9165] DEBUG -- : request-header:
> referer => http://diary:diary at localhost/~rin/diary/update.rb
> D, [2012-01-08T00:25:20.777128 #9165] DEBUG -- : request-header: host
> => localhost
> D, [2012-01-08T00:25:20.777381 #9165] DEBUG -- : request-header:
> accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
> I, [2012-01-08T00:25:20.778814 #9165]  INFO -- : status:
> Net::HTTPUnauthorized 1.1 401 Authorization Required
> D, [2012-01-08T00:25:20.779127 #9165] DEBUG -- : response-header:
> content-type => text/html; charset=iso-8859-1
> D, [2012-01-08T00:25:20.779357 #9165] DEBUG -- : response-header:
> connection => Keep-Alive
> D, [2012-01-08T00:25:20.779569 #9165] DEBUG -- : response-header:
> www-authenticate => Digest realm="tDiary",
> nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
> algorithm=MD5, qop="auth"
> D, [2012-01-08T00:25:20.779818 #9165] DEBUG -- : response-header:
> server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
> D, [2012-01-08T00:25:20.780037 #9165] DEBUG -- : response-header: date
> => Sat, 07 Jan 2012 15:25:20 GMT
> D, [2012-01-08T00:25:20.780231 #9165] DEBUG -- : response-header:
> content-length => 290
> D, [2012-01-08T00:25:20.780439 #9165] DEBUG -- : response-header:
> content-encoding => gzip
> D, [2012-01-08T00:25:20.780628 #9165] DEBUG -- : response-header:
> keep-alive => timeout=5, max=98
> D, [2012-01-08T00:25:20.780847 #9165] DEBUG -- : response-header: vary
> => Accept-Encoding
> D, [2012-01-08T00:25:20.781100 #9165] DEBUG -- : Read 290 bytes (290 total)
> D, [2012-01-08T00:25:20.781460 #9165] DEBUG -- : gzip body
> I, [2012-01-08T00:25:20.787175 #9165]  INFO -- : Net::HTTP::Post:
> /~rin/diary/update.rb
> D, [2012-01-08T00:25:20.787408 #9165] DEBUG -- : request-header:
> accept-language => en-us,en;q=0.5
> D, [2012-01-08T00:25:20.787652 #9165] DEBUG -- : request-header: accept =>
> */*
> D, [2012-01-08T00:25:20.787889 #9165] DEBUG -- : request-header:
> content-type => application/x-www-form-urlencoded
> D, [2012-01-08T00:25:20.788136 #9165] DEBUG -- : request-header:
> user-agent => Mechanize/2.1 Ruby/1.8.7p334
> (http://github.com/tenderlove/mechanize/)
> D, [2012-01-08T00:25:20.788393 #9165] DEBUG -- : request-header:
> accept-encoding => gzip,deflate,identity
> D, [2012-01-08T00:25:20.788710 #9165] DEBUG -- : request-header:
> authorization => Digest username="diary", realm="tDiary", qop=auth,
> uri="/~rin/diary/update.rb",
> nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
> nc=00000001, cnonce="a6a1370113ae4db8792628a3e87927e8",
> response="a903c6e42ed5896cd1da05516fc5b93d"
> D, [2012-01-08T00:25:20.789087 #9165] DEBUG -- : request-header:
> content-length => 211
> D, [2012-01-08T00:25:20.789302 #9165] DEBUG -- : request-header:
> referer => http://diary:diary at localhost/~rin/diary/update.rb
> D, [2012-01-08T00:25:20.789520 #9165] DEBUG -- : request-header: host
> => localhost
> D, [2012-01-08T00:25:20.789711 #9165] DEBUG -- : request-header:
> accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
> I, [2012-01-08T00:25:20.842091 #9165]  INFO -- : status:
> Net::HTTPInternalServerError 1.1 500 Internal Server Error
> D, [2012-01-08T00:25:20.842477 #9165] DEBUG -- : response-header:
> content-type => text/html
> D, [2012-01-08T00:25:20.842718 #9165] DEBUG -- : response-header:
> connection => close
> D, [2012-01-08T00:25:20.842961 #9165] DEBUG -- : response-header:
> authentication-info => rspauth="f92b2000ce402faad6c351d19abdc07c",
> cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000001, qop=auth
> D, [2012-01-08T00:25:20.843216 #9165] DEBUG -- : response-header:
> server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
> D, [2012-01-08T00:25:20.843419 #9165] DEBUG -- : response-header: date
> => Sat, 07 Jan 2012 15:25:20 GMT
> D, [2012-01-08T00:25:20.843644 #9165] DEBUG -- : response-header:
> content-length => 416
> D, [2012-01-08T00:25:20.843865 #9165] DEBUG -- : response-header:
> content-encoding => gzip
> D, [2012-01-08T00:25:20.844076 #9165] DEBUG -- : response-header: vary
> => Accept-Encoding
> D, [2012-01-08T00:25:20.844377 #9165] DEBUG -- : Read 416 bytes (416 total)
> D, [2012-01-08T00:25:20.844808 #9165] DEBUG -- : gzip body
>
> /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:303:in
> `fetch': 500 => Net::HTTPInternalServerError
> (Mechanize::ResponseCodeError)
>        from
> /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:655:in
> `response_authenticate'
>        from
> /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:300:in
> `fetch'
>        from
> /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:1036:in
> `post_form'
>        from
> /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:426:in
> `submit'
>        from
> /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/form.rb:178:in
> `submit'
>        from ./tdwrite.rb:24:in `main'
>        from (eval):23:in `form_with'
>        from ./tdwrite.rb:22:in `main'
>        from ./tdwrite.rb:29
>
>
> [apache web server access_log]
>
> 127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "GET /~rin/diary/update.rb
> HTTP/1.1" 401 290
> 127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "GET
> /~rin/diary/update.rb HTTP/1.1" 200 7160
> 127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "POST /~rin/diary/update.rb
> HTTP/1.1" 401 290
> 127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "POST
> /~rin/diary/update.rb HTTP/1.1" 500 416
>
>
> [apache web server error_log]
>
> (In particular there are no errors)
>
> --
> miwa
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120108/72790050/attachment-0001.html>

From miwarin at gmail.com  Sun Jan  8 10:58:58 2012
From: miwarin at gmail.com (=?UTF-8?B?IuS4iei8quaZiyhNaXdhIFN1c3VtdSki?=)
Date: Mon, 09 Jan 2012 00:58:58 +0900
Subject: [Mechanize-users] 500 error in tdiary using mechanize
In-Reply-To: <CAJmQBMJfOhUGpfHf6ys_0dumcx0iPqD9cofmnJNb31y4=5Y=7A@mail.gmail.com>
References: <CAM_ND6rBaNnxYHtBRYsEh0HvtfUCXN2gfT3yoQLi8Js3kvyUkA@mail.gmail.com>
	<CAJmQBMJfOhUGpfHf6ys_0dumcx0iPqD9cofmnJNb31y4=5Y=7A@mail.gmail.com>
Message-ID: <4F09BD42.3010807@gmail.com>

hi Ryan

(2012/01/08 21:55), Ryan Gerry wrote:
> I am not sure about the root cause, but have you tried changing the 
> user agent string to Firefox?  If that does not work, use the net 
> panel in Firefox to carefully compare the headers to Mechanize to see 
> if there is something different/missing.

install Live HTTP Headers 
(https://addons.mozilla.org/ja/firefox/addon/live-http-headers/) to Firefox.
compared to Firefox and mechanize. referer were different.

mechanize:
   request-header: referer => 
http://test:test at www.area51.gr.jp/~rin/diary/update.rb

Firefox:
   Referer: http://www.area51.gr.jp/~rin/diary/update.rb

added the referer to submit. Then it worked!
       form.submit(form.button_with(:name => "append"), 
headers={"referer" => referer})


The complete code is this.

----
#!/usr/pkg/bin/ruby -Ku

require 'rubygems'
require 'mechanize'
require 'logger'
require 'pp'

def main
   uri = "http://localhost/~rin/diary/update.rb"
   user = "test"
   pass = "test"
   referer = "http://www.area51.gr.jp/~rin/diary/update.rb"

   text = "!test\nhello tdiary!\n"

   agent = Mechanize.new
   agent.log = Logger.new($stdout)
   agent.auth(user, pass)
   agent.get(uri, nil, referer)
   agent.page.form_with(:action => "update.rb") { |form|
     form['body'] = text
     form.submit(form.button_with(:name => "append"), headers={"referer" 
=> referer})
   }

end

main
----

This is ok.

thanks!


From jack at groundbreakingsoftware.com  Sat Jan 14 21:45:58 2012
From: jack at groundbreakingsoftware.com (Jack Royal-Gordon)
Date: Sat, 14 Jan 2012 18:45:58 -0800
Subject: [Mechanize-users] Reload previously retrieved page
Message-ID: <9C53B75D-52AD-401E-97E7-67437BFEF424@GroundbreakingSoftware.com>

I'm a newbie with Mechanize, and I have what seems like a simple question: I'm trying to save a page that has been loaded and then reload it later for further processing.  I see that Mechanize has a method parse(uri, response, body) that (I presume) can load the DOM without physically accessing the site over the internet.  If I save the URI and body from a previous Mechanize.get(), how do I get the "response" from the page that is also required in order to reload the page?  Or is there a better way to save and restore the page without incurring the latency of a second internet access?  I've also tried Mechanize::Page.new() but I have the same problem in that I need a "response" for that call (nil gives an error).



From benmanns at gmail.com  Tue Jan 17 16:45:31 2012
From: benmanns at gmail.com (Benjamin Manns)
Date: Tue, 17 Jan 2012 16:45:31 -0500
Subject: [Mechanize-users] Reload previously retrieved page
In-Reply-To: <9C53B75D-52AD-401E-97E7-67437BFEF424@GroundbreakingSoftware.com>
References: <9C53B75D-52AD-401E-97E7-67437BFEF424@GroundbreakingSoftware.com>
Message-ID: <CAALv9fURNe5PSrd0ZAtmT5mUf7pADj=zAWy6qqNLVsOLprp=AQ@mail.gmail.com>

Hey Jack,

Mechanize uses Nokogiri <http://nokogiri.org/> to parse HTML. You can use
Nokogiri directly to parse HTML or XML from a string or
file<http://nokogiri.org/tutorials/parsing_an_html_xml_document.html>.
However, this returns a Nokogiri::HTML::Document object, while Mechanize
returns a Mechanize::Page object - the Nokogiri document is the same
information that you would get by calling page.root. This is usually all
that you need, but it lacks the convenience methods that a
Mechanize::Pageaffords.

To recreate a Mechanize::Page, use the ::new
method<http://mechanize.rubyforge.org/Mechanize/Page.html#method-c-new>(note:
I haven't fully tested this):

agent = Mechanize.new
content = ... # your content as a string - you could read from a file here.
# Arguments: (uri, response headers, body, response code, mechanize agent)
# The ones I've left nil seemed to not be required, though leaving them out
may affect other methods.
page = Mechanize::Page.new(nil, {'content-type' => 'text/html'}, content,
nil, agent)

As you can tell, this is a bit more complicated than simply running
Nokogiri::HTML(content), but it gives you all the Mechanize::Page methods.


Ben

On Sat, Jan 14, 2012 at 9:45 PM, Jack Royal-Gordon <
jack at groundbreakingsoftware.com> wrote:

> I'm a newbie with Mechanize, and I have what seems like a simple question:
> I'm trying to save a page that has been loaded and then reload it later for
> further processing.  I see that Mechanize has a method parse(uri, response,
> body) that (I presume) can load the DOM without physically accessing the
> site over the internet.  If I save the URI and body from a previous
> Mechanize.get(), how do I get the "response" from the page that is also
> required in order to reload the page?  Or is there a better way to save and
> restore the page without incurring the latency of a second internet access?
>  I've also tried Mechanize::Page.new() but I have the same problem in that
> I need a "response" for that call (nil gives an error).
>
>
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>



-- 
Benjamin Manns
benmanns at gmail.com
(434) 321-8324
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120117/3d8ee574/attachment.html>

From rdpoor at gmail.com  Fri Jan 27 15:41:25 2012
From: rdpoor at gmail.com (Robert Poor)
Date: Fri, 27 Jan 2012 12:41:25 -0800
Subject: [Mechanize-users] dealing with Encoding::UndefinedConversionError:
 "\xA2" from ASCII-8BIT to UTF-8
Message-ID: <CAGHqdqXLd+KkwSS3y9g0c__1HXdBeqZZp2pN1qtvsazCGcRKzw@mail.gmail.com>

I'm 99% certain that this is what's being discussed in

https://github.com/tenderlove/mechanize/issues/168

But I'm not sure what the remedy is.  My production app is running on
Heroku, so I need a fix that I can push there.  Can someone post
instructions that this neophyte can follow?  (I gather I'm supposed to fork
and add the current master version from github to my config, but I need
simple little steps for muddy little feet.)  Thanks...

I'm running Rails 3.2, Mechanize 2.1.  Mechanize is failing with

        Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8

with this (partial) stack trace:

from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize/http/agent.rb:807:in
`block in response_read'
from /usr/local/lib/ruby/1.9.1/net/protocol.rb:387:in `call_block'
from /usr/local/lib/ruby/1.9.1/net/protocol.rb:378:in `<<'
from /usr/local/lib/ruby/1.9.1/net/protocol.rb:88:in `read'
from /usr/local/lib/ruby/1.9.1/net/http.rb:2445:in `read_chunked'
from /usr/local/lib/ruby/1.9.1/net/http.rb:2419:in `read_body_0'
from /usr/local/lib/ruby/1.9.1/net/http.rb:2379:in `read_body'
from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize/http/agent.rb:795:in
`response_read'
from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize/http/agent.rb:267:in
`block in fetch'
from /usr/local/lib/ruby/1.9.1/net/http.rb:1194:in `block in
transport_request'
from /usr/local/lib/ruby/1.9.1/net/http.rb:2342:in `reading_body'
from /usr/local/lib/ruby/1.9.1/net/http.rb:1193:in `transport_request'
from /usr/local/lib/ruby/1.9.1/net/http.rb:1177:in `request'
from
/app/vendor/bundle/ruby/1.9.1/gems/net-http-persistent-2.3.3/lib/net/http/persistent.rb:621:in
`request'
from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize/http/agent.rb:264:in
`fetch'
from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize.rb:319:in
`get'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120127/76b5ea73/attachment.html>

From rdpoor at gmail.com  Fri Jan 27 16:52:27 2012
From: rdpoor at gmail.com (Robert Poor)
Date: Fri, 27 Jan 2012 13:52:27 -0800
Subject: [Mechanize-users] dealing with
 Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8
In-Reply-To: <CAGHqdqXLd+KkwSS3y9g0c__1HXdBeqZZp2pN1qtvsazCGcRKzw@mail.gmail.com>
References: <CAGHqdqXLd+KkwSS3y9g0c__1HXdBeqZZp2pN1qtvsazCGcRKzw@mail.gmail.com>
Message-ID: <CAGHqdqU=07NyRU4y21WxR9Qi08029ODWCEbq6fUvSBU=9JBL+A@mail.gmail.com>

Okay, solved it myself (and feeling ever so slightly chuffed about it
all!).  I noticed that one of the comments in

   https://github.com/tenderlove/mechanize/issues/168

from @zzak said:

   "Using master fixed this for me too, I just
forked<https://github.com/zzak/mechanize> and
added a gemspec using
hoe-gemspec<https://github.com/flavorjones/hoe-gemspec> so
that I could use it with bundler in my project."

So I changed the line in my Gemspec from:

    # gem 'mechanize'

to

    gem 'mechanize', :git => 'git://github.com/zzak/mechanize.git'

... and it fixed the problem.  Thank you, Mr. Scott!!  (Now I have to track
down the 'undefined methods `flush` for nil:NilClass problems, but I see
that's already reported as well!)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20120127/b4856f45/attachment.html>

From drbrain at segment7.net  Fri Jan 27 16:53:35 2012
From: drbrain at segment7.net (Eric Hodel)
Date: Fri, 27 Jan 2012 13:53:35 -0800
Subject: [Mechanize-users] dealing with
	Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8
In-Reply-To: <CAGHqdqXLd+KkwSS3y9g0c__1HXdBeqZZp2pN1qtvsazCGcRKzw@mail.gmail.com>
References: <CAGHqdqXLd+KkwSS3y9g0c__1HXdBeqZZp2pN1qtvsazCGcRKzw@mail.gmail.com>
Message-ID: <D169A892-DDB5-423B-A3D9-ECFA388E0478@segment7.net>

On Jan 27, 2012, at 12:41 PM, Robert Poor wrote:

> I'm 99% certain that this is what's being discussed in 
> 
> 	https://github.com/tenderlove/mechanize/issues/168
> 
> But I'm not sure what the remedy is.  My production app is running on Heroku, so I need a fix that I can push there.  Can someone post instructions that this neophyte can follow?  (I gather I'm supposed to fork and add the current master version from github to my config, but I need simple little steps for muddy little feet.) 

There's several forks of mechanize that have the fix, you can use one of them or downgrade to a previous version of mechanize.

>  Thanks...
> 
> I'm running Rails 3.2, Mechanize 2.1.  Mechanize is failing with 
> 
>         Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8

I should have a release out be the end of next week that fixes this bug.