From therapshow at gmail.com  Wed Apr  1 15:47:01 2009
From: therapshow at gmail.com (Frank Gjildberg)
Date: Wed, 1 Apr 2009 21:47:01 +0200
Subject: [Mechanize-users] My first script (please help)
Message-ID: <656c2abe0904011247h61222b1dg4e62edf319fe468b@mail.gmail.com>

I got this smalltime social network here with about 200 users. In the
beginning we didn't have the "make all users friends of the admin"
function that we do now, so I have to manually add these 200 users as
friend of the admin.

My friend coded this thing in PHP and I ain't got much insight into
its underpinnings (but it's a mess), so I figured this'd be a great
way to learn Mechanize / Nokogiri.

Here's what I'm trying to do:

1) Go to http://members.mysite.com
2) Log in (the login box is on the same page)
3) There's a select box labeled "Sort by" -- select "New members"
4) List all the users on that page, each user looking like
http://my.mysite.com/some_user
5) Do the same for http://members.mysite.com/?page=2 and all the way
up to the max (whatever that is)
6) Then, for each user, visit its profile page and click on "Add
friends" (or http://my.mysite.com/<user>/friendship/add_request)
7) The system will respond with a blank page saying "Friend request to
<strong>some_user</strong> is sent"

So far I have:

-

require "rubygems"
require "mechanize"

agent = WWW::Mechanize.new

agent.get("http://members.mysite.com") do |page|

  page.form_with(:name => /login/) do |form|
    form.username = "hahdfgdf"
    form.user_pass = "sdfsdf"
  end.submit

  page = agent.click page.link_with(:href => /my.mysite.com/)

end

-

The above script produces this error:

undefined method `username' for nil:NilClass (NoMethodError), from
(eval):12:in `form_with'

Even though my login textfield's name is "form[username]" and the
password textfield's name is "form[user_pass]". Incase :name =>
/login/ (should correspond to the form's ID "login-form") was nil, I
also tried :action => "http://mysite.com/login", but without success.

What am I doing wrong here?

And is the structure etc. of my script looking set to complete the
rest of the tasks?

Thank you.

Best regards,
Redd Vinylene

-- 
http://www.home.no/reddvinylene

From andy at r210.com  Thu Apr  9 00:07:26 2009
From: andy at r210.com (Andy Leak)
Date: Wed, 8 Apr 2009 21:07:26 -0700
Subject: [Mechanize-users] Ruby Mechanize / Caching ??
In-Reply-To: <4cb868300904081249h356f82eav81641e4381150585@mail.gmail.com>
References: <4cb868300904081249h356f82eav81641e4381150585@mail.gmail.com>
Message-ID: <4cb868300904082107n11b56e74k386ce2d118ec1cb2@mail.gmail.com>

Dear Mechanize Group -

I am looking for a caching solution that works with Mechanize/Ruby.  I tried
using Mechanize with FakeWeb but couldn't get it to work reliably.

Do you know if anyone has written a caching plugin, or has a reliable
caching solution that works with Mechanize/Ruby?

I found this caching plugin for Mechanize / Perl (
http://github.com/joemcmahon/www-mechanize-plugin-cache/tree/master), but
haven't been able to find anything for Mechanize/Ruby.

Thanks, Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20090408/5055c02b/attachment-0001.html>

From andreas.formiconi at gmail.com  Thu Apr  9 05:07:14 2009
From: andreas.formiconi at gmail.com (Andreas Formiconi)
Date: Thu, 9 Apr 2009 11:07:14 +0200
Subject: [Mechanize-users] undefined method 'text' ...
Message-ID: <c0817ecb0904090207i7667f79ctffa24404f6594eff@mail.gmail.com>

Just trying to run the example

*****
  require 'rubygems'
  require 'mechanize'

  a = WWW::Mechanize.new
  a.get('http://rubyforge.org/') do |page|
    # Click the login link
    login_page = a.click(page.links.text(/Log In/))

    # Submit the login form
    my_page = login_page.form_with(:action => '/account/login.php') do |f|
      f.form_loginname  = ARGV[0]
      f.form_pw         = ARGV[1]
    end.click_button

    my_page.links.each do |link|
      text = link.text.strip
      next unless text.length > 0
      puts text
    end
  end
*****

with command

/opt/local/bin/ruby rubyforge.rb

but getting

rubyforge.rb:7: undefined method `text' for #<Array:0x16d2940>
(NoMethodError)
        from
/opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.2/lib/www/mechanize.rb:232:in
`get'
        from rubyforge.rb:5


Installed mechanize with

sudo /opt/local/bin/gem install mechanize

which resulted in

/opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.2/

System is Mac OS X 10.4.11

ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin8]




-- 
Andreas R. Formiconi
**************************
Blog: http://iamarf.wordpress.com/
Wiki: http://infomedfi.pbwiki.com/
Blog Clown in corsia: http://m-illumino-d-immenso.blogspot.com/
Distribuzione dispense MedWiki: http://www.medwiki.it/
Studente del corso "Connectivism and Connective Knowledge Online Course"
http://ltc.umanitoba.ca/wiki/Connectivism
**************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20090409/6d1fe794/attachment.html>

From michael.c.libby at gmail.com  Thu Apr  9 08:32:58 2009
From: michael.c.libby at gmail.com (Michael Libby)
Date: Thu, 9 Apr 2009 07:32:58 -0500
Subject: [Mechanize-users] undefined method 'text' ...
In-Reply-To: <c0817ecb0904090207i7667f79ctffa24404f6594eff@mail.gmail.com>
References: <c0817ecb0904090207i7667f79ctffa24404f6594eff@mail.gmail.com>
Message-ID: <db8524fb0904090532p59436f94x4d93cacc32863430@mail.gmail.com>

On Thu, Apr 9, 2009 at 4:07 AM, Andreas Formiconi <
andreas.formiconi at gmail.com> wrote:

> Just trying to run the example
>
> *****
>   require 'rubygems'
>   require 'mechanize'
>
>   a = WWW::Mechanize.new
>   a.get('http://rubyforge.org/') do |page|
>     # Click the login link
>     login_page = a.click(page.links.text(/Log In/))


 [snip]


> rubyforge.rb:7: undefined method `text' for #<Array:0x16d2940>
> (NoMethodError)
>         from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.2/lib/www/mechanize.rb:232:in
> `get'
>         from rubyforge.rb:5
>

The error message tells you line 7, which is login_page = ...

A page.links returns an Array, which -- like the error messages says-- does
not have a #text method.

You're going to have to choose one link from the Array somehow and pass that
to a.click().

You probably want something like:

  login_page = a.click(page.links.select{|x| x.text =~ /Log In/}.first)

Personally I would not chain all that stuff together without making sure my
#select and #first return a non-empty, non-null value. But to each his own.

-Michael

-- 
Michael C. Libby
www.mikelibby.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mechanize-users/attachments/20090409/3aef4dc5/attachment.html>

From m.zecko at gmail.com  Sat Apr 11 06:49:17 2009
From: m.zecko at gmail.com (m.zeckinger)
Date: Sat, 11 Apr 2009 12:49:17 +0200
Subject: [Mechanize-users] questions about frames and scraping issues
Message-ID: <8be6656c0904110349r74865676w43e672146796560c@mail.gmail.com>

hi list,
i am completely amazed by www::mechanize, although i have some
questions regarding frames:

1) how can i adress a certain frame directly? even if i know the name
of the frame, i could just adress it via e.g. page.frames[1] -- is
there a way to get it via name?

2) how can i fill out forms in a frame? actually, how do i access any
frame-data?
i tried puts page.frames[1].forms.inspect
but actually this seems not to work out...

is there any more in-depth information about www::mechanize and forms,
or can anybody here help out?

thanks!
mz

From mat.schaffer at gmail.com  Sun Apr 12 11:26:20 2009
From: mat.schaffer at gmail.com (Mat Schaffer)
Date: Sun, 12 Apr 2009 11:26:20 -0400
Subject: [Mechanize-users] filter links based on class
In-Reply-To: <e48fbdf70904111654w55bbc7baufdcde949022e526e@mail.gmail.com>
References: <e48fbdf70904111654w55bbc7baufdcde949022e526e@mail.gmail.com>
Message-ID: <36307976-3C2B-4360-B0E9-1A1F8BAC048F@gmail.com>

I'd probably just locate it with CSS searching. Pretty sure  
page.search('a.foo') still does that, though I'm not as sure since the  
move from hpricot to nokogiri.
-Mat

On Apr 11, 2009, at 7:54 PM, Chris Blumentritt wrote:

> I am trying to filter a link that has an attribute class="foo" in it  
> is there something like:
>
> page.links_with(:class => 'foo') available?
>
> I see there is something for :href and :text   
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users


From mat.schaffer at gmail.com  Sun Apr 12 11:35:31 2009
From: mat.schaffer at gmail.com (Mat Schaffer)
Date: Sun, 12 Apr 2009 11:35:31 -0400
Subject: [Mechanize-users] My first script (please help)
In-Reply-To: <656c2abe0904011247h61222b1dg4e62edf319fe468b@mail.gmail.com>
References: <656c2abe0904011247h61222b1dg4e62edf319fe468b@mail.gmail.com>
Message-ID: <A44467B0-B91C-43BA-A0F6-6E71B4952176@gmail.com>

I'm sure you could do this, but I'm not familiar with the block syntax  
you have here.  This seems to work for me:

agent = WWW::Mechanize.new
form = agent.get('http://google.com').form(:name => /\w+/)
form.q = 'test'
page = agent.submit(form)
puts page.links.length

Also I'm sure redefining page in the block syntax like that could lead  
to confusion. It looks like you're trying to grab the links from the  
original form page, not the resultant page.

-Mat



On Apr 1, 2009, at 3:47 PM, Frank Gjildberg wrote:

> I got this smalltime social network here with about 200 users. In the
> beginning we didn't have the "make all users friends of the admin"
> function that we do now, so I have to manually add these 200 users as
> friend of the admin.
>
> My friend coded this thing in PHP and I ain't got much insight into
> its underpinnings (but it's a mess), so I figured this'd be a great
> way to learn Mechanize / Nokogiri.
>
> Here's what I'm trying to do:
>
> 1) Go to http://members.mysite.com
> 2) Log in (the login box is on the same page)
> 3) There's a select box labeled "Sort by" -- select "New members"
> 4) List all the users on that page, each user looking like
> http://my.mysite.com/some_user
> 5) Do the same for http://members.mysite.com/?page=2 and all the way
> up to the max (whatever that is)
> 6) Then, for each user, visit its profile page and click on "Add
> friends" (or http://my.mysite.com/<user>/friendship/add_request)
> 7) The system will respond with a blank page saying "Friend request to
> <strong>some_user</strong> is sent"
>
> So far I have:
>
> -
>
> require "rubygems"
> require "mechanize"
>
> agent = WWW::Mechanize.new
>
> agent.get("http://members.mysite.com") do |page|
>
>  page.form_with(:name => /login/) do |form|
>    form.username = "hahdfgdf"
>    form.user_pass = "sdfsdf"
>  end.submit
>
>  page = agent.click page.link_with(:href => /my.mysite.com/)
>
> end
>
> -
>
> The above script produces this error:
>
> undefined method `username' for nil:NilClass (NoMethodError), from
> (eval):12:in `form_with'
>
> Even though my login textfield's name is "form[username]" and the
> password textfield's name is "form[user_pass]". Incase :name =>
> /login/ (should correspond to the form's ID "login-form") was nil, I
> also tried :action => "http://mysite.com/login", but without success.
>
> What am I doing wrong here?
>
> And is the structure etc. of my script looking set to complete the
> rest of the tasks?
>
> Thank you.
>
> Best regards,
> Redd Vinylene
>
> -- 
> http://www.home.no/reddvinylene
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users


From wojciech at piekutowski.net  Sat Apr 18 09:55:25 2009
From: wojciech at piekutowski.net (Wojciech Piekutowski)
Date: Sat, 18 Apr 2009 15:55:25 +0200
Subject: [Mechanize-users] Running test suite
Message-ID: <a9ab03a50904180655p7752b69fv5f436643fa7134b7@mail.gmail.com>

Hello,

I have problems with running the test suite. `ruby test` on master
branch gives me '283 tests, 1362 assertions, 4 failures, 49 errors'.
Is it just because master is a development branch? Do you try to have
tests aligned with the code?

I have also 1 failure in REL_0.9.2:

 1) Failure:
test_set_encoding(TestPage) [./test/test_page.rb:29]:
<"UTF-8"> expected but was
<nil>.

274 tests, 1449 assertions, 1 failures, 0 errors

I'm using nokogiri 1.2.3 and libxml2 2.6.32 (dfsg-4ubuntu1.1). Is
mechanize designed for other version of these libs?

Greetings,
Wojciech

From aaron.patterson at gmail.com  Sat Apr 18 10:49:04 2009
From: aaron.patterson at gmail.com (Aaron Patterson)
Date: Sat, 18 Apr 2009 07:49:04 -0700
Subject: [Mechanize-users] Running test suite
In-Reply-To: <a9ab03a50904180655p7752b69fv5f436643fa7134b7@mail.gmail.com>
References: <a9ab03a50904180655p7752b69fv5f436643fa7134b7@mail.gmail.com>
Message-ID: <6959e1680904180749r5f2a5741xd8a4b3029d769e4c@mail.gmail.com>

On Sat, Apr 18, 2009 at 6:55 AM, Wojciech Piekutowski
<wojciech at piekutowski.net> wrote:
> Hello,
>
> I have problems with running the test suite. `ruby test` on master
> branch gives me '283 tests, 1362 assertions, 4 failures, 49 errors'.
> Is it just because master is a development branch? Do you try to have
> tests aligned with the code?
>
> I have also 1 failure in REL_0.9.2:
>
> ?1) Failure:
> test_set_encoding(TestPage) [./test/test_page.rb:29]:
> <"UTF-8"> expected but was
> <nil>.
>
> 274 tests, 1449 assertions, 1 failures, 0 errors
>
> I'm using nokogiri 1.2.3 and libxml2 2.6.32 (dfsg-4ubuntu1.1). Is
> mechanize designed for other version of these libs?

libxml2 version 2.6.32 has problems when it comes to encoding.  If you
upgrade libxml2 to the latest (2.7.3) you should be able to run the
tests.

-- 
Aaron Patterson
http://tenderlovemaking.com/

From wojciech at piekutowski.net  Sat Apr 18 15:17:39 2009
From: wojciech at piekutowski.net (Wojciech Piekutowski)
Date: Sat, 18 Apr 2009 21:17:39 +0200
Subject: [Mechanize-users] Running test suite
In-Reply-To: <6959e1680904180749r5f2a5741xd8a4b3029d769e4c@mail.gmail.com>
References: <a9ab03a50904180655p7752b69fv5f436643fa7134b7@mail.gmail.com> 
	<6959e1680904180749r5f2a5741xd8a4b3029d769e4c@mail.gmail.com>
Message-ID: <a9ab03a50904181217s54db63c0y79eeb07948ce74f8@mail.gmail.com>

On Sat, Apr 18, 2009 at 16:49, Aaron Patterson
<aaron.patterson at gmail.com> wrote:
> On Sat, Apr 18, 2009 at 6:55 AM, Wojciech Piekutowski
> <wojciech at piekutowski.net> wrote:
>> Hello,
>>
>> I have problems with running the test suite. `ruby test` on master
>> branch gives me '283 tests, 1362 assertions, 4 failures, 49 errors'.
>> Is it just because master is a development branch? Do you try to have
>> tests aligned with the code?
>>
>> I have also 1 failure in REL_0.9.2:
>>
>> ?1) Failure:
>> test_set_encoding(TestPage) [./test/test_page.rb:29]:
>> <"UTF-8"> expected but was
>> <nil>.
>>
>> 274 tests, 1449 assertions, 1 failures, 0 errors
>>
>> I'm using nokogiri 1.2.3 and libxml2 2.6.32 (dfsg-4ubuntu1.1). Is
>> mechanize designed for other version of these libs?
>
> libxml2 version 2.6.32 has problems when it comes to encoding. ?If you
> upgrade libxml2 to the latest (2.7.3) you should be able to run the
> tests.

Thanks, now tests work fine.

I'd be a good idea to add this libxml2 version requirement (or
warning) to Mechanize or Nokogiri or maybe both. RMagick does a
similar thing with the imagemagick lib - gem simply won't compile
without proper version of the lib.

Greetings,
Wojciech

From wojciech at piekutowski.net  Sat Apr 18 15:42:28 2009
From: wojciech at piekutowski.net (Wojciech Piekutowski)
Date: Sat, 18 Apr 2009 21:42:28 +0200
Subject: [Mechanize-users] [PATCH] When user provides an encoding then use
	it, otherwise 	autodetect it
In-Reply-To: <a9ab03a50904181235w1469252exdb7d9154aa35f32c@mail.gmail.com>
References: <a9ab03a50904181235w1469252exdb7d9154aa35f32c@mail.gmail.com>
Message-ID: <a9ab03a50904181242n7e5a245ai5b1c0508ee38907f@mail.gmail.com>

Hi,

I've created a small patch to mechanize.

The problem:
The user should be able to set his own page encoding, for example when
<meta /> encoding information can is invalid or auto detection
routines fail to guess correctly.

Actual results:
Mechanize will use the encoding detected by it (or Nokogiri/libxml2)
always when there's a <meta /> with encoding information included.
User can only set his encoding when there's no <meta />.

Expected results:
Mechanize should allow the user to always force his own encoding.

Commit:
http://github.com/wpiekutowski/mechanize/commit/2af5e6b2c520cca96cd5fe18df284f57d09c3cef

Greetings,
Wojciech

From aaron.patterson at gmail.com  Sat Apr 18 16:25:30 2009
From: aaron.patterson at gmail.com (Aaron Patterson)
Date: Sat, 18 Apr 2009 13:25:30 -0700
Subject: [Mechanize-users] [PATCH] When user provides an encoding then
	use it, otherwise autodetect it
In-Reply-To: <a9ab03a50904181242n7e5a245ai5b1c0508ee38907f@mail.gmail.com>
References: <a9ab03a50904181235w1469252exdb7d9154aa35f32c@mail.gmail.com>
	<a9ab03a50904181242n7e5a245ai5b1c0508ee38907f@mail.gmail.com>
Message-ID: <6959e1680904181325i64070aa7vc14031d594003478@mail.gmail.com>

On Sat, Apr 18, 2009 at 12:42 PM, Wojciech Piekutowski
<wojciech at piekutowski.net> wrote:
> Hi,
>
> I've created a small patch to mechanize.
>
> The problem:
> The user should be able to set his own page encoding, for example when
> <meta /> encoding information can is invalid or auto detection
> routines fail to guess correctly.
>
> Actual results:
> Mechanize will use the encoding detected by it (or Nokogiri/libxml2)
> always when there's a <meta /> with encoding information included.
> User can only set his encoding when there's no <meta />.
>
> Expected results:
> Mechanize should allow the user to always force his own encoding.
>
> Commit:
> http://github.com/wpiekutowski/mechanize/commit/2af5e6b2c520cca96cd5fe18df284f57d09c3cef

Looks good.  I've added you to the project, so you get to merge.  :-)

-- 
Aaron Patterson
http://tenderlovemaking.com/