ok

tag. This might be libxml2 failing to parse the character set. Aaron, you know encoding better than I do. Any ideas? -mike On Tue, Jan 13, 2009 at 3:35 AM, Zhenjian YU wrote: > Hi, > > I use nokogiri to scrape some book items ('.//div[@class="book_list"]') > from a Chinese page http://read.dangdang.com/list_4 . > It seems that nokogiri doesn't handle this page very well. nokogiri only > got 16 book_list items, with the last item book_1226, which is wrong. > While hpricot can produce the correct 20 book_list items. > > Can anyone figure out what the problem is? > > Best Regards, > Castor > > -- > http://www.yobo.com > http://www.8sheng.com > > _______________________________________________ > Nokogiri-talk mailing list > Nokogiri-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/nokogiri-talk > > -- mike dalessio mike at csa.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenjian at gmail.com Tue Jan 13 09:20:38 2009 From: zhenjian at gmail.com (Zhenjian YU) Date: Tue, 13 Jan 2009 22:20:38 +0800 Subject: [Nokogiri-talk] Nokogiri doesn't produce the correct list as hpricot In-Reply-To: <618c07250901130520r61ff0818k170acfc25ee1ae08@mail.gmail.com> References: <4ba70b40901130035j2663e6b0tffe3e243b08e6cac@mail.gmail.com> <618c07250901130520r61ff0818k170acfc25ee1ae08@mail.gmail.com> Message-ID: <4ba70b40901130620j160f8856j789115afc13838d0@mail.gmail.com> The encoding of the web page is GB2312. Is there some encoding problem? Best Regards, Castor On Tue, Jan 13, 2009 at 9:20 PM, Mike Dalessio wrote: > Hi, > > I noticed that, if you parse that page with Nokogiri and then output the > parsed document, it is truncated halfway through book_1226's

class='autohr'> tag. This might be libxml2 failing to parse the character > set. > > Aaron, you know encoding better than I do. Any ideas? > > -mike > > On Tue, Jan 13, 2009 at 3:35 AM, Zhenjian YU wrote: > >> Hi, >> >> I use nokogiri to scrape some book items ('.//div[@class="book_list"]') >> from a Chinese page http://read.dangdang.com/list_4 . >> It seems that nokogiri doesn't handle this page very well. nokogiri only >> got 16 book_list items, with the last item book_1226, which is wrong. >> While hpricot can produce the correct 20 book_list items. >> >> Can anyone figure out what the problem is? >> >> Best Regards, >> Castor >> >> -- >> http://www.yobo.com >> http://www.8sheng.com >> >> _______________________________________________ >> Nokogiri-talk mailing list >> Nokogiri-talk at rubyforge.org >> http://rubyforge.org/mailman/listinfo/nokogiri-talk >> >> > > > -- > mike dalessio > mike at csa.net > > _______________________________________________ > Nokogiri-talk mailing list > Nokogiri-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/nokogiri-talk > > -- http://www.yobo.com http://www.8sheng.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenjian at gmail.com Tue Jan 13 09:58:17 2009 From: zhenjian at gmail.com (Zhenjian YU) Date: Tue, 13 Jan 2009 22:58:17 +0800 Subject: [Nokogiri-talk] Nokogiri doesn't produce the correct list as hpricot In-Reply-To: <4ba70b40901130620j160f8856j789115afc13838d0@mail.gmail.com> References: <4ba70b40901130035j2663e6b0tffe3e243b08e6cac@mail.gmail.com> <618c07250901130520r61ff0818k170acfc25ee1ae08@mail.gmail.com> <4ba70b40901130620j160f8856j789115afc13838d0@mail.gmail.com> Message-ID: <4ba70b40901130658l7a0052e1hd0ea94ab8f4daedd@mail.gmail.com> I tried to parse another page http://read.dangdang.com/content_571897 . The parsed document was truncated too. Best Regards, Castor On Tue, Jan 13, 2009 at 10:20 PM, Zhenjian YU wrote: > The encoding of the web page is GB2312. Is there some encoding problem? > > Best Regards, > Castor > > > On Tue, Jan 13, 2009 at 9:20 PM, Mike Dalessio wrote: > >> Hi, >> >> I noticed that, if you parse that page with Nokogiri and then output the >> parsed document, it is truncated halfway through book_1226's

> class='autohr'> tag. This might be libxml2 failing to parse the character >> set. >> >> Aaron, you know encoding better than I do. Any ideas? >> >> -mike >> >> On Tue, Jan 13, 2009 at 3:35 AM, Zhenjian YU wrote: >> >>> Hi, >>> >>> I use nokogiri to scrape some book items ('.//div[@class="book_list"]') >>> from a Chinese page http://read.dangdang.com/list_4 . >>> It seems that nokogiri doesn't handle this page very well. nokogiri only >>> got 16 book_list items, with the last item book_1226, which is wrong. >>> While hpricot can produce the correct 20 book_list items. >>> >>> Can anyone figure out what the problem is? >>> >>> Best Regards, >>> Castor >>> >>> -- >>> http://www.yobo.com >>> http://www.8sheng.com >>> >>> _______________________________________________ >>> Nokogiri-talk mailing list >>> Nokogiri-talk at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/nokogiri-talk >>> >>> >> >> >> -- >> mike dalessio >> mike at csa.net >> >> _______________________________________________ >> Nokogiri-talk mailing list >> Nokogiri-talk at rubyforge.org >> http://rubyforge.org/mailman/listinfo/nokogiri-talk >> >> > > > -- > http://www.yobo.com > http://www.8sheng.com > -- http://www.yobo.com http://www.8sheng.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.patterson at gmail.com Tue Jan 13 11:45:22 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Tue, 13 Jan 2009 08:45:22 -0800 Subject: [Nokogiri-talk] Nokogiri doesn't produce the correct list as hpricot In-Reply-To: <4ba70b40901130035j2663e6b0tffe3e243b08e6cac@mail.gmail.com> References: <4ba70b40901130035j2663e6b0tffe3e243b08e6cac@mail.gmail.com> Message-ID: <6959e1680901130845p7e71475k5ea50d345544f875@mail.gmail.com> On Tue, Jan 13, 2009 at 12:35 AM, Zhenjian YU wrote: > Hi, > > I use nokogiri to scrape some book items ('.//div[@class="book_list"]') from > a Chinese page http://read.dangdang.com/list_4 . > It seems that nokogiri doesn't handle this page very well. nokogiri only got > 16 book_list items, with the last item book_1226, which is wrong. > While hpricot can produce the correct 20 book_list items. > > Can anyone figure out what the problem is? Specifying an encoding works: require 'rubygems' require 'nokogiri' require 'open-uri' doc = Nokogiri::HTML(open('http://read.dangdang.com/list_4'), nil, 'CP936') puts doc.xpath('//div[@class="book_list"]').length # => 20 Is there a good way to detect encoding in ruby? I'd like to make nokogiri just deal with this stuff automatically. -- Aaron Patterson http://tenderlovemaking.com/ From rubikitch at ruby-lang.org Tue Jan 13 12:28:05 2009 From: rubikitch at ruby-lang.org (rubikitch at ruby-lang.org) Date: Wed, 14 Jan 2009 02:28:05 +0900 (JST) Subject: [Nokogiri-talk] Nokogiri doesn't produce the correct list as hpricot In-Reply-To: <6959e1680901130845p7e71475k5ea50d345544f875@mail.gmail.com> References: <4ba70b40901130035j2663e6b0tffe3e243b08e6cac@mail.gmail.com> <6959e1680901130845p7e71475k5ea50d345544f875@mail.gmail.com> Message-ID: <20090114.022805.68093029.rubikitch@ruby-lang.org> From: "Aaron Patterson" Subject: Re: [Nokogiri-talk] Nokogiri doesn't produce the correct list as hpricot Date: Tue, 13 Jan 2009 08:45:22 -0800 > Is there a good way to detect encoding in ruby? I'd like to make > nokogiri just deal with this stuff automatically. charset header? require 'rubygems' require 'nokogiri' require 'open-uri' open('http://read.dangdang.com/list_4') do |f| html = f.read f.charset # => "gb2312" doc = Nokogiri::HTML(html, nil, f.charset) doc.xpath('//div[@class="book_list"]').length # => 16 end But this result is incorrect. Bug? -- rubikitch Blog: http://d.hatena.ne.jp/rubikitch/ Site: http://www.rubyist.net/~rubikitch/ From aaron.patterson at gmail.com Tue Jan 13 13:31:54 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Tue, 13 Jan 2009 10:31:54 -0800 Subject: [Nokogiri-talk] Nokogiri doesn't produce the correct list as hpricot In-Reply-To: <20090114.022805.68093029.rubikitch@ruby-lang.org> References: <4ba70b40901130035j2663e6b0tffe3e243b08e6cac@mail.gmail.com> <6959e1680901130845p7e71475k5ea50d345544f875@mail.gmail.com> <20090114.022805.68093029.rubikitch@ruby-lang.org> Message-ID: <6959e1680901131031i344206a8yc5583496c367f5ed@mail.gmail.com> On Tue, Jan 13, 2009 at 9:28 AM, wrote: > From: "Aaron Patterson" > Subject: Re: [Nokogiri-talk] Nokogiri doesn't produce the correct list as hpricot > Date: Tue, 13 Jan 2009 08:45:22 -0800 > >> Is there a good way to detect encoding in ruby? I'd like to make >> nokogiri just deal with this stuff automatically. > > charset header? > > require 'rubygems' > require 'nokogiri' > require 'open-uri' > > open('http://read.dangdang.com/list_4') do |f| > html = f.read > f.charset # => "gb2312" > doc = Nokogiri::HTML(html, nil, f.charset) > doc.xpath('//div[@class="book_list"]').length # => 16 > end > > But this result is incorrect. Bug? I'm not sure. It works with 'CP936' as the encoding. Perhaps they are synonymous? -- Aaron Patterson http://tenderlovemaking.com/ From andywatts at angrylapdog.com Tue Jan 13 14:45:38 2009 From: andywatts at angrylapdog.com (Andrew Watts-Curnow) Date: Tue, 13 Jan 2009 11:45:38 -0800 Subject: [Nokogiri-talk] XPath expressions other than location paths Message-ID: <6cf2f09c0901131145w3f618552s10a0d520c23d2632@mail.gmail.com> Hi, I'm curious about support for XPath expressions that return booleans, string or numbers. Here's a couple of quotes from the specs. > The primary syntactic construct in XPath is the expression. An expression > matches the production Expr . An > expression is evaluated to yield an object, which has one of the following > four basic types: > > - node-set (an unordered collection of nodes without duplicates) > - boolean (true or false) > - number (a floating-point number) > - string (a sequence of UCS characters) > > One important kind of expression is a location path. .... The result of > evaluating an expression that is a location path is the node-set containing > the nodes selected by the location path. Ultimately, I'm hoping to use XPaths like the following. count(//book) => 2 contains(//gems, 'nokogiri') => True substring-after(//h1, 'Nokogiri is') => 'subarashi' Firefox's document.evaluate handles these, but maybe that's the DOM XPath spec. Does libxml support expressions other than location paths? Would this make sense as an enhancement to nokogiri? Thanks Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.patterson at gmail.com Tue Jan 13 14:53:56 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Tue, 13 Jan 2009 11:53:56 -0800 Subject: [Nokogiri-talk] XPath expressions other than location paths In-Reply-To: <6cf2f09c0901131145w3f618552s10a0d520c23d2632@mail.gmail.com> References: <6cf2f09c0901131145w3f618552s10a0d520c23d2632@mail.gmail.com> Message-ID: <6959e1680901131153x230f701lca213fec698321a9@mail.gmail.com> On Tue, Jan 13, 2009 at 11:45 AM, Andrew Watts-Curnow wrote: > Hi, > > I'm curious about support for XPath expressions that return booleans, string > or numbers. > > Here's a couple of quotes from the specs. >> >> The primary syntactic construct in XPath is the expression. An expression >> matches the production Expr. An expression is evaluated to yield an object, >> which has one of the following four basic types: >> >> node-set (an unordered collection of nodes without duplicates) >> boolean (true or false) >> number (a floating-point number) >> string (a sequence of UCS characters) >> >> One important kind of expression is a location path. .... The result of >> evaluating an expression that is a location path is the node-set containing >> the nodes selected by the location path. > > Ultimately, I'm hoping to use XPaths like the following. > count(//book) => 2 > contains(//gems, 'nokogiri') => True > substring-after(//h1, 'Nokogiri is') => 'subarashi' > Firefox's document.evaluate handles these, but maybe that's the DOM XPath > spec. > > Does libxml support expressions other than location paths? > Would this make sense as an enhancement to nokogiri? I think it is possible, but I have a hard time justifying to myself why you would need it. For example: count(//book) could be doc.xpath('//book').length contains(//gems, 'nokogiri') could be doc.xpath('//gems').any? { |x| x.to_s =~ /nokogiri/ } substring-after(//h1, 'Nokogiri is') could be doc.xpath('//h1').to_s.gsub(/^Nokogiri is/, '') I'm willing to look in to returning something other than a node set more seriously if I could get a compelling example. -- Aaron Patterson http://tenderlovemaking.com/ From andywatts at angrylapdog.com Tue Jan 13 16:41:30 2009 From: andywatts at angrylapdog.com (Andrew Watts-Curnow) Date: Tue, 13 Jan 2009 13:41:30 -0800 Subject: [Nokogiri-talk] XPath expressions other than location paths In-Reply-To: <6959e1680901131153x230f701lca213fec698321a9@mail.gmail.com> References: <6cf2f09c0901131145w3f618552s10a0d520c23d2632@mail.gmail.com> <6959e1680901131153x230f701lca213fec698321a9@mail.gmail.com> Message-ID: <6cf2f09c0901131341n2c4d170ercb2049b49a5ecf9c@mail.gmail.com> Aaron, Thanks for the reply. I use my XPath expressions in different environments and therefore trying to keep them as portable as possible. For example, some of my XPaths will also be used by firefox's document.evaluate command. Supporting more of the spec and thereby XPath portability seems a pretty compelling.....but then I'm the guy that needs it. :) - Andy On Tue, Jan 13, 2009 at 11:53 AM, Aaron Patterson wrote: > On Tue, Jan 13, 2009 at 11:45 AM, Andrew Watts-Curnow > wrote: > > Hi, > > > > I'm curious about support for XPath expressions that return booleans, > string > > or numbers. > > > > Here's a couple of quotes from the specs. > >> > >> The primary syntactic construct in XPath is the expression. An > expression > >> matches the production Expr. An expression is evaluated to yield an > object, > >> which has one of the following four basic types: > >> > >> node-set (an unordered collection of nodes without duplicates) > >> boolean (true or false) > >> number (a floating-point number) > >> string (a sequence of UCS characters) > >> > >> One important kind of expression is a location path. .... The result of > >> evaluating an expression that is a location path is the node-set > containing > >> the nodes selected by the location path. > > > > Ultimately, I'm hoping to use XPaths like the following. > > count(//book) => 2 > > contains(//gems, 'nokogiri') => True > > substring-after(//h1, 'Nokogiri is') => 'subarashi' > > Firefox's document.evaluate handles these, but maybe that's the DOM XPath > > spec. > > > > Does libxml support expressions other than location paths? > > Would this make sense as an enhancement to nokogiri? > > I think it is possible, but I have a hard time justifying to myself > why you would need it. > > For example: > > count(//book) could be doc.xpath('//book').length > contains(//gems, 'nokogiri') could be doc.xpath('//gems').any? { |x| > x.to_s =~ /nokogiri/ } > substring-after(//h1, 'Nokogiri is') could be > doc.xpath('//h1').to_s.gsub(/^Nokogiri is/, '') > > I'm willing to look in to returning something other than a node set > more seriously if I could get a compelling example. > > -- > Aaron Patterson > http://tenderlovemaking.com/ > _______________________________________________ > Nokogiri-talk mailing list > Nokogiri-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/nokogiri-talk > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holtonma at gmail.com Tue Jan 13 18:27:19 2009 From: holtonma at gmail.com (Mark Holton) Date: Tue, 13 Jan 2009 15:27:19 -0800 Subject: [Nokogiri-talk] XPath expressions other than location paths In-Reply-To: <6959e1680901131153x230f701lca213fec698321a9@mail.gmail.com> References: <6cf2f09c0901131145w3f618552s10a0d520c23d2632@mail.gmail.com> <6959e1680901131153x230f701lca213fec698321a9@mail.gmail.com> Message-ID: <8bd9e8730901131527t301e2dd9tb48d2784acbce009@mail.gmail.com> I like 'the saw' simple and effective... not bloated or like a Swiss Army Knife. My two cents. :Mark On Tue, Jan 13, 2009 at 11:53 AM, Aaron Patterson wrote: > On Tue, Jan 13, 2009 at 11:45 AM, Andrew Watts-Curnow > wrote: > > Hi, > > > > I'm curious about support for XPath expressions that return booleans, > string > > or numbers. > > > > Here's a couple of quotes from the specs. > >> > >> The primary syntactic construct in XPath is the expression. An > expression > >> matches the production Expr. An expression is evaluated to yield an > object, > >> which has one of the following four basic types: > >> > >> node-set (an unordered collection of nodes without duplicates) > >> boolean (true or false) > >> number (a floating-point number) > >> string (a sequence of UCS characters) > >> > >> One important kind of expression is a location path. .... The result of > >> evaluating an expression that is a location path is the node-set > containing > >> the nodes selected by the location path. > > > > Ultimately, I'm hoping to use XPaths like the following. > > count(//book) => 2 > > contains(//gems, 'nokogiri') => True > > substring-after(//h1, 'Nokogiri is') => 'subarashi' > > Firefox's document.evaluate handles these, but maybe that's the DOM XPath > > spec. > > > > Does libxml support expressions other than location paths? > > Would this make sense as an enhancement to nokogiri? > > I think it is possible, but I have a hard time justifying to myself > why you would need it. > > For example: > > count(//book) could be doc.xpath('//book').length > contains(//gems, 'nokogiri') could be doc.xpath('//gems').any? { |x| > x.to_s =~ /nokogiri/ } > substring-after(//h1, 'Nokogiri is') could be > doc.xpath('//h1').to_s.gsub(/^Nokogiri is/, '') > > I'm willing to look in to returning something other than a node set > more seriously if I could get a compelling example. > > -- > Aaron Patterson > http://tenderlovemaking.com/ > _______________________________________________ > Nokogiri-talk mailing list > Nokogiri-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/nokogiri-talk > -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodieboy at gmail.com Tue Jan 13 21:06:19 2009 From: goodieboy at gmail.com (Matt Mitchell) Date: Tue, 13 Jan 2009 21:06:19 -0500 Subject: [Nokogiri-talk] beginner questions Message-ID: Hi, I just started using NG (after experiencing some Hpricot pain) and am very happy with it so far. I'm using it to scrape large TEI files to index into Solr, and also chunking out bits of xml fragments to disk. I've got a few questions and well, I'll just cut to it :) is there a nice way to get the next/previous sibling element, not text? how can i get the absolute position of a node? (to the root) how can i get the relative position of a node? (to it's parent) how can i get the depth of a node? is there a way to search using negation? - Hpricot uses a "filter" like //:not(blah) - but does not seem to work correctly! does it support unicode/utf-8? Thanks in advance! And thanks for the awesome library. Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenjian at gmail.com Wed Jan 14 00:54:49 2009 From: zhenjian at gmail.com (Zhenjian YU) Date: Wed, 14 Jan 2009 13:54:49 +0800 Subject: [Nokogiri-talk] Nokogiri doesn't produce the correct list as hpricot In-Reply-To: <6959e1680901131031i344206a8yc5583496c367f5ed@mail.gmail.com> References: <4ba70b40901130035j2663e6b0tffe3e243b08e6cac@mail.gmail.com> <6959e1680901130845p7e71475k5ea50d345544f875@mail.gmail.com> <20090114.022805.68093029.rubikitch@ruby-lang.org> <6959e1680901131031i344206a8yc5583496c367f5ed@mail.gmail.com> Message-ID: <4ba70b40901132154n36014e7dx8eade0613df3fd08@mail.gmail.com> I found the problem. libxml2 actually uses header information to determine the encoding, gb2312. The problem is that in the web page, there are some chars that don't belong to gb2312. Model browsers are robust enough to correctly handle them. But libxml2 uses iconv to do encoding conversion, which breaks when encounters these chars. This leads to truncated document in nokogiri. Explicitly specify the charset as GB18030 will solve this problem. GB18030 is big enough to contain all possible Chinese chars. On Wed, Jan 14, 2009 at 2:31 AM, Aaron Patterson wrote: > On Tue, Jan 13, 2009 at 9:28 AM, wrote: > > From: "Aaron Patterson" > > Subject: Re: [Nokogiri-talk] Nokogiri doesn't produce the correct list as > hpricot > > Date: Tue, 13 Jan 2009 08:45:22 -0800 > > > >> Is there a good way to detect encoding in ruby? I'd like to make > >> nokogiri just deal with this stuff automatically. > > > > charset header? > > > > require 'rubygems' > > require 'nokogiri' > > require 'open-uri' > > > > open('http://read.dangdang.com/list_4') do |f| > > html = f.read > > f.charset # => "gb2312" > > doc = Nokogiri::HTML(html, nil, f.charset) > > doc.xpath('//div[@class="book_list"]').length # => 16 > > end > > > > But this result is incorrect. Bug? > > I'm not sure. It works with 'CP936' as the encoding. Perhaps they > are synonymous? > > -- > Aaron Patterson > http://tenderlovemaking.com/ > _______________________________________________ > Nokogiri-talk mailing list > Nokogiri-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/nokogiri-talk > -- http://www.yobo.com http://www.8sheng.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rainerthiel at zoho.com Sat Jan 17 10:55:23 2009 From: rainerthiel at zoho.com (rainerthiel) Date: Sat, 17 Jan 2009 17:55:23 +0200 Subject: [Nokogiri-talk] XSLT v2.0 support Message-ID: <11ee54dbab1.5813744184432384679.-270860799764988955@zoho.com> Greetings, Warnings ;) 1. Newbie question 2. Windows user i have installed the gem (nokogiri-1.0.5-x86-mswin32-60) and run the tests. All ok. Now i am trying to transform an xml using a stylesheet that outputs to multiple result documents. The code snippet: style = Nokogiri::XSLT.parse(File.read(xslt_file)) fails as follows: compilation error: element result-document xsltStylePreCompute: unknown xsl:result-document Does Nokogiri not support XSLT 2.0? Regards Rainer Thiel -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.patterson at gmail.com Sat Jan 17 16:34:58 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Sat, 17 Jan 2009 13:34:58 -0800 Subject: [Nokogiri-talk] XSLT v2.0 support In-Reply-To: <11ee54dbab1.5813744184432384679.-270860799764988955@zoho.com> References: <11ee54dbab1.5813744184432384679.-270860799764988955@zoho.com> Message-ID: <6959e1680901171334j1b9330b7vfc154e928d2b0b12@mail.gmail.com> On Sat, Jan 17, 2009 at 7:55 AM, rainerthiel wrote: > Greetings, > > Warnings ;) > 1. Newbie question > 2. Windows user > > i have installed the gem (nokogiri-1.0.5-x86-mswin32-60) and run the tests. > All ok. > Now i am trying to transform an xml using a stylesheet that outputs to > multiple result documents. > > The code snippet: > > style = Nokogiri::XSLT.parse(File.read(xslt_file)) > > fails as follows: > > compilation error: element result-document > xsltStylePreCompute: unknown xsl:result-document > > Does Nokogiri not support XSLT 2.0? I'm not sure..... It supports whatever libxslt, and libexslt support. We added support for libexslt pretty recently, so you should upgrade nokogiri and try again. -- Aaron Patterson http://tenderlovemaking.com/ From mike.tracy at gmail.com Wed Jan 21 10:33:01 2009 From: mike.tracy at gmail.com (Michael Tracy) Date: Wed, 21 Jan 2009 09:33:01 -0600 Subject: [Nokogiri-talk] unexpected behavior with xpath search Message-ID: This is a pattern I use alot when testing webapps and Hpricot behaves as I would expect, Nokogiri does something different. The scenario: 1.Simple page with 3 tables and varying numbers of tr/td 2. search for table elements 3. foreach table element search for tr elements 4. foreach tr element search for td elements 5. return the inner_text of td elements as an array The result I expect is to get an array of 6 elements that are the inner_html of each td. Each inner search would be limited to the children of each table node. Am I doing something wrong here?

table1 tr1 td1	table1 tr1 td2
table1 tr2 td1
table1 tr3 td1

table2 tr1 td1

table2 tr1 td2

=> nil irb(main):019:0> doc = Nokogiri::HTML(File.read("nokotest.html"));ret = []; doc.search("//table").each { |tab| tab.search("//tr").each { | tr| tr.search("//td").each { |td| ret << td.inner_text }}};nil; ret.size => 48 irb(main):020:0> doc = Hpricot.parse(File.read("nokotest.html"));ret = []; doc.search("//table").each { |tab| tab.search("//tr").each { | tr| tr.search("//td").each { |td| ret << td.inner_text }}};nil; ret.size => 6 -- --- Michael L. Tracy // matasano security read us on the web: http://www.matasano.com/log -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.patterson at gmail.com Wed Jan 21 11:32:30 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Wed, 21 Jan 2009 08:32:30 -0800 Subject: [Nokogiri-talk] unexpected behavior with xpath search In-Reply-To: References: Message-ID: <6959e1680901210832y1350e23bwcf3df4f8be4b680e@mail.gmail.com> On Wed, Jan 21, 2009 at 7:33 AM, Michael Tracy wrote: > This is a pattern I use alot when testing webapps and Hpricot behaves as I > would expect, Nokogiri does something different. > The scenario: > 1.Simple page with 3 tables and varying numbers of tr/td > 2. search for table elements > 3. foreach table element search for tr elements > 4. foreach tr element search for td elements > 5. return the inner_text of td elements as an array > The result I expect is to get an array of 6 elements that are the inner_html > of each td. Each inner search would be limited to the children of each > table node. > Am I doing something wrong here? You're not doing anything wrong. Your XPath queries probably just aren't expressing what you think they're expressing. :-) Double slash indicates that you want to search relative to the root of the document. Beginning your xpath query with a slash *always* indicates "do this search from the root node". Relative searches start with a dot. Try something like this: doc.search("//table").each { |tab| tab.search(".//tr").each { |tr| tr.search(".//td").each { |td| ret << td.inner_text }}};nil; ret.size Hpricot does not implement XPath spec correctly, so this may cause confusion. I hope my example helps! -- Aaron Patterson http://tenderlovemaking.com/ From mtracy at matasano.com Wed Jan 21 12:55:55 2009 From: mtracy at matasano.com (Michael Tracy) Date: Wed, 21 Jan 2009 11:55:55 -0600 Subject: [Nokogiri-talk] unexpected behavior with xpath search In-Reply-To: <6959e1680901210832y1350e23bwcf3df4f8be4b680e@mail.gmail.com> References: <6959e1680901210832y1350e23bwcf3df4f8be4b680e@mail.gmail.com> Message-ID: That's what I get for relying on Hpricot bugs! irb(main):036:0> doc = Nokogiri::HTML(File.read ("nokotest.html"));doc.search("//table/tr/td").map { |x| x.inner_html }.size => 6 irb(main):037:0> ret = [];doc.search("//table").each { |tab| tab.search(".//tr").each { |tr| tr.search(".//td").each { |td| ret << td.inner_text }}};nil; ret.size => 6 like a charm. Thanks, -mt On Jan 21, 2009, at 10:32 AM, Aaron Patterson wrote: > On Wed, Jan 21, 2009 at 7:33 AM, Michael Tracy > wrote: >> This is a pattern I use alot when testing webapps and Hpricot >> behaves as I >> would expect, Nokogiri does something different. >> The scenario: >> 1.Simple page with 3 tables and varying numbers of tr/td >> 2. search for table elements >> 3. foreach table element search for tr elements >> 4. foreach tr element search for td elements >> 5. return the inner_text of td elements as an array >> The result I expect is to get an array of 6 elements that are the >> inner_html >> of each td. Each inner search would be limited to the children of >> each >> table node. >> Am I doing something wrong here? > > You're not doing anything wrong. Your XPath queries probably just > aren't expressing what you think they're expressing. :-) > > Double slash indicates that you want to search relative to the root of > the document. Beginning your xpath query with a slash *always* > indicates "do this search from the root node". Relative searches > start with a dot. > > Try something like this: > > doc.search("//table").each { |tab| tab.search(".//tr").each { |tr| > tr.search(".//td").each { |td| ret << td.inner_text }}};nil; ret.size > > Hpricot does not implement XPath spec correctly, so this may cause > confusion. I hope my example helps! > > -- > Aaron Patterson > http://tenderlovemaking.com/ > _______________________________________________ > Nokogiri-talk mailing list > Nokogiri-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/nokogiri-talk -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2421 bytes Desc: not available URL: From mike.dalessio at gmail.com Thu Jan 22 09:09:23 2009 From: mike.dalessio at gmail.com (Mike Dalessio) Date: Thu, 22 Jan 2009 09:09:23 -0500 Subject: [Nokogiri-talk] beginner questions In-Reply-To: References: Message-ID: <618c07250901220609x1d89b13du559e9e3c83666148@mail.gmail.com> Hi Matt, It doesn't look like anyone's tried to answer your questions, and you've probably figured out the answers by now, but for posterity, I'll try to answer anyway! On Tue, Jan 13, 2009 at 9:06 PM, Matt Mitchell wrote: > Hi, > > I just started using NG (after experiencing some Hpricot pain) and am very > happy with it so far. I'm using it to scrape large TEI files to index into > Solr, and also chunking out bits of xml fragments to disk. I've got a few > questions and well, I'll just cut to it :) > > is there a nice way to get the next/previous sibling element, not text? node.next or node.previous > > how can i get the absolute position of a node? (to the root) try node.css_path or node.path > > how can i get the relative position of a node? (to it's parent) node.parent.children.to_a.index(node) will give you an offset into the node's siblings, but that will include text nodes (including blank text nodes), so it may not be what you actually want. can you give me a use case for what you're trying to do? > > how can i get the depth of a node? walk back up the node.parent chain until you get nil. (if you think this should be in the API, make a case for it!) > > is there a way to search using negation? > - Hpricot uses a "filter" like //:not(blah) - but does not seem to work > correctly! right, hpricot's implementation was inconsistent, and we couldn't come up with a use case for supporting that operator. can you provide a use case? i'm open to implementing something that will be useful, and that can't be achieved using the existing nokogiri interface. > > does it support unicode/utf-8? you bet. you may need to provide the encoding as an additional argument to the parse method. > > > Thanks in advance! And thanks for the awesome library. > > Matt > > _______________________________________________ > Nokogiri-talk mailing list > Nokogiri-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/nokogiri-talk > > -- mike dalessio mike at csa.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From normelton at gmail.com Fri Jan 23 15:55:15 2009 From: normelton at gmail.com (Norman Elton) Date: Fri, 23 Jan 2009 15:55:15 -0500 Subject: [Nokogiri-talk] Stream Parsing Message-ID: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> I've tried hooking Nokogiri up to a TCP socket to parse incoming XML data. I'd like to parse data as it's coming in, but it looks like my start_document method isn't invoked until the socket is closed. Is there a way to parse data "on the fly"? Thanks! Norman From aaron.patterson at gmail.com Fri Jan 23 17:07:26 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Fri, 23 Jan 2009 14:07:26 -0800 Subject: [Nokogiri-talk] Stream Parsing In-Reply-To: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> References: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> Message-ID: <6959e1680901231407o6be69175p18344f227d852aa5@mail.gmail.com> On Fri, Jan 23, 2009 at 12:55 PM, Norman Elton wrote: > I've tried hooking Nokogiri up to a TCP socket to parse incoming XML > data. I'd like to parse data as it's coming in, but it looks like my > start_document method isn't invoked until the socket is closed. > > Is there a way to parse data "on the fly"? Interesting. It should be. Are you passing it an IO object? Can you provide some sample code? We do support IO streams in the SAX parser, so that *should* work. -- Aaron Patterson http://tenderlovemaking.com/ From normelton at gmail.com Fri Jan 23 21:05:11 2009 From: normelton at gmail.com (Norman Elton) Date: Fri, 23 Jan 2009 21:05:11 -0500 Subject: [Nokogiri-talk] Stream Parsing In-Reply-To: <6959e1680901231407o6be69175p18344f227d852aa5@mail.gmail.com> References: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> <6959e1680901231407o6be69175p18344f227d852aa5@mail.gmail.com> Message-ID: <6b3a7f010901231805v5fc9d6d1pacbcd4a9cce10fd0@mail.gmail.com> >> Is there a way to parse data "on the fly"? > > Interesting. It should be. Are you passing it an IO object? Can you > provide some sample code? We do support IO streams in the SAX parser, > so that *should* work. Yes, I'm passing an IO object. I've pasted a boiled-down test case online: http://pastie.org/369341 It basically defines a simple SAX handler that outputs a timestamped message for a variety of events. Then it forks itself into a read/writer, and the writer feeds XML tags into the reader. On my system, the writer delivers each message, but the reader does not recognize any until the writer closes the pipe. When I replace the nokogiri parser with just plain calls to io_read.gets, it works fine. Apologies if I'm missing something obvious here. Thanks for any help! Norman From rubikitch at ruby-lang.org Sat Jan 24 13:45:55 2009 From: rubikitch at ruby-lang.org (rubikitch at ruby-lang.org) Date: Sun, 25 Jan 2009 03:45:55 +0900 (JST) Subject: [Nokogiri-talk] HEADER_DIRS Message-ID: <20090125.034555.78759419.rubikitch@ruby-lang.org> Hi, I installed the latest snapshot of libxml2 in /usr/local, and there is older libxml2 in /usr. In this situation, nokogiri uses older one. diff --git INDEX:/ext/nokogiri/extconf.rb WORKDIR:/ext/nokogiri/extconf.rb index 8247c05..e217cc2 100644 --- INDEX:/ext/nokogiri/extconf.rb +++ WORKDIR:/ext/nokogiri/extconf.rb @@ -41,8 +41,8 @@ else HEADER_DIRS = [ File.join(INCLUDEDIR, "libxml2"), INCLUDEDIR, + '/usr/local/include/libxml2', '/usr/include/libxml2', - '/usr/local/include/libxml2' ] [ -- rubikitch Blog: http://d.hatena.ne.jp/rubikitch/ Site: http://www.rubyist.net/~rubikitch/ From aaron.patterson at gmail.com Sat Jan 24 14:01:34 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Sat, 24 Jan 2009 11:01:34 -0800 Subject: [Nokogiri-talk] HEADER_DIRS In-Reply-To: <20090125.034555.78759419.rubikitch@ruby-lang.org> References: <20090125.034555.78759419.rubikitch@ruby-lang.org> Message-ID: <6959e1680901241101x1f722454vb6c7b2354455ba85@mail.gmail.com> On Sat, Jan 24, 2009 at 10:45 AM, wrote: > Hi, > > I installed the latest snapshot of libxml2 in /usr/local, > and there is older libxml2 in /usr. > In this situation, nokogiri uses older one. > > diff --git INDEX:/ext/nokogiri/extconf.rb WORKDIR:/ext/nokogiri/extconf.rb > index 8247c05..e217cc2 100644 > --- INDEX:/ext/nokogiri/extconf.rb > +++ WORKDIR:/ext/nokogiri/extconf.rb > @@ -41,8 +41,8 @@ else > HEADER_DIRS = [ > File.join(INCLUDEDIR, "libxml2"), > INCLUDEDIR, > + '/usr/local/include/libxml2', > '/usr/include/libxml2', > - '/usr/local/include/libxml2' > ] Applied! ?????? http://github.com/tenderlove/nokogiri/commit/3c271cd7ee65a3daf3849067858ce722fcacb5c0 -- Aaron Patterson http://tenderlovemaking.com/ From mike.dalessio at gmail.com Mon Jan 26 09:14:56 2009 From: mike.dalessio at gmail.com (Mike Dalessio) Date: Mon, 26 Jan 2009 09:14:56 -0500 Subject: [Nokogiri-talk] Stream Parsing In-Reply-To: <6b3a7f010901231805v5fc9d6d1pacbcd4a9cce10fd0@mail.gmail.com> References: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> <6959e1680901231407o6be69175p18344f227d852aa5@mail.gmail.com> <6b3a7f010901231805v5fc9d6d1pacbcd4a9cce10fd0@mail.gmail.com> Message-ID: <618c07250901260614qd4d03bbu1fda9493bc1aaded@mail.gmail.com> Interesting. Two comments: 1) IO.pipe doesn't seem to be the best way to reproduce this -- even io_read.read blocks until io_write is closed, independent of Nokogiri or libxml2. 2) I've rewritten your test using sockets, and I'm getting the same result. Codes: http://pastie.org/371022 Aaron, I'm not sure what to make of this. it's actually xmlParseDocument() that's hanging, and io_read_callback doesn't get invoked until the socket is closed. Any ideas? On Fri, Jan 23, 2009 at 9:05 PM, Norman Elton wrote: > >> Is there a way to parse data "on the fly"? > > > > Interesting. It should be. Are you passing it an IO object? Can you > > provide some sample code? We do support IO streams in the SAX parser, > > so that *should* work. > > Yes, I'm passing an IO object. I've pasted a boiled-down test case online: > > http://pastie.org/369341 > > It basically defines a simple SAX handler that outputs a timestamped > message for a variety of events. Then it forks itself into a > read/writer, and the writer feeds XML tags into the reader. On my > system, the writer delivers each message, but the reader does not > recognize any until the writer closes the pipe. > > When I replace the nokogiri parser with just plain calls to > io_read.gets, it works fine. > > Apologies if I'm missing something obvious here. Thanks for any help! > > Norman > _______________________________________________ > Nokogiri-talk mailing list > Nokogiri-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/nokogiri-talk > -- mike dalessio mike at csa.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.patterson at gmail.com Mon Jan 26 12:14:28 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Mon, 26 Jan 2009 09:14:28 -0800 Subject: [Nokogiri-talk] Stream Parsing In-Reply-To: <618c07250901260614qd4d03bbu1fda9493bc1aaded@mail.gmail.com> References: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> <6959e1680901231407o6be69175p18344f227d852aa5@mail.gmail.com> <6b3a7f010901231805v5fc9d6d1pacbcd4a9cce10fd0@mail.gmail.com> <618c07250901260614qd4d03bbu1fda9493bc1aaded@mail.gmail.com> Message-ID: <6959e1680901260914g9350763lafdbde1831f9991e@mail.gmail.com> Good morning! On Mon, Jan 26, 2009 at 6:14 AM, Mike Dalessio wrote: > Interesting. Two comments: > > 1) IO.pipe doesn't seem to be the best way to reproduce this -- even > io_read.read blocks until io_write is closed, independent of Nokogiri or > libxml2. > > 2) I've rewritten your test using sockets, and I'm getting the same result. > > Codes: http://pastie.org/371022 > > Aaron, I'm not sure what to make of this. it's actually xmlParseDocument() > that's hanging, and io_read_callback doesn't get invoked until the socket is > closed. Any ideas? Woot. I have an answer. libxml buffers it's input to 4k. No parsing/callbacks happen until the buffer gets filled. Note my puts on line 58. If you run this, you'll see the callbacks get called before the document finishes writing. http://gist.github.com/52867 -- Aaron Patterson http://tenderlovemaking.com/ From normelton at gmail.com Mon Jan 26 14:42:06 2009 From: normelton at gmail.com (Norman Elton) Date: Mon, 26 Jan 2009 14:42:06 -0500 Subject: [Nokogiri-talk] Stream Parsing In-Reply-To: <6959e1680901260914g9350763lafdbde1831f9991e@mail.gmail.com> References: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> <6959e1680901231407o6be69175p18344f227d852aa5@mail.gmail.com> <6b3a7f010901231805v5fc9d6d1pacbcd4a9cce10fd0@mail.gmail.com> <618c07250901260614qd4d03bbu1fda9493bc1aaded@mail.gmail.com> <6959e1680901260914g9350763lafdbde1831f9991e@mail.gmail.com> Message-ID: <6b3a7f010901261142i21cbb9d6s3c7a37bcd4963b61@mail.gmail.com> > Woot. I have an answer. libxml buffers it's input to 4k. No > parsing/callbacks happen until the buffer gets filled. Aaron, Is there any workaround? A way to tell libxml to not buffer? I'd really like to move to Nokogiri, but need to be able to parse streams waiting for the socket to close (or the buffer to fill). Thanks!! Norman From aaron.patterson at gmail.com Mon Jan 26 22:59:57 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Mon, 26 Jan 2009 19:59:57 -0800 Subject: [Nokogiri-talk] Stream Parsing In-Reply-To: <6b3a7f010901261142i21cbb9d6s3c7a37bcd4963b61@mail.gmail.com> References: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> <6959e1680901231407o6be69175p18344f227d852aa5@mail.gmail.com> <6b3a7f010901231805v5fc9d6d1pacbcd4a9cce10fd0@mail.gmail.com> <618c07250901260614qd4d03bbu1fda9493bc1aaded@mail.gmail.com> <6959e1680901260914g9350763lafdbde1831f9991e@mail.gmail.com> <6b3a7f010901261142i21cbb9d6s3c7a37bcd4963b61@mail.gmail.com> Message-ID: <6959e1680901261959t2b446794qd9115d79e27c6fb7@mail.gmail.com> On Mon, Jan 26, 2009 at 11:42 AM, Norman Elton wrote: >> Woot. I have an answer. libxml buffers it's input to 4k. No >> parsing/callbacks happen until the buffer gets filled. > > Aaron, > > Is there any workaround? A way to tell libxml to not buffer? I'd > really like to move to Nokogiri, but need to be able to parse streams > waiting for the socket to close (or the buffer to fill). How did I know this would be the next question? ;-) As far as I can tell, we can't tell libxml not to buffer. Technically, we could decrease the size of the buffer, but that looks very dangerous. I can't seem to find anything in the API documentation that would let us force libxml to make it's callbacks. Given this new info, do you have any ideas Mike? I can seem to find any help in the libxml source. Just out of curiosity, what size chunks are you parsing? I'm trying to visualize a use case where a 4k buffer size is too large. If you're dealing with small chunks, a push parser might be better. I am very close to having the push parser work. :-) -- Aaron Patterson http://tenderlovemaking.com/ From normelton at gmail.com Mon Jan 26 23:28:26 2009 From: normelton at gmail.com (Norman Elton) Date: Mon, 26 Jan 2009 23:28:26 -0500 Subject: [Nokogiri-talk] Stream Parsing In-Reply-To: <6959e1680901261959t2b446794qd9115d79e27c6fb7@mail.gmail.com> References: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> <6959e1680901231407o6be69175p18344f227d852aa5@mail.gmail.com> <6b3a7f010901231805v5fc9d6d1pacbcd4a9cce10fd0@mail.gmail.com> <618c07250901260614qd4d03bbu1fda9493bc1aaded@mail.gmail.com> <6959e1680901260914g9350763lafdbde1831f9991e@mail.gmail.com> <6b3a7f010901261142i21cbb9d6s3c7a37bcd4963b61@mail.gmail.com> <6959e1680901261959t2b446794qd9115d79e27c6fb7@mail.gmail.com> Message-ID: <6b3a7f010901262028v7135a112nb9d2f11e3bc3da12@mail.gmail.com> > Just out of curiosity, what size chunks are you parsing? I'm trying > to visualize a use case where a 4k buffer size is too large. If > you're dealing with small chunks, a push parser might be better. I am > very close to having the push parser work. :-) If I can hook an IO up to a push parser, I'm pretty sure that's exactly what I'd want. The details of my application are a little non-standard. Basically, a TCP socket exists between server and client. The RPC exchange consists of two long-running XML documents, one in each direction. Both machines exchange xml-decl's, followed by an opening tag. The client then burps XML snippets to the server, which responds with a response snippet. The document never actually "ends" until the session is complete and the socket closed. In order for this to work, each speaker needs to recognize each other's communication without the document fully parsing or the socket closing. It sounds like a push parser would do the trick :) Thanks for your help! Norman From aaron.patterson at gmail.com Mon Jan 26 23:50:03 2009 From: aaron.patterson at gmail.com (Aaron Patterson) Date: Mon, 26 Jan 2009 20:50:03 -0800 Subject: [Nokogiri-talk] Stream Parsing In-Reply-To: <6b3a7f010901262028v7135a112nb9d2f11e3bc3da12@mail.gmail.com> References: <6b3a7f010901231255v2d93beebg3eba63dbbb2bccae@mail.gmail.com> <6959e1680901231407o6be69175p18344f227d852aa5@mail.gmail.com> <6b3a7f010901231805v5fc9d6d1pacbcd4a9cce10fd0@mail.gmail.com> <618c07250901260614qd4d03bbu1fda9493bc1aaded@mail.gmail.com> <6959e1680901260914g9350763lafdbde1831f9991e@mail.gmail.com> <6b3a7f010901261142i21cbb9d6s3c7a37bcd4963b61@mail.gmail.com> <6959e1680901261959t2b446794qd9115d79e27c6fb7@mail.gmail.com> <6b3a7f010901262028v7135a112nb9d2f11e3bc3da12@mail.gmail.com> Message-ID: <6959e1680901262050n52bb03a7md12dc12b8638a5d4@mail.gmail.com> On Mon, Jan 26, 2009 at 8:28 PM, Norman Elton wrote: >> Just out of curiosity, what size chunks are you parsing? I'm trying >> to visualize a use case where a 4k buffer size is too large. If >> you're dealing with small chunks, a push parser might be better. I am >> very close to having the push parser work. :-) > > If I can hook an IO up to a push parser, I'm pretty sure that's > exactly what I'd want. > > The details of my application are a little non-standard. Basically, a > TCP socket exists between server and client. The RPC exchange consists > of two long-running XML documents, one in each direction. Both > machines exchange xml-decl's, followed by an opening tag. The client > then burps XML snippets to the server, which responds with a response > snippet. The document never actually "ends" until the session is > complete and the socket closed. > > In order for this to work, each speaker needs to recognize each > other's communication without the document fully parsing or the socket > closing. It sounds like a push parser would do the trick :) Gotcha. Sounds like you've got an XMPP type situation. Yes, you want push parsing. Keep an eye on this ticket: http://nokogiri.lighthouseapp.com/projects/19607/tickets/9-push-parsing-chunked-parsing I think I'll have something tonight or tomorrow. -- Aaron Patterson http://tenderlovemaking.com/

#{h1_euc}

#{h1_euc}