当前位置：文江博客话题详情

逆向工程：arrow.com 上每页的结果

发布于 2024-11-24 08:55:09 字数 227 浏览 0 评论 0原文

我试图增加每页的结果，例如当您打开时： http://components.arrow.com/part/search/BAV99

该网站为您提供通过单击链接 50 来选择显示 10、25、50 或 100 的选项。现在我的问题是如何使用curl 显示 99999 个结果。

原文

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

感受沵的脚步 2024-12-01 08:55:09

我使用 Fiddler2 提出以下 HTTP 请求，看起来它在一页上返回所有 143 个匹配项。我通过观察从下拉列表中选择“100 个结果”时发送的 HTTP 请求来完成此操作。然后，我复制原始请求，将其粘贴到 Fiddler2 的 RequestBuilder 中，更改 limit=200 并执行它。我将 WebView 中的响应与网站上显示的实际页面进行了比较。

POST http://components.arrow.com/part/search/BAV99 HTTP/1.1
Host: components.arrow.com
Proxy-Connection: keep-alive
Referer: http://components.arrow.com/part/search/BAV99
Content-Length: 331
Cache-Control: max-age=0
Origin: http://components.arrow.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: __utmz=199791230.1309618262.1.1.utmcsr=stackoverflow.com|utmccn=(referral)|utmcmd=referral|utmcct=/questions/5359805/problem-while-parsing-html-xml-and-quering-with-xpath; __utma=199791230.901122760.1309618262.1309618262.1309618262.1; region=na; lang=en; JSESSIONID=s4CyTyHKHbwhJc2v1xXyv50s3mvshZmZJJzGYQNjpmHnw2MmcgyT!1070203391

sort1Name=&sort1Order=&start=0&docid=&cat=&filtMultiSelect=&catFiltAddOn=&chkButton=1&search_token=BAV99&limit=200&requestedURL=http%3A%2F%2Fext.partsearch.arrow.com%2Fnacpartservice%2Fsearch%3Fappid%3Dnac%26cc%3DUS%26cn%3DARROW%2FAMERICAS%26lang%3Den%26retfilt%3DY%26retreq%3DY%26srchtxt%3DBAV99%26start%3D0%26limit%3D10&taxonomy=

I used Fiddler2 to come up with the following HTTP request, which looks like it returns all 143 matching items one a single page. I did this by watching the HTTP request that was sent when I selected "100 Results" from the drop-down. I then copied the raw request, pasted it into Fiddler2's RequestBuilder, changed the limit=200, and executed it. I compared the response in the WebView against the actual pages displayed on the website.

POST http://components.arrow.com/part/search/BAV99 HTTP/1.1
Host: components.arrow.com
Proxy-Connection: keep-alive
Referer: http://components.arrow.com/part/search/BAV99
Content-Length: 331
Cache-Control: max-age=0
Origin: http://components.arrow.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: __utmz=199791230.1309618262.1.1.utmcsr=stackoverflow.com|utmccn=(referral)|utmcmd=referral|utmcct=/questions/5359805/problem-while-parsing-html-xml-and-quering-with-xpath; __utma=199791230.901122760.1309618262.1309618262.1309618262.1; region=na; lang=en; JSESSIONID=s4CyTyHKHbwhJc2v1xXyv50s3mvshZmZJJzGYQNjpmHnw2MmcgyT!1070203391

sort1Name=&sort1Order=&start=0&docid=&cat=&filtMultiSelect=&catFiltAddOn=&chkButton=1&search_token=BAV99&limit=200&requestedURL=http%3A%2F%2Fext.partsearch.arrow.com%2Fnacpartservice%2Fsearch%3Fappid%3Dnac%26cc%3DUS%26cn%3DARROW%2FAMERICAS%26lang%3Den%26retfilt%3DY%26retreq%3DY%26srchtxt%3DBAV99%26start%3D0%26limit%3D10&taxonomy=

回复收藏 0 原文

放手` 2024-12-01 08:55:09

您要定位的页面，http://components.arrow.com/part/search/BAV99 ，使用表单提供 POST 数据，服务器使用该数据来识别要返回的记录数、用于分页的起始偏移量、应如何排序等。

正如 @EvanLarsen 在他的答案，你需要使用curl -d“”。

然而，看起来该网站将正确响应仅提供您真正关心的那些字段的请求。

仅返回第一条记录：

curl -d "limit=1" http://components.arrow.com/part/search/BAV99

仅返回第二条记录：

curl -d "limit=1&start=1" http://components.arrow.com/part/search/BAV99

返回 99999 条记录（如您所问）：

curl -d "limit=99999" http://components.arrow.com/part/search/BAV99

如果您想要一个像样的 UI 以便在 Web 浏览器中更轻松地修改和提交 POST 数据以及查看结果（而不是使用 <代码>curl），查看 Request Maker Chrome 扩展程序。

The page you're targeting, http://components.arrow.com/part/search/BAV99, uses a form to supply POST data the server uses to identify how many records to return, what starting offset to use for pagination, how it should be sorted, etc.

As @EvanLarsen's noted in his answer, you need to use curl -d "<POST_DATA>" <URL>.

However, it appears the site will properly respond to a request with only those fields you actually care about supplied.

To return only the first record:

curl -d "limit=1" http://components.arrow.com/part/search/BAV99

To return only the second record:

curl -d "limit=1&start=1" http://components.arrow.com/part/search/BAV99

To return 99999 records (as you asked):

curl -d "limit=99999" http://components.arrow.com/part/search/BAV99

If you want a decent UI to make modifying and submitting POST data and viewing results easier in a web browser (instead of using curl), check out the Request Maker Chrome Extension.

回复收藏 0 原文

凡间太子 2024-12-01 08:55:09

试试这个：

curl -d "sort1Name=&sort1Order=&start=0&docid=&cat=&filtMultiSelect=&catFiltAddOn=&chkButton=1&search_token=BAV99&limit=9999&requestedURL=http%3A%2F%2Fext.partsearch.arrow.com%2Fnacpartservice%2Fsearch%3Fappid%3Dnac%26cc%3DUS%26cn%3DARROW%2FAMERICAS%26lang%3Den%26retfilt%3DY%26retreq%3DY%26srchtxt%3DBAV99%26start%3D0%26limit%3D10&taxonomy=" http://components.arrow.com/part/search/BAV99

Try this:

curl -d "sort1Name=&sort1Order=&start=0&docid=&cat=&filtMultiSelect=&catFiltAddOn=&chkButton=1&search_token=BAV99&limit=9999&requestedURL=http%3A%2F%2Fext.partsearch.arrow.com%2Fnacpartservice%2Fsearch%3Fappid%3Dnac%26cc%3DUS%26cn%3DARROW%2FAMERICAS%26lang%3Den%26retfilt%3DY%26retreq%3DY%26srchtxt%3DBAV99%26start%3D0%26limit%3D10&taxonomy=" http://components.arrow.com/part/search/BAV99

回复收藏 0 原文