以编程方式登录网站以进行屏幕抓取的最佳方法是什么？ （最好是Python）
我希望能够以编程方式登录网站并定期从该网站获取一些信息。 可以使这件事尽可能简单的最佳工具是什么？ 我更喜欢某种类型的 Python 库，因为我想更加精通 Python，但我愿意接受任何建议。
I want to be able to log into a website programmatically and periodically obtain some information from the site. What is the best tool(s) that would make this as simple as possible? I'd prefer a Python library of some type because I want to become more proficient in Python, but I'm open to any suggestions.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（5）
您可以尝试使用 Mechanize (http://wwwsearch.sourceforge.net/mechanize/) 进行程序化网络-浏览，并且一定要使用 Beautiful Soup (http://www.crummy.com/software/BeautifulSoup/ ）用于抓取。
You can try Mechanize (http://wwwsearch.sourceforge.net/mechanize/) for programmatic web-browsing, and definitely use Beautiful Soup (http://www.crummy.com/software/BeautifulSoup/) for the scraping.
我们大多数人使用 urllib2 来获取页面； 它可以处理各种形式的身份验证和 cookie 收集。 然后Beautiful Soup解析结果。
Most of us use urllib2 to get the page; it can handle various forms of authentication and cookie collection. Then Beautiful Soup to parse the results.
我曾经写过一个Python脚本来自动登录vBulletin论坛。 困难的部分是知道如何正确地形成登录请求，而这是图书馆无法帮助您的。 我发现 Live Http Headers - Firefox 的一个插件 - 对于查看客户端之间发送的内容非常有帮助和服务器在登录过程中。
我也同意其他人的观点，美丽的汤非常棒。
I once wrote a Python script to automatically log into vBulletin forums. The difficult part was knowing how to correctly form the login request and that is something that a library won't help you with. I found Live Http Headers - an addon for Firefox - to be pretty helpful in seeing what is sent between the client and server during the login process.
I also agree with everyone else that Beautiful Soup is pretty awesome.
我建议使用 twill 它可以轻松完成登录过程。 然后使用 beautifulsoup 等。
如上所述。 我从未尝试过机械化，但看起来不错。
i recommend using twill it makes it a snap to do the login procedure. then use beautifulsoup etc.
as described above. ive never tried mechanize, but it looks pretty good.
只是为了屏幕抓取，您可以使用 url lib + pyqyery 的组合。
https://pythonhosted.org/pyquery/
just for screen scraping you can use combination of url lib + pyqyery.
https://pythonhosted.org/pyquery/