网站刮擦时需要很长时间才能加载
我不确定要提供什么标签，但是我在Python中使用硒，所以我决定从这里开始。我正在使用Selenium和Python的请求抓取数千次网站。它启动很快，但是在3400页的加载标记附近，它从大约0.1秒降至3或4秒。关于速度减慢网页加载的任何想法。该程序正在以非常低的功率线条（1个共享CPU和1GB的RAM）上运行。 CPU从一开始就固定在快速运行时，并且据我所知，它并非使用所有RAM。我还给了10 GB交换。我的互联网下载和上传高于200 MB/s。我以为网站主持人本身正在限制我，但我不太了解这些东西，无法确定。
I am unsure what tags to give this but I am using Selenium in python so I decided to start here. I am scraping a website thousands of times using selenium and requests in python. It starts fairly quickly but around the 3400 page load mark it slows down from around .1 seconds to 3 or 4 seconds. Any ideas on what is slowing the webpages loading. The program is being run on a very low power Linode (1 shared cpu and 1gb of ram). The cpu is pegged from the beginning when it is still running fast and from what I can tell, it is not using all the RAM. I also gave it a 10 gb swap. My internet download and upload is above 200 MB/s. I was thinking the website host themselves are limiting me but I don't know this stuff well enough to be sure.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
可以肯定这是主人。如果它们通过您的IP限制，则可能需要使用一些代理。如果网站是共享托管或一些低成本托管的，那么代理将无济于事。
Pretty sure it's the host. If they are limiting by your IP, you may want to use some proxies. If the website is on shared hosting or some low cost hosting then proxies won't help.