在一个网站内抓取所有网页的最快方法

Question

我有一个 C# 应用程序，需要尽快抓取某个域内的许多页面。我有一个 Parallel.Foreach 循环遍历所有 url（多线程）并使用下面的代码来抓取它们：

private string ScrapeWebpage(string url, DateTime? updateDate)
        {
            HttpWebRequest request = null;
            HttpWebResponse response = null;
            Stream responseStream = null;
            StreamReader reader = null;
            string html = null;

            try
            {
                //create request (which supports http compression)
                request = (HttpWebRequest)WebRequest.Create(url);
                request.Pipelined = true;
                request.KeepAlive = true;
                request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
                if (updateDate != null)
                    request.IfModifiedSince = updateDate.Value;

                //get response.
                response = (HttpWebResponse)request.GetResponse();
                responseStream = response.GetResponseStream();
                if (response.ContentEncoding.ToLower().Contains("gzip"))
                    responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
                else if (response.ContentEncoding.ToLower().Contains("deflate"))
                    responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);

                //read html.
                reader = new StreamReader(responseStream, Encoding.Default);
                html = reader.ReadToEnd();
            }
            catch
            {
                throw;
            }
            finally
            {//dispose of objects.
                request = null;
                if (response != null)
                {
                    response.Close();
                    response = null;
                }
                if (responseStream != null)
                {
                    responseStream.Close();
                    responseStream.Dispose();
                }
                if (reader != null)
                {
                    reader.Close();
                    reader.Dispose();
                }
            }
            return html;
        }

如您所见，我有 http 压缩支持，并将 request.keepalive 和 request.pipelined 设置为 true。我想知道我使用的代码是否是在同一站点内抓取多个网页的最快方法，或者是否有更好的方法可以使会话针对多个请求保持打开状态。我的代码正在为我点击的每个页面创建一个新的请求实例，我是否应该尝试仅使用一个请求实例来点击所有页面？启用管道和保活是否理想？

Answer 1

事实证明我错过的是这个：

ServicePointManager.DefaultConnectionLimit = 1000000;