链接检查器；如何避免误报

Question

我正在开发一个链接检查器/损坏的链接查找器，我收到了很多误报，经过仔细检查，我注意到许多错误代码返回了 webExceptions，但它们实际上是可下载的，但在其他一些情况下，状态代码是 404 并且我可以从浏览器访问该页面。

这是代码，它非常丑陋，我想有更多的东西，我说实用。所有状态代码都那么大，如果用于过滤我不想添加到断开链接的状态代码，因为它们是有效链接（我测试了它们全部）。我需要修复的是结构（如果可能）以及如何不出现错误 404。

谢谢！

try
{
   HttpWebRequest request = ( HttpWebRequest ) WebRequest.Create ( uri );
   request.Method = "Head";
   request.MaximumResponseHeadersLength = 32; // FOR IE SLOW SPEED
   request.AllowAutoRedirect = true;
   using ( HttpWebResponse response = ( HttpWebResponse ) request.GetResponse() )
   {
      request.Abort();
   }
   /* WebClient wc = new WebClient();
     wc.DownloadString( uri ); */

   _validlinks.Add ( strUri );
}
catch ( WebException wex )
{
   if (    !wex.Message.Contains ( "The remote name could not be resolved:" ) &&
           wex.Status != WebExceptionStatus.ServerProtocolViolation )
   {
      if ( wex.Status != WebExceptionStatus.Timeout )
      {
         HttpStatusCode code = ( ( HttpWebResponse ) wex.Response ).StatusCode;
         if (
            code != HttpStatusCode.OK &&
            code != HttpStatusCode.BadRequest &&
            code != HttpStatusCode.Accepted &&
            code != HttpStatusCode.InternalServerError &&
            code != HttpStatusCode.Forbidden &&
            code != HttpStatusCode.Redirect &&
            code != HttpStatusCode.Found
         )
         {
            _brokenlinks.Add ( new Href ( new Uri ( strUri , UriKind.RelativeOrAbsolute ) , UrlType.External ) );
         }
         else _validlinks.Add ( strUri );
      }
      else _brokenlinks.Add ( new Href ( new Uri ( strUri , UriKind.RelativeOrAbsolute ) , UrlType.External ) );
   }
   else _validlinks.Add ( strUri );
}

Answer 1

您应该添加 UserAgent 标头，因为许多网站都需要它们。