C# WebRequest 与 php 重定向页面

发布于 2024-10-24 06:46:51 字数 424 浏览 4 评论 0原文

我正在尝试编写一个小型网络爬虫。我从一些 html 中解析出 url，有时我会得到一个 php 重定向页面。我正在寻找一种方法来获取重定向页面的 uri。

我正在尝试使用 System.Net.WebRequest 使用这样的代码获取流。

        WebRequest req = WebRequest.Create(link);
        Stream s = req.GetResponse().GetResponseStream();
        StreamReader st =  new StreamReader(WebRequest.Create(link).GetResponse().GetResponseStream());

问题是链接是 PHP 重定向，因此流始终为空。如何获取 php 重定向页面的 URI？

原文

I am messing around trying to write a small web crawler. I parse out a url from some html and sometimes I get a php redirect page. I am looking for a way to get the uri of the redirected page.

I am trying to use System.Net.WebRequest to get a a stream using code like this

        WebRequest req = WebRequest.Create(link);
        Stream s = req.GetResponse().GetResponseStream();
        StreamReader st =  new StreamReader(WebRequest.Create(link).GetResponse().GetResponseStream());

The problem is that the link is a PHP redirect, so the stream is always null. How would I get the URI to the page the php is redirecting?

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

情未る 2024-10-31 06:46:51

    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(link);
    req.AllowAutoRedirect = true;
    reg.AutomaticDecompression = DecompressionMethods.GZip;

    StreamReader _st = new StreamReader(_req.GetResponseStream(), System.Text.Encoding.GetEncoding(req.CharacterSet));

AllowAutoRedirect 会自动将您带到新的 URI；如果那是你想要的效果。自动解压缩将自动解压缩压缩的响应。此外，您应该在 try catch 块中执行获取响应流部分。我的经验是它抛出了很多 WebExceptions。

由于您正在尝试这项技术，请确保您使用正确的编码读取数据。如果您尝试在不使用 Unicode 的情况下从日本站点获取数据，则该数据将无效。

    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(link);
    req.AllowAutoRedirect = true;
    reg.AutomaticDecompression = DecompressionMethods.GZip;

    StreamReader _st = new StreamReader(_req.GetResponseStream(), System.Text.Encoding.GetEncoding(req.CharacterSet));

the AllowAutoRedirect will automatically take you to the new URI; if that is you're desired effect. The AutomaticDecompression will auto decompress compressed responses. Also you should be executing the get response stream part in a try catch block. I my exp it throws alot of WebExceptions.

Since you're experimenting with this technology make sure you read the data with the correct encoding. If you attempt to get data from a japanese site without using Unicode then the data will be invalid.

回复收藏 0 原文