使用 HttpClient 4.1.1 避免循环重定向

Question

如何使用 HttpClient 4.1.1 避免循环重定向。当我收到这样的错误时：-

executing requestGET http://home.somehost.com/Mynet/pages/cHome.xhtml HTTP/1.1
org.apache.http.client.ClientProtocolException
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
    at edu.uci.ics.crawler4j.url.WebURL.setURL(WebURL.java:122)
    at edu.uci.ics.crawler4j.crawler.CrawlController.addSeed(CrawlController.java:207)
    at edu.uci.ics.crawler4j.example.advanced.Controller.main(Controller.java:31)
Caused by: org.apache.http.client.CircularRedirectException: Circular redirect to 'http://home.somehost.com/Mynet/pages/Home.xhtml'
    at org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI(DefaultRedirectStrategy.java:168)
    at org.apache.http.impl.client.DefaultRedirectStrategy.getRedirect(DefaultRedirectStrategy.java:193)
    at org.apache.http.impl.client.DefaultRequestDirector.handleResponse(DefaultRequestDirector.java:1021)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:482)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)

这是我的代码......

DefaultHttpClient client = null;

        try
        {
            // Set url
            //URI uri = new URI(url.toString());

            client = new DefaultHttpClient();

            client.getCredentialsProvider().setCredentials(
                    new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT, AuthScope.ANY_REALM),
                    new UsernamePasswordCredentials("test", "test"));


            URL url1 = new URL (url);
            HttpURLConnection connection = (HttpURLConnection) url1.openConnection();
            connection.setFollowRedirects(false);

            HttpGet request = new HttpGet(url);
            final HttpParams params = new BasicHttpParams();
            HttpClientParams.setRedirecting(params, false);
            HttpContext context = new BasicHttpContext();

            System.out.println("----------------------------------------");
            System.out.println("executing request" + request.getRequestLine());
            HttpResponse response = client.execute(request, context);
            HttpEntity entity = response.getEntity();


            System.out.println(response.getStatusLine());
                    InputStream content = entity.getContent();
                    BufferedReader in   = 
                        new BufferedReader (new InputStreamReader (content));
                    String line;
                    while ((line = in.readLine()) != null) {
                       // System.out.println(line);
                    }
                } catch(Exception e) {
                    e.printStackTrace();
                }

Answer 1

您可以将 ClientPNames.ALLOW_CIRCULAR_REDIRECTS 设置为 true，这将允许重定向到同一位置。

  client.getParams().setParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS, true); 

请在此处查看更多信息

Answer 2

您可以尝试：

RequestConfig requestConfig = RequestConfig.custom()
                              .setCircularRedirectsAllowed(true)
                              .build();

HttpClient httpClient = HttpClients.custom()
                        .setDefaultRequestConfig(requestConfig)
                        .setRedirectStrategy(new LaxRedirectStrategy())
                        .build();

HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setHttpClient(httpClient);

Answer 3

你只是回避了它。 HttpClient 检测到循环重定向并引发异常。如果它没有被“避免”，它将永远继续重定向（直到您决定终止该进程）。如果服务器响应的话，没有太多其他选择。

真正避免循环重定向循环的唯一方法是修复服务器。

如果您想知道发生了什么（例如为什么它似乎可以在浏览器中查找但不能从您的程序中查找），请尝试打开一些额外的 HttpClient 日志记录。特别是，确保您可以看到来回发送的所有 HTTP 标头。然后，您可以查看在浏览器中发出相同请求时发生的对话，并记下差异。可能是缺少 cookie、疯狂的浏览器检测等等……

有多种方法可以跟踪浏览器的通信。以下是我经常使用的几种方法，按从最简单到最难的顺序排列（恕我直言）：

• Firefox + HttpFox（或 LiveHttpHeaders、Firebug 等...）
• Fiddler（仅限 Windows）
• Wireshark/tcpdump

对于低级测试，请尝试使用 telnet（除非您使用 Windows，在这种情况下，您最好使用 PuTTY/plink）并判断哪些更改会导致循环重定向。

Answer 4

从4.0开始，Apache HttpClient中有一个会导致循环重定向的错误，即使在最新版本中也没有修复。

在DefaultRequestDirector.java中，它创建了一个HttpRedirect来执行重定向，并且它将重用原始HttpGet中的所有标头，这里的问题是它还会重用Host标头，这意味着服务器仍然会获取尝试重定向到新 URI 后的原始主机。

我通过重新实现 DefaultRequestDirector:

public class RedirectRequestDirector extends DefaultRequestDirector
{
    RedirectRequestDirector(
            final HttpRequestExecutor requestExec,
            final ClientConnectionManager conman,
            final ConnectionReuseStrategy reustrat,
            final ConnectionKeepAliveStrategy kastrat,
            final HttpRoutePlanner rouplan,
            final HttpProcessor httpProcessor,
            final HttpRequestRetryHandler retryHandler,
            final RedirectHandler redirectHandler,
            final AuthenticationHandler targetAuthHandler,
            final AuthenticationHandler proxyAuthHandler,
            final UserTokenHandler userTokenHandler,
            final HttpParams params) 
    {
        super(requestExec, conman, reustrat, kastrat, rouplan, httpProcessor, retryHandler, redirectHandler, targetAuthHandler, proxyAuthHandler, userTokenHandler, params);

    }
    @Override
    protected RoutedRequest handleResponse(RoutedRequest roureq,
            HttpResponse response,
            HttpContext context)
                    throws HttpException, IOException
    {
        RoutedRequest req = super.handleResponse(roureq, response, context);
        if(req != null)
        {
            String redirectTarget = req.getRoute().getTargetHost().getHostName();
            req.getRequest().getOriginal().setHeader("Host", redirectTarget);
        }
        return req;
    }

}

和 DefaultHttpClient:

public class RedirectHttpClient extends DefaultHttpClient
{
    @Override
    protected RequestDirector createClientRequestDirector(
            final HttpRequestExecutor requestExec,
            final ClientConnectionManager conman,
            final ConnectionReuseStrategy reustrat,
            final ConnectionKeepAliveStrategy kastrat,
            final HttpRoutePlanner rouplan,
            final HttpProcessor httpProcessor,
            final HttpRequestRetryHandler retryHandler,
            final RedirectHandler redirectHandler,
            final AuthenticationHandler targetAuthHandler,
            final AuthenticationHandler proxyAuthHandler,
            final UserTokenHandler stateHandler,
            final HttpParams params) {
        return new RedirectRequestDirector(
                requestExec,
                conman,
                reustrat,
                kastrat,
                rouplan,
                httpProcessor,
                retryHandler,
                redirectHandler,
                targetAuthHandler,
                proxyAuthHandler,
                stateHandler,
                params);
    }
}

解决了这个问题：现在我不会抱怨循环重定向。

Answer 5

检查您的请求在发送到您请求的网址之前是否未发送到代理。

Answer 6

我在春季版本升级时遇到了这个问题，在我的情况下上下文没有正确初始化。

在org.apache.http.impl.client.DefaultRedirectStrategy中：

RedirectLocations redirectLocations = (RedirectLocations) clientContext.getAttribute(
                HttpClientContext.REDIRECT_LOCATIONS);

clientContext的值应该是basicHttpContext，但是Spring Web
(4.3.x.RELEASE) 正在初始化上下文：

org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal();

clientContext 的值正在更改，这会导致循环重定向错误。
Spring Web (3.2.x.RELEASE) 不会初始化上下文，并且值将为 null。