.NET HttpWebRequest 中的压缩(标头)错误?

发布于 2024-10-03 18:52:02 字数 1363 浏览 3 评论 0原文

我刚刚注意到,使用 .NET HttpWebRequest,我没有从服务器(运行 nginx 0.8.52)获取 gzip 内容。通过使用curl 进行测试,我验证了gzip 已在服务器端正确设置,然后使用VS 调试器深入研究了该问题。

罪魁祸首是 Accept-Encoding 标头字段的生成方式:如果我设置,

 request.AutomaticDecompression = DecompressionMethods.GZip | 
                                   DecompressionMethods.Deflate`

我会得到以下标头:

`Accept-Encoding: gzip, deflate,gzip, deflate`. 

如果我设置,

request.AutomaticDecompression = DecompressionMethods.GZip; 

我会得到

Accept-Encoding: gzip,gzip

我没有检查 HTTP 规范所说的内容,但是 nginx应该可以处理这种情况,但它却返回了 Vary: Accept-Encoding 。另一方面,由 HttpWebRequest 生成的 Accept-Encoding 标头看起来绝对不正确,对我来说似乎是一个错误。

如果我不指定 AutomaticDecompression 并手动设置 Accept-Encoding 标头,我会返回 gzip 内容,但 HttpWebRequest 似乎是非常愚蠢,并且不解码压缩流(我猜它依赖于内部 IsCompressed 标志而不是解析响应标头)。

有什么建议吗?

编辑:

应该注意的是,涉及身份验证;我刚刚发现 HttpWebRequest 最初执行的请求不包含提供的凭据。对于此请求,正确指定了 Accept-Encoding 标头。在获取 401 服务器响应并使用凭据重新执行请求后,会发生重复。

编辑2:

我发布了 Microsoft Connect 错误报告< /a>.

I just noticed that using the .NET HttpWebRequest, I wasn't getting gzip'd contents back from the server (running nginx 0.8.52). Testing with curl, I verified that gzip was set up properly server-side, then dug into the issue with the VS debugger.

The culprit turns out to be how the Accept-Encoding header field is generated: if I set

 request.AutomaticDecompression = DecompressionMethods.GZip | 
                                   DecompressionMethods.Deflate`

I get the following header:

`Accept-Encoding: gzip, deflate,gzip, deflate`. 

If I set

request.AutomaticDecompression = DecompressionMethods.GZip; 

I get

Accept-Encoding: gzip,gzip

I haven't checked what the HTTP specification says, but nginx ought to handle the situation, but instead it returns Vary: Accept-Encoding back. On the other hand, the Accept-Encoding header generated by HttpWebRequest definitely doesn't look right, and seems like a bug to me.

If I don't specify AutomaticDecompression and manually set the Accept-Encoding header, I get gzip'd content back, but HttpWebRequest seems to be pretty dumb, and doesn't decode the compressed stream (I guess it relies on an internal IsCompressed flag rather than parsing the response headers).

Any suggestions?

EDIT:

It should be noted that there's authentication involved; I've just found out that HttpWebRequest initially does a request without including the supplied credentials. For this request, the Accept-Encoding header is specified correctly. The duplication occurs after getting the 401 server response and re-doing the request with credentials.

EDIT 2:

I posted a Microsoft Connect bug report.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

魔法唧唧 2024-10-10 18:52:02

看来是一个错误好吧。可能的解决方法:

  1. 不要使用自动解压缩,手动设置“Accept-Encoding”标头字段,如果设置了“Content-Encoding”属性,则手动处理响应流的解压缩。
  2. 不要使用 Credentials 属性,而是手动构建 HTTP 授权标头。这消除了不必要的服务器往返并修复了标头字段重复错误。

Seems to be a bug alright. Possible workarounds:

  1. Don't use AutomaticDecompression, set "Accept-Encoding" header field manually, and manually handle decompression of the response stream if "Content-Encoding" property is set.
  2. Don't use Credentials property, instead manually construct HTTP Authorization header. This gets rid of unnecessary server roundtrip and fixes the header field duplication bug.
你怎么敢 2024-10-10 18:52:02

哈哈,有趣的是你提到了这一点,我在周五就注意到了这一点;P

我没有看到因此造成的任何问题,我的回复仍然很压缩。

我还注意到了几个代理请求。但我一生都无法在 MS ISA 或具有长名称服务器的新 ISA 上进行 GZip 压缩。似乎只是剥离了请求头,而服务器甚至不知道它。明天我将与我们的一位网络管理员开会,看看问题是否与代理相关。

更新:

我应该注意,我在自己和服务器之间放置 Squid 代理没有问题,但这只是使用基本身份验证,而不是 Windows/ISA 使用的花哨的质询响应内容。

更新2:

我忘了提及,当使用浏览器访问该网站时,Accept-Encoding 标头也会“消失”。这就是为什么我认为真正的问题可能不是 .NET 特有的。测试仍在继续。

更新 3:

我使用 FF3.6 通过 ISA 代理进行了捕获,结果几乎相同。通过 ISA,仅对“单个”代理进行身份验证,但最终响应仍然未压缩。

鉴于 MS 产品中遗传性错误的发生率很高,我不得不支持 ISA/FTMG 是罪魁祸首。

<罢工>
更新4(已解决):

刚刚开会,解决了问题(稍后将发布具有适当设置的屏幕截图)。所以预感是正确的(至少就我而言)。

如果托管服务器的网络中使用了 ISA/FTMG 配置,我强烈建议您研究一下。

更新 5(确实已解决):

尝试了另一种设置,结果成功了。在 FTMG 代理上,我们必须为“外部”网络启用 HTTP 压缩,并确保可压缩内容类型正确(必须为 WCF 添加 application/soap+xml)。这允许代理进行压缩,而实际的 Web 服务器则不能,这可能是理想的,也可能不是理想的,但至少它可以工作。 :)

我强烈建议您研究 ISA/FTMG 配置(如果在托管服务器的网络中使用该配置)。

捕获日志:

这是我通过 Wireshark 捕获的 Forefront TMG 代理使用 IE8 访问网站的跟踪记录。如果错误是 .NET 特定的,我预计结果会有所不同,但从我所看到的来看,它的行为是相同的。

初始请求:

GET http://www.foo.com/ClientInstall HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: www.foo.com

HTTP/1.1 407 Proxy Authentication Required 
  ( Forefront TMG requires authorization to fulfill the request. 
  Access to the Web Proxy filter is denied.  )
Via: 1.1 PROXP01
Proxy-Authenticate: Negotiate
Proxy-Authenticate: Kerberos
Proxy-Authenticate: NTLM
Proxy-Authenticate: Basic realm="PROXP01.bar.com"
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 4140  

使用代理进行身份验证,但失败?:

GET http://www.foo.com/ClientInstall HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: www.foo.com
Proxy-Authorization: Negotiate TlRMTVNTUAABAAAA...

HTTP/1.1 407 Proxy Authentication Required ( Access is denied.  )
Via: 1.1 PROXP01
Proxy-Authenticate: Negotiate TlRMTVNTUAACAAAAB...
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 0     

最终序列:

GET http://www.foo.com/ClientInstall HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Proxy-Authorization: Negotiate TlRMTVNTUAADAAAAGAAYAH4AAAA...
Host: www.foo.com

HTTP/1.1 301 OK
Via: 1.1 PROXP01
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 160
Date: Wed, 24 Nov 2010 04:34:31 GMT
Age: 308
Location: http://www.foo.com/ClientInstall/
Content-Type: text/html; charset=UTF-8
Server: Microsoft-IIS/7.0
X-Powered-By: ASP.NET

GET http://www.foo.com/ClientInstall/ HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: www.foo.com

HTTP/1.1 200 OK
Via: 1.1 PROXP01
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 4621
Date: Wed, 24 Nov 2010 04:39:39 GMT
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.0
Cache-Control: private
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET

LOL, funny you mention this, I noted this just on Friday ;P

I have not seen any issues due to this, my response is still compressed.

I have also noted the several proxy requests. But I cannot for the life of me get GZip compression working over MS ISA or the new one with the long name server. It seems to just strip the request header, and the server does not even know about it. I am having a meeting with one of our network admins tomorrow to see if the issue is proxy related.

Update:

I should note that, I have no problem placing a Squid proxy between myself and server, but that was simply using basic authentication, and not the fancy challenge-response stuff Windows/ISA uses.

Update 2:

I forgot to mention, the Accept-Encoding header 'disappears' too when using a browser to access the site. This is why I think the real problem might not be specific to .NET. Testing continues.

Update 3:

I did a capture using FF3.6 going via ISA proxy and pretty much the same result. Going via ISA, only does a 'single' proxy authenticate, but the final response is still not compressed.

Given the high rate of hereditary bugs in MS products, I would have to side with ISA/FTMG being the culprit here.


Update 4 (solved):

Just had the meeting, and solved the issue (will post screenie with the appropriate setting later). So the hunch was correct (in my case at least).

I would strongly advise to look into ISA/FTMG configuration, if that is used in the network hosting the server.

Update 5 (solved, really):

Tried another setting, and that did the 'trick'. On the FTMG proxy, we had to enable HTTP compression for the 'External' network, as well as make sure the compressible content-types were correct (had to add application/soap+xml for WCF). This allows the proxy to do compression, and the actual web server does not, which may or may not be ideal, but at least it works. :)

I would strongly advise to look into ISA/FTMG configuration, if that is used in the network hosting the server.

Capture log:

Here is my trace hitting the site with IE8 via Forefront TMG proxy captured with Wireshark. If the bug is .NET specific, I would expect the result to be different, but from what I can see, it behaves the same.

Initial request:

GET http://www.foo.com/ClientInstall HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: www.foo.com

HTTP/1.1 407 Proxy Authentication Required 
  ( Forefront TMG requires authorization to fulfill the request. 
  Access to the Web Proxy filter is denied.  )
Via: 1.1 PROXP01
Proxy-Authenticate: Negotiate
Proxy-Authenticate: Kerberos
Proxy-Authenticate: NTLM
Proxy-Authenticate: Basic realm="PROXP01.bar.com"
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 4140  

Authenticate with proxy, but fails?:

GET http://www.foo.com/ClientInstall HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: www.foo.com
Proxy-Authorization: Negotiate TlRMTVNTUAABAAAA...

HTTP/1.1 407 Proxy Authentication Required ( Access is denied.  )
Via: 1.1 PROXP01
Proxy-Authenticate: Negotiate TlRMTVNTUAACAAAAB...
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 0     

Final sequence:

GET http://www.foo.com/ClientInstall HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Proxy-Authorization: Negotiate TlRMTVNTUAADAAAAGAAYAH4AAAA...
Host: www.foo.com

HTTP/1.1 301 OK
Via: 1.1 PROXP01
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 160
Date: Wed, 24 Nov 2010 04:34:31 GMT
Age: 308
Location: http://www.foo.com/ClientInstall/
Content-Type: text/html; charset=UTF-8
Server: Microsoft-IIS/7.0
X-Powered-By: ASP.NET

GET http://www.foo.com/ClientInstall/ HTTP/1.1
Accept: image/gif, image/jpeg, ...
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; ...
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: www.foo.com

HTTP/1.1 200 OK
Via: 1.1 PROXP01
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 4621
Date: Wed, 24 Nov 2010 04:39:39 GMT
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.0
Cache-Control: private
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文