在服务器端处理非常大的http上传和下载的最佳方法
我有一个即将开展的项目，需要处理来自浏览器的非常大的上传（经典输入类型=“文件”或Java Applet），并寻找在服务器上完成这项工作的最佳工具。
这些是我需要的东西：
- 服务器上的内存消耗低，
- 能够将文件保存在服务器上的最终目的地（无需复制文件），
- 不会阻止网络服务器完成的其他重要任务，
- 可以很好地处理高达 2 GB 的文件
- 文件授权（权限将在应用程序中给出）
我对于使用什么技术仍然有一定的自由度，所以我想得到一些建议，以便能够选择服务器上最好的技术来处理此任务：
- ASP。网 ？
- 爪哇？
- 亚马逊S3？
- 其他选择？
我更习惯 Microsoft Stack，但愿意在必要时进行更改：如上所述，我只是在寻找最适合这项工作的工具。
谢谢 ！
更新 ： 服务器端是我对这个问题真正感兴趣的部分，而不是客户端。
看起来这可能是微不足道的，但是当您开始深入研究时，您会发现 .NET 的 4 Mb 限制，下载使用大量内存，可能会阻塞其他线程（当您对线程数量有限制时，如果一个线程可以在通过互联网上传/下载 2 GB 文件的持续时间内执行：那么这不会很好地扩展，不是吗？）等等。
I've an upcoming project where I will need to handle very large uploads from browsers (either the classic input type="file" or a Java Applet), and looking for the best tool to do the job on the server.
Theses are the things I need :
- low memory consumption on the server
- ability to save the file in its final destination on the server (no copying the file around)
- no blocking of other important tasks done by the webserver
- good handling of files up to 2 gb
- authorization of files (permissions would be given in the app)
I still have some latitude on what technology to use so I would like to have some advice in order to be able to choose the best technology on the server to handle this task :
- ASP.NET ?
- Java ?
- Amazon S3 ?
- Other choices ?
I'm more used to the Microsoft Stack, but willing to change if necessary : as told above, I'm just looking for the best tool for the job.
Thanks !
Update :
The server side is the part I'm really interested in for this question, not the client side.
It looks like it may be trivial, but when you start to digg a bit you see 4 Mb limitations with .NET, downloads that use a lot of memory, that CAN block other threads (when you have a limit on the number of threads, and if a thread can execute for the duration of 2 Gb file upload/download over the internet : well this ain't gonna scale very well, will it ?), etc.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（7）
您将需要：
我刚刚完成了一个应用程序 < em>完全一样；我会使用 Silverlight (
WebRequestasync)、ASP.NET (IHttpHandler/IHttpAsyncHandler) 和 SQL Server 2005 (UPDATETEXT) code>/READTEXT) 用于文件存储。更新：关于 ASP.NET 服务器端代码：
ASP.NET 默认配置将允许每个处理器 100 个线程；
IHttpAsyncHandler不会阻止您的进程，您可以将文件内容直接写入context.Response.OutputStream。对于上传，您还将发送多个数据块，但在多个 HTTP 连接中；虽然这可能会因 HTTP 标头而带来一些过热，但在我的测试中效果很好。
You'll need:
I just finished an application exactly like that; I'd use Silverlight (
WebRequestasync), ASP.NET (IHttpHandler/IHttpAsyncHandler) and SQL Server 2005 (UPDATETEXT/READTEXT) for file storage.UPDATE: About ASP.NET server-side code:
ASP.NET default config will allow 100 threads per processor;
IHttpAsyncHandlerwon't block your process and there you can write your file content directly tocontext.Response.OutputStream.For upload, you'll also send several data chunks, but in multiple HTTP connections; while this can bring some overheat due HTTP headers, works very well in my tests.
将输入直接写入磁盘而不是内存。
用Java术语来说，使用FileOutputStream/BufferedOutputStream。
参见上文。
每个请求都在自己的线程中运行，因此无需担心。这仅取决于您如何编码。
直接将文件写入磁盘时不会出现问题。
用 Java 术语来说，您可以使用 Apache Commons FileUpload API 来实现此目的。
不确定您所说的授权级别。磁盘文件系统级别？ Web应用程序级别？客户级别？
Write input directly to disk instead of to memory.
In Java terms, use FileOutputStream/BufferedOutputStream.
See above.
Each request runs in its own thread, so there's nothing to worry about. It just depends on how you code it all.
Non-issue when writing file to disk directly.
In Java terms, you can use Apache Commons FileUpload API for this.
Not sure which level of authorization you're talking about. Disk file system level? Webapplication level? Client level?
在客户端，通过 HTTP POST 的
input type="file"有其缺点 - 特别是，它无法压缩上传（可能不是问题），也无法恢复传输（这在以下情况下可能会很痛苦）： 1000 MB 上传在 990 MB 时失败）。 SWFUpload 虽然在其他方面很出色，但它依赖于浏览器的 HTTP POST 实现。我可能会在客户端上使用 Java 小程序 - 这将允许建立连接并在上传之前检查必要的权限；尽管该路径也有其问题：
也提供了一个选项来回退到普通的旧 HTTP POST。
服务器端几乎可以用任何东西编写，只要您可以在数据到达时对其进行处理（即不要等到拥有整个文件）。
On the client side, the
input type="file"through HTTP POST has its shortcomings - notably, it's unable to compress uploads (probably not an issue) nor can it resume transfers (this can be painful when 1000 MB upload fails at 990 MB). SWFUpload, although it's great in other aspects, relies on the browser's HTTP POST implementation.I'd probably go with a Java applet on the client - this would allow to establish the connection and check for necessary permissions prior to uploading; although that path has its problems too:
also give an option to fall back to plain old HTTP POST too.
Server-side can be written in pretty much anything, as long as you can process the data as they arrive (i.e. don't wait until you have the whole file).
您可以使用 ASP.NET 中的异步文件上传来完成此操作。用flash将其呈现给用户。
http://swfupload.org/node/47
You can do it using Asynchronous file upload in asp.net. With flash to present it to the user.
http://swfupload.org/node/47
您可以使用uploadify。我之前已经使用过几次，它总是适合我的需求。它是一个异步文件上传器，如果需要，可以使用闪存一次上传多个文件。
You could use uploadify. I have used this several times before and it has always suited my needs. It is an asynchronous file uploader that uses flash to allow for multiple files to be uploaded all at once if desired.
您可以使用 BITS 进行上传：
http://www.simple-talk.com/dotnet/.net-tools/using-bits-to-upload-files-with-.net/
You can use BITS for uploading:
http://www.simple-talk.com/dotnet/.net-tools/using-bits-to-upload-files-with-.net/
根据我的经验，其他一些快速注释...
它应该工作的方式是java小程序将文件分成可管理的块并从字节创建散列。一旦服务器接收到一个块，它应该将接收到的字节的哈希值与 java applet 提供的哈希值进行比较。
如果哈希值不匹配，请重试该块。如果它们匹配，则继续下一个块。然后使用工具将所有块重新组合在一起。
A few other quick notes based on my experience...
The way it should work is the java applet breaks the file into manageable chunks and creates a hash from the bytes. Once the server receives a chunk, it should compare a hash of the bytes received with the hash provided by the java applet.
If the hashes don't match, retry the chunk. If they do match, move on to next chunk. Then use a tool to bring all the chunks back together.