服务器负载与大规模上传的可扩展性
我想将用户的数百万个音频项目上传到我的服务器。当前的应用程序旨在提供内容，对其进行转码，最后通过 ftp 发送到存储服务器。我想知道：
应用程序服务器在扩展到更多服务器（以承载网络应用程序负载）后是否可以承担用户评论、上传、转码等巨大任务？
如果上述问题的答案是肯定的，那么它是正确且最好的方法吗？因为一个好的架构是将转码发送到存储服务器等待完成工作并向应用服务器发送响应，但同时它具有更多的复杂性和不安全性。
此类网站常用的方法是什么？
如果我将上传和转码作业发送到存储服务器，它在长期可扩展性方面是否与企业存储技术兼容？
5-当前应用程序基于PHP。是否可以将 tmp 文件夹移动到另一台服务器以克服上传过载？
感谢您对 tmp 文件夹问题 5 的回答。我的意思是 Apache 中的 tmp 文件夹。我知道在移动到最终存储目的地（例如：存储服务器或任何解决方案）之前所有上传的文件都存储在apache的tmp文件夹中。我想知道这是否是 apache 的规则，所有上传的文件都应该首先位于应用程序服务器中，那么我如何控制、扩展和重定向这种大量的存储负载到临时存储或服务器？我的意思是服务器或存储解决方案作为 appche 的 tmp 文件夹，在发送到最终存储位置之前只是上传文件的访客。我研究和设计了有关数据库扩展、存储、负载平衡、内存缓存等的所有内容，但这是我未解决的问题之一。用户新到达主服务器的文件将在缩放架构中发生在哪里？对此的常见解决方案是什么？ （在一盒解决方案中，所有文件都将临时存储在 appche 的 tmp 目录中，但对于大量内容和扩展系统？）。 问候
I want to upload millions of audio items by users to my server. The current app has designed to give the contents, transcode them and finally send by ftp to storage servers. I want to know:
Does the app server can bear the enormous tasks by user like commenting, uploading, transcoding after scaling to more servers (to carry web app load)?
If the answer of above question is yes, is it correct and best approach? Because a good architecture will be to send transcoding to storage servers wait for finishing the job and sending respond to app server but at the same time it has more complexity and insecurity.
What is the common method for this type of websites?
If I send the upload and transcoding job to storage servers does it compatible with enterprise storage technologies in a long term scalability?
5- The current App is based on PHP. Is it possible to move tmp folder to another servers to overcome upload overload?
Thanks for answer, for tmp folder question number 5. I mean the tmp folder in Apache. I know that all uploaded files before moving to final storage destination (eg: storage servers or any solution) are stored in tmp folder of apache. I was wondering if this is a rule for apache and all uploaded files should be located first in app server, so how can I control, scale and redirect this massive load of storage to a temporary storage or server? I mean a server or storage solution as tmp folder of appche to just be guest of uploaded files before sending to the final storages places. I have studied and designed all the things about scaling of database, storages, load balancing, memcache etc. but this is one of my unsolved question. Where new arrived files by users to main server will be taken place in a scaled architect? And what is the common solution for this? (In one box solution all files will be temporary in the tmp dir of appche but for massive amount of contents and in a scaled system?).
Regards
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
您可能想看看 Viddler 架构：http://highscalability.com/blog/2011/5/10/viddler-architecture-7-million-embeds-a-day-and-1500-reqsec.html
You might want to take a look at the Viddler architecture: http://highscalability.com/blog/2011/5/10/viddler-architecture-7-million-embeds-a-day-and-1500-reqsec.html
由于我觉得我无法回答这个问题（我想添加评论，但我的文字太长），一些想法：
如果您正在创建如此大的系统（正如听起来的那样），您应该有一些性能测试以查看有多少并发连接/上传，...无论您的架构可以处理什么。正如我总是说的：如果您不知道：“不，它不能”。
我认为处理重负载的最佳方法（这是：大量上传，需要来自应用程序服务器的大量阻塞线程（->这意味着，我不会使用应用程序服务器来处理文件上传）。异步执行所有繁重的操作（转码）（例如，对上传的文件进行排队，然后处理它们）。在任何情况下，应用程序服务器都不应该等待转码系统的响应 -> 只是告诉用户，他的文件正在处理。将被处理并在完成后向他发送消息（或其他内容）。您可以使用 gearman 之类的东西。
我会搜索现有的架构，这些架构也必须处理大量上传/转换（例如 flickr），只需转到幻灯片共享并搜索“flickr”或“可扩展的网络架构”
我不太明白这一点 - 但我会使用服务器基于他们的任务（例如应用程序服务器、数据库服务器、传输服务器、存储……） - 每个服务器应该做什么，他能做得最好。
当您说 tmp 文件夹时，我恐怕我不知道您在说什么。
祝你好运
Since I don't feel I can answer this (I wanted to add a comment, but my text was too long), some thoughts:
If you are creating such a large system (as it sounds) you should have some performance tests to see, how many concurrent connections/uploads,... whatever your architecture can handle. As I always say: If you don't know it: "no, it can't ".
I think the best way to deal with heavy load (this is: a lot of uploads, requiring a lot of blocked Threads from the appserver (-> this means, I would not use the Appserver to handle the fileuploads). Perform all your heavy operations (transcoding) asynchronously (e.g. queue the uploaded files, processess them afterwards). In any case the Applicaiton server should not wait for the response of the transcoding system -> just tell the user, that his file are going to be processed and send him a message (or whatever) when its finished. You can use something like gearman for that.
I would search for existing architectures, that have to handle a lot of uploads/conversion too (e.g. flickr) just go to slideshare and search for "flickr" or "scalable web architecture"
I do not really understand this - but I would use Servers based on their tasks (e.g. Applicaiton server, Database serversm, Transconding servers, Storage,...) - each server should do, what he can do best.
I am afraid I don't know what you are talking about when you say tmp folder.
Good luck