处理大量文件 - 有任何库或实用程序可以提供帮助吗？
我想处理大量文件 - 想想视频转码，例如 youtube，您必须将所有内容转换为 FLV 格式或图像处理，您可以从大型 RAW 文件创建拇指。
是否有任何服务或图书馆可以帮助进行此类处理？ 我指的不是实际的转码，而是组织、启动任务、监视任务并以某种方式处理错误。
理想情况下在 Windows 上，但 Linux 也可以。
更新：我希望 utility/lib 能够在更高级别处理此类任务。 调度多个进程，以某种方式处理和报告任何故障等。
因此，在转码的情况下，我希望这样的库/实用程序获取要转码的文件列表，然后它将处理在多个线程/进程中启动转码。 您将能够定义成功完成任务后要做什么。 如果发生故障，您也可以这样做。
I'd like to process a large number of files - think video transcoding, like youtube, where you have to transform everything into FLV format or image processing where you create thumbs from large RAW files.
Is there any service or library that can help on such processing? I don't mean actual transcoding, but organizing, launching the tasks, monitoring them and handling errors somehow.
Ideally on windows but linux could also be fine.
Update: I'd like utility/lib to handle such tasks at higher level. Dispatch multiple processes, handle and report any failures somehow, etc.
So, in case of transcoding, I'd like such lib/utility to get a list of files to transcode and then it would handle starting the trancoding in multiple threads/processes. You would be able to define what to do on successful completition of the task. And you'd also be able to do this in case of a failure.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
如果我要这样做，我会创建一个 .NET 程序来监视传入文件的目录，然后根据它们的文件类型触发类似 http://ffmpeg.org/ 将视频转换为输出目录或使用类似 ImageMagick。
监视进程可能需要一些思考，尽管应该可以密切关注已创建的 shell 任务，或者至少检查输出目录中您期望创建的文件。
If I was going to do this I would create a .NET program to watch a directory for incoming files and then based on their file type fire off something like http://ffmpeg.org/ to convert the video into an output directory or for the images using something like ImageMagick.
Monitoring the processes might require some thought although it should be possible to keep an eye on a shell task that you have created, or at the very least check for a file in the output directory that you are expecting to have been created.
命令 shell（Windows 上的命令提示符、Unix 上的 sh/bash/csh 等）和脚本语言都可以轻松地对匹配某种模式的所有文件执行相同的操作。
例如，在Windows 2000左右：
for %f in (*.jpg) do...（例如调用imagemagick的convert生成缩略图）获取
forhelp, 在命令提示符下执行for /?如果您想利用并行性，例如一次分离六个以利用所有六个 CPU 核心，那么您可以使用 unix '
make' 程序，该程序也可在 Windows 上使用。 您必须创建一个 Makefile 来指定您想要执行的操作，并使用 --jobs switch用于并行性的 make 方法的替代方法是使用脚本语言。 许多脚本语言很容易支持同时运行多个子进程，例如Python的Popen()。
Command shells (the Command Prompt on Windows, sh/bash/csh etc on Unix) and scripting languages can all trivially do the same operations for all files matching some pattern.
For example, on Windows 2000 or so onwards:
for %f in (*.jpg) do... (e.g. call imagemagick's convert to generate a thumbnail)To get the
forhelp, at the command prompt dofor /?If you want to exploit parallelism, e.g. spin off six at a time to utilise all six of your CPU cores, then you can use the unix '
make' program, which is also available on Windows. You have to make a Makefile specifying what you want to do, and execute make with the --jobs switchAn alternative to the make approach for parallelism is to use a scripting language. Many scripting languages easily support running multiple child processes at once e.g. Python's Popen().