同时解码和显示多个 H264 视频的最快方法 C#
正如您可能从问题标题中猜测的那样，我们需要同时解码和显示多个（例如，八个）H.264 编码视频（并使它们保持所有时间同步，但这是另一个问题）。视频通常为 25 FPS，分辨率为 640x480。
在讨论问题的症结之前，我将提供一些背景知识。
该功能需要嵌入到相当大的 C# 3.5 (WinForms) 应用程序中。视频将占据应用程序中的矩形 - 托管代码需要能够指定每个视频的绘制位置及其大小。
我们用 C# 获取 H264 数据包，并将它们发送到本机 H264 解码器中以获取 YUV12 图像数据。
早期的尝试包括将 YUV12 图像转换为 RGB24 并将它们 BitBlt'ing 为从 C# 传递到本机代码中的 HWND。虽然功能正常，但所有 BitBlt'ing 都必须在 UI 线程上进行，这导致它在显示多个视频（在 2.6 GHZ core 2 duo 上）时陷入困境。
当前的尝试在启动时启动每个 CPU 核心一个线程，并在这些线程之间平衡视频的解码/显示负载。其性能令人震惊（我发现观看任务管理器比显示的视频更有趣）。在用户界面方面，它还有很多不足之处。
当我们开始从非 UI 线程绘制到 UI 线程（例如，停靠在 WinForms 控件中的面板）上创建的 HWND 的毫秒时，由于 WinForms 的非线程安全性，我们开始出现各种奇怪的行为。这导致我们在本机代码中创建 HWND 并绘制到这些 HWND，并使用 C# 提供应在屏幕坐标中绘制它们的矩形。
嘎！ CanOfWorms.Open()。
问题：当 C# 应用程序获得焦点时，它会跳转到 Z 顺序的前面并隐藏视频窗口。 解决方案：将视频窗口始终置于顶部。
问题：当用户切换到另一个应用程序时，视频窗口仍然位于顶部。 解决方案：检测 C# 应用程序的激活和停用，并相应地显示/隐藏视频窗口。
问题：用户说：“我希望在一台显示器上播放视频，而在另一台显示器上编辑 Word 文档！” 解决方案：告诉用户闭嘴，Word 无论如何都很糟糕。
问题：我被解雇了。
。
我想问题的关键是我们在非 UI 线程上创建了 HWND，并且我们想要“模拟”那些嵌入到 C# 应用程序中的 HWND
有什么想法/建议吗？我完全出去吃午饭了吗？
如果存在一种完全不同的方法，我非常愿意采取一种完全不同的方法（这个项目需要大量的学习 - 中奖的可能性比我在路上的每一步都选择最佳方法的可能性更大）。
As you might surmise from the question title, we are required to decode and display multiple (e.g., eight) H.264 encoded videos at the same time (and keep them all time synchronized, but that's another question for another time). The videos are usually at at 25 FPS with a resolution of 640x480.
I'm going to provide a bit of background before I get to the crux of the problem.
The feature needs to be baked into a fairly large C# 3.5 (WinForms) application. The videos will occupy rectangles in the application - the managed code needs to be able to specify where each video is drawn as well as it's size.
We get the H264 packets in C# and fire them into a native H264 decoder to get YUV12 image data.
An early attempt consisted of converting the YUV12 images to RGB24 and BitBlt'ing them to a HWND passed into the native code from C#. While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos (on a 2.6 GHZ core 2 duo).
The current attempt spins up one-thread-per-cpu-core on startup and load balances the decoding/displaying of videos across these threads. The performance of this is mind-blasting (I find watching task manager much more interesting than the videos being displayed). UI-wise, it leaves a lot to be desired.
The millisecond we started drawing to an HWND created on the UI thread (e.g., a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.
Gah! CanOfWorms.Open().
Problem: When the C# application receives focus, it jumps to the front of the Z-Order and hides the video windows.
Solution: Place the video windows Always On Top.
Problem: When the user switches to another application, the video windows are still on top.
Solution: Detect activation and deactivation of the C# application and show/hide the video windows accordingly.
Problem: User says, "I want my videos playing on one monitor while I edit a Word document in the other!"
Solution: Tell user to shut up and that Word sucks anyways.
Problem: I get fired.
etc. etc.
I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.
Any thoughts/suggestions? Am I completely out to lunch here?
I'm more than open to taking a completely different approach if one exists (This project required a lot of learning - winning the lottery would have a greater likelihood than me having picked the best approach at every step along the road).
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
我喜欢之前发布的 DirectShow 答案，但我想根据您的问题摘录添加一个可能更容易实现的附加选项：
我的想法是从该代码开始，并使用 Visual Studio 2010 的异步 CTP，即 当前可用并包含上线许可证。从那里开始，修改现有代码以提高响应速度应该相对简单：只需在几个地方添加await和async关键字，其余代码应该基本不变。
I like the DirectShow answer posted earlier, but I wanted to include an additional option that might be easier for you to implement, based on this excerpt from your question:
My idea is to start from that code, and use the Async CTP for Visual Studio 2010 that is currently available and includes a go-live license. From there it should be a relatively simple to modify this existing code to be more responsive: just add await and async keywords in a few places and the rest of the code should be largely unchanged.
忘记 BitBlt-ing 并执行以下操作：
此外，我想您将能够将原始 YUV12 放入缓冲区，因为 VMRenderer 能够直接显示它们。
使用 DirectShowNet 库。
编辑：
是的，顺便说一句，如果视频位于同一“画布”上，您可以对渲染器使用相同的技术并仅创建一个大窗口，然后“手动”移动解码的视频矩形并将它们放入帧缓冲区缓冲区中。
另一种编辑：
BitBlts 始终是序列化的，即它们不能并行运行。
Forget about BitBlt-ing and do this:
In addition, I guess that you'll be able to put raw YUV12 into the buffer, as VMRenderer is able to display them directly.
Use DirectShowNet library.
EDIT:
And yes, BTW, if the videos are on the same 'canvas', you can use same technique with renderer and create only one large window, then shift decoded video rectangles 'by hand' and put them into the framebuffers buffer.
YET ANOTHER EDIT:
BitBlts are ALWAYS serialized, i.e. they can't run in parallel.
什么样的古怪行为？
如果您的意思是闪烁或绘图延迟，您是否尝试过 lock() 面板或任何其他类以实现线程/绘图同步？
再说一次：当您将数据发送到解码器、接收图像、转换它然后使用 OnPaint 处理程序绘制它时，确切的问题是什么。 （设置一个以 25fps 循环的不同线程，然后调用 panel1.Invalidate()）
不要那样做。尝试在您的 C# 应用程序中绘制接收到的数据。
一般来说，我不建议混合本机代码和 C#。在本机代码中使用 h264 解码器是这里唯一的例外。
使用您的线程解码视频数据包（就像您已经做的那样），然后让一个线程循环并调用 Invalidate（如上所述）。然后为要在其中显示视频的每个面板设置一个 OnPaint 处理程序。在此处理程序中获取最新的视频图片并绘制它（例如图形）。
我希望这有帮助，但也需要有关该问题的更多信息......
What kind of funky behavior?
If you mean flickering or drawing delay, have you tried to lock() the panel or any other class for thread/drawing synchronisation?
Again: Whats the exact problem when you send the data to the decoder, receive a image, convert it and then draw it with an OnPaint handler. (Setup a different thread that loops at 25fps, call panel1.Invalidate() then)
Don't do that. Try to draw the received data in your c# application.
In general, I wouldn't reccomend mixing native code and c#. Having the h264 decoder in native code is the only exception here.
Use your threads to decode the video packets (as you already do) then have one thread that loops and calls Invalidate(as said above). Then have an OnPaint handler for each panel you are displaying a video in. In this handler get the most recent video picture and draw it (e.Graphics).
I hope this helps, but also need more information about the problem...