如何使用Delphi写入超过物理RAM的巨大JPEG?
问题就在这里。我有一大组 512x512 像素的 JPEG 图块作为常规 jpg 文件。
我写了一个软件,它可以做很多事情,并且最后需要将所有这些文件拼接成一个巨大的 JPEG。
首先,我不想使用 ImageMagick 来执行此操作,而是在我的软件中执行它!
在Delphi中,不可能将JPG文件复制到另一个JPG画布上,因此必须首先创建TBitmap,然后将图块复制到TBitmap画布上,然后将TBitmap转换为jpeg图片并保存到文件中。
当生成的文件尺寸太大(例如 20 000 x 20 000 像素)时,就会出现问题。当我调用 TBitmap.SetSize 时,我自然会收到错误(内存不足或类似的错误)。
我在同一台机器上使用 Photoshop 进行了一些测试,并能够创建一个复杂的(非空白)30 000 x 30 000 文件并将其保存为 JPEG。
所以问题是,我怎样才能做同样的事情呢?有什么方法可以通过将结果直接写入磁盘或使用其他一些技巧来拼接所有这些 JPEG?...
尽管 20k x 20k 像素看起来足够大,但该值仅适用于我的机器(4 GB 内存),因此较小的内存对软件的限制甚至更大!
谢谢
编辑:澄清一下:
我想要的是找到一种方法来拼接这些小 JPG 图像并写入大图像,而不将大图像保留在 RAM 中。显然,位图流可以直接在磁盘上读/写(不确定),但这会导致文件非常大。因此,如果 JPG 格式不允许这样做,任何其他压缩格式(如 TIFF 或 PNG)都可以。我还想避免过多的重新压缩,以免失去(已压缩的)初始 JPG 质量。
因此,完美的解决方案是一种直接读取小文件并以某种方式写入大文件的方法。切片的尺寸为 256x256 或 512x512,以防有助于 JPEG 压缩内容的对齐。
Here is the problem. I have a large set of 512x512 pixels JPEG tiles as regular jpg files.
I have written a piece of software that does a bunch of things and does need to stitch all those files into a single huge JPEG at the end.
First of all, I do NOT wish to use ImageMagick to do this, but perform it inside my software !
In Delphi it is not possible to copy a JPG file onto another JPG canvas, so a TBitmap must be created first, then the tiles are copied onto the TBitmap canvas and then the TBitmap is converted to a jpeg picture and saved to a file.
The issue arises when the resulting file dimensions are too big (like 20 000 x 20 000 pixels). When I call TBitmap.SetSize I naturally get an error (out of memory or something like this).
I made some tests using Photoshop on the very same machine and was able to create a complex (non blank) 30 000 x 30 000 file and save it to JPEG.
So the question is, how could I perform the same thing ? Fing some way to stitch all those JPEGS by writing the result directly to the disk or use some other trick ?...
Even though 20k x 20k pixels seems big enough, this value does only apply to my machine (4 GB ram) so a smaller amount of ram would be even more limiting to the software !
Thanks
Edit : To clarify :
What I would like is to find a way of stitching those small JPG images and write the large one without keeping the large image in RAM. Apparently a bitmap stream read/write is possible directly on disk (not sure) but this would result in a VERY large file. So, if the JPG format does not allow to do this, any other compressed format like TIFF or PNG would do. I would also like to avoid too much recompression not to lose the (already compressed) initial JPG quality.
Hence the perfect solution would be a way to directly read the small files and write into the large one somehow. The dimensions of the tiles are 256x256 or 512x512 in case it would help for some alignment on JPEG compression stuff.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(8)
谢谢大家!
实际上,答案和可能的解决方案是像 Photoshop 那样进行,即将图块中的位图流写入磁盘上的大 bmp 文件(例如 20 000 x 30 000 文件将为 2.4 Gb),然后使用NativeJpg 库通过逐条输入位图数据来将这个大位图转换为 jpg,每个条带高 8 像素。
还可以缝合单行图块(512 像素高),然后将其按 8 × 8 馈送到 NativeJpg 库,然后继续处理下一行图块!
Erik Turner 的一些示例代码:
NativeJpg 库: http://www.simdesign.nl/nativejpg.html< /a>
Thanks to everyone !
Actually, the answer and possible solution is to proceed as Photoshop does, that is write a bitmap stream from the tiles to a big bmp file on the disk (for example a 20 000 x 30 000 file would be 2.4 Gb) and then use the NativeJpg library to convert this big bitmap to a jpg by feeding the bitmap data stripe by stripe, each of them being 8 pixels high.
It would also be possible to stitch a single line of tiles (512 pixels high) and then feed it to the NativeJpg library 8 by 8 and then move on to the next line of tiles !
Some sample code by Erik Turner :
NativeJpg library : http://www.simdesign.nl/nativejpg.html
这是一个相当艰难的领域,正如 @Peter 已经说过的那样,Photoshop 人员从那时起就一直在这个领域1990 年。您可能无法使用编程语言的内置 JPG 解码库,因为它们可能会在 RAM 中加载(并解压)整个图像。
我建议寻找处理此问题的外部库 - 我不知道是否有免费的像样的库,但很可能是这样。
This is a pretty tough field, and as @Peter already says, the Photoshop folks have been at this since 1990. You will probably not be able to work with your programming language's built-in JPG decoding libraries as they will likely load (and unpack) the whole image in the RAM.
I'd recommend looking for external libraries that deal with this - whether there are decent ones available for free I don't know, but it may well be the case.
Delphi 中图像的最大大小(至少在以前的版本中)更多地取决于 Windows 图形驱动程序而不是系统内存量
对此进行了一些实验:
EFG 计算机实验室
The maximum size of images in Delphi (at least in previous versions) was more dependent on the windows graphics drivers than on the amount of system memory
Some experiments on this:
EFG's computer lab
与许多其他面向媒体的程序一样,Photoshop 必须长期处理处理大于主内存的文件的问题。对于大照片,一种技术是平铺以仅处理图像的一部分。
实际上,这比简单的剪切和粘贴更复杂(双关语)。
我不是 Delphi 程序员,但让我担心的是,当您创建图像时内存不足时,当您尝试使用该图像时是否会发生同样的情况?
Photoshop like many other media oriented programs had to deal with the issue of working on files bigger than main memory for a long time. For large photos one technique is tiling to only work a a part of the image.
In practice this is more convoluted (pun intended) than simply cut and paste.
I am no Delphi programmer, but something which worries me is that when you run out of memory creating the image, will the same not happen when you try to use that image?
在进行这样的繁重工作之前,我求助于 http://www.graphicsmagick.org/
这里还有一组帕斯卡单位 http://graphics32.org/ 但它们非常数学和复杂(并且因此,我没有让他们工作),但也是为了艰苦的工作而建立的。
Fore heavy duty work like this, I resort to http://www.graphicsmagick.org/
There're also a set of Pascal units here http://graphics32.org/ but they're pretty mathematical and complex (and I therefor haven't gotten them to work), but also built for the hard work.
您可能需要实现一个自定义类来处理此问题。
在视频内存中,图像(或屏幕缓冲区)是线性数组。水平行按顺序存储,每个像素对应于
y * width + x
处的数组偏移。因此,在 320x200 图像中,5,2 处的像素将位于数组索引 5*320+2-1 或 1601 处。回到硬件加速之前的日子,您可以 malloc() 一个屏幕大小的缓冲区并以数学方式执行绘制纹理、形状、灯光效果等操作,然后将缓冲区 BLT 输出到视频 RAM。
在您的情况下,您可以使用内置位图和图像类来处理适合内存的较小图像,然后将其像素数据复制到一个大数组或一系列数组中(我忘记虚拟内存是否允许您创建缓冲区>物理RAM的大小)。然后,使用直接在该阵列上工作的 JPEG 库(与计算机上安装的 RAM 大小无关),您应该能够将阵列输入到库中并将内容保存到磁盘上。 LZW 压缩非常简单,我希望它们能提供大量有关在网络上手动实现 JPEG 压缩的材料。
对此需要注意的是:如果您使用的是 32 位操作系统,则您的地址空间应限制为 4GB。我能想到的解决这个问题的唯一方法是创建较小的缓冲区(例如,一次一行),用该行对应的像素数据部分填充数据您要缝合的图像,保存它并循环,直到覆盖整个图像区域。
我希望这是清楚的。祝你好运!
You may need to implement a custom class to handle this.
In video memory, an image (or a screen buffer) is a linear array. Horizontal rows are stored sequentially, and each pixel corresponds to the array offset at
y * width + x
.Thus, in a 320x200 image, the pixel at 5,2 would be at array index 5*320+2-1, or 1601. Back in the days before hardware acceleration, you'd malloc() a buffer the size of your screen and mathematically perform an operations like drawing textures, shapes, lighting effects etc, then BLT the buffer out to video RAM.
In your case, you could use the built-in bitmap and image classes to work with smaller images that fit in memory, then copy their pixel data into a large array or series of arrays (I forget if the virtual memory will allow you to create buffers > the size of physical RAM). Then, using a JPEG library that works directly on that array (which is independent of the size of RAM installed on the machine), you should be able to feed the array into the library and have it save the contents out to disk. LZW compression is pretty straightforward and I would expect their to be a lot of material on manually implementing JPEG compression on the web.
One caveat to this: If you're using a 32-bit OS, your address space should be limited to 4GB. The only way I can think of off the top of my head to get around this would be to create smaller buffers (say, one row at a time), throw fill the data with the part of the pixel data that the row corresponds to in your to-be-stiched images, save it, and loop until you've covered your entire image area.
I hope that's clear. Good luck!
你的问题也让我想到了电子表格。当然只是人烟稀少。您可以尝试查看一些图像压缩库,它们可能会给您一些想法。
您还可以做的就是看看其他人是如何做的。我注意到 PixeLook 开发组有 他们的 PixeLook 库,它们是 Delphi 的组件6 用于创建图像和数据处理应用程序。
他们声称大型图像和数据矩阵很容易处理,在他们的屏幕截图页面上,他们展示了 5200 x 5200 图像 (26 MB) 的显示,并表示他们的测试也是在图像大小高达 220 MB 的情况下进行的。
如果您确实需要直接在应用程序中进行大图像处理,这个软件包可能适合您,只需 50 美元。如果它很接近但不太正确(我不知道它是否会加入 jpeg),那么您可能会考虑以 299 美元购买源代码,看看它的作用,并扩展它。
免责声明:我与该公司没有任何关系。
Your problem also makes me think of spreadsheets. which of course only are sparsely populated. You may try looking at some image compression libraries that might give you some ideas.
What you could also do is see how someone else does it. I note that the PixeLook Development Group has their PixeLook library which are components for Delphi 6 for creating image and data processing applications.
They claim that large images and data matrices are easily handled and on their screenshot page, they show the display of a 5200 x 5200 image (26 MB) and they say their tests were also performed with the image size up to 220 MB.
If you really need large image processing directly in your application, this package might work for you for $50. If it's close but not quite right (I don't know if it will join jpegs), then you might consider buying the source for $299, seeing what it does, and extending it.
Disclaimer: I have no connection with this company.
部分外部软件/内部调用解决方案怎么样? IJG(独立 JPEG 组织)有一个出色的命令行工具 jpegtran,我在查看器中使用它进行无损旋转。在您自己的代码中使用 CreateProcess、WaitForsingleObject 来使之看起来像您自己的代码是没有问题的。您甚至可以将可执行文件打包到资源中并临时提取它,
因此它们还具有 jpegjoin 实用程序(可以在 http:// jpegclub.org/jpegtran/),可以以相同的方式使用。更新:该实用程序用于无损连接,因此需要更小的内存/磁盘占用空间
What about partial external software/internal call solution? IJG (Independent JPEG Group) has an excellent command-line tool jpegtran that I used in my viewer for lossless rotation. There's no problem to use CreateProcess, WaitForsingleObject inside your own code to make to look like your own code. You even can pack the executables inside your resource and extract it temporary
So they also has jpegjoin utility (find it at http://jpegclub.org/jpegtran/) that can be used the same way. UPDATE: this utility is for lossless joining, so require much smaller memory/disk footprint