使用 JNA 从 Java 进行本机 C 调用需要占用大量内存
我有一个本机 C 库，可以在非常大的数据集（数百 mb 到 GB 的数量级）上运行一些算法。这是使用 JNA 从 Java 框架内调用的。 Java 加载数据并通过 JNA 将其传递到 C 库。
问题是，似乎使用了过多的内存。对于一个数据集，在 Java 端完成所有加载后，该过程使用了大约 3.0GB，而 C 库正在使用 2.0GB（根据内部内存管理确定）。但是，一旦调用 C 库，该过程最终会达到约 9.5GB！
那么具体的问题是：
Java 和 C 端之间没有重叠吗？也就是说，JNA 是否会生成 Java 数据的 C 有效副本（顺便说一句，所有 int 和 double 数组）并将其传递给本机库，而不是传递给 Java 中包含数据的相同块？
即使假设没有重叠，并且本机库正在使用 JVM 中包含的数据的副本，那么额外的 4.5GB 来自哪里？这大约使该进程占用的系统内存量增加了一倍，我无法想象它都去哪里了。关于 JNA 这些方面的文档似乎非常有限，但我想知道比我更熟悉 JNA 的人是否知道它为什么使用这么多内存，以及我是否以及如何能够减少其占用空间。
编辑：启用 JNA 的 Java 类如下所示：
public interface MyNativeLibrary extends Library {
MyNativeLibrary INSTANCE = (MyNativeLibrary) Native.loadLibrary(
"native_library", MyNativeLibrary.class);
int native_library_function(int num_inputs, int inputs[], int max_num_outputs, int preallocated_outputs[]);
}
在这种情况下，本机函数的返回值将是返回的输出数或错误代码。 C 接口用 int32_t 指定，以确保大小匹配。
I have a native C library that runs some algorithms on very large datasets (on the order of hundreds of mb to gb). This is being called from within a Java framework using JNA. Java loads the data and passes it through JNA to the C library.
The problem is, there seems to be an inordinate amount of memory being used. For one dataset, the process uses about 3.0gb after all the loading is finished on the Java end, and 2.0gb is in use by the C library (as determined using in-house memory management). But the process, once the C library is called, ends up maxing out at about 9.5gb!
Specific questions, then:
Is there no overlap between the Java and C side? That is to say, does JNA produce a C-valid copy of the Java data (all int and double arrays, by the way) and pass that to the native library instead of the same blocks that contain the data in Java?
Even presuming there is no overlap, and the native library is using a copy of the data contained within the JVM, where is that extra 4.5gb coming from? That about doubles the amount of system memory taken up by the process, and i can't imagine where it's all going. The documentation on these aspects of JNA seems very limited, but I wonder if someone more familiar than I am with JNA might know why it's using so much memory, and if and how I might be able to reduce its footprint.
EDIT: The JNA-enabled Java class looks like this:
public interface MyNativeLibrary extends Library {
MyNativeLibrary INSTANCE = (MyNativeLibrary) Native.loadLibrary(
"native_library", MyNativeLibrary.class);
int native_library_function(int num_inputs, int inputs[], int max_num_outputs, int preallocated_outputs[]);
}
In this case, the return value of the native function would be the number of outputs returned, or an error code. The C interface is specified with int32_t, to make sure the sizes match.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
如果只想在本机端分配内存，请使用 Memory 类并使用 java.nio 访问内存.ByteBuffer （内存有一个 getByteBuffer() 方法）。您可以将整数数组作为 JNA 指针传递给分配的内存。这应该可以为您节省大量空间并提高性能。只需确保您在 Java 端管理本机内存资源即可。
If you want to allocate memory only on the native side, use the Memory class in JNA and access the memory using java.nio.ByteBuffer (Memory has a getByteBuffer() method). You can pass your arrays of integers as JNA Pointers the allocated memory. This should save you quite a bit of space and improve performance. Just make sure you manage the native memory resources on the Java side.