寻找“做所有事情的缓冲区”在java中-决定使用Netty

发布于 2024-12-09 18:49:57 字数 841 浏览 7 评论 0原文

一些背景:

我正在开发一些java包,这些包需要获取数据,将其分割并在线分发到许多服务器。根据用户代码(使用我的包的人),我会知道要使用哪种缓冲区。我已经看到 Netty 项目,其中包含一些有趣的概念以及网站中建议的其他解决方案,但并不完全是我正在寻找的。我想我可以实现我的要求,但我更喜欢重用。

实际问题:字节缓冲区要求

将实现与接口分开。 两个实现之间的变化。

固定大小:

  1. 直接内存使用。
  2. 分发时零复制 IO。

成长尺寸

  1. 可以成长到一定尺寸。
  2. 在重新分配和使用的内存量之间保持平衡。

更新 1:由于还有很多其他好处,决定使用 Netty 项目库。

我想提供一些有关我的系统的详细信息。我需要在整个服务器网络中分发大量数据。我对缓冲区大小的了解非常少。有时可以是固定尺寸,有时可以是未知尺寸模式。我希望能够建立一种双赢的机制。当我收到最终缓冲区进行检查时,我确实有一个定义的事件。

我知道缓冲区直接映射到内存时可以表现得更好。 Netty 为我提供了这个,但我无法让这个缓冲区增长。我有可以增长的动态缓冲区 -Netty 也提供了这一点。

有几个问题:

  1. 我是否应该将缓冲区从动态​​复制到直接,以防缓冲区发生更改?有没有办法切换模式?

  2. 您如何看待实现一个类,该类保存 ChannleBuffers 数组并公开与 ChannleBuffer 相同的接口,同时包含“内部”缓冲区数组,根据需要分配新的直接内存。

  3. 您还有其他解决方案吗?

Some background:

I working on some java packages that need to take data, divide it and distribute to many Servers online. According to the user code (person using my packages), I will know which kind of buffer to use. I have seen The Netty Project that has some interesting concepts in mind and other solutions suggested here in the site, but not exactly what I'm looking for. I guess I can implement my requirements, but I like reusing better.

The actual Issue: Byte Buffer Requirements

Separate implementation from interface.
Change between the two implementation.

Fixed Size:

  1. Direct memory usage.
  2. Zero Copy IO on distribution.

Growing Size

  1. Can grow to up to a certain Size.
  2. Keeps balance between reallocation and amount of memory in use.

Update 1: Decided to go with the Netty project library due to a lot of other benefits.

I wanted to give some details about my system. I need to distribute large amounts of data throughout a network of servers. I have very low amount of knowledge regarding the size of my buffers. Sometimes it can be fixed sizes, sometimes it can be in unknown size mode. I want to be able to develop a mechanism that enjoys the benefit of both worlds. I do have a defined event when I receive the final buffer for inspection.

I understand that buffers can perform better when they are directly mapped to memory. Netty provides this for me, but I can't make this buffer grow. I have dynamic buffer which can grow -Netty provides that too.

Couple of questions:

  1. should I copy the buffer to from dynamic to direct in case it is changed? Is there a way to switch modes?

  2. What do you think of implementing a class that holds an array of ChannleBuffers and exposes the same interface like ChannleBuffer while containing "inside" an array of buffers allocating new direct memory as needed.

  3. Do you have other solutions in mind ?

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

清旖 2024-12-16 18:49:57

我会仔细研究直接字节缓冲区数组上的分散/聚集 NIO。直接性为您提供了无复制 I/O,而数组为您提供了增长机制。

I would have a good look at scatter/gather NIO on arrays of direct byte buffers. The direct-ness gives you no-copy I/O, and the array gives you the mechanism to grow.

野の 2024-12-16 18:49:57

我发现 Netty 为我提供了我需要的所有东西,甚至更多。

I found out that Netty supplied me with all the things I need and more.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文