如何在生产区域以分布方式运行Kafka连接

发布于 2025-02-10 17:03:10 字数 536 浏览 2 评论 0 原文

根据Confluent上的文档我们可以使用Connect Rest API来开始/停止/管理Kafka Connect-只想了解当我们在云上部署资源时,最好的方法是什么(在我的情况下AWS)在我们限制UI访问的位置。

  1. 我们在私人子网中拥有所有应用程序和kafka。
  2. 我们根本无法使用任何公共IP来暴露GUI或直接通过Bowser或Postman直接击中Kafka Connect API。
  3. 由于访问限制,我们也无法进行端口转发。

想了解人们如何使用直接访问限制来运行生产级Kafka-我们可以直接使用的一些开源工具?在REST CONNECT API周围创建并使OpenSOURCE吗? 请指导这一点。

谢谢

AS per documentation on confluent https://docs.confluent.io/platform/current/connect/references/restapi.html we can use connect rest api to start/stop/manage kafka connect - just want to understand what is the preferable way when we are deploying our resource on cloud (in my case aws) where we have restricted UI access.

  1. We have all the application and kafka in private subnet.
  2. We can not use any public ip at all to expose the GUI or hit the REST Kafka connect API directly via bowser or POSTMAN.
  3. We can't do port forwarding as well due to access restriction.

Want to understand how people runs production grade Kafka with direct access restriction - some open source tool which we can use directly? anything creating and made opensource around REST Connect API?
Kindly guide on this.

Thank you

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

花想c 2025-02-17 17:03:10

不能做端口转发

即使使用ELB/ALB也 ?这将是访问API的必要步骤

限制UI访问

Kafka Connect没有UI,因此您无需担心。 MSK Connect仪表板确实可以,但是只有在您登录到AWS帐户时才可以访问。

完全无法使用任何公共IP来暴露GUI或直接通过Bowser或Postman直接击中Kafka Connect API

它绝对不应该是公开的,但是只要您拥有,您就可以使用袜子代理来解决该限制SSH访问。不过,我建议不要使用桌面/浏览器与API进行交互。您应该将配置保留为代码;例如,“ nofollow noreferrer”> Terraform> Terraform 是MSK连接器。

例如,您也可以使用EKS和部署Strimzi操作员,并将连接部署作为Kubernetes资源管理。然后,按照GitOps风格,连接器在私有子网中通过GIT操作进行控制,而不是直接与任何连接HTTP端点进行交互。

can't do port forwarding

Even with an ELB/ALB? That will be a necessary step to access the API at all

restricted UI access

Kafka Connect does not have a UI, so you don't need to worry about that. MSK Connect dashboard does, but that is only accessible when you login to the AWS Account.

can not use any public ip at all to expose the GUI or hit the REST Kafka connect API directly via bowser or POSTMAN

It definitely shouldn't be public, but you could use a SOCKS proxy to get around that limitation, as long as you have SSH access. I would suggest not using desktop/browser to interact with the API, though. You should keep your configurations as code; for example, Terraform an MSK Connector.

You could also use EKS and deploy Strimzi operator, for example, and manage Connect deployments as Kubernetes resources. Then, following GitOps-style, connectors are controlled with Git actions within a private subnet, rather than interact directly with any Connect HTTP endpoints.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文