我们有一个2个Clickhouse节点的群集。我们将其用于从Kafka流式传输,因此我们使用Kafka发动机桌。数据格式为 proto
,我们将每种原始格式都放在Clickhouse的特定文件夹中。
我正在处理的问题是:
我有一个旧的原始原型,一段时间后我应该更改数据模式,因此我删除了旧的Kafka引擎表,并创建了一个带有相同表名称的新的。我还创建了一个新文件夹并在此处复制了新的原始文件,但是我发现ClickHouse Caches Proto架构每张表格,并且没有识别新的Proto Path,因此我创建了一个带有不同名称的新的Kafka引擎表,但是此时Clickhouse找到了原始文件但它发生了错误和错误:
<Error> void DB::StorageKafka::threadFunc(size_t): Code: 36. DB::Exception: Not found a message named 'MyMessage' in the schema file 'test/my_message.proto': While executing Kafka. (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):
我想知道如何解决此问题,我应该删除缓存,如果是,我在哪里可以找到缓存文件路径?
We have a cluster of 2 clickhouse nodes. We use it for streaming from kafka, so we use kafka engine table. The data format is proto
and we put each proto format in a specific folder in clickhouse.
The issue I am dealing with is:
I had an old proto, after a while I should change the schema of data, so I removed the old kafka engine table and created a new one with the same table name. I also created a new folder and copied the new proto here, but I found that Clickhouse caches proto schema per table and didn't recognize the new proto path, so I created a new kafka engine table with different name, but this time Clickhouse found the proto filee but it rasied and error:
<Error> void DB::StorageKafka::threadFunc(size_t): Code: 36. DB::Exception: Not found a message named 'MyMessage' in the schema file 'test/my_message.proto': While executing Kafka. (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):
I wonder how to resolve this issues, should I delete cache, If so, where can I find the cache files path?
发布评论
评论(1)
我解决了这个问题,我做了以下步骤:
更改kafka引擎表
更改消费者组名称添加新版本,该名称应在Kafka Engine Table
中指定
更改文件夹和原始文件名(可能不需要文件夹名称更改)
例如,在原始文件中更改消息名称:
消息消息1 {.....}
消息消息1 {.....}
在原始文件中更改消息的名称,然后在存在的任何地方更改Message1和Message2。
您无需更改实现的视图名称(消费者名称)和事件最终表名称。
I resolved this issue, I did these steps:
Change kafka engine table for example add a new version to it's name
Change consumer group name which should be specified in kafka engine table
Change the folder and proto file name(folder name change may not be needed)
Change the name of messages in proto file for example:
message MESSAGE1{.....}
message MESSAGE1 {.....}
Then change MESSAGE1 and MESSAGE2 wherever exists.
you don't need to change materialized view name (consumer name) and event the final table name.