背景（跳过“问题”的最终目标）
以示例端点/Process-batch 处理给定单个请求的一批项目。收到该请求后，它将记录以下行：
{"endpoint_type": "process-batch", "batch_size": <integer>, "are_there_warnings": <boolean>}
我可以为此配置基于日志的度量：
< img src =“ https://i.sstatic.net/cuzvf.png” alt =“在此处输入映像
”上面创建的基于日志的公制，指定它可以从全球任何地方发射）：
结果仪表板看起来像这样：
问题是 batch_size 和 eas_there_warnings 不要自动显示为我可以用来操纵线路图的字段。请参阅下面的“组”和“添加过滤器”：
为了启用这些字段，我可以为每个字段添加一个标签（我们将其称为“明确标签”步骤，以供将来参考）：
添加了这些标签后：
我可以回到仪表板，然后查看“组”和“添加过滤器”的“组”：
并且可以操纵仪表板以使用这些字段：
问题
在尝试改善现有代码库的可观察性时，我必须添加所有现有字段已记录为显式标签在GCP侧，而不是GCP根据其名称和类型推断出记录的字段是标签。对于极少数情况，当我将一个新字段添加（例如 COPT 向/Process-Batch >的日志行中）时，我必须在GCP上进行明确的标签监视和记录端，而不是GCP推断新添加的字段应为组和过滤器中隐式添加的标签。
GCP是否有可能自动处理 JSONPAYLOAD 作为标签的所有字段？以下是我想到的一些合理的默认值：
- “ log Metric Name”以领域本身。
- “描述”将被留空。
- “标签类型”将从字段的类型中确定。
- “字段名称”将是
jsonpayload。＆lt; log-metric-name＆gt; 。
如果不可能，如果隐式将 JSONPAYLOAD 字段视为标签是一个坏主意，我也要感谢任何推理。在我看来，这将大大减少具有功能性仪表板的努力。
Background (skip to "The Problem" for end goal)
Take an example endpoint /process-batch that processes a batch of items given a single request. Upon receiving that request, it logs the following line:
{"endpoint_type": "process-batch", "batch_size": <integer>, "are_there_warnings": <boolean>}
I can configure a logs-based metric for this via:
Then I can create a line chart (changing the "Resource & Metric" to be the logs-based metric created above, specifying that it can be emitted from anywhere globally):
Resulting dashboard looks like this:
The issue is that batch_size and are_there_warnings don't automatically appear as fields that I can use to manipulate the line graph. See "Group By" and "Add Filter" below:
To enable those fields, I can add a label for each of them (let's call this the "explicit labeling" step for future reference):
After confirming those labels are added:
I can go back to the dashboard and see that "Group By" and "Add Filter" have been updated accordingly:
And can manipulate the dashboard to use those fields:
The Problem
When trying to improve observability for an existing codebase, I have to add all the existing fields already logged as explicit labels on the GCP side, instead of GCP inferring that the logged fields are labels based on their name and type. And for the rare cases when I add a new field (such as cost to the log lines emitted by /process-batch), I have to do explicit labeling on the GCP Monitoring and Logging side instead of GCP inferring that the newly added field should be a new implicitly added label in group by and filter.
Is it possible for GCP to automatically treat all fields part of jsonPayload as labels? Here are some reasonable defaults I had in mind:
- "Log metric name" takes on the name of the field itself.
- "Description" would be left empty.
- "Label type" would be determined from the type of the field.
- "Field name" would be
jsonPayload.<log-metric-name>.
If this is not possible, I'd also appreciate any reasoning if implicitly treating jsonPayload fields as labels is a bad idea. In my eyes, it would significantly reduce the effort to have functional dashboards.
发布评论
评论（2）
检查后，此类功能尚不可用。但是，我们已经提出了这个请求。此请求没有ETA，但我们可以使用以下链接进行监视。
您可以在线程中添加公共问题跟踪器功能请求，并在线程中添加“我也”。随着越来越多的用户要求支持它，这将引起人们对请求的更多关注。
功能请求跟踪器：
从JSON有效载荷发出的自动检索标签
到目前为止，以下是一些指南。
配置代理商
special-fields“ rel =“ nofollow noreferrer”>处理有效载荷>
https://cloud.google.com/stackdriver/docs/solutions/agents/agents/logging/configuration？hl = en#en# ///cloud.google.com/stackdriver/docs/solutions/agents/agents/logging/configuration?hl = en#special-fields“ rel =“ nofollow noreferrer”>结构性有效载荷中的特殊领域
Upon checking, This kind of feature is not yet available. However we already raise this request. This request does not have an ETA but we can monitor this using below link.
You can star the public issue tracker feature requests and add ‘Me too’ in the thread. This will bring more attention to the request as more users request support for it.
Feature Request Tracker :
Autodetect Labels from JSON Payload
As of now below are some guides that can help you.
Google Cloud fluentd output plugin configuration
Configuring the agent
Processing payloads
Special fields in structured payloads
我来自云记录工程团队。
当前在技术上不可能将从日志传播到基于日志的公制（LBM）标签的自动传播JSONPAY LOADIANS。 LBM是具有某些可自定义标签的指标。随着日志的摄入，公制计数器会用标签值增加。标签数量及其基于日志的指标的限制，而日志中的JSONPAY负载具有无限的基数和不断发展的模式。
对于这样的用例， log Analytics 是一个更合适。我们正在在日志分析中添加对图表和仪表板的支持，以便您可以通过直接使用SQL直接查询日志来直接添加以任何JSONPAYLOAD值作为尺寸或故障的图表。在接下来的几周内敬请关注此功能。
I am from the Cloud Logging Engineering team.
Auto propagating jsonPayload fields from logs into logs-based metric (LBM) labels is currently not technically possible. LBMs are metrics that are created with certain customizable labels. As logs are ingested, the metric counters are incremented with the label values. There are limits on the number of labels and their cardinality in logs-based metrics, whereas jsonPayload in logs have unbounded cardinality and evolving schemas.
For a use-case like this, Log Analytics is a better fit. We are adding support for charting and dashboarding in Log Analytics such that you can directly add a chart with any jsonPayload values as dimensions or breakdowns by directly querying the logs using SQL. Stay tuned for this feature in the coming weeks.