Vertex Workbench-如何在Jupyter笔记本中运行BigQueryExamplegen

Question

问题

试图运行

InvalidUserInputError: Request missing required parameter projectId [while running 'InputToRecord/QueryTable/ReadFromBigQuery/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction']

”

import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "path_to_credential_file"


from tfx.v1.extensions.google_cloud_big_query import BigQueryExampleGen
from tfx.v1.components import (
    StatisticsGen,
    SchemaGen,
)
from tfx.orchestration.experimental.interactive.interactive_context import InteractiveContext
%load_ext tfx.orchestration.experimental.interactive.notebook_extensions.skip
context = InteractiveContext(pipeline_root='./data/artifacts')

运行BigQueryExample。

query = """
SELECT 
    * EXCEPT (trip_start_timestamp, ML_use)
FROM 
    {PROJECT_ID}.public_dataset.chicago_taxitrips_prep
""".format(PROJECT_ID=PROJECT_ID)

example_gen = context.run(
    BigQueryExampleGen(query=query)
)

有错误。

InvalidUserInputError: Request missing required parameter projectId [while running 'InputToRecord/QueryTable/ReadFromBigQuery/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction']

数据

请参见为芝加哥出租车旅行数据集设置BigQuery数据集。

Answer 1

在GCP中运行的项目ID

需要通过 beam_pipeline_args 参数提供项目ID。

• colab Interactivecontext无法确定bq＃882 的ProjectIDID

已提出＃888来进行这项工作。通过这种更改，您将能够做

  context.run（...，beam_pipeline_args = [' -  project'，'my-project']）
 

query = """
SELECT 
    * EXCEPT (trip_start_timestamp, ML_use)
FROM 
    {PROJECT_ID}.public_dataset.chicago_taxitrips_prep
""".format(PROJECT_ID=PROJECT_ID)

example_gen = context.run(
    BigQueryExampleGen(query=query),
    beam_pipeline_args=[
        '--project', PROJECT_ID,
    ]
)

，但它仍然会因其他错误而失败。

ValueError: ReadFromBigQuery requires a GCS location to be provided. Neither gcs_location in the constructor nor the fallback option --temp_location is set. [while running 'InputToRecord/QueryTable/ReadFromBigQuery/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction']

GCS存储桶

它在GCP中，交互式上下文通过数据流运行BigQueryExampleGen，因此需要通过 beam_pipeline_args 参数来提供GCS存储桶URL。

• bigqueryexamplegen破碎的readfrombigquery需要GCS位置才能提供＃2293
• ：//stackoverflow.com/questions/68325195/valueError-in-dataflow-invalid-gcs-location-none> valueerror in dataFlow中：无效的GCS位置：无

运行数据流管线时通过参数-temp_location gs：// bucket/subfolder/

query = """
SELECT 
    * EXCEPT (trip_start_timestamp, ML_use)
FROM 
    {PROJECT_ID}.public_dataset.chicago_taxitrips_prep
""".format(PROJECT_ID=PROJECT_ID)

example_gen = context.run(
    BigQueryExampleGen(query=query),
    beam_pipeline_args=[
        '--project', PROJECT_ID,
        '--temp_location', BUCKET
    ]
)

statistics_gen = context.run(
    StatisticsGen(examples=example_gen.component.outputs['examples'])
)
context.show(statistics_gen.component.outputs['statistics'])

schema_gen = SchemaGen(
    statistics=statistics_gen.component.outputs['statistics'],
    infer_feature_shape=True
)
context.run(schema_gen)
context.show(schema_gen.outputs['schema'])

documentation

• 使用tfx和vertex pipelines读取BigQuery的数据

本基于笔记本的教程将使用Google Cloud BigQuery作为训练ML模型的数据源。 ML管道将使用TFX构建，并在Google Cloud Cloud顶点管道上运行。在本教程中，我们将使用bigQueryExampleGen组件，该组件读取从BigQuery到TFX管道的数据。

我们还需要通过Beam_pipeline_args 进行BigQueryExample。它包括之类的配置，例如GCP项目的名称和临时存储 for BigQuery执行。