I am not sure if pandas-gbq uses load jobs or streaming inserts under the hood.
Anyway, you can also use the BigQuery Python Client and the method insert_rows_from_dataframe which uses the streaming API.
For your requirement, you can use Google provided Dataflow Templates that contain templates where you can use Pub/Sub to stream data into BigQuery using Dataflow. A streaming pipeline is used to read the JSON formatted data from Cloud Pub/Sub and then write it to BigQuery.
发布评论
评论（2）
提到的限制仅适用于 load load of load ofers 。
您可以使用，而没有那些 limits 。
我不确定 pandas-gbq 是否使用负载作业或引擎盖下的流式插件。
无论如何，您也可以使用 bigquery python python python客户端 =“ https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.client.client.html#google.cloud.bigquery.bigquery.bigquery.client.client.client.client.client.client.client.client _from_from_from_dataafame_from_dataafame_from_dataaframe” 使用流API。
The mentioned limits only apply to load jobs.
You can use streaming inserts instead, which do not have those limits.
I am not sure if pandas-gbq uses load jobs or streaming inserts under the hood.
Anyway, you can also use the BigQuery Python Client and the method insert_rows_from_dataframe which uses the streaming API.
为了您的要求，您可以使用Google提供的数据流模板，该模板包含模板，您可以使用Pub/sub使用DataFlow将数据传输到BigQuery中。流媒体管道用于读取从云公共/sub/sub的JSON格式化数据，然后将其写入BigQuery。
您可以选择模板或。
For your requirement, you can use Google provided Dataflow Templates that contain templates where you can use Pub/Sub to stream data into BigQuery using Dataflow. A streaming pipeline is used to read the JSON formatted data from Cloud Pub/Sub and then write it to BigQuery.
You can choose either Pub/Sub Topic to BigQuery template or Pub/Sub Subscription to BigQuery template according to the requirement. It will incur some cost in BigQuery for data ingestion, for which you can check the pricing given in this document.