Batch loading CSV files into bigquery 代码示例

以下是使用Python和Google Cloud Client库将CSV文件批量加载到BigQuery的示例代码：

from google.cloud import bigquery
import os

# Set up Google Cloud credentials
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/path/to/credentials.json"

# Initialize BigQuery client
client = bigquery.Client()

# Set dataset and table name
dataset_id = "my_dataset"
table_id = "my_table"

# Set path to folder containing CSV files
folder_path = "/path/to/csv/folder"

# Get list of CSV file paths
file_paths = []
for file in os.listdir(folder_path):
    if file.endswith(".csv"):
        file_paths.append(os.path.join(folder_path, file))

# Set up table schema
schema = [
    bigquery.SchemaField("column1", "STRING"),
    bigquery.SchemaField("column2", "INTEGER"),
    bigquery.SchemaField("column3", "FLOAT"),
]

# Set up job configuration
job_config = bigquery.LoadJobConfig(
    schema=schema,
    skip_leading_rows=1,
    source_format=bigquery.SourceFormat.CSV,
)

# Load each CSV file to BigQuery
for file_path in file_paths:
    with open(file_path, "rb") as source_file:
        job = client.load_table_from_file(
            source_file,
            dataset_id + "." + table_id,
            job_config=job_config
        )

    job.result()  # Wait for the job to complete

print("CSV files loaded to BigQuery table successfully!")

这段代码假设您已经将CSV文件放在一个文件夹中，并且拥有BigQuery的凭据。它将遍历文件夹中的每个CSV文件，使用指定的架构和作业配置将其加载到指定的BigQuery表中。