Module: Google::Cloud::Bigquery::External

Defined in:

lib/google/cloud/bigquery/external.rb,
lib/google/cloud/bigquery/external/csv_source.rb,
lib/google/cloud/bigquery/external/avro_source.rb,
lib/google/cloud/bigquery/external/data_source.rb,
lib/google/cloud/bigquery/external/json_source.rb,
lib/google/cloud/bigquery/external/sheets_source.rb,
lib/google/cloud/bigquery/external/parquet_source.rb,
lib/google/cloud/bigquery/external/bigtable_source.rb,
lib/google/cloud/bigquery/external/bigtable_source/column.rb,
lib/google/cloud/bigquery/external/bigtable_source/column_family.rb

Overview

External

Creates a new DataSource (or subclass) object that represents the external data source that can be queried from directly, even though the data is not stored in BigQuery. Instead of loading or streaming the data, this object references the external data source.

See DataSource, CsvSource, JsonSource, SheetsSource, BigtableSource

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

csv_url = "gs://bucket/path/to/data.csv"
csv_table = bigquery.external csv_url do |csv|
  csv.autodetect = true
  csv.skip_leading_rows = 1
end

data = bigquery.query "SELECT * FROM my_ext_table",
                      external: { my_ext_table: csv_table }

# Iterate over the first page of results
data.each do |row|
  puts row[:name]
end
# Retrieve the next page of results
data = data.next if data.next?

Hive partitioning options:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

gcs_uri = "gs://cloud-samples-data/bigquery/hive-partitioning-samples/autolayout/*"
source_uri_prefix = "gs://cloud-samples-data/bigquery/hive-partitioning-samples/autolayout/"
external_data = bigquery.external gcs_uri, format: :parquet do |ext|
  ext.hive_partitioning_mode = :auto
  ext.hive_partitioning_require_partition_filter = true
  ext.hive_partitioning_source_uri_prefix = source_uri_prefix
end

external_data.hive_partitioning? #=> true
external_data.hive_partitioning_mode #=> "AUTO"
external_data.hive_partitioning_require_partition_filter? #=> true
external_data.hive_partitioning_source_uri_prefix #=> source_uri_prefix

Defined Under Namespace

Classes: AvroSource, BigtableSource, CsvSource, DataSource, JsonSource, ParquetSource, SheetsSource