Class: Google::Cloud::Bigquery::External::ParquetSource

Inherits:
DataSource
  • Object
show all
Defined in:
lib/google/cloud/bigquery/external/parquet_source.rb

Overview

ParquetSource

ParquetSource is a subclass of DataSource and represents a Parquet external data source that can be queried from directly, even though the data is not stored in BigQuery. Instead of loading or streaming the data, this object references the external data source.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

parquet_url = "gs://bucket/path/to/data.parquet"
parquet_table = bigquery.external parquet_url do |parquet|
  parquet.enable_list_inference = 1
end

data = bigquery.query "SELECT * FROM my_ext_table",
                      external: { my_ext_table: parquet_table }

# Iterate over the first page of results
data.each do |row|
  puts row[:name]
end
# Retrieve the next page of results
data = data.next if data.next?

Instance Method Summary collapse

Methods inherited from DataSource

#autodetect, #autodetect=, #avro?, #backup?, #bigtable?, #compression, #compression=, #csv?, #format, #hive_partitioning?, #hive_partitioning_mode, #hive_partitioning_mode=, #hive_partitioning_require_partition_filter=, #hive_partitioning_require_partition_filter?, #hive_partitioning_source_uri_prefix, #hive_partitioning_source_uri_prefix=, #ignore_unknown, #ignore_unknown=, #json?, #max_bad_records, #max_bad_records=, #orc?, #parquet?, #sheets?, #urls

Instance Method Details

#enable_list_inference ⇒ Boolean

Indicates whether to use schema inference specifically for Parquet LIST logical type.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

parquet_url = "gs://bucket/path/to/data.parquet"
parquet_table = bigquery.external parquet_url do |parquet|
  parquet.enable_list_inference = true
end

parquet_table.enable_list_inference #=> true

Returns:

  • (Boolean) 


76
77
78
 
# File 'lib/google/cloud/bigquery/external/parquet_source.rb', line 76

def enable_list_inference
  @gapi.parquet_options.enable_list_inference
end

#enable_list_inference=(new_enable_list_inference) ⇒ Object

Sets whether to use schema inference specifically for Parquet LIST logical type.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

parquet_url = "gs://bucket/path/to/data.parquet"
parquet_table = bigquery.external parquet_url do |parquet|
  parquet.enable_list_inference = true
end

parquet_table.enable_list_inference #=> true

Parameters:

  • new_enable_list_inference (Boolean) —

    The new enable_list_inference value.



97
98
99
100
 
# File 'lib/google/cloud/bigquery/external/parquet_source.rb', line 97

def enable_list_inference= new_enable_list_inference
  frozen_check!
  @gapi.parquet_options.enable_list_inference = new_enable_list_inference
end

#enum_as_string ⇒ Boolean

Indicates whether to infer Parquet ENUM logical type as STRING instead of BYTES by default.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

parquet_url = "gs://bucket/path/to/data.parquet"
parquet_table = bigquery.external parquet_url do |parquet|
  parquet.enum_as_string = true
end

parquet_table.enum_as_string #=> true

Returns:

  • (Boolean) 


119
120
121
 
# File 'lib/google/cloud/bigquery/external/parquet_source.rb', line 119

def enum_as_string
  @gapi.parquet_options.enum_as_string
end

#enum_as_string=(new_enum_as_string) ⇒ Object

Sets whether to infer Parquet ENUM logical type as STRING instead of BYTES by default.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

parquet_url = "gs://bucket/path/to/data.parquet"
parquet_table = bigquery.external parquet_url do |parquet|
  parquet.enum_as_string = true
end

parquet_table.enum_as_string #=> true

Parameters:

  • new_enum_as_string (Boolean) —

    The new enum_as_string value.



140
141
142
143
 
# File 'lib/google/cloud/bigquery/external/parquet_source.rb', line 140

def enum_as_string= new_enum_as_string
  frozen_check!
  @gapi.parquet_options.enum_as_string = new_enum_as_string
end