Class: Google::Cloud::Bigquery::Schema

Inherits:
Object
  • Object
show all
Defined in:
lib/google/cloud/bigquery/schema.rb,
lib/google/cloud/bigquery/schema/field.rb

Overview

Table Schema

A builder for BigQuery table schemas, passed to block arguments to Dataset#create_table and Table#schema. Supports nested and repeated fields via a nested block.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.create_table "my_table"

table.schema do |schema|
  schema.string "first_name", mode: :required
  schema.record "cities_lived", mode: :repeated do |cities_lived|
    cities_lived.string "place", mode: :required
    cities_lived.integer "number_of_years", mode: :required
  end
end

See Also:

Defined Under Namespace

Classes: Field

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.dump(schema, destination) ⇒ Schema

Write a schema as JSON to a file.

The JSON schema file is the same as for the bq CLI.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"
schema = Google::Cloud::Bigquery::Schema.dump(
  table.schema,
  "schema.json"
)

Parameters:

  • schema (Schema) —

    A Google::Cloud::Bigquery::Schema.

  • destination (IO, String) —

    An IO to which to write the schema, or a String containing the filename to write to.

Returns:

  • (Schema) —

    The schema so that commands are chainable.



105
106
107
 
# File 'lib/google/cloud/bigquery/schema.rb', line 105

def dump schema, destination
  schema.dump destination
end

.load(source) ⇒ Schema

Load a schema from a JSON file.

The JSON schema file is the same as for the bq CLI consisting of an array of JSON objects containing the following:

  • name: The column name
  • type: The column's data type
  • description: (Optional) The column's description
  • mode: (Optional) The column's mode (if unspecified, mode defaults to NULLABLE)
  • fields: If type is RECORD, an array of objects defining child fields with these properties

Examples:

require "google/cloud/bigquery"

schema = Google::Cloud::Bigquery::Schema.load(
  File.read("schema.json")
)

Parameters:

  • source (IO, String, Array<Hash>) —

    An IO containing the JSON schema, a String containing the JSON schema, or an Array of Hashes containing the schema details.

Returns:



77
78
79
 
# File 'lib/google/cloud/bigquery/schema.rb', line 77

def load source
  new.load source
end

Instance Method Details

#boolean(name, description: nil, mode: :nullable) ⇒ Object

Adds a boolean field to the schema.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



364
365
366
 
# File 'lib/google/cloud/bigquery/schema.rb', line 364

def boolean name, description: nil, mode: :nullable
  add_field name, :boolean, description: description, mode: mode
end

#bytes(name, description: nil, mode: :nullable) ⇒ Object

Adds a bytes field to the schema.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



380
381
382
 
# File 'lib/google/cloud/bigquery/schema.rb', line 380

def bytes name, description: nil, mode: :nullable
  add_field name, :bytes, description: description, mode: mode
end

#date(name, description: nil, mode: :nullable) ⇒ Object

Adds a date field to the schema.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



443
444
445
 
# File 'lib/google/cloud/bigquery/schema.rb', line 443

def date name, description: nil, mode: :nullable
  add_field name, :date, description: description, mode: mode
end

#datetime(name, description: nil, mode: :nullable) ⇒ Object

Adds a datetime field to the schema.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



427
428
429
 
# File 'lib/google/cloud/bigquery/schema.rb', line 427

def datetime name, description: nil, mode: :nullable
  add_field name, :datetime, description: description, mode: mode
end

#dump(destination) ⇒ Schema

Write the schema as JSON to a file.

The JSON schema file is the same as for the bq CLI.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"
table.schema.dump "schema.json"

Parameters:

  • destination (IO, String) —

    An IO to which to write the schema, or a String containing the filename to write to.

Returns:

  • (Schema) —

    The schema so that commands are chainable.



275
276
277
278
279
280
281
282
283
284
 
# File 'lib/google/cloud/bigquery/schema.rb', line 275

def dump destination
  if destination.respond_to?(:rewind) && destination.respond_to?(:write)
    destination.rewind
    destination.write JSON.dump(fields.map(&:to_hash))
  else
    File.write String(destination), JSON.dump(fields.map(&:to_hash))
  end

  self
end

#empty? ⇒ Boolean

Whether the schema has no fields defined.

Returns:

  • (Boolean) —

    true when there are no fields, false otherwise.



205
206
207
 
# File 'lib/google/cloud/bigquery/schema.rb', line 205

def empty?
  fields.empty?
end

#field(name) {|f| ... } ⇒ Field

Retrieve a field by name.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

field = table.schema.field "name"
field.required? #=> true

Yields:

  • (f)

Returns:

  • (Field) —

    A field object.



193
194
195
196
197
198
 
# File 'lib/google/cloud/bigquery/schema.rb', line 193

def field name
  f = fields.find { |fld| fld.name == name.to_s }
  return nil if f.nil?
  yield f if block_given?
  f
end

#fields ⇒ Array<Field>

The fields of the table schema.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

schema = table.schema

schema.fields.each do |field|
  puts field.name
end

Returns:

  • (Array<Field>) —

    An array of field objects.



127
128
129
130
131
132
133
 
# File 'lib/google/cloud/bigquery/schema.rb', line 127

def fields
  if frozen?
    Array(@gapi.fields).map { |f| Field.from_gapi(f).freeze }.freeze
  else
    Array(@gapi.fields).map { |f| Field.from_gapi f }
  end
end

#float(name, description: nil, mode: :nullable) ⇒ Object

Adds a floating-point number field to the schema.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



330
331
332
 
# File 'lib/google/cloud/bigquery/schema.rb', line 330

def float name, description: nil, mode: :nullable
  add_field name, :float, description: description, mode: mode
end

#headers ⇒ Array<Symbol>

The names of the fields as symbols.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.create_table "my_table"

schema = table.schema

schema.headers.each do |header|
  puts header
end

Returns:

  • (Array<Symbol>) —

    An array of column names.



153
154
155
 
# File 'lib/google/cloud/bigquery/schema.rb', line 153

def headers
  fields.map(&:name).map(&:to_sym)
end

#integer(name, description: nil, mode: :nullable) ⇒ Object

Adds an integer field to the schema.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



314
315
316
 
# File 'lib/google/cloud/bigquery/schema.rb', line 314

def integer name, description: nil, mode: :nullable
  add_field name, :integer, description: description, mode: mode
end

#load(source) ⇒ Schema

Load the schema from a JSON file.

The JSON schema file is the same as for the bq CLI consisting of an array of JSON objects containing the following:

  • name: The column name
  • type: The column's data type
  • description: (Optional) The column's description
  • mode: (Optional) The column's mode (if unspecified, mode defaults to NULLABLE)
  • fields: If type is RECORD, an array of objects defining child fields with these properties

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table" do |t|
  t.schema.load File.read("path/to/schema.json")
end

Parameters:

  • source (IO, String, Array<Hash>) —

    An IO containing the JSON schema, a String containing the JSON schema, or an Array of Hashes containing the schema details.

Returns:

  • (Schema) —

    The schema so that commands are chainable.



239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
 
# File 'lib/google/cloud/bigquery/schema.rb', line 239

def load source
  if source.respond_to?(:rewind) && source.respond_to?(:read)
    source.rewind
    schema_json = String source.read
  elsif source.is_a? Array
    schema_json = JSON.dump source
  else
    schema_json = String source
  end

  schema_json = %({"fields":#{schema_json}})

  @gapi = Google::Apis::BigqueryV2::TableSchema.from_json schema_json

  self
end

#numeric(name, description: nil, mode: :nullable) ⇒ Object

Adds a numeric number field to the schema. Numeric is a fixed-precision numeric type with 38 decimal digits, 9 that follow the decimal point.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



348
349
350
 
# File 'lib/google/cloud/bigquery/schema.rb', line 348

def numeric name, description: nil, mode: :nullable
  add_field name, :numeric, description: description, mode: mode
end

#param_types ⇒ Hash

The types of the fields, using the same format as the optional query parameter types.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.create_table "my_table"

schema = table.schema

schema.param_types

Returns:

  • (Hash) —

    A hash with column names as keys, and types as values.



174
175
176
 
# File 'lib/google/cloud/bigquery/schema.rb', line 174

def param_types
  Hash[fields.map { |field| [field.name.to_sym, field.param_type] }]
end

#record(name, description: nil, mode: nil) {|field| ... } ⇒ Object

Adds a record field to the schema. A block must be passed describing the nested fields of the record. For more information about nested and repeated records, see Loading denormalized, nested, and repeated data .

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.create_table "my_table"

table.schema do |schema|
  schema.string "first_name", mode: :required
  schema.record "cities_lived", mode: :repeated do |cities_lived|
    cities_lived.string "place", mode: :required
    cities_lived.integer "number_of_years", mode: :required
  end
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: nil) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.

Yields:

  • (field) —

    a block for setting the nested record's schema

Yield Parameters:

  • field (Field) —

    the object accepting the nested schema

Raises:

  • (ArgumentError) 


481
482
483
484
485
486
487
488
 
# File 'lib/google/cloud/bigquery/schema.rb', line 481

def record name, description: nil, mode: nil
  # TODO: do we need to raise if no block was given?
  raise ArgumentError, "a block is required" unless block_given?

  nested_field = add_field name, :record, description: description, mode: mode
  yield nested_field
  nested_field
end

#string(name, description: nil, mode: :nullable) ⇒ Object

Adds a string field to the schema.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



298
299
300
 
# File 'lib/google/cloud/bigquery/schema.rb', line 298

def string name, description: nil, mode: :nullable
  add_field name, :string, description: description, mode: mode
end

#time(name, description: nil, mode: :nullable) ⇒ Object

Adds a time field to the schema.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



411
412
413
 
# File 'lib/google/cloud/bigquery/schema.rb', line 411

def time name, description: nil, mode: :nullable
  add_field name, :time, description: description, mode: mode
end

#timestamp(name, description: nil, mode: :nullable) ⇒ Object

Adds a timestamp field to the schema.

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



395
396
397
 
# File 'lib/google/cloud/bigquery/schema.rb', line 395

def timestamp name, description: nil, mode: :nullable
  add_field name, :timestamp, description: description, mode: mode
end