Class: Google::Cloud::Bigquery::Data

Inherits:
Array
  • Object
show all
Defined in:
lib/google/cloud/bigquery/data.rb

Overview

Data

Represents a page of results (rows) as an array of hashes. Because Data delegates to Array, methods such as Array#count represent the number of rows in the page. In addition, methods of this class include result set metadata such as total and provide access to the schema of the query or table. See Project#query, Google::Cloud::Bigquery::Dataset#query and Table#data.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

sql = "SELECT word FROM `bigquery-public-data.samples.shakespeare`"
job = bigquery.query_job sql

job.wait_until_done!
data = job.data

data.count # 100000
data.total # 164656
data.each do |row|
  puts row[:word]
end
data = data.next if data.next?

Instance Method Summary collapse

Instance Method Details

#all(request_limit: nil) {|row| ... } ⇒ Enumerator

Retrieves all rows by repeatedly loading #next until #next? returns false. Calls the given block once for each row, which is passed as the parameter.

An enumerator is returned if no block is given.

This method may make several API calls until all rows are retrieved. Be sure to use as narrow a search criteria as possible. Please use with caution.

Examples:

Iterating each rows by passing a block:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

table.data.all do |row|
  puts row[:word]
end

Using the enumerator by not passing a block:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

words = table.data.all.map do |row|
  row[:word]
end

Limit the number of API calls made:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

table.data.all(request_limit: 10) do |row|
  puts row[:word]
end

Parameters:

  • request_limit (Integer) (defaults to: nil) —

    The upper limit of API requests to make to load all data. Default is no limit.

Yields:

  • (row) —

    The block for accessing each row of data.

Yield Parameters:

  • row (Hash) —

    The row object.

Returns:

  • (Enumerator) —

    An enumerator providing access to all of the data.



473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
 
# File 'lib/google/cloud/bigquery/data.rb', line 473

def all request_limit: nil
  request_limit = request_limit.to_i if request_limit

  return enum_for :all, request_limit: request_limit unless block_given?

  results = self
  loop do
    results.each { |r| yield r }
    if request_limit
      request_limit -= 1
      break if request_limit.negative?
    end
    break unless results.next?
    results = results.next
  end
end

#ddl? ⇒ Boolean

Whether the query that created this data was a DDL statement.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
data = bigquery.query "CREATE TABLE my_table (x INT64)"

data.statement_type #=> "CREATE_TABLE"
data.ddl? #=> true

Returns:

  • (Boolean)

See Also:



272
273
274
275
 
# File 'lib/google/cloud/bigquery/data.rb', line 272

def ddl?
  ["CREATE_MODEL", "CREATE_TABLE", "CREATE_TABLE_AS_SELECT", "CREATE_VIEW", "\n", "DROP_MODEL", "DROP_TABLE",
   "DROP_VIEW"].include? statement_type
end

#ddl_operation_performed ⇒ String?

The DDL operation performed, possibly dependent on the pre-existence of the DDL target. (See #ddl_target_table.) Possible values (new values might be added in the future):

  • "CREATE": The query created the DDL target.
  • "SKIP": No-op. Example cases: the query is CREATE TABLE IF NOT EXISTS while the table already exists, or the query is DROP TABLE IF EXISTS while the table does not exist.
  • "REPLACE": The query replaced the DDL target. Example case: the query is CREATE OR REPLACE TABLE, and the table already exists.
  • "DROP": The query deleted the DDL target.

Returns:

  • (String, nil) —

    The DDL operation performed.



315
316
317
 
# File 'lib/google/cloud/bigquery/data.rb', line 315

def ddl_operation_performed
  job_gapi&.statistics&.query&.ddl_operation_performed
end

#ddl_target_routine ⇒ Google::Cloud::Bigquery::Routine?

The DDL target routine, in reference state. (See Routine#reference?.) Present only for CREATE/DROP FUNCTION/PROCEDURE queries. (See #statement_type.)

Returns:



327
328
329
330
331
332
 
# File 'lib/google/cloud/bigquery/data.rb', line 327

def ddl_target_routine
  ensure_service!
  routine = job_gapi&.statistics&.query&.ddl_target_routine
  return nil if routine.nil?
  Google::Cloud::Bigquery::Routine.new_reference_from_gapi routine, service
end

#ddl_target_table ⇒ Google::Cloud::Bigquery::Table?

The DDL target table, in reference state. (See Table#reference?.) Present only for CREATE/DROP TABLE/VIEW queries. (See #statement_type.)

Returns:



342
343
344
345
346
347
 
# File 'lib/google/cloud/bigquery/data.rb', line 342

def ddl_target_table
  ensure_service!
  table = job_gapi&.statistics&.query&.ddl_target_table
  return nil if table.nil?
  Google::Cloud::Bigquery::Table.new_reference_from_gapi table, service
end

#dml? ⇒ Boolean

Whether the query that created this data was a DML statement.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
data = bigquery.query "UPDATE my_table " \
                      "SET x = x + 1 " \
                      "WHERE x IS NOT NULL"

data.statement_type #=> "UPDATE"
data.dml? #=> true

Returns:

  • (Boolean)

See Also:



296
297
298
 
# File 'lib/google/cloud/bigquery/data.rb', line 296

def dml?
  ["INSERT", "UPDATE", "MERGE", "DELETE"].include? statement_type
end

#etag ⇒ String

An ETag hash for the page of results represented by the data instance.

Returns:

  • (String) —

    The ETag hash.



88
89
90
 
# File 'lib/google/cloud/bigquery/data.rb', line 88

def etag
  @gapi_json[:etag]
end

#fields ⇒ Array<Schema::Field>

The fields of the data, obtained from the schema of the table from which the data was read.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

data = table.data

data.fields.each do |field|
  puts field.name
end

Returns:



176
177
178
 
# File 'lib/google/cloud/bigquery/data.rb', line 176

def fields
  schema.fields
end

#headers ⇒ Array<Symbol>

The names of the columns in the data, obtained from the schema of the table from which the data was read.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

data = table.data

data.headers.each do |header|
  puts header
end

Returns:

  • (Array<Symbol>) —

    An array of column names.



199
200
201
 
# File 'lib/google/cloud/bigquery/data.rb', line 199

def headers
  schema.headers
end

#kind ⇒ String

The resource type of the API response.

Returns:

  • (String) —

    The resource type.



79
80
81
 
# File 'lib/google/cloud/bigquery/data.rb', line 79

def kind
  @gapi_json[:kind]
end

#next ⇒ Data

Retrieves the next page of data.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

sql = "SELECT word FROM `bigquery-public-data.samples.shakespeare`"
job = bigquery.query_job sql

job.wait_until_done!
data = job.data

data.count # 100000
data.total # 164656
data.each do |row|
  puts row[:word]
end
data = data.next if data.next?

Returns:

  • (Data) —

    A new instance providing the next page of data.



410
411
412
413
414
415
416
417
418
 
# File 'lib/google/cloud/bigquery/data.rb', line 410

def next
  return nil unless next?
  ensure_service!
  data_json = service.list_tabledata \
    @table_gapi.table_reference.dataset_id,
    @table_gapi.table_reference.table_id,
    token: token
  self.class.from_gapi_json data_json, @table_gapi, job_gapi, @service
end

#next? ⇒ Boolean

Whether there is a next page of data.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

sql = "SELECT word FROM `bigquery-public-data.samples.shakespeare`"
job = bigquery.query_job sql

job.wait_until_done!
data = job.data

data.count # 100000
data.total # 164656
data.each do |row|
  puts row[:word]
end
data = data.next if data.next?

Returns:

  • (Boolean) —

    true when there is a next page, false otherwise.



383
384
385
 
# File 'lib/google/cloud/bigquery/data.rb', line 383

def next?
  !token.nil?
end

#num_dml_affected_rows ⇒ Integer?

The number of rows affected by a DML statement. Present only for DML statements INSERT, UPDATE or DELETE. (See #statement_type.)

Returns:

  • (Integer, nil) —

    The number of rows affected by a DML statement, or nil if the query is not a DML statement.



356
357
358
 
# File 'lib/google/cloud/bigquery/data.rb', line 356

def num_dml_affected_rows
  job_gapi&.statistics&.query&.num_dml_affected_rows
end

#param_types ⇒ Hash

The types of the fields in the data, obtained from the schema of the table from which the data was read. Types use the same format as the optional query parameter types.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

data = table.data

data.param_types

Returns:

  • (Hash) —

    A hash with field names as keys, and types as values.



221
222
223
 
# File 'lib/google/cloud/bigquery/data.rb', line 221

def param_types
  schema.param_types
end

#schema ⇒ Schema

The schema of the table from which the data was read.

The returned object is frozen and changes are not allowed. Use Table#schema to update the schema.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

data = table.data

schema = data.schema
field = schema.field "name"
field.required? #=> true

Returns:

  • (Schema) —

    A schema object.



152
153
154
155
 
# File 'lib/google/cloud/bigquery/data.rb', line 152

def schema
  return nil unless @table_gapi
  Schema.from_gapi(@table_gapi.schema).freeze
end

#statement_type ⇒ String?

The type of query statement, if valid. Possible values (new values might be added in the future):

Returns:

  • (String, nil) —

    The type of query statement.



251
252
253
 
# File 'lib/google/cloud/bigquery/data.rb', line 251

def statement_type
  job_gapi&.statistics&.query&.statement_type
end

#token ⇒ String

A token used for paging results. Used by the data instance to retrieve subsequent pages. See #next.

Returns:

  • (String) —

    The pagination token.



98
99
100
 
# File 'lib/google/cloud/bigquery/data.rb', line 98

def token
  @gapi_json[:pageToken]
end

#total ⇒ Integer

The total number of rows in the complete table.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

sql = "SELECT word FROM `bigquery-public-data.samples.shakespeare`"
job = bigquery.query_job sql

job.wait_until_done!
data = job.data

data.count # 100000
data.total # 164656
data.each do |row|
  puts row[:word]
end
data = data.next if data.next?

Returns:

  • (Integer) —

    The number of rows.



125
126
127
128
129
 
# File 'lib/google/cloud/bigquery/data.rb', line 125

def total
  Integer @gapi_json[:totalRows]
rescue StandardError
  nil
end