BigQueryWriteClient

BigQueryWriteClient

BigQuery Write API.

The Write API can be used to write data to BigQuery.

For supplementary information about the Write API, see: https://cloud.google.com/bigquery/docs/write-api

Constructor

new BigQueryWriteClient(optionsopt, gaxInstanceopt)

Construct an instance of BigQueryWriteClient.

Parameters:
Name Type Attributes Description
options object <optional>

The configuration object. The options accepted by the constructor are described in detail in this document. The common options are:

Properties
Name Type Attributes Description
credentials object <optional>

Credentials object.

Properties
Name Type Attributes Description
client_email string <optional>
private_key string <optional>
email string <optional>

Account email address. Required when using a .pem or .p12 keyFilename.

keyFilename string <optional>

Full path to the a .json, .pem, or .p12 key downloaded from the Google Developers Console. If you provide a path to a JSON file, the projectId option below is not necessary. NOTE: .pem and .p12 require you to specify options.email as well.

port number <optional>

The port on which to connect to the remote host.

projectId string <optional>

The project ID from the Google Developer's Console, e.g. 'grape-spaceship-123'. We will also check the environment variable GCLOUD_PROJECT for your project ID. If your app is running in an environment which supports Application Default Credentials, your project ID will be detected automatically.

apiEndpoint string <optional>

The domain name of the API remote host.

clientConfig gax.ClientConfig <optional>

Client configuration override. Follows the structure of gapicConfig.

fallback boolean <optional>

Use HTTP/1.1 REST mode. For more information, please check the documentation.

gaxInstance gax <optional>

loaded instance of google-gax. Useful if you need to avoid loading the default gRPC version and want to use the fallback HTTP implementation. Load only fallback version and pass it to the constructor: const gax = require('google-gax/build/src/fallback'); // avoids loading google-gax with gRPC const client = new BigQueryWriteClient({fallback: true}, gax);

Members

apiEndpoint

The DNS address for this API service.

apiEndpoint

The DNS address for this API service - same as servicePath.

port

The port for this API service.

scopes

The scopes needed to make gRPC calls for every method defined in this service.

servicePath

The DNS address for this API service.

Methods

appendRows(optionsopt) → {Stream}

Appends data to the given stream.

If offset is specified, the offset is checked against the end of stream. The server returns OUT_OF_RANGE in AppendRowsResponse if an attempt is made to append to an offset beyond the current end of the stream or ALREADY_EXISTS if user provides an offset that has already been written to. User can retry with adjusted offset within the same RPC connection. If offset is not specified, append happens at the end of the stream.

The response contains an optional offset at which the append happened. No offset information will be returned for appends to a default stream.

Responses are received in the same order in which requests are sent. There will be one response for each successful inserted request. Responses may optionally embed error information if the originating AppendRequest was not successfully processed.

The specifics of when successfully appended data is made visible to the table are governed by the type of stream:

  • For COMMITTED streams (which includes the default stream), data is visible immediately upon successful append.

  • For BUFFERED streams, data is made visible via a subsequent FlushRows rpc which advances a cursor to a newer offset in the stream.

  • For PENDING streams, data is not made visible until the stream itself is finalized (via the FinalizeWriteStream rpc), and the stream is explicitly committed via the BatchCommitWriteStreams rpc.

Parameters:
Name Type Attributes Description
options object <optional>

Call options. See CallOptions for more details.

Returns:
Type Description
Stream

An object stream which is both readable and writable. It accepts objects representing AppendRowsRequest for write() method, and will emit objects representing AppendRowsResponse on 'data' event asynchronously. Please see the documentation for more details and examples.

Example
  /**
   * This snippet has been automatically generated and should be regarded as a code template only.
   * It will require modifications to work.
   * It may require correct/in-range values for request initialization.
   * TODO(developer): Uncomment these variables before running the sample.
   */
  /**
   *  Required. The write_stream identifies the append operation. It must be
   *  provided in the following scenarios:
   *  * In the first request to an AppendRows connection.
   *  * In all subsequent requests to an AppendRows connection, if you use the
   *  same connection to write to multiple tables or change the input schema for
   *  default streams.
   *  For explicitly created write streams, the format is:
   *  * `projects/{project}/datasets/{dataset}/tables/{table}/streams/{id}`
   *  For the special default stream, the format is:
   *  * `projects/{project}/datasets/{dataset}/tables/{table}/streams/_default`.
   *  An example of a possible sequence of requests with write_stream fields
   *  within a single connection:
   *  * r1: {write_stream: stream_name_1}
   *  * r2: {write_stream: /*omit* /}
   *  * r3: {write_stream: /*omit* /}
   *  * r4: {write_stream: stream_name_2}
   *  * r5: {write_stream: stream_name_2}
   *  The destination changed in request_4, so the write_stream field must be
   *  populated in all subsequent requests in this stream.
   */
  // const writeStream = 'abc123'
  /**
   *  If present, the write is only performed if the next append offset is same
   *  as the provided value. If not present, the write is performed at the
   *  current end of stream. Specifying a value for this field is not allowed
   *  when calling AppendRows for the '_default' stream.
   */
  // const offset = {}
  /**
   *  Rows in proto format.
   */
  // const protoRows = {}
  /**
   *  Id set by client to annotate its identity. Only initial request setting is
   *  respected.
   */
  // const traceId = 'abc123'
  /**
   *  A map to indicate how to interpret missing value for some fields. Missing
   *  values are fields present in user schema but missing in rows. The key is
   *  the field name. The value is the interpretation of missing values for the
   *  field.
   *  For example, a map {'foo': NULL_VALUE, 'bar': DEFAULT_VALUE} means all
   *  missing values in field foo are interpreted as NULL, all missing values in
   *  field bar are interpreted as the default value of field bar in table
   *  schema.
   *  If a field is not in this map and has missing values, the missing values
   *  in this field are interpreted as NULL.
   *  This field only applies to the current request, it won't affect other
   *  requests on the connection.
   *  Currently, field name can only be top-level column name, can't be a struct
   *  field path like 'foo.bar'.
   */
  // const missingValueInterpretations = [1,2,3,4]
  /**
   *  Optional. Default missing value interpretation for all columns in the
   *  table. When a value is specified on an `AppendRowsRequest`, it is applied
   *  to all requests on the connection from that point forward, until a
   *  subsequent `AppendRowsRequest` sets it to a different value.
   *  `missing_value_interpretation` can override
   *  `default_missing_value_interpretation`. For example, if you want to write
   *  `NULL` instead of using default values for some columns, you can set
   *  `default_missing_value_interpretation` to `DEFAULT_VALUE` and at the same
   *  time, set `missing_value_interpretations` to `NULL_VALUE` on those columns.
   */
  // const defaultMissingValueInterpretation = {}

  // Imports the Storage library
  const {BigQueryWriteClient} = require('@google-cloud/bigquery-storage').v1;

  // Instantiates a client
  const storageClient = new BigQueryWriteClient();

  async function callAppendRows() {
    // Construct request
    const request = {
      writeStream,
    };

    // Run request
    const stream = await storageClient.appendRows();
    stream.on('data', (response) => { console.log(response) });
    stream.on('error', (err) => { throw(err) });
    stream.on('end', () => { /* API call completed */ });
    stream.write(request);
    stream.end();
  }

  callAppendRows();

close() → {Promise}

Terminate the gRPC channel and close the client.

The client will no longer be usable and all future behavior is undefined.

Returns:
Type Description
Promise

A promise that resolves when the client is closed.

getProjectId() → {Promise}

Return the project ID used by this class.

Returns:
Type Description
Promise

A promise that resolves to string containing the project ID.

initialize() → {Promise}

Initialize the client. Performs asynchronous operations (such as authentication) and prepares the client. This function will be called automatically when any class method is called for the first time, but if you need to initialize it before calling an actual method, feel free to call initialize() directly.

You can await on this method if you want to make sure the client is initialized.

Returns:
Type Description
Promise

A promise that resolves to an authenticated service stub.

matchDatasetFromTableName(tableName) → {string}

Parse the dataset from Table resource.

Parameters:
Name Type Description
tableName string

A fully-qualified path representing Table resource.

Returns:
Type Description
string

A string representing the dataset.

matchDatasetFromWriteStreamName(writeStreamName) → {string}

Parse the dataset from WriteStream resource.

Parameters:
Name Type Description
writeStreamName string

A fully-qualified path representing WriteStream resource.

Returns:
Type Description
string

A string representing the dataset.

matchLocationFromReadSessionName(readSessionName) → {string}

Parse the location from ReadSession resource.

Parameters:
Name Type Description
readSessionName string

A fully-qualified path representing ReadSession resource.

Returns:
Type Description
string

A string representing the location.

matchLocationFromReadStreamName(readStreamName) → {string}

Parse the location from ReadStream resource.

Parameters:
Name Type Description
readStreamName string

A fully-qualified path representing ReadStream resource.

Returns:
Type Description
string

A string representing the location.

matchProjectFromProjectName(projectName) → {string}

Parse the project from Project resource.

Parameters:
Name Type Description
projectName string

A fully-qualified path representing Project resource.

Returns:
Type Description
string

A string representing the project.

matchProjectFromReadSessionName(readSessionName) → {string}

Parse the project from ReadSession resource.

Parameters:
Name Type Description
readSessionName string

A fully-qualified path representing ReadSession resource.

Returns:
Type Description
string

A string representing the project.

matchProjectFromReadStreamName(readStreamName) → {string}

Parse the project from ReadStream resource.

Parameters:
Name Type Description
readStreamName string

A fully-qualified path representing ReadStream resource.

Returns:
Type Description
string

A string representing the project.

matchProjectFromTableName(tableName) → {string}

Parse the project from Table resource.

Parameters:
Name Type Description
tableName string

A fully-qualified path representing Table resource.

Returns:
Type Description
string

A string representing the project.

matchProjectFromWriteStreamName(writeStreamName) → {string}

Parse the project from WriteStream resource.

Parameters:
Name Type Description
writeStreamName string

A fully-qualified path representing WriteStream resource.

Returns:
Type Description
string

A string representing the project.

matchSessionFromReadSessionName(readSessionName) → {string}

Parse the session from ReadSession resource.

Parameters:
Name Type Description
readSessionName string

A fully-qualified path representing ReadSession resource.

Returns:
Type Description
string

A string representing the session.

matchSessionFromReadStreamName(readStreamName) → {string}

Parse the session from ReadStream resource.

Parameters:
Name Type Description
readStreamName string

A fully-qualified path representing ReadStream resource.

Returns:
Type Description
string

A string representing the session.

matchStreamFromReadStreamName(readStreamName) → {string}

Parse the stream from ReadStream resource.

Parameters:
Name Type Description
readStreamName string

A fully-qualified path representing ReadStream resource.

Returns:
Type Description
string

A string representing the stream.

matchStreamFromWriteStreamName(writeStreamName) → {string}

Parse the stream from WriteStream resource.

Parameters:
Name Type Description
writeStreamName string

A fully-qualified path representing WriteStream resource.

Returns:
Type Description
string

A string representing the stream.

matchTableFromTableName(tableName) → {string}

Parse the table from Table resource.

Parameters:
Name Type Description
tableName string

A fully-qualified path representing Table resource.

Returns:
Type Description
string

A string representing the table.

matchTableFromWriteStreamName(writeStreamName) → {string}

Parse the table from WriteStream resource.

Parameters:
Name Type Description
writeStreamName string

A fully-qualified path representing WriteStream resource.

Returns:
Type Description
string

A string representing the table.

projectPath(project) → {string}

Return a fully-qualified project resource name string.

Parameters:
Name Type Description
project string
Returns:
Type Description
string

Resource name string.

readSessionPath(project, location, session) → {string}

Return a fully-qualified readSession resource name string.

Parameters:
Name Type Description
project string
location string
session string
Returns:
Type Description
string

Resource name string.

readStreamPath(project, location, session, stream) → {string}

Return a fully-qualified readStream resource name string.

Parameters:
Name Type Description
project string
location string
session string
stream string
Returns:
Type Description
string

Resource name string.

tablePath(project, dataset, table) → {string}

Return a fully-qualified table resource name string.

Parameters:
Name Type Description
project string
dataset string
table string
Returns:
Type Description
string

Resource name string.

writeStreamPath(project, dataset, table, stream) → {string}

Return a fully-qualified writeStream resource name string.

Parameters:
Name Type Description
project string
dataset string
table string
stream string
Returns:
Type Description
string

Resource name string.