v1

v1

Source:

Members

(static) AuthorizationType :number

The type of authorization needed for this data source.

Properties:
Name Type Description
AUTHORIZATION_TYPE_UNSPECIFIED number

Type unspecified.
AUTHORIZATION_CODE number

Use OAuth 2 authorization codes that can be exchanged for a refresh token on the backend.
GOOGLE_PLUS_AUTHORIZATION_CODE number

Return an authorization code for a given Google+ page that can then be exchanged for a refresh token on the backend.
Source:

(static) DataRefreshType :number

Represents how the data source supports data auto refresh.

Properties:
Name Type Description
DATA_REFRESH_TYPE_UNSPECIFIED number

The data source won't support data auto refresh, which is default value.
SLIDING_WINDOW number

The data source supports data auto refresh, and runs will be scheduled for the past few days. Does not allow custom values to be set for each transfer config.
CUSTOM_SLIDING_WINDOW number

The data source supports data auto refresh, and runs will be scheduled for the past few days. Allows custom values to be set for each transfer config.
Source:

(static) MessageSeverity :number

Represents data transfer user facing message severity.

Properties:
Name Type Description
MESSAGE_SEVERITY_UNSPECIFIED number

No severity specified.
INFO number

Informational message.
WARNING number

Warning message.
ERROR number

Error message.
Source:

(static) RunAttempt :number

Represents which runs should be pulled.

Properties:
Name Type Description
RUN_ATTEMPT_UNSPECIFIED number

All runs should be returned.
LATEST number

Only latest run per day should be returned.
Source:

(static, constant) TransferState :number

Represents data transfer run state.

Properties:
Name Type Description
TRANSFER_STATE_UNSPECIFIED number

State placeholder.
PENDING number

Data transfer is scheduled and is waiting to be picked up by data transfer backend.
RUNNING number

Data transfer is in progress.
SUCCEEDED number

Data transfer completed successfully.
FAILED number

Data transfer failed.
CANCELLED number

Data transfer is cancelled.
Source:

(static, constant) TransferType :number

DEPRECATED. Represents data transfer type.

Properties:
Name Type Description
TRANSFER_TYPE_UNSPECIFIED number

Invalid or Unknown transfer type placeholder.
BATCH number

Batch data transfer.
STREAMING number

Streaming data transfer. Streaming data source currently doesn't support multiple transfer configs per project.
Source:

(static) Type :number

Parameter type.

Properties:
Name Type Description
TYPE_UNSPECIFIED number

Type unspecified.
STRING number

String parameter.
INTEGER number

Integer parameter (64-bits). Will be serialized to json as string.
DOUBLE number

Double precision floating point parameter.
BOOLEAN number

Boolean parameter.
RECORD number

Deprecated. This field has no effect.
PLUS_PAGE number

Page ID for a Google+ Page.
Source:

Type Definitions

CheckValidCredsRequest

A request to determine whether the user has valid credentials. This method is used to limit the number of OAuth popups in the user interface. The user id is inferred from the API call context. If the data source has the Google+ authorization type, this method returns false, as it cannot be determined whether the credentials are already valid merely based on the user id.

Properties:
Name Type Description
name string

The data source in the form: projects/{project_id}/dataSources/{data_source_id}
Source:
See:

CheckValidCredsResponse

A response indicating whether the credentials exist and are valid.

Properties:
Name Type Description
hasValidCreds boolean

If set to true, the credentials exist and are valid.
Source:
See:

CreateTransferConfigRequest

A request to create a data transfer configuration. If new credentials are needed for this transfer configuration, an authorization code must be provided. If an authorization code is provided, the transfer configuration will be associated with the user id corresponding to the authorization code. Otherwise, the transfer configuration will be associated with the calling user.

Properties:
Name Type Description
parent string

The BigQuery project id where the transfer configuration should be created. Must be in the format projects/{project_id}/locations/{location_id} If specified location and location of the destination bigquery dataset do not match - the request will fail.
transferConfig Object

Data transfer configuration to create.

This object should have the same structure as TransferConfig
authorizationCode string

Optional OAuth2 authorization code to use with this transfer configuration. This is required if new credentials are needed, as indicated by CheckValidCreds. In order to obtain authorization_code, please make a request to https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id=&scope=<data_source_scopes>&redirect_uri=<redirect_uri>

  • client_id should be OAuth client_id of BigQuery DTS API for the given data source returned by ListDataSources method.
  • data_source_scopes are the scopes returned by ListDataSources method.
  • redirect_uri is an optional parameter. If not specified, then authorization code is posted to the opener of authorization flow window. Otherwise it will be sent to the redirect uri. A special value of urn:ietf:wg:oauth:2.0:oob means that authorization code should be returned in the title bar of the browser, with the page text prompting the user to copy the code and paste it in the application.
versionInfo string

Optional version info. If users want to find a very recent access token, that is, immediately after approving access, users have to set the version_info claim in the token request. To obtain the version_info, users must use the "none+gsession" response type. which be return a version_info back in the authorization response which be be put in a JWT claim in the token request.
Source:
See:

DataSource

Represents data source metadata. Metadata is sufficient to render UI and request proper OAuth tokens.

Properties:
Name Type Description
name string

Output only. Data source resource name.
dataSourceId string

Data source id.
displayName string

User friendly data source name.
description string

User friendly data source description string.
clientId string

Data source client id which should be used to receive refresh token.
scopes Array.<string>

Api auth scopes for which refresh token needs to be obtained. These are scopes needed by a data source to prepare data and ingest them into BigQuery, e.g., https://www.googleapis.com/auth/bigquery
transferType number

Deprecated. This field has no effect.

The number should be among the values of TransferType
supportsMultipleTransfers boolean

Deprecated. This field has no effect.
updateDeadlineSeconds number

The number of seconds to wait for an update from the data source before the Data Transfer Service marks the transfer as FAILED.
defaultSchedule string

Default data transfer schedule. Examples of valid schedules include: 1st,3rd monday of month 15:30, every wed,fri of jan,jun 13:15, and first sunday of quarter 00:00.
supportsCustomSchedule boolean

Specifies whether the data source supports a user defined schedule, or operates on the default schedule. When set to true, user can override default schedule.
parameters Array.<Object>

Data source parameters.

This object should have the same structure as DataSourceParameter
helpUrl string

Url for the help document for this data source.
authorizationType number

Indicates the type of authorization.

The number should be among the values of AuthorizationType
dataRefreshType number

Specifies whether the data source supports automatic data refresh for the past few days, and how it's supported. For some data sources, data might not be complete until a few days later, so it's useful to refresh data automatically.

The number should be among the values of DataRefreshType
defaultDataRefreshWindowDays number

Default data refresh window on days. Only meaningful when data_refresh_type = SLIDING_WINDOW.
manualRunsDisabled boolean

Disables backfilling and manual run scheduling for the data source.
minimumScheduleInterval Object

The minimum interval for scheduler to schedule runs.

This object should have the same structure as Duration
Source:
See:

DataSourceParameter

Represents a data source parameter with validation rules, so that parameters can be rendered in the UI. These parameters are given to us by supported data sources, and include all needed information for rendering and validation. Thus, whoever uses this api can decide to generate either generic ui, or custom data source specific forms.

Properties:
Name Type Description
paramId string

Parameter identifier.
displayName string

Parameter display name in the user interface.
description string

Parameter description.
type number

Parameter type.

The number should be among the values of Type
required boolean

Is parameter required.
repeated boolean

Deprecated. This field has no effect.
validationRegex string

Regular expression which can be used for parameter validation.
allowedValues Array.<string>

All possible values for the parameter.
minValue Object

For integer and double values specifies minimum allowed value.

This object should have the same structure as DoubleValue
maxValue Object

For integer and double values specifies maxminum allowed value.

This object should have the same structure as DoubleValue
fields Array.<Object>

Deprecated. This field has no effect.

This object should have the same structure as DataSourceParameter
validationDescription string

Description of the requirements for this field, in case the user input does not fulfill the regex pattern or min/max values.
validationHelpUrl string

URL to a help document to further explain the naming requirements.
immutable boolean

Cannot be changed after initial creation.
recurse boolean

Deprecated. This field has no effect.
deprecated boolean

If true, it should not be used in new transfers, and it should not be visible to users.
Source:
See:

DeleteTransferConfigRequest

A request to delete data transfer information. All associated transfer runs and log messages will be deleted as well.

Properties:
Name Type Description
name string

The field will contain name of the resource requested, for example: projects/{project_id}/transferConfigs/{config_id}
Source:
See:

DeleteTransferRunRequest

A request to delete data transfer run information.

Properties:
Name Type Description
name string

The field will contain name of the resource requested, for example: projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}
Source:
See:

GetDataSourceRequest

A request to get data source info.

Properties:
Name Type Description
name string

The field will contain name of the resource requested, for example: projects/{project_id}/dataSources/{data_source_id}
Source:
See:

GetTransferConfigRequest

A request to get data transfer information.

Properties:
Name Type Description
name string

The field will contain name of the resource requested, for example: projects/{project_id}/transferConfigs/{config_id}
Source:
See:

GetTransferRunRequest

A request to get data transfer run information.

Properties:
Name Type Description
name string

The field will contain name of the resource requested, for example: projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}
Source:
See:

ListDataSourcesRequest

Request to list supported data sources and their data transfer settings.

Properties:
Name Type Description
parent string

The BigQuery project id for which data sources should be returned. Must be in the form: projects/{project_id}
pageToken string

Pagination token, which can be used to request a specific page of ListDataSourcesRequest list results. For multiple-page results, ListDataSourcesResponse outputs a next_page token, which can be used as the page_token value to request the next page of list results.
pageSize number

Page size. The default page size is the maximum value of 1000 results.
Source:
See:

ListDataSourcesResponse

Returns list of supported data sources and their metadata.

Properties:
Name Type Description
dataSources Array.<Object>

List of supported data sources and their transfer settings.

This object should have the same structure as DataSource
nextPageToken string

Output only. The next-pagination token. For multiple-page list results, this token can be used as the ListDataSourcesRequest.page_token to request the next page of list results.
Source:
See:

ListTransferConfigsRequest

A request to list data transfers configured for a BigQuery project.

Properties:
Name Type Description
parent string

The BigQuery project id for which data sources should be returned: projects/{project_id}.
dataSourceIds Array.<string>

When specified, only configurations of requested data sources are returned.
pageToken string

Pagination token, which can be used to request a specific page of ListTransfersRequest list results. For multiple-page results, ListTransfersResponse outputs a next_page token, which can be used as the page_token value to request the next page of list results.
pageSize number

Page size. The default page size is the maximum value of 1000 results.
Source:
See:

ListTransferConfigsResponse

The returned list of pipelines in the project.

Properties:
Name Type Description
transferConfigs Array.<Object>

Output only. The stored pipeline transfer configurations.

This object should have the same structure as TransferConfig
nextPageToken string

Output only. The next-pagination token. For multiple-page list results, this token can be used as the ListTransferConfigsRequest.page_token to request the next page of list results.
Source:
See:

ListTransferLogsRequest

A request to get user facing log messages associated with data transfer run.

Properties:
Name Type Description
parent string

Transfer run name in the form: projects/{project_id}/transferConfigs/{config_Id}/runs/{run_id}.
pageToken string

Pagination token, which can be used to request a specific page of ListTransferLogsRequest list results. For multiple-page results, ListTransferLogsResponse outputs a next_page token, which can be used as the page_token value to request the next page of list results.
pageSize number

Page size. The default page size is the maximum value of 1000 results.
messageTypes Array.<number>

Message types to return. If not populated - INFO, WARNING and ERROR messages are returned.

The number should be among the values of MessageSeverity
Source:
See:

ListTransferLogsResponse

The returned list transfer run messages.

Properties:
Name Type Description
transferMessages Array.<Object>

Output only. The stored pipeline transfer messages.

This object should have the same structure as TransferMessage
nextPageToken string

Output only. The next-pagination token. For multiple-page list results, this token can be used as the GetTransferRunLogRequest.page_token to request the next page of list results.
Source:
See:

ListTransferRunsRequest

A request to list data transfer runs. UI can use this method to show/filter specific data transfer runs. The data source can use this method to request all scheduled transfer runs.

Properties:
Name Type Description
parent string

Name of transfer configuration for which transfer runs should be retrieved. Format of transfer configuration resource name is: projects/{project_id}/transferConfigs/{config_id}.
states Array.<number>

When specified, only transfer runs with requested states are returned.

The number should be among the values of TransferState
pageToken string

Pagination token, which can be used to request a specific page of ListTransferRunsRequest list results. For multiple-page results, ListTransferRunsResponse outputs a next_page token, which can be used as the page_token value to request the next page of list results.
pageSize number

Page size. The default page size is the maximum value of 1000 results.
runAttempt number

Indicates how run attempts are to be pulled.

The number should be among the values of RunAttempt
Source:
See:

ListTransferRunsResponse

The returned list of pipelines in the project.

Properties:
Name Type Description
transferRuns Array.<Object>

Output only. The stored pipeline transfer runs.

This object should have the same structure as TransferRun
nextPageToken string

Output only. The next-pagination token. For multiple-page list results, this token can be used as the ListTransferRunsRequest.page_token to request the next page of list results.
Source:
See:

ScheduleOptions

Options customizing the data transfer schedule.

Properties:
Name Type Description
disableAutoScheduling boolean

If true, automatic scheduling of data transfer runs for this configuration will be disabled. The runs can be started on ad-hoc basis using StartManualTransferRuns API. When automatic scheduling is disabled, the TransferConfig.schedule field will be ignored.
startTime Object

Specifies time to start scheduling transfer runs. The first run will be scheduled at or after the start time according to a recurrence pattern defined in the schedule string. The start time can be changed at any moment. The time when a data transfer can be trigerred manually is not limited by this option.

This object should have the same structure as Timestamp
endTime Object

Defines time to stop scheduling transfer runs. A transfer run cannot be scheduled at or after the end time. The end time can be changed at any moment. The time when a data transfer can be trigerred manually is not limited by this option.

This object should have the same structure as Timestamp
Source:
See:

ScheduleTransferRunsRequest

A request to schedule transfer runs for a time range.

Properties:
Name Type Description
parent string

Transfer configuration name in the form: projects/{project_id}/transferConfigs/{config_id}.
startTime Object

Start time of the range of transfer runs. For example, "2017-05-25T00:00:00+00:00".

This object should have the same structure as Timestamp
endTime Object

End time of the range of transfer runs. For example, "2017-05-30T00:00:00+00:00".

This object should have the same structure as Timestamp
Source:
See:

ScheduleTransferRunsResponse

A response to schedule transfer runs for a time range.

Properties:
Name Type Description
runs Array.<Object>

The transfer runs that were scheduled.

This object should have the same structure as TransferRun
Source:
See:

StartManualTransferRunsRequest

A request to start manual transfer runs.

Properties:
Name Type Description
parent string

Transfer configuration name in the form: projects/{project_id}/transferConfigs/{config_id}.
requestedTimeRange Object

Time range for the transfer runs that should be started.

This object should have the same structure as TimeRange
requestedRunTime Object

Specific run_time for a transfer run to be started. The requested_run_time must not be in the future.

This object should have the same structure as Timestamp
Source:
See:

StartManualTransferRunsResponse

A response to start manual transfer runs.

Properties:
Name Type Description
runs Array.<Object>

The transfer runs that were created.

This object should have the same structure as TransferRun
Source:
See:

TimeRange

A specification for a time range, this will request transfer runs with run_time between start_time (inclusive) and end_time (exclusive).

Properties:
Name Type Description
startTime Object

Start time of the range of transfer runs. For example, "2017-05-25T00:00:00+00:00". The start_time must be strictly less than the end_time. Creates transfer runs where run_time is in the range betwen start_time (inclusive) and end_time (exlusive).

This object should have the same structure as Timestamp
endTime Object

End time of the range of transfer runs. For example, "2017-05-30T00:00:00+00:00". The end_time must not be in the future. Creates transfer runs where run_time is in the range betwen start_time (inclusive) and end_time (exlusive).

This object should have the same structure as Timestamp
Source:
See:

TransferConfig

Represents a data transfer configuration. A transfer configuration contains all metadata needed to perform a data transfer. For example, destination_dataset_id specifies where data should be stored. When a new transfer configuration is created, the specified destination_dataset_id is created when needed and shared with the appropriate data source service account.

Properties:
Name Type Description
name string

The resource name of the transfer config. Transfer config names have the form of projects/{project_id}/locations/{region}/transferConfigs/{config_id}. The name is automatically generated based on the config_id specified in CreateTransferConfigRequest along with project_id and region. If config_id is not provided, usually a uuid, even though it is not guaranteed or required, will be generated for config_id.
destinationDatasetId string

The BigQuery target dataset id.
displayName string

User specified display name for the data transfer.
dataSourceId string

Data source id. Cannot be changed once data transfer is created.
params Object

Data transfer specific parameters.

This object should have the same structure as Struct
schedule string

Data transfer schedule. If the data source does not support a custom schedule, this should be empty. If it is empty, the default value for the data source will be used. The specified times are in UTC. Examples of valid format: 1st,3rd monday of month 15:30, every wed,fri of jan,jun 13:15, and first sunday of quarter 00:00. See more explanation about the format here: https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml#the_schedule_format NOTE: the granularity should be at least 8 hours, or less frequent.
scheduleOptions Object

Options customizing the data transfer schedule.

This object should have the same structure as ScheduleOptions
dataRefreshWindowDays number

The number of days to look back to automatically refresh the data. For example, if data_refresh_window_days = 10, then every day BigQuery reingests data for [today-10, today-1], rather than ingesting data for just [today-1]. Only valid if the data source supports the feature. Set the value to 0 to use the default value.
disabled boolean

Is this config disabled. When set to true, no runs are scheduled for a given transfer.
updateTime Object

Output only. Data transfer modification time. Ignored by server on input.

This object should have the same structure as Timestamp
nextRunTime Object

Output only. Next time when data transfer will run.

This object should have the same structure as Timestamp
state number

Output only. State of the most recently updated transfer run.

The number should be among the values of TransferState
userId number

Deprecated. Unique ID of the user on whose behalf transfer is done.
datasetRegion string

Output only. Region in which BigQuery dataset is located.
Source:
See:

TransferMessage

Represents a user facing message for a particular data transfer run.

Properties:
Name Type Description
messageTime Object

Time when message was logged.

This object should have the same structure as Timestamp
severity number

Message severity.

The number should be among the values of MessageSeverity
messageText string

Message text.
Source:
See:

TransferRun

Represents a data transfer run.

Properties:
Name Type Description
name string

The resource name of the transfer run. Transfer run names have the form projects/{project_id}/locations/{location}/transferConfigs/{config_id}/runs/{run_id}. The name is ignored when creating a transfer run.
scheduleTime Object

Minimum time after which a transfer run can be started.

This object should have the same structure as Timestamp
runTime Object

For batch transfer runs, specifies the date and time of the data should be ingested.

This object should have the same structure as Timestamp
errorStatus Object

Status of the transfer run.

This object should have the same structure as Status
startTime Object

Output only. Time when transfer run was started. Parameter ignored by server for input requests.

This object should have the same structure as Timestamp
endTime Object

Output only. Time when transfer run ended. Parameter ignored by server for input requests.

This object should have the same structure as Timestamp
updateTime Object

Output only. Last time the data transfer run state was updated.

This object should have the same structure as Timestamp
params Object

Output only. Data transfer specific parameters.

This object should have the same structure as Struct
destinationDatasetId string

Output only. The BigQuery target dataset id.
dataSourceId string

Output only. Data source id.
state number

Data transfer run state. Ignored for input requests.

The number should be among the values of TransferState
userId number

Deprecated. Unique ID of the user on whose behalf transfer is done.
schedule string

Output only. Describes the schedule of this transfer run if it was created as part of a regular schedule. For batch transfer runs that are scheduled manually, this is empty. NOTE: the system might choose to delay the schedule depending on the current load, so schedule_time doesn't always match this.
Source:
See:

UpdateTransferConfigRequest

A request to update a transfer configuration. To update the user id of the transfer configuration, an authorization code needs to be provided.

Properties:
Name Type Description
transferConfig Object

Data transfer configuration to create.

This object should have the same structure as TransferConfig
authorizationCode string

Optional OAuth2 authorization code to use with this transfer configuration. If it is provided, the transfer configuration will be associated with the authorizing user. In order to obtain authorization_code, please make a request to https://www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id=&scope=<data_source_scopes>&redirect_uri=<redirect_uri>

  • client_id should be OAuth client_id of BigQuery DTS API for the given data source returned by ListDataSources method.
  • data_source_scopes are the scopes returned by ListDataSources method.
  • redirect_uri is an optional parameter. If not specified, then authorization code is posted to the opener of authorization flow window. Otherwise it will be sent to the redirect uri. A special value of urn:ietf:wg:oauth:2.0:oob means that authorization code should be returned in the title bar of the browser, with the page text prompting the user to copy the code and paste it in the application.
updateMask Object

Required list of fields to be updated in this request.

This object should have the same structure as FieldMask
versionInfo string

Optional version info. If users want to find a very recent access token, that is, immediately after approving access, users have to set the version_info claim in the token request. To obtain the version_info, users must use the "none+gsession" response type. which be return a version_info back in the authorization response which be be put in a JWT claim in the token request.
Source:
See: