com.google.cloud.spring.bigquery.core

Class BigQueryTemplate

java.lang.Object

com.google.cloud.spring.bigquery.core.BigQueryTemplate

All Implemented Interfaces:

BigQueryOperations

public class BigQueryTemplate
extends Object
implements BigQueryOperations

Helper class which simplifies common operations done in BigQuery.

Since:

1.2

Constructor Summary

Constructors

Constructor and Description
BigQueryTemplate(com.google.cloud.bigquery.BigQuery bigQuery, com.google.cloud.bigquery.storage.v1.BigQueryWriteClient bigQueryWriteClient, Map<String,Object> bqInitSettings, org.springframework.scheduling.TaskScheduler taskScheduler) A Full constructor which creates the BigQuery template.
BigQueryTemplate(com.google.cloud.bigquery.BigQuery bigQuery, String datasetName) Deprecated. As of release 3.3.1, use BigQueryTemplate(BigQuery,BigQueryWriteClient,Map,TaskScheduler) instead
BigQueryTemplate(com.google.cloud.bigquery.BigQuery bigQuery, String datasetName, org.springframework.scheduling.TaskScheduler taskScheduler) Deprecated. As of release 3.3.1, use BigQueryTemplate(BigQuery,BigQueryWriteClient,Map,TaskScheduler) instead

Method Summary

Modifier and Type	Method and Description
com.google.cloud.bigquery.Table	createTable(String tableName, com.google.cloud.bigquery.Schema schema)
BigQueryJsonDataWriter	getBigQueryJsonDataWriter(com.google.cloud.bigquery.storage.v1.TableName parentTable)
com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse	getCommitResponse(com.google.cloud.bigquery.storage.v1.TableName parentTable, BigQueryJsonDataWriter writer)
String	getDatasetName()
int	getJsonWriterBatchSize()
WriteApiResponse	getWriteApiResponse(String tableName, InputStream jsonInputStream)
void	setAutoDetectSchema(boolean autoDetectSchema) Sets whether BigQuery should attempt to autodetect the schema of the data when loading data into an empty table for the first time.
void	setCreateDisposition(com.google.cloud.bigquery.JobInfo.CreateDisposition createDisposition) Sets the JobInfo.CreateDisposition which specifies whether a new table may be created in BigQuery if needed.
void	setJobPollInterval(Duration jobPollInterval) Sets the Duration amount of time to wait between successive polls on the status of a BigQuery job.
void	setWriteDisposition(com.google.cloud.bigquery.JobInfo.WriteDisposition writeDisposition) Sets the JobInfo.WriteDisposition which specifies how data should be inserted into BigQuery tables.
org.springframework.util.concurrent.ListenableFuture<com.google.cloud.bigquery.Job>	writeDataToTable(String tableName, InputStream inputStream, com.google.cloud.bigquery.FormatOptions dataFormatOptions) Writes data to a specified BigQuery table.
org.springframework.util.concurrent.ListenableFuture<com.google.cloud.bigquery.Job>	writeDataToTable(String tableName, InputStream inputStream, com.google.cloud.bigquery.FormatOptions dataFormatOptions, com.google.cloud.bigquery.Schema schema) Writes data to a specified BigQuery table with a manually-specified table Schema.
org.springframework.util.concurrent.ListenableFuture<WriteApiResponse>	writeJsonStream(String tableName, InputStream jsonInputStream) This method uses BigQuery Storage Write API to write new line delimited JSON file to the specified table.
org.springframework.util.concurrent.ListenableFuture<WriteApiResponse>	writeJsonStream(String tableName, InputStream jsonInputStream, com.google.cloud.bigquery.Schema schema) This method uses BigQuery Storage Write API to write new line delimited JSON file to the specified table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigQueryTemplate

@Deprecated
public BigQueryTemplate(com.google.cloud.bigquery.BigQuery bigQuery,
                                    String datasetName)

Deprecated. As of release 3.3.1, use BigQueryTemplate(BigQuery,BigQueryWriteClient,Map,TaskScheduler) instead

Creates the BigQuery template.

Parameters:

bigQuery - the underlying client object used to interface with BigQuery

datasetName - the name of the dataset in which all operations will take place

BigQueryTemplate

@Deprecated
public BigQueryTemplate(com.google.cloud.bigquery.BigQuery bigQuery,
                                    String datasetName,
                                    org.springframework.scheduling.TaskScheduler taskScheduler)

Deprecated. As of release 3.3.1, use BigQueryTemplate(BigQuery,BigQueryWriteClient,Map,TaskScheduler) instead

Creates the BigQuery template.

Parameters:

bigQuery - the underlying client object used to interface with BigQuery

datasetName - the name of the dataset in which all operations will take place

taskScheduler - the TaskScheduler used to poll for the status of long-running BigQuery operations

BigQueryTemplate

public BigQueryTemplate(com.google.cloud.bigquery.BigQuery bigQuery,
                        com.google.cloud.bigquery.storage.v1.BigQueryWriteClient bigQueryWriteClient,
                        Map<String,Object> bqInitSettings,
                        org.springframework.scheduling.TaskScheduler taskScheduler)

A Full constructor which creates the BigQuery template.

Parameters:

bigQuery - the underlying client object used to interface with BigQuery

bigQueryWriteClient - the underlying BigQueryWriteClient reference use to connect with BigQuery Storage Write Client

bqInitSettings - Properties required for initialisation of this class

taskScheduler - the TaskScheduler used to poll for the status of long-running BigQuery operations

Method Detail

setAutoDetectSchema

public void setAutoDetectSchema(boolean autoDetectSchema)

Sets whether BigQuery should attempt to autodetect the schema of the data when loading data into an empty table for the first time. If set to false, the schema must be defined explicitly for the table before load.

Parameters:

autoDetectSchema - whether data schema should be autodetected from the structure of the data. Default is true.

setWriteDisposition

public void setWriteDisposition(com.google.cloud.bigquery.JobInfo.WriteDisposition writeDisposition)

Sets the JobInfo.WriteDisposition which specifies how data should be inserted into BigQuery tables.

Parameters:

writeDisposition - whether to append to or truncate (overwrite) data in the BigQuery table. Default is WriteDisposition.WRITE_APPEND to append data to a table.

setCreateDisposition

public void setCreateDisposition(com.google.cloud.bigquery.JobInfo.CreateDisposition createDisposition)

Sets the JobInfo.CreateDisposition which specifies whether a new table may be created in BigQuery if needed.

Parameters:

createDisposition - whether to never create a new table in the BigQuery table or only if needed.

setJobPollInterval

public void setJobPollInterval(Duration jobPollInterval)

Sets the Duration amount of time to wait between successive polls on the status of a BigQuery job.

Parameters:

jobPollInterval - the Duration poll interval for BigQuery job status polling

writeDataToTable

public org.springframework.util.concurrent.ListenableFuture<com.google.cloud.bigquery.Job> writeDataToTable(String tableName,
                                                                                                            InputStream inputStream,
                                                                                                            com.google.cloud.bigquery.FormatOptions dataFormatOptions)

Description copied from interface: BigQueryOperations

Writes data to a specified BigQuery table.

Specified by:

writeDataToTable in interface BigQueryOperations

Parameters:

tableName - name of the table to write to

inputStream - input stream of the table data to write

dataFormatOptions - the format of the data to write

Returns:

ListenableFuture containing the BigQuery Job indicating completion of operation

writeDataToTable

public org.springframework.util.concurrent.ListenableFuture<com.google.cloud.bigquery.Job> writeDataToTable(String tableName,
                                                                                                            InputStream inputStream,
                                                                                                            com.google.cloud.bigquery.FormatOptions dataFormatOptions,
                                                                                                            com.google.cloud.bigquery.Schema schema)

Description copied from interface: BigQueryOperations

Writes data to a specified BigQuery table with a manually-specified table Schema.

Example:

 Schema schema = Schema.of(
    Field.of("CountyId", StandardSQLTypeName.INT64),
    Field.of("State", StandardSQLTypeName.STRING),
    Field.of("County", StandardSQLTypeName.STRING)
 );

 ListenableFuture<Job> bigQueryJobFuture =
     bigQueryTemplate.writeDataToTable(
          TABLE_NAME, dataFile.getInputStream(), FormatOptions.csv(), schema);
 

Specified by:

writeDataToTable in interface BigQueryOperations

Parameters:

tableName - name of the table to write to

inputStream - input stream of the table data to write

dataFormatOptions - the format of the data to write

schema - the schema of the table being loaded

Returns:

ListenableFuture containing the BigQuery Job indicating completion of operation

writeJsonStream

public org.springframework.util.concurrent.ListenableFuture<WriteApiResponse> writeJsonStream(String tableName,
                                                                                              InputStream jsonInputStream,
                                                                                              com.google.cloud.bigquery.Schema schema)

This method uses BigQuery Storage Write API to write new line delimited JSON file to the specified table. This method creates a table with the specified schema.

Specified by:

writeJsonStream in interface BigQueryOperations

Parameters:

tableName - name of the table to write to

jsonInputStream - input stream of the json file to be written

Returns:

ListenableFuture containing the WriteApiResponse indicating completion of operation

createTable

public com.google.cloud.bigquery.Table createTable(String tableName,
                                                   com.google.cloud.bigquery.Schema schema)

writeJsonStream

public org.springframework.util.concurrent.ListenableFuture<WriteApiResponse> writeJsonStream(String tableName,
                                                                                              InputStream jsonInputStream)

This method uses BigQuery Storage Write API to write new line delimited JSON file to the specified table. The Table should already be created as BigQuery Storage Write API doesn't create it automatically.

Specified by:

writeJsonStream in interface BigQueryOperations

Parameters:

tableName - name of the table to write to

jsonInputStream - input stream of the json file to be written

Returns:

ListenableFuture containing the WriteApiResponse indicating completion of operation

getBigQueryJsonDataWriter

public BigQueryJsonDataWriter getBigQueryJsonDataWriter(com.google.cloud.bigquery.storage.v1.TableName parentTable)
                                                 throws com.google.protobuf.Descriptors.DescriptorValidationException,
                                                        IOException,
                                                        InterruptedException

Throws:

com.google.protobuf.Descriptors.DescriptorValidationException

IOException

InterruptedException

getWriteApiResponse

public WriteApiResponse getWriteApiResponse(String tableName,
                                            InputStream jsonInputStream)
                                     throws com.google.protobuf.Descriptors.DescriptorValidationException,
                                            IOException,
                                            InterruptedException

Throws:

com.google.protobuf.Descriptors.DescriptorValidationException

IOException

InterruptedException

getCommitResponse

public com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse getCommitResponse(com.google.cloud.bigquery.storage.v1.TableName parentTable,
                                                                                              BigQueryJsonDataWriter writer)

getDatasetName

public String getDatasetName()

getJsonWriterBatchSize

public int getJsonWriterBatchSize()