Class: Google::Apis::DataflowV1b3::WorkerPool

Inherits:
Object
  • Object
show all
Includes:
Core::Hashable, Core::JsonObjectSupport
Defined in:
generated/google/apis/dataflow_v1b3/classes.rb,
generated/google/apis/dataflow_v1b3/representations.rb,
generated/google/apis/dataflow_v1b3/representations.rb

Overview

Describes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Core::JsonObjectSupport

#to_json

Methods included from Core::Hashable

process_value, #to_h

Constructor Details

#initialize(**args) ⇒ WorkerPool

Returns a new instance of WorkerPool



1198
1199
1200
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1198

def initialize(**args)
   update!(**args)
end

Instance Attribute Details

#autoscaling_settings ⇒ Google::Apis::DataflowV1b3::AutoscalingSettings

Settings for WorkerPool autoscaling. Corresponds to the JSON property autoscalingSettings

Returns:



1153
1154
1155
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1153

def autoscaling_settings
  @autoscaling_settings
end

#data_disks ⇒ Array<Google::Apis::DataflowV1b3::Disk>

Data disks that are used by a VM in this workflow. Corresponds to the JSON property dataDisks

Returns:



1132
1133
1134
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1132

def data_disks
  @data_disks
end

#default_package_set ⇒ String

The default package set to install. This allows the service to select a default set of packages which are useful to worker harnesses written in a particular language. Corresponds to the JSON property defaultPackageSet

Returns:

  • (String) 


1165
1166
1167
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1165

def default_package_set
  @default_package_set
end

#disk_size_gb ⇒ Fixnum

Size of root disk for VMs, in GB. If zero or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property diskSizeGb

Returns:

  • (Fixnum) 


1103
1104
1105
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1103

def disk_size_gb
  @disk_size_gb
end

#disk_source_image ⇒ String

Fully qualified source image for disks. Corresponds to the JSON property diskSourceImage

Returns:

  • (String) 


1196
1197
1198
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1196

def disk_source_image
  @disk_source_image
end

#disk_type ⇒ String

Type of root disk for VMs. If empty or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property diskType

Returns:

  • (String) 


1121
1122
1123
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1121

def disk_type
  @disk_type
end

#ip_configuration ⇒ String

Configuration for VM IPs. Corresponds to the JSON property ipConfiguration

Returns:

  • (String) 


1143
1144
1145
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1143

def ip_configuration
  @ip_configuration
end

#kind ⇒ String

The kind of the worker pool; currently only harness and shuffle are supported. Corresponds to the JSON property kind

Returns:

  • (String) 


1127
1128
1129
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1127

def kind
  @kind
end

#machine_type ⇒ String

Machine type (e.g. "n1-standard-1"). If empty or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property machineType

Returns:

  • (String) 


1115
1116
1117
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1115

def machine_type
  @machine_type
end

#metadata ⇒ Hash<String,String>

Metadata to set on the Google Compute Engine VMs. Corresponds to the JSON property metadata

Returns:

  • (Hash<String,String>) 


1158
1159
1160
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1158

def metadata
  @metadata
end

#network ⇒ String

Network to which VMs will be assigned. If empty or unspecified, the service will use the network "default". Corresponds to the JSON property network

Returns:

  • (String) 


1171
1172
1173
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1171

def network
  @network
end

#num_threads_per_worker ⇒ Fixnum

The number of threads per worker harness. If empty or unspecified, the service will choose a number of threads (according to the number of cores on the selected machine type for batch, or 1 by convention for streaming). Corresponds to the JSON property numThreadsPerWorker

Returns:

  • (Fixnum) 


1191
1192
1193
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1191

def num_threads_per_worker
  @num_threads_per_worker
end

#num_workers ⇒ Fixnum

Number of Google Compute Engine workers in this pool needed to execute the job. If zero or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property numWorkers

Returns:

  • (Fixnum) 


1184
1185
1186
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1184

def num_workers
  @num_workers
end

#on_host_maintenance ⇒ String

The action to take on host maintenance, as defined by the Google Compute Engine API. Corresponds to the JSON property onHostMaintenance

Returns:

  • (String) 


1092
1093
1094
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1092

def on_host_maintenance
  @on_host_maintenance
end

#packages ⇒ Array<Google::Apis::DataflowV1b3::Package>

Packages to be installed on workers. Corresponds to the JSON property packages

Returns:



1068
1069
1070
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1068

def packages
  @packages
end

#pool_args ⇒ Hash<String,Object>

Extra arguments for this worker pool. Corresponds to the JSON property poolArgs

Returns:

  • (Hash<String,Object>) 


1097
1098
1099
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1097

def pool_args
  @pool_args
end

#subnetwork ⇒ String

Subnetwork to which VMs will be assigned, if desired. Expected to be of the form "regions/REGION/subnetworks/SUBNETWORK". Corresponds to the JSON property subnetwork

Returns:

  • (String) 


1138
1139
1140
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1138

def subnetwork
  @subnetwork
end

#taskrunner_settings ⇒ Google::Apis::DataflowV1b3::TaskRunnerSettings

Taskrunner configuration settings. Corresponds to the JSON property taskrunnerSettings

Returns:



1148
1149
1150
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1148

def taskrunner_settings
  @taskrunner_settings
end

#teardown_policy ⇒ String

Sets the policy for determining when to turndown worker pool. Allowed values are: TEARDOWN_ALWAYS, TEARDOWN_ON_SUCCESS, and TEARDOWN_NEVER. TEARDOWN_ALWAYS means workers are always torn down regardless of whether the job succeeds. TEARDOWN_ON_SUCCESS means workers are torn down if the job succeeds. TEARDOWN_NEVER means the workers are never torn down. If the workers are not torn down by the service, they will continue to run and use Google Compute Engine VM resources in the user's project until they are explicitly terminated by the user. Because of this, Google recommends using the TEARDOWN_ALWAYS policy except for small, manually supervised test jobs. If unknown or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property teardownPolicy

Returns:

  • (String) 


1086
1087
1088
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1086

def teardown_policy
  @teardown_policy
end

#worker_harness_container_image ⇒ String

Required. Docker container image that executes the Cloud Dataflow worker harness, residing in Google Container Registry. Corresponds to the JSON property workerHarnessContainerImage

Returns:

  • (String) 


1109
1110
1111
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1109

def worker_harness_container_image
  @worker_harness_container_image
end

#zone ⇒ String

Zone to run the worker pools in. If empty or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property zone

Returns:

  • (String) 


1177
1178
1179
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1177

def zone
  @zone
end

Instance Method Details

#update!(**args) ⇒ Object

Update properties of this object



1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
 
# File 'generated/google/apis/dataflow_v1b3/classes.rb', line 1203

def update!(**args)
  @packages = args[:packages] if args.key?(:packages)
  @teardown_policy = args[:teardown_policy] if args.key?(:teardown_policy)
  @on_host_maintenance = args[:on_host_maintenance] if args.key?(:on_host_maintenance)
  @pool_args = args[:pool_args] if args.key?(:pool_args)
  @disk_size_gb = args[:disk_size_gb] if args.key?(:disk_size_gb)
  @worker_harness_container_image = args[:worker_harness_container_image] if args.key?(:worker_harness_container_image)
  @machine_type = args[:machine_type] if args.key?(:machine_type)
  @disk_type = args[:disk_type] if args.key?(:disk_type)
  @kind = args[:kind] if args.key?(:kind)
  @data_disks = args[:data_disks] if args.key?(:data_disks)
  @subnetwork = args[:subnetwork] if args.key?(:subnetwork)
  @ip_configuration = args[:ip_configuration] if args.key?(:ip_configuration)
  @taskrunner_settings = args[:taskrunner_settings] if args.key?(:taskrunner_settings)
  @autoscaling_settings = args[:autoscaling_settings] if args.key?(:autoscaling_settings)
  @metadata = args[:metadata] if args.key?(:metadata)
  @default_package_set = args[:default_package_set] if args.key?(:default_package_set)
  @network = args[:network] if args.key?(:network)
  @zone = args[:zone] if args.key?(:zone)
  @num_workers = args[:num_workers] if args.key?(:num_workers)
  @num_threads_per_worker = args[:num_threads_per_worker] if args.key?(:num_threads_per_worker)
  @disk_source_image = args[:disk_source_image] if args.key?(:disk_source_image)
end