Class: Google::Apis::DataprocV1beta2::SparkJob

Inherits:
Object
  • Object
show all
Includes:
Core::Hashable, Core::JsonObjectSupport
Defined in:
generated/google/apis/dataproc_v1beta2/classes.rb,
generated/google/apis/dataproc_v1beta2/representations.rb,
generated/google/apis/dataproc_v1beta2/representations.rb

Overview

A Dataproc job for running Apache Spark (http://spark.apache.org/) applications on YARN. The specification of the main method to call to drive the job. Specify either the jar file that contains the main class or the main class name. To pass both a main jar and a main class in that jar, add the jar to CommonJob.jar_file_uris, and then specify the main class name in main_class.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(**args) ⇒ SparkJob

Returns a new instance of SparkJob.



3064
3065
3066
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3064

def initialize(**args)
   update!(**args)
end

Instance Attribute Details

#archive_uris ⇒ Array<String>

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip. Corresponds to the JSON property archiveUris

Returns:

  • (Array<String>) 


3019
3020
3021
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3019

def archive_uris
  @archive_uris
end

#args ⇒ Array<String>

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission. Corresponds to the JSON property args

Returns:

  • (Array<String>) 


3026
3027
3028
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3026

def args
  @args
end

#file_uris ⇒ Array<String>

Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks. Corresponds to the JSON property fileUris

Returns:

  • (Array<String>) 


3032
3033
3034
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3032

def file_uris
  @file_uris
end

#jar_file_uris ⇒ Array<String>

Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks. Corresponds to the JSON property jarFileUris

Returns:

  • (Array<String>) 


3038
3039
3040
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3038

def jar_file_uris
  @jar_file_uris
end

#logging_config ⇒ Google::Apis::DataprocV1beta2::LoggingConfig

The runtime logging config of the job. Corresponds to the JSON property loggingConfig

Returns:



3043
3044
3045
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3043

def logging_config
  @logging_config
end

#main_class ⇒ String

The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris. Corresponds to the JSON property mainClass

Returns:

  • (String) 


3049
3050
3051
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3049

def main_class
  @main_class
end

#main_jar_file_uri ⇒ String

The HCFS URI of the jar file that contains the main class. Corresponds to the JSON property mainJarFileUri

Returns:

  • (String) 


3054
3055
3056
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3054

def main_jar_file_uri
  @main_jar_file_uri
end

#properties ⇒ Hash<String,String>

Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code. Corresponds to the JSON property properties

Returns:

  • (Hash<String,String>) 


3062
3063
3064
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3062

def properties
  @properties
end

Instance Method Details

#update!(**args) ⇒ Object

Update properties of this object



3069
3070
3071
3072
3073
3074
3075
3076
3077
3078
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3069

def update!(**args)
  @archive_uris = args[:archive_uris] if args.key?(:archive_uris)
  @args = args[:args] if args.key?(:args)
  @file_uris = args[:file_uris] if args.key?(:file_uris)
  @jar_file_uris = args[:jar_file_uris] if args.key?(:jar_file_uris)
  @logging_config = args[:logging_config] if args.key?(:logging_config)
  @main_class = args[:main_class] if args.key?(:main_class)
  @main_jar_file_uri = args[:main_jar_file_uri] if args.key?(:main_jar_file_uri)
  @properties = args[:properties] if args.key?(:properties)
end