Class: Google::Apis::DataprocV1beta2::SparkJob

Inherits:
Object
  • Object
show all
Includes:
Core::Hashable, Core::JsonObjectSupport
Defined in:
lib/google/apis/dataproc_v1beta2/classes.rb,
lib/google/apis/dataproc_v1beta2/representations.rb,
lib/google/apis/dataproc_v1beta2/representations.rb

Overview

A Dataproc job for running Apache Spark (http://spark.apache.org/) applications on YARN. The specification of the main method to call to drive the job. Specify either the jar file that contains the main class or the main class name. To pass both a main jar and a main class in that jar, add the jar to CommonJob.jar_file_uris, and then specify the main class name in main_class.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(**args) ⇒ SparkJob

Returns a new instance of SparkJob.



3075
3076
3077
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3075

def initialize(**args)
   update!(**args)
end

Instance Attribute Details

#archive_uris ⇒ Array<String>

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip. Corresponds to the JSON property archiveUris

Returns:

  • (Array<String>) 


3030
3031
3032
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3030

def archive_uris
  @archive_uris
end

#args ⇒ Array<String>

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission. Corresponds to the JSON property args

Returns:

  • (Array<String>) 


3037
3038
3039
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3037

def args
  @args
end

#file_uris ⇒ Array<String>

Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks. Corresponds to the JSON property fileUris

Returns:

  • (Array<String>) 


3043
3044
3045
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3043

def file_uris
  @file_uris
end

#jar_file_uris ⇒ Array<String>

Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks. Corresponds to the JSON property jarFileUris

Returns:

  • (Array<String>) 


3049
3050
3051
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3049

def jar_file_uris
  @jar_file_uris
end

#logging_config ⇒ Google::Apis::DataprocV1beta2::LoggingConfig

The runtime logging config of the job. Corresponds to the JSON property loggingConfig

Returns:



3054
3055
3056
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3054

def logging_config
  @logging_config
end

#main_class ⇒ String

The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris. Corresponds to the JSON property mainClass

Returns:

  • (String) 


3060
3061
3062
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3060

def main_class
  @main_class
end

#main_jar_file_uri ⇒ String

The HCFS URI of the jar file that contains the main class. Corresponds to the JSON property mainJarFileUri

Returns:

  • (String) 


3065
3066
3067
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3065

def main_jar_file_uri
  @main_jar_file_uri
end

#properties ⇒ Hash<String,String>

Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code. Corresponds to the JSON property properties

Returns:

  • (Hash<String,String>) 


3073
3074
3075
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3073

def properties
  @properties
end

Instance Method Details

#update!(**args) ⇒ Object

Update properties of this object



3080
3081
3082
3083
3084
3085
3086
3087
3088
3089
 
# File 'lib/google/apis/dataproc_v1beta2/classes.rb', line 3080

def update!(**args)
  @archive_uris = args[:archive_uris] if args.key?(:archive_uris)
  @args = args[:args] if args.key?(:args)
  @file_uris = args[:file_uris] if args.key?(:file_uris)
  @jar_file_uris = args[:jar_file_uris] if args.key?(:jar_file_uris)
  @logging_config = args[:logging_config] if args.key?(:logging_config)
  @main_class = args[:main_class] if args.key?(:main_class)
  @main_jar_file_uri = args[:main_jar_file_uri] if args.key?(:main_jar_file_uri)
  @properties = args[:properties] if args.key?(:properties)
end