Class: Google::Apis::DataprocV1beta2::SparkJob
- Inherits:
-
Object
- Object
- Google::Apis::DataprocV1beta2::SparkJob
- Includes:
- Core::Hashable, Core::JsonObjectSupport
- Defined in:
- generated/google/apis/dataproc_v1beta2/classes.rb,
generated/google/apis/dataproc_v1beta2/representations.rb,
generated/google/apis/dataproc_v1beta2/representations.rb
Overview
A Dataproc job for running Apache Spark (http://spark.apache.org/) applications on YARN. The specification of the main method to call to drive the job. Specify either the jar file that contains the main class or the main class name. To pass both a main jar and a main class in that jar, add the jar to CommonJob.jar_file_uris, and then specify the main class name in main_class.
Instance Attribute Summary collapse
-
#archive_uris ⇒ Array<String>
Optional.
-
#args ⇒ Array<String>
Optional.
-
#file_uris ⇒ Array<String>
Optional.
-
#jar_file_uris ⇒ Array<String>
Optional.
-
#logging_config ⇒ Google::Apis::DataprocV1beta2::LoggingConfig
The runtime logging config of the job.
-
#main_class ⇒ String
The name of the driver's main class.
-
#main_jar_file_uri ⇒ String
The HCFS URI of the jar file that contains the main class.
-
#properties ⇒ Hash<String,String>
Optional.
Instance Method Summary collapse
-
#initialize(**args) ⇒ SparkJob
constructor
A new instance of SparkJob.
-
#update!(**args) ⇒ Object
Update properties of this object.
Constructor Details
#initialize(**args) ⇒ SparkJob
Returns a new instance of SparkJob.
3064 3065 3066 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3064 def initialize(**args) update!(**args) end |
Instance Attribute Details
#archive_uris ⇒ Array<String>
Optional. HCFS URIs of archives to be extracted into the working directory of
each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
Corresponds to the JSON property archiveUris
3019 3020 3021 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3019 def archive_uris @archive_uris end |
#args ⇒ Array<String>
Optional. The arguments to pass to the driver. Do not include arguments, such
as --conf, that can be set as job properties, since a collision may occur that
causes an incorrect job submission.
Corresponds to the JSON property args
3026 3027 3028 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3026 def args @args end |
#file_uris ⇒ Array<String>
Optional. HCFS URIs of files to be placed in the working directory of each
executor. Useful for naively parallel tasks.
Corresponds to the JSON property fileUris
3032 3033 3034 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3032 def file_uris @file_uris end |
#jar_file_uris ⇒ Array<String>
Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver
and tasks.
Corresponds to the JSON property jarFileUris
3038 3039 3040 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3038 def jar_file_uris @jar_file_uris end |
#logging_config ⇒ Google::Apis::DataprocV1beta2::LoggingConfig
The runtime logging config of the job.
Corresponds to the JSON property loggingConfig
3043 3044 3045 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3043 def logging_config @logging_config end |
#main_class ⇒ String
The name of the driver's main class. The jar file that contains the class must
be in the default CLASSPATH or specified in jar_file_uris.
Corresponds to the JSON property mainClass
3049 3050 3051 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3049 def main_class @main_class end |
#main_jar_file_uri ⇒ String
The HCFS URI of the jar file that contains the main class.
Corresponds to the JSON property mainJarFileUri
3054 3055 3056 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3054 def main_jar_file_uri @main_jar_file_uri end |
#properties ⇒ Hash<String,String>
Optional. A mapping of property names to values, used to configure Spark.
Properties that conflict with values set by the Dataproc API may be
overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf
and classes in user code.
Corresponds to the JSON property properties
3062 3063 3064 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3062 def properties @properties end |
Instance Method Details
#update!(**args) ⇒ Object
Update properties of this object
3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 3069 def update!(**args) @archive_uris = args[:archive_uris] if args.key?(:archive_uris) @args = args[:args] if args.key?(:args) @file_uris = args[:file_uris] if args.key?(:file_uris) @jar_file_uris = args[:jar_file_uris] if args.key?(:jar_file_uris) @logging_config = args[:logging_config] if args.key?(:logging_config) @main_class = args[:main_class] if args.key?(:main_class) @main_jar_file_uri = args[:main_jar_file_uri] if args.key?(:main_jar_file_uri) @properties = args[:properties] if args.key?(:properties) end |