Class: Google::Apis::DataprocV1beta2::SparkJob

Inherits:
Object
  • Object
show all
Includes:
Core::Hashable, Core::JsonObjectSupport
Defined in:
generated/google/apis/dataproc_v1beta2/classes.rb,
generated/google/apis/dataproc_v1beta2/representations.rb,
generated/google/apis/dataproc_v1beta2/representations.rb

Overview

A Dataproc job for running Apache Spark (http://spark.apache.org/) applications on YARN. The specification of the main method to call to drive the job. Specify either the jar file that contains the main class or the main class name. To pass both a main jar and a main class in that jar, add the jar to CommonJob.jar_file_uris, and then specify the main class name in main_class.

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Core::JsonObjectSupport

#to_json

Methods included from Core::Hashable

process_value, #to_h

Constructor Details

#initialize(**args) ⇒ SparkJob

Returns a new instance of SparkJob.



2816
2817
2818
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2816

def initialize(**args)
   update!(**args)
end

Instance Attribute Details

#archive_uris ⇒ Array<String>

Optional. HCFS URIs of archives to be extracted in the working directory of Spark drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, and . zip. Corresponds to the JSON property archiveUris

Returns:

  • (Array<String>) 


2771
2772
2773
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2771

def archive_uris
  @archive_uris
end

#args ⇒ Array<String>

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission. Corresponds to the JSON property args

Returns:

  • (Array<String>) 


2778
2779
2780
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2778

def args
  @args
end

#file_uris ⇒ Array<String>

Optional. HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks. Corresponds to the JSON property fileUris

Returns:

  • (Array<String>) 


2784
2785
2786
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2784

def file_uris
  @file_uris
end

#jar_file_uris ⇒ Array<String>

Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks. Corresponds to the JSON property jarFileUris

Returns:

  • (Array<String>) 


2790
2791
2792
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2790

def jar_file_uris
  @jar_file_uris
end

#logging_config ⇒ Google::Apis::DataprocV1beta2::LoggingConfig

The runtime logging config of the job. Corresponds to the JSON property loggingConfig

Returns:



2795
2796
2797
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2795

def logging_config
  @logging_config
end

#main_class ⇒ String

The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris. Corresponds to the JSON property mainClass

Returns:

  • (String) 


2801
2802
2803
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2801

def main_class
  @main_class
end

#main_jar_file_uri ⇒ String

The HCFS URI of the jar file that contains the main class. Corresponds to the JSON property mainJarFileUri

Returns:

  • (String) 


2806
2807
2808
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2806

def main_jar_file_uri
  @main_jar_file_uri
end

#properties ⇒ Hash<String,String>

Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code. Corresponds to the JSON property properties

Returns:

  • (Hash<String,String>) 


2814
2815
2816
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2814

def properties
  @properties
end

Instance Method Details

#update!(**args) ⇒ Object

Update properties of this object



2821
2822
2823
2824
2825
2826
2827
2828
2829
2830
 
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2821

def update!(**args)
  @archive_uris = args[:archive_uris] if args.key?(:archive_uris)
  @args = args[:args] if args.key?(:args)
  @file_uris = args[:file_uris] if args.key?(:file_uris)
  @jar_file_uris = args[:jar_file_uris] if args.key?(:jar_file_uris)
  @logging_config = args[:logging_config] if args.key?(:logging_config)
  @main_class = args[:main_class] if args.key?(:main_class)
  @main_jar_file_uri = args[:main_jar_file_uri] if args.key?(:main_jar_file_uri)
  @properties = args[:properties] if args.key?(:properties)
end