Class: Google::Apis::DataprocV1beta2::PySparkJob
- Inherits:
-
Object
- Object
- Google::Apis::DataprocV1beta2::PySparkJob
- Includes:
- Core::Hashable, Core::JsonObjectSupport
- Defined in:
- generated/google/apis/dataproc_v1beta2/classes.rb,
generated/google/apis/dataproc_v1beta2/representations.rb,
generated/google/apis/dataproc_v1beta2/representations.rb
Overview
A Cloud Dataproc job for running Apache PySpark (https://spark.apache.org/docs/ 0.9.0/python-programming-guide.html) applications on YARN.
Instance Attribute Summary collapse
-
#archive_uris ⇒ Array<String>
Optional.
-
#args ⇒ Array<String>
Optional.
-
#file_uris ⇒ Array<String>
Optional.
-
#jar_file_uris ⇒ Array<String>
Optional.
-
#logging_config ⇒ Google::Apis::DataprocV1beta2::LoggingConfig
The runtime logging config of the job.
-
#main_python_file_uri ⇒ String
Required.
-
#properties ⇒ Hash<String,String>
Optional.
-
#python_file_uris ⇒ Array<String>
Optional.
Instance Method Summary collapse
-
#initialize(**args) ⇒ PySparkJob
constructor
A new instance of PySparkJob.
-
#update!(**args) ⇒ Object
Update properties of this object.
Methods included from Core::JsonObjectSupport
Methods included from Core::Hashable
Constructor Details
#initialize(**args) ⇒ PySparkJob
Returns a new instance of PySparkJob
2429 2430 2431 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2429 def initialize(**args) update!(**args) end |
Instance Attribute Details
#archive_uris ⇒ Array<String>
Optional. HCFS URIs of archives to be extracted in the working directory of .
jar, .tar, .tar.gz, .tgz, and .zip.
Corresponds to the JSON property archiveUris
2383 2384 2385 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2383 def archive_uris @archive_uris end |
#args ⇒ Array<String>
Optional. The arguments to pass to the driver. Do not include arguments, such
as --conf, that can be set as job properties, since a collision may occur that
causes an incorrect job submission.
Corresponds to the JSON property args
2390 2391 2392 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2390 def args @args end |
#file_uris ⇒ Array<String>
Optional. HCFS URIs of files to be copied to the working directory of Python
drivers and distributed tasks. Useful for naively parallel tasks.
Corresponds to the JSON property fileUris
2396 2397 2398 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2396 def file_uris @file_uris end |
#jar_file_uris ⇒ Array<String>
Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver
and tasks.
Corresponds to the JSON property jarFileUris
2402 2403 2404 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2402 def jar_file_uris @jar_file_uris end |
#logging_config ⇒ Google::Apis::DataprocV1beta2::LoggingConfig
The runtime logging config of the job.
Corresponds to the JSON property loggingConfig
2407 2408 2409 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2407 def logging_config @logging_config end |
#main_python_file_uri ⇒ String
Required. The HCFS URI of the main Python file to use as the driver. Must be a
.py file.
Corresponds to the JSON property mainPythonFileUri
2413 2414 2415 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2413 def main_python_file_uri @main_python_file_uri end |
#properties ⇒ Hash<String,String>
Optional. A mapping of property names to values, used to configure PySpark.
Properties that conflict with values set by the Cloud Dataproc API may be
overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf
and classes in user code.
Corresponds to the JSON property properties
2421 2422 2423 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2421 def properties @properties end |
#python_file_uris ⇒ Array<String>
Optional. HCFS file URIs of Python files to pass to the PySpark framework.
Supported file types: .py, .egg, and .zip.
Corresponds to the JSON property pythonFileUris
2427 2428 2429 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2427 def python_file_uris @python_file_uris end |
Instance Method Details
#update!(**args) ⇒ Object
Update properties of this object
2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 |
# File 'generated/google/apis/dataproc_v1beta2/classes.rb', line 2434 def update!(**args) @archive_uris = args[:archive_uris] if args.key?(:archive_uris) @args = args[:args] if args.key?(:args) @file_uris = args[:file_uris] if args.key?(:file_uris) @jar_file_uris = args[:jar_file_uris] if args.key?(:jar_file_uris) @logging_config = args[:logging_config] if args.key?(:logging_config) @main_python_file_uri = args[:main_python_file_uri] if args.key?(:main_python_file_uri) @properties = args[:properties] if args.key?(:properties) @python_file_uris = args[:python_file_uris] if args.key?(:python_file_uris) end |