com.google.api.services.dataproc.v1beta2.model

Class SparkJob

java.lang.Object

java.util.AbstractMap<String,Object>

com.google.api.client.util.GenericData

com.google.api.client.json.GenericJson

com.google.api.services.dataproc.v1beta2.model.SparkJob

All Implemented Interfaces:

Cloneable, Map<String,Object>

public final class SparkJob
extends GenericJson

A Dataproc job for running Apache Spark (http://spark.apache.org/) applications on YARN. The specification of the main method to call to drive the job. Specify either the jar file that contains the main class or the main class name. To pass both a main jar and a main class in that jar, add the jar to CommonJob.jar_file_uris, and then specify the main class name in main_class.

This is the Java data model class that specifies how to parse/serialize into the JSON that is transmitted over HTTP when working with the Cloud Dataproc API. For a detailed explanation see: https://developers.google.com/api-client-library/java/google-http-java-client/json

Author:

Google, Inc.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.api.client.util.GenericData

GenericData.Flags

Nested classes/interfaces inherited from class java.util.AbstractMap

AbstractMap.SimpleEntry<K,V>, AbstractMap.SimpleImmutableEntry<K,V>

Nested classes/interfaces inherited from interface java.util.Map

Map.Entry<K,V>

Constructor Summary

Constructors

Constructor and Description
SparkJob()

Method Summary

Modifier and Type	Method and Description
SparkJob	clone()
List<String>	getArchiveUris() Optional.
List<String>	getArgs() Optional.
List<String>	getFileUris() Optional.
List<String>	getJarFileUris() Optional.
LoggingConfig	getLoggingConfig() Optional.
String	getMainClass() The name of the driver's main class.
String	getMainJarFileUri() The HCFS URI of the jar file that contains the main class.
Map<String,String>	getProperties() Optional.
SparkJob	set(String fieldName, Object value)
SparkJob	setArchiveUris(List<String> archiveUris) Optional.
SparkJob	setArgs(List<String> args) Optional.
SparkJob	setFileUris(List<String> fileUris) Optional.
SparkJob	setJarFileUris(List<String> jarFileUris) Optional.
SparkJob	setLoggingConfig(LoggingConfig loggingConfig) Optional.
SparkJob	setMainClass(String mainClass) The name of the driver's main class.
SparkJob	setMainJarFileUri(String mainJarFileUri) The HCFS URI of the jar file that contains the main class.
SparkJob	setProperties(Map<String,String> properties) Optional.

Methods inherited from class com.google.api.client.json.GenericJson

getFactory, setFactory, toPrettyString, toString

Methods inherited from class com.google.api.client.util.GenericData

entrySet, equals, get, getClassInfo, getUnknownKeys, hashCode, put, putAll, remove, setUnknownKeys

Methods inherited from class java.util.AbstractMap

clear, containsKey, containsValue, isEmpty, keySet, size, values

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Map

compute, computeIfAbsent, computeIfPresent, forEach, getOrDefault, merge, putIfAbsent, remove, replace, replace, replaceAll

Constructor Detail

SparkJob

public SparkJob()

Method Detail

getArchiveUris

public List<String> getArchiveUris()

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Returns:

value or null for none

setArchiveUris

public SparkJob setArchiveUris(List<String> archiveUris)

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Parameters:

archiveUris - archiveUris or null for none

getArgs

public List<String> getArgs()

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

Returns:

value or null for none

setArgs

public SparkJob setArgs(List<String> args)

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

Parameters:

args - args or null for none

getFileUris

public List<String> getFileUris()

Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.

Returns:

value or null for none

setFileUris

public SparkJob setFileUris(List<String> fileUris)

Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.

Parameters:

fileUris - fileUris or null for none

getJarFileUris

public List<String> getJarFileUris()

Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.

Returns:

value or null for none

setJarFileUris

public SparkJob setJarFileUris(List<String> jarFileUris)

Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.

Parameters:

jarFileUris - jarFileUris or null for none

getLoggingConfig

public LoggingConfig getLoggingConfig()

Optional. The runtime log config for job execution.

Returns:

value or null for none

setLoggingConfig

public SparkJob setLoggingConfig(LoggingConfig loggingConfig)

Optional. The runtime log config for job execution.

Parameters:

loggingConfig - loggingConfig or null for none

getMainClass

public String getMainClass()

The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris.

Returns:

value or null for none

setMainClass

public SparkJob setMainClass(String mainClass)

The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris.

Parameters:

mainClass - mainClass or null for none

getMainJarFileUri

public String getMainJarFileUri()

The HCFS URI of the jar file that contains the main class.

Returns:

value or null for none

setMainJarFileUri

public SparkJob setMainJarFileUri(String mainJarFileUri)

The HCFS URI of the jar file that contains the main class.

Parameters:

mainJarFileUri - mainJarFileUri or null for none

getProperties

public Map<String,String> getProperties()

Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.

Returns:

value or null for none

setProperties

public SparkJob setProperties(Map<String,String> properties)

Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.

Parameters:

properties - properties or null for none

set

public SparkJob set(String fieldName,
                    Object value)

Overrides:

set in class GenericJson

clone

public SparkJob clone()

Overrides:

clone in class GenericJson