com.google.api.services.dataproc.v1beta2.model

Class PySparkJob

java.lang.Object

java.util.AbstractMap<String,Object>

com.google.api.client.util.GenericData

com.google.api.client.json.GenericJson

com.google.api.services.dataproc.v1beta2.model.PySparkJob

All Implemented Interfaces:

Cloneable, Map<String,Object>

public final class PySparkJob
extends GenericJson

A Dataproc job for running Apache PySpark (https://spark.apache.org/docs/0.9.0/python- programming-guide.html) applications on YARN.

This is the Java data model class that specifies how to parse/serialize into the JSON that is transmitted over HTTP when working with the Cloud Dataproc API. For a detailed explanation see: https://developers.google.com/api-client-library/java/google-http-java-client/json

Author:

Google, Inc.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.api.client.util.GenericData

GenericData.Flags

Nested classes/interfaces inherited from class java.util.AbstractMap

AbstractMap.SimpleEntry<K,V>, AbstractMap.SimpleImmutableEntry<K,V>

Nested classes/interfaces inherited from interface java.util.Map

Map.Entry<K,V>

Constructor Summary

Constructors

Constructor and Description
PySparkJob()

Method Summary

Modifier and Type	Method and Description
PySparkJob	clone()
List<String>	getArchiveUris() Optional.
List<String>	getArgs() Optional.
List<String>	getFileUris() Optional.
List<String>	getJarFileUris() Optional.
LoggingConfig	getLoggingConfig() Optional.
String	getMainPythonFileUri() Required.
Map<String,String>	getProperties() Optional.
List<String>	getPythonFileUris() Optional.
PySparkJob	set(String fieldName, Object value)
PySparkJob	setArchiveUris(List<String> archiveUris) Optional.
PySparkJob	setArgs(List<String> args) Optional.
PySparkJob	setFileUris(List<String> fileUris) Optional.
PySparkJob	setJarFileUris(List<String> jarFileUris) Optional.
PySparkJob	setLoggingConfig(LoggingConfig loggingConfig) Optional.
PySparkJob	setMainPythonFileUri(String mainPythonFileUri) Required.
PySparkJob	setProperties(Map<String,String> properties) Optional.
PySparkJob	setPythonFileUris(List<String> pythonFileUris) Optional.

Methods inherited from class com.google.api.client.json.GenericJson

getFactory, setFactory, toPrettyString, toString

Methods inherited from class com.google.api.client.util.GenericData

entrySet, equals, get, getClassInfo, getUnknownKeys, hashCode, put, putAll, remove, setUnknownKeys

Methods inherited from class java.util.AbstractMap

clear, containsKey, containsValue, isEmpty, keySet, size, values

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Map

compute, computeIfAbsent, computeIfPresent, forEach, getOrDefault, merge, putIfAbsent, remove, replace, replace, replaceAll

Constructor Detail

PySparkJob

public PySparkJob()

Method Detail

getArchiveUris

public List<String> getArchiveUris()

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Returns:

value or null for none

setArchiveUris

public PySparkJob setArchiveUris(List<String> archiveUris)

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Parameters:

archiveUris - archiveUris or null for none

getArgs

public List<String> getArgs()

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

Returns:

value or null for none

setArgs

public PySparkJob setArgs(List<String> args)

Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

Parameters:

args - args or null for none

getFileUris

public List<String> getFileUris()

Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.

Returns:

value or null for none

setFileUris

public PySparkJob setFileUris(List<String> fileUris)

Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.

Parameters:

fileUris - fileUris or null for none

getJarFileUris

public List<String> getJarFileUris()

Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.

Returns:

value or null for none

setJarFileUris

public PySparkJob setJarFileUris(List<String> jarFileUris)

Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.

Parameters:

jarFileUris - jarFileUris or null for none

getLoggingConfig

public LoggingConfig getLoggingConfig()

Optional. The runtime log config for job execution.

Returns:

value or null for none

setLoggingConfig

public PySparkJob setLoggingConfig(LoggingConfig loggingConfig)

Optional. The runtime log config for job execution.

Parameters:

loggingConfig - loggingConfig or null for none

getMainPythonFileUri

public String getMainPythonFileUri()

Required. The HCFS URI of the main Python file to use as the driver. Must be a .py file.

Returns:

value or null for none

setMainPythonFileUri

public PySparkJob setMainPythonFileUri(String mainPythonFileUri)

Required. The HCFS URI of the main Python file to use as the driver. Must be a .py file.

Parameters:

mainPythonFileUri - mainPythonFileUri or null for none

getProperties

public Map<String,String> getProperties()

Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.

Returns:

value or null for none

setProperties

public PySparkJob setProperties(Map<String,String> properties)

Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.

Parameters:

properties - properties or null for none

getPythonFileUris

public List<String> getPythonFileUris()

Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

Returns:

value or null for none

setPythonFileUris

public PySparkJob setPythonFileUris(List<String> pythonFileUris)

Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

Parameters:

pythonFileUris - pythonFileUris or null for none

set

public PySparkJob set(String fieldName,
                      Object value)

Overrides:

set in class GenericJson

clone

public PySparkJob clone()

Overrides:

clone in class GenericJson