Namespace Google.Apis.Dataproc.v1.Data

Classes

AcceleratorConfig

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine (https://cloud.google.com/compute/docs/gpus/).

AccessSessionSparkApplicationEnvironmentInfoResponse

Environment details of a Saprk Application.

AccessSessionSparkApplicationJobResponse

Details of a particular job associated with Spark Application

AccessSessionSparkApplicationResponse

A summary of Spark Application

AccessSessionSparkApplicationSqlQueryResponse

Details of a query for a Spark Application

AccessSessionSparkApplicationSqlSparkPlanGraphResponse

SparkPlanGraph for a Spark Application execution limited to maximum 10000 clusters.

AccessSessionSparkApplicationStageAttemptResponse

Stage Attempt for a Stage of a Spark Application

AccessSessionSparkApplicationStageRddOperationGraphResponse

RDD operation graph for a Spark Application Stage limited to maximum 10000 clusters.

AccessSparkApplicationEnvironmentInfoResponse

Environment details of a Saprk Application.

AccessSparkApplicationJobResponse

Details of a particular job associated with Spark Application

AccessSparkApplicationResponse

A summary of Spark Application

AccessSparkApplicationSqlQueryResponse

Details of a query for a Spark Application

AccessSparkApplicationSqlSparkPlanGraphResponse

SparkPlanGraph for a Spark Application execution limited to maximum 10000 clusters.

AccessSparkApplicationStageAttemptResponse

Stage Attempt for a Stage of a Spark Application

AccessSparkApplicationStageRddOperationGraphResponse

RDD operation graph for a Spark Application Stage limited to maximum 10000 clusters.

AnalyzeOperationMetadata

Metadata describing the Analyze operation.

ApplicationEnvironmentInfo

Details about the Environment that the application is running in.

ApplicationInfo

High level information corresponding to an application.

AutoscalingConfig

Autoscaling Policy config associated with the cluster.

AutoscalingPolicy

Describes an autoscaling policy for Dataproc cluster autoscaler.

AutotuningConfig

Autotuning configuration of the workload.

AuxiliaryNodeGroup

Node group identification and configuration information.

AuxiliaryServicesConfig

Auxiliary services configuration for a Cluster.

BasicYarnAutoscalingConfig

Basic autoscaling configurations for YARN.

Batch

A representation of a batch workload in the service.

BatchOperationMetadata

Metadata describing the Batch operation.

Binding

Associates members, or principals, with a role.

Cluster

Describes the identifying information, config, and status of a Dataproc cluster

ClusterMetrics

Contains cluster daemon metrics, such as HDFS and YARN stats.Beta Feature: This report is available for testing purposes only. It may be changed before final release.

ClusterOperation

The cluster operation triggered by a workflow.

ClusterSelector

A selector that chooses target cluster for jobs based on metadata.

ClusterStatus

The status of a cluster and its instances.

ConfidentialInstanceConfig

Confidential Instance Config for clusters using Confidential VMs (https://cloud.google.com/compute/confidential-vm/docs)

ConsolidatedExecutorSummary

Consolidated summary about executors used by the application.

DiagnoseClusterRequest

A request to collect cluster diagnostic information.

DiskConfig

Specifies the config of disk options for a group of VM instances.

A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); }

EncryptionConfig

Encryption settings for the cluster.

EndpointConfig

Endpoint config for this cluster

EnvironmentConfig

Environment configuration for a workload.

ExecutionConfig

Execution configuration for a workload.

ExecutorMetrics

ExecutorMetricsDistributions

ExecutorPeakMetricsDistributions

ExecutorResourceRequest

Resources used per executor used by the application.

ExecutorStageSummary

Executor resources consumed by a stage.

ExecutorSummary

Details about executors used by the application.

Expr

Represents a textual expression in the Common Expression Language (CEL) syntax. CEL is a C-like expression language. The syntax and semantics of CEL are documented at https://github.com/google/cel-spec.Example (Comparison): title: "Summary size limit" description: "Determines if a summary is less than 100 chars" expression: "document.summary.size() < 100" Example (Equality): title: "Requestor is owner" description: "Determines if requestor is the document owner" expression: "document.owner == request.auth.claims.email" Example (Logic): title: "Public documents" description: "Determine whether the document should be publicly visible" expression: "document.type != 'private' && document.type != 'internal'" Example (Data Manipulation): title: "Notification string" description: "Create a notification string with a timestamp." expression: "'New message received at ' + string(document.create_time)" The exact variables and functions that may be referenced within an expression are determined by the service that evaluates it. See the service documentation for additional information.

FlinkJob

A Dataproc job for running Apache Flink applications on YARN.

GceClusterConfig

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

GetIamPolicyRequest

Request message for GetIamPolicy method.

GetPolicyOptions

Encapsulates settings provided to GetIamPolicy.

GkeClusterConfig

The cluster's GKE config.

GkeNodeConfig

Parameters that describe cluster nodes.

GkeNodePoolAcceleratorConfig

A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a node pool.

GkeNodePoolAutoscalingConfig

GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.

GkeNodePoolConfig

The configuration of a GKE node pool used by a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/concepts/jobs/dataproc-gke#create-a-dataproc-on-gke-cluster).

GkeNodePoolTarget

GKE node pools that Dataproc workloads run on.

GoogleCloudDataprocV1WorkflowTemplateEncryptionConfig

Encryption settings for encrypting workflow template job arguments.

HadoopJob

A Dataproc job for running Apache Hadoop MapReduce (https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html) jobs on Apache Hadoop YARN (https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YARN.html).

HiveJob

A Dataproc job for running Apache Hive (https://hive.apache.org/) queries on YARN.

IdentityConfig

Identity related configuration, including service account based secure multi-tenancy user mappings.

InjectCredentialsRequest

A request to inject credentials into a cluster.

InputMetrics

Metrics about the input data read by the task.

InputQuantileMetrics

InstanceFlexibilityPolicy

Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.

InstanceGroupAutoscalingPolicyConfig

Configuration for the size bounds of an instance group, including its proportional size to other groups.

InstanceGroupConfig

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

InstanceReference

A reference to a Compute Engine instance.

InstanceSelection

Defines machines types and a rank to which the machines types belong.

InstanceSelectionResult

Defines a mapping from machine types to the number of VMs that are created with each machine type.

InstantiateWorkflowTemplateRequest

A request to instantiate a workflow template.

Interval

Represents a time interval, encoded as a Timestamp start (inclusive) and a Timestamp end (exclusive).The start must be less than or equal to the end. When the start equals the end, the interval is empty (matches no time). When both start and end are unspecified, the interval matches any time.

Job

A Dataproc job resource.

JobData

Data corresponding to a spark job.

JobMetadata

Job Operation metadata.

JobPlacement

Dataproc job config.

JobReference

Encapsulates the full scoping used to reference a job.

JobScheduling

Job scheduling options.

JobStatus

Dataproc job status.

JobsSummary

Data related to Jobs page summary

JupyterConfig

Jupyter configuration for an interactive session.

KerberosConfig

Specifies Kerberos related configuration.

KubernetesClusterConfig

The configuration for running the Dataproc cluster on Kubernetes.

KubernetesSoftwareConfig

The software configuration for this Dataproc cluster running on Kubernetes.

LifecycleConfig

Specifies the cluster auto-delete schedule configuration.

ListAutoscalingPoliciesResponse

A response to a request to list autoscaling policies in a project.

ListBatchesResponse

A list of batch workloads.

ListClustersResponse

The list of all clusters in a project.

ListJobsResponse

A list of jobs in a project.

ListOperationsResponse

The response message for Operations.ListOperations.

ListSessionTemplatesResponse

A list of session templates.

ListSessionsResponse

A list of interactive sessions.

ListWorkflowTemplatesResponse

A response to a request to list workflow templates in a project.

LoggingConfig

The runtime logging config of the job.

ManagedCluster

Cluster that is managed by the workflow.

ManagedGroupConfig

Specifies the resources used to actively manage an instance group.

MemoryMetrics

MetastoreConfig

Specifies a Metastore configuration.

Metric

A Dataproc custom metric.

NamespacedGkeDeploymentTarget

Deprecated. Used only for the deprecated beta. A full, namespace-isolated deployment target for an existing GKE cluster.

NodeGroup

Dataproc Node Group. The Dataproc NodeGroup resource is not related to the Dataproc NodeGroupAffinity resource.

NodeGroupAffinity

Node Group Affinity for clusters using sole-tenant node groups. The Dataproc NodeGroupAffinity resource is not related to the Dataproc NodeGroup resource.

NodeGroupOperationMetadata

Metadata describing the node group operation.

NodeInitializationAction

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

NodePool

indicating a list of workers of same type

Operation

This resource represents a long-running operation that is the result of a network API call.

OrderedJob

A job executed by the workflow.

OutputMetrics

Metrics about the data written by the task.

OutputQuantileMetrics

ParameterValidation

Configuration for parameter validation.

PeripheralsConfig

Auxiliary services configuration for a workload.

PigJob

A Dataproc job for running Apache Pig (https://pig.apache.org/) queries on YARN.

Policy

An Identity and Access Management (IAM) policy, which specifies access controls for Google Cloud resources.A Policy is a collection of bindings. A binding binds one or more members, or principals, to a single role. Principals can be user accounts, service accounts, Google groups, and domains (such as G Suite). A role is a named list of permissions; each role can be an IAM predefined role or a user-created custom role.For some types of Google Cloud resources, a binding can also specify a condition, which is a logical expression that allows access to a resource only if the expression evaluates to true. A condition can add constraints based on attributes of the request, the resource, or both. To learn which resources support conditions in their IAM policies, see the IAM documentation (https://cloud.google.com/iam/help/conditions/resource-policies).JSON example: { "bindings": [ { "role": "roles/resourcemanager.organizationAdmin", "members": [ "user:mike@example.com", "group:admins@example.com", "domain:google.com", "serviceAccount:my-project-id@appspot.gserviceaccount.com" ] }, { "role": "roles/resourcemanager.organizationViewer", "members": [ "user:eve@example.com" ], "condition": { "title": "expirable access", "description": "Does not grant access after Sep 2020", "expression": "request.time < timestamp('2020-10-01T00:00:00.000Z')", } } ], "etag": "BwWWja0YfJA=", "version": 3 } YAML example: bindings: - members: - user:mike@example.com - group:admins@example.com - domain:google.com - serviceAccount:my-project-id@appspot.gserviceaccount.com role: roles/resourcemanager.organizationAdmin - members: - user:eve@example.com role: roles/resourcemanager.organizationViewer condition: title: expirable access description: Does not grant access after Sep 2020 expression: request.time < timestamp('2020-10-01T00:00:00.000Z') etag: BwWWja0YfJA= version: 3 For a description of IAM and its features, see the IAM documentation (https://cloud.google.com/iam/docs/).

PoolData

Pool Data

PrestoJob

A Dataproc job for running Presto (https://prestosql.io/) queries. IMPORTANT: The Dataproc Presto Optional Component (https://cloud.google.com/dataproc/docs/concepts/components/presto) must be enabled when the cluster is created to submit a Presto job to the cluster.

ProcessSummary

Process Summary

ProvisioningModelMix

Defines how Dataproc should create VMs with a mixture of provisioning models.

PyPiRepositoryConfig

Configuration for PyPi repository

PySparkBatch

A configuration for running an Apache PySpark (https://spark.apache.org/docs/latest/api/python/getting_started/quickstart.html) batch workload.

PySparkJob

A Dataproc job for running Apache PySpark (https://spark.apache.org/docs/latest/api/python/index.html#pyspark-overview) applications on YARN.

Quantiles

Quantile metrics data related to Tasks. Units can be seconds, bytes, milliseconds, etc depending on the message type.

QueryList

A list of queries to run on a cluster.

RddDataDistribution

Details about RDD usage.

RddOperationCluster

A grouping of nodes representing higher level constructs (stage, job etc.).

RddOperationEdge

A directed edge representing dependency between two RDDs.

RddOperationGraph

Graph representing RDD dependencies. Consists of edges and a root cluster.

RddOperationNode

A node in the RDD operation graph. Corresponds to a single RDD.

RddPartitionInfo

Information about RDD partitions.

RddStorageInfo

Overall data about RDD storage.

RegexValidation

Validation based on regular expressions.

RepairClusterRequest

A request to repair a cluster.

RepairNodeGroupRequest

RepositoryConfig

Configuration for dependency repositories

ReservationAffinity

Reservation Affinity for consuming Zonal reservation.

ResizeNodeGroupRequest

A request to resize a node group.

ResourceInformation

ResourceProfileInfo

Resource profile that contains information about all the resources required by executors and tasks.

RuntimeConfig

Runtime configuration for a workload.

RuntimeInfo

Runtime information about workload execution.

SearchSessionSparkApplicationExecutorStageSummaryResponse

List of Executors associated with a Spark Application Stage.

SearchSessionSparkApplicationExecutorsResponse

List of Executors associated with a Spark Application.

SearchSessionSparkApplicationJobsResponse

A list of Jobs associated with a Spark Application.

SearchSessionSparkApplicationSqlQueriesResponse

List of all queries for a Spark Application.

SearchSessionSparkApplicationStageAttemptTasksResponse

List of tasks for a stage of a Spark Application

SearchSessionSparkApplicationStageAttemptsResponse

A list of Stage Attempts for a Stage of a Spark Application.

SearchSessionSparkApplicationStagesResponse

A list of stages associated with a Spark Application.

SearchSessionSparkApplicationsResponse

A list of summary of Spark Applications

SearchSparkApplicationExecutorStageSummaryResponse

List of Executors associated with a Spark Application Stage.

SearchSparkApplicationExecutorsResponse

List of Executors associated with a Spark Application.

SearchSparkApplicationJobsResponse

A list of Jobs associated with a Spark Application.

SearchSparkApplicationSqlQueriesResponse

List of all queries for a Spark Application.

SearchSparkApplicationStageAttemptTasksResponse

List of tasks for a stage of a Spark Application

SearchSparkApplicationStageAttemptsResponse

A list of Stage Attempts for a Stage of a Spark Application.

SearchSparkApplicationStagesResponse

A list of stages associated with a Spark Application.

SearchSparkApplicationsResponse

A list of summary of Spark Applications

SecurityConfig

Security related configuration, including encryption, Kerberos, etc.

Session

A representation of a session.

SessionOperationMetadata

Metadata describing the Session operation.

SessionStateHistory

Historical state information.

SessionTemplate

A representation of a session template.

SetIamPolicyRequest

Request message for SetIamPolicy method.

ShieldedInstanceConfig

Shielded Instance Config for clusters using Compute Engine Shielded VMs (https://cloud.google.com/security/shielded-cloud/shielded-vm).

ShufflePushReadMetrics

ShufflePushReadQuantileMetrics

ShuffleReadMetrics

Shuffle data read by the task.

ShuffleReadQuantileMetrics

ShuffleWriteMetrics

Shuffle data written by task.

ShuffleWriteQuantileMetrics

SinkProgress

SoftwareConfig

Specifies the selection and config of software inside the cluster.

SourceProgress

SparkApplication

A summary of Spark Application

SparkBatch

A configuration for running an Apache Spark (https://spark.apache.org/) batch workload.

SparkConnectConfig

Spark connect configuration for an interactive session.

SparkHistoryServerConfig

Spark History Server configuration for the workload.

SparkJob

A Dataproc job for running Apache Spark (https://spark.apache.org/) applications on YARN.

SparkPlanGraph

A graph used for storing information of an executionPlan of DataFrame.

SparkPlanGraphCluster

Represents a tree of spark plan.

SparkPlanGraphEdge

Represents a directed edge in the spark plan tree from child to parent.

SparkPlanGraphNode

Represents a node in the spark plan tree.

SparkPlanGraphNodeWrapper

Wrapper user to represent either a node or a cluster.

SparkRBatch

A configuration for running an Apache SparkR (https://spark.apache.org/docs/latest/sparkr.html) batch workload.

SparkRJob

A Dataproc job for running Apache SparkR (https://spark.apache.org/docs/latest/sparkr.html) applications on YARN.

SparkRuntimeInfo

SparkSqlBatch

A configuration for running Apache Spark SQL (https://spark.apache.org/sql/) queries as a batch workload.

SparkSqlJob

A Dataproc job for running Apache Spark SQL (https://spark.apache.org/sql/) queries.

SparkStandaloneAutoscalingConfig

Basic autoscaling configurations for Spark Standalone.

SparkWrapperObject

Outer message that contains the data obtained from spark listener, packaged with information that is required to process it.

SpeculationStageSummary

Details of the speculation task when speculative execution is enabled.

SqlExecutionUiData

SQL Execution Data

SqlPlanMetric

Metrics related to SQL execution.

StageAttemptTasksSummary

Data related to tasks summary for a Spark Stage Attempt

StageData

Data corresponding to a stage.

StageInputMetrics

Metrics about the input read by the stage.

StageMetrics

Stage Level Aggregated Metrics

StageOutputMetrics

Metrics about the output written by the stage.

StageShufflePushReadMetrics

StageShuffleReadMetrics

Shuffle data read for the stage.

StageShuffleWriteMetrics

Shuffle data written for the stage.

StagesSummary

Data related to Stages page summary

StartClusterRequest

A request to start a cluster.

StartupConfig

Configuration to handle the startup of instances during cluster create and update process.

StateHistory

Historical state information.

StateOperatorProgress

Status

The Status type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by gRPC (https://github.com/grpc). Each Status message contains three pieces of data: error code, error message, and error details.You can find out more about this error model and how to work with it in the API Design Guide (https://cloud.google.com/apis/design/errors).

StopClusterRequest

A request to stop a cluster.

StreamBlockData

Stream Block Data.

StreamingQueryData

Streaming

StreamingQueryProgress

SubmitJobRequest

A request to submit a job.

SummarizeSessionSparkApplicationExecutorsResponse

Consolidated summary of executors for a Spark Application.

SummarizeSessionSparkApplicationJobsResponse

Summary of a Spark Application jobs.

SummarizeSessionSparkApplicationStageAttemptTasksResponse

Summary of tasks for a Spark Application stage attempt.

SummarizeSessionSparkApplicationStagesResponse

Summary of a Spark Application stages.

SummarizeSparkApplicationExecutorsResponse

Consolidated summary of executors for a Spark Application.

SummarizeSparkApplicationJobsResponse

Summary of a Spark Application jobs.

SummarizeSparkApplicationStageAttemptTasksResponse

Summary of tasks for a Spark Application stage attempt.

SummarizeSparkApplicationStagesResponse

Summary of a Spark Application stages.

TaskData

Data corresponding to tasks created by spark.

TaskMetrics

Executor Task Metrics

TaskQuantileMetrics

TaskResourceRequest

Resources used per task created by the application.

TemplateParameter

A configurable parameter that replaces one or more fields in the template. Parameterizable fields: - Labels - File uris - Job properties - Job arguments - Script variables - Main class (in HadoopJob and SparkJob) - Zone (in ClusterSelector)

TerminateSessionRequest

A request to terminate an interactive session.

TestIamPermissionsRequest

Request message for TestIamPermissions method.

TestIamPermissionsResponse

Response message for TestIamPermissions method.

TrinoJob

A Dataproc job for running Trino (https://trino.io/) queries. IMPORTANT: The Dataproc Trino Optional Component (https://cloud.google.com/dataproc/docs/concepts/components/trino) must be enabled when the cluster is created to submit a Trino job to the cluster.