Namespace Google.Apis.Dataproc.v1.Data
Classes
AcceleratorConfig
Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine (https://cloud.google.com/compute/docs/gpus/).
AccessSessionSparkApplicationEnvironmentInfoResponse
Environment details of a Saprk Application.
AccessSessionSparkApplicationJobResponse
Details of a particular job associated with Spark Application
AccessSessionSparkApplicationResponse
A summary of Spark Application
AccessSessionSparkApplicationSqlQueryResponse
Details of a query for a Spark Application
AccessSessionSparkApplicationSqlSparkPlanGraphResponse
SparkPlanGraph for a Spark Application execution limited to maximum 10000 clusters.
AccessSessionSparkApplicationStageAttemptResponse
Stage Attempt for a Stage of a Spark Application
AccessSessionSparkApplicationStageRddOperationGraphResponse
RDD operation graph for a Spark Application Stage limited to maximum 10000 clusters.
AccessSparkApplicationEnvironmentInfoResponse
Environment details of a Saprk Application.
AccessSparkApplicationJobResponse
Details of a particular job associated with Spark Application
AccessSparkApplicationResponse
A summary of Spark Application
AccessSparkApplicationSqlQueryResponse
Details of a query for a Spark Application
AccessSparkApplicationSqlSparkPlanGraphResponse
SparkPlanGraph for a Spark Application execution limited to maximum 10000 clusters.
AccessSparkApplicationStageAttemptResponse
Stage Attempt for a Stage of a Spark Application
AccessSparkApplicationStageRddOperationGraphResponse
RDD operation graph for a Spark Application Stage limited to maximum 10000 clusters.
AccumulableInfo
AnalyzeBatchRequest
A request to analyze a batch workload.
AnalyzeOperationMetadata
Metadata describing the Analyze operation.
AppSummary
ApplicationAttemptInfo
Specific attempt of an application.
ApplicationEnvironmentInfo
Details about the Environment that the application is running in.
ApplicationInfo
High level information corresponding to an application.
AutoscalingConfig
Autoscaling Policy config associated with the cluster.
AutoscalingPolicy
Describes an autoscaling policy for Dataproc cluster autoscaler.
AutotuningConfig
Autotuning configuration of the workload.
AuxiliaryNodeGroup
Node group identification and configuration information.
AuxiliaryServicesConfig
Auxiliary services configuration for a Cluster.
BasicAutoscalingAlgorithm
Basic algorithm for autoscaling.
BasicYarnAutoscalingConfig
Basic autoscaling configurations for YARN.
Batch
A representation of a batch workload in the service.
BatchOperationMetadata
Metadata describing the Batch operation.
Binding
Associates members, or principals, with a role.
CancelJobRequest
A request to cancel a job.
Cluster
Describes the identifying information, config, and status of a Dataproc cluster
ClusterConfig
The cluster config.
ClusterMetrics
Contains cluster daemon metrics, such as HDFS and YARN stats.Beta Feature: This report is available for testing purposes only. It may be changed before final release.
ClusterOperation
The cluster operation triggered by a workflow.
ClusterOperationMetadata
Metadata describing the operation.
ClusterOperationStatus
The status of the operation.
ClusterSelector
A selector that chooses target cluster for jobs based on metadata.
ClusterStatus
The status of a cluster and its instances.
ClusterToRepair
Cluster to be repaired
ConfidentialInstanceConfig
Confidential Instance Config for clusters using Confidential VMs (https://cloud.google.com/compute/confidential-vm/docs)
ConsolidatedExecutorSummary
Consolidated summary about executors used by the application.
DataprocMetricConfig
Dataproc metric config.
DiagnoseClusterRequest
A request to collect cluster diagnostic information.
DiagnoseClusterResults
The location of diagnostic output.
DiskConfig
Specifies the config of disk options for a group of VM instances.
DriverSchedulingConfig
Driver scheduling configuration.
Empty
A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); }
EncryptionConfig
Encryption settings for the cluster.
EndpointConfig
Endpoint config for this cluster
EnvironmentConfig
Environment configuration for a workload.
ExecutionConfig
Execution configuration for a workload.
ExecutorMetrics
ExecutorMetricsDistributions
ExecutorPeakMetricsDistributions
ExecutorResourceRequest
Resources used per executor used by the application.
ExecutorStageSummary
Executor resources consumed by a stage.
ExecutorSummary
Details about executors used by the application.
Expr
Represents a textual expression in the Common Expression Language (CEL) syntax. CEL is a C-like expression language. The syntax and semantics of CEL are documented at https://github.com/google/cel-spec.Example (Comparison): title: "Summary size limit" description: "Determines if a summary is less than 100 chars" expression: "document.summary.size() < 100" Example (Equality): title: "Requestor is owner" description: "Determines if requestor is the document owner" expression: "document.owner == request.auth.claims.email" Example (Logic): title: "Public documents" description: "Determine whether the document should be publicly visible" expression: "document.type != 'private' && document.type != 'internal'" Example (Data Manipulation): title: "Notification string" description: "Create a notification string with a timestamp." expression: "'New message received at ' + string(document.create_time)" The exact variables and functions that may be referenced within an expression are determined by the service that evaluates it. See the service documentation for additional information.
FlinkJob
A Dataproc job for running Apache Flink applications on YARN.
GceClusterConfig
Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.
GetIamPolicyRequest
Request message for GetIamPolicy method.
GetPolicyOptions
Encapsulates settings provided to GetIamPolicy.
GkeClusterConfig
The cluster's GKE config.
GkeNodeConfig
Parameters that describe cluster nodes.
GkeNodePoolAcceleratorConfig
A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a node pool.
GkeNodePoolAutoscalingConfig
GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.
GkeNodePoolConfig
The configuration of a GKE node pool used by a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/concepts/jobs/dataproc-gke#create-a-dataproc-on-gke-cluster).
GkeNodePoolTarget
GKE node pools that Dataproc workloads run on.
GoogleCloudDataprocV1WorkflowTemplateEncryptionConfig
Encryption settings for encrypting workflow template job arguments.
HadoopJob
A Dataproc job for running Apache Hadoop MapReduce (https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html) jobs on Apache Hadoop YARN (https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YARN.html).
HiveJob
A Dataproc job for running Apache Hive (https://hive.apache.org/) queries on YARN.
IdentityConfig
Identity related configuration, including service account based secure multi-tenancy user mappings.
InjectCredentialsRequest
A request to inject credentials into a cluster.
InputMetrics
Metrics about the input data read by the task.
InputQuantileMetrics
InstanceFlexibilityPolicy
Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.
InstanceGroupAutoscalingPolicyConfig
Configuration for the size bounds of an instance group, including its proportional size to other groups.
InstanceGroupConfig
The config settings for Compute Engine resources in an instance group, such as a master or worker group.
InstanceReference
A reference to a Compute Engine instance.
InstanceSelection
Defines machines types and a rank to which the machines types belong.
InstanceSelectionResult
Defines a mapping from machine types to the number of VMs that are created with each machine type.
InstantiateWorkflowTemplateRequest
A request to instantiate a workflow template.
Interval
Represents a time interval, encoded as a Timestamp start (inclusive) and a Timestamp end (exclusive).The start must be less than or equal to the end. When the start equals the end, the interval is empty (matches no time). When both start and end are unspecified, the interval matches any time.
Job
A Dataproc job resource.
JobData
Data corresponding to a spark job.
JobMetadata
Job Operation metadata.
JobPlacement
Dataproc job config.
JobReference
Encapsulates the full scoping used to reference a job.
JobScheduling
Job scheduling options.
JobStatus
Dataproc job status.
JobsSummary
Data related to Jobs page summary
JupyterConfig
Jupyter configuration for an interactive session.
KerberosConfig
Specifies Kerberos related configuration.
KubernetesClusterConfig
The configuration for running the Dataproc cluster on Kubernetes.
KubernetesSoftwareConfig
The software configuration for this Dataproc cluster running on Kubernetes.
LifecycleConfig
Specifies the cluster auto-delete schedule configuration.
ListAutoscalingPoliciesResponse
A response to a request to list autoscaling policies in a project.
ListBatchesResponse
A list of batch workloads.
ListClustersResponse
The list of all clusters in a project.
ListJobsResponse
A list of jobs in a project.
ListOperationsResponse
The response message for Operations.ListOperations.
ListSessionTemplatesResponse
A list of session templates.
ListSessionsResponse
A list of interactive sessions.
ListWorkflowTemplatesResponse
A response to a request to list workflow templates in a project.
LoggingConfig
The runtime logging config of the job.
ManagedCluster
Cluster that is managed by the workflow.
ManagedGroupConfig
Specifies the resources used to actively manage an instance group.
MemoryMetrics
MetastoreConfig
Specifies a Metastore configuration.
Metric
A Dataproc custom metric.
NamespacedGkeDeploymentTarget
Deprecated. Used only for the deprecated beta. A full, namespace-isolated deployment target for an existing GKE cluster.
NodeGroup
Dataproc Node Group. The Dataproc NodeGroup resource is not related to the Dataproc NodeGroupAffinity resource.
NodeGroupAffinity
Node Group Affinity for clusters using sole-tenant node groups. The Dataproc NodeGroupAffinity resource is not related to the Dataproc NodeGroup resource.
NodeGroupOperationMetadata
Metadata describing the node group operation.
NodeInitializationAction
Specifies an executable to run on a fully configured node and a timeout period for executable completion.
NodePool
indicating a list of workers of same type
Operation
This resource represents a long-running operation that is the result of a network API call.
OrderedJob
A job executed by the workflow.
OutputMetrics
Metrics about the data written by the task.
OutputQuantileMetrics
ParameterValidation
Configuration for parameter validation.
PeripheralsConfig
Auxiliary services configuration for a workload.
PigJob
A Dataproc job for running Apache Pig (https://pig.apache.org/) queries on YARN.
Policy
An Identity and Access Management (IAM) policy, which specifies access controls for Google Cloud resources.A Policy is a collection of bindings. A binding binds one or more members, or principals, to a single role. Principals can be user accounts, service accounts, Google groups, and domains (such as G Suite). A role is a named list of permissions; each role can be an IAM predefined role or a user-created custom role.For some types of Google Cloud resources, a binding can also specify a condition, which is a logical expression that allows access to a resource only if the expression evaluates to true. A condition can add constraints based on attributes of the request, the resource, or both. To learn which resources support conditions in their IAM policies, see the IAM documentation (https://cloud.google.com/iam/help/conditions/resource-policies).JSON example: { "bindings": [ { "role": "roles/resourcemanager.organizationAdmin", "members": [ "user:mike@example.com", "group:admins@example.com", "domain:google.com", "serviceAccount:my-project-id@appspot.gserviceaccount.com" ] }, { "role": "roles/resourcemanager.organizationViewer", "members": [ "user:eve@example.com" ], "condition": { "title": "expirable access", "description": "Does not grant access after Sep 2020", "expression": "request.time < timestamp('2020-10-01T00:00:00.000Z')", } } ], "etag": "BwWWja0YfJA=", "version": 3 } YAML example: bindings: - members: - user:mike@example.com - group:admins@example.com - domain:google.com - serviceAccount:my-project-id@appspot.gserviceaccount.com role: roles/resourcemanager.organizationAdmin - members: - user:eve@example.com role: roles/resourcemanager.organizationViewer condition: title: expirable access description: Does not grant access after Sep 2020 expression: request.time < timestamp('2020-10-01T00:00:00.000Z') etag: BwWWja0YfJA= version: 3 For a description of IAM and its features, see the IAM documentation (https://cloud.google.com/iam/docs/).
PoolData
Pool Data
PrestoJob
A Dataproc job for running Presto (https://prestosql.io/) queries. IMPORTANT: The Dataproc Presto Optional Component (https://cloud.google.com/dataproc/docs/concepts/components/presto) must be enabled when the cluster is created to submit a Presto job to the cluster.
ProcessSummary
Process Summary
ProvisioningModelMix
Defines how Dataproc should create VMs with a mixture of provisioning models.
PyPiRepositoryConfig
Configuration for PyPi repository
PySparkBatch
A configuration for running an Apache PySpark (https://spark.apache.org/docs/latest/api/python/getting_started/quickstart.html) batch workload.
PySparkJob
A Dataproc job for running Apache PySpark (https://spark.apache.org/docs/0.9.0/python-programming-guide.html) applications on YARN.
Quantiles
Quantile metrics data related to Tasks. Units can be seconds, bytes, milliseconds, etc depending on the message type.
QueryList
A list of queries to run on a cluster.
RddDataDistribution
Details about RDD usage.
RddOperationCluster
A grouping of nodes representing higher level constructs (stage, job etc.).
RddOperationEdge
A directed edge representing dependency between two RDDs.
RddOperationGraph
Graph representing RDD dependencies. Consists of edges and a root cluster.
RddOperationNode
A node in the RDD operation graph. Corresponds to a single RDD.
RddPartitionInfo
Information about RDD partitions.
RddStorageInfo
Overall data about RDD storage.
RegexValidation
Validation based on regular expressions.
RepairClusterRequest
A request to repair a cluster.
RepairNodeGroupRequest
RepositoryConfig
Configuration for dependency repositories
ReservationAffinity
Reservation Affinity for consuming Zonal reservation.
ResizeNodeGroupRequest
A request to resize a node group.
ResourceInformation
ResourceProfileInfo
Resource profile that contains information about all the resources required by executors and tasks.
RuntimeConfig
Runtime configuration for a workload.
RuntimeInfo
Runtime information about workload execution.
SearchSessionSparkApplicationExecutorStageSummaryResponse
List of Executors associated with a Spark Application Stage.
SearchSessionSparkApplicationExecutorsResponse
List of Executors associated with a Spark Application.
SearchSessionSparkApplicationJobsResponse
A list of Jobs associated with a Spark Application.
SearchSessionSparkApplicationSqlQueriesResponse
List of all queries for a Spark Application.
SearchSessionSparkApplicationStageAttemptTasksResponse
List of tasks for a stage of a Spark Application
SearchSessionSparkApplicationStageAttemptsResponse
A list of Stage Attempts for a Stage of a Spark Application.
SearchSessionSparkApplicationStagesResponse
A list of stages associated with a Spark Application.
SearchSessionSparkApplicationsResponse
A list of summary of Spark Applications
SearchSparkApplicationExecutorStageSummaryResponse
List of Executors associated with a Spark Application Stage.
SearchSparkApplicationExecutorsResponse
List of Executors associated with a Spark Application.
SearchSparkApplicationJobsResponse
A list of Jobs associated with a Spark Application.
SearchSparkApplicationSqlQueriesResponse
List of all queries for a Spark Application.
SearchSparkApplicationStageAttemptTasksResponse
List of tasks for a stage of a Spark Application
SearchSparkApplicationStageAttemptsResponse
A list of Stage Attempts for a Stage of a Spark Application.
SearchSparkApplicationStagesResponse
A list of stages associated with a Spark Application.
SearchSparkApplicationsResponse
A list of summary of Spark Applications
SecurityConfig
Security related configuration, including encryption, Kerberos, etc.
Session
A representation of a session.
SessionOperationMetadata
Metadata describing the Session operation.
SessionStateHistory
Historical state information.
SessionTemplate
A representation of a session template.
SetIamPolicyRequest
Request message for SetIamPolicy method.
ShieldedInstanceConfig
Shielded Instance Config for clusters using Compute Engine Shielded VMs (https://cloud.google.com/security/shielded-cloud/shielded-vm).
ShufflePushReadMetrics
ShufflePushReadQuantileMetrics
ShuffleReadMetrics
Shuffle data read by the task.
ShuffleReadQuantileMetrics
ShuffleWriteMetrics
Shuffle data written by task.
ShuffleWriteQuantileMetrics
SinkProgress
SoftwareConfig
Specifies the selection and config of software inside the cluster.
SourceProgress
SparkApplication
A summary of Spark Application
SparkBatch
A configuration for running an Apache Spark (https://spark.apache.org/) batch workload.
SparkConnectConfig
Spark connect configuration for an interactive session.
SparkHistoryServerConfig
Spark History Server configuration for the workload.
SparkJob
A Dataproc job for running Apache Spark (https://spark.apache.org/) applications on YARN.
SparkPlanGraph
A graph used for storing information of an executionPlan of DataFrame.
SparkPlanGraphCluster
Represents a tree of spark plan.
SparkPlanGraphEdge
Represents a directed edge in the spark plan tree from child to parent.
SparkPlanGraphNode
Represents a node in the spark plan tree.
SparkPlanGraphNodeWrapper
Wrapper user to represent either a node or a cluster.
SparkRBatch
A configuration for running an Apache SparkR (https://spark.apache.org/docs/latest/sparkr.html) batch workload.
SparkRJob
A Dataproc job for running Apache SparkR (https://spark.apache.org/docs/latest/sparkr.html) applications on YARN.
SparkRuntimeInfo
SparkSqlBatch
A configuration for running Apache Spark SQL (https://spark.apache.org/sql/) queries as a batch workload.
SparkSqlJob
A Dataproc job for running Apache Spark SQL (https://spark.apache.org/sql/) queries.
SparkStandaloneAutoscalingConfig
Basic autoscaling configurations for Spark Standalone.
SparkWrapperObject
Outer message that contains the data obtained from spark listener, packaged with information that is required to process it.
SpeculationStageSummary
Details of the speculation task when speculative execution is enabled.
SqlExecutionUiData
SQL Execution Data
SqlPlanMetric
Metrics related to SQL execution.
StageAttemptTasksSummary
Data related to tasks summary for a Spark Stage Attempt
StageData
Data corresponding to a stage.
StageInputMetrics
Metrics about the input read by the stage.
StageMetrics
Stage Level Aggregated Metrics
StageOutputMetrics
Metrics about the output written by the stage.
StageShufflePushReadMetrics
StageShuffleReadMetrics
Shuffle data read for the stage.
StageShuffleWriteMetrics
Shuffle data written for the stage.
StagesSummary
Data related to Stages page summary
StartClusterRequest
A request to start a cluster.
StartupConfig
Configuration to handle the startup of instances during cluster create and update process.
StateHistory
Historical state information.
StateOperatorProgress
Status
The Status type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by gRPC (https://github.com/grpc). Each Status message contains three pieces of data: error code, error message, and error details.You can find out more about this error model and how to work with it in the API Design Guide (https://cloud.google.com/apis/design/errors).
StopClusterRequest
A request to stop a cluster.
StreamBlockData
Stream Block Data.
StreamingQueryData
Streaming
StreamingQueryProgress
SubmitJobRequest
A request to submit a job.
SummarizeSessionSparkApplicationExecutorsResponse
Consolidated summary of executors for a Spark Application.
SummarizeSessionSparkApplicationJobsResponse
Summary of a Spark Application jobs.
SummarizeSessionSparkApplicationStageAttemptTasksResponse
Summary of tasks for a Spark Application stage attempt.
SummarizeSessionSparkApplicationStagesResponse
Summary of a Spark Application stages.
SummarizeSparkApplicationExecutorsResponse
Consolidated summary of executors for a Spark Application.
SummarizeSparkApplicationJobsResponse
Summary of a Spark Application jobs.
SummarizeSparkApplicationStageAttemptTasksResponse
Summary of tasks for a Spark Application stage attempt.
SummarizeSparkApplicationStagesResponse
Summary of a Spark Application stages.
TaskData
Data corresponding to tasks created by spark.
TaskMetrics
Executor Task Metrics
TaskQuantileMetrics
TaskResourceRequest
Resources used per task created by the application.
TemplateParameter
A configurable parameter that replaces one or more fields in the template. Parameterizable fields: - Labels - File uris - Job properties - Job arguments - Script variables - Main class (in HadoopJob and SparkJob) - Zone (in ClusterSelector)
TerminateSessionRequest
A request to terminate an interactive session.
TestIamPermissionsRequest
Request message for TestIamPermissions method.
TestIamPermissionsResponse
Response message for TestIamPermissions method.
TrinoJob
A Dataproc job for running Trino (https://trino.io/) queries. IMPORTANT: The Dataproc Trino Optional Component (https://cloud.google.com/dataproc/docs/concepts/components/trino) must be enabled when the cluster is created to submit a Trino job to the cluster.
UsageMetrics
Usage metrics represent approximate total resources consumed by a workload.
UsageSnapshot
The usage snapshot represents the resources consumed by a workload at a specified time.
ValueValidation
Validation based on a list of allowed values.
VirtualClusterConfig
The Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/guides/dpgke/dataproc-gke-overview).
WorkflowGraph
The workflow graph.
WorkflowMetadata
A Dataproc workflow template resource.
WorkflowNode
The workflow node.
WorkflowTemplate
A Dataproc workflow template resource.
WorkflowTemplatePlacement
Specifies workflow execution target.Either managed_cluster or cluster_selector is required.
WriteSessionSparkApplicationContextRequest
Write Spark Application data to internal storage systems
WriteSessionSparkApplicationContextResponse
Response returned as an acknowledgement of receipt of data.
WriteSparkApplicationContextRequest
Write Spark Application data to internal storage systems
WriteSparkApplicationContextResponse
Response returned as an acknowledgement of receipt of data.
YarnApplication
A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.Beta Feature: This report is available for testing purposes only. It may be changed before final release.