Google Cloud Storage C++ Client Library

The Google Cloud Storage (GCS) C++ Client library offers types and functions access GCS from C++11 applications. It offers full access to the GCS API, including operations to list, read, write, and delete GCS objects and buckets. The library also provides functions to modify the IAM permissions on buckets, read and modify the metadata associated with objects and buckets, configure encryption keys, configure notifications via Cloud Pub/Sub, and change the access control list of object or buckets.

This library requires a C++11 compiler. It is supported (and tested) on multiple Linux distributions, as well as Windows and macOS. The README on GitHub provides detailed instructions to install the necessary dependencies, as well as how to compile the client library.

Setting up your repo

In order to use the Cloud Cloud Storage C++ client library from your own code, you'll need to configure your build system how to fetch and compile the Cloud C++ client library. The Cloud C++ client library natively supports the Bazel and CMake build systems. We've created a minimal, "Hello world", [quickstart repo][quickstart-link] that includes detailed instructions on how to compile the library for use in your application. You can fetch the source from GitHub as normal:

git clone https://github.com/googleapis/google-cloud-cpp.git
cd google-cloud-cpp/google/cloud/storage/quickstart

Example: Quickstart

The following shows the code that you'll run in the google/cloud/storage/quickstart/ directory, which should give you a taste of the Cloud Storage C++ client library API.

// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
 
#include "google/cloud/storage/client.h"
#include <iostream>
 
int main(int argc, char* argv[]) {
  if (argc != 2) {
    std::cerr << "Missing bucket name.\n";
    std::cerr << "Usage: quickstart <bucket-name>\n";
    return 1;
  }
  std::string const bucket_name = argv[1];
 
  // Create aliases to make the code easier to read.
  namespace gcs = ::google::cloud::storage;
 
  // Create a client to communicate with Google Cloud Storage. This client
  // uses the default configuration for authentication and project id.
  auto client = gcs::Client();
 
  auto writer = client.WriteObject(bucket_name, "quickstart.txt");
  writer << "Hello World!";
  writer.Close();
  if (writer.metadata()) {
    std::cout << "Successfully created object: " << *writer.metadata() << "\n";
  } else {
    std::cerr << "Error creating object: " << writer.metadata().status()
              << "\n";
    return 1;
  }
 
  auto reader = client.ReadObject(bucket_name, "quickstart.txt");
  if (!reader) {
    std::cerr << "Error reading object: " << reader.status() << "\n";
    return 1;
  }
 
  std::string contents{std::istreambuf_iterator<char>{reader}, {}};
  std::cout << contents << "\n";
 
  return 0;
}

API Notes

• GOOGLE_CLOUD_PROJECT=... is used in examples and integration tests to configure the GCP project.
• CLOUD_STORAGE_ENABLE_TRACING=raw-client enables logging for each "RPC" generated by the library. Unless you have configured you own logging backend, you should also set GOOGLE_CLOUD_CPP_ENABLE_CLOG to produce any output on the program's console.
• CLOUD_STORAGE_ENABLE_TRACING=http enables logging for the full http traffic generated by the library.
• CLOUD_STORAGE_ENABLE_TRACING=raw-client,http enables all logging.
• CLOUD_STORAGE_EMULATOR_ENDPOINT=... override the default endpoint used by the library, intended for testing only.
• CLOUD_STORAGE_TESTBENCH_ENDPOINT=... DEPRECATED please use CLOUD_STORAGE_EMULATOR_ENDPOINT instead.
• GOOGLE_CLOUD_CPP_ENABLE_CLOG=yes turns on logging in the library, basically the library always "logs" but the logging infrastructure has no backend to actually print anything until the application sets a backend or they set this environment variable.

Experimental

• GOOGLE_CLOUD_CPP_STORAGE_REST_CONFIG=... configuration for the REST protocol, currently only the disable-xml value has any effect. Sometimes the application developer may want to test using an emulator that does not support XML, while the library defaults to XML for some media operations.
• GOOGLE_CLOUD_CPP_STORAGE_GRPC_CONFIG=... used with google::cloud::storage_experimental::DefaultGrpcClient() to configure configure the gRPC protocol. Setting this to media enables gRPC for just media operations (reading and writing data), while setting this to metadata enables gRPC for all operations. Note that gRPC support is an early access program, contact Google Cloud support for details.

Using GOOGLE_CLOUD_PROJECT to set the default project

Some of the GCS APIs need a project as a parameter. For such APIs the application can (a) call an overload that takes the project id as a parameter, (b) define the project explicitly when the Client object is constructed, or (c) set the GOOGLE_CLOUD_PROJECT environment variable to the desired project id. If you have set this environment variable the code can be even simpler:

  namespace gcs = ::google::cloud::storage;
  using ::google::cloud::StatusOr;
  [](gcs::Client client, std::string const& bucket_name) {
    StatusOr<gcs::BucketMetadata> bucket_metadata =
        client.CreateBucket(bucket_name, gcs::BucketMetadata());
 
    if (!bucket_metadata) {
      throw std::runtime_error(bucket_metadata.status().message());
    }
 
    std::cout << "Bucket " << bucket_metadata->name() << " created."
              << "\nFull Metadata: " << *bucket_metadata << "\n";
  }

Error Handling

This library never throws exceptions to signal error. In general, the library returns a StatusOr<T> if an error is possible. Some functions return objects that already have an existing error handling mechanism, such as types derived from std::ostream where the application can check the state flags to determine if there was an error. In these cases no StatusOr wrapper is used.

Error Handling Example (without exceptions):

Applications that do not use exceptions to signal errors should check if the StatusOr<T> contains a value before using it. If the StatusOr<T> does contain a value then the StatusOr<T> can be used as a smart pointer to T. That is, operator->() and operator*() work as you would expect. If the StatusOr<T> does not contain a value then the error details are available using the .status() member function (and trying to access the value produces undefined behavior).

using gcs = ::google::cloud::storage;
[](gcs::Client client) {
  google::cloud::StatusOr<gcs::BucketMetadata> metadata =
    client.GetBucketMetadata("my-bucket");
 
  if (!metadata) {
    std::cerr << "GetBucketMetadata: " << metadata.status() << "\n";
    return;
  }
 
  // use `metadata` as a smart pointer to `BucketMetadata`
  std::cout << "The metadata for bucket " << metadata->name()
            << " is " << *metadata << "\n";
}

Error Handling Example (with exceptions):

Applications that use exceptions to signal errors can simply call .value() on the StatusOr<T> object. This will return a T if the StatusOr<T> object contains a value, and will otherwise throw an exception.

using gcs = ::google::cloud::storage;
[](gcs::Client client) {
  gcs::BucketMetadata metadata = client.GetBucketMetadata(
        "my-bucket").value();  // throws on error
  std::cout << "The metadata for bucket " << metadata.name()
            << " is " << metadata << "\n";
}

Retry, Backoff, and Idempotency Policies.

The library automatically retries requests that fail with transient errors, and follows the recommended practice to backoff between retries. Application developers can override the default retry and backoff policies:

  namespace gcs = ::google::cloud::storage;
  using ::google::cloud::StatusOr;
  [](std::string const& bucket_name, std::string const& object_name,
     std::string const& contents) {
    // Create a client that only gives up on the third error. The default policy
    // is to retry for several minutes.
    auto client =
        gcs::Client(google::cloud::Options{}.set<gcs::RetryPolicyOption>(
            gcs::LimitedErrorCountRetryPolicy(3).clone()));
 
    StatusOr<gcs::ObjectMetadata> object_metadata =
        client.InsertObject(bucket_name, object_name, std::move(contents),
                            gcs::IfGenerationMatch(0));
 
    if (!object_metadata) {
      throw std::runtime_error(object_metadata.status().message());
    }
 
    std::cout << "The object " << object_metadata->name()
              << " was created in bucket " << object_metadata->bucket()
              << "\nFull metadata: " << *object_metadata << "\n";
  }

The default policies are to continue retrying for up to 15 minutes, and to use truncated (at 5 minutes) exponential backoff, doubling the maximum backoff period between retries.

By default the library retries all operations, even those that are not idempotent. Application developers can override the idempotency policy to follow a more conservative approach:

  namespace gcs = ::google::cloud::storage;
  using ::google::cloud::StatusOr;
  [](std::string const& bucket_name, std::string const& object_name,
     std::string const& contents) {
    // Create a client that only retries idempotent operations, the default is
    // to retry all operations.
    auto client =
        gcs::Client(google::cloud::Options{}.set<gcs::IdempotencyPolicyOption>(
            gcs::StrictIdempotencyPolicy().clone()));
    StatusOr<gcs::ObjectMetadata> object_metadata =
        client.InsertObject(bucket_name, object_name, std::move(contents),
                            gcs::IfGenerationMatch(0));
 
    if (!object_metadata) {
      throw std::runtime_error(object_metadata.status().message());
    }
 
    std::cout << "The object " << object_metadata->name()
              << " was created in bucket " << object_metadata->bucket()
              << "\nFull metadata: " << *object_metadata << "\n";
  }

See also

LimitedTimeRetryPolicy and LimitedErrorCountRetryPolicy for alternative retry policies.

ExponentialBackoffPolicy to configure different parameters for the exponential backoff policy.

AlwaysRetryIdempotencyPolicy and StrictIdempotencyPolicy for alternative idempotency policies.

Next Steps

The documentation for each member function in the Client class includes short snippets on how to call the function. The snippets for ReadObject(), WriteObject(), ListObjects(), and DeleteObject() are good places to continue learning about the library.

Example for mocking GCS C++ Client