Google Cloud Storage C++ Client  1.31.1
A C++ Client Library for Google Cloud Storage
Public Member Functions | List of all members
google::cloud::storage::v1::ObjectWriteStream Class Reference

Defines a std::basic_ostream<char> to write to a GCS object. More...

#include <google/cloud/storage/object_write_stream.h>

Public Member Functions

 ObjectWriteStream ()
 Creates a stream not associated with any buffer. More...
 
 ObjectWriteStream (std::unique_ptr< internal::ObjectWriteStreambuf > buf)
 Creates a stream associated with the give request. More...
 
 ObjectWriteStream (ObjectWriteStream &&rhs) noexcept
 
ObjectWriteStreamoperator= (ObjectWriteStream &&rhs) noexcept
 
void swap (ObjectWriteStream &rhs)
 
 ObjectWriteStream (ObjectWriteStream const &)=delete
 
ObjectWriteStreamoperator= (ObjectWriteStream const &)=delete
 
 ~ObjectWriteStream () override
 Closes the stream (if necessary). More...
 
bool IsOpen () const
 Return true if the stream is open to write more data. More...
 
void Close ()
 Close the stream, finalizing the upload. More...
 
StatusOr< ObjectMetadata > const & metadata () const &
 Access the upload results. More...
 
StatusOr< ObjectMetadata > && metadata () &&
 
std::string const & received_hash () const
 The received CRC32C checksum and the MD5 hash values as reported by GCS. More...
 
std::string const & computed_hash () const
 The locally computed checksum and hashes, as a string. More...
 
HeadersMap const & headers () const
 The headers (if any) returned by the service. More...
 
std::string const & payload () const
 The returned payload as a raw string, for debugging only. More...
 
std::string const & resumable_session_id () const
 Returns the resumable upload session id for this upload. More...
 
std::uint64_t next_expected_byte () const
 Returns the next expected byte. More...
 
void Suspend () &&
 Suspends an upload. More...
 
Status last_status () const
 Returns the status of partial errors. More...
 

Detailed Description

Defines a std::basic_ostream<char> to write to a GCS object.

This class is used to upload objects to GCS. It can handle objects of any size, but keep the following considerations in mind:

Unformatted I/O
On a .write() call this class attempts to send the data immediately, this this the unbuffered API after all. If any previously buffered data and the data provided in the .write() call are larger than an upload quantum the class sends data immediately. Any data in excess of a multiple of the upload quantum are buffered for the next upload.

These examples may clarify how this works:

  1. Consider a fresh ObjectWriteStream that receives a .write() call with 257 KiB of data. The first 256 KiB are immediately sent and the remaining 1 KiB is buffered for a future upload.
  2. If the same stream receives another .write() call with 256 KiB then it will send the buffered 1 KiB of data and the first 255 KiB from the new buffer. The last 1 KiB is buffered for a future upload.
  3. Consider a fresh ObjectWriteStream that receives a .write() call with 4 MiB of data. This data is sent immediately, and no data is buffered.
  4. Consider a stream with a 256 KiB buffer from previous buffered I/O (see below to understand how this might happen). If this stream receives a .write() call with 1024 KiB then both the 256 KiB and the 1024 KiB of data are uploaded immediately.
Formatted I/O
When performing formatted I/O, typically used via operator<<, this class will buffer data based on theClientOptions::upload_buffer_size() setting. Note that this setting is expressed in bytes, but it is always rounded (up) to an upload quantum.
Recommendations
For best performance uploading data we recommend using exclusively the unbuffered I/O API. Furthermore, we recommend that applications use data in multiples of the upload quantum in all calls to .write(). Larger buffers result in better performance. Note that our empirical results show that these improvements tapper off around 32MiB or so.
Suspending Uploads
Note that, as it is customary in C++, the destructor of this class finalizes the upload. If you want to prevent the class from finalizing an upload, use the Suspend() function.
Example: starting a resumable upload.
namespace gcs = ::google::cloud::storage;
return [](gcs::Client client, std::string const& bucket_name,
std::string const& object_name) {
bucket_name, object_name, gcs::NewResumableUploadSession(),
auto session_id = stream.resumable_session_id();
std::cout << "Created resumable upload: " << session_id << "\n";
// Because this stream was created with `AutoFinalizeDisabled()` its
// destructor will *not* finalize the upload, allowing a separate process or
// function to resume and continue the upload.
stream << "This data will not get uploaded, it is too small\n";
return session_id;
}
The Google Cloud Storage (GCS) Client.
Definition: client.h:195
ObjectWriteStream WriteObject(std::string const &bucket_name, std::string const &object_name, Options &&... options)
Writes contents into an object.
Definition: client.h:1231
Defines a std::basic_ostream<char> to write to a GCS object.
std::string const & resumable_session_id() const
Returns the resumable upload session id for this upload.
UseResumableUploadSession NewResumableUploadSession()
Create a UseResumableUploadSession option that requests new sessions.
AutoFinalize AutoFinalizeDisabled()
Configure a stream to leave uploads pending (not finalized) on destruction.
Definition: auto_finalize.h:54
Contains all the Google Cloud Storage C++ client APIs.
Example: resuming a resumable upload.
namespace gcs = ::google::cloud::storage;
using ::google::cloud::StatusOr;
[](gcs::Client client, std::string const& bucket_name,
std::string const& object_name, std::string const& session_id) {
// Restore a resumable upload stream, the library automatically queries the
// state of the upload and discovers the next expected byte.
client.WriteObject(bucket_name, object_name,
if (!stream.IsOpen() && stream.metadata().ok()) {
std::cout << "The upload has already been finalized. The object "
<< "metadata is: " << *stream.metadata() << "\n";
}
if (stream.next_expected_byte() == 0) {
// In this example we create a small object, smaller than the resumable
// upload quantum (256 KiB), so either all the data is there or not.
// Applications use `next_expected_byte()` to find the position in their
// input where they need to start uploading.
stream << R"""(
Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim
ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea
commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit
esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat
non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
)""";
}
stream.Close();
StatusOr<gcs::ObjectMetadata> metadata = stream.metadata();
if (!metadata) throw std::runtime_error(metadata.status().message());
std::cout << "Upload completed, the new object metadata is: " << *metadata
<< "\n";
}
std::uint64_t next_expected_byte() const
Returns the next expected byte.
StatusOr< ObjectMetadata > const & metadata() const &
Access the upload results.
bool IsOpen() const
Return true if the stream is open to write more data.
void Close()
Close the stream, finalizing the upload.
UseResumableUploadSession RestoreResumableUploadSession(std::string session_id)
Create a UseResumableUploadSession option that restores previous sessions.

Definition at line 106 of file object_write_stream.h.

Constructor & Destructor Documentation

◆ ObjectWriteStream() [1/4]

google::cloud::storage::v1::ObjectWriteStream::ObjectWriteStream ( )

Creates a stream not associated with any buffer.

Attempts to use this stream will result in failures.

Definition at line 39 of file object_write_stream.cc.

◆ ObjectWriteStream() [2/4]

google::cloud::storage::v1::ObjectWriteStream::ObjectWriteStream ( std::unique_ptr< internal::ObjectWriteStreambuf >  buf)
explicit

Creates a stream associated with the give request.

Reading from the stream will result in http requests to get more data from the GCS object.

Parameters
bufan initialized ObjectWriteStreambuf to upload the data.

Definition at line 42 of file object_write_stream.cc.

◆ ObjectWriteStream() [3/4]

google::cloud::storage::v1::ObjectWriteStream::ObjectWriteStream ( ObjectWriteStream &&  rhs)
noexcept

Definition at line 50 of file object_write_stream.cc.

◆ ObjectWriteStream() [4/4]

google::cloud::storage::v1::ObjectWriteStream::ObjectWriteStream ( ObjectWriteStream const &  )
delete

◆ ~ObjectWriteStream()

google::cloud::storage::v1::ObjectWriteStream::~ObjectWriteStream ( )
override

Closes the stream (if necessary).

Definition at line 76 of file object_write_stream.cc.

Member Function Documentation

◆ Close()

void google::cloud::storage::v1::ObjectWriteStream::Close ( )

Close the stream, finalizing the upload.

Closing a stream completes an upload and creates the uploaded object. On failure it sets the badbit of the stream.

The metadata of the uploaded object, or a detailed error status, is accessible via the metadata() member function. Note that the metadata may be empty if the application creates a stream with the Fields("") parameter, applications cannot assume that all fields in the metadata are filled on success.

Exceptions
Ifthe application has enabled the exception mask this function may throw std::ios_base::failure.

Definition at line 84 of file object_write_stream.cc.

◆ computed_hash()

std::string const& google::cloud::storage::v1::ObjectWriteStream::computed_hash ( ) const
inline

The locally computed checksum and hashes, as a string.

This object computes the CRC32C checksum and MD5 hash of the uploaded data. There are several cases where these values may be empty or irrelevant, for example:

  • When performing resumable uploads the stream may not have had access to the full data.
  • The application may disable the CRC32C and/or the MD5 hash computation.

The string has the same format as the value returned by received_hash(). Note that the format of this string is also subject to change without notice.

See also
https://cloud.google.com/storage/docs/hashes-etags for more information on checksums and hashes in GCS.

Definition at line 231 of file object_write_stream.h.

◆ headers()

HeadersMap const& google::cloud::storage::v1::ObjectWriteStream::headers ( ) const
inline

The headers (if any) returned by the service.

For debugging only.

Warning
the contents of these headers may change without notice. Unless documented in the API, headers may be removed or added by the service. Also note that the client library uses both the XML and JSON API, choosing between them based on the feature set (some functionality is only available through the JSON API), and performance. Consequently, the headers may be different on requests using different features. Likewise, the headers may change from one version of the library to the next, as we find more (or different) opportunities for optimization.

Definition at line 245 of file object_write_stream.h.

◆ IsOpen()

bool google::cloud::storage::v1::ObjectWriteStream::IsOpen ( ) const
inline

Return true if the stream is open to write more data.

Note
write streams can be "born closed" when created using a previously finalized upload session. Applications that restore a previous session should check the state, for example:
auto stream = client.WriteObject(...,
if (!stream.IsOpen() && stream.metadata().ok()) {
std::cout << "Yay! The upload was finalized previously.\n";
return;
}

Definition at line 167 of file object_write_stream.h.

◆ last_status()

Status google::cloud::storage::v1::ObjectWriteStream::last_status ( ) const
inline

Returns the status of partial errors.

Application may write multiple times before closing the stream, this function gives the capability to find out status even before stream closure.

This function is different than metadata() as calling metadata() before Close() is undefined.

Definition at line 295 of file object_write_stream.h.

◆ metadata() [1/2]

StatusOr<ObjectMetadata>&& google::cloud::storage::v1::ObjectWriteStream::metadata ( ) &&
inline

Definition at line 194 of file object_write_stream.h.

◆ metadata() [2/2]

StatusOr<ObjectMetadata> const& google::cloud::storage::v1::ObjectWriteStream::metadata ( ) const &
inline

Access the upload results.

Note that calling these member functions before Close() is undefined behavior.

Definition at line 193 of file object_write_stream.h.

◆ next_expected_byte()

std::uint64_t google::cloud::storage::v1::ObjectWriteStream::next_expected_byte ( ) const
inline

Returns the next expected byte.

For non-resumable uploads this is always zero. Applications that use resumable uploads can use this value to resend any data not committed in the GCS.

Definition at line 269 of file object_write_stream.h.

◆ operator=() [1/2]

ObjectWriteStream& google::cloud::storage::v1::ObjectWriteStream::operator= ( ObjectWriteStream &&  rhs)
inlinenoexcept

Definition at line 128 of file object_write_stream.h.

◆ operator=() [2/2]

ObjectWriteStream& google::cloud::storage::v1::ObjectWriteStream::operator= ( ObjectWriteStream const &  )
delete

◆ payload()

std::string const& google::cloud::storage::v1::ObjectWriteStream::payload ( ) const
inline

The returned payload as a raw string, for debugging only.

Definition at line 248 of file object_write_stream.h.

◆ received_hash()

std::string const& google::cloud::storage::v1::ObjectWriteStream::received_hash ( ) const
inline

The received CRC32C checksum and the MD5 hash values as reported by GCS.

When the upload is finalized (via Close()) the GCS server reports the CRC32C checksum and, if the object is not a composite object, the MDF hash of the uploaded data. This class compares the reported hashes against locally computed hash values, and reports an error if they do not match.

The values are reported as comma separated tag=value pairs, e.g. crc32c=AAAAAA==,md5=1B2M2Y8AsgTpgAmY7PhCfg==. The format of this string is subject to change without notice, they are provided for informational purposes only.

See also
https://cloud.google.com/storage/docs/hashes-etags for more information on checksums and hashes in GCS.

Definition at line 212 of file object_write_stream.h.

◆ resumable_session_id()

std::string const& google::cloud::storage::v1::ObjectWriteStream::resumable_session_id ( ) const
inline

Returns the resumable upload session id for this upload.

Note that this is an empty string for uploads that do not use resumable upload session ids. Client::WriteObject() enables resumable uploads based on the options set by the application.

Definition at line 258 of file object_write_stream.h.

◆ Suspend()

void google::cloud::storage::v1::ObjectWriteStream::Suspend ( ) &&

Suspends an upload.

This is a destructive operation. Using this object after calling this function results in undefined behavior. Applications should copy any necessary state (such as the value resumable_session_id()) before calling this function.

namespace gcs = ::google::cloud::storage;
return [](gcs::Client client, std::string const& bucket_name,
std::string const& object_name) {
bucket_name, object_name, gcs::NewResumableUploadSession());
auto session_id = stream.resumable_session_id();
std::cout << "Created resumable upload: " << session_id << "\n";
// As it is customary in C++, the destructor automatically closes the
// stream, that would finish the upload and create the object. For this
// example we want to restore the session as-if the application had crashed,
// where no destructors get called.
stream << "This data will not get uploaded, it is too small\n";
std::move(stream).Suspend();
return session_id;
}

Definition at line 105 of file object_write_stream.cc.

◆ swap()

void google::cloud::storage::v1::ObjectWriteStream::swap ( ObjectWriteStream rhs)
inline

Definition at line 134 of file object_write_stream.h.