google::cloud::storage::ObjectWriteStream Class Reference

Defines a std::basic_ostream<char> to write to a GCS object. More...

#include <google/cloud/storage/object_write_stream.h>

Public Member Functions
	ObjectWriteStream ()
	Creates a stream not associated with any buffer. More...
	ObjectWriteStream (std::unique_ptr< internal::ObjectWriteStreambuf > buf)
	Creates a stream associated with the give request. More...
	ObjectWriteStream (ObjectWriteStream &&rhs) noexcept
ObjectWriteStream &	operator= (ObjectWriteStream &&rhs) noexcept
void	swap (ObjectWriteStream &rhs)
	ObjectWriteStream (ObjectWriteStream const &)=delete
ObjectWriteStream &	operator= (ObjectWriteStream const &)=delete
	~ObjectWriteStream () override
	Closes the stream (if necessary). More...
bool	IsOpen () const
	Return true if the stream is open to write more data. More...
void	Close ()
	Close the stream, finalizing the upload. More...
StatusOr< ObjectMetadata > const &	metadata () const &
	Access the upload results. More...
StatusOr< ObjectMetadata > &&	metadata () &&
std::string const &	received_hash () const
	The received CRC32C checksum and the MD5 hash values as reported by GCS. More...
std::string const &	computed_hash () const
	The locally computed checksum and hashes, as a string. More...
HeadersMap const &	headers () const
	The headers (if any) returned by the service. More...
std::string const &	payload () const
	The returned payload as a raw string, for debugging only. More...
std::string const &	resumable_session_id () const
	Returns the resumable upload session id for this upload. More...
std::uint64_t	next_expected_byte () const
	Returns the next expected byte. More...
void	Suspend () &&
	Suspends an upload. More...
Status	last_status () const
	Returns the status of partial errors. More...

Detailed Description

Defines a std::basic_ostream<char> to write to a GCS object.

This class is used to upload objects to GCS. It can handle objects of any size, but keep the following considerations in mind:

• This API is designed for applications that need to stream the object payload. If you have the payload as one large buffer consider using storage::Client::InsertObject(), it is simpler and faster in most cases.
• This API can be used to perform unformatted I/O, as well as formatted I/O using the familiar operator<< APIs. Note that formatted I/O typically implies some form of buffering and data copying. For best performance, consider using the .write() member function.
• GCS expects to receive data in multiples of the upload quantum (256KiB). Sending a buffer that is not a multiple of this quantum terminates the upload. This constraints the implementation of buffered and unbuffered I/O as described below.

Unformatted I/O

On a .write() call this class attempts to send the data immediately, this this the unbuffered API after all. If any previously buffered data and the data provided in the .write() call are larger than an upload quantum the class sends data immediately. Any data in excess of a multiple of the upload quantum are buffered for the next upload.

These examples may clarify how this works:

1. Consider a fresh ObjectWriteStream that receives a .write() call with 257 KiB of data. The first 256 KiB are immediately sent and the remaining 1 KiB is buffered for a future upload.
2. If the same stream receives another .write() call with 256 KiB then it will send the buffered 1 KiB of data and the first 255 KiB from the new buffer. The last 1 KiB is buffered for a future upload.
3. Consider a fresh ObjectWriteStream that receives a .write() call with 4 MiB of data. This data is sent immediately, and no data is buffered.
4. Consider a stream with a 256 KiB buffer from previous buffered I/O (see below to understand how this might happen). If this stream receives a .write() call with 1024 KiB then both the 256 KiB and the 1024 KiB of data are uploaded immediately.

Formatted I/O

When performing formatted I/O, typically used via operator<<, this class will buffer data based on theClientOptions::upload_buffer_size() setting. Note that this setting is expressed in bytes, but it is always rounded (up) to an upload quantum.

Recommendations

For best performance uploading data we recommend using exclusively the unbuffered I/O API. Furthermore, we recommend that applications use data in multiples of the upload quantum in all calls to .write(). Larger buffers result in better performance. Note that our empirical results show that these improvements tapper off around 32MiB or so.

Suspending Uploads

Note that, as it is customary in C++, the destructor of this class finalizes the upload. If you want to prevent the class from finalizing an upload, use the Suspend() function.

Example: starting a resumable upload.

  namespace gcs = ::google::cloud::storage;
  return [](gcs::Client client, std::string const& bucket_name,
            std::string const& object_name) {
    gcs::ObjectWriteStream stream = client.WriteObject(
        bucket_name, object_name, gcs::NewResumableUploadSession(),
        gcs::AutoFinalizeDisabled());
    auto session_id = stream.resumable_session_id();
    std::cout << "Created resumable upload: " << session_id << "\n";
    // Because this stream was created with `AutoFinalizeDisabled()` its
    // destructor will *not* finalize the upload, allowing a separate process or
    // function to resume and continue the upload.
    stream << "This data will not get uploaded, it is too small\n";
    return session_id;
  }

Example: resuming a resumable upload.

  namespace gcs = ::google::cloud::storage;
  using ::google::cloud::StatusOr;
  [](gcs::Client client, std::string const& bucket_name,
     std::string const& object_name, std::string const& session_id) {
    // Restore a resumable upload stream, the library automatically queries the
    // state of the upload and discovers the next expected byte.
    gcs::ObjectWriteStream stream =
        client.WriteObject(bucket_name, object_name,
                           gcs::RestoreResumableUploadSession(session_id));
    if (!stream.IsOpen() && stream.metadata().ok()) {
      std::cout << "The upload has already been finalized.  The object "
                << "metadata is: " << *stream.metadata() << "\n";
    }
    if (stream.next_expected_byte() == 0) {
      // In this example we create a small object, smaller than the resumable
      // upload quantum (256 KiB), so either all the data is there or not.
      // Applications use `next_expected_byte()` to find the position in their
      // input where they need to start uploading.
      stream << R"""(
Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim
ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea
commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit
esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat
non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
)""";
    }
 
    stream.Close();
 
    StatusOr<gcs::ObjectMetadata> metadata = stream.metadata();
    if (!metadata) throw std::runtime_error(metadata.status().message());
    std::cout << "Upload completed, the new object metadata is: " << *metadata
              << "\n";
  }

Definition at line 103 of file object_write_stream.h.

Constructor & Destructor Documentation

ObjectWriteStream() [1/4]

google::cloud::storage::ObjectWriteStream::ObjectWriteStream

(

)

Creates a stream not associated with any buffer.

Attempts to use this stream will result in failures.

ObjectWriteStream() [2/4]

google::cloud::storage::ObjectWriteStream::ObjectWriteStream ( std::unique_ptr< internal::ObjectWriteStreambuf >  buf )

explicit

Creates a stream associated with the give request.

Reading from the stream will result in http requests to get more data from the GCS object.

Parameters

buf	an initialized ObjectWriteStreambuf to upload the data.

ObjectWriteStream() [3/4]

google::cloud::storage::ObjectWriteStream::ObjectWriteStream ( ObjectWriteStream &&  rhs )

noexcept

ObjectWriteStream() [4/4]

google::cloud::storage::ObjectWriteStream::ObjectWriteStream ( ObjectWriteStream const &  )

delete

~ObjectWriteStream()

google::cloud::storage::ObjectWriteStream::~ObjectWriteStream ( )

override

Closes the stream (if necessary).

Member Function Documentation

Close()

void google::cloud::storage::ObjectWriteStream::Close

(

)

Close the stream, finalizing the upload.

Closing a stream completes an upload and creates the uploaded object. On failure it sets the badbit of the stream.

The metadata of the uploaded object, or a detailed error status, is accessible via the metadata() member function. Note that the metadata may be empty if the application creates a stream with the Fields("") parameter, applications cannot assume that all fields in the metadata are filled on success.

Exceptions

If	the application has enabled the exception mask this function may throw std::ios_base::failure.

computed_hash()

std::string const& google::cloud::storage::ObjectWriteStream::computed_hash ( ) const

inline

The locally computed checksum and hashes, as a string.

This object computes the CRC32C checksum and MD5 hash of the uploaded data. There are several cases where these values may be empty or irrelevant, for example:

• When performing resumable uploads the stream may not have had access to the full data.
• The application may disable the CRC32C and/or the MD5 hash computation.

The string has the same format as the value returned by received_hash(). Note that the format of this string is also subject to change without notice.

See also

https://cloud.google.com/storage/docs/hashes-etags for more information on checksums and hashes in GCS.

Definition at line 228 of file object_write_stream.h.

headers()

HeadersMap const& google::cloud::storage::ObjectWriteStream::headers ( ) const

inline

The headers (if any) returned by the service.

For debugging only.

Warning

The contents of these headers may change without notice. Unless documented in the API, headers may be removed or added by the service. Also note that the client library uses both the XML and JSON API, choosing between them based on the feature set (some functionality is only available through the JSON API), and performance. Consequently, the headers may be different on requests using different features. Likewise, the headers may change from one version of the library to the next, as we find more (or different) opportunities for optimization.

Definition at line 242 of file object_write_stream.h.

IsOpen()

bool google::cloud::storage::ObjectWriteStream::IsOpen ( ) const

inline

Return true if the stream is open to write more data.

Note

write streams can be "born closed" when created using a previously finalized upload session. Applications that restore a previous session should check the state, for example:

auto stream = client.WriteObject(...,
    gcs::RestoreResumableUploadSession(session_id));
if (!stream.IsOpen() && stream.metadata().ok()) {
  std::cout << "Yay! The upload was finalized previously.\n";
  return;
}

Definition at line 164 of file object_write_stream.h.

last_status()

Status google::cloud::storage::ObjectWriteStream::last_status ( ) const

inline

Returns the status of partial errors.

Application may write multiple times before closing the stream, this function gives the capability to find out status even before stream closure.

This function is different than metadata() as calling metadata() before Close() is undefined.

Definition at line 292 of file object_write_stream.h.

metadata() [1/2]

StatusOr<ObjectMetadata>&& google::cloud::storage::ObjectWriteStream::metadata ( ) &&

inline

Definition at line 191 of file object_write_stream.h.

metadata() [2/2]

StatusOr<ObjectMetadata> const& google::cloud::storage::ObjectWriteStream::metadata ( ) const &

inline

Access the upload results.

Note that calling these member functions before Close() is undefined behavior.

Definition at line 190 of file object_write_stream.h.

next_expected_byte()

std::uint64_t google::cloud::storage::ObjectWriteStream::next_expected_byte ( ) const

inline

Returns the next expected byte.

For non-resumable uploads this is always zero. Applications that use resumable uploads can use this value to resend any data not committed in the GCS.

Definition at line 266 of file object_write_stream.h.

operator=() [1/2]

ObjectWriteStream& google::cloud::storage::ObjectWriteStream::operator= ( ObjectWriteStream &&  rhs )

inlinenoexcept

Definition at line 125 of file object_write_stream.h.

operator=() [2/2]

ObjectWriteStream& google::cloud::storage::ObjectWriteStream::operator= ( ObjectWriteStream const &  )

delete

payload()

std::string const& google::cloud::storage::ObjectWriteStream::payload ( ) const

inline

The returned payload as a raw string, for debugging only.

Definition at line 245 of file object_write_stream.h.

received_hash()

std::string const& google::cloud::storage::ObjectWriteStream::received_hash ( ) const

inline

The received CRC32C checksum and the MD5 hash values as reported by GCS.

When the upload is finalized (via Close()) the GCS server reports the CRC32C checksum and, if the object is not a composite object, the MDF hash of the uploaded data. This class compares the reported hashes against locally computed hash values, and reports an error if they do not match.

The values are reported as comma separated tag=value pairs, e.g. crc32c=AAAAAA==,md5=1B2M2Y8AsgTpgAmY7PhCfg==. The format of this string is subject to change without notice, they are provided for informational purposes only.

See also

https://cloud.google.com/storage/docs/hashes-etags for more information on checksums and hashes in GCS.

Definition at line 209 of file object_write_stream.h.

resumable_session_id()

std::string const& google::cloud::storage::ObjectWriteStream::resumable_session_id ( ) const

inline

Returns the resumable upload session id for this upload.

Note that this is an empty string for uploads that do not use resumable upload session ids. Client::WriteObject() enables resumable uploads based on the options set by the application.

Definition at line 255 of file object_write_stream.h.

Suspend()

void google::cloud::storage::ObjectWriteStream::Suspend

(

)

&&

Suspends an upload.

This is a destructive operation. Using this object after calling this function results in undefined behavior. Applications should copy any necessary state (such as the value resumable_session_id()) before calling this function.

  namespace gcs = ::google::cloud::storage;
  return [](gcs::Client client, std::string const& bucket_name,
            std::string const& object_name) {
    gcs::ObjectWriteStream stream = client.WriteObject(
        bucket_name, object_name, gcs::NewResumableUploadSession());
    auto session_id = stream.resumable_session_id();
    std::cout << "Created resumable upload: " << session_id << "\n";
    // As it is customary in C++, the destructor automatically closes the
    // stream, that would finish the upload and create the object. For this
    // example we want to restore the session as-if the application had crashed,
    // where no destructors get called.
    stream << "This data will not get uploaded, it is too small\n";
    std::move(stream).Suspend();
    return session_id;
  }

swap()

void google::cloud::storage::ObjectWriteStream::swap ( ObjectWriteStream &  rhs )

inline

Definition at line 131 of file object_write_stream.h.