Google Cloud Datastore

Google Cloud Datastore is a fully managed, schemaless database for storing non-relational data. You should feel at home if you are familiar with relational databases, but there are some key differences to be aware of to make the most of using Datastore.

The goal of google-cloud is to provide an API that is comfortable to Rubyists. Your authentication credentials are detected automatically in Google Cloud Platform (GCP), including Google Compute Engine (GCE), Google Kubernetes Engine (GKE), Google App Engine (GAE), Google Cloud Functions (GCF) and Cloud Run. In other environments you can configure authentication easily, either directly in your code or via environment variables. Read more about the options for connecting in the Authentication Guide.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new(
  project_id: "my-todo-project",
  credentials: "/path/to/keyfile.json"
)

task = datastore.find "Task", "sampleTask"
task["priority"] = 5
datastore.save task

To learn more about Datastore, read the Google Cloud Datastore Concepts Overview .

Retrieving records

Records, called "entities" in Datastore, are retrieved by using a key. The key is more than a numeric identifier, it is a complex data structure that can be used to model relationships. The simplest key has a string kind value and either a numeric id value or a string name value. A single record can be retrieved by calling Google::Cloud::Datastore::Dataset#find and passing the parts of the key:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"

Optionally, Google::Cloud::Datastore::Dataset#find can be given a key object:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", 123456
task = datastore.find task_key

See Google::Cloud::Datastore::Dataset#find

Querying records

Multiple records can be found that match criteria. (See Google::Cloud::Datastore::Query#where)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)

tasks = datastore.run query

Records can also be ordered. (See Google::Cloud::Datastore::Query#order)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  order("created")

tasks = datastore.run query

The number of records returned can be specified. (See Google::Cloud::Datastore::Query#limit)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  limit(5)

tasks = datastore.run query

When using Datastore in a multitenant application, a query may be run within a namespace using the namespace option. (See Multitenancy)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)

tasks = datastore.run query, namespace: "example-ns"

Records' key structures can also be queried. (See Google::Cloud::Datastore::Query#ancestor)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_list_key = datastore.key "TaskList", "default"

query = datastore.query("Task").
  ancestor(task_list_key)

tasks = datastore.run query

See Google::Cloud::Datastore::Query and Google::Cloud::Datastore::Dataset#run

Paginating records

All records may not return at once, but multiple calls can be made to Datastore to return them all.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task")
tasks = datastore.run query
tasks.all do |t|
  puts t["description"]
end

See Google::Cloud::Datastore::Dataset::LookupResults and Google::Cloud::Datastore::Dataset::QueryResults

Creating records

New entities can be created and persisted buy calling Google::Cloud::Datastore::Dataset#save. The entity must have a key to be saved. If the key is incomplete then it will be completed when saved.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end
task.key.id #=> nil
datastore.save task
task.key.id #=> 123456

Multiple new entities may be created in a batch.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task1 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

task2 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 5
  t["description"] = "Integrate Cloud Datastore"
end

tasks = datastore.save(task1, task2)
task_key1 = tasks[0].key
task_key2 = tasks[1].key

Entities in Datastore form a hierarchically structured space similar to the directory structure of a file system. When you create an entity, you can optionally designate another entity as its parent; the new entity is a child of the parent entity.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"
task_key.parent = datastore.key "TaskList", "default"

task = datastore.entity task_key do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 5
  t["description"] = "Integrate Cloud Datastore"
end

Setting properties

Entities hold properties. A property has a name that is a string or symbol, and a value that is an object. Most value objects are supported, including String, Integer, Date, Time, and even other entity or key objects. Changes to the entity's properties are persisted by calling Google::Cloud::Datastore::Dataset#save.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"
# Read the priority property
task["priority"] #=> 4
# Write the priority property
task["priority"] = 5
# Persist the changes
datastore.save task

Array properties can be used to store more than one value.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask" do |t|
  t["tags"] = ["fun", "programming"]
  t["collaborators"] = ["alice", "bob"]
end

Deleting records

Entities can be removed from Datastore by calling Google::Cloud::Datastore::Dataset#delete and passing the entity object or the entity's key object.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"
datastore.delete task

Multiple entities may be deleted in a batch.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key1 = datastore.key "Task", "sampleTask1"
task_key2 = datastore.key "Task", "sampleTask2"
datastore.delete task_key1, task_key2

Transactions

Complex logic can be wrapped in a transaction. All queries and updates within the Google::Cloud::Datastore::Dataset#transaction block are run within the transaction scope, and will be automatically committed when the block completes.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"

datastore.transaction do |tx|
  if tx.find(task_key).nil?
    task = datastore.entity task_key do |t|
      t["type"] = "Personal"
      t["done"] = false
      t["priority"] = 4
      t["description"] = "Learn Cloud Datastore"
    end
    tx.save task
  end
end

Alternatively, if no block is given the transaction object is returned allowing you to commit or rollback manually.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"

tx = datastore.transaction
begin
  if tx.find(task_key).nil?
    task = datastore.entity task_key do |t|
      t["type"] = "Personal"
      t["done"] = false
      t["priority"] = 4
      t["description"] = "Learn Cloud Datastore"
    end
    tx.save task
  end
  tx.commit
rescue
  tx.rollback
end

A read-only transaction cannot modify entities; in return they do not contend with other read-write or read-only transactions. Using a read-only transaction for transactions that only read data will potentially improve throughput.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_list_key = datastore.key "TaskList", "default"
query = datastore.query("Task").
  ancestor(task_list_key)

tasks = nil

datastore.transaction read_only: true do |tx|
  task_list = tx.find task_list_key
  if task_list
    tasks = tx.run query
  end
end

See Google::Cloud::Datastore::Transaction and Google::Cloud::Datastore::Dataset#transaction

Querying metadata

Datastore provides programmatic access to some of its metadata to support meta-programming, implementing backend administrative functions, simplify consistent caching, and similar purposes. The metadata available includes information about the entity groups, namespaces, entity kinds, and properties your application uses, as well as the property representations for each property.

The special entity kind __namespace__ can be used to find all the namespaces used in your application entities.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("__namespace__").
  select("__key__").
  where("__key__", ">=", datastore.key("__namespace__", "g")).
  where("__key__", "<", datastore.key("__namespace__", "h"))

namespaces = datastore.run(query).map do |entity|
  entity.key.name
end

The special entity kind __kind__ can be used to return all the kinds used in your application.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("__kind__").
  select("__key__")

kinds = datastore.run(query).map do |entity|
  entity.key.name
end

Property queries return entities of kind __property__ denoting the indexed properties associated with an entity kind. (Unindexed properties are not included.)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("__property__").
  select("__key__")

entities = datastore.run(query)
properties_by_kind = entities.each_with_object({}) do |entity, memo|
  kind = entity.key.parent.name
  prop = entity.key.name
  memo[kind] ||= []
  memo[kind] << prop
end

Property queries support ancestor filtering on a __kind__ or __property__ key, to limit the query results to a single kind or property. The property_representation property in the entity representing property p of kind k is an array containing all representations of p's value in any entity of kind k.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

ancestor_key = datastore.key "__kind__", "Task"
query = datastore.query("__property__").
  ancestor(ancestor_key)

entities = datastore.run(query)
representations = entities.each_with_object({}) do |entity, memo|
  property_name = entity.key.name
  property_types = entity["property_representation"]
  memo[property_name] = property_types
end

Property queries can also be filtered with a range over the pseudo-property __key__, where the keys denote either __kind__ or __property__ entities.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

start_key = datastore.key "__property__", "priority"
start_key.parent = datastore.key "__kind__", "Task"
query = datastore.query("__property__").
  select("__key__").
  where("__key__", ">=", start_key)

entities = datastore.run(query)
properties_by_kind = entities.each_with_object({}) do |entity, memo|
  kind = entity.key.parent.name
  prop = entity.key.name
  memo[kind] ||= []
  memo[kind] << prop
end

Configuring timeout

You can configure the request timeout value in seconds.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new timeout: 120

Additional information

Google Cloud Datastore can be configured to use an emulator or to enable gRPC's logging. To learn more, see the Emulator guide and Logging guide.