Migrating from Python 2 version of NDB¶
While every attempt has been made to keep compatibility with the previous version of ndb, there are fundamental differences at the platform level, which have made necessary in some cases to depart from the original implementation, and sometimes even to remove existing functionality altogether.
One of the main objectives of this rewrite was to enable ndb for use in any Python environment, not just Google App Engine. As a result, many of the ndb APIs that relied on GAE environment and runtime variables, resources, and legacy APIs have been dropped.
Aside from this, there are many differences between the Datastore APIs provided by GAE and those provided by the newer Google Cloud Platform. These differences have required some code and API changes as well.
Finally, in many cases, new features of Python 3 have eliminated the need for some code, particularly from the old utils module.
If you are migrating code, these changes can generate some confusion. This document will cover the most common migration issues.
Setting up a connection¶
The most important difference from the previous ndb version, is that the new ndb requires the use of a client to set up a runtime context for a project. This is necessary because ndb can now be used in any Python environment, so we can no longer assume it’s running in the context of a GAE request.
The ndb client uses
google.auth for authentication, consistent with other
Google Cloud Platform client libraries. The client can take a credentials
parameter or get the credentials using the GOOGLE_APPLICATION_CREDENTIALS
environment variable, which is the recommended option. For more information
about authentication, consult the Cloud Storage Client Libraries documentation.
After instantiating a client, it’s necessary to establish a runtime context,
Client.context method. All interactions with the database must
be within the context obtained from this call:
from google.cloud import ndb client = ndb.Client() with client.context() as context: do_something_with_ndb()
The context is not thread safe, so for threaded applications, you need to generate one context per thread. This is particularly important for web applications, where the best practice would be to generate a context per request. However, please note that for cases where multiple threads are used for a single request, a new context should be generated for every thread that will use the ndb library.
The following code shows how to use the context in a threaded application:
import threading from google.cloud import datastore from google.cloud import ndb client = ndb.Client() class Test(ndb.Model): name = ndb.StringProperty() def insert(input_name): with client.context(): t = Test(name=input_name) t.put() thread1 = threading.Thread(target=insert, args=['John']) thread2 = threading.Thread(target=insert, args=['Bob']) thread1.start() thread2.start()
Note that the examples above are assuming the google credentials are set in the environment.
There are some methods from the
key module that are not implemented in
this version of ndb:
These methods were used to pass keys to and from the db Datastore API, which is no longer supported (db was ndb’s predecessor).
There are some methods from the
model module that are not implemented in
this version of ndb. This is because getting the indexes relied on GAE
There are various small changes in some of the model properties that might trip you up when migrating code. Here are some of them, for quick reference:
The BlobProperty constructor only sets _compressed if explicitly passed. The original set _compressed always.
In the exact same fashion the JsonProperty constructor only sets _json_type if explicitly passed.
Similarly, the DateTimeProperty constructor only sets _auto_now and _auto_now_add if explicitly passed.
TextProperty(indexed=True) and StringProperty(indexed=False) are no longer supported. That is, TextProperty can no longer be indexed, whereas StringProperty is always indexed.
The Property() constructor (and subclasses) originally accepted both unicode and str (the Python 2 versions) for name (and kind) but now only accept str.
QueryOptions and Query Order¶
The QueryOptions class from
google.cloud.ndb.query, has been reimplemented,
google.appengine.datastore.datastore_rpc.Configuration is no longer
available. It still uses the same signature, but does not support original
google.appengine.datastore.datastore_query.Order is no
longer available, the
ndb.query.PropertyOrder class has been created to
MessageProperty and EnumProperty¶
These properties, from the
ndb.msgprop module, depend on the Google
Protocol RPC Library, or protorpc, which is not an ndb dependency. For
this reason, they are not part of this version of ndb.
When writing a tasklet, it is no longer necessary to raise a Return exception for returning the result. A normal return can be used instead:
@ndb.tasklet def get_cart(): cart = yield CartItem.query().fetch_async() return cart
Note that “raise Return(cart)” can still be used, but it’s not recommended.
There are some methods from the
tasklet module that are not implemented in
this version of ndb, mainly because of changes in how an ndb context is
created and used in this version:
The previous version of ndb included an
ndb.utils module, which defined
a number of methods that were mostly used internally. Some of those have been
made obsolete by new Python 3 features, while others have been discarded due
to implementation differences in the new ndb.
Possibly the most used utility from this module outside of ndb code is the
positional decorator, which declares that only the first n arguments of
a function or method may be positional. Python 3 can do this using keyword-only
arguments. What used to be written as:
@utils.positional(2) def function1(arg1, arg2, arg3=None, arg4=None): pass
Should be written like this in Python 3:
def function1(arg1, arg2, *, arg3=None, arg4=None): pass
positional remains available and works in Python 3.
App Engine’s legacy exceptions are no longer available, but ndb provides shims for most of them, which can be imported from the ndb.exceptions package, like this:
from google.cloud.ndb.exceptions import BadRequestError, BadArgumentError
There are many differences between the current Datastore API and the legacy App Engine Datastore. In most cases, where the public API was generally used, this should not be a problem. However, if you relied in your code on the private Datastore API, the code that does this will probably need to be rewritten.
Specifically, the old NDB library included some undocumented APIs that dealt directly with Datastore protocol buffers. These APIs will no longer work. Rewrite any code that used the following classes, properties, or methods:
Property._db_set_compressed_meaning and Property._db_set_uncompressed_meaning.
Model._deserialize and Model._serialize.
In the previous version,
google.appengine.api.namespacemanager was used
to determine the default namespace when not passed in to constructors which
require it, like
Key. In this version, the client class can be instantiated
with a namespace, which will be used as the default whenever it’s not included
in the constructor or method arguments that expect a namespace:
from google.cloud import ndb client=ndb.Client(namespace="my namespace") with client.context() as context: key = ndb.Key("SomeKind", "SomeId")
In this example, the key will be created under the namespace my namespace, because that’s the namespace passed in when setting up the client.
The Django middleware that was part of the GAE version of ndb has been discontinued and is no longer available in current ndb. The middleware basically took care of setting the context, which can be accomplished on modern Django with a simple class middleware, similar to this:
from google.cloud import ndb class NDBMiddleware(object): def __init__(self, get_response): self.get_response = get_response self.client = ndb.Client() def __call__(self, request): context = self.client.context() request.ndb_context = context with context: response = self.get_response(request) return response
__init__ method is called only once, during server start, so it’s a
good place to create and store an ndb client. As mentioned above, the
recommended practice is to have one context per request, so the
method, which is called once per request, is an ideal place to create it.
After we have the context, we add it to the request, right before the response
is processed. The context will then be available in view and template code.
Finally, we use the
with statement to generate the response within our
Another way to get an ndb context into a request, would be to use a context processor, but those are functions called for every request, which means we would need to initialize the client and context on each request, or find another way to initialize and get the initial client.
Note that the above code, like other ndb code, assumes the presence of the GOOGLE_APPLICATION_CREDENTIALS environment variable when the client is created. See Django documentation for details on setting up the environment.