How to avoid datastore_v3.Next calls


My datastore processing takes lot of times. So, I checked what process takes time with StackDriver -> Trace. Then, "datastore_v3.Next" is called many times.

I found following document. https://cloud.google.com/appengine/docs/standard/python/tools/appstats

Datastore queries usually involve a datastore_v3.RunQuery followed by zero or more datastore_v3.Next calls. (RunQuery returns the first few results, so the API only uses Next when fetching many results. Avoiding unnecessary Next calls may speed up your app!)

But, I don't understand

  • In what cases will datastore_v3.Next be called many time?
  • How to avoid Next calls?

Add: MyCode is below

    @classmethod
    def get_foo(cls, user_key, foo):
      search_key = ndb.Key('UserInformation', user_key.id(), 'Foo', foo)
      return search_key.get()

UserInformation and Foo kind have 1 million entity.



The primary way is the one mentioned in the doc, simply don't return an unnecessarily large result set.

If you only care about the first few results in the query, you can look into using fetch which can return a limited set of results instead of the whole thing.

In addition, you can also make good use of memcache on NDB, less datastore calls means less .Next calls overall.