google app engine - Is there a limit on the number of entities you can query from the GAE datastore? -

my gcm endpoint derived code @ /github.com/googlecloudplatform/gradle-appengine-templates/tree/master/gcmendpoints/root/src/main. each android client device registers endpoint. message can sent first 10 registered devices using code:

@api(name = "messaging", version = "v1", namespace = @apinamespace(ownerdomain = "${endpointownerdomain}", ownername = "${endpointownerdomain}", packagepath="${endpointpackagepath}")) public class messagingendpoint { private static final logger log = logger.getlogger(messagingendpoint.class.getname());  /** api keys can obtained google cloud console */ private static final string api_key = system.getproperty("gcm.api.key");  /**  * send first 10 devices (you can modify send number of devices or specific device)  *  * @param message message send  */ public void sendmessage(@named("message") string message) throws ioexception {     if(message == null || message.trim().length() == 0) {         log.warning("not sending message because empty");         return;     }     // crop longer messages     if (message.length() > 1000) {         message = message.substring(0, 1000) + "[...]";     }     sender sender = new sender(api_key);     message msg = new message.builder().adddata("message", message).build();     list<registrationrecord> records = ofy().load().type(registrationrecord.class).limit(10).list();     for(registrationrecord record : records) {         result result = sender.send(msg, record.getregid(), 5);         if (result.getmessageid() != null) {             log.info("message sent " + record.getregid());             string canonicalregid = result.getcanonicalregistrationid();             if (canonicalregid != null) {                 // if regid changed, have update datastore                 log.info("registration id changed " + record.getregid() + " updating " + canonicalregid);                 record.setregid(canonicalregid);                 ofy().save().entity(record).now();             }         } else {             string error = result.geterrorcodename();             if (error.equals(constants.error_not_registered)) {                 log.warning("registration id " + record.getregid() + " no longer registered gcm, removing datastore");                 // if device no longer registered gcm, remove datastore                 ofy().delete().entity(record).now();             }             else {                 log.warning("error when sending message : " + error);             }         }     } } 

}

the above code sends first 10 registered devices. send registered clients. according http://objectify-appengine.googlecode.com/svn/branches/allow-parent-filtering/javadoc/com/googlecode/objectify/cmd/query.html#limit(int) setting limit(0) accomplishes this. i'm not convinced there not problem large numbers of registered clients due memory constraints or time takes execute query. https://code.google.com/p/objectify-appengine/source/browse/queries.wiki?repo=wiki states "cursors let take "checkpoint" in query result set, store checkpoint elsewhere, , resume left off later. used in combination task queue api iterate through large datasets cannot processed in 60s limit of single request".

note comment 60s limit of single request.

so question - if modified sample code @ /github.com/googlecloudplatform/gradle-appengine-templates/tree/master/gcmendpoints/root/src/main request objects datastore, replacing limit(10) limit(0), ever fail large number of objects? , if fail, number of objects?

this poor pattern, cursors. @ least, you'll hit hard 60s limit single request. , since you're doing updates on registrationrecord, need transaction, slow down process more.

this task queue for. best way in 2 tasks:

1. your api endpoint enqueues "send message everyone" , returns immediately.
2. that first task "mapper" iterates registrationrecords keys-only query. each key, enqueue "reducer" task "send x message record".
3. the reducer task sends message , (in transaction) performs record update.

using deferred isn't code @ all.

the first task frees client , gives 10m iterating registrationrecord keys rather 60s limit normal request. if have chunking right , batch queue submissions, should able generate thousands of reducer tasks per second.

this effortlessly scale hundreds of thousands of users, , might millions. if need scale higher, can apply map/reduce approach parallelize mapping. it's question of how many instances want throw @ problem.

i have used approach great effect in past sending out millions of apple push notifications @ time. task queue friend, use heavily.