AppEngine Tuning – Schlemiel, you’re fired!

Hop it, Schlemiel

This is the concluding post in the series that begins with The Amazing Story of AppEngine and the Two Orders Of Magnitude.

You’ll recall in my initial post that I detected a, well, somewhat suboptimal algorithm, where I was touching the AppEngine datastore on the order of 10^8 times per day? Liz Fong made the comment that “Schlemiel the Painter algorithms are bad”. What? Who’s Schlemiel the painter?

Well, it turns out he’s a hard worker, but not a smart one. The reference comes from a classic Joel on Software post:

Shlemiel gets a job as a street painter, painting the dotted lines down the middle of the road. On the first day he takes a can of paint out to the road and finishes 300 yards of the road. “That’s pretty good!” says his boss, “you’re a fast worker!” and pays him a kopeck.

The next day Shlemiel only gets 150 yards done. “Well, that’s not nearly as good as yesterday, but you’re still a fast worker. 150 yards is respectable,” and pays him a kopeck.

The next day Shlemiel paints 30 yards of the road. “Only 30!” shouts his boss. “That’s unacceptable! On the first day you did ten times that much work! What’s going on?”

“I can’t help it,” says Shlemiel. “Every day I get farther and farther away from the paint can!”

Is it Schlemiel or Shlemiel? Whichever it is, we need to fire them both.

Now, you’ll recall that this was all academic. AppEngine currently has no way to detect excessive datastore reads, apart from the billing info. So, I made changes to the code to give Schlemiel the flick, then we waited.

But wait no longer! Here’s the hard data.

The Hard Data

Usage Report for 2011-09-08

Resource Used Free Billable Charge
CPU Time:
$0.10/CPU hour
12.83 6.50 6.33 $0.64
Bandwidth Out:
0.15 1.00 0.00 $0.00
Bandwidth In:
1.32 1.00 0.32 $0.04
Stored Data:
0.48 1.00 0.00 $0.00
Recipients Emailed:
$0.10/1000 Emails
0.00 2.00 0.00 $0.00
Backend Usage:
$0.00 $0.72 $0.00 $0.00
Always On:
No $0.00
Total: $0.68

Estimated Charges Under New Pricing

The charges below are estimates of what you would be paying once App Engine’s new pricing model goes live. The amounts shown below are for your information only, they are not being charged and therefore do not affect your balance.

If you would like to optmize your application to reduce your costs in the future, make sure to read our Optimization Article. If you have any additional questions or concerns, please contact us at:

Frontend Instance Hour costs reflect a 50% price reduction active until November 20th, 2011.
Resource Used Free Billable Charge
Frontend Instance Hours:
41.31 24.00 17.31 $0.70
Backend Instance Hours:
0.00 9.00 0.00 $0.00
Datastore Storage:
0.48 1.00 0.00 $0.00
Blobstore Storage:
0.00 5.00 0.00 $0.00
Datastore Writes:
$1.00/Million Ops
0.40 0.05 0.35 $0.35
Datastore Reads:
$0.70/Million Ops
2.30 0.05 2.25 $1.58
Small Datastore Operations:
$0.10/Million Ops
0.04 0.05 0.00 $0.00
Bandwidth In:
1.32 1.00 0.32 $0.04
Bandwidth Out:
0.15 1.00 0.00 $0.00
$0.01/100 Messages
0.00 1.00 0.00 $0.00
XMPP Stanzas:
$0.01/1000 Stanzas
0.00 1.00 0.00 $0.00
Opened Channels:
$0.01/100 Opens
0.00 1.00 0.00 $0.00
Total*: (before clipping to daily budget) $2.67

* Note this total does not take into account the minimum per-application charge in the new pricing model.

Let me just take a moment to say


Oh yeah. It’s w00t, because this, although being a higher number, is one I can afford. I’m out of danger territory.

Anyway, let’s look at this in a little detail. Here’s what Schlemiel was projected to cost me:

Datastore Reads:
$0.70/Million Ops
59.06 0.05 59.01 $41.31

And here’s the post-Schlemiel picture:

Datastore Reads:
$0.70/Million Ops
2.30 0.05 2.25 $1.58

That, people, is a win.

There’s still a decent cost there, $1.58/day is considerable money for a self funded pet project. So where’s that coming from?

Recall I projected there would still be a good chunk of reads being performed by the fix:

1500 * 720 = 1,008,000 datastore reads per day

That’s in the ballpark, and probably accounts for around half of this. I can totally remove this processing with a careful change to my initialization code for the offending objects, and I haven’t done so already purely in the interests of science. So maybe it’s now time to do that.

It does look like there are still a fair chunk of reads happening elsewhere, possibly another bit of algorithm iterating a thousand or so records on every 2 minute cron job run. Odds are that’s also unnecessary. I’ll worry about that later though; I’ll remove the previously mentioned processing first, see how that works out, then take it from there.

But what about Instances, I hear you cry?

Sweet, Sweet Instances

Previously on Point7: <law and order-esque Da Dum!>

If I can get the average instances to 3, I’ll be paying 4 cents X (3-1) X 24 = US$1.92/day . That’s still a chunk more than I’m currently paying, but it’s doable (as long as everything else stays lowish).  Cautious optimism!

But what did the data say?

Frontend Instance Hours:
41.31 24.00 17.31 $0.70

Wow, that’s much better! Compare that to my old CPU bill:

CPU Time:
$0.10/CPU hour
12.83 6.50 6.33 $0.64

What did I do? I moved a couple of sliders, and spread my tasks out with a couple of lines of code. That’s pretty tough stuff. No wonder people are bitching & moaning!

I’ve got a few days of billing data now, and it’s all similar, so I’d say it’s a solid result. I actually can’t explain why this is so cheap, it doesn’t appear to match the instance data. Here’s the instance graph:

AppEngine Instances for Syyncc, 7 Days, 10 September 2011

Whatever it is, I’m happy with it.

So I’ve Got That Going For Me Which Is Nice

Not only are the instances right down to super cheap territory, I’m pretty sure I can get the datastore reads down into free territory with just a little more work. It’s all coming up Millhouse.

Just as a fun aside, this post took a little longer than I thought, due to the billing taking a while to come through. Why’d that happen? Because my credit card was maxed out (yeah, that’ll happen!), the weekly payment for AppEngine couldn’t be processed, and so my billing info was frozen until I fixed it. It turns out they give you no more billing info, and don’t let you change your billing settings, until you pay for the thing. Oh the humanity! Well, at least your app doesn’t get turned off, that’s great. Could be a gotcha if you have your billing settings cranked up though!

I’d love an ability to throw a bunch of money in ahead of time, ie: be able to have a positive account that the weekly billing draws from (my credit card is notoriously unreliable). I guess I could implement that by getting a debit card purely for the app. But then I have to think about money and stuff, and that’s boring and awful 😉

What’s it all mean for AppEngine developers?

Straightforwardly, it means this: optimisation isn’t difficult, and you should quit whining and do it. You might learn something.

I know there are some people who are optimised and can’t fix this. Their apps are predicated on using more instances than is financially viable. That’s hard luck.

On the other hand, the vast majority of apps will just be running fat & happy, and can be fixed with a bit of attention. Just do it, it’s not that tough. In fact, if you want a hand, yell out, I’m happy to look at stuff (as long as you’re ok for me to talk about it publicly, anonymity can be preserved to protect the guilty).

I’m a huge fan of not prematurely optimising, absolutely. But premature wont last forever. Here’s my new rule of thumb for AppEngine development:

You get to be willfully stupid or cheap but not both

Bullheaded idiocy is reserved for the rich. The rest of us need to use our brains just a little bit. And then it can be super cheap. So we little guys get a platform which can have longevity, and we get our way paid by wealthy and/or dumb people. I’m good with that.

Where to next?

This comment just in on the original “Amazing Story” post:

My app has the exact same scenario as yours – I initiate ~250 URL Fetches every 15 minutes. If I allow multiple front-end instances to get created, all these fetches will occur in parallel, but I’d be paying a lot of money for all these mostly idle instances. My optimization centers around using one single backend that processes these URL fetches one at a time (totally acceptable for my app), by leasing them one at time from a pull queue. The pull queue is populated by random events in a front-end instances, and the back-end is triggered by a cron every 15 minutes to lease these URL Fetch tasks from the pull-queue. This way all my work gets done by a single backend that runs 24×7. I could easily change my cron to run once every hour instead of 15 minutes, and then my backend is running for free (just under 9 instance hours a day).

Another level of optimization is kicking off multiple URL Fetches using create_rpc() calls, so that my backend can do other things while the URL fetch completes (which, like in your case, can take several seconds to complete or timeout). With all this, I hope to stay under the free instance hour quota.

Some people, unlike me, can just say something awesome in a couple of paragraphs without going all tl;dr on it.

Firstly, Rishi has done the backend style optimisation which seemed like the way to go. And what’s that you’re saying, it could run for free? Now you’ve got my attention. That’s worth some thought.

Secondly, what’s this create_rpc() call of which Rishi speaks? Oh, it must be this:

I really should RTFM, OMFG!

“Asynchronous Requests: A Python app can make an asynchronous request to the URL Fetch service to fetch a URL in the background, while the application code does other things.”

The doco says that you create an rpc object using create_rpc(), then execute it with make_fetch_call(), then finally wait for it to complete with wait() or get_result(). You can have it call a callback function on completion, too (although it requires the call to wait() or get_result() to invoke it).

My code is full of long fetches to external urls. It might be possible to do other things while they execute, with a code restructure. I smell some coding and another blog post in my future.

AppEngine Tuning – Schlemiel, you’re fired!

21 thoughts on “AppEngine Tuning – Schlemiel, you’re fired!

  1. Emlyn, There’s another important optimization that you can do. Take a look here:

    If you specify X-AppEngine-FailFast for taskqueue tasks those tasks won’t spin up new instances. If there’s no free instance they will immediately fail and will be retried later. This will result in slow process, but in more efficient resource usage consumption. Unfortunately the documentation doesn’t make clear that this works for push queues too. And you’ll also see 500 errors in your logs without any messages for these tasks. Try it out, this should help.

  2. Emlyn says:

    That’s really interesting, a hugely useful technique. Now in my case, it might not be ok, because something bad might happen if two tasks for the same monitor happen concurrently. A fail-retry approach might cause something like that. So due to my poor coding, that is probably not advisable for me. Although, on the other hand, if I make my bucket size 1 then there’s no concurrency to be had. That’s starting to get perverse though.

    I’d also be a bit worried about starvation in that scenario.

    But, for people for whom tasks were truly independent and not time critical, that’s a great idea.

  3. deferred doesn’t solve the instance hour issue – it’s just a (rather nice, if I say so myself) interface on top of the task queue. Using asynchronous calls is still a good way to reduce instance time.

      1. I did indeed write deferred. And yes, your instance is active/serving for the period between when the request arrives and when you finish processing, regardless of what you do during that period.

        1. Emlyn says:

          I’m implementing some commercial code right now using deferred – taking a break to comment here and grab a cuppa while I see if a recurring schedule works correctly. Fine work, thanks very much.

            1. Emlyn says:

              A penny has just dropped I think. Here’s my situation:

              I’m defering a call to a method on a datastore object (ie: a persistent object). That method does something, then re-defers itself.

              Now, I’ve also got an “enabled” property on that object. The method described above first checks enabled, before it does something and then re-defers itself.

              So the idea is, I can go change enabled to false, put() the object, then when the method runs again, it’ll see enabled is false and instead do nothing (and not reschedule itself).

              Except, I think you are ahead of me by now, enabled is always true.

              … because this is not the object loaded from the datastore, it is a deserialised pickled stale object.

              And the fix would be to reload the object and work with that, not on the unpickled self.

      2. (Replying here to your nested response)

        You’re correct – deferred is serializing the datastore object as it exists when you called defer, so later changes are not reflected.

        Instead, you should pass the key of the object to a function or class method, which fetches the entity before working on it.

          1. You definitely shouldn’t do that – you’re serializing the whole entity, then deserializing it and overwriting it. Instead, use a class method, send it the key, and do “self = db.get(key)” if you want to minimize the number of changes you have to make.

    1. Emlyn says:

      Currently somewhere between 500 and 1000 users – my instrumenting is almost non-existent 😦 .

      It’ll support far more than that when I get the G+ api; that’ll allow me to make it event driven. When I made the transition from polling to events with FB and Buzz, the engine quieted way down; it jumped up again when I added polling G+. Polling is expensive!

  4. lyxera says:

    you are have a million datastore reads per day, which is costing a lot of money. have you thought about making use of memcached so as to reduce your datastore reads to a minimum?

    I don’t understand the nature of your application, but if you do read the same record for multiple times a day, memcached can make a significant difference.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s