Being truly asynchronous with Tornado

In the past few days I’ve had the chance to play a bit with Tornado, a non-blocking web server for Python similar to node.js. Tornado works by using something like epoll or whatever I/O event notification exists in the system. Tornado is single threaded and works like a fancy while (True) loop. As a consequence, whenever we do some blocking I/O (e.g. database call) the web server cannot process other requests. While that may not be a big deal on very low traffic sites or if your I/O subsystem is extremely fast, it starts becoming an issue as your traffic grows or if you’re hosted on something like Amazon AWS; notorious for its slow and inconsistent disk I/O performance.  The biggest problems that I encountered with Tornado are: 1) the lack of robust asynchronous libraries for things like DB access and 2) Python’s own awkwardness when dealing with async code. NodeJS programmers don’t have any of these problems because the community writes software with async in mind and Javascript as a language offers anonymous blocks which make callback style code tolerable. The purpose of this post is to encourage Python programmers using Tornado to pay attention to their blocking calls or if they don’t need any of the features that Tornado offers, stick to a WSGI server such as uwsgi or gunicorn.

We’re going to start by writing a pretend service that takes half a second to return a response. Half a second is not a lot of time and doing things like calling the Facebook API for example can take much longer for that sometimes.

class PretendService(tornado.web.RequestHandler):
  def get(self):
     """ Pretend some work is being done by sleeping for 500ms """
     ioloop = tornado.ioloop.IOLoop.instance()
     ioloop.add_timeout(time.time() + 0.5, self._finish_req)

  def _finish_req(self):

Now we’re going to write two handlers, one of them will call the service in the way Python programmers are used to writing code. The other will make the same call the proper way that Tornado expects, using an asynchronous http client that supports callbacks. First the normal synchronous

class MainHandlerBlocking(tornado.web.RequestHandler):
  def get(self):
      req = httpclient.HTTPRequest(pretend_service_url, method='GET')
      # we could use something like requests or urllib here
      client = tornado.httpclient.HTTPClient()
      response = client.fetch(req)
      # do something with the response

And now the Tornado way

class MainHandlerAsync(tornado.web.RequestHandler):
  def get(self):
     req = httpclient.HTTPRequest(pretend_service_url, method='GET')
     client = tornado.httpclient.AsyncHTTPClient()
     # don't let the yield call confuse you, it's just Tornado helpers to make
     # writing async code a bit easier. This is the same as doing
     # client.fetch(req, callback=_some_other_helper_function)
     response = yield gen.Task(client.fetch, req)
     ### do something with the response ###

and finally some plumbing code:

application = tornado.web.Application([
 (r"/async", MainHandlerAsync),
 (r"/external-api", PretendService),
 (r"/blocking", MainHandlerBlocking)

if __name__ == "__main__":
 define("port", default=8888, help="run on the given port", type=int)
 http_server = tornado.httpserver.HTTPServer(application)

Now that we have everything setup, lets fire up some benchmarks. First lets start with 5 concurrent users and try a total of 100 requests.

> ab -n 100 -c 5
Requests per second:    1.96 [#/sec] (mean)
Time per request:       2549.939 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95%   2552ms

> ab -n 100 -c 5
Requests per second:    9.75 [#/sec] (mean)
Time per request:       513.067 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95%   521ms

As you can see from the results above, code written in a blocking fashion can only handle 2 requests per second and under a moderate load of 5 concurrent users takes 2.5 seconds to serve 95% of the requests. Code written async lets Tornado go on and handle more requests while we are blocked on that network call, resulting in 10 reqs/sec which 95% of the requests taking 521ms, 500 of which is due to the pretend service we wrote. But lets see how far we can take it, lets assume our service got popular and we now have 10 users at the same time.

> ab -n 500 -c 10
Requests per second:    1.96 [#/sec] (mean)
Time per request:       5099.994 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95%   5102ms

> ab -n 500 -c 10
Requests per second:    19.06 [#/sec] (mean)
Time per request:       524.732 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95%   534ms

As we expected, our blocking code is still only able to handle 2 requests per second and now 95% of our users have to wait 5 seconds for a response. Our async code however lets Tornado scale and show its true power by handling 19 requests per sec with no noticeable increase in the time it takes to serve a request. Lets raise the bar one more time:

> ab -n 1000 -c 20
Requests per second:    1.96 [#/sec] (mean)
Time per request:       10200.390 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95%   10203ms

> ab -n 1000 -c 20
Requests per second:    19.44 [#/sec] (mean)
Time per request:       1028.586 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95%   1044ms

We are beating a dead horse here but obviously not only the async code scales better but Tornado is not kind to any sort of blocking on I/O. In that last test, we would need 10 Tornado instances to match the performance of a single async one.

Note that while I throw the word async around a lot, not any asynchronous library would work. For example, you can’t use grequests (gevent + requests), use AsyncHttpClient instead. That is because Tornado makes certain assumptions and uses a callback style API. In most cases, libraries would need to be rewritten specifically for Tornado but luckily some already have.

Finally, I recommend that if you are set on using Python and don’t need to build a real-time app stick with a WSGI server, mostly because of the lack of async libraries for Tornado and that callback-style code is not something that Python shines in. If you do plan on using Tornado, hopefully I’ve convinced you on the importance of not doing any blocking I/O.


8 thoughts on “Being truly asynchronous with Tornado

  1. yavorgeorgiev

    Is the yield/task model built into python or does Tornado compile that down into some callback soup?

  2. papercruncher Post author

    Yield as a keyword is built in, but the whole yield Task() model is actually a really clever hack in Tornado that handles the callback soup for you.

  3. breaking news

    Attractive part of content. I simply stumbled upon your weblog and in accession capital to say that I acquire in fact loved account your blog posts. Anyway I will be subscribing to your augment and even I success you get admission to persistently quickly.

  4. Orion

    Hey, Could you please post the full code example online. I tried to recreate your results here:

    but I keep getting timeout issues.

  5. Phekuchand

    I don’t want the server part. Can I file like 100s of requests using just AsyncHttpClient, and scan for responses one by one after that ? To replicate the way Perl allows to do ?

  6. cypeSpeandy

    With only 1 week left until Father¡¯s Day time, I¡¯m sure a lot of you are still scrambling to look for the perfect gift to your dear old Dad. How about a very snazzy watch? Citizen sent me certainly one of their latest Eco-Drive designer watches – the CA0467-11H Eco-Drive Primo Chronograph Watch to be exact to test. With a racing motivated design, this is a cool watch that can keep time and flip heads. Let¡¯s take a closer look. And of course the item offers the ultra-convenient (and eco-friendly) charging design and that is powered by light – virtually any light. It never requires a battery change but will charge in the sunrays, in the house, anywhere where there¡¯s background light. When fully charged, it will run for up to 6 months. Light is absorbed with the crystal and the dial the location where the solar cell converts your light to energy.
    [url=][/url] 【正規品】オリエント 時計 オリエント ユー ORIENT YOU ソーラー 腕時計 レディース WY0011WG【オリエント ユー 2013 新作】
    [url=][/url] [CITIZEN]シチズン 時計 プロマスター [PROMASTER] アルティクロン BN4026-09F メンズ/腕時計 #107692 ■12月下旬発売予定 予約商品
    Although the watch doesn¡¯t have a straightforward to see charging status indicator, when the reserve electrical power becomes too low, the 2nd hand will become moving in 2 second increments rather then 1 second increments. This is an alert to tell you it¡¯s time to reveal to the Primo to lead light. Placing the watch within a window sill on a new sunny day for 5-6 hours will fully charge it. I wanted to get started with a little Gadgeteer trivia¡­ Fourteen prohibited, back in 1999, we posted a writeup on then brand new Resident Eco-Drive watch. That review continues for making our yearly top 25 most read reviews despite all these years. That goes to indicate that Citizen Watch Organization and Eco-Drive watches are just as popular as ever. It¡¯s easy to see why these watches are and so popular. The new Primo watch is known for a sporty style that shows a chunky stainless steel grey ion-plated case including a clear mineral crystal. It has a 1/5 second chronograph measuring approximately 60 minutes, tachymeter, rotating 360 degree bezel it is water resistant to one hundred meters.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s