At Hulu, our development teams (and individual developers) have a lot of freedom in their choice of development tools. Our overridding principle is “make the best choice for your project”, which means that we expect you to evaluate tools and platforms that are already used within the company as well as those that are new to us. While sometimes the project’s needs will drive its development team to choose something new, there are a lot of people at Hulu who make the choice of Python.
This year, as part of our commitment to Python, we are helping to sponsor PyCon 2013. Look for a few Hulugans on the conference floor and at our booth too. If you are there, stop by our booth to chat about what kinds of things we do with Python.
We like Python for its speed of development and execution, its diverse libraries, and for being easy enough to read to easily let new developers get acquainted with a project. We use Python widely, for tasks big and small. The small might include scripts to help with deployment or monitoring, or wrappers around git or other tools. The large includes systems that are core to our application API. Some internal examples are below.
At the core of our devices is an application we call Deejay. When our desktop, living room, and mobile apps start up, they connect to Deejay to learn about the Hulu environment they’re connecting to. An iPhone in Japan needs to know to use a different information architecture than an iPhone in the US. The PS3 app will need a different set of icons than an Xbox. In addition to general configuration, the Deejay service is used to help in streaming video.
Python and the Tornado web server are behind the service that we at Hulu use to resize, crop, and otherwise manipulate images that get displayed by our site and by our device apps. We’d open sourced the core of this service a bit ago, and have used it extensively. The service provides a consistent HTTP front-end to some imagemagick capabilities, and allows an app to request an image of a particular size with a particular effect applied — whatever an app needs at the moment. Given our scale, this service benefits greatly from living behind a cache (whether local or CDN-based).
There comes a time in every company’s life when devops gets sick and tired of developers saying, “I’d like another machine for my service”. We’d gotten to this point some time ago, and built Sod. This platform is the foundation of our private cloud. It allows any of our developers to create a Xen-based virtual machine (running Centos, Ubuntu, or Windows), get it up on the correct part of our network, endow it with the desired amount of RAM and drive space, and launch it — within less than a minute. Whether using a web user interface, an API, or a command-line tool, Sod is our centralized interface to a cluster of Xen nodes. It abstracts the interaction with Xen and handles its peculiarities. It also helps our devops team to manage VM operations like cross-host moves. By integrating with our internal authenticaion system it also helps us keep track of machine and service ownership throughout our environment. We built Sod with CherryPy, gunicorn, gevent, and based its data store on MySQL.
There comes a further time in a company’s life when developers start spinning up VMs to host a one-off web service. The developers want the service to be fault-tolerant — so they spin up multiple instances and get them load-balanced. They want to know how well the service runs, so they build log collection systems. They want to have the service geo-distributed, so they create their own schemes for hosting this in multiple data centers.
Donki was built to make this much more simple. At its core it’s a service for hosting other services. Developers write any wsgi-compliant service, then
git push it to donki, which handles the rest. This includes deployment, pre-release smoke testing, setting up DNS and load balancing, handling data center distribution, log collection, and much more. Donki guarantees that at least 2 instances (or more, depending on load) are always up. Under the covers, Donki uses Sod and other internal devops services to orchestrate its hosting. In fact, the Sod service itself is hosted by Donki. We wrote Donki using Django for the front-end.
What started off as a Hack Day project for a couple of us has become an important part of many people’s lives at Hulu. After noticing that we pay significant money for a telco phone conferencing service, we hacked on a Django-based system to use the Twilio API to create a phone conferencing service. After a weekend it was a working prototype, and after a few more weeks we launched it to the company. By January of this year, we had 400+ users participating in 1,300+ calls per month — at a fraction of the cost our telco-based service cost us, and with a great deal of flexibility. Like many other such apps, Parley runs as a Donki-managed service.
There are many more Python projects at Hulu. From small to big, we depend on the language and its ecosystem to drive our business. So it’s with pride that we sponsor this year’s PyCon, and hope to continue to have a wonderful symbiotic relationship with the language, its future advances, and its great influence for years to come.