• Hulu
  • TV
  • Movies
Hulu Tech Blog

Hostess Menu System

January 3rd, 2012 by Scott Post

When people ask me what it is I do at Hulu, I usually have a few answers prepared ranging from, “I work on computers” (this is surprisingly satisfactory sometimes) to specific details on day-to-day tech projects.  For this post, I’d like to crack open the hood and dive deep into one challenge I’ve helped solve here at Hulu:  A project aptly named Hostess.

No More Control

In late 2008, our Hulu Desktop application was in full-on development.  It became clear that this app was to be the first of many out-of-browser Hulu experiences. (Today, of course, we have a substantial and growing device line-up.)  The fundamental problem we foresaw with these applications was control: we’d lose the luxury of swift deployments, patches, and bug fixes and instead have to deal with third-party quality assurance approval processes and the probable latency associated with them.  This meant each new build deployed could potentially be delayed for weeks before going live. If we weren’t careful, this would include trivial user interface changes, additional content views, or—more specifically—implementation details such as content organization.

Our Solution

The fix was a relatively simple one: let the server determine as much of the experience as possible. Rather than managing this all on the client, our solution was to have the server maintain all of the views, filters, and even UI layout specifications in a database.  This enables the structure and content of our applications to be data driven and decoupled from our deployed bits.  Because the navigation capabilities of devices can be limited (sometimes remote control), the structure of an application is composed of a familiar set of simple Menu objects.  At the root of a tree structure sits the main menu, and each submenu its child.

In addition to Menu objects, the server also maintains associations with Query objects.  These objects basically represent SQL fragments—which are stored in the database themselves (very meta, I know).  Simple menus don’t need to have a Query associated with them. Perhaps only metadata such as the menu’s textual name and ranking relative to its siblings is enough to navigate (look at Now Playing, Popular, Recently Added, or Help, for example). But for menus that represent actual Hulu content—like a most popular shows listing—there is a Query object on the server that represents the command that will ultimately get executed on the database.  This fragmentation scheme allows for cascading of queries and provides a mechanism for children to inherit parent filters.  In addition, it allows us to have a centralized service to manage menus for all of our devices, removes redundancy, and cuts down on the amount of SQL we have to write (and any developer can vouch for SQL as sometimes being cumbersome to deal with in code. Formatting strings anyone?).


Example Menu

 It’s not obvious, but the above screenshot actually represents three levels of the Hulu TV application’s menu hierarchy.  Beginning at the top level, notice the menu called Most Popular.  This menu has siblings (which are not shown) and children: Shows, Episodes, Movies, etc.  Looking at the Shows menu, it has a single child of type ShowThumbnailCarousel which the client knows how to render.  Here is the same view represented in the actual database:

mysql> select * from menus where parent_menu_id = 185332;
| child_menu_id | parent_menu_id | display         | query_id | app_data                             |
|        185333 |         185332 | Search          |       -1 | <view_name>SearchView</view_name>    |
|        185334 |         185332 | Browse Movies   |    24856 | <view_name>BrowseView</view_name>    |
|        185371 |         185332 | Browse TV       |    24881 | <view_name>BrowseView</view_name>    |
|        185383 |         185332 | Recently Added  |       -1 | <view_name>ContentView</view_name>   |
|        185389 |         185332 | Most Popular    |       -1 | <view_name>ContentView</view_name>   |
|        185395 |         185332 | Queue / Profile |       -1 | <view_name>ContentView</view_name>   |
|        185404 |         185332 | Recommended     |       -1 | <view_name>ContentView</view_name>   |
|        185407 |         185332 | Help            |       -1 | <view_name>HelpView</view_name>      |

mysql> select * from menus where parent_menu_id = 185389; +---------------+----------------+----------+----------+---------------------------------------------+ | child_menu_id | parent_menu_id | display | query_id | app_data | +---------------+----------------+----------+----------+---------------------------------------------+ | 185390 | 185389 | Shows | -1 | <cmtype>ShowThumbnailCarousel</cmtype> | | 185391 | 185389 | Episodes | -1 | <cmtype>VideoThumbnailCarousel</cmtype> | | 185392 | 185389 | Movies | -1 | <cmtype>ShowThumbnailCarousel</cmtype> | | 185393 | 185389 | Clips | -1 | <cmtype>VideoThumbnailCarousel</cmtype> | | 185394 | 185389 | Trailers | -1 | <cmtype>VideoThumbnailCarousel</cmtype> | +---------------+----------------+----------+----------+---------------------------------------------+

mysql> select * from menus where parent_menu_id = 185390; +---------------+----------------+---------+----------+----------+ | child_menu_id | parent_menu_id | display | query_id | app_data | +---------------+----------------+---------+----------+----------+ | 185337 | 185390 | {name} | 24895 | NULL | +---------------+----------------+---------+----------+----------+

mysql> select * from queries where id = 24895; +-------+--------+--------+--------+-------------------------+-----------------------+ | id | object | params | fields | filter | order_by | +-------+--------+--------+--------+-------------------------------------------------+ | 24895 | Show | NULL | NULL | full_episodes_count > 0 | view_count_today DESC | +-------+--------+--------+--------+-------------------------+-----------------------+

The Menu object responsible for the show carousel is represented by the last two rows above.  It has an associated Query object, so the items of this menu aren’t just further text submenus, but rather are objects of type Show with a particular filter and sorting.  Upon request of this menu, the server looks up the menu data, pieces together a SQL query looking something like “SELECT * FROM shows WHERE full_episodes_count > 0 ORDER BY view_count_today DESC” and finally formats a response containing these items. If we continue on down the tree, each item in this carousel points to yet another submenu, and so on. Below is the child menu you’ll see when you select Parks and Recreation:

And has corresponding data:

mysql> select * from menus where parent_menu_id = 185337;
| child_menu_id | parent_menu_id | display      | query_id | app_data                                |
|        185339 |         185337 | Episodes     |       -1 | <cmtype>VideoThumbnailCarousel</cmtype> |
|        185342 |         185337 | Clips        |       -1 | <cmtype>VideoThumbnailCarousel</cmtype> |
|        185343 |         185337 | Seasons      |       -1 | <cmtype>NonSlidingMenu</cmtype>         |
|        185345 |         185337 | Rate         |       -1 | <cmtype>RatingControl</cmtype>          |
|        185346 |         185337 | Subscribe    |       -1 | <cmtype>SubscriptionControl</cmtype>    |
|        185347 |         185337 | Related      |       -1 | <cmtype>ShowThumbnailCarousel</cmtype>  |
|        185348 |         185337 | Home         |       -1 | <cmtype>TextControl</cmtype>            |

mysql> select * from menus where parent_menu_id = 185343; +---------------+----------------+------------------------+----------+-----------------------------------------+ | child_menu_id | parent_menu_id | display | query_id | app_data | +---------------+----------------+------------------------+----------+-----------------------------------------+ | 185344 | 185343 | Season {season_number} | 24862 | <cmtype>VideoThumbnailCarousel</cmtype> | +---------------+----------------+------------------------+----------+-----------------------------------------+ mysql> select * from queries where id = 24862; +--------+--------+---------+-----------------------+------------------+---------------+-----------------------+ | id | object | params | fields | filter | order_by | group_by | +--------+--------+---------+-----------------------+------------------+---------------+-----------------------+ | 24862 | Video | show_id | season_number,show_id | show_id=:show_id | season_number | season_number,show_id | +--------+--------+---------+-----------------------+------------------+---------------+-----------------------+

mysql> select * from menus where parent_menu_id = 185344; +---------------+----------------+---------+----------+----------+ | child_menu_id | parent_menu_id | display | query_id | app_data | +---------------+----------------+---------+----------+----------+ | 185340 | 185344 | {title} | 24863 | NULL | +---------------+----------------+---------+----------+----------+ mysql> select * from queries where id = 24863; +--------+--------+-----------------------+---------------------------------------------------+----------------+ | id | object | params | filter | order_by | +--------+--------+-----------------------+---------------------------------------------------+----------------+ | 24863 | Video | show_id,season_number | show_id=:show_id AND season_number=:season_number | episode_number | +--------+--------+-----------------------+---------------------------------------------------+----------------+

The above represents another three levels deeper into the menu hierarchy (the previous view can be considered off screen).  Notice the server requires a show_id parameter to properly construct a query for these children menus. In this case, Parks and Recreation was selected and show_id is passed down.  The client accesses these children via URLs which are returned in each response from the server.  So for the above view, the Hulu TV client had to make a few calls to the server:  one to retrieve the menu for Parks and Recreation containing metadata and links to its children, another to follow the link to the Seasons menu, and finally a call to get the menu with episodes for the second season. Now, let’s say we discovered it would be better to sort the Seasons menu by current seasons first and shorten the display text to S2, S1.  We could simply make an update in our CMS so display is ‘S{season_number}’ and order_by is ‘season_number DESC’ and we are done for any device that reads this menu.

Components and Technology Stack

The system has three main components:  the data sync, the web services, and the CMS.  All components were written in Python and built using the CherryPy web framework and the Sqlalchemy ORM module.  The CMS component has a front end UI written in Javascript with jQuery and uses jinja HTML5 templates.  All requests to the Hostess web services are proxied through a caching layer using Varnish.  There is also another Memcached layer to store Menu objects in shared memory since they’re not changed frequently.

Above is a diagram of the entire system and interaction between components.  The data sync component simply syncs metadata from Hulu’s master site database into its own schema.  This is mainly to keep traffic to the site database and devices separate, but devices also have additional metadata to query and join against—and Hulu’s website does not.  The master runs the data sync and CMS.  On a handful of slaves, MySQL replication keeps data synchronized with the master.  Each of these slaves runs as many Hostess web service components as there are CPU cores on that machine. As we scale, we can simply add more slaves to the system.  The web services are load balanced and serve the majority of metadata (in XML or JSON) found on Hulu devices.  This includes living room devices like Xbox, PS3, and Roku and mobile devices like the iPad, iPhone, and Android phones.  Each device uses a particular set of menus (starting at different roots) which we call Hostess applications.  Managing these applications is done via the CMS component.

Hostess CMS

The CMS allows our developers to manage, create, edit, reorganize, and deploy new applications as often as we want, as was our ultimate goal. Applications are edited offline and then exported to either staging environments for testing or production for deployment.  When they’re exported, the application’s menus are inserted into the master database, and these trickle down to all the replicated slaves.  As cache expires, devices pick up the new root menus and their fresh child menus.  In the future we hope to expand the CMS to abstract away having to know SQL to manage apps.  This would enable any project manager to be able create and maintain basic menus for their apps.  A challenge we have (and will continue to face) is making sure tables are properly indexed and queries are optimal.

For me, it’s been fun working with smart problem-solvers here on the Hulu team to build and scale Hostess as our Hulu Plus device offerings have grown.  If you’re still curious why we chose the name Hostess: the job of a hostess  is to serve menus as you prepare to consume something really delicious. We thought this name was a good fit.  It has ultimately solved our deployment woes, enabling us to keep our applications flexible, reusable, and agile as we continue to build on this framework.  And in the process, it has grown into something more than originally intended—but that’s a topic for another post.