Toward HyperDrupal

Back in September I had the privilege of attending a web development conference in my new hometown, Greenville, SC. The occasion was REST Fest, which was primarily a barcamp format gathering of REST API developers and thought leaders from around the world.

I got to meet several local web developers along with encouraging and helpful out of towners like Mike Amundsen (author of Building Hypermedia APIs with HTML5 & Node, highly recommended) and Sam Ramji (erstwhile champion of open source at Microsoft, now with Apigee). I also got a second chance to meet David Zuelke, who I somehow missed at DrupalCon Munich where he presented on RESTful Web Services. (In addition to REST, he and I geeked out over the BMW Performance Track, which I enjoyed the day before thanks to demo day at The Iron Yard, our local startup accelerator and host to REST Fest and other tech events.)

The event was quite the eye opener, an opportunity for me to be out of my element with nary a Drupal developer in sight. However, the topics covered and examples shared were incredibly practical for me, as we were in the middle of researching and implementing RESTful Drupal Commerce at Commerce Guys for a client project and for a Commerce Kickstart mobile app developed with the help of Sumit Kataria.

Fast forward now three months and I'm finally circling back around to pump up the Commerce Services module to turn any Drupal Commerce site into a REST API server. My focus thus far has been on improving GET requests to support a variety of query parameters offering:

  • Response format specification (i.e. a subset of an entity's properties / fields when you don't need the entire entity)
  • Filtering by property names, field names, and multi-column field column names
  • Sorting by those same parameters
  • Paging through the result set via limit and offset parameters
  • The ability to include referenced entities in the response
  • The ability to flatten field value arrays to the current language and from arrays to scalars for single value, mono-column fields

As a best practices guide, I've been using an incredibly helpful eBook from Apigee called Web API Design: Creating Interfaces that Developers Love. The idea here is to follow standard patterns for all of our entity resources that allow API clients to easily browse the API.

Additionally, my goal is for Drupal Commerce API servers to actually be RESTful, not simply RESTish. If you aren't familiar, a great articule to bookmark and read when you need it is A Brief Introduction to REST by Stefan Tilkov. As we build out the Commerce Services module and deploy projects from client sites to mobile apps, we'll also publish developer documentation that defines and demonstrates best practices use of the system.

My end goal here is to keep pushing forward to a holy grail of sorts - HyperDrupal. By simply creating powerful resources that respect HTTP methods (and other headers) as defined, we're two thirds of the way there. The final step (as identified by the Richardson Maturity Model) is to format our responses using a "hypermedia type" - a media type that includes the hypermedia controls we're used to in HTML like links (a), forms (forms that POST), query templates (forms that GET), and the like. See Mike Amundsen's book (referenced above) for more detail.

The media type I'll be focusing on first is Mike's Collection+JSON media type. Even as I do my testing with Commerce Services, I'm using the beginnings of what I hope will become the Collection+JSON response formatter for Services based REST servers. The "big idea" here is that an API client like a mobile app could browse your API like a user would browse your website.

For example, the app would start at the "front page" of your API where it might find the link to your product list. When the customer chooses which product to view, in addition to the product information, the GET response includes a form with the necessary parameters to submit a POST request to add the product to the cart. If built to consume this information properly, the client never actually need to know how your URLs are constructed, whether or not you've recently changed API responses, or when you've added new features to the site. That information would all be communicated through API request responses.

This process has been awkwardly acronymized as "HATEOS" (pronounced had-ee-us) - Hypermedia as the Engine of Application State - and is further described in Haters Gonna HATEOS by Steve Klabnik, proprietor of the living eBook Designing Hypermedia APIs. It may sound a bit "pie in the sky", but the advantages of Drupal / Drupal Commerce offering this level of API functionality out of the box are too good not to pursue.

The good news is that Drupal 8 is well on its way toward beefing up our REST cred. Larry Garfield, Klaus Purer, Lin Clark and others have been hard at work laying the groundwork for solid REST in core through the "Web Services and Context in Core" D8 initiative. I don't believe hypermedia is on the menu for D8 release, but they were just granted a reprieve in the form of the D8 feature completion phase to ensure solid core support for levels 1 and 2 in the Richardson Maturity Model of REST APIs.

Being able to focus specifically on Drupal Commerce helps me research and iterate quickly to implement and use hypermedia with Commerce Services, but it does keep me a bit downstream of all that's going on in core (and for D7 in modules like Services, RESTws, and others). What I can say from here is that the future looks bright, and more members of the community are engaging around and communicating about proper REST API design. Together, I'm confident we'll see a successful push toward HyperDrupal in the very near future. Cool

Comments

..but technically it's "Hypermedia as the Engine of Application State."

This is the 4th constraint Roy Fielding defined in his dissertation on Representational State Transfer (the other 3 being identification of resources, manipulation of resources through representations and self-descriptive messages.

D'oh. Silly mistake, fixing now. And I definitely should've linked to Fielding's dissertation. Lol

For those interested, I keep it bookmarked. Haven't read it all yet, but it's a nice "someday" goal:

http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm

Thanks for the great blog post - I just started work on an open source iOS mobile app for Drupal Commerce QuickStart on Monday. I've got it up on GitHub at https://github.com/jefflinwood/ShoppingAppIOS

Everything you mentioned about improving GET requests would definitely be a huge addition to Drupal's Services - it's a little strange in that I find that using the JSON Views data source solves a lot of problems with the Drupal Services node resource.

I'm using the Drupal Commerce Services module, and I was definitely curious to see how it would progress in the future. I really like the idea of HATEOS, but I'm not sold that it's actually useful for mobile apps - any non-trivial app will have to still know how to display/search each resource - you wouldn't just present each field, for instance.

Again, great ideas, and I'm planning to continue working on apps for Drupal Commerce, so I'd be happy to help any way I can.

Hey Jeff, thanks for your comments, and thanks for sharing your project. I'll definitely have a look at it and maybe try to get 'er running locally. Cool

Re: Views and Services, yeah, a lot of my desire to really beef up our resources w/ respect to GET requests comes out of the mobile app project we were working on. It basically became a thing that was entirely driven by Views output, but I didn't like the idea of someone having to configure a View to get at something resembling a robust web service.

One of the huge wins here is the native support for filtering and sorting on any property, field, or field column without you having to add an exposed filter to the View for each potential option. In my G+ thread on this topic, we discussed a bit more the pros and cons of Views (or another tool, like Search API) to power these responses. I'm obviously pretty excited about what I have now, but I'll keep an open mind and see where we can go in the future. Smile

Re: HATEOS + Drupal Commerce, I'm not sure I have a solid picture of the types of emergent behavior we should expect to happen either. However, I do expect it to happen. It might come in the form of client side libraries that know how to parse API responses and present some standard components within the target UI. One very simple example might be endless scrolling in a mobile device that knows how to read the paging links in a response and keep showing a "Tap to see more" type queue (or just autoload additional products) when appropriate.

The immediate benefit, though, would be client development that doesn't have to depend on hardcoded URIs. If it knows how to traverse one Commerce REST API, you could point it at another API "root" and it could do its thing just as well. That probably doesn't matter as much for apps that have a 1:1 relationship between the client and the Commerce REST API (i.e. a store specific shopping app), but there may be other applications where it's more useful.

The nice thing about emergent behavior is that we don't necessarily have to know what to be looking for. Someone will just end up doing something cool, and we can all sit back and say, "Aha! I knew we did this for a reason." Tongue

Hi, Ryan. As always, great piece.

I believe (and wrote in Drupal Watchdog #4) that the future of the web isn't in pages or nodes, but in *streams* of data. We're seeing that already, as information providers mash up dynamic stuff (weather, traffic) with more static stuff (maps, public filings) to create practical, to-the-minute information products.

Drupal Commerce is a good example of a realization of the "data stream" concept, in the sense that product information comprises just another stream: It's ever-changing and atomic.

Anyway, that's my view. So it's good to see you (unconsciously?) agreeing. Smile

--Tom

Thanks, Tom. And I would even consciously agree with you. Lol

One of the advantages I saw in making full use of the fieldable entity system was that we would be able to use the same tools we have for importing / exporting content in Drupal for the store data. I apparently never finished my blog series "From Ubercrart to Drupal Commerce", but in part two I mention the big picture idea that eCommerce sites are conduits not containers, so we should do whatever we can to make it easier to get data into and out of the site.

Two clear examples illustrate the necessity:

  1. An store with an appreciable order volume will manage their accounting and fulfillment in third party systems. Orders may have come in through the front end of the website, but they just as quickly need to work their way out of the back end to these other services.
  2. A multi-channel store will likely need the same product content to be delivered to a variety of different places with differing format and localization requirements. We implemented such a site for a client recently where the Commerce backend was completely redesigned around managing these multi-channel requirements for their product catalog.

So I'm right with you, and I'll see about a follow-up post focusing on how adhering to REST standards will make Drupal Commerce stronger just like architecting it around the fieldable entity system did inside of Drupal.

The good news is that Drupal 8 is well on its way toward beefing up our REST cred. Larry Garfield, Klaus Purer, Lin Clark and others have been hard at work laying the groundwork for solid REST in core through the "Web Services and Context in Core" D8 initiative. I don't believe hypermedia is on the menu for D8 release, ...

All the data exposed by D8's Web API will be in JSON-LD (an upcoming W3C standard) so it will go beyond simple hypermedia support. You might wanna check out the specs at json-ld.org or this presentation.

I was under the impression that JSON-LD was more RDF for JSON than a hypermedia type. Watching this video it sounds like a pluggable media type through the @context parameter, using metadata in the JSON to make the actual data understandable (assuming you can parse the metadata). I don't see anything along the lines of templates or links beyond entity relationships, but I could be mistaken.

Mike Kelly offers hal+json, which was another strong contender for personal evaluation, as an alternative hypermedia type to JSON-LD sans RDF.

We recently decided to shift to HAL for the primary format for Drupal 8. HAL is solid on the hypermedia front. The challenge is making it generic. I don't believe we'll have a complete site traversal available via HAL, because too much of that would be site specific. All the plumbing will be there, though, including generic entity support. That means building a hypermedia API on Drupal 8 should be orders of magnitude easier on Drupal 8 than Drupal 7.

There is still much work to do, so we'd love to see you in the core queues with us. :-)