Toward HyperDrupal

Back in September I had the privilege of attending a web development conference in my new hometown, Greenville, SC. The occasion was REST Fest, which was primarily a barcamp format gathering of REST API developers and thought leaders from around the world.

I got to meet several local web developers along with encouraging and helpful out of towners like Mike Amundsen (author of Building Hypermedia APIs with HTML5 & Node, highly recommended) and Sam Ramji (erstwhile champion of open source at Microsoft, now with Apigee). I also got a second chance to meet David Zuelke, who I somehow missed at DrupalCon Munich where he presented on RESTful Web Services. (In addition to REST, he and I geeked out over the BMW Performance Track, which I enjoyed the day before thanks to demo day at The Iron Yard, our local startup accelerator and host to REST Fest and other tech events.)

The event was quite the eye opener, an opportunity for me to be out of my element with nary a Drupal developer in sight. However, the topics covered and examples shared were incredibly practical for me, as we were in the middle of researching and implementing RESTful Drupal Commerce at Commerce Guys for a client project and for a Commerce Kickstart mobile app developed with the help of Sumit Kataria.

Fast forward now three months and I'm finally circling back around to pump up the Commerce Services module to turn any Drupal Commerce site into a REST API server. My focus thus far has been on improving GET requests to support a variety of query parameters offering:

  • Response format specification (i.e. a subset of an entity's properties / fields when you don't need the entire entity)
  • Filtering by property names, field names, and multi-column field column names
  • Sorting by those same parameters
  • Paging through the result set via limit and offset parameters
  • The ability to include referenced entities in the response
  • The ability to flatten field value arrays to the current language and from arrays to scalars for single value, mono-column fields

As a best practices guide, I've been using an incredibly helpful eBook from Apigee called Web API Design: Creating Interfaces that Developers Love. The idea here is to follow standard patterns for all of our entity resources that allow API clients to easily browse the API.

Additionally, my goal is for Drupal Commerce API servers to actually be RESTful, not simply RESTish. If you aren't familiar, a great articule to bookmark and read when you need it is A Brief Introduction to REST by Stefan Tilkov. As we build out the Commerce Services module and deploy projects from client sites to mobile apps, we'll also publish developer documentation that defines and demonstrates best practices use of the system.

My end goal here is to keep pushing forward to a holy grail of sorts - HyperDrupal. By simply creating powerful resources that respect HTTP methods (and other headers) as defined, we're two thirds of the way there. The final step (as identified by the Richardson Maturity Model) is to format our responses using a "hypermedia type" - a media type that includes the hypermedia controls we're used to in HTML like links (a), forms (forms that POST), query templates (forms that GET), and the like. See Mike Amundsen's book (referenced above) for more detail.

The media type I'll be focusing on first is Mike's Collection+JSON media type. Even as I do my testing with Commerce Services, I'm using the beginnings of what I hope will become the Collection+JSON response formatter for Services based REST servers. The "big idea" here is that an API client like a mobile app could browse your API like a user would browse your website.

For example, the app would start at the "front page" of your API where it might find the link to your product list. When the customer chooses which product to view, in addition to the product information, the GET response includes a form with the necessary parameters to submit a POST request to add the product to the cart. If built to consume this information properly, the client never actually need to know how your URLs are constructed, whether or not you've recently changed API responses, or when you've added new features to the site. That information would all be communicated through API request responses.

This process has been awkwardly acronymized as "HATEOS" (pronounced had-ee-us) - Hypermedia as the Engine of Application State - and is further described in Haters Gonna HATEOS by Steve Klabnik, proprietor of the living eBook Designing Hypermedia APIs. It may sound a bit "pie in the sky", but the advantages of Drupal / Drupal Commerce offering this level of API functionality out of the box are too good not to pursue.

The good news is that Drupal 8 is well on its way toward beefing up our REST cred. Larry Garfield, Klaus Purer, Lin Clark and others have been hard at work laying the groundwork for solid REST in core through the "Web Services and Context in Core" D8 initiative. I don't believe hypermedia is on the menu for D8 release, but they were just granted a reprieve in the form of the D8 feature completion phase to ensure solid core support for levels 1 and 2 in the Richardson Maturity Model of REST APIs.

Being able to focus specifically on Drupal Commerce helps me research and iterate quickly to implement and use hypermedia with Commerce Services, but it does keep me a bit downstream of all that's going on in core (and for D7 in modules like Services, RESTws, and others). What I can say from here is that the future looks bright, and more members of the community are engaging around and communicating about proper REST API design. Together, I'm confident we'll see a successful push toward HyperDrupal in the very near future. Cool

Commerce Shipping brings a lot to the table

This week we finally packaged the full release of Commerce Shipping 2.0, and I couldn't be happier. Thanks to the efforts of my co-maintainer, googletorp, and contributors helior, cvangysel, and andyg5000, we have a solid shipping framework for Drupal Commerce that offers more flexibility and granular control than any other shipping system I've seen, Drupal or not.


Commerce Shipping checkout panes on the default checkout form.

What's the big deal?

As I wrote in the release notes, this milestone really is a big deal for Drupal Commerce. Among the many benefits this module brings to the table are:

  • The ability to use any number of shipping methods to calculate shipping rates on a site, using Rules to control when a particular method should be enabled. You can use one carrier to quote domestic rates, another to quote international rates, and a flat rate to offer free shipping if certain conditions are met.
  • An architecture that subdivides shipping methods into their individual shipping services (e.g. Ground, 2nd Day Air, Next Day Air, etc.), giving you granular control through Rules over which services should be available to an order. You might use this to ensure perishable products only ship overnight or to only offer guaranteed delivery dates for items in stock.
  • A shipping rate caching system that allows carrier rated quotes to be fetched in a single API request and stored locally (with an optional timeout) for the duration of the checkout process.
  • A shipping rate calculation API that lets you perform additional calculations on top of the base shipping rate for each service through custom modules or Rules. This allows you to add taxes, handling fees, discounts, and more to your shipping rates. It also turns the Flat Rate module into the basis for more complex shipping calculation schemes - weight based, quantity based, etc.
  • A checkout pane that can automatically recalculate shipping rates as customers enter their billing and shipping information. This lets you display accurate shipping rates that depend on address data on a single page checkout form.

Additionally, in conjunction with last month's Commerce 1.4 release, Drupal Commerce supports copying addresses and other matching field data between your billing information and shipping information checkout panes. Many thanks to the contributors mentioned above for working out all the kinks between this feature and the recalculation of shipping services.

This architecture removes limitations found in other open source eCommerce systems, like Ubercart and Magento, that often take an "all or nothing" approach to showing shipping services from any given carrier on the checkout form. Even better, features that this module delivers for free would actually cost you extra to add to systems like Magento that nickel and dime you through plugin marketplaces just to make your checkout form more customer friendly. Tongue

Next steps for Commerce Shipping

With this release there are already modules to support Flat Rate shipping, carrier rated quotes from UPS, FedEx, USPS, and others, and in store pickup facilitated by Kiala. Additional contributed modules focus on extending the system with other features, like joachim's Shipping Weight Tariff module that offers a simplified table based interface for setting shipping costs by total order weight.

Next steps involve bringing these contributed modules and their dependencies (like the Physical Fields module and Commerce Physical Products) to a full release, an effort already underway thanks to andyg5000 and other contributors / maintainers.

This work has been a community effort, and it has paid off for everyone. Smile

Executing arbitrary JavaScript from a Forms API #ajax callback

Drupal 7's Forms API included several sweet improvements making it easier than ever to build dynamic forms (with or) without writing custom JavaScript. I'm sure there are better summaries than I have time to bake up right now, but a quick list of pertinent properties includes #attached, #states, and #ajax. This post focuses on the #ajax property and how you can use it to execute arbitrary JavaScript during a form refresh.


A form element that uses #ajax to allow address copying in checkout.

Briefly stated, what I'm calling the form refresh is what happens when Drupal finally asks the callback function pictured above what response to send to the client. This can be one of two things:

  1. A piece of the form that will be rendered to replace the contents of the wrapper element specified in the #ajax array
  2. An array of AJAX commands to execute in the browser

We do both in Drupal Commerce, sometimes simply refreshing part of the form that has been updated and other times sending back a nice fat batch of commands to update various parts of the page. We use commands arrays to great effect to make attribute widgets on the Add to Cart form dynamically update product fields that have been rendered into node pages. This is how product images change based on the color selected, for example.

To create these commands, you use functions like ajax_command_replace(), which simply swaps out the contents of an element on the page with something else as if you'd used the first type of response. The other commands are mostly similar manipulations of the DOM, with the full list of commands available here.

I personally think we need more modules brave enough to use ajax_command_alert() in their #ajax callbacks. Cool

One thing you won't find in that list is something like ajax_command_eval() to execute some arbitrary JavaScript back at the client. We've had just such a need a couple times now, once in Commerce Shipping where I wanted to trigger a shipping rate recalculation when an address copy button was clicked and another time on a client site where updating attribute widgets on the Add to Cart form needed to trigger updates in a Flash based product customizer.

Our solution was to use ajax_command_invoke(), which executes a jQuery method with whatever arguments you pass it. Your module should first define a jQuery plugin to hold your code, which is surprisingly easy. Your form them needs to use the #attached property to ensure the JavaScript file is included when your #ajax enabled form element is rendered. Then you just have to invoke your custom jQuery method via ajax_command_invoke(), and you're golden.

This diff from the Commerce Shipping module shows this (hack?) in action. Theoretically, ajax_command_invoke() invokes a jQuery method on some element selected from the DOM, but this example currently just triggers another element's click event using a small jQuery plugin that breaks a design rule by returning nothing to chain to. C'est la vie.

Next Steps for Drupal Commerce

It was my pleasure to demo Commerce Kickstart 2.x at DrupalCon Munich as I presented Next Steps for Drupal Commerce. The relaunched distribution has been a long time coming, and I couldn't be happier with the release our Kickstart team at Commerce Guys has produced. They've proven the idea that we could keep the core of Drupal Commerce a slim eCommerce framework and wrap it in a usability layer that makes it both attractive and easier to administer.

Beyond the sheer amount of polish that has gone into the UI from the installer to the "Shiny" administration theme, its two most exciting features for me are the Inline Entity Form and Views Megarow functionality.


An inline entity form on a product display node with multiple variations.

Inline Entity Form allows us to embed product details forms inside node forms, simplifying the product vs. product display architecture that has made Drupal Commerce notorious (and truly flexible). I could write a whole post about just how we've implemented it, but I'll save the gory details for a later time.

I will at least point out that if this were a single value product reference field, instead of the drag-and-drop table pictured here, the module would simply embed the product related form elements directly into the node form.


A Views Megarow in action after clicking the "Quick Edit" link.

An equally impressive technical implementation, Views Megarow makes it simple to edit groups of products and orders inline in a View through some fancy AJAX row expansion. It gives you easy access to bulk product edit forms and order summaries complete with order activity streams and administrator comments.

Both of these features have been meticulously planned, designed, user tested, and revised to provide a very streamlined user experience for store administrators. With my time mostly focused on other development, I happily applaud our team for putting out such high quality work.

However, even with this release under our belts, we're hardly ready to sit back and relax. We have plenty more to do in the development of Drupal Commerce distributions, contributed modules, and the core framework itself. I spent the final part of my session at DrupalCon proposing the topics and timeline under consideration for a Drupal Commerce 2.x roadmap targeting Drupal 8.

To make sure we're meeting these challenges head on, I proposed a Commerce 2.x planning and development sprint at our offices in Paris in mid-October. The primary goal of the sprint is to research and plan development on Drupal 8, including how to best utilize and work with the various core initiatives, what new modules to add as dependencies or bring into core as new features, and what core code needs refactoring.

This sprint is on the calendar for October 15-19, and while we likely won't be laying down a lot of code, we're still looking for collaborators. We're particularly interested in folks with experience developing for Drupal Commerce on Drupal 7, bonus points awarded to those who have had to work around its limitations.

I'll post more on the topic in the near future, including specific items we intend to address in 2.x development. In the meantime, it's about time for a Commerce 1.4.

Pages