Drupal Commerce in a git.d.o world

Because we were doing our Commerce module development on GitHub and exporting to CVS, we need to make some changes now that the Git migration on d.o is complete. Obviously the main Git repository for the project is now on d.o instead of GitHub. To clone the latest code, you can follow the helpful Git instructions included on Drupal project pages.

All of my D7 projects now include a 7.x-1.x branch that you should track to get the latest commits. I also updated the various dev releases of Commerce, Address Field, and my Commerce contribs to be snapshots of this branch, essentially deprecating master. I'm trying to decide the best thing to do about those pesky master branches and will likely just git rm everything in them and leave behind a README indexing the various HEADs in the repository.

Many people had forks of both the old Commerce repository and my development repository on GitHub. Additionally, some of you may have pending commits that haven't been pulled yet. If this is the case for you, please generate a patch instead of a new pull request and post it to the issue queue. As of this point I have no plans to rewrite the commit history of the new official repository to match the existing history on GitHub, meaning I cannot cleanly pull from your repositories.

(I have read Ben's blog post, and I'm sure there are some other finer aspects of Git that I'm just ignorant of to accommodate this... but I'm not sure it's worth the hassle when forks will need to be updated across the board anyways.)

This seems a little brute force, but I think it's for the best. The commit graph on d.o is now a nice tidy straight line, whereas the old graph on GitHub was a fractured mess that included several instances of duplicated commits thanks to botched merges and branching. We were still learning an appropriate Git workflow at the time and made several mistakes that not only dirtied up the history but also made it hard to sort out how to pull from various forks. (Accordingly, current contributor statistics on Ohloh are inaccurate anyways.)

This does mean some contributor credit in the commits will be lost, but I'll be updating various project pages to credit beta contributors for their commits. Regardless, I've been crediting contributors in commit messages all along, and merge commits show whose repository the merged commits came from. As of this point, I think the loss of a little meta information is worth the clean slate with the new repositories. Flame away in the comments if this is the stupidest idea you've ever heard... just want to make sure my plans are clear.

While we figure out how to handle pull requests in an environment that doesn't natively support them, in your issue please indicate your repository URL (whether it's a d.o sandbox of GitHub fork) and branch that I should pull from to compare. Bonus points for attaching a patch for quick review on site (or linking to the diff on the appropriate web interface). I'll add your repository as a remote so you don't have to post it every time. Additionally, I'd prefer it if you used a single branch per issue you're addressing with the branch name simply being the nid of the issue - makes it quite easy for me to branch and pull locally.

(I should mention that the good ol' patch process is perfectly fine if you don't want to maintain a public repository for me to pull from. You can just make your commits locally and diff 'em all in one fell swoop to create a patch for review.)

This is essentially what we've been doing on GitHub, so no surprises here for existing contributors. I'll leave the GitHub repos up for a while, but I'm updating their READMEs to point to the new repository on d.o. Once tested and approved, these guidelines will find their way into a handbook page for posterity's sake.

Calling all pre-beta testers

This is a quick call for help before I package up a beta release of Drupal Commerce. The last few weeks I've been working around the clock on blocking issues, dependency issues, Rules based pricing and tax (incl. VAT) support, and last minute database tweaks. My wife has been incredibly supportive of me thus far, but now I need help from my friends with a bit more technical acumen.

Here's the deal: for all intents and purposes, this code is beta ready. Database schema changes have mellowed out. The modules have solidified. The core feature set lets you start selling quickly. The Views and Rules integration provides for some fine standard interfaces and workflows. The last few weeks, pcambra has been our SimpleTest hero, and now he's working with recidive to flesh out the functional tests even further.

What it really needs is eyes. Your eyes. If you have a few spare moments, you could greatly help by testing an installation and ensuring there aren't any crazy bugs that I just don't see because of my development environment or selective blindness. These could take the form of installation bugs, odd default configurations in Views / Rules, typos, broken access control, and more. There are bound to be things I'm missing, and I really need your help to know what they are.

Here's how to get started very quickly:

  1. Download Drupal 7.
  2. Download into your sites/all/modules directory the latest dev versions of the following dependencies:
    1. Address Field
    2. Ctools
    3. Views 3
    4. Entity API
    5. Rules 2
  3. (Note: Getting these from CVS would be better to get their latest commits.)

  4. Download into your sites/all/modules directory my Git dev version of Drupal Commerce (or just git clone git://github.com/rszrama/drupalcommerce.git).
  5. Download into your profiles directory my Commerce Dev installation profile and download into your sites/all/modules directory Admin Menu (a dependency of the installation profile).
  6. Now install Drupal 7 using the Commerce Dev installation profile and see what happens!
  7. If you're feeling adventurous, use the Standard installation profile and then try to enable the Commerce modules manually to make sure the standard install process works. (You'll have to adjust some permissions, enable the cart block, and build a product display node type from scratch - read more about that in the first issue of Drupal Watchdog. )

Once it's all installed, cruise through the various admin interfaces, settings forms, and front end features and let me know what breaks. Note, I'm not looking for support requests, "How do I...?", or feature requests, "Needs flat rate shipping support." Rather, I want to know what, if anything, of the core feature set just doesn't work.™

One of the latest features to test will be to configure a tax rate for your country / state. Simply browse to admin/commerce/config/taxes and add the tax rate. If you made it a sales tax, it'll show up once you get to checkout. If you made it a VAT, it'll show up inclusive in product prices. Additionally, if you made it a VAT, you can edit products and enter their prices including VAT. Taxes work through the same Rules event as other product pricing rules, so you can try adding a discount rule and fiddle with its weight to see how it interplays with your tax rules.

Drupal Commerce comes with an example payment method module that lets you just put a name in on the checkout form to test payment. You can test additional payment methods if you have developer accounts for Authorize.Net, CyberSource, or PayPal.

Please report any bugs you can turn up (or +1 existing bugs) in our issue tracker. I'll also be hanging out in #drupalcommerce on irc.freenode.net if you're into IRC.

While you do this, I'll be running tests of my own, reviewing the names and descriptions of all our permissions, and ensuring the customer profile integration with Checkout and Order administration is up to snuff.

With any luck, we'll all be beta testers tomorrow.

To App or Not to App?

I started to answer this question with a discussion of my own motivations for module development. They are many and varied, but they pretty much all come down to me either having an idea and some time or me getting paid to make time for an idea. However, I realize a person's motivations for writing his or her intangible "Drupal App" don't really matter if a particular motivation can co-exist with the existing model of community module development (whether it's evil or not, ignored or not, mirrored or not)... and ultimately I didn't want to do Robert's problem solving for him on what that could look like.

I then decided to take the angle of problems with the architecture of Drupal modules vs. an App store model, current support and maintenance channels, and a host of other problems I see with the idea of Drupal apps. Some points were established elsewhere... Yes, we can't really sell modules. Yes, people are selling themes. But no, I don't think selling "Features" would work... or at least I don't think it would be good for the consumer.

The traditional "App" (I hate that word - really? are we selling full applications?) must encapsulate its functionality, and it can't be effectively supported if its innards are free to be customized and mixed in with (i.e. broken by) other modules. This lack of interoperability between paid plugins on other platforms was actually quite a shock to me at last year's CMS Expo and is a big weakness. It also increases the development burden on the "App developer" making it a net loss for the consumer and provider. (Really, I saw an e-commerce plugin baking in its own Views system.) In fact, the only person I see getting a good deal is the "App distributor."

But I digress... these are just software / social engineering problems. If distributors really wanted developers and consumers to buy into their distribution models, they just have to provide solutions for these things. Once again, I'm not trying to do anyone's problem solving... unless they want to cut me in on their good deal.

What an App Store won't solve

I think other threads of this conversation have been dealt with adequately elsewhere, and overall I think an App store could work if all caveats are accounted for and the store's success is mitigated enough by the strengths of drupal.org to not blast holes in the community development model we currently have. One thing I don't see getting a lot of play in the discussion, though, is whose problems an App store wouldn't solve. I've been thinking about this since taking part in the discussion with ICanLocalize (sorry, can't find the link).

A paid distribution channel will not provide you with the time or money to code or support your module. You still have to make the initial investment in time or money to get your "App" written, and long before you're turning a profit you'll need to be proving your intent and ability to support your product. If you then rely solely on distribution fees to recover your investment, you're relying on the wroooong license to get 'er done. If you intend to bundle support and maintenance with the distribution, I just need to point out again... this is first and foremost a good deal for the distributor unless he turns your App into Angry Birds.

Really, though, this is freaking Drupal... all we need is for enough interested people to build an installation profile for paid module support. Does that mean every module's support would go behind a paywall? I sincerely doubt it, as again, developers still have to prove their competence to support their project to non-paying new customers to convert them in the first place. They're also not the only ones who support modules on d.o... every one of their users is a potential support provider.

I do think that publicizing intentionally scaled back support in the absence of a paid distribution channel is not a good way to win hearts and minds. You don't have to say it, just do what you need to make money and offer premium support for people willing to pay for it. There are also other ways to build support around your module in the community, whether it's a bug squad or a sponsored development sprint.

Leaving module developers behind

Finally, something that just hit me... the only "Apps" I see as being viable at present are SaaS plugins and themes (thanks to licensing opportunities). Others see distributions and Features-esque configurations as viable. Perhaps there are still other viable options, but at the end of the day, none of this translates into dollars to fund the module development that any of these products depends on. And isn't that what started the conversation in the first place? Are we trying to figure out how to support module developers, or are we looking for a different way to make money using Drupal + contributed modules beyond professional services?

Drupal Commerce Alpha 4

Well, Christmas is already over for a lot of you, but there's still an hour left here in the States for me to wish everyone a Merry Christmas with a Drupal Commerce Alpha 4 release. I drove down to Louisiana to celebrate with my wife's family, and for once I took the passenger seat and got a good bit of work in to tighten up our Entity / Rules integration. Éowyn had a meltdown after 8 hours, too, so thanks to the Super 8 Motel in Wheatley, AR, I made even more headway on the dynamic Add to Cart form and its dependent attribute display.

However, integration and maintenance issues aren't the most exciting things in this release. I'm much more excited to announce the extensive work done on Drupal Commerce's Rules powered dynamic pricing system. We now have a centralized system empowering site builders and store administrators to implement pricing rules for discounts, tax calculation and display, and currency conversion through the user interface. While some of the subsystems still need attention, the Rules are in and working on the Calculating the sell price of a product event. You can see the results immediately on product field displays and in the shopping cart.

For example, I might want to give members to my Wombat Dating site a 10% discount. Never mind the absurdity of the scenario (if that's even possible) and see the basic product display node showing the product image and price fields:

The price field is displayed with the Formatted amount display formatter, which knows that modules are enabled that allow for dynamic price calculation:

This particular calculation option allows product sell prices to be calculated through Rules, so I've setup a 10% Members Discount using this simple rule configuration:

The end result, as you would expect, is a much cheaper night out on the town with our lovely wombat:

This system is a huge step forward from what we attempted in Ubercart, which lacked a baked in UI, touched every price on the site, and involved a pretty heavy caching system that couldn't address the problem of executing queries based on calculated prices. The system eventually got the boot, but we got a chance to tackle the challenges afresh with Drupal Commerce. Through some good discussion at DrupalCon Copenhagen with Damien, Miro Dietiker, and others, we realized the problem was actually in how we understood our dynamic pricing needs.

The first thing we did was restrict our initial scope to calculating product sell prices. That solved a lot of performance problems and API confusion on its own. However, we still had to solve the querying problem since we're depending on Rules to perform the calculations. This means we require PHP to execute to find the actual sell price of a product for any given customer. Simple queries that order and filter lists of products based on the sell price would be inaccurate once the price was altered for display, so a customer with a special discount on a product won't see it at the top of a catalog View when he orders by price. We've run into this problem several times at Commerce Guys, especially when displaying prices with tax included.

What we determined was that we needed to pre-calculate dynamic prices so the data would exist in the database before a customer ever needed it. Since we know the Rules attached to the price calculation event, we can pre-calculate prices for every combination of applicable Rules. The pre-calculated price data can then be joined into queries for any given set of Rules, enabling ordering, filtering, and faceting to work with actual sell prices instead of just the base price stored on the field. There are some guidelines to follow when constructing Rules for pre-calculation, and we've also devised a few simple ways for larger sites to keep the calculated price table from breaching epic row counts.

If I lost you, my apologies. It's very exciting stuff for the project, and I tend to talk too much about it. I've already explained it to all of my in-laws, including my 75 year-old grandfather-in-law whom I'll be teaching Drupal while I'm down here for his personal website.

I'll be blogging more about the dynamic price pre-calculation system, especially once it gets an actual user interface. For now it's a small API and a database table, but it's itching to be put through its paces. I'll be testing it first with Views and welcome additional eyes on the code. For more information on how it functions, refer to the development specification and the copious amount of comments on the pertinent functions in the Product Reference module.

Anyways, I've been closeted up with the laptop enough for one Christmas, so I'm going to join the rest of the family with some doctored eggnog and a seat in front of the fire. I hope you all had a Merry Christmas, and I look forward to feedback on the new hotness in Alpha 4.

Check out the release notes for the full changelog since Alpha 3, If you're thinking of resolving to visit Paris for the New Year, check out the details on our Paris Commerce Sprint, January 17-21, 2011 and consider joining us in Paris to push this code from a planned beta to 1.0 in time for DrupalCon Chicago. We'd love to have you.