Getting to work on a stronger Drupal Commerce 2.x

I don't like being sandy but I love building sand castles. The potential is high but the stakes are low. You have unlimited raw material to build whatever you can imagine, and if you screw up you can just knock it over (to the delight of your children) and start again.

When your constant is impermanence, you adjust your expectations accordingly. If the sands shift or the tide rises, you shrug it off and start again or go back to bocce. However, if you were somehow bound by the fruit of your labor, you'd think twice about the when, where, and how of your building.

Lyle and I started developing Ubercart when Drupal 5 was still in beta, and we put up with the impermanence because we were just two dudes learning a lot and working from a clean slate.

Moving that forward to Drupal 6 took even longer because our codebase grew unwieldy, so I decided at Commerce Guys to trim the fat and start over. We didn't learn every lesson moving to Drupal 7, though, as it was still unstable when we began developing Drupal Commerce. With an unstable Views module. And an unstable Rules module. And an incomplete Entity API. Tongue

I don't think we would've done any differently, as the flip side of the instability is the opportunity to positively impact the development of the projects you depend on. For Drupal 8 we ended up contributing to core in other ways while tackling a full Drupal Commerce rewrite more slowly than we hoped. Even the code that we did develop against Drupal 8 is now outdated, as we had to juggle managing our existing ecosystem, writing new code, and rewriting that code to track changes in Drupal 8 itself.

However, we've recently been given the opportunity to host a variety of developers / documenters in Paris from June 30 - July 4 to re-evaluate our Commerce 2.x roadmap with the direct help and guidance of key members of the Symfony project. Specifically, we'll be looking to both move our code "upstream" from our Drupal modules into generalized libraries and take advantage of existing Symfony projects where possible.

Our target is an even leaner codebase with connections to the broader world of PHP based eCommerce. If you have any insights in this direction, please comment or contact me directly and consider joining us in Paris to learn and contribute.

Photo credit: j.s. clark

Beyond Wombats

I accidentally started publishing open source software in 2006, the first integration of the QuickBooks Web Connector with anything. This was pre-Ubercart when I was just cutting my teeth on PHP / MySQL development at Prima Supply, and I thought it would be fun to claim the code was written by wombats while I just published it online. I decided to own the silliness and start blogging on bywombats.com using Drupal 4.7 at the time - and immediately picked up a freelance contract doing QuickBooks integration work. Tongue

Fast forward a few years, a couple of eCommerce projects, and a Drupal startup, and the moniker has apparently run its course. A lot of folks over the years assumed "By Wombats" was the name of a Drupal shop and tried to contact me for work. Little did they know my site's contact form hasn't been working for years. Shock

This past week I finally got around to fixing that by moving from Rackspace to Digital Ocean and configuring my server to send mail through Mandrill. I upgraded to Drupal 7 (about time?) and played around with Mike Crittenden's Meedjum theme until I liked the look. He's presently trying to bribe me to help maintain the theme, but I'm holding out for better beer.

Migrating was the fun part, as it was my first opportunity to make use of the Drupal-to-Drupal data migration module. I couldn't get moving files to work for some reason, but I figured out how to make options settings work in the configuration form (leave the select lists alone and put the value in the "Default value" text field) to make use of the preserve_files option. I copied the old files to the Drupal 7 files directory and ran the migration with that flag set for it to start managing the files that were already in the directory.

Migrating taxonomy terms was straightforward, but it took me a while to get blog posts right. I learned I had to select the taxonomy term migration in the "Source migration" select list to ensure blog posts got created in the Drupal 7 site referencing the correct term ID (since they changed during migration). I had to do the same for files, choosing the "Upload [upload]" source field and selecting the file migration I used as the source migration. I also had to set the optional MigrateFile class to "MigrateFileFid" to ensure the new blog's file field referenced the appropriate file IDs.

Perhaps the best thing about Migrate is its ability to rollback migrations and re-import them to debug the process. I did this several times for the various migrations. I also made use of its ability to import a representative subset of the awaiting content while I was debugging the term / file association process. Clever module - Mike Ryan rocks.

The future for this blog is a consolidation of various other things I'm doing online, including blogging about life in Greenville, SC (hint: it rocks) and Roguelikes / indie games (again with the Wordpress?). Once that's done, I'll carry on with the blogging about Drupal, startups, and a few other things, including my recent explorations with R and Commerce Guys's Platform.

I hope to hear from you soon. Smile

On Cats, Bags, and Letting it all out

One of the unexpected challenges in raising money to grow your business is keeping mum about the deal until it's time. Time is likely among the many terms you'll find defined in your contract, and between the day you sign the papers and the day that time actually arrives, you're glowing inside because your investors believe in the potential of your business and want to see you do more.

Investors don't magically make a business plan succeed, nor do they single out the sole source of success behind a business or an idea. This is certainly the case with Commerce Guys' raise announced last week. We know for a fact that our investors get open source as much as they do eCommerce. Even as they evaluated us on our ability to execute our business plan, they evaluated us on how well we work within and alongside the larger Drupal community. When they took a close look, they saw the strengths of the community and the caliber of developers collaborating with us to build Drupal Commerce.

That's what makes it so exciting to share the news - investors and developers who have grown their own businesses and, in the case of the team at Open Ocean with MySQL, their own open source projects have looked closely into both Commerce Guys and Drupal Commerce and felt confident enough to front some serious cash for us to kick our efforts up a notch. Many of these guys have built their own eCommerce systems and understand the challenges we're in a unique position to solve through Drupal 7, Views, Rules, and Commerce, and they're guys who understand the importance of the community in the success of any open source project.

So, we're not crazy after all, and what we've been trying to build with our friends at Commerce Guys and in the Drupal community isn't crazy. Ambitious, sure, but achievable. Our vision for Drupal Commerce remains the same - to see Drupal Commerce become the world's leading open source eCommerce framework. For the last two and a half years, my time has been set aside by Commerce Guys to develop the code (with plenty of help from other brilliant Commerce Guys and community contributors) and grow the community needed to make it happen. Now we've sold the vision to some very smart people with deep pockets outside our normal circles and are eager to see what happens next.

Their affirmation is much appreciated, but so is the money that will let us hire and set aside even more developers to "scale me" out a bit. We need to address immediate concerns pertaining to documentation and community support on DrupalCommerce.org. We'll need to make sure we follow-through on our longstanding 2.x strategy to bring some sanity to the user experience for administrators even as 1.x has privileged developers. All the while, there will be more than enough module maintenance and distribution work to go around!

Addressing these needs for Drupal Commerce should only require a fraction of the money we've raised, but it's a good start that will have an immediate positive impact on the thousands of people already using Drupal Commerce to power their online businesses. If you think you can stomach working with me on a daily basis and have the chops to help us succeed, be sure to get in touch.

Drupal Commerce Alpha 4

Well, Christmas is already over for a lot of you, but there's still an hour left here in the States for me to wish everyone a Merry Christmas with a Drupal Commerce Alpha 4 release. I drove down to Louisiana to celebrate with my wife's family, and for once I took the passenger seat and got a good bit of work in to tighten up our Entity / Rules integration. Éowyn had a meltdown after 8 hours, too, so thanks to the Super 8 Motel in Wheatley, AR, I made even more headway on the dynamic Add to Cart form and its dependent attribute display.

However, integration and maintenance issues aren't the most exciting things in this release. I'm much more excited to announce the extensive work done on Drupal Commerce's Rules powered dynamic pricing system. We now have a centralized system empowering site builders and store administrators to implement pricing rules for discounts, tax calculation and display, and currency conversion through the user interface. While some of the subsystems still need attention, the Rules are in and working on the Calculating the sell price of a product event. You can see the results immediately on product field displays and in the shopping cart.

For example, I might want to give members to my Wombat Dating site a 10% discount. Never mind the absurdity of the scenario (if that's even possible) and see the basic product display node showing the product image and price fields:

The price field is displayed with the Formatted amount display formatter, which knows that modules are enabled that allow for dynamic price calculation:

This particular calculation option allows product sell prices to be calculated through Rules, so I've setup a 10% Members Discount using this simple rule configuration:

The end result, as you would expect, is a much cheaper night out on the town with our lovely wombat:

This system is a huge step forward from what we attempted in Ubercart, which lacked a baked in UI, touched every price on the site, and involved a pretty heavy caching system that couldn't address the problem of executing queries based on calculated prices. The system eventually got the boot, but we got a chance to tackle the challenges afresh with Drupal Commerce. Through some good discussion at DrupalCon Copenhagen with Damien, Miro Dietiker, and others, we realized the problem was actually in how we understood our dynamic pricing needs.

The first thing we did was restrict our initial scope to calculating product sell prices. That solved a lot of performance problems and API confusion on its own. However, we still had to solve the querying problem since we're depending on Rules to perform the calculations. This means we require PHP to execute to find the actual sell price of a product for any given customer. Simple queries that order and filter lists of products based on the sell price would be inaccurate once the price was altered for display, so a customer with a special discount on a product won't see it at the top of a catalog View when he orders by price. We've run into this problem several times at Commerce Guys, especially when displaying prices with tax included.

What we determined was that we needed to pre-calculate dynamic prices so the data would exist in the database before a customer ever needed it. Since we know the Rules attached to the price calculation event, we can pre-calculate prices for every combination of applicable Rules. The pre-calculated price data can then be joined into queries for any given set of Rules, enabling ordering, filtering, and faceting to work with actual sell prices instead of just the base price stored on the field. There are some guidelines to follow when constructing Rules for pre-calculation, and we've also devised a few simple ways for larger sites to keep the calculated price table from breaching epic row counts.

If I lost you, my apologies. It's very exciting stuff for the project, and I tend to talk too much about it. I've already explained it to all of my in-laws, including my 75 year-old grandfather-in-law whom I'll be teaching Drupal while I'm down here for his personal website.

I'll be blogging more about the dynamic price pre-calculation system, especially once it gets an actual user interface. For now it's a small API and a database table, but it's itching to be put through its paces. I'll be testing it first with Views and welcome additional eyes on the code. For more information on how it functions, refer to the development specification and the copious amount of comments on the pertinent functions in the Product Reference module.

Anyways, I've been closeted up with the laptop enough for one Christmas, so I'm going to join the rest of the family with some doctored eggnog and a seat in front of the fire. I hope you all had a Merry Christmas, and I look forward to feedback on the new hotness in Alpha 4.

Check out the release notes for the full changelog since Alpha 3, If you're thinking of resolving to visit Paris for the New Year, check out the details on our Paris Commerce Sprint, January 17-21, 2011 and consider joining us in Paris to push this code from a planned beta to 1.0 in time for DrupalCon Chicago. We'd love to have you.

Pages