The transition from writing free-form web applications, to writing modules for Drupal, or APEX customizations for Salesforce, is another set of lessons in protocols and constraints. First, it’s not enough to understand the syntax and form of the language (this is especially true for APEX – and beware the required test coverage!) One has to understand how the surrounding application works – what APIs or methods one can use, and how. And unlike long standing languages, there aren’t lots of detailed cookbooks and that sort of thing lying around – a lot of it is learning from other folks, as well as just learning by trial and error.

And, in my small forays into learning frameworks like CakePHP, Ruby On Rails, and others, it seems like these days, coding for the web is many lessons in constraints – which is a good thing, I think. Even though it feels like beating my head against a wall, it’s nice to know that I won’t “dump core” and break Salesforce (although I for sure have broken Drupal on occasion!)

A few months ago, in preparation for a couple of projects, and a big push into this area for our company, I was faced with a strategic choice – go it alone, and build our own integration module for client projects,  or plunge into using and working with the contributed salesforce modules. Truth is, it wasn’t really a choice for me – I’ve got using and contributing back to open source projects in my DNA somehow. Although we certainly could have chosen, like others, to go our own way, we have committed ourselves to using, and contributing to the modules on drupal.org.

What we lose:

• Complete control over development process and direction
• Not having to fix other people’s bugs in order for stuff to work

What we gain:

• Not having to reinvent a number of wheels
• An easier upgrade path
• Build on the work of others
• Collaborate and learn

The work done so far on the modules is really solid – and it’s getting better. There is a great new maintainer, and increasing activity and contributions. There are also a number of other module integrations (like Ubercart, Webform, and FeedAPI) that are moving forward. Integrations with Views and Actions are also moving being considered (it’s instructive to look at the issues queue). This is stuff that would be hard to match, and makes building integrations for different kinds of sites easier.

So beyond just my own personal preference, I think that there is much benefit, both for our clients, and for us as a company, in hitching our wagon to theses contributed modules instead of going it alone.

I’ve considered myself pretty successful at both helping clients navigate the tough waters of web development projects, as well as accomplishing web projects for them. Recently, though, I had two projects that ended up, for wont of a better term, clusterfracks. And I’ve spent a lot of time lately trying to figure out what lessons I need to learn from those projects – what can I take away from them so I don’t make the same mistakes again. They were both custom web applications, both projects that I was a strategic, rather than engineering, partner on. Both projects were attempting to accomplish pretty sophisticated database functionality (such as case management). Functionality I knew how to get done, because I’d accomplished it before – so I had a very good feeling for what kind of code it would take to accomplish the task (and, ergo cost and time.) But what I hadn’t taken into consideration is how slick, AJAXy, easy to navigate, and easy to understand user interfaces people have gotten used to in the last few years. And, frankly, have come to expect. And how unwilling people are to sacrifice that for raw functionality.

I did a lot of self-examination: where did I go wrong? What could I have done differently? Was it the client? The developers? Me? I realized a fairly simple truth. It was all three.  In reality, I should have looked at the budgets of those projects, and looked at the clients straight in the eye and said, “double, or triple the budget at least, or don’t do the project.” And walked away if they insisted. The vendors should have bid triple what they did, and had more user interface expertise on board. The clients should not have expected to get slick 2009 functionality for a mid 5-figure budget.

The easier a user interface is to use, the more money and time it took to create. It’s that simple. What most nonprofit decision makers don’t completely realize is that the interfaces they work in every day when they shop,  or tweet and facebook, or use other online tools, are the product of millions and millions of dollars of venture capital, or, in some cases, hundreds of thousands of person hours of work in open source projects (or some combination of both.) Ground-up custom applications, even when written in great frameworks like Ruby on Rails or CakePHP, which save all sorts of development time, just are not going to have the user experience people are getting more and more used to without very serious investment of time and expertise. In addition, most small development shops don’t have the user interface expertise on hand to accomplish that task, even with a hefty budget.

So the lessons:

1) If you are embarking on a custom development project (such as a case management, for example) exhaust every possible option of using and customizing/modifying existing tools (Salesforce, CiviCRM, SugarCRM, other open source tools) before you begin to consider custom development from scratch.

2) If you have a budget of less than $100,000, go back, and stay, at step 1. I know this is high, but I’m serious. Obviously, simpler projects won’t need a budget of this sort. But simpler projects generally don’t need custom databases.

3) If you’ve got the cash to spend, and have exhausted all other options, when choosing a vendor, make sure the vendor you choose has UE expertise on hand. Look at other custom database work they’ve done. Dig in. Make sure it has the ease of user experience that you are expecting.

4) Remember the mantra: the easier it is to use, the more expensive it is to build.

The organization should be able to create several different kinds of events, and determine for each kind of event which detailed information will be taken. Those events can be displayed in a list or calendar format. Users can register for events, and pay using a credit card.

There are many ways to describe this story – it certainly can be a lot more detailed, but what’s clear is the result of this functionality. And, of course, user stories are great for agile development process.

Developers would determine how much this function would cost (based on our knowledge of the tools we use, and the time it takes using those tools to generate this sort of functionality), and clients would know exactly what they are getting from a functionality standpoint. When this functionality is complete, everyone is happy. The developers get reasonable compensation for a job well done, and the clients get the mission-based functionality they asked for.

And it would avoid a situation which I have become recently far too familiar with – vendors who underbid projects, and then feel the need to resort not to the intent of the contract, but the letter. Everyone knows it is utterly impossible to specify every detail in the letter of a contract – sometimes letter of the contract, unfortunately, details things like fields and queries, not functionality. The letter of a contract will be, almost by definition unless based on functionality, an inadequate representation of the final result needed. In this case, no one really wins. The clients either don’t get the functionality they expected, or they pay extra for it, and they leave the project with a bad taste in their mouth about the vendors, which will only come around to hurt the vendors later.

All tools have their strengths and weaknesses. And, in addition, the best tool does nothing without good human project management skills using it. As a shop that practices Agile development (we use an adaptation of scrum methodology that seems to work for a shop that does multiple projects with small teams,) finding a good tool that facilitates instead of hobbles Agile was critical for us.

We found, and have chosen to use Redmine for our project management/ticketing system. You can think of it as a multi-project version of Trac, which is a fabulous ticketing/wiki system that we were initially going to go with. Redmine has the elements of Trac that we liked, with the added ability to track multiple projects. Like Basecamp, Redmine has document storage and messaging systems. It doesn’t have milestones per se, but it does allow you to see tasks in calendar and Gantt views, which is very helpful. Unlike Basecamp, you can add custom fields to tickets, users and other features. Having spent many hours in Basecamp, I actually like Redmine much better. It does even do time tracking, which we won’t use, but is nice to know is there. And the wiki is nice. Basecamp’s Writeboards seem so much more like an add on than integrated.

It’s a Ruby on Rails application, and that was actually kind of fun to finally get to install and play with RoR a tiny bit. And it’s great that it’s free and open source. Although that wasn’t an absolute requirement for us, it is most definitely a plus, given so much of our work is implementing open source web tools. And it’s nice to save a few bucks per month.

Human processes mold around software. We see this all the time. A CRM gives you these 5 canned reports, and you get used to making do with what’s there. A legacy client database requires a certain order of data entry, and your intake forms have been produced to copy that order. Your email software has particular limitations, and you find behavioral workarounds.

What’s also true in the realm of customized software, is that software is molded around people. You put in your RFP that a package spit out data in X,Y and Z ways because your ED is used to data in that form (maybe because a package they had at their previous organization had those canned reports.) You have a requirement that data be entered into the system in one particular way, probably because that’s the way you’ve always done it. Sometimes, you feel the need to replicate a process that the person 3 administrative assistants ago put in place that was molded around their particular limitations, just because that’s what you know.

When you are undergoing the process of creating or implementing a new system of any sort, whether it be a CMS for a website, a CRM, some internal system, it is a really good exercise to be like a 3 year-old, and keep asking “why?” Why do we need this feature? Why will this report be important? Why should the software work this way? Once you peel the layers down to the bottom, you’ll either have “we don’t know” or “because we believe it will help us meet our mission in this specific way.”  Then you know what you should take, and what you can leave behind.

As you know, I have built a long-time expertise in open source software and web applications, particularly Content Management Systems (CMS) and online database systems, including CRM. Thomas Groden, my new business partner, has expertise in Software-as-a-Service Constituent Relationship Management Systems (CRM), as well as much more broad expertise in technology infrastructure.

All technology implementors have to choose their tools (unless they run a very large shop) and we have decided to focus on implementation of both Salesforce.com and CiviCRM as CRMs, and Drupal as a CMS. We are keenly interested in building on our expertise to integrate these open platforms in really rich ways, to allow organizations to create great online applications.

I’m excited to be a part of a team – I’ve been a soloist for a while, and it’s nice to build collaborations, and work together with people with shared ideals on larger projects than I’d be able to take on alone. And I’m really excited by the set of technologies we’re working on, and the kinds of applications we’ll be building with these technologies.

And you can follow us on twitter.

One thing I learned is that it is incredibly sensitive to typos. One space accidentally inserted between the “<?” and the “php” led to a completely blank page. Ah well. I’m certainly learning what mistakes can lead to what kinds of issues, which is good. Eventually that becomes second nature.

But, in any event, by the end of an hour or so of hacking, I’d turned a template that I found online at Open Web Design into a Drupal template. I felt accomplished! I’m going to do a few more, and see how sophisticated with it I can get.

One thing I ran into (and haven’t been able to solve yet) is that it’s not easy to have navigation that requires more than just the standard <ul><li> tags. Adding <span> tags, for instance (which makes possible some more interesting looking navigation buttons) seems, at least at first, far from trivial.

I’m making a list of little(ish) projects that I want to do – sort of like problems I think I want to know how to solve.

• Drupal and google docs single sign on. There is already a SSO Module for Drupal 5.x, and someone submitted a patch for it, but it’s still up for review. I’d also have to cough up $50/year to get a google account that has the SSO API, but it might be worth it.
• Drupal sidebar connecting with the NPR API – perhaps to provide a targeted news stream?
• Doing a google map mashup of data in Drupal
• Working with getting flickr photostreams to show in Drupal

I’m still looking for a good project to try out in Cake. Unfortunately, the module Drake, which is meant to be a bridge allowing you to run Cake applications within Drupal, seems moribund. There is only a development snapshot for the 5.x branch, and no one seems to be picking it up for 6.x. Sigh. There is, for sure, another whole blog entry about Drupal modules.

The approaches are similar and different to each other. I set up both on my laptop, and tried out some really simple app development. In Cake, the database build is separate from the application building (you do it yourself), whereas in Symfony, you use Symfony to build the database with schema files written in YAML. Then, you build forms and such using the schema as a foundation.

They both use the MVC pattern, and both use object oriented PHP, which is great. I got a lot further with Cake in one evening of playing with both than I did with Symfony. At this point, I really prefer Cake – it feels like it jives with my own coding sensibilities better. I also don’t like the overhead of learning YAML. I can imagine, though, that the Symfony approach can be powerful.

Looking at Ohloh, Cake is more popular than Symfony (on Ohloh, who knows about in general), but Symfony has a lot more developers (81 vs 17). They both have good documentation and active communities.

For now, unless something strange happens, I’ll settle on Cake – although I’ll not be spending too much time on it, since I’m working hard to grok Drupal. But perhaps a cool project will manifest, and I can use it.

Update: I learned that Yahoo and delicious have a huge investment in Symfony (which, I guess, might be why they have so many more developers.)

• Jane, at FIO Partners, talks about how important it is to focus on the value that consultants provide to their clients.
• The Nonprofits Assistance Fund blog talks about how during uncertain times, organizations should think about moving into “turnaround mode.”

Somewhat tangentially related:

• Although they didn’t say this was related to the economy, Strategynut gives seven tips for building trust with your consulting clients.
• A Small Change blog talks about annual board solicitations.
• Donor Power blog asks whether brand kills fundraising? Answer: no.

I have been doing a lot of thinking myself about the economic meltdown, and what it means personally for me as a consultant. I’ve talked a little before about some changes I’m making, both because of personal interest as well as what I feel is changes in the wind in terms the kinds of new priorities that might be emerging.