Now that the Idealware CMS report is out, I get to have my say about it. Here’s the first post, there might be more to come.
The thing that is prompting this post is the little storm about the security metric that we used to try and get a handle on the security of the 4 different systems we reviewed. More on that in a bit.
You might think that comparing four different open source packages that, in essence, do pretty the same thing (in a broad sense) would be a cakc walk. In fact, nothing could be farther from the truth. The developers of each project have completely different sets of assumptions about what the right way to do things is, and completely different philosophies and ethos when it comes to building interfaces and functionality. Making apples-to-apples comparisons of these systems was one of the most difficult analytical tasks I’ve taken on in a while (and, actually much of the heavy lifting of designing the analysis was done by Laura Quinn), and until you attempt such a thing, please be somewhat tempered in your complaints about it.
Now the security issue. One of the 12 different aspects we are comparing is “Scalability and security”. The report isn’t about security, it’s a very, very broad comparison of the systems, with security as a very small component. That’s just the context. Two (yes, just two) questions out of many relate to security. First, a simple metric relating to security reports, and second, what processes are in place in the communities to deal with security. This wasn’t designed to be an in-dept, complex analysis of security. If it had been, we would have done a lot more work on how to measure security. On the Four Kitchens blog, they say, “While both reports above seem to identify Drupal (and Joomla! and WordPress, to be fair) as having notably bad [emphasis mine] security, they’re also both based on one superficial metric: self-reported vulnerabilities.” Now I can’t speak about the IBM report (I haven’t even read it yet), but our report says no such thing. Drupal gets a “Solid” on Scalability and security. Solid, which is only one step below Excellent. And you know why it got a “Solid”? Because, indeed, it does have more reported security vulnerabilities than Plone (as do Joomla and WordPress.)
David Geilhufe, who also takes issue with the security metric, has some good points. Yes, sheer numbers of vulnerabilities are not anywhere near the best metric of whether or not a system is secure or not. As a quick comparative look between a small number of open source systems, it’s hard to argue that it contributes no information. Four Kitchens seems to suggest that part of the reason for more vulnerabilities in Drupal compared to Plone is that it’s more popular. But, if you’ve been an observer to the Linux/Windows FUD wars, you’ll remember that Microsoft has that exact same argument about why there are more security vulnerabilities in Windows as compared to Linux. And the Linux folks say, in response, “It’s not popularity, it’s design.” I’m sure that Four Kitchens, and most open source software developers agree with that perspective. In reviewing Plone, and talking with people who develop for Plone, I was convinced that the reason that Plone had fewer reported vulnerabilities was not just because it was less popular – it’s because it (and Python and Zope) was more secure by design.
I am completely happy with Drupal’s security (otherwise, it wouldn’t have gotten a “Solid.”) I think the Drupal community takes security extremely seriously, and if they didn’t, I wouldn’t have chosen it as a platform for development. I also think that the Joomla and WordPress communities take security seriously. In our estimation, they were all really good. But Plone was just that much better.
{ 8 comments }
(from