Now that the Idealware CMS report is out, I get to have my say about it. Here’s the first post, there might be more to come.
The thing that is prompting this post is the little storm about the security metric that we used to try and get a handle on the security of the 4 different systems we reviewed. More on that in a bit.
You might think that comparing four different open source packages that, in essence, do pretty the same thing (in a broad sense) would be a cakc walk. In fact, nothing could be farther from the truth. The developers of each project have completely different sets of assumptions about what the right way to do things is, and completely different philosophies and ethos when it comes to building interfaces and functionality. Making apples-to-apples comparisons of these systems was one of the most difficult analytical tasks I’ve taken on in a while (and, actually much of the heavy lifting of designing the analysis was done by Laura Quinn), and until you attempt such a thing, please be somewhat tempered in your complaints about it.
Now the security issue. One of the 12 different aspects we are comparing is “Scalability and security”. The report isn’t about security, it’s a very, very broad comparison of the systems, with security as a very small component. That’s just the context. Two (yes, just two) questions out of many relate to security. First, a simple metric relating to security reports, and second, what processes are in place in the communities to deal with security. This wasn’t designed to be an in-dept, complex analysis of security. If it had been, we would have done a lot more work on how to measure security. On the Four Kitchens blog, they say, “While both reports above seem to identify Drupal (and Joomla! and WordPress, to be fair) as having notably bad [emphasis mine] security, they’re also both based on one superficial metric: self-reported vulnerabilities.” Now I can’t speak about the IBM report (I haven’t even read it yet), but our report says no such thing. Drupal gets a “Solid” on Scalability and security. Solid, which is only one step below Excellent. And you know why it got a “Solid”? Because, indeed, it does have more reported security vulnerabilities than Plone (as do Joomla and WordPress.)
David Geilhufe, who also takes issue with the security metric, has some good points. Yes, sheer numbers of vulnerabilities are not anywhere near the best metric of whether or not a system is secure or not. As a quick comparative look between a small number of open source systems, it’s hard to argue that it contributes no information. Four Kitchens seems to suggest that part of the reason for more vulnerabilities in Drupal compared to Plone is that it’s more popular. But, if you’ve been an observer to the Linux/Windows FUD wars, you’ll remember that Microsoft has that exact same argument about why there are more security vulnerabilities in Windows as compared to Linux. And the Linux folks say, in response, “It’s not popularity, it’s design.” I’m sure that Four Kitchens, and most open source software developers agree with that perspective. In reviewing Plone, and talking with people who develop for Plone, I was convinced that the reason that Plone had fewer reported vulnerabilities was not just because it was less popular – it’s because it (and Python and Zope) was more secure by design.
I am completely happy with Drupal’s security (otherwise, it wouldn’t have gotten a “Solid.”) I think the Drupal community takes security extremely seriously, and if they didn’t, I wouldn’t have chosen it as a platform for development. I also think that the Joomla and WordPress communities take security seriously. In our estimation, they were all really good. But Plone was just that much better.
{ 1 trackback }
{ 7 comments }
Now, before I address the arguments you hand-picked for responses, I’ll reaffirm my arguments about the obvious conflict of interest surrounding self-reporting and the lack of data showing past (and fixed) vulnerability reports correlate in any way with future security breaches. I can only assume you concede these points. You also ignore my argument about the 2008 security audits of Drupal that resulted in numerous vulnerability reports, but more on that later.
“[...] but our report says no such thing [...]”
In the context of your report, I probably could have said “notably worse.” The IBM report specifically puts Drupal on a top 10 list for its vulnerability reports, so “bad” applies there. Also, when the range of your security evaluations extends from “good” to “excellent” without characterizing what’s worse than “good,” I don’t have any alternative but to treat “good” as the worst on your spectrum. Saying “everyone gets an ‘A’!” does not get you a free pass from criticism of your methodology.
“But, if you’ve been an observer to the Linux/Windows FUD wars, you’ll remember that Microsoft has that exact same argument about why there are more security vulnerabilities in Windows as compared to Linux.”
That’s an absurd mischaracterization. Microsoft’s argument is that more people choose to target and exploit Windows because the larger number of deployments provided a more attractive payload. My argument is that more eyes on Drupal’s code and more people performing formal security audits causes Drupal to uncover a greater percentage of issues.
“In reviewing Plone, and talking with people who develop for Plone, I was convinced that the reason that Plone had fewer reported vulnerabilities was not just because it was less popular – it’s because it (and Python and Zope) was more secure by design.”
Naturally, you’re now expected to say why. It’s simply not enough to say “Plone feels more secure” without giving something concrete.
I’ll conclude here the same way I did in my post at Four Kitchens: the absence of good metrics does not excuse the use of misleading ones. If you can’t publish a responsible security evaluation, then don’t publish one.
First off, you can feel free to say that Drupal is “notably worse” in security than Plone, while I would choose to say Plone is “notably better.” But I guess that’s a glass 1/2 empty/full sort of thing.
Of course, I totally agree with you that more eyeballs on code leads to more vulnerabilities shown. But in this case, that’s not the entire issue at all. Joomla had by far the most vulnerabilities reported, and I’m not sure I’d argue that there are a lot more eyeballs on Joomla code than on Drupal code. (WP and Drupal had about the same number.)
Anyway, the metrics were not meant to be misleading, and they weren’t meant as a measure of the security of a particular system – they were meant as a means of comparison. One of the more difficult things in writing this report (for me) was that the structure of the report demanded somewhat short, abbreviated conclusions, and there was in fact a lot of detail that didn’t make it into the report.
And I did not say Plone “feels” more secure – that’s insulting.
You want details? Here are some:
- Because Plone does not use MySQL (or a SQL-based backend) it is immune to SQL injection attacks.
- While Drupal’s ACL and roles are quite good (quite better than WP and Joomla) Zope’s additional capacity, and it’s much more granular workflow capacity provides a bit more protection against possible vulnerabilities.
- Plone uses SHA1 to store passwords, the others use MD5. I think the consensus now is SHA1 is more secure.
- In general, Plone comes more locked-down after installation than Drupal.
- I could also talk about PHP vs. Python and security, but that might lead to a holy war.
Most of the programming problems are misuse of the well documented abstract layers. Even though the documents online explains it well, people still use the T() and the DB_QUERY() functions incorrectly. That would eliminate most injection attacks like XSS, SQL injection, and CDRF.
Most of the administrative problems are: 1) using an outdated versions, 2) using a module before it is “tried and true”, 3) using the PHP/FullHTML input filters that specifically allows for dangerous output code, and 4) not correctly setting permissions for the file system and database grants. Issue #1 and #3 are more difficult in Drupal6 and 7, #2 involves someone ignoring the well displayed version numbers, and #4 is beyond Drupal’s realm of influence.
I have more details which I will put on my own site soon.
Instead of going for the “Drupal is so popular, all bugs are shallow, that’s why we have so many security holes” rhetoric, I’d suggest addressing the list of the 10 most common security vulnerabilities in web applications from OWASP. It’s a good checklist that lists the most common attack vectors for web applications these days. If the PHP-based projects (not just Drupal :) can show how they address these, they are on their way to show that they take security seriously.
Plone’s version is here: http://plone.org/products/plone/security/overview
Please note that I’m not trying to stir up this issue, Plone has chosen to focus on security, which of course comes with its own tradeoffs — the “why can’t untrusted visitors embed a YouTube video in my site” being the most common complaint.
Dave Keays wrote:
A good observation. That’s why strong security really needs to be implemented on the application-server level and not just on the application (in this case CMS) or plug-in level. Add-on developers will always be less sophisticated than core developers. They need a stack that makes it very hard to make the most common mistakes. Though I think Plone’s developers are conscientious about security, it’s really Zope’s security model that keeps add-on developers from routinely shooting everyone in the foot.
Well, Michelle, you know I love you, and…
First, I must say I am not without sin and am not casting the first stone. Doing such a comparison is brutally hard and you guys did a great job. At the same time you made, IMHO, a very critical mistake by teaching people something about security that just isn’t useful.
“itâs hard to argue that it contributes no information”
Mmmmm… for a project with zero reported security vulnerabilities:
Does it contribute the information that the system has fewer security vulnerabilities than other systems?
Does it contribute the information that fewer people are looking for security vulnerabilities?
Does it contribute the information that the the project does not report their security vulnerabilities?
In the end David is right… the absence of a good metric does not excuse the publication of a bad one.
And finally, I must say I agree with the conclusion that Plone is “more” secure than Drupal by design, however, the idealware audience needs to understand that they don’t buy a secure CMS (which the metric implies)… I can install plone or drupal or hire a hack to do it and the systems will be equally INSECURE.
Actually, Plone didn’t have zero – it had two.
Your points are well taken, David. And I totally agree that the implementation matters more than the project itself.
Comments on this entry are closed.