Overview As part of the new product development for the fall of 2009, a new external site is to be developed.  One of the goals is to have the external application be a separate application than that of the internal site…….

Your points are well taken, David. And I totally agree that the implementation matters more than the project itself.

First, I must say I am not without sin and am not casting the first stone. Doing such a comparison is brutally hard and you guys did a great job. At the same time you made, IMHO, a very critical mistake by teaching people something about security that just isn’t useful.

“it’s hard to argue that it contributes no information”

Mmmmm… for a project with zero reported security vulnerabilities:
Does it contribute the information that the system has fewer security vulnerabilities than other systems?
Does it contribute the information that fewer people are looking for security vulnerabilities?
Does it contribute the information that the the project does not report their security vulnerabilities?

In the end David is right… the absence of a good metric does not excuse the publication of a bad one.

And finally, I must say I agree with the conclusion that Plone is “more” secure than Drupal by design, however, the idealware audience needs to understand that they don’t buy a secure CMS (which the metric implies)… I can install plone or drupal or hire a hack to do it and the systems will be equally INSECURE.

Most of the programming problems are misuse of the well documented abstract layers. Even though the documents online explains it well, people still use the T() and the DB_QUERY() functions incorrectly. That would eliminate most injection attacks like XSS, SQL injection, and CDRF.

A good observation. That’s why strong security really needs to be implemented on the application-server level and not just on the application (in this case CMS) or plug-in level. Add-on developers will always be less sophisticated than core developers. They need a stack that makes it very hard to make the most common mistakes. Though I think Plone’s developers are conscientious about security, it’s really Zope’s security model that keeps add-on developers from routinely shooting everyone in the foot.

Plone’s version is here: http://plone.org/products/plone/security/overview

Please note that I’m not trying to stir up this issue, Plone has chosen to focus on security, which of course comes with its own tradeoffs — the “why can’t untrusted visitors embed a YouTube video in my site” being the most common complaint.

Most of the administrative problems are: 1) using an outdated versions, 2) using a module before it is “tried and true”, 3) using the PHP/FullHTML input filters that specifically allows for dangerous output code, and 4) not correctly setting permissions for the file system and database grants. Issue #1 and #3 are more difficult in Drupal6 and 7, #2 involves someone ignoring the well displayed version numbers, and #4 is beyond Drupal’s realm of influence.

I have more details which I will put on my own site soon.

Of course, I totally agree with you that more eyeballs on code leads to more vulnerabilities shown. But in this case, that’s not the entire issue at all. Joomla had by far the most vulnerabilities reported, and I’m not sure I’d argue that there are a lot more eyeballs on Joomla code than on Drupal code. (WP and Drupal had about the same number.)

Anyway, the metrics were not meant to be misleading, and they weren’t meant as a measure of the security of a particular system – they were meant as a means of comparison. One of the more difficult things in writing this report (for me) was that the structure of the report demanded somewhat short, abbreviated conclusions, and there was in fact a lot of detail that didn’t make it into the report.

And I did not say Plone “feels” more secure – that’s insulting.

You want details? Here are some:

- Because Plone does not use MySQL (or a SQL-based backend) it is immune to SQL injection attacks.
- While Drupal’s ACL and roles are quite good (quite better than WP and Joomla) Zope’s additional capacity, and it’s much more granular workflow capacity provides a bit more protection against possible vulnerabilities.
- Plone uses SHA1 to store passwords, the others use MD5. I think the consensus now is SHA1 is more secure.
- In general, Plone comes more locked-down after installation than Drupal.
- I could also talk about PHP vs. Python and security, but that might lead to a holy war.

“[...] but our report says no such thing [...]”

In the context of your report, I probably could have said “notably worse.” The IBM report specifically puts Drupal on a top 10 list for its vulnerability reports, so “bad” applies there. Also, when the range of your security evaluations extends from “good” to “excellent” without characterizing what’s worse than “good,” I don’t have any alternative but to treat “good” as the worst on your spectrum. Saying “everyone gets an ‘A’!” does not get you a free pass from criticism of your methodology.

“But, if you’ve been an observer to the Linux/Windows FUD wars, you’ll remember that Microsoft has that exact same argument about why there are more security vulnerabilities in Windows as compared to Linux.”

That’s an absurd mischaracterization. Microsoft’s argument is that more people choose to target and exploit Windows because the larger number of deployments provided a more attractive payload. My argument is that more eyes on Drupal’s code and more people performing formal security audits causes Drupal to uncover a greater percentage of issues.

“In reviewing Plone, and talking with people who develop for Plone, I was convinced that the reason that Plone had fewer reported vulnerabilities was not just because it was less popular – it’s because it (and Python and Zope) was more secure by design.”

Naturally, you’re now expected to say why. It’s simply not enough to say “Plone feels more secure” without giving something concrete.

I’ll conclude here the same way I did in my post at Four Kitchens: the absence of good metrics does not excuse the use of misleading ones. If you can’t publish a responsible security evaluation, then don’t publish one.