Beth Kanter tweeted about an article by Gale Berkowitz relating to evaluation, which I found really fascinating – it is a must read. In this article, Gale points to an interesting shift within her organization (the Packard Foundation):
“Over the past four years we have been shifting from evaluation for proof or accountability (“Did the program work?”) to evaluation for program improvement (“What did we learn that can help us make the program better?”).”
In some ways, it’s a subtle shift – but as she says, the latter leads to “real time” evaluation – something that happens as one moves through projects, not just at the end.
Nonprofit organizations often have their feet to the fire to evaluate their programs and projects, because funders and contributors often demand proof that their programs work. And there has been an overall movement in the sector in the direction of increased evaluation and learning. The community I’m a part of, the group of for-profit (“for-little-profit” as is often said – we’re small and lean) companies that serve the technical needs of nonprofits, evaluation is generally not part of the process of the work we do. But it should be.
I’ve talked about this before. A lot. In a variety of different contexts. To me, evaluation, both internal (“how could we have done this process better?” “”How could we have worked together as a team better?”) as well as externally with the client (“How did we do?” “What could we have done better?” “How could we have communicated better?”) is a critical part of the work we do.
It’s a tough balance. We’re geeks, often busy deep in the command-line, SQL and code. We’re often extremely busy, juggling lots of projects and demands at once. The bottom line, of course, for us, is always a measure of how well we are doing, but I don’t think that’s enough. As our sector as a whole moves further and further along the path of a commitment to evaluation and learning, I think it behooves us to follow.
So, you ask, what are good strategies to start with? I can give you what we try to do. Some of it is well worked out, and some is nascent. All of these we aim to do, but it’s easy to miss the target. Evaluation is a learning process, like anything else, and the most important thing is an intention and commitment to being a learning organization. The rest will eventually follow over time.
- Spend time at the beginning of each project outlining evaluation steps and process for the project.
- Spend time at the end of every project asking internally “what worked, and what didn’t work?”
- Ask clients at the end of the project a set of questions about the process and the result.
- If its an ongoing engagement, ask periodically (we aim for every 6 months or so) for an evaluation meeting or call with the client.
- Write a report at the end of each project with lessons learned.
- When a proposal isn’t accepted, ask a few questions, both internally and externally, and write up a short report with lessons learned.
- Ask internally how earlier lessons learned, are being applied to current projects.
- Always be open to learning how to make things better.
Tagged as:
evaluation,
learning,
Nonprofits,
nptech
Some great tech and nonprofit tech stuff I’ve come across lately:
I’ve been a fan of user stories for several years now. User stories are a way to describe a set of functionalities of an application in a way that is focused on results – it’s easy to connect to mission. An example from an events management application:
The organization should be able to create several different kinds of events, and determine for each kind of event which detailed information will be taken. Those events can be displayed in a list or calendar format. Users can register for events, and pay using a credit card.
There are many ways to describe this story – it certainly can be a lot more detailed, but what’s clear is the result of this functionality. And, of course, user stories are great for agile development process.
Developers would determine how much this function would cost (based on our knowledge of the tools we use, and the time it takes using those tools to generate this sort of functionality), and clients would know exactly what they are getting from a functionality standpoint. When this functionality is complete, everyone is happy. The developers get reasonable compensation for a job well done, and the clients get the mission-based functionality they asked for.
And it would avoid a situation which I have become recently far too familiar with – vendors who underbid projects, and then feel the need to resort not to the intent of the contract, but the letter. Everyone knows it is utterly impossible to specify every detail in the letter of a contract – sometimes letter of the contract, unfortunately, details things like fields and queries, not functionality. The letter of a contract will be, almost by definition unless based on functionality, an inadequate representation of the final result needed. In this case, no one really wins. The clients either don’t get the functionality they expected, or they pay extra for it, and they leave the project with a bad taste in their mouth about the vendors, which will only come around to hurt the vendors later.
Tagged as:
Consulting,
Development,
nptech
The python community has started a conversation about diversity, with the ultimate goal of creating basically a welcoming statement. It comes out of Kirrily Robert’s keynote at OSCON about women and open source. There is a cool site from the Ruby community called Railsbridge, and one of their guidelines is to “Reach out to individuals and groups who are underrepresented in the community.”
There has been, of course, a lot said about the fact that although women make up 20% of the tech field, they only are approximately 1.5% of open source communities. There have been long standing groups that have tried to address this, and new efforts as well. Some open source communities are more diverse than others. In her keynote, Kirrily talks about two open source projects, Archive of Our Own and Dreamwidth that have a majority of women involved, which is rather unusual.
A short twitter conversation I had with a colleague brought up the issue of whether or not this is just an exercise – will this actually lead to any lasting change? That’s a good question.
Kirrily has a set of really good guidelines for open source communities:
- Recruit diversity
- Say it, mean it
- Tools (Tools are easy)
- Transparency
- Don’t Stare
- Value all contributions
- Call people on their crap
- Pay Attention
As a long time open source user and advocate, even though I am someone who rarely finds people like me in open source projects (i.e other women of color), I’ve always seen the open source movement a potential avenue for the greater involvement of people other than white, straight, young men, because theoretically (this is the important part) one’s involvement in a community is pure meritocracy. But so many open source communities have so far to go when it comes to being welcoming. I’m reminded of sitting in Drupalcon in DC and hearing Dries talk about the “beard length” of the developers. And of course there was the huge brou-ha-ha around a presentation at a recent Ruby conference.
And, of course, there are other factors as well. There are far too few places like The Community Software Lab of Lowell, MA, who’s mission is:
We write, administer and maintain open source software to serve the underserved.
We use and improve the skills of people with underused skills
We work to make hacker sub culture values (transparency, meritocracy and generosity) the values of the entire culture and bring about the post scarcity society.
We work toward our mission by trying to achieve our short term goals transparently and generously while accumulating only necessary wealth.
So what will it take? Will this effort in the python community pan out? I think it’s a great start. I think the first step is definitely a focus on community environment. Is it friendly? Is it welcoming? Is it easy for new developers to start, and get deeper in? Are there good mentoring models? All of that makes a huge difference. And having a statement doesn’t at all guarantee anything, but it provides something people can point to and say “this is our goal.” Better than nothing, and a lot better than many open source communities are doing.
Here at OpenIssue, we think a lot about the web. I mean, a LOT. And we’ve been thinking a lot about web hosting, and the varied flavors it comes in. We’re working to figure out what makes sense for us to use and implement, and what makes sense for us to recommend to our clients. A while ago, we decided, like many folks, virtual private servers were going to be the preferred hosting set up. Not that it’s right for all organizations – but for many who invest significant dollars into implementation of a website or CiviCRM, the advantages of a VPS will likely outweigh the higher monthly cost.
We started using Slicehost, which was incredibly easy to set up and use, and was acquired by Rackspace, which is considered the premium dedicated server hosting company. I then soon discovered a service called Cloudkick, which allowed us to monitor all of our slices and our clients slices in one dashboard. That was very cool.
It turns out that in the process of creating Cloudkick, the folks there came up with libcloud – a library that service providers could use to give developers access to the services needed by the servers – list, restart, create, destroy, etc. There are now a number of cloud hosting service providers, such as Rackspace cloud servers (used to be Mosso), Slicehost, and Amazon, that are beginning to support libcloud. Libcloud has become it’s own open source project, and is under active development.
Hopefully, this will provide a plethora of options for folks in terms of being able to monitor and manage the varied cloud servers they’ve got going. It certainly has already made our lives a lot easier.
Tagged as:
cloud,
hosting,
libcloud
Here’s a broad ranging list of interesting tidbits I’ve found recently.
Tagged as:
nptech
Any consulting shop that does significant amounts of implementation and development (as we do) needs a project management and ticketing tool. Basecamp seems to be a standard that many people have reached for. We were using Intervals for a while, which is really a fabulous tool if you do a lot of hourly consulting. We also have been using Google spreadsheets for some elements of project management.
All tools have their strengths and weaknesses. And, in addition, the best tool does nothing without good human project management skills using it. As a shop that practices Agile development (we use an adaptation of scrum methodology that seems to work for a shop that does multiple projects with small teams,) finding a good tool that facilitates instead of hobbles Agile was critical for us.
We found, and have chosen to use Redmine for our project management/ticketing system. You can think of it as a multi-project version of Trac, which is a fabulous ticketing/wiki system that we were initially going to go with. Redmine has the elements of Trac that we liked, with the added ability to track multiple projects. Like Basecamp, Redmine has document storage and messaging systems. It doesn’t have milestones per se, but it does allow you to see tasks in calendar and Gantt views, which is very helpful. Unlike Basecamp, you can add custom fields to tickets, users and other features. Having spent many hours in Basecamp, I actually like Redmine much better. It does even do time tracking, which we won’t use, but is nice to know is there. And the wiki is nice. Basecamp’s Writeboards seem so much more like an add on than integrated.
It’s a Ruby on Rails application, and that was actually kind of fun to finally get to install and play with RoR a tiny bit. And it’s great that it’s free and open source. Although that wasn’t an absolute requirement for us, it is most definitely a plus, given so much of our work is implementing open source web tools. And it’s nice to save a few bucks per month.
Tagged as:
nptech,
opensource,
tools,
web
I did a kind of radical experiment a couple of weeks ago: I de-friended almost all of my nptech and client Facebook friends (cutting my friend count by more than 60%). I had a few reasons for this, and over the past couple of weeks that I’ve been living this experiment, it’s made me quite happy. Of course, everyone is still on Twitter, and Linked in, etc., so I still feel connected.
Even though I tend not to blog anywhere near as much as most of my colleagues about social networks (because it’s really not my passion,) I’ve been a fairly early adopter, in the broad sense (of course, if I compare myself to Beth Kanter, I’m a laggard.) I have an account on all of the major social networks (and some of the obscure ones, too,) listen often, and update fairly regularly. A while ago, I realized that I would keep hearing the same nonprofit technology related stuff, over and over again, and I realized I was contributing to that by using Ping.fm to send the same status notices everywhere, or connecting my twitter account to my facebook and linked in accounts, etc. (actually, I think it might even be possible to create an infinite loop doing that stuff.) I stopped doing that a while back.
Now of course it used to be that all of my Facebook “friends” were other nptech early adopters. But around two years ago, a steady stream of my real friends started to come on, and then about 40% of my Facebook friends were non-nptech related. I noticed two important things: first, a status notice that a real friend was having a hard time would get buried in the cacophany of new reports, new campaigns, new blog posts, etc. Not a good thing. Also, I noticed that I censored myself on Facebook – I wouldn’t say things to friends, or play games, or take silly quizzes because I felt the need to be “professional.”
So all of that lead me to make Facebook a “work-free” space. I left work-related groups, disconnected this blog from Facebook, etc.
And doing that led me to think a little bit about how we nonprofit technology leaders use these social networks, and how we work with our clients to use these services. I do think that still, the majority of nonprofit organizations aren’t all that connected to social networks. I’m not entirely utterly convinced yet that all of them should. And I do wonder about the echo effect – if you are an early adopter, and you are on multiple networks, you are going to hear the same stuff over and over. Is that a good thing, or a bad thing? Should we be suggesting that organizations tailor much more specifically their messages, rather than using the services that allow them (and us) to send the same updates everywhere at once?
The technology behind social network strategy and implementation is way more my bad than communications strategy, but this experiment has opened my eyes to some of the things we may be doing wrong. And, of course, there is an entirely interesting conversation to be had about the issues of work and personal life, but I’ll save that for my other blog.
Tagged as:
facebook,
nptech
I spent a big chunk of my day dealing with a project that is, in no uncertain terms, a trainwreck. The client has sunk a ton of money into a product which is in, its current (first phase supposedly finished) state, unusable (client and vendor shall remain unnamed.) My role in the project has been strategic and as a liason, not technical, which to some extent gives me a bit of a distanced view.
Web development trainwrecks are, sadly, far from isolated cases – they happen all the time, even when all of the parties have good intentions. And as someone who is building a business around doing this sort of work, it is of keen interest to me as to why some projects end up in the state that this project is in, and I want to make sure to avoid these kinds of situations. So how do we avoid trainwrecks? Some trainwrecks we can see coming miles away, but yet we are in complete denial about them. Some trainwrecks are like sudden derailments – it’s not at all clear where it comes from. But I think all trainwreck projects have the seed of the wreck somewhere in the history of the project.
The hallmarks of this particular trainwreck were so clear, that in retrospect, they scream out at me:
- Lack of transparency about development process
- Lack of transparency about cost implications of increased scope
- Waterfall development process (well, the vendor said they practiced Agile, but in practice, it’s been waterfall)
As a practitioner of the Agile development process (we use a somewhat modified form of Scrum, in particular,) I’m beginning to really see the value of this kind of process. It makes visible all sorts of things that are often hidden. It seems like the Agile methodology helps in a number of ways:
- Once educated, clients have a window into the development process. They know what small chunks of development are going to happen in a given time interval, and they know what they will get at the end of that time interval
- Things are developed in priority order
- Clients can critique things early
- New functionality becomes a part of the “product backlog” and it is easier to have clarity about what is and is not within scope
Of course, it is theoretically possible to be completely transparent in a traditional waterfall methodology, and completely opaque using Agile, but I do think that the Agile methodology makes it way more difficult to be opaque. But it also takes some work and education of clients unfamiliar with the methodology (as well as making mistakes along the way on our part as developers.)
And I’ve been able to watch this process work well, not only with our own projects, but also with a project I was a strategic lead on. I was pretty skeptical a year or so ago, but now I’m sold. And since transparency has always been something of real importance to me, a development process that encourages transparency is a good thing.
Tagged as:
agile,
Development,
nptech,
web
As you can tell, I haven’t had much time to blog lately. Here are some great links I’ve come across that I thought were worth sharing:
Tagged as:
nptech,
opensource