You searched for:

evaluation

Evaluation and being a learning organization

September 12, 2009

Beth Kanter tweeted about an article by Gale Berkowitz relating to evaluation, which I found really fascinating – it is a must read. In this article, Gale points to an interesting shift within her organization (the Packard Foundation):

“Over the past four years we have been shifting from evaluation for proof or accountability (“Did the program work?”) to evaluation for program improvement (“What did we learn that can help us make the program better?”).”

In some ways, it’s a subtle shift – but as she says, the latter leads to “real time” evaluation – something that happens as one moves through projects, not just at the end.

Nonprofit organizations often have their feet to the fire to evaluate their programs and projects, because funders and contributors often demand proof that their programs work. And there has been an overall movement in the sector in the direction of increased evaluation and learning.  The community I’m a part of, the group of for-profit (“for-little-profit” as is often said – we’re small and lean)  companies that serve the technical needs of nonprofits, evaluation is generally not part of the process of the work we do. But it should be.

I’ve talked about this before. A lot. In a variety of different contexts. To me, evaluation, both internal (“how could we have done this process better?” “”How could we have worked together as a team better?”) as well as externally with the client (“How did we do?” “What could we have done better?” “How could we have communicated better?”) is a critical part of the work we do.

It’s a tough balance. We’re geeks, often busy deep in the command-line, SQL and code. We’re often extremely busy, juggling lots of projects and demands at once. The bottom line, of course, for us, is always a measure of how well we are doing, but I don’t think that’s enough. As our sector as a whole moves further and further along the path of a commitment to evaluation and learning, I think it behooves us to follow.

So, you ask, what are good strategies to start with? I can give you what we try to do. Some of it is well worked out, and some is nascent. All of these we aim to do, but it’s easy to miss the target. Evaluation is a learning process, like anything else, and the most important thing is an intention and commitment to being a learning organization. The rest will eventually follow over time.

  1. Spend time at the beginning of each project outlining evaluation steps and process for the project.
  2. Spend time at the end of every project asking internally “what worked, and what didn’t work?”
  3. Ask clients at the end of the project a set of questions about the process and the result.
  4. If its an ongoing engagement, ask periodically (we aim for every 6 months or so) for an evaluation meeting or call with the client.
  5. Write a report at the end of each project with lessons learned.
  6. When a proposal isn’t accepted, ask a few questions, both internally and externally, and write up a short report with lessons learned.
  7. Ask internally how earlier lessons learned, are being applied to current projects.
  8. Always be open to learning how to make things better.

Tagged as: , , ,

{ 3 comments }

My Theory of Practice

July 10, 2008

I finally had the reason to begin to more completely articulate my theory of practice. My theory of practice is different than my consulting philosophy. They certainly are consistent with each other, but they are distinct. A theory of practice, in my mind, outlines the methods and ideals behind how I get work done with clients. This theory includes the following elements that I think are key to my work:

  • Listening. Listening, both at the beginning, and consistently through an engagement, to their goals, ideals, “points of pain”, and points of confusion.
  • Educating. One of the most important roles I play is educating clients about the technology that they will be engaging with, based upon what I’ve heard while I’ve listened. This is also an ongoing process.
  • Intermediation. The role I play most often currently is providing a clear and understandable avenue between the client and a technology vendor (such as web or database development shop). The client is quite knowledgeable about their organization, mission, and goals for a project, but often not knowledgeable about technology. The vendor is expert at what they do, but cannot always provide a channel of communication that the client can really work with. I provide that clear channel, so both sides benefit.
  • Learning. Those first three elements make up the communication arm of my practice. The other arm is learning. I can’t do what I do without being a technology expert. And I can’t stay a technology expert without continually learning. Reading, research, collaborating with others, getting my hands dirty with servers and code, playing with new applications and new APIs – all of those things keep my technology expertise fresh.

More specifically, what methods do I use to help clients make their way through the entire process of a technology project:

  • Qualitative and Quantitative (where appropriate) assessment of requirements and needs, including surveys and interviews with internal (and/or external) stakeholders
  • Research – both standard internet research as well as outreach and interviews with relevant people
  • Writing – writing requirements, RFPs, documentation
  • Project management – keeping a project on track
  • Evaluation – evaluating projects as they are happening, and when they are done.

{ 2 comments }

NPTECH Punk

June 19, 2008

Beth, of course, suggested this, and I’m jumping on her bandwagon. I realized, in being introduced to Edupunk, that I have been doing it for, oh, almost 20 years now.

In 1989, I joined the faculty of Hampshire College (and stayed for 10 years). Hampshire’s motto is “Non Satis Scire” – to know is not enough. From their website:

Some of the features that distinguish Hampshire from more traditional liberal arts colleges include student-designed academic concentrations; an active, collaborative, inquiry-based pedagogy; an interdisciplinary curriculum; and a narrative evaluation system.

Sounds a lot like Edupunk, doesn’t it?

But in the nonprofit realm, my perspective on helping nonprofit organizations with technology issues has a lot to do with client empowerment, learning based on what’s needed at the moment, and active collaboration.

I got a chance to test this out in a more orchestrated way (as opposed to the usual consultant/client interactions) when I facilitated/taught an OpenOffice.org “untraining” earlier this month at Google HQ in NYC (some more details are on the Google Blog.) I learned a lot. The unconference/camp model of learning about technology issues is really great, but falls a little short when dealing with a specific tool, and an audience that is mostly unfamiliar with it. So the model that I am coming up with is a combination of that model, and what I would call an “inquiry based” model – helping people in a more structured way come up with specific questions and problems before the event, and then use the event to collaboratively answer those questions, and solve those problems. The questions and problems are generated exactly from the needs of the participants – what do they need to do?

Anyway, I do hope at some point to have a chance to do this kind of thing again. And I think it would be great to have an nptechpunk mini movement!

{ 2 comments }

Reflection and Evaluation

March 10, 2008

Michele Martin, one of my fave bloggers, has a great post today on Reflective Practice. Both reflective practice – that is the process of reflecting on what you do, and how you do it, as well as conscious, deliberate evaluation of projects, are things that are not very common in our field, nor things that are valued or encouraged.

In many ways, we are focused on solving technology problems, or completing projects.  But I have really come to believe that the way that we work with people is as important as the “final” outcome. We might be able to build the most wizz-bang amazing website ever (in a technological sense) but if we haven’t really thought about how we moved through the project, never evaluated how the project really went, and didn’t learn from the process, in the end, the project wasn’t the success it seemed to be. In fact, it’s amazing how much we can learn from projects that might be considered failures by technological criteria.

In the last few months, I was involved in helping three organizations choose vendors for varied technology projects, and in the course of that time, I talked with almost a dozen technology vendors of one type or another. One question I asked all of them was about whether they had a process of reflection and evaluation of their work, as it was going on, and when the project was coming to a close. Unfortunately, none of them had an answer to that question. That is something I would love to see change.

{ 2 comments }

Circuit Rider School

June 11, 2007

Way back when (last month – I’ve been busy) Deborah Finn blogged about the "New England School for Circuit Riders." That blog entry came about because she and I had a long conversation about what kinds of skills nonprofit technology providers needed, and what we felt was missing.

Hot on the heels of that (OK, not so hot – about 3 weeks later) I had a great conversation with my old colleague Marc Osten about some work Lasa is doing around providing support to technology providers in their neck of the woods (that’d be the UK.)

I realize that we’ve been having this conversation ever since NTC used to be called the "Circuit Rider Roundup." Its not that there is a lack of technology vendors and support. It’s that there is a lack of really good support – responsive, empowering, educational, integrative, and knowledgeable about, and invested in, the sector.

For those of us who’d like to see organizations get better support – how do we do that? I think part of the answer has to be to provide the resources for people to become better providers – whether it be to help budding accidental techies get off the ground to become great IT staff or independent consultants, or helping individual and small consulting firms learn what makes really good nonprofit support.

There are many challenges – how do you teach self-reflection and self-evaluation? How do you teach the ins and outs of the nonprofit sector? How do you get providers to invest time and energy in what is really a marginally profitable business?

I don’t have too many answers today, but living inside the questions for a while is always a good start.

{ 0 comments }

Technology Support as Teaching

April 24, 2007

I’ve been thinking a lot about technology support lately. Really a lot. Part of it is being prompted by my own technology support experiences with my satellite “broadband” provider (which have been largely frustrating). A lot of it has been because I have lately been exposed to situations where I have felt organizations haven’t gotten the support they need, which, in our world, I think is all too common. As I move out of doing implementation, and into more evaluation, planning and facilitation of technology change within organizations, I wanted to spend some time articulating what I have tried my best to practice when I’ve been in a place of providing technology support.

All technology providers have to deal at some level with support. Whether they implement a system, or build it, they will inevitably have the job of supporting that technology. Providers have many different ways of handling that challenge. Unfortunately, the most recent trend, which I have experienced all too much (and I’m sure you all have too), is to simply follow a script with the person who needs support. It drives me simply nuts that every single time I call my satellite provider about a problem with the service, and I’m saying “I’m seeing 80% packet loss, and doing a traceroute suggests that it’s about 2 hops after your modem” and they respond with “OK, first, we’re going to clear out your browser cache. Go to preferences …” It has been a challenge to resist uttering strings of obscenities.

But also, the question is – is providing technology support simply just an end in itself, or is it also a means to another end – that is, can it be a means to empower clients in appropriate technology use to further their mission?

I realized, in thinking about all of this, that the model of technology support that makes the most sense to me is to think of it similarly as a teacher-student relationship. I know, I’m a born educator, and I’m sure someone out there is saying “if you have a hammer, every problem looks like a nail…” But I do think there is some validity to this approach. Certainly, if you are a technology provider that values empowerment of your clients, this is probably a good model to consider.

So what is it about a teacher-student relationship that we can learn from to provide really good technical support? From my perspective, there are four elements to a technology support process with this as a model:

  1. Assessment – where is the client – both in terms of technology knowledge, as well as in terms of what they need at the moment?
  2. Empowerment – as you help them with a problem, teach them about the problem, and ways to troubleshoot (or possibly solve) the problem themselves in the future.
  3. Relationship – an ongoing relationship with the client
  4. Solution – providing the solution to their problem

First, Assessment. Where is this client, now? First, there is the question of what they know. If you have a relationship with them (see #3) you’ll already be familiar with their technical expertise – so you’ll know where to start. But there is more than that to assessment. What is going on for them? Is this a problem that is critical to their work, or a “pebble in the shoe” kind of problem – annoying, but not urgent? Are they trying to get a grant out, and they are scared they won’t meet the deadline because of a technical issue? Are they angry? All of these are important to know and understand, so it’s possible to meet them where they are. That’s one of the hallmarks of a good educator – meeting a student where they are, tailoring the education to meet the needs of the student. It’s also, I think, a hallmark of a good provider of technology support.

Second, Empowerment. One of the most common problems that someone who has built websites has, is the client calls up, and says “the website is down”. And you hurriedly go to your browser, and, voila, the website isn’t down. So now you take them through all of the steps to figure out why it was they can’t see their own website. You can choose to take them through this problem so that they figure out at the moment what’s up, and who to call, or you can take them through it so that next time it happens, they won’t need to call you, because they’ve figured out the problem really belongs to “insert_some_other_technology_provider_here.” Or, they’ll call you because the website really is down. Teaching them about the technology behind the problem they are having, and helping them to understand what’s involved in it, not only empowers them to deal with problems more on their own, but it also empowers them to solve other technology problems, and be more engaged in technology planning in the future.

Third, Relationship. All of this works within whatever relationship you have with a client. As mentioned above, if you’ve worked consistently with a client, you know what their level of expertise is – this makes assessment easier. Also, you remember how much work you got done when a substitute teacher came to class? Not a lot of learning, but certainly a lot of spitballs. Consistency in relationship is as important to students as it is to people who get support from a technology provider. Usually, of course, with the huge technology providers, that sort of thing isn’t possible. But with smaller providers it certainly is. Sometimes, even with larger providers, they manage to get around this by having detailed logs of conversations with you. I’ve found that quite helpful in the past – it has surprised me when someone has said something like “I see you called a couple of months ago with a problem regarding x. How has that worked for you since then?” It was nice to feel like someone actually bothered to write it down, and for the person talking with me bothered to read it. In the past, for me, my ongoing support relationships with clients have been the way that I have learned the most about their organizations. It has allowed me to be proactive in working with them on technology, and incredibly informative in helping future planning. The relationship is a two-way street: just as they let us know about challenges they face with their technology problems – it’s important for us to tell them about the challenges that we run into in working to support them. There is a level of trust that’s important to this relationship. Honestly, it is the relationship I cherish most highly (even more highly than whatever they pay me).

Fourth, Solution. This is where the provider-client relationship differs most from the teacher-student relationship. Of course, in the end, the client needs their technology problem solved, as quickly and efficiently as possible. But I’d argue that good assessment of where the client is, and where the problem fits in their work and organizational lives, empowerment of them to troubleshoot problems on their own, and an ongoing, stable relationship, will make the eventual solution of the problem a lot easier, more economical and less stressful for both the client and the provider than it might be otherwise.

Technorati Tags: ,

{ 2 comments }

Speaking too soon

April 15, 2007

I’ve been doing a lot of thinking since I wrote my post, a few weeks ago, saying I was done with technology consulting. In one sense, I spoke too soon, although in another, I was right on. And, to some extent, this post is a bit self-indulgent, so if you’re looking for some concrete technology talk, you might want to wait for the next post on Joomla. :-)

I first started doing technology consulting for nonprofit organizations in 1996, with a project for a local public television station (WGBY in Springfield, MA), to design a technology center for teachers to learn about technology and the internet, so they could apply that in their classrooms. It was a great project, and a success, since that technology center is still in operation today. Understandably, it has come to be somewhat different than I designed it back then, but it still feels good that something that I worked hard on is still serving people. And it was the sheer enjoyment of that project – of talking to many different people about needs and desires, thinking about how to appropriately use technology to those ends, that got me out of academia, and into the nonprofit and educational technology world.

I did a lot of planning, evaluation and training in the beginning – some on my own, some with Summit Collaborative. it was what I enjoyed most, and it was what I thought I was best at. But, somewhere along the line, I started to do more and more implementation, because, honestly, that was what my clients needed most at the time. I put in a few networks in the late ’90s (ugh, really, I pulled cable.) I started to do databases for organizations, and then, in 1999, I flew headlong into web application development, which became my specialty and mainstay until I took a break to go to seminary in 2005. At first, I liked it a lot. I liked being able to create things that I thought my clients wanted (and they thought they wanted.) I stumbled a fair bit along the way. I had a hard time being a successful business owner with employees (I pretty much suck at it, so I hitched my wagon to Database Designs Associates from 2003 until this year.) And I struggled mightily with my own capacity to build really good applications mostly without other developers to help out. It was really hard to try and write new applications building on a framework I’d written a while ago, while simultaneously improving that framework, and keeping up with new things such as Ajax and RSS, mostly by myself. It just wasn’t happening very well.

And as time wore on, I lost touch with people and organizations. I sat for hours (or days) at a time in front of my screen without contact with the folks I was doing the work for. And, if there was contact, it was most often on the level of “can you fix this?” “can you add this feature?” I don’t blame them – they needed the fixes, and the features. But that was a pale shadow of the kind of work and contact I wanted with my clients. And I also struggled with the consulting business model. In the early days, as a business owner, I needed to think a lot about sustaining business (I had employees, and I wanted them to eat.) And later, even though it wasn’t a large part of my job description, it still was something that I had a hard time with – like getting yanked out of my flow to answer RFPs.

For one long time client (I had this client for just about all of the span of my consulting career – they were my second client), I had a much fuller, richer role, even though much of the work I did for them was database and web application development, we’d built a great rapport over time, and it felt wonderful when I got the chance to talk with them about bigger picture issues. But that was not so often, and, as staff in that organization left over time, that relationship changed.

When I came back from seminary, I was very clear that I couldn’t do technology consulting in the way that I had come to do it. I couldn’t bring myself to code or design databases, or write connections to APIs, or do any of those things that had become my bread and butter over the past 6 years. I wanted to work directly with organizations and people. So, it seemed to me that I needed simply to leave technology consulting behind, and move into doing things in a more spiritual vein, perhaps.

But then, I had something of an epiphany. And that epiphany was in my post about “Technology Consulting 2.0.” And the more I thought about it, the more it made sense to me, and the more I liked it. And the more I talked with other people about it, the more it made sense for me to do it. I will hold off for a while yet in my life working with people directly on spiritual issues, and work now with what could certainly be called the spirituality of nonprofit technology – finding balance and looking at the bigger picture. I’m creating a new practice, called MetaCentric Techology Advising. It will include visioning and planning, evaluation and training. All of the stuff that I liked the most about nonprofit technology, and, honestly, what I’m probably best at. And it’s nice to know that all of the last 8 years or so as a “technology vendor” as it were, will be there as good experience and guidance as I work with clients.

I won’t talk much about it in this blog again, but I thought it might be something people would want to hear about.

Technorati Tags: ,

{ 4 comments }

NTC Summary, and Nonprofit Technology Consulting 2.0

April 8, 2007

As I write this, I’m hurtling through small towns and big cities on the train home. We’ve passed through Baltimore – which reminds me of a project I did once, way back when, to work with a group of mostly small and medium-sized organizations on technology planning. In those days, the buzzwords were “internet connectivity,” “networks,” “websites,” and “email.” This was in the solidly web 1.0 world where many organizations still weren’t even networked, still used dial-up internet connections, and had websites written in the earliest version of Front Page, or were done by the CFO’s nephew.

I’ve emerged from this week’s frenzy of buzzwords like “blogging,” “open API,” “e-advocacy,” “municipal wireless” and “social networking” not surprised at how much things have changed, really, but how much they have stayed exactly the same. From the stories I’ve heard this week, nonprofits of the size that I’m most familiar with (small to medium-sized) still don’t have in-house technology expertise to make evaluations about what directions to go in. They sometimes deal with vendors and developers that don’t really understand their mission, don’t speak their language, and don’t tell them the truth (whether intentionally, or by a lack of self-examination.) They struggle mightily with software, no matter whether it’s free/open source or proprietary, shrink-wrapped or custom-built, on their desktops or web-hosted, which they generally spend extraordinary amounts of time and/or money on. The buzzwords have changed and the technology has gotten more sophisticated – but the problems many nonprofits are facing are exactly the same. So I hate to throw cold water on the whole enterprise – but if the core issues that most nonprofits are facing haven’t changed, and the situation isn’t getting better, how is it that have we helped?

I also saw the conference with some different, post-seminary eyes. I was looking for the deeper purposes behind the implementation of technology. I was looking for the discriminating approach to adopt technology appropriately. I was looking for the big conversation – why are we doing this anyway? Is it still just in the pursuit of “efficiency”? Is it all just TCO arguments? And I also looked at this with post-implementation eyes. I spent 8 years implementing technology “solutions” for nonprofit organizations. I wrote thousands of lines of code and designed more databases than I can count. I think I truly did some good, and I know I made mistakes along the way. Mistakes I hope to learn from, now that I won’t be doing implementation anymore.

Sometimes, the forward march of technology seems like this train I’m riding on – inexorably traveling down the track of capitalist profit while nonprofits are hanging on to those little hand-powered trucks that we, the people who serve them in this realm are working really hard to pump up and down, so we can try and gamely keep up. And while they watch really large organizations zip by them in bigger, better vehicles, looking exactly like they know where they are going. But no one seems to be asking “why are we on this track in the first place?” “Is being on this track going to really help me save the whales/feed people/organize/save the planet?”

And it’s making me think a lot about what I’m going to start calling “Nonprofit Technology Consulting 2.0″ (and yes, I’m subverting the dominant paradigm.) I don’t know yet whether I’ll actually start practicing it, but I’d like to think about it more. What would it be like if we could help nonprofits with the following:

  • Asking whether technology implementations in their organization in the past have really facilitated their mission? In what ways have they not?
  • Asking whether technology played a beneficiary, damaging or neutral role in internal organizational dynamics and staff morale?
  • Asking, before implementing a new technology – what problem is really attempting to be solved? is it a problem that can be solved in any other ways?
  • How does increasing use of networking technology, on-line presence, and internet communications facilitate or hinder work that is done face to face?
  • Making choices about technology not just based on cost/TCO or feature set – but to bring in issues of the effects on staff, organizational dynamics, and the role of factors such as organizational determination of data destiny, source and ownership of software, and environmental impact.
  • Being mediators between vendors and nonprofits – to look at issues that are technological, and issues that are about personality, behavior and organizational structure and dynamics (on both sides)
  • Looking at the bigger picture – how does what an organization does with technology affect the larger community, and the planet?

I’m looking for ways that it might be possible to practice nonprofit technology consulting with head and heart, with a view to the bigger picture of our society and our planet, and the precarious place we are in as human beings at this time, and with a view that reflects my emerging belief that increasing human touch and human contact will do more, in the end, than many of our attempts to increase efficiency by using technology.

When I re-started this blog 6 months ago, I named it Zen and the Art of Nonprofit Technology for a good reason. I want us to pay attention. I want us to pay attention to what we are doing, and how we are doing it. I’m very clear that there are technology implementations that are completely appropriate, mission-facilitating, and even good for the greater community, and good for the planet. I want to make sure that every single technology implementation is like that. My bet is that we might do a lot fewer of them if that were so.

As I keep thinking more about this, I’ll be blogging about it. I welcome any feedback and conversation, either by email, or on comments and trackbacks on this blog.

{ 6 comments }