The following is our best attempt to summarize at least some of the salient issues that arose in the intense and brilliant June 15, 2010 Tweet Chat.  You can see the full archive of the chat here.

Topic: Evaluating the Consultant’s Own Effectiveness

Many thanks to Nick Perona (@rawr_nickzilla) for the hours it took to assemble this!

Question 1: Why Evaluate?
Before considering HOW we measure our effectiveness, we should probably consider WHY to measure. At its highest potential, what would evaluating consultant effectiveness make possible – for the client and for the consultant?

Responses and conversation included:

• For the consultant, evaluating effectiveness allows opportunity to compare what the client sees as effective versus what consultant thinks is effective.
• Allows the consultant to measure the extent of their role in client’s success in addition to the benchmarks the client was helped to achieve.
• For the client, measurement of consultant effectiveness could determine whether client is moving closer to achieving their vision.
• At its highest potential, evaluating consultant success makes possible achieving an organization’s vision, while ensuring the consultant is continually improving his/her ability to do so.

Question 2: What to Measure?
What would consultants have to measure to accomplish that? What might be indicators of success?

Responses and conversation included:

• Success as defined by the consultant can be very different from how it is defined by the client. Importance of consultant and client arriving at a single definition for their success and then measure against it?
• Importance of establishing trust and an ongoing relationship between consultant and client, facilitating further collaboration. Once a mutual definition of success is established, it becomes far easier to arrive at “deliverables” on the part of the consultant and metrics for that success on the part of the client.
• Question raised: How can qualitative indicators be quantified? For example, an organization can measure “learning” by weighing what it knows / can do after the engagement versus prior to the engagement.
• Noted by @SpurDave (Dave Svet): Ethnographic Research (Wikipedia link here) as a valuable tool for creating success indicators. Dave suggested Margaret LeCompte’ss Ethnographers Toolkit.
• Organizational learning and social change as important aspects to measure, as the effects continue to add value after the consultant leaves.
• Question re: Grant proposal writing. If client is energized about the process and creates change within their organization to qualify for the grant, but doesn’t win the grant in the end, how to measure the consultant’s success?
• Need for mutual client/consultant definition of success. Before work begins, consultant and client arrive at indicators of success for that engagement. May include the grant AND other aspects of “success” in the process of grant preparation.

Question 3: Deliverables vs. Results
Is a consultant’s job to provide “deliverables” or the result of those deliverables? Put another way, do clients want “a drill” or do they want “a hole”? Or “to hang a towel rack”? Or really, “a dry towel”?

Responses and conversation included:

• The consultant’s responsibility to work with the client to determine their ultimate goal: Is it the “towel rack” they want? Or a “shelf”?
• @MarkRiffey noted (with tongue perhaps halfway in cheek): “Clients dont want social media, they want more/better relationships/interaction. They put up with social media to get it :) ”
• @JDeanCoffey (Jara Dean-Coffey) noted: “We work to increase clients’ adaptive capacity to think, assess, determine & act. And to do it without us. Process is also a tool.
• Once these goals are established, the consultant brings his/her own tools to work with the client, but the client also has their own resources, tools, organizational strengths upon which to build success. The consultant helps align the work at the end result but most of the “heavy lifting” comes from the client.
• As @SeattleDrury (Peter Drury) noted: “If the consultant is working harder than the client to achieve success, then the outcomes will not be sustainable.”
• The need for a true desire on the part of the client to create change. Organizations may recognize internal problems but may be fearful of the change necessary to get them where they want to be.
• If stakeholders and key decision-makers are involved in the process every step of the way, they can help identify the “pain” to be addressed during the engagement.
• Is it the consultant’s place to focus on outcomes when a client wants outputs (a deliverable)?
• How to measure whether our work is contributing to the client’s big picture mission / vision / outcomes? Are there ways for a social media or grant writing or board development consultant to move deliverables beyond “I finished the project and that’s all that matters; it’s up to the client to use it how they wish”? Can the same narrowly focused project be positioned to accomplish the client’s larger outcomes? What up front discussions would have to happen for consultants to be able to fit their work into the client’s ultimate goals?
• As @CharlieKalech noted in a question to @JoppaThoughts (Erica Holthausen): Is your goal to finish your assessment or to convey your findings & make it meaningful towards their goal?

Question 4: Measurement Methods / Tools
What are you currently using to measure real results / outcomes / effectiveness?
How are you discovering the outcomes clients want beyond deliverables? How is that creating your scope of work?

Responses and conversation included:

• Are consultants measuring their own “outputs” or “outcomes?” Are they measuring the consulting equivalent of cans of food distributed at a Food Bank vs. a reduction in the number of hungry families in the community?
• If in some circumstances outputs are all there are to measure, should that be the only metric for success? If so, how can a consultant measure the ultimate outcomes / impact of his/her work?
• How can we measure whether the engagement has led to the client being transformed, or the extent to which they are aiming towards their vision?
• @JoppaThoughts (Erica Holthausen) noted: “Our projects short term, so maybe more output oriented. Client goals are long-term, more outcome oriented. *think*”
• The need for pre-engagement discussion came up here as well. Open-ended questions early on, involving the board and key decision-makers in the discussion of outcomes at the beginning of the process. What change do you want to see? What do you want to be different when we’re done?
• @SpurDave (Dave Svet) suggested: “Count quantitative outputs to compare qualitative changes. Goes back to pre-test, post test and control group. Example: More condoms distributed can result in happier, wealthier, smaller families. [Another example, later] All pet shelter clients = control. Before social media, after social media and how you compare to others. All DV shelter clients = control. Current situation is pre-test. Result of work is difference with post test.

Question 5: Reflection
Any key learnings from today’s chat? What stood out for you? Any aha’s?

Many people quoted @SeattleDrury and @AskDebra in their “biggest takeaways.” Those quotes were summed up by @JoppaThoughts: “Biggest take away: Idea that nothing we do is sustainable if we are more invested & working harder than the client.”

@amykincaid Lessons learned are probably about fit, and about expectations up front! I’m going to work on measures/ing work with individual clients, but then also for our portfolio as a whole.

@NancyIannone Seems like consultants searching for ways to measure community impact by orgs as much as the organizations themselves

@ltwhite (Leslie White) Takeaway – thinking outputs v. outcomes. How to apply to my type of consulting (risk assessment/management)

@wrightmomentum (Susan Wright) Challenge & opportunity to discover co-creative effectiveness/evaluation.

@socialchngediva (Ericka Hines) Make assessment a part of the planning in early stages. And research that for soft stuff like “leadership development”

Lastly, the following sums up how many folks thought of the chat:

@JoppaThoughts (Erica Holthausen)
Participating in #NPCons — and learning from, joking with and debating my colleagues is just another reason #whyIlovetwitter! This is sacred professional development time!

@SeattleDrury (Peter Drury)
I humbly confess I need a support group and training for managing to follow #NPCons ! Crazily powerful – and also crazy to follow! 8^)

And the Best Reflection of the Day Award goes to @socialchngediva (Ericka Hines)
#NPCons is like a brainiac cocktail party where all of the convos are uberpowerful and you dont want to miss anything.

What’s Next
First, this isn’t all the conversation, but many of the highlights. We encourage you to read  the complete transcript of the chat here.

The meatiness of this chat suggests the need for deeper conversation. Whatever topic, quote, thought you see above or in the archives here – or if you have new thoughts to add – let’s see what emerges as we have more quiet reflective time to think about these important topics.