EES Conference 2016 Presentation: Modular Evaluator Competency Profiles as a Framework for Global Education, Training and Practice

Strand 1 - Evaluation Ethics, Governance, and Professionalism

12^th EES Biannual Conference

Evaluation Futures in Europe and Beyond: Connectivity, Innovation and Use

26-30 September, 2016,

MECC, Maastricht, Netherlands

Presented by:

Gunter Rochow

 Credentialed Evaluator,

Chairman and President, Capra International Inc., Canada

 

As we are all aware, global evaluation practice is making enormous strides on all fronts, stimulated to a considerable extent by the activities of the International Year of Evaluation 2015 and the Global Evaluation Agenda 2016-2020 that resulted from it. In this presentation I will look at some implications and requirements resulting from the third leg of the Global Evaluation Agenda, i.e. developing individual evaluator capacities, which calls for more evaluation-related education and training. In particular, this presentation responds to the need that Voluntary Organizations of Professional Evaluators (VOPEs) may require support with professionalization initiatives by offering an approach that has been implemented successfully worldwide for many other professions and occupations. It could do the same for evaluation. Regrettably, it is evident that the Supply of evaluation-related education and training has been increasing before the global evaluation community had carefully defined the nature and extent of Demand for it! Thus, Supply is appealing for Demand to come forward, and Demand is not steering the offering of Supply, as it should!

 

In that environment, the global evaluation community is being bombarded with advertisements from universities and colleges (example 1; example 2; example 3), programs and seminars offered by national and international evaluation societies and organizations (example 1; example 2; example 3), as well as by private sector trainers (example 1; example 2). It is a market place in which the sellers are competing for buyers, and the buyers are often confused about what they should buy in order best to meet their needs. Nevertheless, in all fairness, I must point out that in the case of Canada, the Canadian Evaluation Society has defined competencies for evaluators (although these still require polishing) and the CES offers Credentialed Evaluator status to evaluators who can demonstrate that they have achieved a strong level of the required competencies and that they abide by the applicable standards. In that global context, evaluation managers appeal to evaluation providers to offer their evaluation services often, to the extent possible, with the support of evaluators coming from diverse national, cultural, educational and linguistic backgrounds, while rightly insisting on delivering top quality products. The dilemma for many evaluation providers is how they can be sure that they are selecting the right horses to pull their wagons.

Put more precisely, while international evaluation providers welcome the advantages of having evaluators available in the global market place, they often face a major problem and uncertainty when attempting to select evaluation teams that are fully capable of delivering high quality products. Why is that so? Evaluation Providers often find it difficult to assess with confidence the extent and quality of the prior education or training in evaluation, either in a formal or informal environment, which new team members might bring to the table. The reason is that the prospective team members present to the Evaluation Providers a degree, a diploma, a certificate, a record of attendance that makes assertions while the Evaluation Provider does not have a satisfactory common yardstick to assess the contents of these records of achievement!
 

That is, of course, a problem that is common to most educational records and it applies to most education and training institutions. To defend their personal credibility in that environment, students often consciously select institutions of learning on the basis of the institutions’ reputations, track records and published rankings, but the documentary evidence of their learning does little to provide detailed comparable information to prospective employers.

I believe that the global evaluation community should and can do better than that, as it sets out on the third leg of the Global Evaluation Agenda 2016-2020, and we can do better without impinging on the time-honoured and expected “independence” of institutional education providers. 

An approach to evaluation capacity building using competency profiles would not only help public, private and educational entities in curricular planning, but the resulting profiles would also be useful to commissioners for evaluation services and service providers in gender-equitable personnel recruitment, training, performance assessment and the development of remuneration scales.


Before I set out on describing how the evaluation community can do better, I have a question for Evaluation Providers in our midst: Have you had problems in identifying and understanding the needed substance of educational and training credentials of prospective evaluators before you risk putting them on your teams? Or have you muddled through, as I have had to do on occasion? In my view, a solution for the global evaluation community is to develop and maintain a detailed and unequivocal listing of demand for evaluation-related education and training in accordance with the third leg of the Global Evaluation Agenda 2016-2020 and to present that to potential suppliers of the needed education and training, and insisting that potential suppliers demonstrate the extent to which their offerings in the market place respond to the demand specified by the global evaluation community.
 
The key feature of a DACUM competency profile is that it is a bottom-up, not top-down, description of the observable and measurable skill competencies required in a given area of professional practice, which is developed by groups of practitioners. The net effect is not only the production of descriptions of skill profiles that reflect the realities inherent in professional practice in terms of professional and personal skills, but also the fact that these profiles point to evolving good or best practices that can help to shape the future application of these professional skills.
                
Professional competency is defined as “the demonstrated ability – including knowledge, skills and attitudes – to perform a task successfully according to an established standard.” These competencies are typically analysed in up to six levels, plus a description of the required knowledge.

I will illustrate how such an approach works by means of an example from Brazil. In 1998 the Brazilian National Industrial Training Service (SENAI) had a problem (see: OccasionalPaper4Dacum): The Brazilian Ministry of Labour and Employment (Ministério do Trabalho e Emprego) had published a new Classification and Dictionary of Occupations in accordance with which SENAI was expected to deliver an updated training program. SENAI had difficulty understanding the precise skill content of the defined occupations. With a view to finding a solution, SENAI contacted the Government of Canada, among representatives from several other countries, with the request to send a three person public-private sector team to Brazil, at Brazil’s expense, to participate in a Labour Market Transformations International Seminar. All invited guests, Canada’s included, were asked to make presentations on how SENAI’s problem could be solved. I was both surprised and pleased when our firm’s (Capra International Inc.) proposed approach was accepted! Consequently, from 1999 until 2002 our firm, in collaboration with the Canadian Vocational Association, trained Brazilian facilitators for using the DACUM (see origin and further development) (Developing a Curriculum) method and supervising the production of competency profiles for about 300 of the 500 occupational families contained in Brazil’s new Classification and Dictionary of Occupations. Since then, until as recently as 2014, other DACUM-related engagements followed, including with the Institute of Economic Research  (FIPE) at the University of São Paulo, which undertook further DACUM-related research under contract to the Ministry of Labour and Employment.

Global planning for education and training for evaluation is best done at Level 2 General Areas of Competencies and Level 3 Tasks/ Skills to ensure overall comparable coverage, while leaving it to the Voluntary Organizations of Professional Evaluators (VOPEs) to plan at other DACUM levels, as needed. 

An example of Level 2 and Level 3 profiles looks as follows:

Level 2: Identifying the General Areas of Competencies (GACs)

Possible GACs of A Professional Evaluator

A Professional Evaluator must be able to …

GACs	Professional / Technical Competencies
A.      Circumscribe An Evaluation  Mandate
B.      Design …
C.      Collect …
D.      Analyse …
E.      …
F.       …

And to accomplish the above duties and tasks, a Professional Evaluator must be able to …

GACs	Non Occupation Specific Competencies
X.   Demonstrate General Competencies	…	…	…	…

Level 3: Identifying Competencies

Possible Competencies of A Professional Evaluator

A Professional Evaluator must be able to …

GACs	Professional / Technical Competencies
A. Circumscribe an Evaluation       Mandate	1.       Determine the purpose for an evaluation	…	…	…
B. Design …	…	Establish evaluation feasibility and appropriateness	Anticipate design issues	…
C. Collect …	Negotiate access to data	Address quality control issues	Ensure adherence to established data control protocols	…
D. Analyse …	…	…	…	…
…	…	…	…	…

And to accomplish the above duties and tasks, a Professional Evaluator must be able to …

GACs	Non Occupation Specific Competencies
X.   Demonstrate General    Competencies	1.       Act ethically	2.       Demonstrate written communication skills	3.       Demonstrate verbal communi-cation skills	4.       Demonstrate analytical skills
	5.       Negotiate	6.       …	7.       …	8.       …

The following steps are needed to develop a global competency framework for evaluation professionals:
 1.      Identify a lead organization responsible for guiding the development of a global strategy and plan of action for implementing the third leg of the Global Evaluation Agenda. That organization might be the International Organization for Cooperationin Evaluation (IOCE).2.      Mandate the chosen organization to consult with its member organizations (VOPEs) on the development of the General Areas of Competencies (Level 2) and Tasks/Skills (Level 3) for evaluation practice, in accordance with applicable norms and standards.
3.      To execute step 2, organize DACUM workshops, which normally require 4-5 days each under the leadership of a qualified DACUM facilitator.
4.      Once the global evaluation community has reached agreement on the profiles for the General Areas of Competencies for Evaluation Practice (Level 2) and Tasks/Skills (Level 3), offer these as guidelines to potential education and training providers and to recruiters of evaluators. This is the Demand side.
5.      Request potential education and training providers to offer their education and training services with specific reference to the guidelines issues by the global evaluation community and the VOPEs. This is the Supply side.
6.      Encourage potential recruiters of evaluators to specify their Demand in accordance with the adopted global and VOPE guidelines.
7.      Encourage potential evaluators to document their evaluation expertise in accordance with the adopted guidelines.
8.      Publicize the existence of the guidelines through professional associations and VOPE websites.
9.      Update the guidelines from time to time, as evaluation practice evolves.
10.  Request global organizations and VOPEs to report annually on their progress in demanding, offering and using education and training in accordance with the adopted guidelines.


From what you know so far about the proposed approach to use Modular Evaluator Competency Profiles as a Framework for Global Education, Training and Practice, do you think that this could help you and your organization in executing the third leg of the Global Evaluation Agenda; i.e. in developing individual evaluator capacities?


I would be grateful to you for now using the open floor to comment on how the proposed approach might help you, your organization, your VOPE, or why you think that it might not. If you can think of additional steps that may be  needed for implementing this approach, identifying these would be particularly helpful. Your comments will be valuable in helping to chart a way forward. So, please step forward and let us discuss.

 

 

 

Thank you!