EES Conference 2016 Presentation: Modular Evaluator Competency Profiles as a Framework for Global Education, Training and Practice

Strand 1 - Evaluation Ethics, Governance, and Professionalism

12^th EES Biannual Conference

Evaluation Futures in Europe and Beyond: Connectivity, Innovation and Use

26-30 September, 2016,

MECC, Maastricht, Netherlands

Presented by:

Gunter Rochow

 Credentialed Evaluator,

Chairman and President, Capra International Inc., Canada

 

As we are all aware, global evaluation practice is making enormous strides on all fronts, stimulated to a considerable extent by the activities of the International Year of Evaluation 2015 and the Global Evaluation Agenda 2016-2020 that resulted from it. In this presentation I will look at some implications and requirements resulting from the third leg of the Global Evaluation Agenda, i.e. developing individual evaluator capacities, which calls for more evaluation-related education and training. In particular, this presentation responds to the need that Voluntary Organizations of Professional Evaluators (VOPEs) may require support with professionalization initiatives by offering an approach that has been implemented successfully worldwide for many other professions and occupations. It could do the same for evaluation. Regrettably, it is evident that the Supply of evaluation-related education and training has been increasing before the global evaluation community had carefully defined the nature and extent of Demand for it! Thus, Supply is appealing for Demand to come forward, and Demand is not steering the offering of Supply, as it should!

 

In that environment, the global evaluation community is being bombarded with advertisements from universities and colleges (example 1; example 2; example 3), programs and seminars offered by national and international evaluation societies and organizations (example 1; example 2; example 3), as well as by private sector trainers (example 1; example 2). It is a market place in which the sellers are competing for buyers, and the buyers are often confused about what they should buy in order best to meet their needs. Nevertheless, in all fairness, I must point out that in the case of Canada, the Canadian Evaluation Society has defined competencies for evaluators (although these still require polishing) and the CES offers Credentialed Evaluator status to evaluators who can demonstrate that they have achieved a strong level of the required competencies and that they abide by the applicable standards. In that global context, evaluation managers appeal to evaluation providers to offer their evaluation services often, to the extent possible, with the support of evaluators coming from diverse national, cultural, educational and linguistic backgrounds, while rightly insisting on delivering top quality products. The dilemma for many evaluation providers is how they can be sure that they are selecting the right horses to pull their wagons.

Put more precisely, while international evaluation providers welcome the advantages of having evaluators available in the global market place, they often face a major problem and uncertainty when attempting to select evaluation teams that are fully capable of delivering high quality products. Why is that so? Evaluation Providers often find it difficult to assess with confidence the extent and quality of the prior education or training in evaluation, either in a formal or informal environment, which new team members might bring to the table. The reason is that the prospective team members present to the Evaluation Providers a degree, a diploma, a certificate, a record of attendance that makes assertions while the Evaluation Provider does not have a satisfactory common yardstick to assess the contents of these records of achievement!
 

That is, of course, a problem that is common to most educational records and it applies to most education and training institutions. To defend their personal credibility in that environment, students often consciously select institutions of learning on the basis of the institutions’ reputations, track records and published rankings, but the documentary evidence of their learning does little to provide detailed comparable information to prospective employers.

I believe that the global evaluation community should and can do better than that, as it sets out on the third leg of the Global Evaluation Agenda 2016-2020, and we can do better without impinging on the time-honoured and expected “independence” of institutional education providers. 

An approach to evaluation capacity building using competency profiles would not only help public, private and educational entities in curricular planning, but the resulting profiles would also be useful to commissioners for evaluation services and service providers in gender-equitable personnel recruitment, training, performance assessment and the development of remuneration scales.


Before I set out on describing how the evaluation community can do better, I have a question for Evaluation Providers in our midst: Have you had problems in identifying and understanding the needed substance of educational and training credentials of prospective evaluators before you risk putting them on your teams? Or have you muddled through, as I have had to do on occasion? In my view, a solution for the global evaluation community is to develop and maintain a detailed and unequivocal listing of demand for evaluation-related education and training in accordance with the third leg of the Global Evaluation Agenda 2016-2020 and to present that to potential suppliers of the needed education and training, and insisting that potential suppliers demonstrate the extent to which their offerings in the market place respond to the demand specified by the global evaluation community.
 
The key feature of a DACUM competency profile is that it is a bottom-up, not top-down, description of the observable and measurable skill competencies required in a given area of professional practice, which is developed by groups of practitioners. The net effect is not only the production of descriptions of skill profiles that reflect the realities inherent in professional practice in terms of professional and personal skills, but also the fact that these profiles point to evolving good or best practices that can help to shape the future application of these professional skills.
                
Professional competency is defined as “the demonstrated ability – including knowledge, skills and attitudes – to perform a task successfully according to an established standard.” These competencies are typically analysed in up to six levels, plus a description of the required knowledge.

I will illustrate how such an approach works by means of an example from Brazil. In 1998 the Brazilian National Industrial Training Service (SENAI) had a problem (see: OccasionalPaper4Dacum): The Brazilian Ministry of Labour and Employment (Ministério do Trabalho e Emprego) had published a new Classification and Dictionary of Occupations in accordance with which SENAI was expected to deliver an updated training program. SENAI had difficulty understanding the precise skill content of the defined occupations. With a view to finding a solution, SENAI contacted the Government of Canada, among representatives from several other countries, with the request to send a three person public-private sector team to Brazil, at Brazil’s expense, to participate in a Labour Market Transformations International Seminar. All invited guests, Canada’s included, were asked to make presentations on how SENAI’s problem could be solved. I was both surprised and pleased when our firm’s (Capra International Inc.) proposed approach was accepted! Consequently, from 1999 until 2002 our firm, in collaboration with the Canadian Vocational Association, trained Brazilian facilitators for using the DACUM (see origin and further development) (Developing a Curriculum) method and supervising the production of competency profiles for about 300 of the 500 occupational families contained in Brazil’s new Classification and Dictionary of Occupations. Since then, until as recently as 2014, other DACUM-related engagements followed, including with the Institute of Economic Research  (FIPE) at the University of São Paulo, which undertook further DACUM-related research under contract to the Ministry of Labour and Employment.

Global planning for education and training for evaluation is best done at Level 2 General Areas of Competencies and Level 3 Tasks/ Skills to ensure overall comparable coverage, while leaving it to the Voluntary Organizations of Professional Evaluators (VOPEs) to plan at other DACUM levels, as needed. 

An example of Level 2 and Level 3 profiles looks as follows:

Level 2: Identifying the General Areas of Competencies (GACs)

Possible GACs of A Professional Evaluator

A Professional Evaluator must be able to …

GACs	Professional / Technical Competencies
A.      Circumscribe An Evaluation  Mandate
B.      Design …
C.      Collect …
D.      Analyse …
E.      …
F.       …

And to accomplish the above duties and tasks, a Professional Evaluator must be able to …

GACs	Non Occupation Specific Competencies
X.   Demonstrate General Competencies	…	…	…	…

Level 3: Identifying Competencies

Possible Competencies of A Professional Evaluator

A Professional Evaluator must be able to …

GACs	Professional / Technical Competencies
A. Circumscribe an Evaluation       Mandate	1.       Determine the purpose for an evaluation	…	…	…
B. Design …	…	Establish evaluation feasibility and appropriateness	Anticipate design issues	…
C. Collect …	Negotiate access to data	Address quality control issues	Ensure adherence to established data control protocols	…
D. Analyse …	…	…	…	…
…	…	…	…	…

And to accomplish the above duties and tasks, a Professional Evaluator must be able to …

GACs	Non Occupation Specific Competencies
X.   Demonstrate General    Competencies	1.       Act ethically	2.       Demonstrate written communication skills	3.       Demonstrate verbal communi-cation skills	4.       Demonstrate analytical skills
	5.       Negotiate	6.       …	7.       …	8.       …

The following steps are needed to develop a global competency framework for evaluation professionals:
 1.      Identify a lead organization responsible for guiding the development of a global strategy and plan of action for implementing the third leg of the Global Evaluation Agenda. That organization might be the International Organization for Cooperationin Evaluation (IOCE).2.      Mandate the chosen organization to consult with its member organizations (VOPEs) on the development of the General Areas of Competencies (Level 2) and Tasks/Skills (Level 3) for evaluation practice, in accordance with applicable norms and standards.
3.      To execute step 2, organize DACUM workshops, which normally require 4-5 days each under the leadership of a qualified DACUM facilitator.
4.      Once the global evaluation community has reached agreement on the profiles for the General Areas of Competencies for Evaluation Practice (Level 2) and Tasks/Skills (Level 3), offer these as guidelines to potential education and training providers and to recruiters of evaluators. This is the Demand side.
5.      Request potential education and training providers to offer their education and training services with specific reference to the guidelines issues by the global evaluation community and the VOPEs. This is the Supply side.
6.      Encourage potential recruiters of evaluators to specify their Demand in accordance with the adopted global and VOPE guidelines.
7.      Encourage potential evaluators to document their evaluation expertise in accordance with the adopted guidelines.
8.      Publicize the existence of the guidelines through professional associations and VOPE websites.
9.      Update the guidelines from time to time, as evaluation practice evolves.
10.  Request global organizations and VOPEs to report annually on their progress in demanding, offering and using education and training in accordance with the adopted guidelines.


From what you know so far about the proposed approach to use Modular Evaluator Competency Profiles as a Framework for Global Education, Training and Practice, do you think that this could help you and your organization in executing the third leg of the Global Evaluation Agenda; i.e. in developing individual evaluator capacities?


I would be grateful to you for now using the open floor to comment on how the proposed approach might help you, your organization, your VOPE, or why you think that it might not. If you can think of additional steps that may be  needed for implementing this approach, identifying these would be particularly helpful. Your comments will be valuable in helping to chart a way forward. So, please step forward and let us discuss.

 

 

 

Thank you!

Perfiles Modulares de Competencia de Evaluadores como Marco de Referencia para la Educación, Formación y Práctica Globales

Se presentó este tema en la 12ª Conferencia Bianual de la Sociedad Europea de Evaluación (EES), en Maastricht, Países Bajos, el 28 de septiembre de 2016.

La práctica de la evaluación global está dando grandes pasos en todos los frentes, estimulados en gran medida por las actividades del Año Internacional de la Evaluación 2015  y la Agenda Global de Evaluación 2016-2020 que resultó de ella. La tercera etapa de la Agenda Global de Evaluación, es decir, el desarrollo de las capacidades individuales de los evaluadores, exige una mayor educación y capacitación relacionadas con la evaluación. En particular, las Organizaciones Voluntarias de Evaluadores Profesionales (VOPE) pueden necesitar apoyo para desarrollar iniciativas de profesionalización adecuadas.

Al perseguir tales iniciativas, el peligro es que la Oferta de educación y formación relacionada con la evaluación intenta atraer la Demanda, y la Demanda no está dirigiendo la Oferta, como debería! Es un mercado en el que los vendedores están compitiendo por los compradores, y los compradores a menudo se confunden sobre lo que deben comprar con el fin de satisfacer mejor sus necesidades.

Al abordar la tercera etapa de la Agenda Global de Evaluación 2016-2020, sería en beneficio de la comunidad de evaluación global determinar las necesidades locales concretas, VOPE por VOPE, y, sobre la base de ellas, invitar a los proveedores de la educación y la formación (la Oferta) para responder a la Demanda específica de educación y formación relacionadas con la evaluación. 

El mejor enfoque para definir las competencias requeridas en una profesión, incluida la evaluación, es el método DACUM el cual fue desarrollado por la Asociación Vocacional Canadiense (CVA). Por invitación del Serviço Nacional de Aprendizagem Industrial en Brasil, tuve el privilegio, entre 1998 y 2002, en colaboración con la CVA, de utilizar el método DACUM para definir las competencias de unas 300 de las 500 familias ocupacionales en la entonces nueva Clasificación de Ocupaciones (Classificação Brasileira de Ocupações).   El aspecto fundamental del método DACUM es que es de abajo hacia arriba, no de arriba hacia abajo, involucrando a los profesionales que entienden completamente el contexto técnico, educativo y cultural local.   Si los VOPE individuales definen sus requisitos utilizando este método común, la comunidad de evaluación global puede identificar fácilmente las competencias básicas comunes, así como las variaciones regionales o específicas de cada país o requisitos únicos, con miras a desarrollar un marco global de competencias de evaluación. Basándose en las competencias definidas, los proveedores de capacitación pueden ofrecer curso modulares que se ajusten a la demanda local especificada.

Un enfoque para el fortalecimiento de la capacidad de evaluación utilizando perfiles de competencia no sólo ayudaría a las entidades públicas, privadas y educativas en la planificación curricular, sino que los perfiles resultantes también serían útiles para los servicios de evaluación y proveedores de servicios de reclutamiento, capacitación, evaluación de desempeño, así como el desarrollo de las escalas de remuneración.

Las VOPE u otras entidades pertinentes que deseen obtener más información sobre cómo utilizar el método propuesto para definir sus competencias de evaluación requeridas pueden dirigir sus investigaciones con confianza a:

Gunter Rochow

Evaluador acreditado (Sociedad Canadiense de Evaluación)

Correo electrónico: gunter@rochow.info 

See the English version of this post.

Modular Evaluator Competency Profiles as a Framework for Global Education, Training and Practice

This subject had been presented at the 12th European Evaluation Society (EES) Biannual Conference, in Maastricht, Netherlands on 28 September 2016.

Global evaluation practice is making enormous strides on all fronts, stimulated to a considerable extent by the activities of the International Year of Evaluation 2015 and the Global Evaluation Agenda 2016-2020 that resulted from it. The third leg of the Global Evaluation Agenda, i.e. developing individual evaluator capacities, calls for more evaluation-related education and training. In particular, Voluntary Organizations of Professional Evaluators (VOPEs) may require support with developing suitable professionalization initiatives.

In pursuing such initiatives, the danger is that the Supply of evaluation-related education and training is appealing for Demand to come forward, and Demand is not steering the offering of Supply, as it should! It is a market place in which the sellers are competing for buyers, and the buyers are often confused about what they should buy in order best to meet their needs.

In addressing the third leg of the Global Evaluation Agenda 2016-2020, it would be to the advantage of the global evaluation community to determine, on a VOPE by VOPE basis, its precise local needs, and on the basis of these, to invite suppliers of education and training to respond to the specific demand for evaluation-related education and training.

The best time-honoured approach for defining required competencies in a profession, evaluation included, is the DACUM method developed by the Canadian Vocational Association (CVA). At the invitation of Brazil’s National Industrial Training Service (Serviço Nacional de Aprendizagem Industrial) I had the privilege between 1998 and 2002, in collaboration with the CVA, to use the DACUM method to define the skill competencies of about 300 out of about 500 occupational families in Brazil’s then new Classification of Occupations (Classificação Brasileira de Ocupações).

The fundamental aspect of the DACUM method is that it is bottom-up, not top-down, involving practitioners who thoroughly understand the local technical, educational and cultural context.

If individual VOPEs define their requirements using this common method, then the global evaluation community can readily identify common core competencies, as well as regional or country-specific variations or unique requirements, with a view to developing a global framework of evaluation competencies. Building on the defined competencies, training providers can then offer modular course work that fits specified demand.

An approach to evaluation capacity building using competency profiles would not only help public, private and educational entities in curricular planning, but the resulting profiles would also be useful to commissioners for evaluation services and service providers in gender-equitable personnel recruitment, training, performance assessment and the development of remuneration scales.

VOPEs, or other relevant entities, desiring further information on how to use the proposed method in defining their required evaluation competencies may direct their inquiries in confidence (in English, French or Spanish) to:

Gunter Rochow
Credentialed Evaluator (Canadian Evaluation Society)
(gunter@rochow.info)

Buscar versión Español aquí.