Like all UN organizations, ILO has an internal evaluation system. This is done for the purpose of donor accountability, increased experience, and work improvement. Like the rest of the UN system, we follow the standard criteria developed by the Organization for Economic Co-operation and Development (OECD). There are five basic principles: relevance, effectiveness, efficiency, sustainability, and impact. Two years ago, another one was added coherence, Irina Sinelina says.
Marina Mikhailova learned the principles and approaches to evaluation (including organizational development evaluation and personnel evaluation) from Process Consulting and the Siberian Civic Initiatives Support Center, and international schools.
All the small pieces got together, one after another, to build the big theoretical experience. In practice, it is, first of all, extensive experience in evaluating projects submitted for grant competitions. We, at the Garant Center, have been organizing these contests since 2001 in Arkhangelsk Region. Later I took part in reviewing applications for other contests and evaluating the results of the projects we had supported. I also had some experience in evaluation from the times I was the director of a recruitment agency. as such, I had to evaluate personnel and the management system in the organization, Marina Mikhailova continues.
KNOWLEDGE AND COMPETENCIES
The topic of knowledge and competencies in evaluation is closely related to the topic of learning. If we know what the competencies are needed, we know what to teach them, and how to write job descriptions.
Since the late 1980s, various professional associations of evaluators started publishing lists of competencies. For example, the UN Evaluation Group believes that an evaluator should have basic professional competencies, technical skills, management skills, interpersonal skills, and promote a culture of evaluation.
Canada, a country that has established a certification system for evaluators, suggests the following areas of competence for a practicing evaluation specialist[18]:
1. Reflexive competencies related to fundamental norms and values in the field of evaluation, as well as awareness of ones own expertise and needs for professional growth.
2. Technical competencies related to specific aspects of evaluation: planning, data collection, data analysis, reporting.
3. Situational competencies related to evaluative thinking applied to analyze and consider the interests, problems, and circumstances specific to each evaluation.
4. Managerial competencies related to the project/evaluation management process (budgeting, resourcing, management).
5. Interpersonal competencies related to communication, negotiation, conflict resolution, cooperation.
The Association of Specialists in Program and Policy Evaluation has not yet produced a similar list of competencies. Natalia Kosheleva, ASPPE President, emphasizes an understanding of how programs and projects work as one of the core competencies of evaluation specialists in Russia. Equally important is the ability to collect, analyze and present data. Communication and interpersonal competencies are necessary for those involved in external evaluations. Finally, visual communication skills are increasingly demanded lately.
Alena Bogomolova says that an evaluator needs to be able to exchange opinions and knowledge freely.
You need groups of people or platforms where you can talk about what youre getting/not getting and discuss everything before embarking on the journey of evaluation: the best ways to proceed, questions to formulate, design to use. You need to be able to talk to other people about it. Because it is very difficult to develop in a vacuum. What else is important? In my opinion, it is vital to have your own value-based professional position, from which you will make an evaluation, she says.
Marina Mikhailova agrees: To be a professional, you have to be part of the professional community, sharing experiences, learning from others, communicating. And it is important that would-be evaluators follow ethical standards. Still, evaluation is not an audit, not an inspection. And here it is important to remember that evaluation is not about giving an A or an F; it is about answering the right questions and making the right decisions. An evaluator is a person who has come to help sort things out, Marina Mikhailova explains.
Evaluation is an incredibly interesting field, Irina Efremova-Garth notes, because an evaluator must have skills from many different areas.
You have to be extremely proficient at doing different kinds of research, extremely good at data crunching. You must have very good management skills. After all, evaluation is rarely done by just one person. It usually requires a team, so you have to be able to build processes properly, allocate roles in the team and manage results. You have to have reflection, critical thinking skills, the expert adds.
Irina Sinelina believes that it is important to have a good basic education in liberal arts, which builds analytical skills, and a good knowledge of English, because much of the material about evaluation is written in English. The rest you can learn by practice.
PROFESSIONAL STANDARD
As mentioned earlier, one of the stages in the development of a profession is a professional standard. Why do we need a professional standard in any activity? As noted in Decree No.23 of the Russian Government dated January 22, 2013 professional standards are applied[19]
Примечания
1
Здесь и далее под оценкой понимается «систематический сбор информации о деятельности в рамках программы, ее характеристиках и результатах, который проводится для того, чтобы вынести суждение о программе, повысить эффективность программы и/или разработать планы на будущее». Источник: Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks: Sage Publications.
2
Wikipedia. (2022). Michael Quinn Patton. Retrieved from: https://en.wikipedia.org/wiki/Michael_Quinn_Patton.
3
Здесь и далее под оценщиком понимается специалист по оценке.
4
The Blue Marble фотография планеты Земля, сделанная 7 декабря 1972 года экипажем космического корабля «Аполлон-17» с расстояния примерно в 29 тыс. км от поверхности Земли. Для астронавтов Земля имела размер и вид голубого марбла (игрушки, обычно представляющей собой цветной стеклянный шарик).
5
Hereinafter, evaluation is understood as the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. Source: Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks: Sage Publications.
6
Wikipedia. (2022). Michael Quinn Patton. Retrieved from: https://en.wikipedia.org/wiki/Michael_Quinn_Patton.
7
Hereinafter, an evaluator is defined as an evaluation specialist.
8
The Blue Marble is a photograph of Planet Earth taken on December 7, 1972 by the crew of Apollo 17 spacecraft from a distance of around 29,000 kilometers from the planets surface. For the astronauts, Earth was the size and appearance of a blue marble (a colored glass ball toy).
9
Wilensky, L. (1964). The professionalization of everyone? American Journal of Sociology, 70(2), 137158. Ayoo, S., Wilcox, Y., LaVelle, J. M., Podems, D., & Barrington, G. V. (2020). Grounding the 2018 AEA Evaluator Competencies in the broader context of professionalization. In J. A King (Ed.), The American Evaluation Associations Program Evaluator Competencies. New Directions for Evaluation, 2020, 13 30.