This article provides a brief overview of an Evaluation Needs Assessment framework developed by Lirata and provided as a resource to our partner organisations. The framework provides a structure that organisations can use in undertaking an Evaluation Needs Assessment to help guide their work in strengthening Monitoring, Evaluation and Learning (MEL) in programs, teams or organisation wide.

This framework was developed by Mark Planigale through a partnership project in 2018 with Djirra, a specialist Aboriginal family violence and women’s support service.

We thank others whose work has informed our thinking about evaluation capacity building and needs assessment, in particular Shanelle Boyle, Jean King, Johanna Morariu, Hallie Preskill and Boris Volkov. We thank Fran Demetriou and Trini Espinosa Abascal for their work on the Evaluation Needs Assessment with Djirra, and Kathryn Robb and all the Djirra stakeholders for their feedback and willingness to pilot the framework.

Icon for Creative Commons Attribution-ShareAlike 4.0 International License

The Framework and this article are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

 

What is an Evaluation Needs Assessment?

An Evaluation Needs Assessment is a process through which we can identify the capacities and areas for development in Monitoring, Evaluation and Learning within a team or organisation. It involves looking at the ways that data collection, analysis, reporting, reflection and learning currently occur, and how they could be strengthened. It also considers the types of MEL that would be most useful for achieving organisational goals.

Evaluation Capacity Building

An Evaluation Needs Assessment provides a foundation for Evaluation Capacity Building - an intentional process through which a group or organisation strengthens its ability to design, implement and use effective MEL processes.

In this framework, the concept of 'capacity' encompasses both individual capacity - for example, the skills, knowledge, confidence and attitudes of individual staff - and organisational capacity - the systems, processes, structures, leadership, resources, culture and relationships which support MEL.

Why do a needs assessment?

An Evaluation Needs Assessment identifies strengths, weaknesses, and opportunities for improvement which can provide useful guidance for developing MEL systems and capacities. It can provide a more strategic approach to MEL capacity building by identifying priorities and sequences of work. This will inform effective work to strengthen MEL, for example by increasing skills and knowledge, improving processes and developing better tools. An Evaluation Needs Assessment also provides a baseline for assessing the success of Evaluation Capacity Building initiatives.

Organisations can benefit enormously from investment in well-targeted initiatives to strengthen MEL capacity. Such benefits include identifying what the organisation is doing well and where improvements are needed, gathering useful information to plan and manage service delivery, reporting more efficiently to funders, and generating strong evidence of the value and impact of their work for more powerful advocacy and to attract funding.

The framework

The MEL capacity of a team or organisation is made up of many different factors. In assessing MEL needs, it is useful to examine capacity through a number of different lenses. This framework uses three lenses:

  • Individual MEL capacity - abilities that individual staff are able to contribute.
  • Team and organisational MEL capacity - systems and capabilities that an agency or organisational unit has in place.
  • MEL life cycle - capacity to implement the key steps and stages associated with MEL.

Within these three lenses, we identify 11 capacity domains that are the focus of the Evaluation Needs Assessment. These are summarised in the table below, and explained in more detail in the remainder of the article.

Table 1: Evaluation needs assessment lenses and domains

LENSES

DOMAINS

Individual MEL capacity

1. MEL knowledge, skills & experience

2. MEL attitudes & confidence

Team and organisational MEL capacity

3. MEL focus

4. MEL systems

5. MEL leadership, roles & relationships

6. MEL resources

7. MEL culture, support & development

MEL life cycle

8. MEL design

9. Data collection

10. Analysis & reporting

11. Use & integration

Each domain is assessed individually for strengths, weaknesses, opportunities and risks, and given an overall capacity rating. We often use a five-point scale from ‘Beginning’ to ‘Strong and sustainable’ (see below) which emphasises that evaluation capacity building is a journey and not a destination or a matter of competitive performance. Other scales might be appropriate in different contexts.

Diagram of five point rating scale

Figure 1: Five-point rating scale

Lens 1: Individual MEL capacity

This first lens includes two domains which highlight what individual staff and stakeholders bring to MEL initiatives.

1. MEL knowledge, skills & experience

The extent to which staff and stakeholders understand the key concepts of MEL, and have the skills to effectively play their part in its implementation.

Not everyone needs to be a MEL expert - staff need to know enough to contribute in ways that are appropriate for their role. However, for MEL to work best, it is important that the team or organisation as a whole has the full set of skills required, or can readily access them.

2. MEL attitudes & confidence

The perspectives that stakeholders bring about whether MEL is worthwhile, relevant and achievable, and their level of confidence to be involved.

Attitudes play a big role in whether, and how, staff engage with MEL. If staff see MEL as unimportant or irrelevant to their role, MEL systems may be poorly used or even actively resisted. Individual confidence also influences how proactively people apply their MEL skills.

Lens 2: Team and organisational MEL capacity

This second lens includes five domains which highlight shared capacities that teams and organisations bring to MEL initiatives.

3. MEL focus

The extent to which stakeholders have a shared, articulated focus for MEL.

The heart of this is identifying agreed priorities about what will be evaluated, for what purpose, and which questions are most important to answer. Shared focus also means identifying and agreeing on intended outcomes, which provide a basis for monitoring and evaluating effectiveness and impact.

4. MEL systems

The extent to which sustainable operating systems (guiding documents, defined processes and structures) govern MEL, knowledge management and related areas.

Effective MEL systems help coordinate effort so that important MEL outputs and outcomes occur reliably. They enable MEL activities to occur in consistent, efficient and sustainable ways.

5. MEL leadership, roles & relationships

The nature and allocation of roles and responsibilities for MEL, including the exercise of MEL leadership, and presence of supportive MEL relationships.

Introducing MEL involves substantial learning and change, and therefore needs leadership and endorsement from senior management. Staff also need clarity about who is responsible for undertaking MEL work - including designing frameworks, processes and tools, collecting data, analysing and reporting, and interpreting and communicating findings. Relationships across teams and with external partners or communities can also contribute to the success of MEL.

6. MEL resources

The capacity to allocate staff time and financial resources to MEL activities.

The primary resource required is staff time. Funds might also be needed for items such as participant reimbursements, software or hardware, materials, training, or consultant fees. As an important but often non-urgent activity, it can be difficult for MEL to make it to the top of the priority list unless adequate budget and staff time are allocated and protected.

7. MEL culture, support & development

The ways in which organisational and team culture enable or restrict reflection and learning, and the processes through which MEL capacity is built.

A culture of trust and openness is an important foundation for teams to be able to have honest conversations about what is working well and what needs to be improved. Teams which devalue data and just want to "do the doing without wasting time on thinking too hard about it" will struggle to engage with MEL. Staff and stakeholders may therefore require a range of supports in order to facilitate their participation in MEL activities and develop evaluation capacity.

Lens 3: MEL life cycle

The third lens includes four domains which highlight strengths and weaknesses in relation to different phases of designing and implementing MEL.

8. MEL design

Processes of planning and preparation which establish frameworks and tools for MEL.

During the design phase, stakeholders decide what the focus of MEL will be and what questions it aims to answer, what methods will be used, and what outcomes and indicators are relevant. Assessing capacity in this domain involves considering the extent to which MEL design work has been undertaken, and the availability of skills and knowledge which enable effective design.

9. Data collection

Processes through which data is collected, stored and managed.

Data collection processes need to fulfil operational, compliance and MEL requirements. Effective data collection occurs when organisations clearly delegate responsibility for data collection tasks, provide clear and up-to-date training and instructions, use fit-for-purpose tools, monitor and improve data quality, and have sufficient resources to enable these activities to occur in a timely manner. Issues of ethics and risk are also important, including informed consent, privacy, data security and any impact on client experience.

Assessing capacity in this domain involves considering the extent to which effective and ethical data collection is in place, and the availability of skills and knowledge which support data collection.

10. Analysis & reporting

Processes through which data is analysed to extract information of value to stakeholders, and the ways in which this information is communicated.

Reporting often requires substantial technical skills in data analysis. It also involves sense-making or interpretation, which depends on knowledge of the context. Effective reporting requires clear frameworks, proper data definitions and sound tools and templates, and is targeted to the needs of specific audiences. Assessing capacity in this domain involves considering the reporting systems already in place and their level of functioning, as well as the skills and knowledge required to develop and use these systems.

11. Use & integration

Processes through which information generated through MEL is used by the organisation and external stakeholders to achieve goals.

In this important phase of the life cycle, stakeholders use MEL findings to achieve organisational objectives. This includes internal use - for coordinating and overseeing operations, planning, developing and improving services, and allocating resources - and externally for promoting the organisation’s work, for advocacy, and seeking funding. Assessing capacity at this phase involves considering the processes in place for using MEL findings, and the skills and knowledge available to drive these processes.

Evaluation Needs Assessment process

Data collection

Assessing an organisation’s capacity on the 11 domains requires the collection and analysis of data. We need to understand what stakeholders are thinking about MEL, what MEL-related processes are currently in place and how well they are working, and what the organisation’s internal and external MEL objectives and requirements are.

There are a range of options for collecting data from stakeholders about evaluation capacity and needs. These include surveys, interviews, focus groups, workshops, review of documents, data audits, and observation. Organisations should choose the methods that best fit their budget and context. A sample staff survey is available from Lirata, which organisations can use and tailor as required.

Think carefully about which stakeholders are involved in the Evaluation Needs Assessment process. A Needs Assessment can be a powerful way to understand the diversity of experience and needs that people bring to MEL, and a chance to engage people in the journey and to begin to shift attitudes about MEL. For these reasons, we recommend maximising participation, especially from staff.

Analysis and reporting

Evaluation Needs Assessment findings can be reported using the 11 domains as a structure.

For each domain it is useful to provide:

  • A summary assessment of "Where are we on our journey", using the selected rating scale.
  • Identified Strengths, Challenges, Opportunities and Risks, which are likely to influence the success of evaluation capacity building initiatives.
  • A summary of stakeholder views - including illustrative comments, and quantitative data (e.g. scalar responses from surveys) where relevant.
  • Recommendations for strengthening evaluation capacity.

As always, while the written report is valuable, engaging organisational leaders and stakeholders in discussing the findings will be of equal or even greater value. This can help ensure findings are well understood and that needs can be addressed cohesively.

Action plan

Having reviewed the Evaluation Needs Assessment findings, develop an action plan which prioritises the next steps in building evaluation capacity for your organisation.

  

Assistance with evaluation capacity building

Lirata Ltd assists service providers, advocacy organisations and funders to build their MEL capacity and undertake effective monitoring and evaluation.

For further information or assistance, please contact the Lirata team:

Phone: +61 (0)3 9457 2547
Email: This email address is being protected from spambots. You need JavaScript enabled to view it.
  

Download

Evaluation Needs Assessment: A foundation for building MEL capacity (PDF 309 KB)

Related articles

Evaluation needs assessment in an Aboriginal Community Controlled context

Evaluation Needs Assessment: Staff survey template

Theory of Change: A tool for more effective change making