As the COVID-19 pandemic continues its global march, overwhelming even the most sophisticated health systems, there is an urgent need for real-time, reliable evidence to allow for rapid responses to changing and challenging operating settings.
This makes Monitoring Evaluation Research and Learning (MERL) more important than ever, and we must adapt approaches and tools to remain relevant. Tetra Tech’s in-house Research Monitoring and Evaluation (RME) team has developed the following advice for consideration:
Do No Harm
Practitioners should also: In the current COVID-19 environment, the health and safety of staff and communities remains the top priority when considering research, monitoring and adaptation activity. In line with social distancing recommendations, face-to-face contact should be avoided or carefully managed in line with the latest health advice.
- Be mindful of the desire to collect additional information on the impact of COVID-19, and that any data collection is not overly onerous and is directly related to the project’s scope of work
- Recognise that some populations will be more difficult to reach, and unintentionally excluded, and that some sensitive information (e.g. increased domestic violence) is less likely to surface
- When using mobile phones to collect data, ensure that any changes will not be a burden for beneficiaries i.e. to buy credit to participate, and
- When working remotely, remember the importance of properly storing and safeguarding data collected.
Resources
AAAS, Do No Harm: An Ethical Data Life Cycle
‘Go lean’ and enable adaptive management approaches
There should be a greater emphasis on: generating real-time data about activities as well asthe operating context; collecting formal and informal evidence; creating short feedback loops; facilitating collaborative sense-making which includes multiple perspectives; and documenting and communicating evidence-based adaptations. During COVID-19, it is critical that any MERL activity adds value and provides timely decision-making information to frontline workers, local managers, decision makers and funders. Practitioners should review MERL frameworks based on changes to program objectives, delivery models, timeframes and resource allocations.
Resources
ODI, Supporting adaptive management: monitoring and evaluation tools and approaches
Embrace technology
Software solutions and online platforms easily enable remote communication directly with partners, beneficiaries and stakeholders. It is important to rapidly revise and customize approaches and research tools for remote engagement:
- Contact participants through a variety of media (email, phone, social media);
- Develop a process for obtaining virtual consent;
- Ensure questionnaires are short and focused;
- Develop detailed guidelines for interviewers to manage the flow of discussions with limited or absent visual cues;
- Review protocols for data storage and validation.
Helpful tools:
- Phone surveys – Surveys can be conducted via short message service (SMS), interactive voice response (IVR) or computer-assisted telephone interviewing (CATI). The survey approach should consider the literacy of the population, their engagement with the program and motivation for participating, the length of the survey, the complexity of questions and the sensitivity of information sought.
- Group discussions – Platforms such as Skype and Microsoft Teams are useful to enable virtual discussions. Use the ‘message’ function to capture attendance, feedback and people’s reactions to topics discussed and be mindful of information security when choosing your platform.
- Interactive workshops – Interactive, participatory platforms such as Mentimeter and Slido can be useful in obtaining real-time feedback from workshop participants through ranking and rating functions, live polls, quizzes, word clouds and real-time questions and answers (Q&A).
Building credibility and rapport with participants is critical to generating useful, reliable information. Poor connectivity, long delays, confusion and multiple attempts at connection will affect the quality of your data – not to mention everyone’s mental health! Prepare a Plan B.
Resources
J – PAL: Best Practices for Phone Surveys
RTI, Collecting Data with Mobile Surveys in Low- and Middle-Income Countries During COVID-19
Mathematica, Tips to Quickly Switch from Face-to-Face to Home-Based Telephone Interviewing
Better Evaluation, Mobile Data Collection
Harness local capacity
Remember, it is imperative that field staff are provided up-to-date health information, clear advice on the risks and opportunities of engagement, techniques for social distancing and the opportunity to refuse to undertake action they believe put themselves, their community or program participants at risk. Current travel restrictions present an opportunity to build the networking and skills capacity of locally based researchers and evaluators. The role of the international MEL adviser will shift to that of a trainer, mentor and critical friend, while local staff will lead on data collection, collation and analysis. Locally led activity will result in to more locally led decision making as project teams are encouraged and empowered to work adaptively.
Resources
Guidelines For Adapting Third-Party Monitoring in the Context of the COVID-19 Outbreak
Understand the needs of the most vulnerable
Rapid appraisal: An approach which draws on multiple evaluation methods (including interviews, community discussions, mini-surveys, direct observation, community mapping and secondary data collection) to quickly collect data to understand issues and inform decisions.COVID-19 pandemic and response is likely to exacerbate pre-existing inequalities and disproportionately impact already vulnerable groups: it is more important than ever that their specific needs and priorities are understood and addressed:
- Rapid gender analysis: A tool used by humanitarian responders to gather essential information about gender roles and responsibilities, capacities and vulnerabilities and then generate programming recommendations.
- Action research: A continuous spiral of steps which involves continuous action, assessment and adaptation becomes an important feature of the way activities are implemented and improved. Implementers and beneficiaries own this process.
- Real-time evaluations: Participatory evaluations designed to provide real-time feedback in order to inform operational management and decision making.
Resources
USAID TIPS: Using Rapid Appraisal Methods
CARE, Gender in Emergencies Guidance Note
Handbook for Participatory Action Research, Planning & Evaluation
ALNAP (2009) Real-time Evaluations of Humanitarian Action – an ALNAP Guide
Use remote monitoring approaches and techniques
Remember that remote monitoring is the norm when working in fragile and conflict affected regions. Here are some lessons learned:
- Take a reality check – Be realistic about what is possible. This refers to the outcomes, level of data, frequency of monitoring and reporting that can be expected from MERL systems and partners
- Partnerships matter (even more than usual!) – Investing in high quality partnerships with trusted partners becomes even more important when on-the-ground monitoring and evaluation is not possible. Make sure you are communicating regularly (ideally using video conferencing facilities), sharing information and maintaining accountability for any changes that might affect the partnership
- Only monitor what matters – Focus on fewer, higher quality indicators. This acknowledges the costs and risks associated with collecting data (financial cost, beneficiary impact, security/health risks in collecting data etc)
- Get creative – Investigate using innovative confirmation methods in complex environments – such as geographical information system (GIS) mapping, big data approaches, or sentinel indicators drawing on large publicly available data sets to track change at the system level, etc.
- Unintended consequences – Put extra effort into monitoring for unintended consequences. Adaptive management and politically aware approaches (e.g. Strategy Testing) can be helpful with horizon scanning to ensure you are actively managing risks
- Collaborate – Work with others to see what’s possible in terms of sharing data, information and approaches, joint or third-party monitoring etc
Resources
Partnership Brokers Association, Remote Partnering Work Book
GSDRC, Approaches to remote monitoring in fragile states
Current example:
Tetra Tech worked with the Afghanistan team at DFAT to develop a performance assessment framework to assist with and management of the Afghanistan aid program. An overview of the approach to remote monitoring is provided on the AES website
Focus on the data you already have
Literature reviews form an important part of most formal evaluations, but are rarely used often enough to develop theories that drive change or assess progress.
Now is the time to examine what relevant data sets and information you could share with partners (subject to confidentiality arrangements) while analyzing existing data to identify and share lessons learned.
Example: Tetra Tech has been commissioned to evaluate ten years of New Zealand’s investment in the Solomon Islands. The original scope of work involved significant consultation with stakeholders in Wellington and the Solomon Islands. In light of COVID-19, the team will now focus on a desk-based review to: summarise New Zealand’s investments, networks and reach in-country; highlight key achievements and the factors which contributed to success; consolidate lessons learned that should inform future programming and outline key considerations for New Zealand’s ongoing country-programme. This will directly support New Zealand’s response to the COVID pandemic in the Solomon Islands by providing the evidence base for decision making.
Keep communicating!
However, without effective, transparent communication both within, across and between teams your program outcomes won’t be as effective and efficient as planned. Remember, monitoring and evaluation activity is not only possible, but critical during COVID-19. Rapidly collaboration and communicating evidence-based research and data will keep staff safe, keep partners abreast of the evolving operating environment and enable decision makers to meet the needs of the most vulnerable
Stay well.
For further information, contact:
Catriona Flavel,
Senior Principal, Research Monitoring and Evaluation
