Tsitsa PMERL - A Departure from Business as Usual
While monitoring and evaluation (M&E) is widely recognised as important or even essential for restoration and development initiatives, the way in which M&E is conventionally implemented and reported often does not fulfil its promised potential (Woodhill, 2007). This is in large part because it does not support learning and strategic adaptive management (Taylor & Soal, 2003; Bornstein, 2006;
Mueller-Hirth, 2012; Kachur et al., 2016; Mudau, 2020) for reasons including the following:
M&E systems are often treated as static; something to adopt at the start of a project and then simply implement in the same way through the life of the project. By not taking into account the complex nature of interventions and contexts in which changes inevitably occur over time, indicators and targets become outdated and implementers find themselves having to “work around” them.
Accounting for expenditure and compliance with standards are often the exclusive focus, if donors and government default to performance auditing (Smith et al., 2019), focus on inputs, activities and outputs, and impose this focus to the exclusion of process- and principle-oriented M&E.
It is standard practice to report almost exclusively against quantitative indicators. This may enable the aggregation of data on various scales. However, large scale aggregations are seldom contextual enough for learning, and where M&E is exclusively about quantifiable progress, the opportunity for (second-loop) learning is greatly reduced. Reports are likely to omit unexpected outcomes, failures and mistakes (Mudau, 2020), whereas reporting and reflecting on disruptions and dissonances are vital for learning (Schulz, 2010; Wals, 2007).
In a separation between monitoring and evaluation, monitoring (the routine collection of data) is seen as the task of project implementers, while evaluation (sense-making based on the monitoring data) is done by external experts, midway through or at the end of the project. This excludes the implementers from the sense-making that could lead to ongoing learning towards improved practice (Woodhill, 2007).
The M&E system of the Tsitsa Project (TP) was explicitly designed to address these and other limitations of conventional M&E. The acronym “PMERL” reflects some of the changes: “Participatory”
M&E, with a strong emphasis on “Reflection and Learning” as part of and supported by integrated monitoring and evaluation processes. These design features are linked to the TP’s theory of change and strategic objectives, including its transdisciplinary and adaptive approach, and its intentions to build agency and innovate. It is also fundamentally linked to the understanding of restoration programmes as taking place in and through complex Social-Ecological Systems (SES). In complex SES, the pathways to success need to be worked out during and through action-taking and reflection. The dynamic, largely open-ended nature of contexts, featuring complex feedback loops and emergent properties, requires real-time learning. Such learning must be fed and informed by real-time data and less formal observations, and shared reflection on what this information means.
Embedding reflection as a practice within all the partner organisations (that is, a participatory approach) is a foundation for learning and the essential ongoing feedbacks needed for adaptive management. The TP’s PMERL processes connect data and sense-making to improve understanding
The Tsitsa Approach: Supporting Information (Appendix 2 to 7)
41 of “what is happening” by beneficiaries, implementers and sponsors, all along the way. This is done by collating and organizing data and observations, facilitating conversations, making information available when needed, and creating opportunities for reflection, joint evaluation and synthesis. There is no blueprint for exactly how to do this, and the TP continues to refine the process, as outlined next.The Development and Practical Features of PMERL in the Tsitsa Project
Early in the Tsitsa Project (2015) catchment residents were contracted to contribute to monitoring data (Bannatyne et al., 2017) and project leads developed strategic objectives and a theory of change (Fabricius et al., 2016). With the intention to develop a more extensive participatory M&E system, TP leadership in 2017 engaged with experts who had experience in designing a complexity-sensitive M&E system for another catchment-based programme, namely the RESILIM-O programme implemented by the Association for Water and Rural Development (AWARD) in the Limpopo-Olifants Basin. The experience from implementing and refining AWARD’s MERL system informed a PMERL plan for the TP (Botha et al. 2017). In the first years of its implementation, the TP PMERL plan was refined and detailed (Rosenberg & Human, 2018; Human et al., 2019), thus addressing the first limitation of ‘static’ M&E systems noted above.
A key development was the establishment of a PMERL team (including an M&E Coordinator, Hanli Human, succeeded by Karen Kotschy) to drive and support the integration of PMERL into other TP processes. Also important was situating the PMERL team within a Knowledge and Learning group (called a Community of Practice or CoP in TP parlance, to underscore the collaborative and ongoing innovative nature of the work required). The Knowledge and Learning CoP drew together PMERL, knowledge management and capacity development functions into a group that met regularly and also interfaced regularly with the Project Coordinator. This allowed for M&E, reporting and information management decisions to be made collectively, which is regarded as best practice internationally (e.g.
Villaneuva et al., undated).
The ongoing conceptual and strategic guidance provided by Dr Harry Biggs and Prof Eureta Rosenberg have proved important for maintaining institutional memory and linking work to the broader context.
Prof Rosenberg’s research on evaluation for learning in the Environmental Learning Research Centre at Rhodes University has been one of the key ways in which the experience from other complexity sensitive projects has been shared into the TP. Under Prof. Rosenberg’s supervision, Human (2019) conducted research that informed the development of social indicators for the TP, while Mtati (2020) studied the experiences of the catchment-based monitors, and how the benefits of citizen-based monitoring could be expanded.
The PMERL team worked with other TP CoPs on the participatory development of indicators (Human, 2019), an extensive process which involved valuable deliberations within and between project groups on their objectives and how they were going to achieve them (that is, theories of change on a sub- project level). This process is likely to be repeated biennially. Indicator protocols were developed for all indicators and roles and responsibilities of the various contributors to the monitoring system were agreed (these include the PMERL team, researchers, monitors, monitor managers, the Catchment Coordinator, and knowledge management staff). A PhD study in progress aims to identify essential variables for the TP monitoring system (Itzkin, 2019).
Early efforts to get TP team members to reflect on what they were doing saw the establishment of CoP quarterly reports and Back-to-Office field reports inspired by the successful Back-to-Office reports
The Tsitsa Approach: Supporting Information (Appendix 2 to 7)
42 developed by AWARD. Adjustments to the reporting templates were necessary, following feedback, to make this process more valuable and less onerous (templates were adjusted to make questions more open-ended).The PMERL team reads all quarterly reports produced by the CoPs as well as reflection reports and research papers produced by students and other partners. The experience of the community-based monitors is also included through reports compiled by the managers of the various monitor groups.
The PMERL team then produces synthetic reflection reports and papers, also in a consultative manner (e.g. Cockburn et al., 2018; Cockburn et al., 2019; Cockburn et al., 2020; Kotschy et al., 2020; Kotschy
& Mvulane, 2020). Quarterly rather than annual TP-wide reflection reports were instituted in 2020.
These reports include a detailed reflection of progress against the TP objectives and principles, indicator data, and a narrative summary of what has happened over the quarter. The intention is to provide a synthetic overview of data from many different sources, to highlight successes and challenges, to note progress towards the intended outcomes even if these are still a long way off, and to identify recommendations and actions that can be taken forward.
The KL CoP found, as AWARD did, that reports need to be complemented with other reflective activities and events to promote engagement with the findings (knowledge mediation). In the TP these include ‘Reflection and Wellbeing Teas’ and ‘Reflect and Reconnect’ meetings, preferably in person but now also online. The ‘Reflection and Wellbeing Teas’ arose out of an early finding that TP team members felt a need for support and connection with colleagues while navigating terrain that was often outside of their comfort zones (Cockburn et al., 2018). The PMERL team has also played a key role in supporting other cross-TP events such as Science-Management events, Science-Management- Society events, and the Research Colloquium. This support has included bringing reflection activities into these events to help with building the culture of reflection and learning, and supporting knowledge sharing and synthesis.
The overall Theory of Change has been a key focus for periodic project-wide reflection. Significantly, the TP Theory of Change reflects a social-ecological systems (SES) perspective in that it is not linear and its multi-dimensionality mitigates against narrow output-based indicators.
Most recently (2020) the TP PMERL processes were complemented with a wider set of social and biophysical monitoring processes with the addition of Community Liaison Officers and Eco-Rangers to the Citizen Monitors and Citizen Technicians already established in the catchment. Data is shared with sponsored cell phones and suitable software applications send rural residents’ findings to the university. A key next step, which has been delayed by COVID restrictions, is a sensible interweaving of the TP PMERL system outlined here, with the monitoring and reporting that government (Department of Environment, Forestry and Fisheries and Treasury) requires of restoration project implementers. This will be a fulfilment of the intention to have a hybrid model of complexity-sensitive approaches interwoven with mainstream M&E processes.
In summary, the PMERL system supports members to work according to the TP principles of SES thinking; transdisciplinarity; a collaborative, reflexive and adaptive orientation; participation; and a strong scientific evidence base. Emphasis is placed on developing appropriate feedback processes for strategic adaptive management and connecting these to subsequent actions, both overall and in different subsystems within the project. The key features of the system are captured in the PMERL acronym: participatory monitoring, evaluation, reflection and learning (with the “M” potentially also
The Tsitsa Approach: Supporting Information (Appendix 2 to 7)
43 addressing the issue of motivation and moral support for tackling complexity together...). Although at present less emphasis is being placed on the evaluation component, evaluation takes place informally in an ongoing way through the practice of reflection. The wealth of quantitative and qualitative data being collected in the TP will support and enhance more “conventional” evaluation processes in future.It should be clear from the above account that having a formal M&E plan is just part of the process, and that iterative work is necessary to integrate an innovative approach that matches a project’s particular features, to increase the possibility of optimal participation by all role players.
At the time of writing the Knowledge and Learning CoP consisted of six members:
• A CoP Coordinator (PhD level)
• A Capacity Development Coordinator (PhD level)
• A PMERL Coordinator (PhD level)
• A Knowledge & Learning Support Officer (Masters level, role divided between PMERL and knowledge management and mediation)
• Two Strategic Advisors
The CoP Coordinator, Capacity Development Coordinator and one of the Strategic Advisors are partly supported by funding from Rhodes University as they perform other roles within the university. The PMERL Coordinator, Knowledge & Learning Support Officer and the other Strategic Advisor are supported directly from TP funds.
In the experience of the group, successful functioning of the KL CoP has been enabled by the distinct but overlapping roles of the different members, which has allowed flexibility in tackling the varied demands of the work. The PMERL function is distinct from but highly integrated with the project management, knowledge management and capacity development functions. The culture of mutual support and encouragement within the group is also experienced as highly valuable, given that what needs to be done is often not clear and requires constant re-evaluation against the changing context of the project and its strategic imperatives. It is very comforting to walk this journey with supportive colleagues!
The Tsitsa Approach: Supporting Information (Appendix 2 to 7)
44 ReferencesBannatyne, L., Rowntree, K., van der Waal, B. & Nyamela, N. 2017. Design and implementation of a citizen technician–based suspended sediment monitoring network: Lessons from the Tsitsa River catchment, South Africa. Water SA 43(3): 365–377.
Bornstein, L. 2006. Systems of accountability, webs of deceit? Monitoring and evaluation in South African NGOs. Development 49(2), 52–61. http://dx.doi.org/10.1057/palgrave.development.1100261 Botha, L., Rosenberg, E., Biggs, H., Kotschy, K. & Conde-Aller, L. 2017. Ntabelanga-Lalini Ecological Infrastructure Project (NLEIP): Participatory Monitoring, Evaluation, Reflection & Learning (PMERL) Framework. NLEIP Internal Report, Rhodes University, Grahamstown.
Villanueva, P., Gould, C. & Pichon, F. Undated. Routes to Resilience: Insights from BRACED Year 1 (Synthesis Paper). Building Resilience and Adaptation to Climate Extremes and Disasters (BRACED) Knowledge Manager.
Cockburn, J., Biggs, H., Rosenberg, E. & Palmer, C.G. 2018. Learning through reflective praxis: Lessons from integrated sustainability research with a governance focus in a complex social-ecological system, Eastern Cape, South Africa. Tsitsa Project Internal Report, Rhodes University, Makhanda.
Cockburn, J., Human, H., Rosenberg, E. & Biggs, H. 2019. Participatory Monitoring, Evaluation, Reflection and Learning (PMERL): Meta-Reflection Report 2018-2019. Tsitsa Project Internal Report, Rhodes University, Makhanda.
Cockburn, J., Human, H., Kotschy, K., Rosenberg, E., Wolff, M. & Biggs, H. 2020. Meta-Reflection Report 2019-2020. Tsitsa Project Internal Report, Rhodes University, Makhanda.
Fabricius, C., Biggs, H.C. & Powell, M. 2016. Research Investment Strategy: Ntabelanga and Lalini Ecological Infrastructure Project (NLEIP). Department of Environmental Science, Rhodes University, Grahamstown.
Human, H. 2019. Developing social indicators for the evaluation of natural resource management programmes using a capability approach in the Eastern Cape, South Africa. M.Ed. thesis, Faculty of Education, Rhodes University.
Human, H., Cockburn, J., Rosenberg, E., Lunderstedt, K. & Kotschy, K. 2019. Knowledge and Learning CoP: PMERL and Knowledge Management and Mediation Inception Document 2019-2020.
Environmental Learning Research Centre, Rhodes University, Makhanda.
Kachur, D., Soal, S. & Van Blerk, R. 2016. Stretching between learning and accountability: Experiences of South African non-governmental organisations. African Evaluation Journal 4(1), a71.
http://dx.doi.org/10.4102/aej.v4i1.71
Kotschy, K., Mvulane, P. & Cockburn, J. 2020. Tsitsa Project Quarterly Reflection Report (Q1), July 2020. Tsitsa Project Internal Report, Rhodes University, Makhanda.
Kotschy, K. & Mvulane, P. 2020. Tsitsa Project Quarterly Reflection Report (Q2), October 2020. Tsitsa Project Internal Report, Rhodes University, Makhanda.
The Tsitsa Approach: Supporting Information (Appendix 2 to 7)
45 Mtati, N. 2020. Towards realising the benefits of citizen participation in environmental monitoring: A case study in an Eastern Cape Natural Resource Management Programme. M.Ed. thesis, Faculty of Education, Rhodes University.Mudau, V. 2020. A study on reporting and learning in three Natural Resource Management Programmes in South Africa. M.Ed. thesis, Faculty of Education, Rhodes University.
Mueller-Hirth, N., 2012. If you don’t count, you don’t count: Monitoring and evaluation in South African NGOs, Development and Change 43(3), 649–670. http://dx.doi.org/10.1111/j.1467- 7660.2012.01776.x
Rosenberg, E. & Human, H. 2018. Tsitsa Project Participatory Monitoring, Evaluation, Reflection and Learning Inception Document, April 2018. Environmental Learning Research Centre, Rhodes University, Makhanda.
Schulz, K. 2010. Being Wrong: Adventures in the Margin of Error: The Meaning of Error in an Age of Certainty. Portobello, London.
Smith, L., Pophiwa, N. & Tirivanhu, P., 2019. Introduction in Blaser Mapitsa, C., Tirivanhu, P. &
Pophiwa, N. (Eds), Evaluation landscape in Africa: Context, methods and capacity. SUN Press, Stellenbosch.
Taylor, J. & Soal, S. 2003. Measurement in Developmental Practice: From the Mundane to the Transformational. Community Development Resource Association, p. 13, www.cdra.org.za.
Wals, A. (Ed) 2007. Social learning towards a sustainable world: Principles, perspectives, and praxis.
Wageningen, The Netherlands: Wageningen Academic Publishers.
Woodhill, J. 2007. M&E as learning: Rethinking the dominant paradigm: In J. de Graaf, J. Camerson, S.
Sombatpanit, C. Pieri & J. Woodhill (Eds.) Monitoring and Evaluation of Social Conservation and Watershed Development Projects (pp. 83–107). Enfield, New Hampshire: Science Publishers.