Vivienne Kennedy, Fiona Cram & Nan Wehipeihana, for Research and Evaluation Consultancy Limited, Wellington

Funder: Department of Child, Youth and Family

Timeframe: 2007-08

Ensuring Māori children and young persons realise their potential as Māori (TPK, 2000)

Capacity is ‘the ability of individuals, organisations and whole societies to define and solve problems, make informed choices, order their priorities and plan for their futures, as well as implement programs and projects to sustain them’ (Nair, 2003:1).  Capacity building is the process by which individuals, groups, organisations and society increase their capacity.  This is often defined within a context of sustainability and a societal ethic of care (Torjman, 1998). 

The Department of Child Youth and Family's (CYF) capacity building initiative – the Iwi Māori Provider Development Fund (IMPDF) – began in 2000 with the release of Māori and Iwi capacity building fund in the government’s budget.  The first evaluation of this fund described the implementation of the IMPDF along with preliminary outcomes (O’Reilly, 2004).  A clear finding from this evaluation was that providers wanted more freedom to decide what their capacity needs were and how their capacity was built.  In response CYF contracted the development of a capacity self-assessment tool, along with training and coaching support, so that providers could: self-assess their capacity needs, work out a development plan and apply to the IMPDF for funding. CYF also instigated a register of mentors, coaches and supervisors that providers could contract with directly (whereas previously providers had little choice from a small pool of coaches and supervisors who were contracted centrally by CYF).

Two rounds of training in capacity self-assessment have occurred to date (2005 & 2007) and two rounds of IMPDF funding have been distributed to Māori and Iwi providers following on from their capacity self-assessment, formation of development plans and application to the IMPDF. This scoping report, commissioned by Child, Youth and Family, contributes to an ongoing assessment of CYF’s efforts to build the capacity of Māori and Iwi providers of CYF social services and is a key input into scoping a final evaluation of CYF’s capacity building initiative – the Iwi Māori Provider Development Fund. 

Methodology

Data from the 2005 and 2007 IMPDF applications was anonymised and entered into a database for analysis (e.g., demographics, capacity scores, funding requested).  Overall trends were then examined.

Six providers selected from around the country were invited to participate in the present evaluation and all accepted.  Documentation analysis and in-depth interviews led to the development of a rich description of each provider’s engagement with, and development arising from, the IMPDF, including capacity self-assessment.  These provider narratives were then analysed for common themes.

Findings

The analysis of providers’ funding applications revealed that:

•  A total of 95 providers applied to the IMPDF in 2005; 69 providers in 2007.  Fifty providers applied in both years.
• In 2005, 33 percent of the providers applying to the IMPDF were from the Northern Region while 11 percent were from the South Island.  In 2007 32 percent of providers applying were from the Central (North Island) Region.
• Providers received 65 percent of the funding they applied for in the 2007 round compared to 48 percent in 2005.
• Providers’ ratings of the nine capacity areas ranged from ‘acceptable’ to ‘good’ in both 2005 and 2007. 
• Providers’ capacity scores changed only slightly from 2005 to 2007.

The key themes arising from the provider narratives were:

1. Capacity self-assessment: a worthwhile process. On the whole the providers found the capacity self-assessment process to be worthwhile, flexible, objective and empowering and describe the self-assessment process as a capacity building activity in itself. Issues raised related to the timeframe and the difficulty of getting staff together to undertake the assessment.
2. Engaging governance. Providers were able to identify and then facilitate the changes needed to ensure that their governance bodies were engaged and effective.
3. Quality management. Both the capacity self-assessment process and the capacity building undertaken by providers has enabled them to enhance their practice.
4. Professionalism. The IMPDF supported staff development, including practice supervision.
5. Solid foundations. The capacity self-assessment process and the IMPDF funding enabled providers to strengthen their infrastructure. This was recognised as essential to good service provision and providers.
6. More effective service provision. Providers were being recognised for their service provision capabilities.
7. Future horizons. Providers were seeking more continuity in funding and relationships with funders.

Māori and Iwi providers operate in a complex and dynamic enviroment, in terms of the communities they serve and the funding/politcal context in which they operate.  As well as having contracts for services with CYF, they typically have contracts with other government agencies, with the level of funding from these other agencies often being greater then the funding received from CYF.  Some providers also receive capacity building assistance from other government agencies for activities not funded by IMPDF.

This makes an evaluation of IMPDF particularly challenging, as it is difficult to determine the extent to which any changes in provider capacity, either positive or negative, can be attributed in part or fully to IMPDF, and/or if other factors are responsible for some or all of any change in capacity.  On the other hand, there is also evidence of CYF’s IMPDF funding contributing to specific organisational capacity gains particularly in the area of governance, organisational infrastructure and staff capability and professionalism.  In addition, describing and/or measuring any change in capacity does not address questions of the effectiveness of IMPDF.  How much or what level of change in capacity should be expected?  What constitutes an excellent, good, or poor outcome and how would this be determined?

In summary, any evaluation of the CYF IMPDF needs to take account of the range of factors that impact on providers and ultimately on the capacity building outcomes achieved by providers.  Attribution of change and the effectiveness of the fund are two key areas that should be built into the design of a final evaluation of the IMPDF.  Importantly, the evaluation should reflect the nature of capacity building as both an end in and of itself (i.e., strengthened organisational capacity), and a means to an end (i.e., better service delivery and client outcomes).

This progress report involving some preliminary evaluation activity was designed to set the scene for an outcomes evaluation of the CYF IMPDF initiative.  The preliminary evaluation activities have merely scraped the surface of the implementation of the capacity self-assessment by providers, and their outcomes from the IMPDF funding they received.  An outcomes evaluation of the IMPDF would reveal the full extent of the outcomes experienced by Māori and Iwi providers as a result of their engagement with the IMPDF.

In line with the research on the evaluations of capacity building, the findings from this preliminary evaluation and the evaluators’ knowledge of IMPDF its implemenation and operation from 2002 to 2007, it is recommended that:

• CYF proceed with a full outcomes evaluation of the IMPDF
• CYF fund providers’ capacity self-assessments towards the end of the current IMPDF funding cycle
• CYF support the development of Māori and Iwi providers’ evaluation capacity
• Any evaluation of IMPDF would build and strengthen the internal evaluation capacity of Māori and Iwi providers