Improving Child Development Outcomes in a Child Welfare Agency: Evidence we do make a Difference

By Dr. Michael O'Brien

In 2004, Family and Children‘s Services of Renfrew County expanded its mission from the protection of children from maltreatment to also include the promotion of their actual development. This was a formidable challenge since it necessitated a review of what was required for children to accomplish this additional part of the mission. It also meant that the agency had to define what the desired child welfare outcomes should be in order to determine whether the new mission had, or had not, been accomplished. In retrospect this transition and the steps required at each point of decision-making helped us immensely and may be of interest to other Children‘s Aid Societies contemplating a similar journey.

In selecting outcome indicators, a number of factors were considered in our initial deliberations. They are outlined below:

1. Family and Children‘s Services of Renfrew County took a close look at what we wanted to measure and why.

2. The goals of Ontario‘s Child Welfare Transformation Agenda were factored into the selection of outcomes. This meant we were committed to making risk reduction and safety a paramount priority while also placing greater emphasis on meeting the needs of children and parents, and working with them in a more collaborative fashion.

3. Client satisfaction, building on client strengths, and client perceptions about their needs and problems, were also areas we felt required more attention.

4. Decisions had to be made about assessment requirements since much of what was measured was influenced by the selection of particular assessment instruments.

5. The practicality and utility of gathering and then analyzing the data was a key consideration.

6. We took into account that our resources and influence are more limited for children living in the community than for children in care. Thus we knew that our outcome goals would have to reflect this limitation.

7. We were committed to some re-design of services in order to attain better outcomes, if that was required.

THE OUTCOME SELECTION PROCESS

The child protection outcomes research on child well-being pertaining to children living with their families is limited (McDonald, Lieberman, Poertner, & Hornby, 1989; Fallon, 1998; Poertner, McDonald, & Murray, 2000; D‘antrade, Lemon Osterling, & Austin, 2008). Safety, permanence, and well-being are the three broad categories on which a consensus exists, but what the outcome indicators should be and how those indicators should be measured has not been widely agreed upon. We selected the following indicators to measure outcomes: child safety, child behaviour, school functioning, child‘s health and development, and the child‘s sense of competence. We did so based on our review of the literature on child welfare outcomes, resiliency, and client views about their needs. We also examined developments in the United Kingdom as they had begun a movement towards needs-based approaches to child protection earlier than North America. The following sources summarize some of the options considered in choosing the outcomes that we wished to measure:

Child Welfare Matrix (Trocmé, N. et al.,1998)

1.Child safety (recurrence of maltreatment, serious injuries, or death)

2.Permanence (placement rate, moves in care, time to achieve permanence)

3. Child well-being (school performance, child behaviour)

4. Family and community support (family moves, parenting capacity, ethno-cultural placement matching)

Key Messages from Resiliency Research (Luthar, 2003; Masten, 2001; Rutter, 1979, 1995)

1. Strengthening caring ability of parents

2. Facilitating social support for the child outside the family

3. Promoting good school adjustment helping the child to develop a sense of competence

Client Perceptions of Need (Frencsh, & Cameron, 2003; Magura, 1986; O‘Brien, 2004; Packman, 1986; Williams, 1997)

1. Problems with stress-need for emotional support

2. Child behaviour

3. Instrumental needs-help in obtaining groceries, recreation, housing, etc.

Assessment Framework for Children in Need of Intervention (United Kingdom)

1. Seven dimensions of child development from Looking After Children

2. Parenting capacity

3. Family and environmental problems

Common Language Framework (indicators developed by Darlington Social Research Unit)

1. Living situation

2. Family and social relationships (child and parent)

3. Child‘s social and anti-social behaviour

4. Child‘s physical and psychological health

5. Education and employment for children and youth

THE SELECTION OF MEASUREMENT TOOLS

Next the agency looked at how we should go about measuring the indicators we had selected.

Conceptually it was important to give greater emphasis to the assessment and addressing of the child welfare 'needs‘ of children.

Research methods pertaining to the measurement of a construct such as need, favour the use of more than one instrument (McKillip, 1987). Any one method, because of its limitations, will only partially measure the construct. The use of multiple methods, though more costly, tends to eliminate bias and expand the level of understanding by capturing more than one perspective. For example, service providers and service users each have their own values which will be expressed when asked to define user needs. For the reasons outlined above, both objective and subjective methods are required to measure needs attainment.

Bradshaw‘s taxonomy of needs provides four approaches for measuring need that have stood the test of time since identified in 1972. The taxonomy includes both objective and subjective measures:

1. Expressed need is the demand for service by consumers.

2. Normative need is a standard or level set by the experts or professionals as desirable.

3. Felt need is a person‘s self-perception of his situation.

4. Comparative need is a need that is assessed by comparing those receiving a service with those in a community who have similar characteristics but are not receiving the service (Bradshaw, 1972).

We chose to focus on measuring normative need, a more objective measure, while also tabulating felt need, which is a subjective measure. Fortunately on a number of levels, both of these types of measures were achieved by adapting the everyday work performed by child protection workers to this data collection. We accomplished this by selecting instruments that measure child development, training staff in their use, and supporting them. By incorporating the collection of evidence garnered about child development into front line practice we hoped to build a culture that made child development as important as child safety.

Furthermore, in order to provide staff with some additional tools we created a behaviour management program, and secured a Trillium grant, to be used to enroll children and youth on child protection caseloads in social and recreational activities and in the KUMON supplemental education program. The instruments we use include the following:

The Brigance Developmental Screen (Glascoe, 2002)

The Brigance Developmental Screen, completed by the agency‘s nurse, is a tool used to assess the development of children from birth to 6 years old. It consists of eight scales designed to be administered during the various developmental stages that occur up to the age of 6.

The Strengths and Difficulties Questionnaire (Goodman, 1997)

The questionnaire is a mental health screening tool which we administer when a case opens and then every 6 months until the child protection cases closes. It is completed by the child protection worker using responses provided by parents.

Threshold, (Little, Axford, & Morpeth, 2003)

Threshold, a decision-making tool developed at the Dartington Social Research Unit, was designed to assess levels of impairment to child development, and to predict future impairment. It is completed by social workers at case opening and closing. Threshold provides a structure for making decisions about impairment. It encourages workers to collect data about a child‘s health and development (physical, social, behavioural, intellectual, emotional) and to analyze it in a logical sequence in order to arrive at judgment about impairment. Then, through the examination of the interaction of risk and protective factors affecting a child in each of the major developmental dimensions, the child protection worker is assisted to make predictions about future impairment and to determine the child‘s developmental needs.

Kidscreen (Ravens-Sieberer, 2006)

Kidscreen, developed through the World Health Organization, is an instrument used to ask children between the ages of 8 and 18 how they perceive their quality of life. It is administered early in the life of a child protection case and then 10 to 12 months later.

The Parent Outcome Interview (Magura, & Moses, 1986)

The Parent Outcome Interview is completed at the closing of a child protection case. Both the Parent Outcome Interview and Kidscreen are completed by a research assistant.

FINDINGS

1) The Strengths and Difficulties Questionnaire (SDQ) shows a statistically significant improvement in levels of hyperactivity and conduct problems 6 months after the opening of a child protection case (N=457). Statistical analysis of our data suggests that hyperactivity is the key predictor of the overall SDQ score. Slight improvement is also seen in the overall total difficulties score. It is also important to report that on the pro-social subscale, children and youth are within the norm when initially assessed at case opening (N=845).

2) Table 1 shows the levels of impairment to child development and predictions of future significant impairment at case opening as determined by social workers completing the Threshold decision making tool (N=733). We have found that at closure we see both a substantial decrease in the number of children with a child development impairment (decreasing from 18.8% to 8%) and the number for those whom a significant impairment to development is predicted (decreasing from 35.5% to 16%) (N=196).

Table 1

We have also found by cross-tabulating the initial Threshold with the one completed at case closing, that Threshold has a statistically significant better than expected rate of being able to predict impairment to child development. These findings suggest that Threshold does hold promise for being able to predict future impairment. The importance of this finding is substantial because we now have a tool that can effectively assist us in understanding how child protection issues are likely to impact a child. Combined with professional knowledge and expertise, the Threshold tool can be most valuable in determining how intense and extensive the child protection intervention should be.

3) The Kidscreen quality of life measure shows that 1 year after first being administered, children and youth are reporting a significant improvement in their quality of life (N=80). At mean-time 1 they are found to be within the norm for physical well-being, self-perception, autonomy, social/peer support, and feelings about their school environment. By mean-time 2 they remain within the norm for the dimensions already mentioned but now also fall within the norms for psychological well-being and day-to-day moods and emotions.

Table 2

In respect to self-perception, they substantially exceed the norm by mean-time 2. At mean-time 1 they fall into the 60th percentile and at mean-time 2 fall into the 70th percentile. A t-score of between 49 and 50 is considered to be the norm for the general population of children and youth in Europe where it has been widely tested. Table 2 shows the time series scores for each of the 10 dimensions of Kidscreen.

The reliability of the above data is reinforced by comparing it with the 260 initial Kidscreens completed with children and youth in the agency which show similar results to the mean-time 1 data depicted in Table 2.

4) Outcomes observed through the use of the Brigance Developmental Screen show no improvement 6 months after the initial Brigance assessment; children remained at the 63rd percentile. Of the 130 Brigance assessments of children on child protection caseloads it was found that 15.4% are designated as at-risk regarding their development, 51.5% as normal, and 32.3% as advanced. It is primarily due to failure to attain learning milestones that impedes progress. In comparison to children admitted into care, children on child protection caseloads show better development at the initial assessment, but unlike children in care, do not tend to make as many developmental gains.

5) The Parent Outcome Interview is being used as a client survey at closing. On most items regarding the clients‘ relationship with the child protection worker, about 75% of clients provided a positive report. Clients are also asked about what changes have occurred to the problems they were experiencing at the time the case was opened. On questions having to do with their child‘s problems and their own stress levels, reports of an improvement have been encouraging (N=60).

CONCLUSIONS

The outcomes described rely on parents, social workers, and children and youth as sources of information. The triangulation resulting from combining a number of sources and instruments for the collection of our outcome data strengthens the credibility of the findings. The instruments involved in the data collection have been used for varying lengths of time, ranging from 2 to 5 years.

The evidence to date supports the following conclusions:

1) A modest but significant number of children improve with respect to hyperactivity and conduct issues;

2) Impairment to child development and predictions of future impairment to child development are greatly decreased by the time the case is closing;

3) Children and youth often feel better about themselves after having been involved with our child protection services;

4) At the closing of a case, a high percentage of parents report a positive working relationship with their social worker. Those who had concerns about the functioning of their children often report some improvement; and

5) Children under the age of 6 on child protection caseloads do not progress in their readiness for school during the course of the agency‘s involvement. No increase in maltreatment has occurred during the time that our child development strategy has been implemented.

Although the focus of this paper is primarily to report on the effectiveness of the agency‘s routine child protection intervention, we have gleaned some insight into how better outcomes can be achieved. Through the use of the clinical instruments that have been described we have accumulated substantial sets of data; the data show that many of the children and youth we encounter benefit from a variety of protective factors, in their lives to counter balance the risks to which they are being exposed. Given the evidence about protective factors, it may not be surprising to find the outcomes we have reported. We now know enough about outcomes, strengths, and protective factors to predict that it is likely, with sufficient support, the development of many of the children on the agency‘s child protection caseload can be improved.

Although we cannot definitively interpret the findings, it is clear that many of them are modestly encouraging. The perceptions of children and youth about their quality of life are very encouraging and warrant further research. It would be of interest to determine, for example, what role subjective measures of well-being, like Kidscreen, should play in child welfare assessment since they offer a perspective not captured by clinical instruments that measure functioning. Additionally, it brings to question, how the perceptions of children and youth about their quality of life impact their future capabilities and functioning. Lastly, can the positive developmental improvements emerging from the results of the various assessment instruments used by the agency sustain themselves once child protection intervention is concluded?

In addition to studying the questions raised above, further research is required to generalize the findings beyond the Family and Children‘s Services of Renfrew County, and to uncover the processes that most directly affect outcomes. We believe these outcomes demonstrate the value of good social work practice in child welfare, and suggest the importance of having a well-trained, committed, and effective professional social work staff.

From the perspective of child protection policy, the findings support the utility of providing the resources not only to keep children safe, but also to promote their well-being. Investing in better outcomes in the child protection system is a complex and challenging undertaking that requires commitment from policy makers, practitioners, administrators, and researchers if we are to succeed. Our experience has been that the investment is worth the effort.

ABOUT THE AUTHOR:

Dr. Michael O‘Brien is the Director of Research and Quality Assurance at Family and Children‘s Services of Renfrew County and an assistant professor at the School of Social Work at Dalhousie University.

REFERENCES

Bradshaw, J. (1972). The concept of social need. New Society, March 30, 640-643.

D’antrade, A., Lemon Osterling, K., & Austin, M. (2008). Understanding and measuring child welfare outcomes. Journal of Evidence-Based Social Work, 5(1), 135-156.

Fallon, B., (1998). Outcomes literature review: A preliminary review for the client outcomes in child welfare project. In Thompson, J. & Fallon, B. (eds), The first Canadian roundtable on child welfare outcomes. Toronto: University of Toronto Press.

Frensch, K., & Cameron, G. (2003). Bridging or maintaining distance: A matched comparison of parent and service provider realities. Waterloo (ON): Wilfrid Laurier University.

Glascoe, F. P. (2002). Technical Report for the Brigance Screens. (3rd ed.). North Billerica (MA): Curriculum Associate, Inc.

Goodman, R. (1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology, Psychiatry, and Allied Disciplines, 38, 581-586.

Little, M., Axford, N., & Morpeth, L. (2003). Threshold: Determining the extent of impairment to children's development. Dartington (England): Warren House Press.

Luthar, S. (Ed.), (2003). Resilience and vulnerability: Adaptation in the context of childhood adversities. Cambridge: Cambridge University Press.

Magura, S., & Moses, B.S. (1986). Outcome measures for child welfare services. Washington (DC): Child Welfare League of America, Inc.

Masten, A.S. (2001). Ordinary Magic: Resilience processes in development. American Psychologist, 56, 227-238.

McDonald, T., Lieberman, A., Poertner, J., & Hornby, H. (1989). Child welfare standards for success. Children and Youth Services Review, 11, 319-330.

McKillip, J. (1987). Needs analysis-Tools for the human services and education. Newbury Park (CA): Sage Publications.

O’Brien, M.J. (2005). Exploring and comparing client perception of need and social worker perception of risk: A key to improved intervention in cases of child neglect. (doctoral dissertation, McGill University, Canada). AAT NR12920.

Packman, J. (1986). Who needs child care-Social work decisions about children. Oxford: Basil Blackwell Ltd.

Poertner, J., McDonald, T., & Murray, C. (2000). Child welfare outcomes revisited. Children and Youth Services Review. 3(9/10), 789-810.

Ravens-Sieberer, U. (2006). The Kidscreen questionnaires. Berlin (Germany): Pabst Science Publishers.

Rutter, M. (1979). Protective factors in children’s responses to stress and disadvantage. In J.S. Bruner & A. Garten (Eds.), Primary Prevention of Psychopathology (Vol. 3, pp. 49-74). Hanover (NH): University Press of New England.

Rutter, M. (1995). Psychosocial adversity: Risk, resilience, and recovery. Southern African Journal of Child and Adolescent Psychiatry. 7(2), 75-88.

Trocmé, N., Oxman-Martinez, J., Moreau, J., Fallon, B., MacLaurin, B., Schumaker, K., et. al. (1998). Client outcomes in child welfare: Outcomes framework. Toronto: University of Toronto.

Williams, M. (1997). Parents, children and social workers. Aldershott: Ashgate Publishing Company.

Next article: Book Review - Supporting Young Parents: Pregnancy and Parenthood Among Young People in Care

Previous article: Providing S.M.I.L.E.S to Children of the Waterloo Region Whose Caregiver has a Mental Health Diagnois