Outcomes of a Supplemental Learning Program for Children in Care at Family and Children's Services of Renfrew County
By Dr. Michael O'Brien and John Rutland

Increasingly the child welfare system in Ontario is working actively to remediate the educational deficits of children in its care. As a variety of approaches are being used, it is important to evaluate the effectiveness of the programs being implemented. Family and Children’s Services of Renfrew County has been offering a supplemental learning program for children in care since 2000. We chose the KUMON math and reading supplemental program which is the largest of its kind in the world. The article will examine the outcomes achieved since we embarked on a partnership with KUMON Canada.

The KUMON method was chosen based both on the quality of the program and a belief that children in care would be able to achieve success with it. Our goal for each child enrolled in the program is to achieve grade level success. At the time of enrollment students complete a diagnostic test with their instructor. Using the results of these tests, an individualized program is established for each child. Students begin at a point at which they can complete the material comfortably. From there, students advance through the materials using a mastery learning approach. This means that students advance only when they have demonstrated mastery of the material. Daily practice is a key to success with this method. Typically the expectation is that a child will be enrolled in the program for at least one year. The method is structured to foster continuous improvement via the accurate and timely completion of worksheets. The children attend one of our learning centres twice a week to complete worksheets under the supervision of an instructor and assistants. The commitment of foster parents and social workers is essential as daily assignments that require about 20 minutes a day must be completed in the foster home. Typically we focus on children between the ages of four and 13, however, any high school student who wishes to attend is welcome. At any given time 50 to 60 children in care are enrolled in the program. Family and Children’s Services holds the franchise for Renfrew County. As KUMON is a supplemental education program that can be used for both remediation and enrichment, it can be of benefit to any child. In addition to offsetting the cost of the operation of our program, the involvement of fee paying students also serves to normalize the tutorial experience. We have 90 fee paying participants in the program. Upon entering one of our learning centres one sees no difference between the children in care and the fee paying students. It is remarkable to note how smoothly the learning centres operate when one considers the behavioural difficulties being experienced by so many of our children in care.

The fundamental question to be answered through the evaluative research has been to determine whether a significant improvement in academic skills could be attributed to the math and reading supplemental program. In addition to the goal of evaluating improvement in both academic achievement and skills, the evaluation design has also included the goals of evaluating the suitability of the program for children in care, satisfaction with the program from the perspective of children, foster parents and social workers, as well as any unintended changes in the emotional and behavioural functioning of participants. Quantitative outcomes have been evaluated by examining changes in Wide Range Achievement Test (WRAT) scores and math and reading grades at school. Qualitative outcomes have been addressed through surveys of children in care, foster parents and teachers, and through focus groups with the workers for the children. The outputs that have been considered are retention and progress in completing the levels within the KUMON program in comparison to fee paying participants both in Renfrew County and KUMON sites in Canada.

Ensuring a high level of retention in this supplemental learning program is critical to its success. The agency was aware it would face challenges around retention due to the emotional and behavioural difficulties faced by many children in care, distances in traveling to the program sites in the large geographic area of Renfrew county, and the likelihood that many children would be less than enthused about having more educational demands placed upon them. The average time spent enrolled in KUMON for children in care is usually about 20 months which is very comparable to the data collected by KUMON. The design of the program which includes self-directed learning, immediate success for any children who complete the assigned work, and regular awards and periodic special events all increase motivation and ultimately retention. Volunteer drivers are used in instances when foster parents are unable to bring their foster child to a KUMON site twice a week. Presentations have been made to foster parents and children’s services workers about strategies they might use to support children in care who are participating. At each site an instructor is available to provide one-to-one support to foster parents who are having issues with their foster child’s engagement with the program. The other output that has been measured since the program was launched has been the number of levels completed by children in care. KUMON is comprised of a number of levels of increasing difficulty beginning at the pre-school stage of math and reading and progressing to the level of math and reading required for secondary school graduation. We have found that children in care are completing a satisfactory number of levels, but at a moderately lesser number of levels than our fee paying participants.

School report card results and the WRAT have been used to measure the progress of children enrolled in KUMON over periods of time. The following school report card results depict reading marks for children enrolled in KUMON reading, and math marks for children enrolled in KUMON math. The times series data was extracted from the children’s files corresponding to the time period in which they were enrolled in our program, and each of the times represent consecutive report card results for the children.

Table 1

Table 2

A paired samples t-test taken at time 1 and time 4 for the reading grades found the progress to be statistically significant at the 99 percent level (N=24). A paired samples t-test performed on a larger sample of 48 children at time 1 and time 3 was not quite statistically significant, but did show improvement in reading grades. The math grades were not found to be statistically significant (N=26).

Administration of WRAT occurs each May and June. Depending on the date of entry of a child into the supplementary learning program the initial measure may or may not be a true baseline measure. Given that some of the children are not tested until they have been in the program for a period of time it is our view that it is likely their baseline scores would have been lower had they been tested immediately upon entry into KUMON. A standard score of 100 is the average within the general population of children. Our goal has been to elevate the WRAT scores of children in care up to and beyond the standard score. Tables 3 and 4 illustrate the time series results that have been obtained with the WRAT instrument.

Table 3

Table 4

A paired samples t-test performed at time 1 and time 3 found that change in reading scores to be statistically significant at the 95 percent level (N=20). A paired samples t-test performed at time 1 and time 2 found the change in math scores to be close to, but not statistically significant (N=24). There may be some discrepancy between the improvement in math WRAT scores versus the lack of improvement in math grades at school, but as the two samples do not each contain all the same children no discrepancy may exist at all.

Qualitative information about outcomes has been gathered from a number of sources. In June 2001, after the program had been running for a year, we surveyed teachers about their perceptions of children in care enrolled in the KUMON. The teachers stated that the children had usually improved in math if they were enrolled in our math modules and reading if they were enrolled in our reading modules. Persistence, work completion, and task focus were observed to have improved. Noticeable improvements in confidence were not reported by the teachers. In 2007, as part of a larger survey of foster parents, a set of questions about our supplementary educational program were asked. Of those who had experience with enrolling a foster child in KUMON a large percentage reported that the program had been beneficial and had improved their foster child’s school grades. More than half of that group saw an improvement in the self esteem of their foster children, while few noticed any improvement in behaviour that they felt could be attributed to success in KUMON. The foster parents’ perceptions about behaviour were confirmed by examining the Child and Adolescent Functional Assessment Scale (CAFAS) results for children in our care which showed no statistically significant improvements over time in total CAFAS scores related to the time period in which the children participated in our program. Children’s services workers have consistently praised the effectiveness of KUMON. In addition, discussions with the workers have also been helpful in understanding some of the barriers to attracting and retaining foster children in the program, such as transportation and the extra demands placed on foster parents who often supervise children in completing their KUMON assignments. Perhaps, some of the most important information comes from the children themselves. One youngster related with pride how she was the second last person standing in her class in an oral math contest. Without her involvement in our program this moment would likely never have occurred. Another child was proud of moving from being the weakest in her class in math to becoming the teacher’s math assistant. One of the children would break down in tears any time there was a new concept introduced in KUMON. This problem persisted for a year but she now faces new concepts without hesitation. These are examples of children and youth who grew not only scholastically but also with respect to their self-confidence.

In conclusion, it has been our experience that the commitment of foster parents and parents in the case of the fee paying participants, is absolutely critical to success with KUMON. Since commitment is so important, we would like to end by sharing a situation that happened in one our centres which exemplifies commitment, albeit beyond what you would ever expect. A grandmother, who is a kinship care provider to her grandson, brought her grandson to the centre on the day her husband of 40 years passed away. She is not a very well-educated woman but has rarely missed coming to the centre. We have had the sense that she sees our program as a way out of the obstacles she and her children have faced in their lives. Although it was clearly an incredibly difficult day for her she did not want her grandson to miss a day at our program. We hope that we can all be as passionate and committed to the education of foster children as that grandmother is to her grandson’s future. We have had our share of successes and failures in offering a supplementary educational program. Our biggest challenges are to continue to increase the numbers of foster children participating in the program and to ensure their retention once they have enrolled. From many sources we see evidence we are on the right course and will continue with the investment we have made in realizing better educational outcomes for children in our care. We, will continue to use the WRAT instrument and changes in school grades to measure success. Although we have gathered some evidence about KUMON’s efficacy as a supplementary learning program for children in care it is our intention to develop an increasingly large sample of time series data to buttress the findings presented in this paper.

About Authors

Dr. Michael O’Brien is the Director of Research and Quality Assurance at the Family and Children’s Services of Renfrew County. He is an Assistant Professor at the School of Social Work, Dalhousie University in Halifax, Nova Scotia.

John Rutland, M.ED., is a retired special education consultant and is the Education Coordinator at Family and Children’s Services of Renfrew County in Ontario.

Previous article: The Role of Child and Youth Care Practitioners in Evidence - Based Practice in Group Care: Executive Summary

Next article: Why Do the Youth on My Caseload Harm Themselves? Communications As a Possible Factor in Youth Deliberate Self-Harm