Bringing Together the Child Welfare Team
A Trainer’s Guide

Lessons Learned

A key objective of this project is to disseminate information that the project team has gathered about successful approaches to building the skills of child welfare supervisors and managers. As part of that effort, this section of the curriculum, contains information we collected from the project evaluator and representatives of the four child welfare agencies/training institutes that field-tested the curriculum. We asked for lessons learned, comments on ways to adapt and enrich the curriculum, plans to incorporate the training into the agency’s ongoing training efforts, ideas on how to identify the impact of the training on day to day job performance of supervisors and managers and themes which can inform future users of this curriculum.

The Muskie project team appreciates the advice shared and hope you find it helpful as you consider providing training on ASFA implementation and data use skills for child welfare managers and supervisors in your state/region or county.

Lessons Learned: the Project Evaluator’s Perspective

Although this curriculum is designed to support training transfer, evaluating its impact was challenging.
The project team set out to design an evaluation approach that would evaluate the impact of the training on the practice of the participants. While the pilot trainings included a relatively small number of participants (approximately 20 to 25 each) making the impact of the pilot training difficult to measure, we did want to test an approach that would focus on impact rather than satisfaction. As training professionals know, impact evaluation is challenging at best – and this training renewed our understanding of those challenges.

The curriculum is designed to support the transfer of learning from the classroom to office practice. The briefing to senior executives is designed to create a commitment to the bringing the substance of the curriculum into action. The personal learning plans are designed to support participant’s reflection on their own learning and continued post-training skill development. The pre-training evaluation form asks participants to reflect on what they need/want to learn and the post-training evaluation form asks participants to anticipate putting their learning into practice back at the office.

That said, the primary focus of the pilot sites was on customizing the training-room curriculum to meet their specific local needs. None of the sites used / tested personal learning plans— reportedly as previous experiences with this approach were only modestly productive. While all sites had the blessing of senior executives to participate in this initiative, Cuyahoga County was the only site that used the executive briefing as designed. They briefed their senior management team, including the director, on the curriculum and solicited input as to substance and focus. For example, the director asked that the action planning exercise focus on a problem of particular concern to him (namely, “foster care drift” – the length of time in foster care before permanency is achieved). The results from that exercise were presented to him following the training, as were the results from the systemic factors assessment.

One qualitative measure of impact is the commitment and enthusiasm of each pilot site to delivering this curriculum to a broader audience beyond the pilot itself. Kentucky has integrated it as part of a comprehensive supervisory curriculum; Cuyahoga County is adapting it as three separate, companion training sessions; and New Mexico and Wisconsin are rolling out the curriculum they piloted with some modifications. Clearly, pilot sites believe the curriculum offers important benefits to their managers and supervisors. This is echoed by participants, a large majority of whom reported that they would use “all” or “almost all” of what they learned.

The testing, revision, and validation by the pilot sites of the materials presented here should allow new users of this curriculum to focus on the activities outside the training-room that support training transfer. We encourage new users to take on the challenge of measuring the impact of this training by using the evaluation forms here in combination with the performance measurement tools already in place at your agency.

Curriculum flexibility requires evaluation flexibility.
As noted in other places, this curriculum is designed to be customized to meet local needs and the results of the pilot prove the curriculum is flexibility and adaptable. It has been delivered in states that are relatively rich in training resources as well as those that have relatively few resources. It has been delivered prior to a state’s Child & Family Service Review (CFSR) as well as following a state’s CFSR. Each site successfully adapted the curriculum to meet its unique needs. New users of the curriculum should be sure to adapt the evaluation forms to reflect any customizing changes to the curriculum.

ASFA knowledge is multi-leveled.
Participants rated their knowledge and understanding of ASFA very high both before and after the training. However, the trainers in two sites believe that, prior to the training, participants did not understand how ASFA actually applies to case practice. In a debriefing with trainers from all four sites, we concluded that on the pre-training evaluation participants at these sites were rating their understanding of ASFA as a federal requirement. During the training, they learned about the application of ASFA to case practice so that the post-training evaluation rating reflects a deeper understanding of ASFA.

Increasing the skills required to use data for decision-making is often hindered by factors outside the participants’ control.
Participants rated their skills in using data to support decision-making as being low relative to other skill areas. Their ratings improved in the post-training evaluation, but were still relatively low. Their comments on the evaluation forms identified a number of obstacles to developing and practicing these skills. The most commonly identified obstacles were access to user-friendly reports, concerns about data integrity/quality, and time. The pilot trainers concurred that these factors are challenges with which their entire agencies are grappling. This finding highlights the importance of securing executive support for the curriculum’s content, particularly for areas that might require organizational changes to support training transfer.

New Mexico took an interesting approach to modeling the use of data. They enlisted the assistance of one of their IT managers in analyzing the evaluation results. He was able to data enter the pre and post training responses from all participants at the end of the training and provide the results to the participants. In this way, participants saw the evaluation results (which we believe is important feedback) and an immediate, first hand example of data use.

Lessons Learned: the Pilot Site Perspectives

Cleveland, Ohio
Cuyahoga County Children and Family Services

Contact person information
Name:	Veronica Holloway
Title:	Senior Supervisor of Direct Services, On Going Section
Address:	Cuyahoga County Department of Children and Family Services 3955 Euclid Avenue, Room Number 226 East Cleveland, Ohio 44115
Phone #:	(216) 432-5032
E-mail Address:	vhollowa@www.cuyahoga.oh.us

1. Briefly describe how you implemented the pilot project in your state. Mention who you trained, the format of the training, who the trainers were and the types of adaptations you made to the core curriculum.

The pilot project for Ohio was implemented at the county level. The Cuyahoga Department of Children and Family Services (CFS), located in Cleveland is the child protection agency for Cuyahoga County. CFS, with input from the state level, developed expectations of what we needed the curriculum to offer. We followed with a review of these expectations by thirteen senior managers of CFS, and from this review the curriculum was modified to meet our needs.

The pilot was implemented over two days and consisted of twenty-one supervisors and seven senior supervisors. These individuals were selected from Intake, Ongoing, Foster Care and Adoptions. The curriculum was presented in its current format, with modification in language usage and data reviewed. The training was conducted by senior trainer, Sharita Jackson, and training officer, Becky Thomas.

2. Describe how you see this training (or pieces of the training) fitting in to your ongoing training for supervisors and managers.

In order to ensure that our families receive the best services we can offer, we feel our staff must better understand outcome based work and how to interpret data. We are currently developing ongoing training for all staff, this includes social workers. Three future training were devised from the pilot training:

3. Describe the impact of the training on those you trained, the system and practice. If you can foresee a long-term impact, please discuss that also.

The training has helped our staff see that the work they do can be measured. Those who had the training stated they would start using almost all the information they learned immediately. The long-term effect will be seen in how we collect data and accurately analyze and improve best practice and child welfare.

4. Discuss the lessons learned from your experience, including what worked best for you and what needed improvement.

One thing we noted was that there must be ongoing training of the Adoption and Safe Families Act (ASFA), especially the Child and Family Service Reviews (CFSR). Staff has an overview of what ASFA is, but at times separate the CFSRs from ASFA. What we discovered was that with an overview of ASFA and the CFSR, staff is better able to understand why outcome based decisions have merit. Additionally, the methodology of training needs improving, along with targeting how data is used to inform child welfare work, and its consequent outcomes.

5. What else do you want to tell us about your experience with the ASFA Training Project?

Our agency is in the process of developing our strategic plan, what we decided was to make ASFA a portion of it.

6. To give readers an idea of how their agency compares to yours, please provide brief descriptions of

Our agency of approximately 1,134 employees, the organization consists of:

Our state has just completed the CFSR review and is compiling the written evaluation to inform all counties of how we did. Our state has an evaluation process, “Child Protection Oversight and Evaluation” (CPOE), that measures outcomes such as time to reunification, timely response to reports of abuse and/or neglect and length of time to adoptions. We have been using CPOE since May of 1986.

Our state does not use SACWIS. We utilize the Family and Children Services Information System (FACSIS). This system allows us statistically to track our children and families as they move towards safety, permanency, and child well being.

All managers and supervisors are mandated to take five core classes (60 hours) of training the first year of supervision. After that each manager and supervisor is mandated to take 30 hours of training. The classes include Administrative Supervision, Educational and Supportive Supervision, Transfer of Learning, Supervision and Managing Workgroup Performance.

Kentucky
Department of Community Based Services

Contact person information
Name:	Denis Hommrich
Title:	Family Services Specialist
Address:	908 West Broadway – 4-E, Louisville, KY 40203
Phone #:	(502)595-4732
E-mail Address:	Denis.Hommrich@mail.state.ky.us

1. Briefly describe how you implemented the pilot project in your state. Mention who you trained, the format of the training, who the trainers were and the types of adaptations you made to the core curriculum.

We conducted a pre-pilot training and 4 pilot trainings involving 4 of sixteen regions. Kentucky consolidated Module 2 and 3. Discussions focused on the knowledge, skills and abilities to implement ASFA and discussed problem solving techniques that could be applied to practice, policy, resource, and organizational issues impacting on implementation of ASFA. Module 4 placed an emphasis on interpretation of the data profiles and child welfare outcomes in the context of organizational improvement and changes rather that “scoring” and “ranking” regions against each other. Reports generated Kentucky’s SACWIS system were reviewed in the context of the child and family services reviews.Module 6 focused on assessing needs of clients and communities and building effective collaborations/ service network needs to address specific regional plans and goals.Participants were encouraged to become familiar with the state and regional strategic plans, goals, baselines.

Staff from the Muskie School, Institute for Child and Family Policy, co-trained with state staff in the pre-pilot. Regional Specialists, Central Office Specialists, Regional, and Central Office trainers conducted subsequent trainings.

2. Describe how you see this training (or pieces of the training) fitting in to your ongoing training for supervisors and managers.

Parts of this training will be incorporated into the ongoing training for supervisors, managers, and frontline workers.

3. Describe the impact of the training on those you trained, the system and practice. If you can foresee a long-term impact, please discuss that also.

Staff completing the training felt they had a good understanding of the ASFA requirements prior to the training. After the training they felt they had a better understanding of how to interpret the data and use this learning to implement change in practice and the organization. They also felt they were better able to focus on critical elements of ASFA and its impacts on Kentucky’s child welfare system.

4. Discuss the lessons learned from your experience, including what worked best for you and what needed improvement.

Each group was different and had different levels of experience with ASFA. And different practice and organizational issues Kentucky will probably include an exercise in the introductory module to explore ASFA implementation issues relevant to the regional staff being trained. Training staff by region rather that having staff from many regions seemed to work well. Some individuals were offended that some performance reports identified workers and supervisors by name. Informing staff in advance that team specific and worker specific reports will be discussed and shared is something that will be done in the future.

5. What else do you want to tell us about your experience with the ASFA Training Project?

It is still a work in progress. We are revising it in light of our experiences and feedback from staff.

6. To give readers an idea of how their agency compares to yours, please provide brief descriptions of your agency (for example, caseload size, number of employees, organizational structure, number of offices, where your agency is in the CFSR process, status of your SACWIS implementation and reports available to supervisors from your SACWIS)

New Mexico
Department of Children, Youth and Families

Contact person information
Name:	Kirk Rowe
Title:	Children’s Section Manager
Address:	P.O. Drawer 5160, Santa Fe, NM 87502
Phone #:	(505) 827-8400
E-mail Address:	KRRowe@CYFD.state.nm.us

1. Briefly describe how you implemented the pilot project in your state. Mention who you trained, the format of the training, who the trainers were and the types of adaptations you made to the core curriculum.

A committee of field supervisors and managers reviewed the curriculum. Upon completion of the review, the committee determined that, as New Mexico had completed their CSFR and had also had a fair amount of training on ASFA, the curriculum would be better suited as post report training that would focus on the Program Improvement Plan (PIP). The PIP had not been written or accepted at that point, but features of the report could be pulled out for consideration when making changes to the curriculum. Once the curriculum was finished, the committee became the trainers and the pilot training took place on May 30 and 31. The pilot training included supervisors and managers from the field offices

2. Describe how you see this training (or pieces of the training) fitting in to your ongoing training for supervisors and managers.

The feedback has been positive. Most see the training as being useful as further defining how the supervisors can insure that their employees are focusing their work on best practice and for preparation for the next CSFR. Many also have stated that the training helped bring into focus how the standards were arrived at and where New Mexico needed to work in order to improve.

3. Describe the impact of the training on those you trained, the system and practice. If you can foresee a long-term impact, please discuss that also.

The training is reported, by participants, to help them focus practice more directly on the standards and outcomes, thus improving their ability to also meet ASFA requirements. A checklist was developed for supervisors to use when supervising staff that helps them track ASFA requirements, to know what has been addressed and what still needs addressing. The long-term impact should be seen in the improvement of records and data.

4. Discuss the lessons learned from your experience, including what worked best for you and what needed improvement.

The flexibility of the curriculum and the various exercises were very useful for designing a curriculum that met the state’s needs. Supervisors and managers had considerable training on ASFA before implementation of this training.

Use of the full modules would have been redundant, but the curriculum was open to modification and quite useful in putting together training for a state that had already been through the CSFR and was in the process of preparing for the next CSFR. Feedback from supervisors indicates that the training was useful for showing them how to implement changes according to ASFA.

5. What else do you want to tell us about your experience with the ASFA Training Project?

For those not familiar with ASFA and the standards and outcomes, this training was a good first exposure. For those who are familiar (most of our supervisors) this was described as a good “ah, ha!”

6. To give readers an idea of how their agency compares to yours, please provide brief descriptions of your agency (for example, caseload size, number of employees, organizational structure, number of offices, where your agency is in the CFSR process, status of your SACWIS implementation and reports available to supervisors from your SACWIS)