 |
|
Module 7: Data Analysis: Tips, Tools and TechniquesRationale Increasingly child welfare supervisors are expected to be able to analyze data and information to help with decision making or to advise decision-makers. This module allows participants the opportunity to learn, expand and hone data analysis skills, as well as understand the varying information needs of people at several levels in the Child Welfare agency. Activities
Time 2 hours Objectives When this module is complete, the participant will be able to:
Easel, newsprint, tape, markers and calculators Sample handouts and overheads: 1. Sample Information Needs for Adoption Triangle (7.1) 4. Tips, Formulas and Definitions of Terms (7.4) 5. Table 1: Abuse Reports- Statewide (7.5) 6. Table 2: Abuse Reports Cascadia (7.6) 7. Tables 1 & 2: Abuse Reports Statewide and Cascadia and Answer Box 2 (7.7) 8. Table 3: Abuse and Neglect Fatalities (7.8) 9. Table 4: Age of Victim and Answer Box 4 (7.9) 10. Tables 5, 6 and 7: Alleged Perpetrator of Substantiated Abuse and Answer Box 5 (7.10) 11. Table 8: Type of Maltreatment Statewide and Cascadia (7.11) 12. Table 9: Data Displayed in Three Ways (7.12) 13. Sample Report to the Commissioner (7.13) 14. Sample SACWIS reports selected by the trainer that show multi-year trends Advance Preparation Make sure flipchart, markers, newsprint pad, overheads, calculators and overhead projector are in the room. Have copies of the case study available for all participants. The trainer will need to update the Sample Information Needs for Adoption Triangle to reflect job titles used by the agency. NOTE: The Abuse on the Increase in Cascadia County case study is a challenging, productive training exercise for both the participants and trainers. The trainer may want to customize the case by using agency data, locations and names. To assist the trainer in preparing to instruct on this case, this module includes the standard Trainer's Instructions supplemented by a Trainer's Version of the Case Study Abuse on the Increase in Cascadia County, which includes the case study with attachments, calculations and suggested answers to key questions (7.3). Prior to the seminar, the trainer should review the material in this module to determine if he/she needs to create additional training aids to facilitate preparation for and presentation of the material. Have available sample SACWIS reports displaying multi year trends for the concluding exercise. Glossary of Terms Comparison - An examination of two or more items to establish similarities and dissimilarities. Cumulative percent -A summing of all data or values related to a percent. Data - A recording of facts, concepts or instructions on a storage medium for communication, retrieval and process by automatic means. Percent - One part in a hundred; tells how many out of 100. For example, 5% = 5 out of 100. Percent change - The portion of a change in quantity, amount or value. Rate - Frequency of occurrence. Trend - A line of general direction Bibliography and Suggested Reading Sperling, A.P. and Levinson, Samuel D. (1998) Arithmetic Made Simple. Doubleday, New York, NY. Revised edition Weinbach, Robert W. and Grinnel, Richard M., Jr. (1991) Statistics for Social Workers. Longman Publishing Group, White Plains, NY. 2nd edition. Williams, Edward. (1989) Arithmetic the Easy Way. Barron's Educational Series, Inc, Hauppuage, NY. 2nd edition. Abuse on the Increase in Cascadia County. (January 2000) Data Users Group of the Oregon Department of Human Services, Salem, OR. Trainer's Instructions 1. Introduce the module by presenting the purpose and objectives using the following as a guide: This module offers you the opportunity to understand the types of information needed by people at various levels of responsibility in a child welfare agency, explore why different levels have different needs and practice various data analysis tools and techniques. The goal is not to turn you into a statistician; however the material we will cover in this module should help you feel more comfortable working with numbers, doing data analysis or interpreting data while you are doing your day-to-day work back in your office. When this module is complete, you will be able to:
2. Explain that the first exercise will provide an opportunity to think about how information needs may vary depending on the organizational level of the user. Refer participants to handout 7.1, Sample Information Needs for Adoption Triangle, and put up overhead 7.1. Begin a mini-lecture using the following as a guide: It's frequently said that 'data is data is data is data' and 'information is information is information'. Later on in this module, we are going to look at several ways to analyze and interpret data and we will be better able to determine if we agree that 'data is data is data'. Right now we are going to focus on the information needs of child welfare agency personnel at various levels in an organization. Let's start by looking at this triangle, which provides a sample of the questions key personnel at three levels---the Agency Director, the Statewide Program Manager and a Regional Program Manager--may ask about the adoption program. As you consider the answers to the questions, you can see that the information gathered gets more streamlined, concrete and straightforward, as well as more summarized, the higher you look in the agency. Note, for example, the difference in detail requested by these two questions 'How many children were placed for adoption?' and ' How many families have adoption assistance agreements?' 3. Pause here and ask participants to go into their small groups and to reflect on the potential impact on decision making if executive personnel work primarily with highly summarized information. After 3 minutes reconvene the large group and ask each group to mention 1-2 impacts. Process. 4. Continue to explore this topic by probing the relationship between the audiences for and uses of information and the level of detail. More specifically, ask participants questions such as "How does an Agency Director use information? Who is likely to ask for information from an Agency Director and for what purposes?" Record the answers on a flipchart. Expect the following types of answers:
5. Repeat the questions for both the statewide and regional program managers. Record the answers on a flipchart. Expect answers such as:
6. Wrap-up the discussion by stating that: Executives in all organizations, not just child welfare, are expected to have some level of knowledge about a variety of topics. Every executive has his or her special areas of interest and will probably be more briefed on those areas than others. Executives, thus, rely on people closer to the program or clients to have, be able to gather and to do more analysis on detailed programmatic issues than they do. Executives often turn to staff to get accurate, complete, factual answers to specific questions. Often, the executives may ask questions that do not have readily available answers so answering the question may take research, analysis and /or educating of the executive on the type of data that is available. Staff often plow through 'tons' of detailed data, selecting, analyzing and organizing it, to get the answer to one question. When responding to a question, be sure that the response is framed in a way that is familiar to the questioner. Administrators of a program are likely to be familiar with the unique language of a program; while users of information external to the agency may not be. So, for example, when addressing a question about the number of substantiated reports in a county, talking about 'substantiated reports per 1000 reports' might make perfect sense to a child welfare administrator but none at all to a news reporter or a legislator. They might be interested, however, in the number of reports that were verified per population in the county or state or per number of families. We looked at an adoption example here, however, the same holds true of other child welfare services. The organizational position of the person, the audience and the intended use of the information all are factors that help define the level of detail of the questions asked and information provided. 7. Introduce the case, Abuse on the Increase in Cascadia County using the following as a guide: Even though you as a child welfare supervisor may not need to analyze the kind of data presented in this case study, you may find that the skills practiced in this case study can be applied in variety of situations, perhaps with your unit or when collecting data for your supervisor or when you are reading reports. The point is to develop or increase your comfort in reading numbers, tables and charts, as well enhance data analysis skills that can be applied in a variety of different situations. The goal is to show how using, analyzing and understanding data correctly can help answer basic social work questions. 8. Pass out copies of the Participant's Version of the Case Study, Abuse on the Increase in Cascadia County, (7.2). Ask the group to take a few minutes to read pages 7.23-7.25, stopping at the end of Question 1. 9. Put up Overhead 7.4, Tips, Formulas and Definitions and ask participants to find the same handout. Walk through a summary of the case and the Question 1 tips and definitions (found on 7.4) using the following as a guide: You've read the case and see that Commissioner Smith is stunned by the newspaper's charge that Cascadia County had a 300% increase in the number of abuse and neglect cases last year. Commissioner Smith turns to you, the Program Administrator in Cascadia County, and the CPS supervisor in Cascadia County to help him look into this charge. He asks you to find answers to several questions. Lets stop and think for a moment how to approach this request. First, we need to find a source of data. In this case, we are fortunate that the State Office for Services to Children and Families (SCF) maintains data on a web page and has just released some statistics last week. The first tip in the case points out that locating the correct data and understanding the data is critical first step in any data analysis. We have discovered that the 'correct data' to answer the Commissioner's first question is on Table 1: Abuse Reports Statewide and Table 2: Abuse Reports in Cascadia County. Can you all find those two tables in you case study? They are in Attachments 1 and 2. OK, lets look at Table 1 (Overhead 7.5) for a minute. You often see tables of data organized into rows and columns. This is a fairly complex table presenting a lot of data. Lets look at each column and row. Going across the top, we see the headings Year, # of CPS Reports, Yearly % Change of CPS Reports, # of Substantiated Reports, Yearly % Change of Substantiated Reports and, finally, Rate of Substantiated Reports per 1000 Reports. Below each of these headings are data. So we can see, for example that in 1998, there were 31,456 CPS reports statewide and in 1992 there were 7,265 Substantiated Reports. Does everyone see where I'm getting this data? The Commissioner's first questions asks 'How many Child Protective Service Reports (CPS) and substantiated reports did Cascadia have in each of the past 5 years?' Does Table 1 answer that for us? No, Table 1 gives us that information statewide. Has SCF provided us any data Abuse Reports in Cascadia County? Sure they have…let's look at Table 2: Abuse Reports in Cascadia County. 10. Put up overhead 7.6 , Table 2 Abuse in Cascadia County and continue the presentation using the following as a guide: Table 1 presented certain data statewide; Table 2 presents the same data for Cascadia County. Take a look at this table and locate the answers to the first set of the Commissioners questions about the number of CPS reports and the number of substantiated reports in Cascadia County over the past five years. 11. Ask participants to locate and identify the answers to the questions and then the trainer can circle the correct data on the overhead. Continue the discussion: So we have now answered part of the first question, we need to move on to the second and third parts and look at what percent of Cascadia's CPS were substantiated in 1998 and how that compares with the state's percentage of substantiated reports. Do we have the data to answer those questions? Where is it? Tables 1 and 2 provide the data. The analytic tool that is being used here is percent; we are being asked to calculate a percent and then compare percents for Cascadia and the state. Percent answers the question 'how many out of 100?' and is usually calculated by dividing the smaller number by the larger number. So lets calculate for 1998 the percent of CPS reports that were substantiated in Cascadia County. 12. Ask the group what table should be used and then to locate the data. Review the formula using overhead 7.4 and ask someone in the group to calculate the percent. (217/1105 *100 = 19.6%) Repeat the activity to calculate the statewide percent of substantiated reports in 1998. (7461/31456 *100=23.7%) 13. Ask the group how the figures compare. (In 1998, the % of Cascadia's substantiated reports was lower than the state's.) Ask participants to note the answers to Question 1 in Answer Box #1 and note the answers to Question 1 on a flipchart. 14. Begin a quick summary of the learning from Question 1 using the following as a guide: We've answered the Commissioner's first question, talked about locating the correct source of data and calculated some percents as a comparative analytical tool. 15. Ask for and address any questions. Ask participants to read through Question 2 information in the case study. 16. Put up overhead 7.4 and continue the discussion: Question 2 asks us to think about two analytical tools-- percent change and rate. Lets start by looking at percent change. Percent change may be used to show either an increase or decrease in a value and is calculated using the following formula: (Most recent # - Previous # )/ Previous # *100 = Percent Change Lets find out how we can apply this formula to answer Question 2, 'During the period 1992-1998, how does the change in the percent of reports that are substantiated in Cascadia compare to the state's substantiated report percent change?' 17. Ask the group how they would go about answering this question. Probe with questions such as: Do we have the data to answer this question? If so, where is the data located for the state? For Cascadia? Use overhead 7.7 and ask the group to locate the correct data on Table 1. Calculate the percent change using the identified data on Table 1 and the formula: (7461 - 7265/7265*100 = 2.70%). Look at the data on Table 2. Point out that the data on Tables 1 and 2 present the same time period but the data in the narrative mentions a different time span---1992-1998 for the state, 1993-1998 for Cascadia. Note that a key concept in when making comparisons of entities is to use data from the same time period. Repeat the activity for the Cascadia County information, locating data on Table 2 (8.6) and then calculating the percent change. (217 - 224/224*100 = - 3.13%) Ask the group how the percents of change compare. (Cascadia's percent change has decreased by 3.13% and the state's has increased by 2.7%) Record the answers to Question 2 on a flipchart and ask the participants to note the answers in Answer Box #2. 18. Move into a discussion to the importance of rate. Ask the group to look at the Rate: Substantiated Reports per 1000 Reports data on Table 1 (8.5). Highlight: As Tip 3 on Overhead 7.4 notes, a critical issue in attempting to compare data is knowing that the data is comparable. Rate is a helpful tool here. Rate helps us look at the number of events, which occurred, compared to the population for which that event could have occurred. So when we look at Table 1 we see that in 1998, the Rate of Substantiated Reports per 1000 Reports was 237. Looking at Table 2 we see that in Cascadia County, the Rate of Substantiated Reports per 1000 Reports in 1998 was 196.4. 19. Review with the group the answers to Questions 1 and 2 and then ask the participants to go to Answer Box #2a. Explain that the group has calculated data and now needs to begin synthesizing the data and thinking about any other information that would be useful to include in the Commissioner's report. Ask each person to take 2 minutes to record in Answer Box #2a any preliminary conclusions or thoughts that should be considered for the Commissioner's report. 20. After 2 minutes, reconvene the group and ask participants to share the information they wrote in Answer Box #2a. Look for the following:
21. Ask for and address any questions on the skills practiced in Questions 1 and 2. 22. Ask the group break into their small groups and assign each small group a question to read and respond to. After 15 minutes reconvene the group and ask each group report on what their question was, what the answer was and what the implication are. 23. To answer Question 3, the group must use the percent change calculation --most recent # - previous # /previous # * 100 = percent change. Make sure they mention the following:
24. Ask someone to define the impact of small numbers. 25. Ask the small group to report on Question 4. This question asks the group to locate data and work with percents. Ask what data should go in the Commissioner's report concerning age of the victims. Make sure they mention the following:
26. Ask for the group report on Question 5. Can they find all the data to respond to Question 5? Note the data they would put in the Commissioner's report concerning perpetrators. Make sure they include the following types of comments:
27. Observe that the data indicate that mothers are much more likely to be perpetrators of child abuse and neglect than fathers are; this seems to be a trend. Is this finding surprising? Is it meaningful? Who provides most of the childcare? Who are usually the parents in single-parent households? What could this finding mean to you as you think about providing services to children and families? 28. Ask for the small group's report on question 6. Make sure they mention:
29. Note also that throughout this training they have been looking at data presented in the same kind of tables. However, there are other ways to present data that can be effective. For example, Overhead 7.12 shows data Table 9 and two charts that have been made from the data. Ask participants which presentation of the data seems most clear and readable. 30. Have each small group discuss its response to Question 7. Make sure participants mention:
31. Continue the discussion of the report to the Commissioner: So let's summarize the information we're going to put in the report to the Commissioner. He asked for this research and data analysis because the newspaper said that Cascadia County had a 300% increase in the number of abuse and neglect cases last year. Additionally, the Commissioner believes that child abuse is on the rise in Cascadia, a key factor in that increase was substance abuse and a frequent perpetrator was the mother's boy friend. We've looked at a lot of individual data so let's start to create the introductory paragraph of the report. First of all, we know now that we can find nothing to substantiate the '300% increase in cases'. We did calculate a 300% increase in fatalities in Cascadia but we know that large % increase was caused by a large increase in a small number of fatalities (1 to 4 between 1997 and 1998) and that does not indicate a trend. We know from our research that between 1997 and 1998, Cascadia County had a 12.31% increase in abuse reports and a 2.74% increase in substantiated reports; we know nothing about cases, however. In answer to the Commissioner's 'belief' that substance abuse was a factor in the increase of child abuse cases in Cascadia cases; we need to report that we don't have specific data on substance abuse in Cascadia County, rather we have data on statewide substance abuse. Finally, our research shows that both statewide and in Cascadia County, mothers, not boyfriends are the perpetrators 42% of the time, with fathers following at 26 and 25% of the time respectively. Data on the mother's boyfriend isn't collected. So, where does that leave us? We can't confirm or deny the 300% increases in cases because we don't have the definition of 'case', we have no data on substance abuse in Cascadia specifically as it's not collected and mothers and fathers, not boyfriends, are the most common perpetrators. How do we frame the report to the Commissioner? Do we just tell him what we don't have? Or do we provide him with the meaningful statistics that we do have (including definitions), mention the data that isn't collected (such as substance abuse by county) and offer suggestions for next steps? 32. Pause and poll the room on what they would do under these conditions. Encourage a conclusion aimed at providing the 'facts' as they have been researched, noting missing data and offering suggestions for collecting such data in the future. Reflect back on the previous exercise where the group looked at data and information needs by varying levels in the agency and note that the commissioner should be provided with highly summarized facts, supported by research and analysis whether or not they support his personal 'beliefs' or not. 33. Distribute copies of handout 7.13, Sample Report for the Commissioner. Give 5 minutes for folks to read the report and then highlight the content of the report drawing on comments provided by participants through out the case discussion. 34. Put over head 7.4 up and refer participants to the related handout. Quickly review the definitions and formulas and begin to wrap-up the activity using the following as a guide: In the case study, you just read through and selected data from several tables, as well as calculated percentages, percent changes, and cumulative percents. We've reviewed rates, the effect of small numbers, what to do when a reporting category is added and how to respond when data are missing. Learning how to run numbers through formulas, while a useful skill to have, was not the main point of this exercise. Rather, the goal was to become familiar with these selected analytical tools, learn questions to ask about data and see if any of these tools can be applied in your day to day supervision or other operational activities. 35. Launch the final activity in this module. Distribute the sample reports and ask participants to go into their small groups to discuss and analyze the data and answer the questions listed on handout 7.14. Specifically, have the groups 'interpret' the data. 36. After 10 minutes have the group come back together and report on their discussions. Probe the responses by asking the following types of questions: What are the figures saying or trying to say? Can you make any conclusions using these figures? Can you see any trends? Do these numbers surprise you? Do you think these figures could be misleading in any way? If so, how? Are there different ways the same data could be interpreted? What analysis tools did they use on this data to try to interpret it? How did the tool work? What lessons can be learned from this exercise in interpreting data? 37. Wrap up the exercise by noting that: More and more frequently, child welfare supervisors are expected to have basic analytical skills such as reading and interpreting data on reports. Having such skills is another tool that a child welfare supervisor can rely on to support the complex decision making that he/she is called on to perform everyday. 38. Ask for and address questions. 39. Introduce the next module, Technology is Changing the Job of the Child Welfare Supervisor: Are You a Change Leader?, by commenting that: During the implementation of any change, be it a major new
system like SACWIS, a new technology, or a new or revised policy
or procedure, supervisors are usually asked to play a dual role--leading
the change as well as changing their own behavior. The next module
looks at several techniques for overcoming resistance to change
and enhancing commitment to change building skills.
Due to the size of Module 7, the handouts and overheads are located in separate locations so your load time will be faster. [7.1] [7.2] [7.3] [7.4] [7.5] [7.6] [7.7] [7.8] [7.9] [7.10] [7.11] [7.12] [7.13] [7.14]
|
Notes
To allow trend analysis, the data on the sample SACWIS reports should include multiple years of agency data. Suggested data elements include # of CPS reports, # of CPS substantiated reports, # of investigations closed without a finding, child victims by age, child victims by race, perpetrators by relationship to victim.
The trainer might want to make up an overhead with the words 'The goal is to show how using, analyzing and understanding data correctly can help answer basic social work questions'.
The trainer's approach to presenting this case will depend on the trainer's assessment of the group's comfort level working with data and calculations. After setting up the case, the trainer may, for example, walk the whole group through the first question set, then break into the small groups to go through the remainder of the questions. In another situation, the trainer may walk the whole group through Questions 1& 2 before going into small groups. Some trainer's might find that this case works best if the whole group stays together for the entire case.
The level of detailed description needed will depend on the participants' level of comfort using tables of information. It may be unnecessary to review the table or it may be necessary to review each row and column. |
