| Module
5: Tips for Using Data to Measure Success
Time
Approximately 3 hours and 15 minutes
Rationale
As a result of ASFA, child welfare managers and supervisors
are increasingly expected to be able to use data, information
and reports to guide decision making and to determine what
is working and what isn't working in the organization, with
practice and in the service delivery system. Thus, child
welfare supervisors and managers must know how to select,
interpret and understand appropriate data to ensure accountability
for achievement of assigned outcomes, allocate scarce resources
and improve services for children and families.
Learning
Objectives
When this module is complete, the participant should be
able to:
-
Identify reports that peers find helpful for supervisory
and managerial decision making
-
Know how to use some basic data tools for reading and
interpreting data
-
Be able to discuss the content of reports in terms of
the usefulness of the data and relate the content to outcomes
-
Understand how to use data from reports to monitor the
implementation of ASFA and agency goals and outcomes,
with a focus on the outcomes he/she and his/her unit is
responsible
Activities
-
Exercise: Share helpful reports and ways to use those
reports in decision making (30 minutes))
-
Mini-lecture: Ten Tips for Using Reports to Improve Decision
Making (10 minutes)
-
Exercise: Hawaii Case Study (45 minutes)
-
Exercise: M & M County (45 minutes)
-
Exercise: Take one or two reports and critique format
and content. Ask if and why the data on the report is
important, useful and used. Identify what the report is
linked to -- safety indicators or finance or budget, for
example (20 minutes)
-
Exercise: Look at selected reports that focus on ASFA/agency/unit
goals, outcomes and measures or CFSR/PIP and make some
comparisons between regions, teams, or units, include
reports that relate back to specific (possibly national)
performance indicators and/or state plan and are easily
accessible by the users (45 minutes)
Sample
Materials
-
Reports Helpful to Decision Making (Section II.5.1)
-
Ten Tips for Using Reports to Improve Decision Making
(Section II.5.2)
-
Participant’s Version of the Hawaii Case Study, including
the case study and attachments (Section II.5.3)
-
Trainer’s Version of the Hawaii Case Study, including
the case study with attachments and suggested answers
to key questions (Section II.5.4)
-
Frequency/Percent (Section II.5.5)
-
Averages (Section II.5.6)
-
Pie Chart (Section II.5.7)
-
Bar Chart (Section II.5.8)
-
Sample reports selected by the trainer to form the basis
for a discussion of their usefulness and importance (to
be handed out at the time of training)
- Sample
reports that support agency outcomes and measures and
allow some comparison between units or regions (to be
handed out at the time of training)
-
Interpreting data form (Section II.5.9)
Advance
Preparation
Make sure flipchart, markers, newsprint pad, overheads,
calculators and overhead projector are in the room. If you
are going to use them, have copies of the case studies available
for all participants.
This
module contains two data exercises, the Hawaii Case Study
and the M&M County exercise. If the trainer would like
to shorten the module, or if one or the other is more appropriate
for the group being training, one of the exercises can be
omitted
-
The Hawaii case focuses on interpreting data and linking
practice to data and outcomes. It includes the standard
Trainer’s Instructions supplemented by a Trainer’s Version
of the case, which includes the case study with attachments
and suggested answers to key questions. Prior to the seminar,
the trainer should review the material in this module
to determine if he/she needs to create additional training
aids to facilitate preparation for and presentation of
the material. You will probably want to customize the
case by using familiar names, local data and indicators
of interest to your agency.
- The
M& M County exercise is an opportunity to practice
some basic statistical concepts and ways to apply them
when using data and reading reports. The presentation
of this exercise should be as informal and humorous as
possible. To use the M&M County exercise have a small
bag of M&Ms for each group. Make sure each bag has
M&Ms of assorted colors.
Collect
appropriate samples for the exercises which include a discussion
of reports.
Glossary
of Terms
Average:
the total number of items divided by the number of items.
Bar
Chart: a way of presenting numerical information as
a series of bars or columns of different lengths.
Comparison:
an examination of two or more items to establish similarities
and dissimilarities.
Data:
a recording of facts, concepts or instructions on a storage
medium for communication, retrieval and process by automatic
means.
Frequency:
the number of times an event occurs.
Legend:
An explanatory caption accompanying a map, chart, or illustration.
Mean:
the arithmetic mean, or means, of a set of measurements
is the sum of the measurements divided by the total number
of measurements.
Median:
the median of a set of numbers is defined to be the middle
value when the numbers are arranged in order of magnitude.
Mode:
the mode of a set of numbers is the value that occurs
most often (with the highest frequency).
Percent:
one part in a hundred; tells how many out of 100. For example,
5% = 5 out of 100.
Statistics:
the science of assembling, organizing, and analyzing data,
as well as drawing conclusions about what the data means.
Bibliography
and Suggested Reading
Abuse on the Increase in Cascadia County. (January
2000) Data Users Group of the Oregon Department of Human
Services, Salem, OR. Source: Using Information Management
to Support the Goals of Safety, Permanency and Well Being,
http://www. muskie.usm.maine.edu/SACWIS
Inmon,
W.H., Zachman, J.A., and Geiger, G. (1997) Data Stores,
Data Warehousing and the Zachman Framework: Managing Enterprise
Knowledge. New York: McGraw-Hill.
Milewski,
Emil G. (1989) The Essentials of Statistics I.
Piscataway, NJ: Research and Education Association.
Smoothey,
Marion. (1993) Let's Investigate Statistics. North
Bellmore, NY: Marshall Cavendish Corporation.
Sperling,
A.P. and Levinson, Samuel D. (1998) Arithmetic Made
Simple. Revised edition. New York, NY: Doubleday.
U.S.
Department of Health and Human Services, Administration
for Children and Families. (November 2000) Rethinking
Child Welfare Practice Under the Adoption and Safe Families
Act of 1997. Washington, D.C.: U.S. Government Printing
Office.
Weinbach,
Robert W. and Grinnel, Richard M., Jr. (1991) Statistics
for Social Workers. 2nd edition. White Plains, NY:
Longman Publishing Group.
Williams,
Edward. (1989) Arithmetic the Easy Way. 2nd edition.
Hauppuage, NY: Barron's Educational Series, Inc.
Trainer's
Instructions
1.
Introduce the module by presenting the rationale and objectives.
Refer to the ‘Theme’ flipchart and highlight the theme covered
in this module: increasing reliance on data and reports
to support decision making.
2. Begin
the module using the following as a guide:
Throughout
this training we have been discussing how ASFA has led to
increased use of outcome-based management in child welfare.
Outcome-based, or results oriented management as it is sometimes
called, is a management approach which relies on data, information
and reports to determine if an outcome has been met or not.
Even before ASFA, child welfare organizations collected
data and used it to measure success. One of the major, fairly
recent changes in child welfare is that now data can be
collected by and stored in computers. On the national level
ASFA requires that states report using data obtained from
the big national systems -- AFCARS (Adoption and Foster
Care Analysis and Reporting System) and NCAANDS (National
Child Abuse and Neglect Data System). In addition, most
states use a version of the SACWIS (State Automated Child
Welfare Information System), or some other statewide system
to collect case data and other information. The data collected
can be used to compare how states, or units or regions in
a state are doing in meeting outcomes. The data can be used
to identify successes and areas that might need improvement.
The material that is produced for each measure comes from
the individual pieces of data that are entered in the system.
There
is no question that plenty of data is available. All data
is not equally important, however. As W.H. Immon said in
the book, Data Stores, Data Warehousing and the Zachman
Framework, 'One of the keys to success for modern corporations
is access to the right information at the right time at
the right place in the right form'.
Individual pieces of data in a database mean nothing. People
must decide what pieces of data are important to them, know
how to retrieve them then analyze them to determine in what
ways they can be useful. The computer can't do this procedure
alone. It still requires human intervention. The challenge
for child welfare managers and supervisors is not only to
select the applicable data but to know how to use this data
effectively.
For
example, a child welfare supervisor is getting reports from
her caseworkers that it is very difficult to find substance
abuse treatment programs for the parents of children in
care. Using reports from her computer system, she learns,
that indeed there are several families affected by the lack
of substance abuse facilities. She notices also that some
of the children in these families have been in foster care
for close to six months. Clearly it is time to take action.
At
a unit meeting, she discusses the situation with the caseworkers.
They suggest that possibly the agency could approach the
few substance abuse providers in the area and find out if
by prioritizing the cases, some of those close to deadlines
could be addressed first. The supervisor works on this and
also suggests to her manager that a vigorous effort should
be made to get more substance abuse facilities in the area.
She uses the data she downloaded from her computer to show
her unit's need. From her supervisor she learns that other
supervisors have identified similar problems. They decided
that aggregate data can be used to work on the regional
or state level to address the problem.
This
example is just an illustration of how data and reports
can be used to improve services for children and families.
I am sure many of you have found yourselves in a situation
where the data you have can be organized to support your
position and solve a problem.
3. Transition
to the next exercise using the following as a guide:
Let's discuss for a moment the reports that each of you
need to perform your own job. Look at Overhead Section II.5.1.
It asks you to describe two reports that you use in your
job or if you don't use any reports, two reports that you
wish you had. On handout Section II.5.1 fill in the blanks
for each of these reports, list one way that you use each
of these reports that you would be willing to share with
the group.
4. Give
the group five minutes, then ask for volunteers to share
the name of the reports used most frequently and how they
use the reports in decision making. During each presentation
of a helpful report, probe to ensure that:
-
the title/name of the report is mentioned
-
why and how it works is discussed, and
- the
benefits are mentioned.
Encourage
participants to ask questions of their colleagues about
each presented report to see if others can benefit from
using the report. Determine if there reports mentioned by
the managers are different or the same as those mentioned
by the supervisors.
5. Sum
up the exercise using the following as a guide:
In
all these instances it's the data you have that allows you
to form a basis for decision making and action. Using data,
provides the support for outcome based management that is
so critical to today's child welfare organization. Supervisors
and managers thus must constantly ensure that their day
to day work effort is responsive to an outcomes approach
both at the case and the agency level. This may involve
learning or enhancing skills in planning, managing for results
and evaluating performance. The traditional focus on compliance
with procedures in assessing program performance is no longer
the model. Instead, we must move to results oriented, outcome
based management practices in the child welfare system that
are aimed at increasing accountability and attention to
service performance. This approach puts new emphasis on
responsibility, accountability, effectiveness, and results,
both for child welfare agencies and families.
6. Continue
the discussion using the following as a guide:
Here,
in no priority order, are some tips you might find helpful
as you think about using data, information and reports to
improve decision making. These tips are summarized on handout
Section II.5.2.
-
Assure and constantly look to improve the
quality of the data
If you have inaccurate, incomplete, unreliable, untimely
data in your system, you will have that same lousy data
on your reports. Managers need to add to their list of
things to check in on data quality issues…things like
sponsoring data quality reviews, allowing time for data
clean-up and building procedures that support user responsibility
for entering accurate, timely data into the system.
-
Develop priority performance areas, create reports that
measure performance in the target areas and distribute
the reports to all levels of the agency
Don't attempt to meet every identified report
need immediately. Identifying the management priorities,
communicating the priorities and then phasing in the rollout
of reports on those monitoring the achievement of the
priorities will help assure that the reports are used.
-
Create statewide, regional, area and team versions of
key reports -- consider developing both summary and detail
versions of reports based on user requirements
All users do not need to see the same reports. To act
on the info in the report, some users need all of the
details others only need summary info. Users need to see
information that is most directly relevant to them and
that they recognize.
-
Use data on reports in day to day management, set policy
and inform decision making; incorporate review of key
reports in staff meetings
The message here is USE the reports; don't just
create them. Pick one or two reports that identify priority
data and discuss them at regularly scheduled staff meetings.
Ask questions about why the data is showing what it shows,
discuss progress, agree on next steps either to continue
good progress or improve progress.
-
Avoid the 'gotcha' syndrome
This is very much related to the use of reports. Don't
have every discussion about data be negative…we missed
the mark here, why did we fail to meet the goals? Why
are we behind in our recruitment of foster homes? Why
are you so far behind your colleagues in…whatever? Instead,
use the reports to ask informed questions, focus management,
supervisory and staff attention on a particular issue
or series of issues, identify barriers to success as well
as highlight where success has been achieved. You may
want to think about something as simple as developing
some reports that give both exceptions and positives.
In other words, why can't you design reports that say
85% of our assessments were completed on time and also
say, thus 15% of our initial assessments are overdue?
This method highlights both the done on time as well as
the overdue. Too often, reports just give the 'exceptions'…what
did not meet the target.
-
Ensure that reports are distributed to the appropriate
people
This goes with out saying but it's harder than it seems
to develop a consistent, reliable, up to date report distribution
list and system.
-
Establish an open, user driven report requirement
process that responds to user requests for information,
manages expectations, prioritizes requests and takes frequency
of reports into consideration
States are finding that users will ask for the world but
in reality can only focus on and effectively use so much
data at a time. Setting up a process for prioritizing
report requests, can help shape user expectations, keep
a focus and assure that real (vs. perceived) data needs
are met.
-
Devote resources to meeting data clean up and
report development, assurance and maintenance efforts
The resource needs won't disappear once your
system is live in fact, they probably will only increase
as users get more and more comfortable analyzing data.
-
Provide training to all staff on how to read and use reports
Don't assume that report reading is intuitive, it's not.
Some of the staff who, are now looking at reports haven't
read a graph or chart for several years and may have not
used data analysis skills for at least that long. Just
as people needed to be taught why they needed to use the
SACWIS system and how to read the screens and where to
find key data in the system, they will need that type
of instruction on reports.
-
Develop both on-line and hardcopy reports -- consider
a data warehouse for user defined and canned ad hoc reports
This is an interesting solution to the issue
we just discussed about the need to devote resources to
designing and using reports. Some states that are struggling
to keep up with user access requests have set up a series
of canned, pre-defined reports, let users fill in some
specific query info, like dates or region etc. and then
produce their own reports. Some professions, like finance,
actually let users have access to the whole data base
and train them on how to create and run their own reports
independent of any support.
7. Introduce
the Hawaii Case Study using the following as a guide:
As a child welfare supervisor or manager you are increasingly
being asked to understand how to use data from reports to
monitor the implementation of ASFA and agency goals and
outcomes with a focus on the outcomes for which you are
responsible. The Hawaii case provides us an opportunity
to link data, practice and outcomes.
8. Pass out copies of the Participant’s Version of the Hawaii
Case Study (Section II.5.3). Ask the participants to go
into their small groups, take a few minutes to read the
case study and then begin to answer the questions. Ask that
a recorder be named who will report back to the large group.
Give the group 20 minutes to work on the answers.
9. After
20 minutes, bring all the participants back together and
begin to gather the small group reports by asking one group
to summarize the case situation. Then ask another group
to report how it answered question 1. Process the response
with the whole group and then continue on to questions 2
and 3.
10 Wrap-up
the activity using the following as a guide:
In the case study, you just read through and interpreted
data from several table and linked that data to outcomes
and practice. Lets now look at some real reports and see
what they tell us.
Does
anyone have any questions before we move on to the next
topic in this module?
11.
If you are going to use the M & M County exercise instead
of the Hawaii Case Study begin with this introduction.
Even though you, as a child welfare supervisor or manager
may not need to analyze data, you probably will encounter
situations on the job when you will find it helpful to have
some knowledge of basic data analysis and the way that data
is used. In fact, many supervisors and managers may find
that they need to use data analysis skills on a regular
basis in their job.
What
are statistics anyway? One definition is the science of
assembling, organizing, and analyzing data, as well as drawing
conclusions about what the data means. Sounds interesting
doesn't it? However, people's ideas about statistics vary.
For example, one author (Smoothey) reports, "People
are sometimes suspicious about statistics". A British
Prime Minister, Disraeli, is believed to have said to Queen
Victoria,"There are three kinds of lies: lies, damned
lies and statistics." Or, as someone else said, "Statistics
are like a bikini; shows a lot, but hides the interesting."
This
exercise provides a look at some statistical terms and concepts.
The data we will use comes from M & M County. You know
that because we will use M & M's to illustrate the material,
and when we are done, you can eat the M & M's.
12.
Divide the group into small groups of 4-5 people. Ask each
group to locate handout Section II.5.5, give each group
a package of M&M's and have each group select one person
to record the data on the appropriate tables. Note that
each M&M represents a child with a permanency goal.
The color indicates what kind of goal it is, i.e., reunification,
placement with relative, adoption, etc. Continue:
In
your jobs you probably will not spend much time counting
M & M's so let's imagine that each M & M represents
one child with a permanency goal and that each color represents
what the goal is - reunification, placement with relative,
adoption, etc. Counting the number of children with permanency
goals and the kind of goals they have is something child
welfare supervisors and managers are apt to do.
Now
open the bag and count the total number of M & M's that
you have. Record your total on the sheet in the designated
place.
Then
count the number of M&M's that you have by color. Record
that data in Table 1 under the column headed Frequency.
Frequency is the number of times an event occurs. So, you
can say that the number of red M&M's you have is the
frequency of red M&M's. Remember that red = reunification
so that the number of red M&M’s represents the number
(or frequency) of children with the permanency goal of reunification.
Write on Table 1 the number of each color M &M's that
you have in the appropriate place.
13.
When the group has finished, continue:
The
information you have collected so far is your data. After
you collect your data you have to organize and summarize
it and see what conclusions you can draw from it. Data can
be organized in various ways. One way of organizing data
is to put the data in tables as we have done here. The frequency
of each color M&M lets us compare the number of the
colors we have, or the number of children that have various
kind of permanency goals.
Another
way to compare the number of M&M's that we have is by
determining what percent of each color we have. You may
remember percent from back in grade school. Percent answers
the question 'how many out of 100?' So to find out what
percent of the total that each color represents, you have
to use the following formula.
Total
number of each color of M&M's * 100 =
Total number of M&M's
So
everyone go ahead and calculate the percent of each color
of M&M's the group has and record it on Table 1 on Handout
Section II.5.5. When everyone is finished, ask the group
if they think using percents make it easier or harder to
compare the number of M&M's they have, or the children
on a caseload with different permanency goals.
14.
Ask one group to give you their numbers and write them on
a flipchart. What does that table tell you now? If this
data represented the children on a caseworkers' caseload,
what might the supervisor learn from the table? Can any
conclusions be drawn from this data? What else would you
like to know about this data?
15.
Continue:
Another way of describing what you have done (create a frequency
distribution) is by using what is commonly known as an "average."
In statistics "averages" are called "measures
of central tendency." By using them we can make a reasonable
estimate of the total population.
Using
the M &M's again, let's look at an example of an average.
16.
Show Overhead Section II. 5.6. Ask the group to find the
corresponding handout in their packets. Ask each group how
many of each color M&M's they have and record their
responses on Overhead/handout Section II. 5.6. Then ask
someone to add the responses and divide that number by the
number of groups you have. Explain that the answer you get
is the average of the number of each color M&M's that
we have.
17.
Continue:
Using
information from the same legend that we used before. Each
M & M represents a child with a permanency goal. Each
color represents how that goal will be achieved. If you
are looking at the status of children with permanency goals
in your team’s caseload, what would this table tell you?
How could you use this information?
Could you use this information to compare your unit, or
office with another?
There
are two other measures of central tendency that are sometimes
used. One is the mode which is the value in a set of numbers
that occurs most often (with the highest frequency). The
mode is not used as frequently as the median and the mean
because it is not as precise. The other measure of central
tendency is the median. In a set of numbers it is defined
as the middle value when the numbers are arranged in order
of magnitude. The median is the most stable measure, meaning
it is least affected by extreme values occurring in the
distribution.
Another
way of presenting data is to use charts. Sometime this method
is more effective than using a table.
18.
Show overhead Section II.5.7 and ask the group to find the
handout that shows this overhead in their packets. You can
use the circle on overhead Section II.5.7 to make a Pie
Chart. Using the percentage information they have calculated
in Table 1, have each group draw in the approximate percentage
for each way to achieve the permanency goal.
Ask
the group if this chart would be a good way to compare permanency
goals? Is it easier to understand than a table? What makes
it an effective way to present data?
19.
Then continue:
Or, you can present data in a bar chart. Look at Overhead
Section II.5.8. Using the graph form, draw in how each group's
data would look displayed in this manner. Be sure to label
your axis and draw a scale. Make sure the legend on the
chart represents the ways to achieve the goal of permanency
(not the colors of the M & M's).
Note
that in the examples the frequencies are represented in
the bar chart and the percentages in the pie chart. Does
this arrangement work is displaying these two kinds of data?
20.
When everyone is finished, ask if there are any questions;
then finish the exercise.
We
have reached the end of the M&M's exercise. I hope you
have had some fun learning a little bit about statistics,
data collection, and percent. Learning how to run numbers
through formulas, while a useful skill to have, is not the
main point of this exercise. Rather, the goal was to become
familiar with these analytical tools, learn questions to
ask about data and see if any of these tools can be applied
in your day to day supervision or other operational activities.
And, you can eat the M&M's if you like.
Does
anyone have any questions before we move on to the next
topic?
Now
that you have learned a few data analysis skills, let's
look at some real reports and see what they tell us.
21.
Distribute one or two sample reports. Ask the participants
to look at the reports and launch a discussion by asking
them if the
data on the reports is important, useful, and used. If not
used, what would be more useful?
22.
Distribute the sample reports that focus on ASFA/agency/unit
goals, outcomes and measures or the CFSR/PIP and make some
comparisons between regions, teams, and units. Ask them
to break into their smaller groups to discuss. Show overhead
(Section II.5.9) and ask them to answer the questions which
are also in their handout packet. Specifically, have the
groups 'interpret' the data.
- What
is the data on these reports saying or trying to say?
-
Can you make any conclusions using these figures?
- Can
you see any trends?
-
Do these numbers surprise you? If so, how?
-
What is the connection between your work responsibilities
and the data on these reports?
-
Do you think these figures could be misleading in any
way? If so, how?
23.
After 10 minutes have the group come back together and report
on their discussions. Process the feedback from the small
groups with the large group.
24.
Wrap up the exercise by noting that:
More
and more frequently, child welfare supervisors are expected
to have basic analytical skills such as reading and interpreting
data on reports. Having such skills is another tool that
a child welfare supervisor can rely on to support the complex
decision making that he/she is called on to perform everyday.
25.
Ask for and address questions.
26.
Briefly summarize module.
We have covered a lot of material in this module. We talked
about how you as supervisors and managers use data; reviewed
some basic data analysis tools; analyzed some reports that
you use and shared some helpful practices regarding data
and reports with the rest of the group. This material won't
make you a data expert, but we hope that it will make you
feel more comfortable when you need to use data and reports
on the job.
27.
Introduce the next module, Wrap-up and Evaluation.
|
