Outcome Evaluation Strategies
for Domestic Violence Service Programs
Receiving FVPSA Funding
A Practical Guide
Written by
Eleanor Lyon, Ph.D.
Cris M. Sullivan, Ph.D.
Published by
National Resource Center
on Domestic Violence
6400 Flank Drive, Suite 1300
Harrisburg, PA 17112
November 2007
About the Authors
Dr. Eleanor Lyon is Director of the Institute for Violence Prevention and Reduction, and
Associate Professor in Residence at the School of Social Work at the University of Connecticut
(UConn). She has worked on issues related to violence against women as an advocate and
researcher since 1974. Before returning to UConn in 1998, Eleanor served as coordinator of a
domestic violence shelter program, followed by fifteen years as a researcher at a large community-
based social service / mental health agency.
Eleanor currently teaches classes on violence against women and research methods, in addition to
consulting and directing research and evaluation efforts. She specializes in evaluating programs for
battered women and their children, children who have been abused sexually, and interventions in
public schools; she has also conducted extensive research related to criminal sanctions policy. She
has received many federal, state and local grants to support this work. Among other efforts,
Eleanor helped to coordinate the “Documenting Our Work” project, served for eight years as
evaluator of VAWnet, and is currently the evaluation consultant for the National Center on
Domestic Violence, Trauma and Mental Health. Her most recent project is a national study of
domestic violence shelters. Among her publications is co-authorship of Safety Planning with
Battered Women.
Eleanor also conducts workshops on evaluation and issues related to violence against women. She
can be reached at eleanor.lyon@uconn.edu.
Dr. Cris M. Sullivan is a professor of Ecological/Community Psychology and associate chair of
the Psychology Department at Michigan State University (MSU). She has been an advocate and
researcher in the movement to end violence against women since 1982. In addition to her MSU
appointments, Cris is also the Director of Evaluation for the Michigan Coalition Against Domestic
and Sexual Violence and Senior Research Advisor to the National Resource Center on Domestic
Violence.
Cris’s areas of research expertise include developing and evaluating community interventions for
battered women and their children, improving the community response to violence against
women, and evaluating victim service programs. She has received numerous federal grants to
support her work over the years, including grants from the National Institute of Mental Health,
Centers for Disease Control and Prevention, and the National Institute of Justice. Her most recent
project is a five-year NIMH RISP grant that involves collaborating with a large domestic abuse-
sexual assault victim service organization. The aim of this project is to enhance the organization’s
capacity to engage in collaborative research, and to build a strong partnership for conducting
meaningful and policy-oriented research on violence against women.
In addition to consulting for local, state, federal and international organizations and initiatives, Cris
also conducts workshops on (1) effectively advocating in the community for women with abusive
partners, and their children; (2) understanding the effects of domestic abuse on women and
children; (3) improving system responses to the problem of violence against women; and (3)
evaluating victim service agencies.
Cris can be reached at sulliv22@msu.edu and you can find out more about her work at
www.vaw.msu.edu/core_faculty/cris_sullivan.
Table of Contents
Pages
BACKGROUND
A Brief History of the FVPSA Outcomes Project 1 - 4
A Word About Terminology 4
SECTION I: CONCEPTUAL ISSUES
Chapter 1: Why We Want to Evaluate Our Work
Why Many Domestic Violence Programs Resist Evaluation 5 - 7
Chapter 2: The Difference Between Research and Evaluation
The Difference Between Research and Evaluation 8 - 9
The Impact of Domestic Abuse Victim Services on Survivors’ Safety
and Well-Being: Research Findings to Date 9 - 12
Chapter 3: Important Considerations Before Designing
an Evaluation
Confidentiality and Safety of Survivors 13
Respecting Survivors Throughout the Process 13 - 14
Attending to Issues of Diversity 15 - 16
Chapter 4: A Brief Primer on the Difference Between
Process and Outcome Evaluation 17 -18
Chapter 5: Outcome Evaluation: What Effect Are We Having? 19
The Difference Between Objectives and Outcomes 20 - 21
Why We Caution Against Following Survivors Over Time as
Part of Outcome Evaluation 21 - 22
Choosing Outcomes that Make Sense to Our Programs 22 - 23
“Problematic” Outcome Statements to Avoid 23 - 25
The Hard-to-Measure Outcomes of Domestic Violence Programs 25 - 26
So, What is an Outcome Measure? 27
Chapter 6: The Documenting Our Work (DOW) Project
A Brief History of Documenting Our Work 28 - 30
Results of the DOW Pilot of Shelter Forms 30 - 34
Results of the DOW Pilot of Support Services and Advocacy Forms 34 - 36
Results of DOW Pilot of Support Group Forms 36 - 37
Chapter 7: The FVPSA Outcomes Pilot Project 38 - 43
SECTION II: PRACTICAL ISSUES
Chapter 8: Deciding How Much Information to Gather,
and When
General Guidelines for Using Samples 44 - 46
Special Considerations for Shelter Samples 46
Special Considerations for Support Group Samples 47
Special Considerations for Support Services and Advocacy Samples 48
Chapter 9: Collecting the Information (Data)
Designing a Protocol for Getting Completed Forms Back from Survivors 49 - 51
Creating a Plan With Staff for Collecting Outcome Evaluation Data 52
Collecting Information from Women in the Shelter 53 -54
Collecting Information from Support Clients 54
Collecting Information from Women Using Support Groups or
Group Counseling 55 - 56
Collecting Information from Survivors Participating in Individual
Counseling 56
Alternative Ways to Collect the Information 56 - 57
Chapter 10: Maintaining and Analyzing the Data
Storing the Data 58
Some Data Entry Considerations 59
How to Analyze the Information You Collect 59
Quantitative Information (including Frequencies and Cross tabs) 59 - 63
Qualitative Information 63 - 64
Points of Contact for Additional Information 64
Chapter 11: Sending the Findings to Your FVPSA Administrator 65 - 66
Chapter 12: Making Your Findings Work for You
Using Your Findings Internally 67
Using Your Findings Externally 67 - 68
How to Share Information with Others 69
When Your Findings are “Less than Positive” 70
Using Your Findings to Support the Continuation of Current Programs 70
Using Your Findings to Justify Creating New Programs 70 - 71
Important Points to Remember 71
APPENDICES
Appendix A: Sample Logic Models
Appendix B: The DOW Forms
Appendix C: FVPSA Pilot Project Feedback from Local Programs:
Verbatim Responses to Open-Ended Questions
Appendix D: Instructions for Using the Databases
Appendix E: Annual Report to Send to FVPSA Administrator
Appendix F: Glossary of Terms
Appendix G: Additional Readings
Appendix H: Literature Cited
FVPSA Outcomes Evaluation: A Practical Guide page 1
Outcome Evaluation Strategies
for Domestic Violence Service Programs
Receiving FVPSA Funding
A Practical Guide
The Family Violence Prevention and Services Administration (FVPSA) within the
U.S. Department of Health and Human Services has been the major source of funding for
domestic violence coalitions and programs since 1984. In fiscal year 2007, FVPSA
provided almost $125 million to support the work of community-based domestic violence
programs, state coalitions, and a network of national resource centers. The overall purpose
of this FVPSA outcome evaluation project is to help states develop and implement
outcome evaluation strategies that will accurately capture the impact of FVPSA dollars on
survivors’ safety and wellbeing.
A Brief History of the FVPSA Outcomes Project
In 2005, the Office of Management of the Budget (OMB) reviewed the FVPSA
Program along with other federal grant programs within the Administration for Children
and Families at HHS. The review of the FVPSA program concluded that “results were not
adequately demonstrated.” In response to this finding, a national advisory group of FVPSA
administrators, state coalition directors, local domestic abuse program staff, tribal program
staff and evaluation specialists was convened to develop strategies for more effectively
demonstrating the impact of the FVPSA program.
It was not a simple task to create outcomes that would adequately reflect results
that might be desired across the different services being provided by domestic violence
programs (shelter, support groups, counseling, advocacy, etc.). However, the advisory
group examined evaluation work that had already been occurring in both Michigan and
Pennsylvania, and chose two outcomes that had been accepted by executive directors of
programs in those states and that captured two goals of any service being offered by
FVPSA Outcomes Evaluation: A Practical Guide page 2
domestic violence programs: to safety plan with survivors and to ensure that survivors are
aware of community resources they might need in the future. There was also research
supporting that these two short-term outcomes led to reduced violence and increased
quality of life for survivors over time. (See pages 9-12 of this manual for a summary of this
research.)
This led the advisory group to agree on the following two outcomes to be collected
from all FVPSA grantees by fall of 2008:
• As a result of contact with the domestic violence program, 65% or more of
domestic violence survivors will have strategies for enhancing their safety.
• As a result of contact with the domestic violence program, 65% or more of
survivors will have knowledge of available community resources.
The 65% target was based on programs’ experience and advisors’
recommendations. Although much of the work done by domestic violence programs
involves services related to safety planning and community resources, program staff do not
always have extensive contact with individual survivors, so not all of them would report
changes in these two areas. For this reason, “65% or more” was seen by advisors as a
realistic initial goal. Once programs have begun to collect this information from survivors,
the percentage goals will be changed to reflect figures based on actual data submitted.
Those percentages will then become the outcome goals for the FVPSA funded programs,
and included in the annual report to the Office of Management and Budget (OMB).
These two outcomes were also chosen because they relate not just to individual
level change (the survivor’s safety and well-being), but they also provide evidence,
important to more and more funders, of stronger and safer communities. Specifically,
research has demonstrated that increasing survivors’ knowledge of safety planning and of
community resources leads to increased safety and well-being over time (see Bybee &
Sullivan, 2002; Goodkind, Sullivan, & Bybee, 2004; Sullivan & Bybee, 1999). Since a good
deal of intimate partner abuse happens outside of the home in communities, such as the
workplace (McFarlane et al., 2000; U.S. Dept of Justice Bureau of Justice Statistics, 2001),
safer women means safer communities. Abuse can also have deleterious effects on
survivors’ ability to work and care for themselves and their children. Therefore, again,
improving women’s quality of life directly improves community well-being.
FVPSA Outcomes Evaluation: A Practical Guide page 3
After the national advisory group agreed upon the two outcomes, discussion
centered on how local programs could measure the extent to which those outcomes
occurred, without overburdening them more than necessary. Some of the advisory board
members were also participating in the national Documenting Our Work (DOW) Project,
and that project provided extremely helpful building blocks for the current effort. The
DOW Project had already developed tools that included measuring the two outcomes, and
advisory members discussed how these tools might be shared nationally to assist programs
with evaluating their work. A history of DOW and some of its pilot results are described in
Chapter 6.
The national advisory group was clear in its recommendation that requirements for
collecting and reporting on the two outcomes be phased in for programs, with adequate
training and technical assistance provided. It was suggested that a two-year pilot project be
implemented that would include working with states to determine the best ways to collect
and report these data. This handbook was created as one component of this effort, and is
intended to provide programs with practical strategies for conducting outcome evaluation.
While the manual focuses on collecting the two outcomes mandated for FVPSA
grantees, the strategies can also be used for all outcome evaluations being conducted by
domestic violence organizations. The intent is twofold:
♦ First, programs are feeling external pressure from funding sources to conduct
outcome evaluation, and it is our sincere hope and expectation that the information
gained through the methods in this guidebook will be useful in carrying out such
evaluations in a way that is not overly burdensome.
♦ Second, and more importantly, we hope and expect that the strategies outlined in
this manual will be helpful for programs to conduct evaluations that will be
meaningful to their work and that will lead to providing the most effective services
possible to survivors of domestic violence.
Most immediately, however, we have designed this manual to help programs collect the
two new outcomes for FVPSA grantees.
This manual is divided into three sections. The first focuses on conceptual issues to
consider before conducting an outcome evaluation, and ends with the description of
Documenting Our Work (DOW). The second section provides practical information about
data collection, analysis, interpretation, and presentation of findings. The third section (the
FVPSA Outcomes Evaluation: A Practical Guide page 4
Appendix) includes the actual DOW tools you might want to use, modify or shorten for
your evaluation work, as well as other background and supplemental material we hope
you find helpful.
A Word About The Terminology Used In The Manual
While all those being victimized by an intimate partner deserve effective advocacy,
protection, and support, the overwhelming majority of domestic violence survivors are
women battered by intimate male partners. For that reason, survivors are referred to as
"women" and "she/her" throughout this manual.
A conscious decision was also made to use the term "survivor" instead of "victim"
throughout this manual. Although there is debate about the use of these terms in the field,
the authors are more comfortable describing women, not in terms of their victimization,
but rather by their strengths, courage and resilience.
SECTION 1
CONCEPTUAL ISSUES
FVPSA Outcomes Evaluation: A Practical Guide page 5
Chapter One
Why We Want to Evaluate Our Work
Although the thought of "evaluation" can be daunting, if not downright
intimidating, there are some good reasons why we want to evaluate the job we are
doing. The most important reason, of course, is that we want to understand the impact
of what we are doing on women's lives. We want to build upon those efforts that are
helpful to women with abusive partners; at the same time, we don't want to continue
putting time and resources into efforts that are not helpful or important. Evaluation is
also important because it provides us with "hard evidence" to present to funders,
encouraging them to continue and increase our funding. Most of us would agree that
these are good reasons to examine the kind of job we're doing...BUT...we are still
hesitant to evaluate our programs for a number of reasons.
Why Many Domestic Violence Programs Resist Evaluation
(and reasons to reconsider!)
“Research has been used against women with abusive partners.” It is true that
people can manipulate or misinterpret research data. However, this is actually a
reason why we need to understand and conduct our own evaluations. To effectively
argue against the misinterpretation of other research, we must at least have a general
understanding of how data are collected, analyzed, and interpreted.
(related): “I don't trust researchers.” Too many programs have had bad
experiences with researchers who come into their settings, collect their data, and are
either never heard from again or who then interpret their findings without a basic
understanding of domestic violence issues. In the academic arena we refer to this as
"drive-by data collection," and we would strongly recommend programs turn such
researchers away at the door. But please remember that working with a researcher to
do program evaluation is optional. This handbook is designed to give you the basic
information you will need to conduct your own outcome evaluation.
FVPSA Outcomes Evaluation: A Practical Guide page 6
“Funders (or the public) will use our findings against us.” A common concern
we have heard from program staff is that our own evaluations could be used against
us because they might not "prove" we are effective in protecting women from
intimate violence. This fear usually comes from people who think that the funders
(or the public) expect us, on our own, to end intimate violence against women. We
would argue that it is unrealistic to expect victim service programs to end
victimization -- that is the role of perpetrator service programs as well as the larger
community. We do, however, need to know if we are effectively meeting goals that
are realistic.
"I have no training in evaluation!" That's why you're reading this manual. There
is a scary mystique around evaluation -- the idea that evaluation is something only
highly trained specialists can (or would want to!) understand. The truth is, this
manual will provide you with most, if not all, of the information you need to conduct
a program evaluation.
“We don't have the staff (or money) to do evaluation.” It is true that evaluating
our programs takes staff time and money. One of the ways we need to more
effectively advocate for ourselves is in educating our funding sources that evaluation
demands must come with dollars attached. However, this manual was created to
prevent every program from having to "re-invent the wheel." Hopefully the strategies
outlined in the following chapters will assist you in conducting evaluation without
having to devote more time and money than is necessary to this endeavor.
“Everyone knows you can make data say anything you want to, anyway.”
This actually isn't true. Although data are open to interpretation, such interpretation
has its limits. For example, if you ask survivors, out of context, how often they
slapped their assailants in the last year, and 78% reported they did so at least once,
you could try to make the argument that women are abusive toward men (which is
why it is so important to word questions accurately and ask contextual questions).
On the other hand, if you collected this same information and then claimed women
FVPSA Outcomes Evaluation: A Practical Guide page 7
never slapped their assailants under any circumstances, you would not have the data
to back you up. Data can be manipulated, but only so far. And the more you
understand research and evaluation the more easily you will be able to point out
when and how data are misinterpreted.
“We've already done evaluation [last year, 10 years ago]; we don't need to
do it again.” Things change. Programs change, and staff change. We should
continually strive to evaluate ourselves and improve our work.
*******************
Knowledge is power. And the more service providers and advocates know about
designing and conducting evaluation efforts the better those efforts will be. Evaluating
our work can provide us with valuable information we need to continually improve our
programs.
The next chapter provides a quick description of the distinction between research
and evaluation, and an overview of some of the knowledge we have gained to date from
recent research. As we will explain more fully in the next chapter, it can be helpful to
know what prior research has found about the effectiveness of services for battered
women, so that we can feel confident we are measuring the appropriate short-term
outcomes that will lead to desired long-term outcomes.
FVPSA Outcomes Evaluation: A Practical Guide page 8
Chapter Two
The Difference Between Research and Evaluation
Many people find the distinction between “research” and “evaluation” to be
confusing, but it’s really not complicated.
♦ Research is a broad term that refers to collecting information about a
topic in an organized, systematic way. It can answer many questions that are
interesting and useful to us, such as how widespread domestic violence is in a
particular country, or within a particular age group. It can answer simple
questions such as these (although getting credible answers might be difficult), or
much more complicated questions, such as “what are the primary factors that
contribute to women’s increased safety after an episode of abuse?”
♦ Evaluation is a particular kind of research. It answers questions about
programs or other kinds of efforts to provide services or create change in
some way. Again, the questions can be simple, such as “what did the program
do?” or more complex, such as “how was the program helpful, and for which
people?” Evaluation research, as the term suggests, tries to answer questions
about a program’s “value.”
Both research and evaluation can provide very useful information for domestic
violence programs. Research usually is conducted so that its results can be applied or
“generalized” to broad segments of the population, such as all women who call the
police after an abusive incident. Large evaluation studies may also be designed so that
they can be applied to many programs of a particular type, such as shelter programs.
Most credible research and large evaluations—especially the ones that follow
people over time, to determine long-term outcomes—can be complicated to conduct,
require substantial funding, and are likely to need help from people who have received
specialized training. Without extra resources they are probably beyond the capacity of
most local domestic violence programs to do on their own. Very good and helpful
FVPSA Outcomes Evaluation: A Practical Guide page 9
evaluations can also be done, however, by local programs, and without a huge financial
investment. That is what we hope this manual will help you to do.
Before we turn to more of the conceptual issues involved with your local
evaluation, however, we want to provide an overview of some of the useful results of
recent research and evaluation. Knowing about such results can suggest program ideas,
as well as ideas for questions you can ask about what your program is doing (or not
doing). Using these kinds of research and evaluation results is what is meant by
“evidence-based practice”—something that makes sense and is being urged more and
more frequently. It essentially means using the best scientific evidence you can find to
decide how to provide services or do other things to help people and communities
affected by domestic violence, and to prevent further violence from occurring.
The Impact of Domestic Abuse Victim Services on
Survivors’ Safety and Wellbeing: Research Findings to Date
It can be helpful to know what research studies have found about the
effectiveness of our efforts, so that we can feel confident we are measuring the
appropriate short-term outcomes that will lead to desired long-term outcomes for
survivors. Unfortunately very few studies to date have examined the long-term impact of
victim services on survivors over time. However, the studies that have been conducted
have consistently found such services to be helpful.
Shelter programs have been found to be one of the most supportive, effective
resources for women with abusive partners, according to the residents themselves
(Bennett et al., 2004; Bowker & Maurer, 1985; Gordon, 1996; Sedlak, 1988; Straus,
Gelles, & Steinmetz, 1980; Tutty, Weaver, & Rothery, 1999). For example, Berk,
Newton, and Berk (1986) reported that, for women who were actively attempting other
strategies at the same time, a stay at a shelter dramatically reduced the likelihood they
would be abused again.
One research study used a true experimental design and followed women for two
years in order to examine the effectiveness of a community-based advocacy program for
domestic abuse survivors. Advocates worked with women 4-6 hours a week over 10
FVPSA Outcomes Evaluation: A Practical Guide page 10
weeks, in the women’s homes and communities. Advocates were highly trained
volunteers who could help women across a variety of areas: education, employment,
housing, legal assistance, issues for children, transportation, and other issues. Women
who worked with the advocates experienced less violence over time, reported higher
quality of life and social support, and had less difficulty obtaining community resources
over time. One out of four (24%) of the women who worked with advocates experienced
no physical abuse, by the original assailant or by any new partners, across the two years
of post-intervention follow-up. Only 1 out of 10 (11%) women in the control group
remained completely free of violence during the same period. This low-cost, short-term
intervention using unpaid advocates appears to have been effective not only in reducing
women's risk of re-abuse, but in improving their overall quality of life (Sullivan, 2000;
Sullivan & Bybee, 1999).
Close examination of which short-term outcomes led to the desired long-term
outcome of safety found that women who had more social support and who reported
fewer difficulties obtaining community resources reported higher quality of life and less
abuse over time (Bybee & Sullivan, 2002). In short, then, there is evidence that if
programs improve survivors’ social support and access to resources, these serve as
protective factors that enhance their safety over time. While local programs are not in the
position to follow women over years to assess their safety, they can measure whether
they have increased women’s support networks and their knowledge about available
community resources.
The only evaluation of a legal advocacy program to date is Bell and Goodman’s
(2001) quasi-experimental study conducted in Washington, DC. Their research found
that women who had worked with advocates reported decreased abuse six weeks later,
as well as marginally higher emotional well-being compared to women who did not
work with advocates. Their qualitative findings also supported the use of
paraprofessional legal advocates. All of the women who had worked with advocates
talked about them as being very supportive and knowledgeable, while the women who
did not work with advocates mentioned wishing they had had that kind of support while
they were going through this difficult process. These findings are promising but given the
lack of a control group they should be interpreted with extreme caution.
FVPSA Outcomes Evaluation: A Practical Guide page 11
Another research study examined domestic abuse survivors’ safety planning
efforts (Goodkind, Sullivan, & Bybee, 2004). Survivors were asked what strategies they
had used to stop or prevent the abuser’s violence. For every strategy mentioned, women
were asked if it made the abuse better, worse, or had no effect. Not surprisingly, for
every strategy that made the situation better for one woman, the same strategy made the
situation worse for another. However, the two strategies that were most likely to make
the situation better were contacting a domestic violence program, and staying at a
domestic violence shelter. These results provide strong support for the importance of
domestic violence programs.
It is also important, though, that women who were experiencing the most
violence and whose assailants had engaged in the most behaviors considered to be
indicators of potential lethality were the most actively engaged in safety planning
activities, but remained in serious danger, despite trying everything they could. These
findings highlight the importance of remembering that survivors are not responsible for
whether or not they are abused again in the future. For some women, despite any safety
strategies they employ, the abuser will still choose to be violent.
Evaluations of support groups have unfortunately been quite limited. One notable
exception is Tutty, Bidgood, and Rothery’s (1993) evaluation of 12 “closed” support
groups (i.e., not open to new members once begun) for survivors. The 10-12 week,
closed support group is a common type of group offered to survivors, and typically
focuses on safety planning, offering mutual support and understanding, and discussion of
dynamics of abuse. Tutty et al.’s (1993) evaluation involved surveying 76 women before,
immediately after, and 6 months following the group. Significant improvements were
found in women’s self-esteem, sense of belonging, locus of control, and overall stress
over time; however, fewer than half of the original 76 women completed the 6-month
follow-up assessment (n = 32), and there was no control or comparison group for this
study. Hence, these findings, too, should be interpreted with extreme caution.
Tutty’s findings were corroborated by a more recent study that did include an
experimental design (Constantino, Kim, & Crane, 2005). This 8-week group was led by a
trained nurse and focused on helping women increase their social support networks and
access to community resources. At the end of the eight weeks the women who had
participated in the group showed greater improvement in psychological distress
FVPSA Outcomes Evaluation: A Practical Guide page 12
symptoms and reported higher feelings of social support. They also showed less health
care utilization than did the women who did not receive the intervention.
These research studies are presented to provide you with some evidence
supporting the long-term effectiveness of the types of services you offer. If programs can
show that they have had positive short-term impacts on women’s lives that have been
shown to lead to longer-term impacts on their safety and well-being, this should help
satisfy funders that the services being provided are worthwhile. The two outcomes that
will be required—help with safety planning and increased knowledge of community
resources—are clearly vital short-term outcomes that have been demonstrated to
contribute to improvements in longer-term safety and well-being. These are among the
short-term impacts that this manual will help you to measure.
FVPSA Outcomes Evaluation: A Practical Guide page 13
Chapter Three
Important Considerations Before
Designing any Evaluation
Before even beginning any evaluation efforts, all programs should consider three
important issues: (1) how you will protect the confidentiality and safety of the women
providing you information, (2) how to be respectful to women when gathering and using
information, and (3) how you will address issues of diversity in your evaluation plan.
Confidentiality and Safety of Survivors
The safety of the women with whom we work must always be our top priority.
The need to collect information to help us evaluate our programs must always be
considered in conjunction with the confidentiality and safety of the women and children
receiving our services. It is not ethical to gather information just for the sake of gathering
information; if we are going to ask women very personal questions about their lives,
there should always be an important reason to do so, and their safety should not be
compromised by their participation in our evaluation. The safety and confidentiality of
women must be kept in mind when (1) deciding what questions to ask; (2) collecting the
information; (3) storing the data; and (4) presenting the information to others.
Respecting Survivors Throughout the Process
When creating or choosing questions to ask women who use our services, we
must always ask ourselves whether we really need the information, how we will use it,
whether it is respectful or disrespectful to ask, and who else might be interested in the
answers. As an example, let's assume we are considering asking women a series of
questions about their use of alcohol or drugs. The first question to ask ourselves is:
How will this information be used? – To ensure women are receiving adequate services?
To prevent women from receiving services? Both? If this information is not directly
relevant to our outcome evaluation efforts, do we really need to ask?
FVPSA Outcomes Evaluation: A Practical Guide page 14
Second, how should we ask these questions in a respectful way? First and
foremost, women should always be told why we are asking the questions we're asking.
And whenever possible, an advisory group of women who have used our services should
assist in supervising the development of evaluation questions. The next question is: who
else might be interested in obtaining this information? Assailants' defense attorneys?
Child Protective Services? Women should always know what might happen to the
information they provide. If you have procedures to protect this information from others,
women should know that. If you might share this information with others, women need
to know that as well. Respect and honesty are key.
The words anonymous and confidential have different meanings.
Although many people incorrectly use them interchangeably, the distinction
between these two words is important.
Anonymous - you do not know who the responses came from. For example,
questionnaires without names or other traceable identifiers left in locked boxes
are anonymous.
Confidential - you do know (or can find out) who the responses came from, but
you are committed to keeping this information to yourself. A woman who
participates in a focus group is not anonymous, but she expects her responses to
be kept confidential.
Attending to Issues of Diversity
Most domestic violence service delivery programs are aware that they must meet
the needs of a diverse population of women, children, and men. This requires taking steps
to ensure our programs are culturally competent, as well as flexible enough to meet the
needs of a diverse clientele.
Cultural competence is more than just "expressing sensitivity or concern" for
individuals from all cultures (cultural sensitivity). A culturally competent program is one
that is designed to effectively meet the needs of individuals from diverse cultural
backgrounds and experiences. It involves understanding not only the societal oppressions
faced by various groups of people, but also respecting the strengths and assets inherent in
different communities. This understanding must then be reflected in program services,
staffing, and philosophies.
In addition to diversity in culture, there is a great deal of other variability among
the individuals needing domestic violence service delivery programs, including diversity
across:
♦ age
♦ citizenship status
♦ gender identity
♦ health (physical, emotional, and mental)
♦ language(s) spoken
♦ literacy
♦ physical ability and disability
♦ religious and spiritual beliefs
♦ sexual orientation
♦ socioeconomic status
Although process evaluation is commonly thought of as the best way to understand the
degree to which our programs meet the needs of women from diverse experiences and
cultures (see Chapter 3), outcome evaluation should also attend to issues of diversity.
FVPSA Outcomes Evaluation: A Practical Guide page 16
This handbook takes the position that outcome evaluation must be designed to answer
the question of whether or not women attained outcomes they identified as important to
them. So for example, before asking women if they obtained a protective order, you must
first ask if they wanted a protective order. Before asking if your support group decreased a
woman's isolation, you would want to know if she felt isolated before attending your
group. Not all women seek our services for the same reasons, and our services must be
flexible to meet those diverse needs. Outcome evaluation can inform you about the
different needs and experiences of women and children, and this information can be used
to inform your program as well as community efforts.
Attending to issues of diversity in your outcome evaluation strategies involves: (1)
including the views and opinions of women and children from diverse backgrounds and
experiences in all phases of your evaluation; (2) including "demographic" questions in your
measures (e.g., ethnicity, age, primary language, number of children, sexual orientation)
that will give you important information about respondents' background and situations;
and (3) pilot testing your outcome measures with individuals from diverse cultures,
backgrounds, and experiences.
FVPSA Outcomes Evaluation: A Practical Guide page 17
Chapter Four
A Brief Primer on the Difference Between
Process and Outcome Evaluation
Even though this handbook focuses primarily on outcome, not process, evaluation,
there is enough confusion about the difference between the two to warrant a brief
discussion of process evaluation. Process evaluation assesses the degree to which your
program is operating as intended. It answers the questions:
What (exactly) are we doing?
How are we doing it?
Who is receiving our services?
Who isn't receiving our services?
How satisfied are service recipients?
How satisfied are staff? Volunteers?
How are we changing?
How can we improve?
These are all important questions to answer, and process evaluation serves an
important and necessary function for program development. Examining how a program is
operating requires some creative strategies and methods, including interviews with staff,
volunteers, and service recipients, focus groups, behavioral observations, and looking at
program records. Some of these techniques are also used in outcome evaluation, and are
described later in this handbook.
When designing outcome measures, it is common to include a number of "process-
oriented" questions as well. This helps us determine the connection between program
services received and outcomes achieved. For example, you might find that women who
received three or more hours of face-to-face contact with your legal advocate were more
likely to report understanding their legal rights than were women who only talked with
your legal advocate once over the phone. Or you might discover that residents of your
shelter were more likely to find housing when a volunteer was available to provide them
with transportation.
FVPSA Outcomes Evaluation: A Practical Guide page 18
Process evaluation is also important because we want to assess not just whether a
woman received what she needed (outcome), but whether she felt "comfortable" with the
staff and volunteers, as well as with the services she received. For example, it is not
enough that a woman received the help she needed to obtain housing (outcome), if the
advocate helping her was condescending or insensitive (process). It is also unacceptable if
a woman felt "safe" while in the shelter (outcome) but found the facility so dirty (process)
she would never come back.
In summary…
♦ PROCESS EVALUATION helps us assess what we are doing, how we
are doing it, why we are doing it, who is receiving the services, how
much recipients are receiving, the degree to which staff, volunteers,
and recipients are satisfied, and how we might improve our programs.
♦ OUTCOMES EVALUATION assesses program impact – What
occurred as a result of the program? Outcomes, as we discuss in the
next chapter, must be measurable, realistic, and philosophically tied to
program activities.
FVPSA Outcomes Evaluation: A Practical Guide page 19
Chapter Five
Outcome Evaluation:
What Effect Are We Having?
It is extremely common for people to confuse process evaluation with outcome
evaluation. Although process evaluation is important -- and discussed in the previous
chapter -- it is not the same as outcome evaluation.
OUTCOME EVALUATION
assesses what occurred as a direct result of the program. Outcomes must be
measurable, realistic, and philosophically tied to program activities.
One of the first places many people get "stuck" in the evaluation process is with all of
the terminology involved.
Objectives
Goals
Outcomes
Outcome Measures
These terms have struck fear in the hearts of many, and are often the cause of
abandoning the idea of evaluation altogether. One reason for this is that the terms are not
used consistently by everyone. Some people see goals and objectives as interchangeable, for
example, while others view objectives and outcomes as the same. What is more important
than memorizing terminology is understanding the meaning behind the labels. This manual
will describe the concepts behind the terms so even if a specific funder or evaluator uses
different terminology than you do, you will still be able to talk with each other!
FVPSA Outcomes Evaluation: A Practical Guide page 20
The Difference Between Objectives and Outcomes
Effective evaluation begins by first defining our overarching goals (sometimes also
referred to as objectives). Goals or objectives (and we’re using these terms
interchangeably; not everyone does) are what we ultimately hope to accomplish through
the work we do. Program goals, usually described in our mission statements, are long-term
aims that are difficult to measure in a simple way.
We would say that the OVERALL GOAL OR OBJECTIVE of domestic violence
victim service programs is to –
enhance safety and justice
for battered women and their children
While it is not important that you agree with this overall objective, it is important
that you choose goals and objectives that make sense for your agency. After the program's
overall objective has been established, it is important to consider what we expect to see
happen as a result of our program, that is measurable, that would tell us we are meeting
our objective(s). These are program OUTCOMES.
The critical distinction between goals and outcomes is that outcomes are statements
reflecting measurable change due to your programs' efforts. Depending on the individual
program, PROGRAM OUTCOMES might include:
a survivor's immediate safety
the immediate safety of the survivor's children
a survivor's increased knowledge about domestic violence
a survivor's increased awareness of options
a survivor's decreased isolation
a community's improved response to battered women and their children
the public's increased knowledge about domestic violence
a perpetrator's cessation of violence (NOTE: only for programs that focus
specifically on the abuser)
FVPSA Outcomes Evaluation: A Practical Guide page 21
There are two types of outcome we can evaluate: long-term outcomes and short-
term outcomes. Long-term outcomes involve measuring what we would expect to
ultimately occur, such as:
increased survivor safety over time
reduced incidence of abuse in the community
reduced homicide in the community
improved quality of life of survivors
As we noted in Chapter 2, measuring long-term outcomes is very labor intensive,
time consuming, and costly. Research dollars are generally needed to adequately examine
these types of outcomes. More realistically, you will be measuring the short-term
outcomes that we expect to lead to the longer-term outcomes.
Why We Caution Against Following Survivors Over Time
as Part of Outcome Evaluation
Some funders are now asking grantees to follow their clients over time (sometimes
for as long as six months or a year) to obtain longer-term outcome data. While we
understand the desire for such data, this again is where we must differentiate between the
roles and capabilities of service programs and researchers. Safely tracking, locating, and
interviewing survivors over time is extremely costly, time-consuming, and resource-
intensive to do correctly. And we have yet to hear of a case where the funder mandating
this new activity is also providing additional money to pay for this additional work.
In the study mentioned in Chapter 3 that involved interviewing survivors every six
months over two years, the investigators were able to successfully locate and interview
94% of the participants at any time point. The investigators compared the women who
were easy to find with the women who were more difficult to track, and discovered that
the "easy to find" women were more likely to be white, were more highly educated, were
more likely to have access to cars, were less depressed, and had experienced less
psychological and physical abuse compared to the women who were more difficult to find
(Sullivan et al., 1996).
FVPSA Outcomes Evaluation: A Practical Guide page 22
The moral of the story is: If you do follow-up interviews with clients, be careful in
your interpretation of findings. The survivors you talk to are probably not representative of
all the people using your services. It is therefore our position that programs should not
waste the resources to gather information that is not likely to be accurate. Rather, they
should spend more time and attention engaging in outcome evaluation that is likely to give
them useful and trustworthy data.
Choosing Outcomes That Make Sense to Our Programs
One of the reasons that many domestic violence victim service program staff have
difficulty applying outcome evaluation to their work is that traditional outcome evaluation
trainings and manuals do not apply to our work. Instead they focus on programs that are
designed to change the behaviors of their clients: for instance, literacy programs are
designed to increase people’s reading and writing skills, AA programs are designed to help
people stay sober, and parenting programs are designed to improve the manner in which
people deal with their children. We, however, are working with victims of someone else’s
behavior. They did not do anything to cause the abuse against them, and we therefore are
not about changing their behaviors. For our work, then, we need to take a more expanded
view of what constitutes an outcome:
An OUTCOME
is a change in knowledge, attitude, skill, behavior,
expectation, emotional status, or life circumstance
due to the service being provided.
Some of our activities are designed to increase survivors’ knowledge (for example,
about the dynamics of abuse, typical behaviors of batterers, or how various systems in the
community work). We also often work to change survivors’ attitudes if they come to us
blaming themselves for the abuse, or believing the lies they have been told repeatedly by
the abuser (e.g., that they are crazy, unlovable, or bad mothers). We also teach many
FVPSA Outcomes Evaluation: A Practical Guide page 23
clients skills, such as budgeting and planning, how to behave during court proceedings or
how to complete a resume, and some clients do modify their behavior if they come to us
wanting to stop using drugs or alcohol, or wanting to improve their parenting.
Domestic violence victim service programs also change people’s expectations
about the kinds of help available in the community. For some clients we may lower their
expectations of the criminal legal system (for example if they think their abuser will be put
in prison for a long time for a misdemeanor) while for others we might raise their
expectations (for example if they are from another country and have been told by the
abuser that there are no laws prohibiting domestic violence).
Many of our services are designed to result in improved emotional status for
survivors, as they receive needed support, protection and information, and finally, we
change some clients’ life circumstances by assisting them in obtaining safe and affordable
housing, becoming employed, or going back to school.
REMEMBER: An OUTCOME
is a change in knowledge, attitude, skill, behavior, expectation,
emotional status, or life circumstance
due to the service being provided.
Because survivors come to us with different needs, from different life
circumstances, and with different degrees of knowledge and skills, it is important that our
outcomes first start with where each client is coming from. We do not, for example, want
to say that 90% of our clients will obtain protection orders, because we know that many
survivors do not want such orders or believe they would endanger them further. Instead,
then, we might say that: Of the women who want and are eligible for protection orders,
90% will accurately complete and file them.
"Problematic" Outcome Statements to Avoid
A common mistake made by many people designing project outcomes is
developing statements that are either (1) not linked to the overall program's objectives, or
(2) unrealistic given what the program can reasonably accomplish. Five common
FVPSA Outcomes Evaluation: A Practical Guide page 24
problematic outcome statements are listed on the following pages, with explanations for
why they should be avoided:
Problematic Outcome Statement #1
"50% of the women who use this service will leave their abusive partners."
The expectation that all battered women should leave their abusive partners is
problematic for a number of reasons, including: it wrongly assumes that leaving the
relationship always ends the violence, and it ignores and disrespects the woman's
agency in making her own decision. This type of "outcome" should either be
avoided altogether or modified to read, 'xx% of the women using this service who
want to leave their abusive partners will be effective in doing so.'
Problematic Outcome Statement #2
"The women who use this program will remain free of abuse."
Victim-based direct service programs can provide support, information,
assistance, and/or immediate safety for women, but they are generally not
designed to decrease the perpetrator's abuse. Suggesting that victim
focused programs can decrease abuse implies the survivor is at least
somewhat responsible for the violence perpetrated against her.
Problematic Outcome Statement #3
"The women who work with legal advocates will be more likely to press charges."
Survivors do not press charges; prosecutors press charges. It should also not be
assumed that participating in pressing charges is always in the woman's best
interest. Legal advocates should provide women with comprehensive information
to help women make the best-informed decisions for themselves.
Problematic Outcome Statement #4
"The women who work with legal advocates will be more likely to cooperate with
the criminal justice system."
Again, women should be viewed as competent adults making the best decision(s)
they can for themselves. Women who choose not to participate in pressing
charges should not be viewed as "noncompliant" or "uncooperative." Until the
criminal justice system provides women with more protection, and eliminates
gender and racial bias and other barriers to justice, it should not be surprising when
women choose not to participate in the criminal justice process.
FVPSA Outcomes Evaluation: A Practical Guide page 25
Problematic Outcome Statement #5
"An outcome of this program will be that the number of calls to the police will
decrease."
First, if this is not a well-funded research study you probably will not have the
resources to find out if calls to the police decrease. But more importantly, a
decrease in the number of calls to the police does not necessarily mean violence
has decreased. It could mean women are more hesitant to contact the police or
that perpetrators are more effective in preventing women from calling the police.
That some programs feel compelled by funders to create outcome statements such
as these is understandable. However, the cost is too high to succumb to this urge. It is one
of our goals to educate the public about domestic violence, and that includes our funders.
If funders have money to spend to eradicate domestic violence, we must educate them
about the appropriate ways to spend that money. We can not do that effectively unless
they understand why abuse occurs in relationships, and that survivors are not responsible
for ending the abuse.
The Hard-to-Measure Outcomes of Domestic Violence
Programs
Why is it so difficult to evaluate domestic violence programs? In addition to the
obvious answer of "too little time and money," many domestic violence programs' goals
involve outcomes that are difficult to measure. An excellent resource for designing
outcomes within non-profit agencies is "Measuring program outcomes: A practical
approach," distributed by the United Way of America (see List of Additional Readings in
the back of this manual for more information). In an especially applicable section entitled
"Special problems with hard-to-measure outcomes" (p. 74), the United Way manual lists
nine situations that present special challenges to outcome measurement. They are
included here, since one or more are evident in most domestic violence programs. Where
applicable, the statement is followed by the type of domestic violence service that is
especially susceptible to this problem:
FVPSA Outcomes Evaluation: A Practical Guide page 26
1. Participants are anonymous, so the program cannot later follow up on the
outcomes for those participants. 24-hour crisis line
2. The assistance is very short-term. 24-hour crisis line; sometimes support groups,
counseling, shelter services, some legal advocacy
3. The outcomes sought may appear to be too intangible to measure in any systematic
way. 24-hour crisis line, counseling, support groups, some shelter services
4. Activities are aimed at influencing community leaders to take action on the part of
a particular issue or group, such as advocacy or community action programs.
systems advocacy programs
5. Activities are aimed at the whole community, rather than at a particular, limited set
of participants. public education campaigns
6. Programs are trying to prevent a negative event from ever occurring.
7. One or more major outcomes of the program cannot be expected for many years,
so that tracking and follow-up of those participants is not feasible.
8. Participants may not give reliable responses because they are involved in substance
abuse or are physically unable to answer for themselves.
9. Activities provide support to other agencies rather than direct assistance to
individuals.
On the one hand, it is heartening to know that (1) the United Way of America
recognizes the challenges inherent to some organizations' efforts, and (2) it is not [simply]
our lack of understanding contributing to our difficulty in creating logic models for some of
our programs. On the other hand, just because some of our efforts are difficult to measure
does not preclude us from the task of evaluating them. It just means we have to try harder!
We have included logic models for some of the common domestic violence services being
offered, in case those would be helpful to you with other funders. They can be found in
Appendix A.
FVPSA Outcomes Evaluation: A Practical Guide page 27
So, What is an Outcome Measure?
Outcome measures are sources of information that tell us whether or to what extent
an outcome has been achieved. So, for example, if the desired outcome is that women
who use our services will know more about community resources, how would we know
whether that had occurred? We might develop a brief survey for them to complete, or we
might interview them face-to-face with a questionnaire....these different ways to determine
whether the outcome has been achieved are called outcome measures because they
measure, or document, whether the change has occurred.
Common types of outcomes measures are:
Paper and pencil surveys
Questionnaires completed in interview format
Mail surveys
Telephone surveys
Staff documentation (for example, documentation regarding how many
protection orders were filed)
In the late 1990s the Documenting Our Work (DOW) project was initiated
nationally to examine the efforts, successes and challenges of the Battered Women’s
Movement. One component of that project was to design outcome evaluation strategies
that local programs could use to evaluate their work. Because the DOW project is directly
relevant to the FVPSA outcomes project, the next chapter describes DOW and its findings
in more detail.
FVPSA Outcomes Evaluation: A Practical Guide page 28
Chapter Six
The Documenting Our Work Project
We describe the Documenting Our Work project here in some detail for several reasons:
It provides useful examples of short-term outcomes measures.
It includes the two outcomes that will be required by FVPSA.
It involves collecting information from survivors, which includes
documentation of the services they wanted.
It was created by people who work in the domestic violence movement.
It involved testing the forms and making changes based on the results.
A Brief History of Documenting Our Work
The National Resource Center on Domestic Violence (NRC) initiated the
Documenting Our Work (DOW) Project in 1998 following discussions among state
coalition directors, women of color activists, and FVPSA state administrators. They agreed
on the need to carefully develop tools for the domestic violence field that would document
its work with and on behalf of battered women at both the state and local levels. There
was a commitment to use this documentation to strengthen and inform program, policy
and research, to increase our understanding of its impact on individuals and communities,
and to help guide future directions. The NRC formed a multi-disciplinary advisory group of
evaluators, coalition directions, local program directors, and state administrators to begin
exploring definitions, goals and objectives, and measures.
During the initial stages of the project, a tremendous amount of information was
collected from the field through targeted focus groups with representative from
underserved communities, from advisory group meetings and conference calls, and
through discussions with others engaged in documentation and outcome measurement.
One result of the Documenting Our Work Project was the development of a
number of tools that programs and coalitions can use to evaluate themselves. They are:
FVPSA Outcomes Evaluation: A Practical Guide page 29
State Coalitions Tools
1. A tool for community partners to complete that documents the coalition’s
community collaboration efforts;
2. A tool for staff to complete that documents the coalition’s community
change efforts; and
3. A tool for staff to complete that is an internal assessment of the coalition’s
goals and activities.
Local Program Tools
1. A tool for community partners to complete that documents the program’s
community collaboration efforts;
2. A tool for community members to complete that documents how the
program is perceived and supported in the community;
3. A tool for staff to complete that documents the program’s systems change
efforts; and
4. A tool for staff and volunteers to complete about their experience working
in the program: their activities, training, support, involvement in decisions,
and other issues.
Local program assessment tools that have been designed for survivors to
complete include surveys evaluating the following services: Shelter, Support Services
and Advocacy, Support groups, and Counseling. A 24-hour hotline form was also
developed that staff members complete at the end of crisis calls.
In examining the DOW client-centered surveys it became clear that questions were
already embedded in them that could be modified slightly to measure the two FVPSA
outcomes. Specifically, wording could be in the form of statements that clients can
either agree or disagree with:
Because of the services I have received from this program so far, I feel I
know more about community resources.
Because of the services I have received from this program so far, I feel I
know more ways to plan for my safety.
FVPSA Outcomes Evaluation: A Practical Guide page 30
The DOW tools had also been pilot-tested across programs from four different
states in order to determine if they were brief, clear, easy to use, and viewed favorably
by both survivors and program staff members. Some of the findings from that pilot are
presented below.
Results of the DOW Pilot of Shelter Forms
Two forms were created for shelter residents. The first survey, designed to be
completed by residents shortly after arriving at the shelter, included questions about
how women heard about the shelter, what their preconceived ideas about it were, and
the kinds of help they were looking for. The second survey, which is completed shortly
before women leave shelter, asks about the extent to which women’s needs were met
by the program. Questions also pertain to how long the woman was at shelter, her
experience with rules and other residents, and whether she would recommend the
program to a friend in similar need. The forms were completed by 75 women across
programs in four states.
• 44% completed form 1 only
• 19% completed form 2 only
• 37% completed both forms
A few brief findings from the pilot are presented here, to give you a flavor of the types of
helpful information programs can get from their clients.
Residents were asked what the experience was like for them upon entering shelter.
Women responded:
Table 1. When I First Arrived…
Staff made me feel welcome
95%
Staff treated me with respect
93%
The space felt comfortable
85%
Other women made me feel welcome
78%
It seemed like a place for women like me
73%
FVPSA Outcomes Evaluation: A Practical Guide page 31
Shelter staff seeing these results would feel pleased that survivors felt welcomed
and treated with respect when they first entered the program.
Another interesting finding was that 23% of the women had “concerns” about
contacting shelter. The most common concerns were:
• Shame or embarrassment about abuse
• Safety at the shelter
• Fear of the unknown—didn’t know what to expect
Women were also asked to check off all of the kinds of help they were looking
for while in shelter. Their responses are in the next table.
Table 2. Kinds of Help Women Wanted at Shelter Entry
• Safety for myself
88%
• Paying attention to my children’s wants and needs*
88%
• Learning about my options and choices
85%
• Paying attention to my own wants and needs
85%
• Understanding about domestic violence
83%
• Counseling for myself
83%
• Learning how to handle the stress in my life
81%
• Finding housing I can afford
81%
• Emotional support
80%
• Connections to other people who can help me
78%
• Safety for my children*
73%
• Support from other women
70%
• Dealing with my children when they are upset
• or causing trouble*
65%
• A job or job training
59%
• Health issues for myself
58%
• Strategies for enhancing my own and my children’s safety
56%
• Counseling for my children*
56%
• Planning ways to make my relationship safer
51%
• Custody or visitation questions*
50%
• Budgeting and handling my money
49%
• Education/school for myself
48%
• Education for my children*
46%
• Health issues for my children*
46%
• Transportation
46%
• Other government benefits
44%
• Legal system/legal issues
44%
• Reconnecting with my community
42%
• Leaving my relationship
42%
FVPSA Outcomes Evaluation: A Practical Guide page 32
Table 2. Kinds of Help Women Wanted at Shelter Entry (continued)
• TANF (welfare) benefits
34%
• Child protection system issues*
29%
• Protective/restraining order
27%
• My abuse-related injuries
20%
• My abuser’s arrest
17%
• My own arrest
9%
• My children’s abuse-related injuries*
8%
• Immigration issues
7%
*Items were responded to by mothers only
This simple listing can be enormously helpful for program planning—to make sure
that the program emphasizes the services most needed by women, to the extent it can—
and for fund-raising. Most program staff would expect that women’s safety would be at the
top of the list. Some might be surprised, however, that help with “paying attention to my
own wants and needs” was ranked so high among all of the choices available. Others
might find it worth noting that help with “leaving my relationship” was checked by less
than half of the women in this test. Although help with “immigration issues” was checked
by a small percentage, this result is likely to vary by location. This type of response could
alert a program to unknown gaps in service, and lead to increased resources.
Useful information was obtained from the survey completed by women upon
shelter exit as well. For example, women were asked to indicate, for every need they had
while in shelter, whether they received all the help they needed, some of the help they
needed, or none of the help they needed with that issue. Some of the findings are in the
following table.
Table 3. Extent of Help Women Received While in Shelter *
Type of help needed:
All
Some
Safety for myself
98%
2%
Learning about my choices and options
67%
30%
Learning to handle stress in my life
65%
27%
Finding affordable housing
51%
34%
Budgeting and handling my money
38%
26%
Job or job training
36%
24%
* Results are only for women who indicated they wanted this type of help
FVPSA Outcomes Evaluation: A Practical Guide page 33
It is important that these results distinguish those who wanted the service from
those who did not. It shows that 98% of the women who wanted safety for themselves got
all of that kind of help that they wanted. This was also true for 71% of the women who
wanted help with “understanding about domestic violence” (not shown in the table); the
remainder reported that they got “some of” that kind of help. Interestingly, all of the
women who stayed in shelter for two months or more got all the help they wanted with
understanding domestic violence.
The survey women complete upon shelter exit is also extremely useful for
outcome evaluation. A number of the items on the survey ask specifically about how
the shelter experienced affected women’s lives. Some of these outcome findings are in
the next table.
Table 4. Because of My Experience in the Shelter, I Feel:
(%s do not total 100 because women could check more than one)
Better prepared to keep myself and my children safe
1
92%
More comfortable asking for help
90%
I have more resources to call upon when I need them
87%
More hopeful about the future
87%
I know more about my options
87%
More comfortable talking about things that bother me
82%
That I will achieve the goals I set for myself
79%
I can do more things on my own
79%
More confident in my decision-making
76%
These are certainly positive results—especially when a quarter of the women had
been in shelter a week or less. Over three-quarters of the women who stayed in the shelter
felt more confident of their decisions and abilities, and nearly nine out of ten felt better
informed and more comfortable asking for help. These are among the outcomes most
shelter staff would hope that residents would attain. In addition, the overwhelming
majority of women who participated in the pilot indicated that they had obtained safety
and emotional support during their time in the shelter.
1
This option was on the original shelter #2 form, and clearly was the one most frequently selected by women in the
pilot test. However, discussion with advisors concluded that it sounded too much like the women had control over their
safety (and their abusive partners’ use of violence against them, in particular), so it has been changed to “I know more
ways to plan for my safety” on the latest version of the form.
FVPSA Outcomes Evaluation: A Practical Guide page 34
These findings can be used to identify training needs, or to provide examples to
illustrate training issues programs already cover. For example, at shelter entry 73% of the
respondents checked “[the shelter] seemed like a place for women like me.” However,
when this item was examined across different racial and ethnic groups it was found that
Latinas were less likely than other women to check this response –just 57% did. This
finding might be shared with staff, so that culturally appropriate welcoming strategies
could be emphasized. Alternatively, this finding might highlight a resource issue – a need
for more bicultural staff, or modifications in shelter décor.
The questions on Shelter form #2 that asked about the help women and children
received can also identify training-related needs. For example, about 4 out of 5 women
indicated they wanted help “learning how to handle the stress in my life.” Of those,
however, 8% indicated they did not get help with this issue, and another 27% did not get
as much help as they wanted. This might be a useful topic to elaborate on in training with
staff and volunteers. Similarly, half of the women wanted help with “budgeting and
handling my money,” but over a quarter of this group did not get any help with this issue.
Information about budgeting can be invaluable for women trying to move toward
independence and self-sufficiency for the first time; making sure that staff and volunteers
are prepared to assist effectively may be a very useful part of comprehensive services.
These are just a few examples out of many ways that collecting this information
from survivors in shelter can help programs become more responsive to women’s needs.
Results of the DOW Pilot of Support Services & Advocacy Forms
This survey was created to obtain brief, specific feedback from women receiving
support services. The forms capture the types of help clients wanted, as well as what they
received. Survivors also indicate how many times they met with an advocate, their feelings
about how respected and supported they felt, and overall how satisfied they were with the
services.
Three states participated in piloting these forms, and 42 women responded. Most
(77%) were under the age of 35, and 40% were women of color.
The following table presents the types of assistance women reported wanting from
the domestic violence program’s support services:
FVPSA Outcomes Evaluation: A Practical Guide page 35
Table 5. Types of Advocacy Women Mentioned Wanting
(Percents do not total 100 because women could check more than one)
Help getting safe and adequate housing
52%
Information about the legal system process
38%
Information about my legal rights and options
29%
Help with a protective order
26%
Help with police issues
21%
Help arranging transportation to meet my needs
21%
Someone to go with me to court
19%
Help with government benefits (e.g. welfare/TANF)
19%
Help getting access to mental health services
19%
Help supporting the court case against the person who
abused me
17%
Access to an attorney
17%
Help with budgeting
14%
Help keeping custody of my children
14%
Help getting child support
14%
Help getting access to child care
14%
Help getting a job
12%
Help getting access to health care
12%
Help with health insurance for my children
10%
Help understanding my rights and options related to
residency
10%
Help with probation issues
7%
Help preparing to testify in court
7%
Help meeting my needs related to my disability
7%
Help with safe visitation for my children
7%
Help with child protection hearings or requirements
7%
Help with my children’s school (e.g. records, changing
schools)
7%
Help getting medical benefits (e.g. Medicaid)
7%
Help dealing with my arrest
5%
Help getting job-related training
5%
Help meeting my child’s disability-related needs
5%
Help getting residency status
5%
Help getting benefits as an immigrant
2%
Help getting access to substance abuse services
2%
This information can be quite helpful for program planning and for fund-raising.
The fact that the most common help women reported needing was affordable housing
might surprise some community members and funders, and could positively influence a
program’s application for more money to devote to this. Also, 7% of the women wanted
help with their own disability and 5% needed help for their child’s disability. While not
FVPSA Outcomes Evaluation: A Practical Guide page 36
large percentages, this could still represent a significant number of families in need of these
specialized services.
Outcome information for support services was quite interesting as well, as shown
in the following table. Notice that the numbers are generally lower than what was
reported by shelter residents. This could be a function of fewer contacts with advocates,
but deserves additional attention.
Table 6. Because of the Services I received, I feel…
More comfortable asking for help
67%
I have more resources to call upon
66%
I know more about my options
66%
More hopeful about my future
64%
Better prepared to keep myself and my children safe
59%
Like I can do more things on my own
57%
Results of the DOW Pilot of Support Group Forms
The support group and counseling form was piloted by programs in four states
for about two months. The responses here are based on 160 women:
• 56% participated in support groups only
• 19% participated in counseling only
• 21% participated in both counseling and support groups
As can be seen in the following table, women sought out these supportive
services for different reasons. Some of the most common reasons were:
Table 7. What Women Wanted at the Beginning of Support Groups
Talk to people who understand my experience
78%
Feel better about myself
77%
Learn more about domestic violence
74%
Support to make some changes in my life
70%
Understand myself better
69%
Hear what other women have done
66%
Feel more hopeful about my life
64%
Help with issues related to my children
54%
FVPSA Outcomes Evaluation: A Practical Guide page 37
Once again women also completed a section of the survey that asked about the
effects of the services on the women’s lives. Findings were fairly similar to what was
seen from women receiving advocacy services:
Table 8. Because of the services I received, I feel…
Better about myself
77%
More comfortable asking for help
67%
I have more resources to call upon
66%
I know more about my options
66%
More hopeful about the future
64%
Like I can do more things on my own
57%
In the pilot study, women’s overall rating of support groups was quite high.
Specifically, their ratings were:
Very helpful – 80%
Helpful – 16%
A little helpful – 3%
Not at all helpful – 1%
The Documenting Our Work pilot study found that these surveys were easy for
most women to complete, easy for staff to administer, and resulted in a wealth of
information of interest to program staff. However, in the pilot study, programs sent all
of their forms to Eleanor Lyon to analyze and report on. Some programs have noted
that, while they would like to use these forms on a regular basis, they fear they lack the
resources to handle all of the data themselves. As a result of this, the FVPSA national
advisory board discussed blending the DOW project with the FVPSA outcome pilot
project. That pilot project is discussed in the next chapter.
FVPSA Outcomes Evaluation: A Practical Guide page 38
Chapter Seven
The FVPSA Outcomes Pilot Project
In the fall of 2006, four states agreed to participate in the FVPSA Outcomes
Pilot Project. The state domestic violence coalitions and FVPSA state administrators of
Missouri, Nebraska, Pennsylvania and Wisconsin agreed to work with Eleanor Lyon
and Cris Sullivan to provide training and technical assistance to their local programs as
they phased in the new outcome requirements. All local programs were encouraged
(but not mandated) to use the full DOW forms in order for us to examine, over time,
how they worked when program staff not only collected the data, but analyzed it as
well.
In May of 2007 we asked the four pilot state coalitions to mail a brief survey to
their local programs, which were then sent anonymously to the evaluators for analysis. The
brief surveys asked program staff to indicate, for each type of service they offered:
• Whether the program was using the full DOW forms, shortened versions, or
different forms;
• How easy the forms were for staff to use;
• How clients felt about completing the forms;
• What percentage of clients they were receiving forms from;
• How long data entry was taking;
• How the database was working for them; and
• Whether they had used any of the findings yet.
Missouri conducted the survey through Survey Monkey, and Pennsylvania had just
informally evaluated the process so did not use the forms (but their open-ended responses
are included in this chapter). We received 16 responses from Missouri, 13 from Nebraska,
and 20 from Wisconsin (TOTAL = 49, plus open-ended comments from 28 Pennsylvania
programs).
As can be seen in the table below, most of the programs in Missouri, Nebraska and
Wisconsin were using all but the counseling forms, with few or no modifications. (This
information was not available from Pennsylvania)
FVPSA Outcomes Evaluation: A Practical Guide page 39
FORM
Yes,
as is
Yes, slightly
modified
Yes, but
significantly
modified
No, we
use our
own form
No,
we don’t
measure
Shelter # 1
69%
10%
2%
6%
Shelter # 2
67%
10%
2%
4%
Support &
Advocacy
73%
10%
2%
2%
4%
Support Groups
67%
12%
4%
2%
Counseling
31%
2%
2%
35%
Staff perceptions of clients’ feelings about completing the forms varied quite a bit.
Slightly over a third (38%) said clients’ reactions were “mostly positive,” while the same
number (38%) said clients’ reactions were “neither positive nor negative.” Ten percent
thought clients felt “mostly negative” and 15% did not know how clients felt about
completing the forms. Positive comments included such statements as:
“For the most part, the women have been very understanding when it is
explained to them that these surveys will only take a few minutes of their
time and will help assist us in assisting them with areas that they need the
most help. They also like the fact that these forms are confidential and no
identifying information is on them.”
“We have been surprised by how many persons are completing the surveys.
Especially surprising is how many are willing to mail them in. The clients
have told the ED that they are appreciative that their opinions matter so
much to us.”
On the other hand, some programs noted a more negative response. These
comments generally focused on clients either finding the forms too cumbersome, or on not
receiving the surveys back.
“It seems to be just one more form in the check-in and check-out
paperwork for them.”
“Comments from women completing the forms - overwhelmed, that we are
not truly interested in their needs, only interested in filling out paper work;
the forms are confusing and lengthy, to date - most of the forms that were
given to women to fill out, have not been returned to us.”
“Some of the comments that we have gotten from women completing the
forms are that they are too long/cumbersome. And women have also stated
that for the questions where you circle a response from 0 to 4, it seems like
the scoring is backwards. In the forms, 1 is strongly agree and 4 is strongly
disagree and generally when you are filling out surveys the higher numbers
tend to be positive and the lower numbers are more negative.”
FVPSA Outcomes Evaluation: A Practical Guide page 40
A number of programs mentioned the issue of response categories being confusing,
so forms were modified in May of 2007 to rectify this. Some of the other comments
indicate that more training may be needed in helping staff determine when and how often
to ask clients to complete surveys, as well as how to improve their return rate.
Staff were also asked how clients responded to the surveys, including being asked
to participate in them. The majority (62%) of respondents said that clients were “overall
positive,” while 38% said clients were “very mixed, with quite a few clients having
concerns or hesitations.” No one endorsed that clients were overall negative.
When asked how smoothly data collection was going for staff, slightly over half
(54%) of the respondents endorsed that “there’s a little confusion but we’re working it out
pretty well.” Slightly fewer (44%) said that “staff understand the system and it’s going
smoothly.” Only 2% indicated that “there’s a great deal of confusion.” Positive comments
generally focused on staff appreciating hearing directly from clients:
“Staff likes the forms, they are not intrusive and will be an asset to the
program for our strengths and weaknesses and program expansion.”
“The Outcomes surveys are to be utilized on a continuous basis and while
this has taken some time for staff to adjust to, they appreciate the ongoing
feedback from clients.”
Negative comments generally focused on time concerns, as well as lack of
understanding regarding when and how often to collect the information.
“They are not as confused as they are feeling like it is more paperwork that
they are having a hard time finding time for.”
“It's hard to find time to fill out the surveys and staff gets frustrated on finding the
time.”
Respondents were then asked what percentage of clients they were receiving forms
from. Responses varied considerably across programs, as can be seen in the table below.
Clients
0-10%
11-25%
26-50%
51-75%
76-90%
91-100%
Shelter
10%
5%
12%
29%
32%
12%
Support services
17%
22%
22%
17%
10%
12%
Support groups
14%
8%
18%
21%
21%
18%
Counseling
48%
12%
24%
8%
8%
0%
FVPSA Outcomes Evaluation: A Practical Guide page 41
Staff then indicated how data entry had been going for them. When asked how
simple the spreadsheets were to enter data into, respondents said:
41% very simple
35% a little complicated at first, but now it’s simple
16% still a little complicated
8% too complicated
The open-ended responses varied from extremely positive to extremely negative.
Some program staff found the entire process quite simple, as seen below:
“Easy format and self-explanatory.”
“I have had no problems opening the databases and entering data. I love
the database. If I could change one thing, though, it would be the way the
cursor moves. For example, for the large sections that require a numerical
response, data entry would be faster and easier if the cursor moved down
rather than across.”
“Have had no difficulty in opening database. Entered data for 1 form as a
test. Will be entering the remainder forms in this coming week. I thought it
would be difficult or cumbersome; it is actually quite easy.... THANK
YOU!”
Others either could not get the database to open, could not figure out how to use it,
or worried about the time involved in using it.
“We haven't even begun this process. Our advocacy coordinator basically
said ‘You have got to be kidding me!?’”
“Unfortunately, this task has proven difficult for our program. We have
attempted to download the database program numerous times and have
had some problems opening, entering, and retaining data.”
“We have not tried to open or enter data yet. We are having trouble finding
a staff person who has the time to work on this.”
These concerns suggest that more training may be needed around the database.
Another complaint from a number of programs was that the database was not in Excel. The
Excel database was completed and provided to programs in June of 2007.
The next questions in the survey focused on how long it took to enter forms into the
database. Only those who had entered data answered this question, and the number of
respondents is indicated next to each form:
FVPSA Outcomes Evaluation: A Practical Guide page 42
5 minutes or less
6-10 minutes
11-15 minutes
15+ minutes
Shelter #2 (n=25)
76%
20%
4%
0
Support Services (n=27)
67%
26%
4%
4%
Support Group (n=24)
88%
8%
4%
0
Counseling (n=8)
88%
0
12%
0
In response to being asked if they had been able to use the data they had collected
so far, 42% responded yes. Open-ended responses included:
“I have used it in my monthly report to the agency's board of directors.
Also, I plan to incorporate the information in my report to our local county
agency when applying for continued funding in August.”
“We have changed the way we handle certain issues and have tried to
make ourselves more open to changes.”
“Yes, we made a change in how our office assistant handles phone calls
that will be more helpful to clients. We are anxious to get more data to be
able to see what other changes are needed.”
“Yes, use it to let staff know any areas of weakness or strengths in our
services.”
All of the verbatim responses to all open-ended questions in the survey can be
found in Appendix B.
Conclusions
Overall the pilot went quite well, and we received extremely helpful feedback that
was used to modify the forms, database, and manual. The majority of clients appear willing
to complete the surveys, some enthusiastically. Quite a few staff are also finding the
surveys to be helpful to their programs in a variety of ways, and the process of collecting
the data seems to be getting more routinized. At the same time, and not surprisingly, some
problems and concerns were raised. A number of the problems could be addressed with
additional training and technical assistance. For example, some staff were asking women
in crisis to complete the surveys, and/or were surveying clients too frequently. Other
programs were having a hard time getting surveys returned, and a number of programs
were having problems with the database. Other problems, however, are more related to
program capacity and accessibility of the surveys. Some programs felt that they don’t have
the staff time to fully participate in this level of data collection, or have computer issues
FVPSA Outcomes Evaluation: A Practical Guide page 43
that make using the database difficult. A number of staff thought the surveys should be
shorter, and available in multiple languages (which is in process). Based on this feedback,
we recommend that programs view the forms in the Appendix as a “menu” of items from
which to choose. Programs wanting to use the entire forms are welcome to do so, while
others might want to considerably shorten them or incorporate some questions into surveys
they are already successfully using. As long as the two outcomes are being measured, all
other survey items are completely voluntary.
SECTION II
PRACTICAL ISSUES
FVPSA Outcomes Evaluation: A Practical Guide page 44
Chapter Eight
Deciding How Much Information
To Gather, and When
Ideally, you would want to hear from all of your agency’s clients how your work has
affected them. You want to be accountable to all of the survivors you serve, and you want to
provide all of them with an opportunity to provide safe feedback about their experiences with
your program. You also want to have the most complete and comprehensive information
about the work you do and its impact on survivors so you can share it with funders and the
public, and also use it within your program to feel good about your accomplishments and
make improvements.
However, time and resource constraints may mean that the ideal cannot be reasonably
managed by all programs. If this is the case, you can consider collecting information from a
sample of survivors who receive each type of service. Sampling is an accepted way of
collecting information from a part of a group to represent the views or experiences of the
group as a whole. It is used all the time to gather information about the American public (and
in other countries, as well, of course). Public opinion polls may question only 1,000 people,
but represent all adults fairly, if they are done carefully. Even the Census is a sample, although
strenuous efforts are made to include everyone. Samples of survivors in your program, then,
can be used to collect information for the two FVPSA outcomes (and any other outcomes you
choose to measure, as well).
General Guidelines for Using Samples
The key to collecting information from a sample of survivors is that you must take steps to
make sure that the people you include are as much like (“representative of”) the whole group
of people who receive your services as possible. This means that survivors from all ages, races
and cultural groups, sexual orientations, religious preferences, and abilities must be included.
It also means that survivors who complain must be included along with those who continually
FVPSA Outcomes Evaluation: A Practical Guide page 45
comment that your program is wonderful. Women who have limited contact or stay only
briefly in shelter should be included, along with those who are involved for the maximum time
(if you have one). You cannot select particular survivors based on one of these
characteristics, and exclude others! That would “bias” your sample.
Expensive research and professional opinion polls commonly obtain representative
samples by selecting participants at random. Essentially, this means that everyone on a list of
the population has an equal chance of being selected to be in the sample. Service programs
(which don’t have a list of everyone they will see) sometimes accomplish the same thing by
selecting every other (or every third, or every tenth, etc.) client. This would be very difficult for
most domestic violence programs and services to do and to monitor accurately—it would be
hard to keep track of who was given a form, and when, and to know when the next one was
“due.”
A reasonable alternative approach to sampling for most programs would be to select
one or more times (depending on the type of service and what works best for you—see the
next section for special considerations for each type of service) during each year when you
will use the forms with survivors. Here are some considerations:
Representative/Typical: The time you select should be a “typical” time period, and
one when it would also be easy for you to focus on using the form (perhaps not, for
example, during October or a major holiday period). You know your
program and the survivors you serve, and the normal fluctuations you experience. If,
for example, you have periods of time that are always especially busy or especially
slow, you may want to avoid those times because they are not representative of your
typical client-flow.
Sample Size: The number of survivors you collect information from is not fixed. It
will depend on how big your program is—the number of survivors you typically
provide specific services to in a given year. The idea is that you need to get
information from enough of them that you can say that what you have is a fair and
reasonable reflection of the experience of the whole group. If you have a small shelter
and typically have only ten residents in the course of a year, you should try to get
information from all of them, and it shouldn’t be too burdensome. If you serve
FVPSA Outcomes Evaluation: A Practical Guide page 46
hundreds every year, then collecting information from twenty or twenty-five percent
may be enough, as long as the selection process is consistent and unbiased.
The length of time you select to collect the information will be determined by
the number you decide is your goal for the sample. In general, the larger the number
of women you serve, the smaller the percentage you will need, as long as the time
period is fairly typical and the selection process is consistent and unbiased. Again, for
example, if you have 1000 advocacy clients, sampling 10% or 15% may be enough. If
you have 50 clients, sampling half of them would be better.
Protecting Anonymity: This is so important that it bears repeating. The sample
should be large enough that individual identities and responses cannot be determined.
The time period selected (as well as the process, discussed in Chapter 9) should last
long enough so that the survivors who participate can (and feel that they can) do so
safely and without being identified.
The general considerations just discussed apply to sampling for all types of services.
However, each type of service has its own considerations, as well. The “special
considerations” are covered briefly in the next sections.
Special Considerations for Shelter Samples
If you use brief client feedback surveys, the approach for most shelter samples that is
simplest and easiest to manage will be to collect information from all residents all year. As
residents seem close to leaving shelter, they can be approached to complete a survey. We do
not recommend waiting until “exit” to ask women to complete surveys. Too often we don’t
know when someone will be leaving shelter, and women have a lot of other things on their
minds when they are moving out.
FVPSA Outcomes Evaluation: A Practical Guide page 47
Special Considerations for Support Group Samples
Open support groups: Clearly, using forms during each session would be too much!
Instead, you can pick a regular interval, such as once a month, and ask all group members
who are present to fill out a form at that time. The DOW forms ask for information on the
number of groups the person has attended, and whether or not they have completed a form
before. In Chapter 11 we discuss how to take this information into account when reporting
your findings. For instance, although some participants will complete a form several times,
analyses can focus on responses at a particular time point and avoid most duplication. For
example, you could report that after attending 4 sessions, more than 85% of women in
your groups knew more about community resources.
Closed support groups: Sampling is much easier with closed groups, because they have
a clear beginning and end. If you hold several general closed support groups in the course
of a year, you may not need to collect information from participants in all of them. If you
hold one or two each year, and have some that are specialized, you should include all of
them.
Even with closed support groups, not all women will attend all meetings or stay until
the last one. To guard against missing survivors who leave before the final meeting, you
can ask group members to complete a form about mid-way through: after the 5
th
session in
a 10-week group, for example. Then you could ask them to complete the form after the
ninth meeting as well.
Individual counseling: Some individual counseling will last only one or two sessions,
and some will last much longer. You will know what is most common for your program.
If most women come for two counseling sessions, for example, all women (during the
sampling period) should be asked to fill out a form after their second session. They would
then be asked to fill it out again after the session the counselor and the survivor agree is the
last one.
FVPSA Outcomes Evaluation: A Practical Guide page 48
Special Considerations for Support Services and Advocacy
Samples
For support services, the primary consideration is timeframe, since much advocacy
involves fewer contacts than groups or counseling. You could sample for a month or a few
weeks at a time at different times in the year, as long as you select representative times, and
think this through in advance. The timing and rationale (e.g. doing it quarterly, so that you can
account for any seasonal variations) should be determined in advance. Again, you should not
select a time because of particular characteristics of clients that you observe (e.g. “we really
got a lot of women after the Latina Health Fair, let’s do it now!”).
Specific recommendations for sampling across the different types of service areas are
provided in the following chapter.
FVPSA Outcomes Evaluation: A Practical Guide page 49
Chapter Nine
Collecting the Information (Data)
There are a number of things to consider before you actually start collecting data from
clients. First you’ll want to decide who on your staff should be involved in deciding which
forms to use and with whom. You’ll also want to develop a realistic timeline for phasing the
outcome evaluation in to program activities. For example, you might just start using the
shelter forms, and then after a period of time start implementing the support group forms,
followed by the support services forms. On the other hand, if you have a large staff doing
quite separate work, you might start using all of the forms at the same time.
Once you decide to start collecting data, be sure that everyone who could possibly
work with clients is aware of this project and how to use the forms. You might introduce this
project during a staff meeting, where people can ask questions. Stress to staff the importance
of asking every woman to complete them at the times you have agreed.
Using the data collection forms will be easier for staff to remember to do if you
incorporate their use into the way you normally organize your work. For instance, in shelter
if you have a “packet” of forms that you use at intake, include the data collection form here
as well. For all programs, make sure forms are available and visible so that staff remember to
use them.
Designing a Protocol for Getting Completed Forms
Back from Survivors
It is important to think about ways to get forms back from survivors in a way that
protects their anonymity. Different programs will make different decisions about this based
on size of your organization, number of staff, types of services offered, etc., but we offered a
number of guidelines here to help you make the best choice possible.
First, regardless of the service offered, survivors should be confident that you can not
trace their comments directly back to them. Some women will not want to give negative
feedback to the person who just provided them with services, either because they do not
want to hurt the staff member’s feelings or because they might think staff will hold their
FVPSA Outcomes Evaluation: A Practical Guide page 50
comments against them. Therefore, some time and effort needs to go into reassuring clients
that steps have been taken to ensure their comments are completely anonymous.
Any staff member who will be involved in collecting surveys from survivors should be
familiar with the following protocol:
1. The staff member who asks the survivor to complete the form should ideally NOT be
the person who has just delivered the service (the advocate, group facilitator,
counselor, etc.). For small programs where this is not possible, be sure to follow the
next guidelines even more carefully.
2. Stress the following things to the survivor when asking her to complete a survey:
a. Explain that you understand she’s busy and that you really appreciate her
taking the time to complete a survey.
b. Explain that your program takes survey results seriously and makes changes
to services based on feedback received.
c. Stress that the survey will only take a few minutes to complete.
d. Stress that while you really would appreciate her feedback, completing the
survey is completely voluntary on her part.
e. Explain that it’s very important staff do not know who completed what
survey and that a number of procedures are in place to make sure staff don’t
know who said what. Explain those procedures.
3. Make sure clients receive either a pencil, or black or blue pen to complete the
survey.
4. Survivors need a private space to complete the survey uninterrupted.
5. Identify a visible, convenient, and secure place for the completed forms to
be returned. You may want to ask survivors what would help them feel most
comfortable and trusting: the type of container (a covered box? something with a
lock?) and its location. For small programs, with few clients, it is especially important
to explain to clients that the box is only opened every month or every quarter
(depending on number of clients) to ensure anonymity of clients.
FVPSA Outcomes Evaluation: A Practical Guide page 51
We have summarized this information into a one-page handout you can copy and
share with all staff. It is on the next page and is also available on the FVPSA Outcomes Project
website.
FVPSA Outcomes Evaluation: A Practical Guide page 52
CREATING A PLAN WITH STAFF
FOR COLLECTING OUTCOME EVALUATION DATA
1. Meet with key staff to explain the need for the evaluation and how it can be useful to the
organization.
2. Decide with staff who will collect the data, how often, and from whom
3. The importance of sampling clients
a. Do not collect data when clients are in crisis
b. Collect data often enough that you don’t miss those clients who receive
short-term services, BUT not so often it’s a burden to clients
c. Sampling shelter residents:
• Ideally, try to ask every shelter resident to participate as they get
closer to shelter exit (other than those in crisis).
d. Sampling support group participants:
• Ideally, every 3-4 weeks pass out forms to all group members at the end of a
meeting, and invite them to stay an extra 5 minutes to complete the form. Pens
or pencils should be provided, a locked box or sealed envelope should be
provided, and the facilitator should leave the room.
e. Sampling advocacy program participants:
• Ideally, after 2 contacts with the advocate unless the advocate believes they’ll
see the client again. You want to allow enough time for change to occur, but
not miss those clients receiving short-term advocacy.
4. The key to sampling is that you must make sure that the people you include are as much
like (“representative of”) the whole group of people who receive your services as possible.
a. Survivors from all ages, races and cultural groups, sexual orientations,
religious preferences, and abilities must be included.
b. Dissatisfied as well as satisfied clients need to be included.
5. Copy enough blank forms that they are readily available to staff; they should be in a visible
area that will remind staff to use them.
6. Design a way that clients can return completed forms in an anonymous way. You can
make or buy a locked box with a hole in the top, or can provide envelopes that clients can
seal themselves and place in a safe place. Consider:
a. Clients need to feel that no one will look at their form in the near future.
b. Clients need to feel that they will not be identified by their survey.
c. Before you begin, you could ask some clients what place or approach would feel
best to them.
d. You might need to figure this out through trial and error.
7. Decide with staff how often to discuss how the data collection is going; this should be
quite often in the beginning while staff is getting used to the new procedures and to decide
together what strategy works well and what doesn’t.
8. All staff who might invite clients to participate in completing a survey should have a copy
of the “Directions for inviting clients to participate in outcome evaluation.”
FVPSA Outcomes Evaluation: A Practical Guide page 53
Collecting Information from Survivors Participating in
Various Programs
The following sections provide specific suggestions for collecting information
from clients of the following programs: (1) shelter, (2) support services & advocacy, (3)
support groups and group counseling, and (4) individual counseling.
Collecting Information from Women in Shelter
It is important to discuss with staff when it is appropriate to have women in
shelter give feedback on their stays. This decision is not as simple as it might seem.
There are different considerations for each form.
We have included a survey you can ask women to complete shortly after they
arrive in shelter (“shelter resident survey #1”). This form does NOT contain outcome
information and is not needed for the FVPSA outcome evaluation project. However,
many local programs have found that it provides them with useful information about
women’s initial feelings, experiences, and needs.
Among the issues for this first form (“shelter resident survey #1”) are:
o Should this form be included with the other information collected during your
“intake” process? We recommend that you give it to them at that time, and
explain it, so that it is understood as a regular part of what you do. It is part of
your commitment to them to listen and respond to them as individuals, as
completely as you can.
o This form should be filled out as soon as possible after the women arrive in
shelter. However, if they are very upset and in crisis, you should wait. You
should agree on how you will handle this situation, so that you are sensitive to
each individual’s particular circumstances, and still get completed forms from
nearly all of those entering shelter. Remember that this information will help
you to improve your services, so you don’t want to neglect including the
women who need them most.
o You should decide what you want to do if you forget to give a woman this form
within the first few days she is in the shelter. She will already have been
affected by her shelter experience, and her understanding of her needs and
FVPSA Outcomes Evaluation: A Practical Guide page 54
goals may have changed. You can decide, at this point, to ask her to complete
the first form anyway, so you can obtain the information on the first page, or
you can ask her to complete the second form, so you are certain to have
information about her experience in the shelter. Whatever choice you make,
you need to apply it consistently, and remember it when you look at your
results. After more than a week or so has passed, starting with the first form
does not make sense, unless most women who come to your shelter stay
several months or more.
Shelter Resident Survey #2 was designed to gather outcome data as well as other
contextual and process information. Should you use this or a different client feedback survey,
you would want to invite residents to participate shortly before they leave the shelter. While
we don’t always know when someone will leave the shelter, we often have a good idea that
the time is approaching. Women find housing, start packing, discuss upcoming plans, etc., and
this is a good time to invite them to complete a quick survey about their experience.
Collecting Information from Support Services Clients
Support services can be broadly understood as helping survivors access services or
resources in a supportive manner. These services are non-residential, and are offered either in-
person or over the phone. Since programs differ across the country in what they offer and how
they offer it, every agency must decide for itself how best to collect outcome information from
clients receiving support services.
Discuss with support service staff when it is appropriate to have women give feedback
about the services they have received. Ideally, women will complete the form right before they
stop services, so their responses will reflect their full experience with your program. However,
women commonly stop coming for services without saying anything in advance—they simply
stop. Other women have only a brief, one-time interaction with your program staff. For this
reason, we suggest that program staff and volunteers thoroughly discuss how and when you
want to use this form. You might reasonably decide that women who receive extremely short-
term advocacy (for example, a five-minute meeting in court where women’s rights and options
are explained) do not need to complete this form, since such brief contact is unlikely to cause
major changes. Our recommendation is to ask a survivor to complete a brief survey after a
minimum of two contacts with the advocate unless the advocate believes they will see the
client again. You want to allow enough time for change to occur, but not miss those clients
receiving shorter-term support and advocacy.
FVPSA Outcomes Evaluation: A Practical Guide page 55
Collecting Information from Women Using Support Groups or
Group Counseling
Nearly all domestic violence programs offer support groups for survivors of domestic
violence. Support groups are organized in different ways: some programs offer separate groups
for women in shelter and women in the community, while others offer one general group
regardless of residence; some programs offer groups that are open-ended, while others offer
groups for specific periods of time; some groups follow a pre-determined sequence of topics,
while others focus on immediate issues raised by group members; some programs offer general
groups, while others have specialized groups that focus on particular topics or populations;
some programs offer groups for children as well as adults. Some programs offer group
counseling, rather than (or in addition to) support groups, but data collection for either type of
activity is quite similar so they are grouped together here for that reason.
Support groups and group counseling are offered to assist women as they consider their
options and choices, and try to make sense of the abuse they have experienced. Some women
are more comfortable talking in groups with other women who have had similar experiences;
they may feel that these are the only people who can genuinely understand their situation, and
can offer realistic suggestions and examples. Others prefer an opportunity to talk in private
and have attention focused only on their concerns. Both approaches hope to promote healing
and enhance women’s understanding of the choices they have.
The Support Group Feedback Form, which can be found in the appendix, can be
modified to use with group counseling programs as well. It was designed to be anonymous
and confidential, and can help your program:
• learn more about why survivors sought out these particular services;
• document the extent to which clients felt they got what they wanted;
• learn which survivors feel most supported and respected; and
• document short-term outcomes.
For obvious reasons it is much easier to collect information from survivors participating
in “closed” groups. Many agencies simply ask the women to stay an additional five minutes to
fill out the brief survey after every 4-6 sessions. Surveys are then placed in locked boxes or
envelopes that can be sealed, and facilitators often assure participants that they only look at
surveys every 3-4 months in order to ensure client anonymity.
FVPSA Outcomes Evaluation: A Practical Guide page 56
For “open” groups, when participants change from week to week, you can still ask
people to complete surveys every 4-6 sessions but you will be including women who have
been in group once with women who have been in group for many weeks. It is important,
therefore, to make sure a question on the survey asks women how many groups they have
attended (as the form in the appendix does) so you can decide if enough time has passed for a
desired outcome to be achieved.
The more frequently you ask women to fill out the forms, the more complete the
information you will get. However, you do not want to burden the women with too many
forms, and you want them to be honest and thoughtful when they fill out the information, so
they should not have to fill them out too many times. You should decide on the intervals that
make most sense for your program, based partly on how long most women continue with
counseling or support groups.
Collecting Information from Survivors Participating in
Individual Counseling
You will notice that the form created to evaluate individual counseling is very similar
to the Support Group Feedback Form. However, the procedure for collecting the information
would differ somewhat, since extra care must be taken to ensure that responses are
anonymous. As is true for all forms, a counselor should never ask a survivor to fill out a form
and hand it back directly. Instead, a place in the program that is accessible and secure should
be identified for returning forms. It could be the same place as other forms are returned, since
the survivor would check off the type of service she is describing. The counselor should
explain to the survivor that other people will record the information, and that forms are only
retrieved from designated spot every few weeks, after many forms have been left, so that
individuals cannot be identified.
Alternative Ways to Collect the Information
Nonprofit organizations commonly use brief, written client feedback surveys to collect
outcome information because they are relatively simple for both staff and clients. However,
relying solely on such surveys, especially if they are only offered in English, means that
FVPSA Outcomes Evaluation: A Practical Guide page 57
programs will not be hearing from all of their clients equally. If someone either does not read
English well or has a physical disability preventing them from comfortably completing the
form, their opinions and experiences will not get counted. We therefore recommend creatively
designing ways to obtain this information in multiple ways. Discussing strategies with staff
and an advisory group of survivors will maximize the likelihood of creating the best options.
What Languages Are Needed?
Individual programs will need forms translated into a variety of languages, depending
on their geographic and social location. The forms provided in this manual and on the FVPSA
Outcomes Project website have been translated into Spanish as of this writing, with more
languages being added over time. Please check in with the website periodically for updates:
http://pubs.pcadv.net/FVPSA_Outcome/
user name and password are both: outcomes
For other languages you may need to contact local resources for assistance. You may
not have the resources to translate the forms into all languages spoken by your clients but if
you can translate the most common languages used, that’s a great start!
Verbally Asking the Survey Questions
Verbally asking clients the survey questions can be helpful when dealing with literacy,
language and/or many disability issues. However, as mentioned many times previously in this
manual, you do not want the person who provided the services to be the person asking the
questions because clients may not feel comfortable giving negative feedback. However, there
are ways that programs have gotten around this. Some use other staff members who have had
no contact with the survivor complete the forms with them. Other programs use interns or
volunteers to help with this; still others have used local translation services to ask the questions
by telephone. These are individual decisions that need to be made by each program based on
need and resources available.
FVPSA Outcomes Evaluation: A Practical Guide page 58
Chapter Ten
Maintaining and Analyzing the Data
A critical component of evaluation is to correctly interpret our findings. Although it is
not true that "you can make data say anything you want," as some critics of evaluation would
suggest, data are open to interpretation. This chapter presents some basics for analyzing and
interpreting findings, as well as some common mistakes to be avoided.
Storing the Data
The first question, before deciding how to analyze your data, is: how and where will
you store your data? We strongly recommend investing in some type of computerized
database, or computer program designed for storing and organizing data. This does not have
to be anything extremely elaborate that only a computer whiz can understand -- as a matter of
fact, that is exactly the kind of database you don't want -- but it should be capable of
organizing your data for you in a simple, manageable way. Most programs have copies of
common spreadsheet programs, such as Excel and Lotus, that they use for budgeting and other
purposes, and most also have Microsoft Access. These programs can also be used to enter and
analyze data. Access and Excel databases have been created for FVPSA grantees, along with
instructions for using them (see Appendix D).
NOTE: Regardless of whether you will be entering the data into a computerized
database, or calculating your findings by hand, determine how and where
you will store your data to maximize confidentiality of participants and to
minimize the opportunity for someone to mistakenly delete or misplace
your files.
FVPSA Outcomes Evaluation: A Practical Guide page 59
Some Data Entry Considerations
Every program must decide for itself who will “enter” the information into a
database, and how and when that will be done. It is ideal if a small number of people are
identified to do this, and on a regular schedule—e.g. weekly or biweekly. That way, the data
are always up-to-date, and the task consumes little time. If one person is identified to perform
data entry, at least one alternate person should also be selected and knowledgeable, to avoid
problems from sickness, vacations, turnover, or other unforeseen circumstances. The process
will work most smoothly if the data file is already set up and ready to be used at the time you
decide to begin asking women to fill out the forms. Again, you have been provided with an
empty Access file for the four DOW forms described in this manual, that you are encouraged
(but not required) to use as part of this pilot.
How to Analyze the Information You Collect
You will have two types of information when it is all entered: “quantitative” data that
can easily be shown numerically—as counts or frequencies, or as percentages—and
“qualitative” data that are in the form of words. The hints that follow show different types of
analysis and provide examples from the pilot test of the DOW forms.
Quantitative Information
The most common types of quantitative analysis you are likely to use are frequencies
and crosstabs.
1
Frequencies tell you how many of something you had. They are sometimes
called “counts,” and are often shown as percentages. Crosstabs allow you to find out the
relationship between two items that have frequencies in groups or categories (such as gender
or age groups). These two types of analysis are described separately.
1
As part of the Access data files you are provided with the four DOW forms, all of the frequencies have been
pre-programmed for you, so the primary analyses will be simple (and explicit instructions for using the data file
and creating your own analyses, including crosstabs, are provided in Appendix D).
FVPSA Outcomes Evaluation: A Practical Guide page 60
Frequencies can be invaluable, even if they are simple. For example, in the pilot test of
shelter forms, we had the results shown on Table 9 for the first question. The table shows that
domestic violence advocates were a source of information about the shelter at over twice the
rate of any other source for the programs involved in this test. Family members and the police
were the next two most common sources. One way a program might use this information is to
see if the percentages of survivors who heard about the shelter from family members increase
after a community education campaign, or if the percentages who heard about it from police
increase after a police training.
Table 9. Using a Frequency Table to Illustrate Findings From the Question: Where
Have You Heard About This Shelter?
Domestic violence advocate
31%
Family member
15%
Police
15%
Friend
12%
Telephone book
10%
Social services agency staff
10%
Health care provider
5%
People in court
3%
People from my religious/spiritual community
2%
CPS staff
2%
TANF staff
2%
Flyer/brochure/poster
2%
Other source
21%
What other stories can frequencies tell? Frequencies can also be shown after particular
responses have been selected. For example, when you look at frequencies of services women
in your program have received, it is important to know first whether the women wanted those
services. If they didn’t want or need a service, you (and any funders who receive your results)
should not be concerned that they didn’t get it. For that reason, the shelter #2 form asks
residents to select from a list of possible services the ones they wanted and received fully,
those they wanted and received partly, and the ones they wanted but did not receive. They
also have the option of indicating that they didn’t want the particular service.
FVPSA Outcomes Evaluation: A Practical Guide page 61
You can “select” only the forms on which the women indicated they wanted the service, and
then get frequencies that show the extent to which they got it. This can make a big difference.
In the pilot, for example, only half of the women who completed the shelter form said they
received all of the help with TANF (welfare) benefits that they wanted. However, 40% of the
women who completed the form indicated they didn’t want help with TANF benefits. When
they were left out, 83% of the women who wanted help with TANF got all the help they
wanted, and nearly 9% more got some of this help. These results are much more positive
indications that the participating shelters provided a service that the residents wanted.
This same principle applies to decisions about whether or not you should include
missing data when you report results of frequencies. Sometimes people who fill out forms
decide not to answer some of the questions. For example, half of the women might indicate
that they wanted a particular service, a quarter might indicate that they didn’t want it, and the
other quarter might not answer the question at all. In that case, you would have two basic
options:
1. Report the results only for those who answered the question: two-thirds wanted the
service and one-third did not. This would usually be preferable. You should also
note the number or percentage of people who did not answer. If it is large, you
might want to think about why so many people did not answer the question.
2. Report the results with the missing information included as one of the percentages.
Again, this is a less common practice. It might be useful, however, if the
percentage of people who did not answer the question is large, and excluding it
would distort your results. For example, you might want to add a question about
whether or not women want help with substance abuse or child protection issues.
Many women might choose not to answer these questions for a variety of reasons.
If you reported percentages only of women who responded, however, and few
wanted help with these issues, you might reach the wrong impression that these
services were not needed.
In general, the choice you make depends partly on the percentage of missing
responses, partly on the reasons you think they are missing, and partly on how you want to use
the results.
FVPSA Outcomes Evaluation: A Practical Guide page 62
Cross tabs can be very helpful for providing more detailed answers to many questions. For
example, you can compare outcomes for women based on how many contacts they had had
with an advocate. See the following table for an example of how this might look:
Number of Advocacy Contacts
Because of the services I
received, I know more about
community resources that I
might need in the future
One
(n = 10)
Two or More
(n = 18)
Total
(N = 28)
no
80%
28%
46%
yes
20%
72%
54%
Total
100%
100%
100%
These results suggest that women who have more contact with an advocate are much more
likely to feel they have more resources to call upon than women who have had more limited
contact.
The next table reports the same outcome but compares women participating in either
counseling, support groups, or both:
Type of Service Received
Because of the services I
received, I know more about
community resources that I
might need in the future
Counseling
(n = 23)
Support
(n = 34)
Both
(n = 23)
Total
(N=80)
no
44%
32%
13%
30%
yes
56%
68%
87%
70%
Total
100%
100%
100%
100%
These results suggest that women who participated only in support groups were more
likely to report that they felt they had more resources after services than those who took part
only in counseling. Perhaps more important, women who took part in both counseling and
support groups were substantially more likely to feel they had more resources than those who
experienced just one of these services.
FVPSA Outcomes Evaluation: A Practical Guide page 63
You are likely to have many different questions you will want to answer in this way,
and not just about the services women wanted or received. For example, you might also want
to learn:
• Does the number of advocacy contacts affect the likelihood that women will feel they
have been helped in a particular way?
Is there a relationship between a woman’s race and ethnicity, or age, or sexual
orientation, and the extent to which she feels respected, or her feelings about the help
she has received? In the pilot, although 79% of the women said they would “strongly
recommend” the program to a friend who needed it (and all of the rest said they would
“recommend” it) just 67% of the African American women responded in this way. This
result might lead a program to obtain more feedback from women about their
experiences.
Does length of time in the shelter affect the likelihood that women will receive
particular services? For example, some services may require appointments or involve
other issues that take more time. Women who are in shelter for less than a week may
be less likely than others to obtain help with TANF benefits, but just as likely to feel
emotionally supported while they are there.
Does the number of counseling or support group sessions affect the likelihood that
women will feel they have been helped in a particular way?
Qualitative Information
The qualitative information on these forms comes in response to “open-ended”
questions—the ones with space provided to write in the answer, and no choices already
provided—and questions with “other” categories or space for comments. Qualitative
information is commonly used to provide examples or explanations, and can be very useful in
that way.
Qualitative information can also be grouped into meaningful categories, and help you
to understand something new. For example, nearly a quarter of the women in the pilot test of
the shelter forms reported that they had had concerns about contacting the shelter. All of them
described their concerns. Most of their answers fell into one of three categories: 1) shame or
embarrassment about their abuse, 2) safety at the shelter, and 3) fear of the unknown—they
FVPSA Outcomes Evaluation: A Practical Guide page 64
didn’t know what to expect at the shelter. One noted that she had been at a different shelter
that “wasn’t very nice;” she was concerned that this one would not be different. These results
could be incorporated into training for staff and volunteers who spend time with women when
they first arrive at shelter. Such training could remind shelter workers that many women are
nervous when they first arrive, and women’s concerns may center on shelter realities as much
or more than on their own physical safety. The woman’s comment about the other shelter
might also reflect confusion between domestic violence and other shelters, and could alert
program staff that they need to provide clearer descriptions in public presentations of what to
expect in a battered women’s shelter.
FVPSA Outcomes Evaluation: A Practical Guide page 65
Chapter 11
Sending the Findings to Your FVPSA Administrator
Once a year FVPSA-funded programs will submit their outcome information to
their FVPSA state administrators. This will be handled somewhat differently across states so
you will want to check with your state administrator and state coalition about how your state
will handle this. We have developed a very simple form (Annual Report to Send to FVPSA
Administrator, found in Appendix E and on the FVPSA Outcome Project website) that programs
can use to submit their outcome information annually. As you can see on that form, you
simply have to fill in your agency’s name at the top, along with the date and the reporting
period (since this will differ by state). For each service you are collecting outcome data for
(e.g., shelter, support services & advocacy, support groups, counseling) you simply fill in the
number of surveys completed and the number of “yes” responses to the two outcome
questions. If you don’t offer a particular service just write N/A for “not applicable.” So if, for
example, you don’t have a shelter program you would write N/A across that row of
information. If you received 73 feedback forms about your support services, and on those
forms 68 clients checked that they knew more about community resources and 63 checked
that they knew more strategies for enhancing their safety, you would write in:
Support Services and Advocacy
# of surveys completed: 73
# of yes responses to resource outcome: 68
# of yes responses to safety outcome: 63
At the bottom of the form you simply add up all of the columns of information so your
FVPSA administrator has total numbers for each agency. It is important to send in actual
numbers, not percentages, to your FVPSA administrator. They will then tally all of the
numbers across all of the state programs, and send this information on to the Administration
for Children and Families at the U.S. Department of Health and Human Services.
We have been asked by some programs if they can just use the same client feedback
form to evaluate all of their services. The rationale behind this question is that programs would
only need to keep track of one form, and they could use the same database across all services.
While we understand the desire for this simplicity, the problem with doing this is that you
FVPSA Outcomes Evaluation: A Practical Guide page 66
wouldn’t know how to accurately interpret your information. If, for example, 20% of your
clients reported feeling disrespected, you wouldn’t know if this was a problem across all of
your services, or only in shelter, only in support groups, etc. So, while some of your questions
across all forms can certainly be identical, we strongly recommend you differentiate the forms
by service area. It’s also a good idea to include a question on each form indicating how much
of the service the client received (e.g., number of days in shelter, number of support groups
attended), and this question would need to be asked differently by service area.
We hope the form we created for you to send to your state administrator greatly
simplifies the reporting process. More specifics about when the information is due will come
directly from your state.
FVPSA Outcomes Evaluation: A Practical Guide page 67
Chapter Twelve
Making Your Findings Work for You
As discussed in Chapter 1, outcome findings can be used internally to improve your
program and externally to encourage others to support your efforts.
Using Your Findings Internally
If you are not already doing so, we would recommend setting aside specific times to
review the outcome information you've gathered as a staff. This sends a message that these
outcomes are important, and gives you an opportunity to discuss, as a group, what is working
and what needs improvement. It would also be helpful to invite volunteers and service
recipients to share in these discussions and brainstorming sessions. As improvements are
made in response to the data you've gathered, broadcast these changes through posters on
walls, announcements, and word-of-mouth. As staff, volunteers, and service recipients see
that your agency is responsive to feedback, they will be more likely to feel invested in and
respected by your organization.
Many examples have been provided throughout this manual on ways the results from the
surveys provided in the appendix and on the website could be used by a program. Depending
on the particular form, results can be used internally to identify unmet needs, areas of needed
staff training, public or survivor perceptions of the program, and particular issues that can arise
for survivors from different cultures, or different groups defined by age, sexual orientation,
parenting status, or others.
Using Your Findings Externally
It is important to give careful thought to how you want to present outcome findings to the
public and to funders. Some words of advice:
Keep it positive
Keep it simple
FVPSA Outcomes Evaluation: A Practical Guide page 68
Keep it Positive
Just like a glass is half empty when it is also half full, outcome findings can be presented in
both negative and positive lights. So keep it honest, but keep it positive!
First, don't hesitate to let others know about the great work you are doing. Contact media
sources (television, radio, newspapers) when you develop new programs, help pass
legislation, and in the case of outcome evaluation, when you have numbers to back up
your successes.
Keep It Simple
When presenting your findings for public consumption it's very important to keep it
simple. If you are talking to the television or radio media you will be lucky to get 30
seconds of air time, so learn to talk in sound bites. Remember, people are not likely to
remember specific numbers but they are likely to remember phrases like "most of," "the
majority," "all" and "none." So instead of reporting:
"87% of the women using our legal services were able to get their needs
addressed"
you could say:
"the vast majority of the women using our legal services were able to get their
needs addressed"
Another way to keep it simple when presenting your findings is to pick and choose what to
share with others. You will be gathering quite a bit of information about your programs
and you certainly can't present it all. Decide on the top two or three findings that would
be of most interest -- and that would present you in a positive light -- and focus on those.
FVPSA Outcomes Evaluation: A Practical Guide page 69
How to Share the Information with Others
There are a number of different ways to visually present your data to others. You can
create fact sheets and informational brochures that include some of your evaluation
findings, and you can also use line graphs, tables, bar charts, and pie charts to display
your data more graphically.
Consider the data you are presenting as well as the audience when deciding how to
present your findings.
Bar Graphs can be especially useful to illustrate differences between groups.
For example, the following graph was created from the same data shown in Chapter 10. It
shows quite clearly that having more than one advocacy contact is associated with women
feeling they have more resources (represented by “yes” in the graph).
Relationship Between Number of Contacts and Women Feeling They
Have More Resources
0%
20%
40%
60%
80%
one contact 2+ contacts
no
yes
Your choice of presentation will also be based on the type of computer programs you
have available to you, and the amount of time you have to devote to this project. One other
technical point: if you are preparing information for an "overhead" or a powerpoint
presentation make sure your font size is 18 or larger (the larger the better) to maximize
people's ability to read your findings easily.
FVPSA Outcomes Evaluation: A Practical Guide page 70
When Your Findings are "Less than Positive"
So what do you do when your findings are not as positive as you had hoped or if your
findings show your program was not as successful in certain respects as you had expected?
Again the same principles apply: keep it positive and keep it simple. Avoid using negative
words like:
problem mistake error failure
and instead use words like:
obstacle difficulty challenge unexpected complication
Remember, one person's "failure" is another person's "obstacle to be overcome!" If you have
to present negative findings to the public, don't just leave them hanging out there. Discuss
how you addressed the obstacle or how you plan to address it in the future. What valuable
lesson did you learn and how will you incorporate this knowledge into your program in the
future? Presented correctly, even "negative" findings can be used to enhance your image with
the public. They will also add credibility to your more “positive” findings.
Using Your Findings to Support the Continuation of Current
Programs
One of the problems we hear agencies complain of repeatedly regarding funders is that
many funding sources want to give money to "new, innovative" programs instead of to current
day-to-day activities. When this is the case for your organization, you might try using your
outcome data to justify the need for your current operations. Let the funder know how
worthwhile and important your current services are instead of always adding new services that
stretch staff to the breaking point.
Using Your Findings to Justify Creating New Programs
There are of course also situations when you will want to use outcome findings to
request funds for a new program. Say for example that your current "Support and Education
Program for 7-10 Year Olds" has demonstrated some positive results. The majority of the
FVPSA Outcomes Evaluation: A Practical Guide page 71
children who have attended the group have reported that they (1) enjoyed the program, (2)
appreciated having a safe place to discuss their feelings, (3) learned about keeping themselves
safe, and (4) learned that they were not to blame for the violence happening. You could use
these findings to justify the need for creating another similarly structured group for either
adolescents or for pre-schoolers.
You could also use your positive findings to justify expanding a popular program.
Perhaps your current Housing Advocate is doing a terrific job but can not handle the heavy
caseload. Having data that illustrate for the funder (1) how many people currently use your
program, (2) how many are turned away due to lack of personnel, and (3) how effective
service recipients find the program to be can be an effective strategy for securing additional
funds for expansion.
Important Points to Remember:
The safety and well-being of the survivors you serve must always take precedence over
gathering data. Design your questions and procedures accordingly, and include
feedback and input from survivors who use your services.
Always take the time to explain why you are asking for information. If you explain that
their input will be used to improve your services, survivors will usually be happy to
answer some questions. It is disrespectful to introduce questions with only "I need you
to answer some questions" or "I need you to fill this out."
Don't request any more information than is necessary, and be committed to using this
information to understand and improve upon your services.
Consider issues of diversity in designing your outcome evaluation. Such issues include
but are not limited to literacy, language, and culture. Again, including input from the
survivors who use your program is vital.
The tools and strategies developed for this manual may or may not make sense for your
specific program. They were created only to provide a foundation from which to begin
your evaluation efforts. You will need to tailor your strategies to fit your specific
program.
Design outcome questions that will answer whether or not survivors attained outcomes
they identified as important to them.
