Evaluation and Policy Analysis

The simple answer is that it is important to know what works, so that effective efforts can be replicated and unsuccessful efforts discarded or modified. Yet perhaps the more important answer to this question is that it is important for policymakers, leaders, teachers, and stakeholders to "think" like an evaluator, and continually strive to understand if what they are doing is working, and to continually improve - to be "improvement-oriented."

Evaluation findings are often politicized and interpreted in different ways. Even when the results of an evaluation appear to be accurate and point towards a particular action or change in policy, various stakeholders, groups, and organizations often have different perspectives as to the direction that should be taken as a result of specific evaluation findings. While it is always important that educators and policymakers look to pertinent research when making decisions about how to best improve public schools, it is equally, if not more crucial that educators evaluate and think critically about their work on an ongoing basis. 

Public education has a history of adopting new strategies without carefully examining the research for these strategies, or building in mechanisms to evaluate the effectiveness of new strategies. It is when educators fail to actively assess and evaluate the effectiveness of potentially innovative ideas that "innovations" becomes "the latest fad" and are easily dismissed by parents, teachers, educational leaders and policymakers. The spread of innovation has the potential to lead to transformational change, but doing so entails that all educators become conscious evaluators of their own work and practice. 

Public  education is filled with numerous policies and regulations that are  intended, in one way or another, to ensure that students receive an  equitable and high-quality education. We have policies to ensure  fairness and equitable access and opportunity for all students, policies  to regulate the use of federal and state funds, and policies that serve  as incentives and sanctions to motivate districts, schools, teachers,  and students to improve. 

This policy environment,  broadly defined as the myriad of rules, regulations and written  expectations within which educators and students live and make  decisions, is not always as coherent as educators may desire. Rather, the collection of policies and regulations tend to  create what can best be characterized as policy noise,  which serves to hinder educators from working effectively and making decisions that best meet the needs of their organizations, their teachers, and students. Coupled with the reality that individuals are inclined to interpret policy based on their own perspective and mental mindset, it easy to understand how educational leaders often feel constrained and unable to find clarity within the policy noise that  serves as their everyday working environment. 

"Truly  innovative and sustainable change requires a re-examination of how  education policies impact improvement efforts, and renewed efforts to  construct a productive and enabling policy environment conducive to  innovative change." 

What is the focus of our evaluation services?  

Evaluation design and data analysis: We  work with clients to conceptualize and design high quality evaluations  and data analysis strategies that are meaningful, practical, and designed to capture the information needed to address and accomplish  client objectives and goals. 

During this work, we: (1) engage clients (e.g., teams from state departments of education,  districts, schools, and constituents) in an iterative and collaborative  process that provides time for individuals to discuss the purpose, goals, and questions that will provide the  foundation for the evaluation design; and (2) convene a set of facilitated and  semi-structured conversations and meetings are used to collect information and provide opportunities for clients to articulate their  theory of action for the strategy, intervention, or policy being  implemented, and inform the evaluation design that will best assess the impact of their work. 

Formative and impact evaluation: We  design and conduct evaluations of education policies, interventions,  and strategies, with a particular focus on evaluations of district and  school improvement strategies and state-level initiatives. 

Our evaluation services are customized, and typically includes (1) an initial evaluation design session in order to clarify evaluation questions and identify and agree upon data collection  strategies and methodologies, followed by (2) formal evaluation activities (e.g., data  collection, surveys, focus groups and interviews, quantitative and  qualitative data analysis, dissemination) designed to meet client  needs while ensuring the integrity and rigor of evaluation activities  and findings.

All evaluation design and consultation involves work towards the following outcomes:  

• Articulating the purpose, expectations, and intended audience(s) of the evaluation and evaluation findings; 
• Crafting detailed evaluation questions and data elements aligned to each evaluation question; 
• Constructing  a logic model depicting the theory of action and intended outcomes of  the proposed policy/strategy/intervention. 
• Formal  evaluation studies (formative and impact) include detailed formative  reports and recommendations, customized to meet client needs and  objectives. 

What is the focus of our policy analysis work?

We  work with state and district leaders to jointly analyze policy, in some  instances conducting a policy audit, focused on identifying areas of  policy coherence (e.g., situations in which policy works to support  improvement efforts) and areas of policy fragmentation (e.g., situations  in which policies and regulations tend to constrain improvement  efforts). We use our analysis of the policy landscape to design specific  action strategies for how policy might be adapted, modified, or  communicated in ways that will enhance overall policy coherence.  

Policy Analysis and Audit activities occur over a period of four months, and include: (1) the  co-development of an analytic policy framework for understanding the  relationship among policies and regulations, and their intended and real  impact; (2) an historical analysis of policy shifts and trends over time, with respect to the particular focus or need of the client; and (3) a detailed analysis of how local, state and federal policy intersect, as a  means of exploring and illuminating areas of policy coherence and  areas of policy fragmentation.

Policy Implementation activities  involve ongoing work with a state or district policy design team (or an  existing state or district leadership team), focusing on how to  construct, modify, and communicate new or existing policies to build  upon areas of policy coherence (strengths) and minimize areas of policy  fragmentation (for instance, by reducing the number of planning  requirements that schools must meet). 

What are the outcomes of this work?

• The development of a customized analytic policy framework for analyzing the coherence of education policy 
• A written policy analysis containing  detailed audit findings regarding the extent to which existing policy  supports and constrains innovation and sustainable improvement efforts.

Capacity building outcomes include:

• Increased ability of a state or district leadership team to analyze how various aspects of education policy influences improvement efforts.
• Increased ability of a state or district leadership team to modify and communicate policies, including strategies to co-construct policy with key stakeholders, to support the effective implementation of education policy.