Data-Driven Decision Making: What It Takes To Make Decisions

Philip A. Streifer

A year or so ago, U.S. automakers offered zero percent financing, sending tens of thousands of us to the showrooms. Many of us walked out with a new car. The automakers' pitch was not very different than before, yet so many more of us took this bite at the apple. Why?

We were prompted to action by what we perceived as a dramatic change from the norm: zero percent financing! The same is true in education. In his book Making the Grade: Reinventing America's Schools , Tony Wagner argues that we react to data that is persuasive, data that points out dramatic differences in achievement between groups or against some standard, like graduation rates.

Yet, sometimes the most persuasive data are not obvious, but hidden in the myriad information we deal with every day. Like stargazing, we need more sophisticated procedures and powerful instruments to unveil the true beauties and mysteries of what we see only on one level.

The Orion Nebula, barely visible to the naked eye, becomes more and more detailed as we look through binoculars, then through a 10-inch telescope, and so on. It is in this detailed view that we learn what this phenomenon really is: a vast area of dust and matter "lit up" by emerging stars.

Data-driven decision making yields the same phenomena. When we drill down into the detailed data using more sophisticated analyses we often uncover new and important information that can be persuasive enough to move us to action. When that data addresses a need or concern, we act.

The Call to Action
There is a strong and ever growing accountability movement in education spurred on even more by the No Child Left Behind Act, which is adding pressure at both the state and local levels.

The law (or rather, the state implementation of the federal definition of "adequate yearly progress") will provide us with the compelling news: which schools did not make adequate yearly progress and, as a result, will face sanctions. The prospect of facing sanctions will surely persuade us to act, but will it tell us exactly what to change if we are compelled to do so?

Let's return to our automaker example. Many of us passed up on this zero percent financing opportunity even though the automakers presented some pretty compelling information about total cost of ownership. Why? It is likely that many of us looked at the information we had, obtained the additional information we needed, and performed a cost-benefit analysis: Do I really need a new car? How long will mine last? What will it cost even at zero percent interest? And, many of us decided that even such a great deal was not worth incurring an unnecessary expense.

We are used to doing these cost-benefit analyses when we calculate the pros and cons of a particular decision at home. But what useful data and information do we have on which to base major education decisions in our schools—decisions about curriculum and assessment, about what programs to keep and what programs to change?

The most readily accessible information we have is most likely a report based on aggregate school data that compares how our school does verses another. These reports do not tell us which students do well over time, which do not, which are contributing to flat performance, and so on. In fact, when you think about it, the most common unit of analysis used by most of these "state report cards" is the school, not the child. But children should be the basis for No Child Left Behind.

The Power of Detailed Data
From a simplistic point of view, a straightforward report of how the school did from year to year may suffice—at least from the federal perspective. But to know specifically what to change, we need the detailed data to do more complex "drill-down" analyses. Just seeing the commercial will not move me to buy the car. Only after careful analysis of the details and a comparison of the available options will I be able to make a reasoned decision.

Persuasive data, data that moves us to take action, comes in two forms: the big picture data and the details that show us what, if anything, should be changed. When we have these data—the details that provide a complete picture—we are likely to decide to do something productive, something that benefits the school and its students. (By the way, sometimes a reasonable decision is a conscious one to do nothing.)

Thus, we can see that learning about an issue from a simple report or seeing it at a basic level (learning about zero percent financing or seeing the Orion Nebula through binoculars) is the first stage of decision making or learning about an issue. But action or deeper learning most often requires some level of analysis and more sophisticated tools and techniques.

Three Stages of Data-Driven Decision Making
We know that educators have a great deal of knowledge about their school. Even with drill-down analysis, however, we rarely come to a clear and concise decision point. Often, we learn more about the problem and potential solution, thereby better "informing our intuition" about what needs to be improved. Data-driven decision-making analysis adds to that knowledge base and provides a clearer pathway to action.

A three-stage model of data-driven decision making helps guide and flesh out the data report-to-data analysis continuum. Stage I relies on the aggregate reporting of roll-up averages, such as those provided on school "report cards." Displaying how one school performed as compared with others paints a big picture of school effectiveness and progress, but it doesn't provide the data for the "drill-down" detailed analyses we need to determine if change is necessary.

Stage II uses more in-depth analyses based on detailed, individual student data and includes reporting based on these analyses. The initial conclusions drawn from Stage I reports are often misleading and downright wrong when subsequently analyzed more deeply at Stage II. Stage II work:

• Provides a deeper and clearer picture of performance of specific groups of students and their members
• Allows us to test performance trends for statistical and practical importance
• Identifies sub-groups, even specific members, who need intervention
• Identifies students who are contributing to academic improvement
• Allows for initial program evaluation where logical links between student course grades and standardized test scores exist.

At this level, more revealing and persuasive findings come to light, like discovering the birth of stars in a nebula. However, for these Stage II reports to be truly useful, all of this number crunching needs to be followed up with observation and interviews to help us better understand trends and drive potential actions. The reports provide a useful compass direction for decisions, but often do not lead to specific and discernable actions.

That may sound curious, if not frustrating, as one would think that after all this work a decision or direction should become obvious. Occasionally that does happen, but not often. It's a lot like buying that car: at the end of all the analysis we are still left with a tough and personal decision (unless of course your auto just hit 200,000 miles and died on the road).

With the results of the analyses from Stage II, educators can begin to discuss not just the big picture, but the minute details of what is really going on in the school. It is those discussions about the relevance of the data that are the persuasion factor. Meaningful actions and change are the results.

Stage III is the application of data-mining tools specially designed to determine "root cause." The field is just now exploring use of these tools and techniques to determine what exactly causes and inhibits achievement in the classroom.

Finding the Riches
Educational decision making to improve achievement is challenging because the variables are often fuzzy, missing, or unmatched with those we want to compare. And, the analysis necessary to get the most useful information is hard to do. Fortunately, there are tools available to help, but there is still a lot of work to do to get it done. Even then, we are often left with just a better sense of informed intuition about what to do. What we can say with some assuredness is that review of simple Stage I reports is not likely to inform our intuition and guide decision making. It's a needed first step to decide what to look at more deeply.

To use another analogy, it's like going to the general physician who looks you over and then decides to send you to a specialist. Without this necessary first step, we don't know what specialist to go to and can waste precious time and resources. Then, we begin looking more deeply. It is here that we can find riches in data that can truly guide our decision making.

In the pages that follow, middle level colleagues share their experiences with the various stages of data-driven decision making. We invite you to join us on this expedition.

Philip A. Streifer, a former superintendent, is associate professor of educational leadership at the University of Connecticut and president of EDsmart, Inc., a company that provides data warehousing and analysis to school districts. He is the author of Using Data To Make Better Educational Decisions , published by Scarecrow Education with the American Association of School Administrators.