Your Data Is Lying to You

Why collecting data isn’t enough and the questions your district needs to start asking

A superintendent slid a thick report across the table to me last spring of color-coded dashboards, trend lines moving in the right direction, and fidelity scores most districts would envy.

“We’re doing everything right,” she said. “The data proves it.”

I looked at the report. Then I looked at her and said, “Tell me about your chronic absenteeism numbers.”

She paused. “They’re up. But that’s a post-pandemic thing every district is dealing with.”

“And your suspension rates for students with IEPs?”

Another pause. “Also up. But we’ve had some tough cases this year.”

“And staff retention in your intervention roles?”

The color drained from her face.

Here was a district that could prove, with charts and percentages, that it was implementing its programs with fidelity. And yet the students those programs were designed to serve were falling further behind, disappearing from school more often, and being removed from classrooms at higher rates than ever.

The data wasn’t wrong. It just wasn’t being questioned.

The absenteeism numbers were there, so were suspension rates and retention data. But each one had been met with an explanation of post-pandemic, tough cases, or a hard year, rather than a question. No one had asked what the data was pointing toward. No one had followed the thread to the system underneath.

That’s the real problem in most districts I work with. Not that the data doesn’t exist. Not that leaders don’t have access to it. But that the culture of inquiry needed to use it honestly, to sit with uncomfortable numbers and ask why rather than reaching for a justification, is missing.

The Comfort of Explanation

There's a reason data presentations in most districts feel reassuring. They're constructed to show that work is happening, not whether it's working. Activity metrics, the number of trainings completed, the fidelity scores, the percentage of Tier 2 slots filled, are easy to collect, report, and defend. They tell the story of effort to confirm that things are happening.

What they can’t tell you on their own, without someone willing to push past the surface is whether any of it is working.

The problem isn’t the data itself. Activity metrics have their place. Knowing whether a program is being implemented, whether staff are trained, whether the structure exists are legitimate things to track. The problem is what happens when those metrics become the end of the conversation rather than the beginning..

When a district reports 85% PBIS fidelity and stops there, the implicit message is, we’ve done our part. When chronic absenteeism is rising but attributed to post-pandemic trends, the implicit message is, this is outside our control. When suspension rates are up but explained by “tough cases,” the implicit message is, the system is fine; the students are the variable.

Each of those explanations may contain some truth. None of them constitute analysis. And the difference between an explanation and an analysis is the difference between a district that feels like it’s managing and a district that actually is.

Genuine data use requires the willingness to be uncomfortable. It requires sitting with a number that doesn’t look good and asking not “what explains this?” but “what is this telling us about our system?” That’s a harder question. It implicates decisions that were made, structures that were built, and leadership choices that are now visible in the outcomes. It’s also the only question that leads anywhere useful.

The superintendent across the table from me had data. What she didn’t have was the practice, or perhaps the permission,  to interrogate it. She had learned, as many leaders do, that the data’s job was to support the narrative, not complicate it.

That’s the habit I’m asking you to break.

The Five Questions Districts Aren’t Asking

These aren’t hypothetical gaps. They’re the specific places where I most commonly see district leaders accept a surface-level answer when a deeper question is sitting right underneath it. Each one represents data the district already has and isn’t fully using.

1. “Our referral numbers are down” — but what happened to the students?

Fewer referrals can mean your Tier 1 systems are stronger and students are getting support earlier. It can also mean staff have stopped referring because they’ve lost faith that anything will happen, or that students have learned which behaviors are tolerated and which aren’t.

The referral number is the starting point. The question that matters is: what happened after the referral? Are students who were referred receiving documented support? Are they showing different patterns afterward? Are the referrals coming earlier in the escalation cycle, or later? A declining referral number without answers to those questions isn’t progress. It’s an incomplete story.

2. “We have 85% PBIS fidelity” — but what is the behavioral culture actually like?

Fidelity tells you whether staff are implementing the program as designed. It does not tell you whether the program is producing results in your specific context, with your specific student population.

The question underneath the fidelity score is, what do we see when we look at student outcomes over time? When you walk through the campus, are students demonstrating the behaviors the program is designed to build? Are the students with the highest behavioral needs benefiting from the universal system, or are they cycling through it without meaningful change? High fidelity and poor outcomes together are important data. They tell you the program is being implemented correctly and still isn’t producing the results you need,  which is a different problem requiring a different response than low fidelity.

3. “Staff attended all the trainings” — but what changed in practice?

Training attendance is perhaps the most commonly reported activity metric in education and one of the least predictive of anything meaningful. We know from decades of adult learning research that attendance doesn’t predict implementation, and implementation doesn’t predict integration.

The question underneath attendance data is, what are we seeing in classrooms and on teams as a result of this training? Are the practices showing up? Are staff using the shared language? Are the tools being applied? If you can’t answer those questions, you don’t have training data, you have attendance data. Attendance data tells you where staff were on a given Tuesday, not what they’re doing on every other day of the year.

4. “Our Tier 2 slots are filled” — but what are the outcomes for students who completed the program?

Full slots tell you the program exists and that students are in it. They tell you nothing about whether it’s working.

The questions that matter are, what is happening for students after they complete Tier 2 support? Are they returning to Tier 1, or cycling back through Tier 2 again? What percentage is being escalated to Tier 3? How long is the waitlist, and what is happening for students while they wait? A program with full slots and no outcome tracking isn’t a support system. It’s a well-documented waiting room.

5. “Suspension rates are lower” — but are student behaviors actually changing?

Reduced suspension rates can represent genuine progress and I want to be careful not to dismiss that. But they can also represent displacement. Students who would have been suspended are instead managed informally, kept in school without appropriate support, or exited through processes that don’t appear in suspension data.

The question underneath the number is, what is happening for the students who are no longer being suspended? Are they receiving support that addresses the behavior that used to generate suspensions? Are we seeing different behavioral patterns over time for those students, or the same patterns managed differently? Genuine progress and data manipulation can produce identical suspension numbers. Only the follow-up question tells you which one you’re looking at.

What It Looks Like to Actually Use Your Data

Shifting from data collection to data use isn’t about gathering more information. Most districts are already drowning in data. It’s about building the habit and the structure to ask harder questions of the data you already have.

One of the most useful frameworks for this is the distinction between leading indicators and lagging indicators. Lagging indicators tell you what has already happened, such as suspension rates at the end of the year, annual assessment scores, and year-end behavior incident totals. These do matter but by the time you have them, it’s too late to change anything for the students represented in those numbers.

Leading indicators tell you what is likely to happen. They measure early warning signals. These are the things that predict future outcomes before those outcomes are fixed, such as, chronic absenteeism trajectories by October, the number of students seen by a counselor in the first six weeks of school, referral-to-support timelines, and Tier 2 completion rates and what follows them.

Leading indicators give you something lagging indicators never can —  time to ask the question before the outcome is already decided.

Here are the questions your data infrastructure should be able to answer. If it can’t, that’s the starting point:

• For every student who received a suspension this year, was there a documented intervention designed to address the behavior that led to the suspension?
• What percentage of students referred to Tier 2 support received that support within two weeks of referral?
• For students who completed a Tier 2 intervention, what happened in the six months following completion?
• Which schools have the highest rates of staff identifying concerns early and what are those schools doing differently?
• Can you identify a student who is becoming at risk before they generate a formal referral?

If your current systems can’t answer these questions, you don’t have a data gap. You have an inquiry gap of a missing practice of following the data somewhere, rather than presenting it and moving on.

The other essential shift is connecting your data across programs so your teams can see the whole student, not just the slice each program tracks. A student can be flagged in your MTSS system, referred to the counselor, receiving special education services, and showing early warning signs in attendance data with none of those systems talking to each other. Each team holds a fragment, so nobody sees the pattern.

Integrated data isn’t primarily a technology problem. It’s a culture and structure problem. You can have the most sophisticated platform on the market and still have siloed information if your teams aren’t structured to share, discuss, and act on what they’re seeing together. The data connection has to happen in the room with staff asking tough questions before it can happen in the system.

The System Audit Underneath the Data Problem

Here’s what I consistently find when I conduct ALIGN Audits in school districts; the data problem is almost never actually a data problem. It’s a culture problem that shows up in how data is used or more precisely, in how little it’s interrogated. Districts have data. They present it, they accept the first-level explanation, and they move on. Then the cycle repeats.

Fragmented programs produce fragmented inquiry. When PBIS, MTSS, SEL, threat assessment, and mental health supports operate as separate initiatives, each with their own teams, their own reporting cycles, their own metrics, and no one is positioned to ask the cross-system questions. The PBIS team looks at behavioral data. The MTSS team looks at academic and attendance data. The mental health team looks at counseling referrals. The student who is showing early warning signs across all three domains is visible to no one, because the question “what do we see when we look at everything together?” is no one’s job to ask.

A true diagnostic review of a district’s data culture asks three foundational questions:

What are we actually trying to learn? Most districts have inherited metrics from grant requirements, state reporting mandates, and program vendors. Those metrics were designed to serve external accountability purposes, not internal improvement. Getting honest about what you actually need to know as opposed to what you’re required to report is the necessary first conversation. The data you collect should be driven by the questions you’re trying to answer, not the other way around.

Who is responsible for asking the next question? Data without an owner and a decision-making process attached to it is just storage. If your MTSS team reviews attendance data monthly but has no protocol for what triggers a deeper look, the data isn’t functioning, it’s being managed. The question is never just whether data is being reviewed. It’s whether someone is accountable for following it somewhere.

What would it look like if our systems were working, and how would we know? This is the question most districts have never formally answered. Define success before you collect another data point. If your PBIS system is working, what will be true for students in three years that isn’t true today? If your Tier 2 interventions are effective, what outcomes should you be able to demonstrate? Build your data questions backward from the outcomes you’re working toward, not forward from the programs you’ve implemented.

The Leadership Challenge: Curiosity Over Reassurance

I want to acknowledge something that makes all of this harder than it sounds; the systems around leaders often reward reassurance and penalize uncertainty.

Boards want to be told that the programs they funded are working. Communities want evidence that their schools are safe and effective. Cabinet members want confirmation that the initiatives they championed are producing results. So leaders learn to present the data that supports those narratives not out of dishonesty, but because the culture around them has made explanation safer than inquiry.

This is the leadership trap that unexplored data enables. When you accept the first-level explanation — when absenteeism is “a post-pandemic thing” and suspensions are “tough cases” and fidelity scores are “good enough” — the gap between what your data shows and what’s actually happening for students keeps widening quietly. Eventually, the gap becomes impossible to close.

Here’s the reframe I offer to every district leader I work with: saying “our data is raising questions we haven’t answered yet” is not an admission of failure. It’s the most important thing you can tell your board this year.

Because it signals that you are asking harder questions, which is something boards rarely hear from district leadership. It says that you are not willing to accept a surface reading of numbers that deserve more and that you understand the difference between a district that looks fine on a dashboard and a district that is actually functioning well for students.

Presenting an honest data picture doesn’t require triggering panic. It requires context and a clear forward path. “This is what our data shows. This is what we’re still trying to understand about it. This is the question we’re currently investigating, and here’s what we expect to be able to tell you in ninety days that we can’t tell you today.” That’s not weakness. That’s the kind of leadership that builds long-term trust.

The bravest thing a district leader can do in a board presentation is put up a slide that says, “This number concerns us, and here is how we’re digging into it.” That’s not a failure to report; instead it’s a commitment to actually lead.

The Question Worth Asking

I’ll leave you with the same question I asked that superintendent when I looked up from her beautifully formatted report:

“If your programs stopped working tomorrow, when would you know and how?”

If the honest answer is “we’d know when the board asks why our numbers changed,” that’s a district that is waiting for lagging indicators to tell a story that leading indicators could have surfaced months earlier.

If the honest answer is “we’d know within a few weeks, because we’re reviewing outcome data at the student level and our teams are structured to ask the hard questions when something doesn’t add up,” that’s a district with a functioning inquiry culture. That’s a district that can course-correct before the crisis arrives.

The districts making real progress aren’t the ones with the cleanest dashboards or the most sophisticated data platforms. They’re the ones where someone in leadership is willing to look at a number that doesn’t look good and ask, with genuine curiosity and without defensiveness, what is this telling us, and what are we going to do about it?

That question is yours to ask and the students in your district are waiting on the answer.