School Information System: "When controlling for demographic characteristics, the effects of additional years in MMSD on WKCE scores are largely ambiguous": An Update on Madison's Transfer Students & The Achievement Gap

January 12, 2013

"When controlling for demographic characteristics, the effects of additional years in MMSD on WKCE scores are largely ambiguous": An Update on Madison's Transfer Students & The Achievement Gap

MAJOR FINDINGS 1. Students who have spent more time in MMSD perform better on the WKCE than their peers who have spent less time in MMSD.
2. Students who have spent more time in MMSD are demographically different from recent arrivals, who are less likely to be white and more likely to be low-income.

3. When controlling for demographic characteristics, the effects of additional years in MMSD on WKCE scores are largely ambiguous.

Based on these findings, MMSD may be better served by refining its core curriculum to meet students' needs based on demographic characteristics rather than the recency of their arrival in MMSD. The recent arrivals report is attached. Our official statement about our findings follows.

The most notable anomaly is among 10th grade students. In both Reading and Math, 10th grade students who had spent one year in MMSD performed as well as students who had spent their entire careers in MMSD and substantially better than new students as well as students who had spent between 2 and 9 years in MMSD. This suggests that students who enter MMSD in 9th grade are altogether different from students who enter in other grades. The high performance level for students spending one year in MMSD prior to 10th grade may reflect students entering MMSD in 9th grade after attending private schools through 8th grade.

--------------------------------

It is the district's responsibility to meet students where they are in their learning and identify needed interventions, enrichment or other programs to advance that learning. That means we need to have curriculum and programs that work for all of the students we serve, regardless of demographic background or how long they have been in the district.

Unfortunately, we know that achievement gaps exist in schools across the country, and no single district has entirely eliminated them. Focusing only on how long a student has been in our district does not underscore the complexity of the issue and is not the most effective predictor of achievement.

Instead, strengthening classroom instruction and ensuring interventions and enrichment that advance learning for every student regardless of demographic characteristics will yield the best results.

However, we do know that mobility, including moving from another district or moves within MMSD, does have some impact on achievement. Exploring community solutions to enhance stability throughout a student's education could both increase achievement and help close gaps.

Larry Winkler kindly published a more detailed analysis, here.

I asked several observers for their perspective on the rhetoric, assertions and the Friday report. Here's one:

"When the data were first presented, the argument put forth was that the performance of newly arrived students explained much of the performance gap that we see in our schools. However, when the District examined the effects of race and socioeconomic status in the analysis, they found that the performance of low income and minority students who had been in the MMSD for many years was not significantly different from the performance of low income and minority students who were new to the District.

It is disappointing that the District and the Mayor's office ran so far and so fast with their initial, incomplete analysis."

UPDATE: Larry Winkler kindly created a set of charts scaled by percentages.

Posted by Jim Zellmer at January 12, 2013 5:57 AM
Subscribe to this site via RSS/Atom:

Newsletter signup | Send us your ideas

Comments

I'm bringing this to the top of the comments section because I think it's important for people to read and understand. ... Upon further analysis of the data, it's become clear that the District and the Mayor jumped the gun, that the achievement gap IS NOT due to poorly prepared transfer students. ... Thanks, Larry, for the additional analyses.

Posted by: Laurie Frost at January 16, 2013 8:35 AM

I understand the view that the Mayor jumped the gun in his analysis but I am not aware of representatives of the school district making the same sort of assertions. Am I missing something?

Posted by: Ed Hughes at January 16, 2013 9:10 AM

Hello Ed:

Thank you for writing. Do you have any idea how the mayor came upon the "transfer from lower quality districts" assertion:

"There is an achievement gap. A significant part of the achievement gap is not because of the failure on the part of the Madison public schools, but it is because of the number of students who have transferred here from other districts, districts like Chicago," he says.

"Those kids come here unprepared. They come from poorly performing schools. There is a reluctance to discuss this factor. The reluctance to discuss it has at least two consequences. The first is that we come to erroneous conclusions about the quality of education in Madison. The second problem is that we don't develop strategies for these kids so that we can close that achievement gap.""

http://www.schoolinfosystem.org/archives/2013/01/madisons_mayor_.php

Posted by: Jim Zellmer at January 16, 2013 4:04 PM

Jim -- I know that the mayor has been interested in the impact of students transferring into the school district but I don’t know anything more than that.

Posted by: Ed Hughes at January 17, 2013 8:48 AM

Also see: http://www.schoolinfosystem.org/archives/2012/04/superintendent_20.php, in which Superintendant Nerad asserted that "Tenure matters. The length of time a student is with MMSD or in one of its high schools has an impact on the likelihood he or she will earn a diploma or equivalency." This may be one of the sources for the mayor's assertion.

Posted by: Steve Rankin at January 17, 2013 7:27 PM

I think the data, even scaled as they are, point out the problem of mobility, and lend support to the District's and Mayor's position. This is not the time to stop teasing out the real issues in the data. The study needs to be refined, but even with this study it is clear that mobility is an issue. More statistical analyses will yield interesting insights. Let's not stop.

The entire student pool is compared, which is not the best way to look at this. It would be better to break down into income levels and/or racial categories and compare. For example, poor kids coming into the district should be compared with poor kids that have spent varying years in the district and compare both to wealthy students. And it may be different for each school.

In a school district that is doing its job, you hope the differences between mobile students non-mobile students and will become narrow in 2-3 years.

Posted by: Donald Pay at January 17, 2013 10:59 PM

"The entire student pool is compared, which is not the best way to look at this."

Bingo. The initial analysis says only that there is a main effect for short tenure when you look at all kinds of students combined, i.e., that mobility matters. That is important (even if not particularly new information) and deserves to be addressed.

But as the later analyses make clear, the initial analysis does not support the argument that the MMSD achievement gap is due largely to poorly prepared transfer students (nor its implication that poor minority students who grow up in the MMSD do just fine). Think about who you're comparing when you compare all students who have been in the MMSD for a long time to all students who are new. Think about the demographic differences between new versus long-tenured students. (Note: There was also the problem in the original analysis that some of the students who took the 8th grade WKCE but not the 4th grade WKCE may have done so for reasons other than not yet being MMSD students -- e.g., IEP's, language issues, non-attendance on test day, etc.)

To return to the main point --

When the District initially released the data in September; when the Mayor and others initially told the story that the achievement gap is due to the influx of poorly prepared transfer students; when the District failed to correct the Mayor immediately and do further analyses to test his hypothesis right away ... all erred, all acted irresponsibly.

IMHO, apologies, a clearly written and accurate update, and front-page coverage aimed at helping the public understand what the data do and do not say* are in order.

*short tenure in the MMSD -- though important in and of itself -- does not add to our understanding of the achievement gap beyond what we already know based on race and SES

Posted by: Laurie Frost at January 18, 2013 9:24 AM

Thank you Laurie, Larry, and Jim for the skilled analyses and for making your work available to others. These are important questions, and the devil is - per usual - in the detail. Unfortunately, the narrative presented in the original roll-out is likely to persist unless the questions about conclusions and meaning are repeated and pursued in the media, district administration, and the full board of education.

I would be remiss if I did not note that staff and some board members have been asking for programs that assess student preparedness and develop remedial programs as needed for students transferring into the district. Those proposals and requests go back at least 8 years if not more.

Alas, those ideas have fallen on infertile ground and the district continues to handle student transfers as a numerical administrative function rather than a question of integrating learners into an active educational system.

If people truly believe that the big issue is students transferring into the district, shouldn't more have been done to develop appropriate programs and supports to create the best opportunity for student success? The ideas are there to be implemented IF there is a commitment to addressing the problems. Even if the problems identified are only a small part of the bigger picture.

Posted by: Lucy Mathiak at January 29, 2013 12:30 PM