There seems to be a recent push to validate the TEI through the media and not research with some dubious claims that are not supported by proper data analysis. Therefore I want to recommend some data concerns and how to possibly test them. A grossly inaccurate merit system will destroy DISD as some superior teachers will either exit voluntarily or through non-renewal while some incompetent teachers will be rewarded and this pattern will accumulate over the years.
TEI Data Concern #1: The claim is the best teachers are retained and the worst exit. The merit pay system in Harrison 2, according to the Colorado state data, made the teacher turnover rate explode and lowered the years of teacher experience. As a result discipline issues skyrocketed, and the AP program collapsed. Unfortunately, DISD is starting to mirror this trend. I am most concerned about losing the best and would look at changes in STAAR scale score converted to percentile ranks within a school and see what happened to the lowest and highest growth performing teachers within a school.
TEI Data Concern #2: The most critical claim is that the TEI is accurate in identifying teacher effectiveness. The main student achievement criteria for teachers in the DISD merit pay scheme is not the CEI but either absolute scores or a gain model called the Colorado Student Median Percentile Growth Model. That growth model does not adjust for any factors outside the school and uses only four wide achievement bands as pretest classification. It has a high likelihood of making teachers at high performing schools look good in large part due to parental influence, not teacher effectiveness. Look at Holly Hacker’s recent list at the DMN. The magnets are at the top. That might be correct but I would like to make sure the achievement portion of the TEI does not correlate to emphasis of education in the home or economic status of the parents, which I suspect does.
TEI Data Concern #3: There is a claim the district has a robust measurement model. The district lost an appeal and the AG made them release the item level data on the ACP. The district now creates 255 tests per year without the time or expertise to write and analyze them properly. I was sent that data file and analyzed a few tests. Each had too many errors to justify use for merit pay. One ACP, 5th grade ELA in fall 2014, was so bad 90 percent of the items had 90 percent of the students answering correctly and the other 10% of the items were so poorly written they confused the brightest kids. How can teachers be judged on a test almost everyone scores high and the remaining items are invalid? The poorly designed tests were never used as predictors in the old CEI model unless they met the criteria of a good predictor and few did. Now they are forced to be predictors in at least the student median growth model. I would use the TEI algorithms to develop simulations ranking teachers on the achievement portion of TEI in 2013, 2014, and 2015 to make sure the TEI is measuring something besides noise after controlling for poverty. A few years ago I took teachers with three years of CEI, ranked them and put the data in a 3-D rotational Excel plot. The plot almost mimicked a random number generator.
TEI Data Concern #4: There is a claim that teacher experience and education are negatively correlated to student achievement: I have seen a dissertation with a retrospective analysis where those DISD students who earned a 4 year college degree significantly had more experienced and educated teachers after controlling or prior achievement. Of course most students do not earn a degree so I would like to see a properly designed study that controls for factors like prior achievement when looking at a predictive model. I would define experience as time teaching the subject that the teacher is qualified to teach. We all know teachers are forced to teach outside their expertise.
TEI Data Concern #5: There is an assumption that spot observations are valid and reliable. Teachers are reporting fall spot observations as artificially lower than spring observations to feign growth and that scores are dependent on compliance to an unknown theory of education. Having observations time dependent, and thus problematic, would be easy to show. Also, there are advanced statistical techniques that would use the analytical scores of each rater to measure agreement upon the structure of the observation space and the placement of a teacher in that structure space by the various raters. If the different raters agree on the final score of a teacher but for different underlying reasons the validity of the observation has to be questioned. Think of wine testers. There are universal experts who agree on what is a good wine and they agree for the same reasons. Now you have novice wine raters and you want to know how close they are to the experts and do they have similar reasons. The TEI has never established the set of expert ratings to emulate nor has it analyzed any underlying structure of the observations. Do not equate this to inter-rater reliability where everyone agrees to a summative rating, often to show conformity.
TEI Data Concern #6: There is a claim that student surveys can accurately indicate teacher effectiveness. Most of us remember the teachers who pushed us and we disliked the most turning out to be the ones later in life we respected the most. Analysis of past student surveys in DISD indicate a significant portion of students may not have even read the questions but merely filled down in a global manner. Since no reversely worded items exist to check this, I would do a structural analysis of the survey of good versus poor readers and see if the same underlying structures and scales exist for both groups. If not, teacher ratings may be a function of reading ability or attitude of the student and not true teacher effectiveness.
Dr. Michael A Dryden LLC, is a retired 20 year evaluator of DISD. He evaluated most of the recent DISD reform programs such as the Learning Centers, School Centered Education, Urban Systemic Initiative in math and science, and Edison Schools Project. Dr. Dryden has a doctorate in research and evaluation with an emphasis on math and science education. He taught in New York, Australia, Samoa, and Indonesia. He evaluated state (AZ), national (US), and international educational systems, now called TIMSS. Despite official retirement, Dr. Dryden continues to learn and do research in his spare time.