Research Progress

I just spoke with Sharon (our programmer) about how the integrated writing tasks are going (we are test piloting them with Levels 1, 3, and 5 today). It was good to see that Level 1 (the only level that handwrote) was able to summarize – and even synthesize – the listening and reading passages. We talked about how Level 3 might be especially difficult (given that it is not Level 3 adjusted and it TOEFL level with just a few vocabulary changes to make it easier).

We also discussed writing ratings and the evaluation study. She is very open to programming an interface for online writing rating feedback since she previously developed something similar to it in the past (for speaking rating). I explained that raters would want to see what others had rated a portfolio after they submitted their own rating to the system. She thought I meant all current raters, but I explained that it would be a database of previous ratings (collected from previous semesters and maybe including a recent set of ratings that I solicit from current teacher to be added to the database of benchmark portfolios).

I also think that the database should include more than just scores; information from the evaluation surveys indicates that raters want to see/hear the justifications for a rating. So, perhaps I could get current raters to write justifications (or maybe audio record them and then transcribe them) and add those to the database along with their scores.

Until this week, the evaluation had been progressing slowly. At the end of last semester I distributed surveys to all raters, but my research assistants never picked up the surveys before they both left for out of town jobs at the end of the semester. Frustrated, but not discouraged, I resent the surveys at the beginning of this semester, but half of the raters had left the ELC (due to graduation and full-time job offers) so I was only able to send out seven surveys. At the beginning of this week I had only received two of the seven, so I sent out another reminder and by this morning I had gotten two more. I will follow-up with the remaining raters in order to get the maximum number of responses possible.

The next step for the evaluation? I need to schedule interviews or a focus group with the raters to follow-up on the interviews. TREC had counseled that I not lead the interviews/focus group in an effort to minimize rater inhibition for free expression. I could a grad student to do it for me, but I want to use someone who is familiar with the rating system. I could also use a current teacher, who is one of my raters, but will her personal bias influence how she interprets feedback from others?

The next step for the dissertation study will involve sending a pilot survey to teachers and students based on the practice LAT is being piloting this week and next. The survey piloting will help me to revise the survey for actual use at the end of the semester during LATs. I have received IRB approval, but I am still awaiting final approval on my release form (the ORCA office wants me to update it).