Vocabulary Measures

Because I plan to have some type of linguistic measurement as part of my validation study, I have been reading about types of language statistics. The most recent is a study by Batia Laufer and Paul Nation.

Laufer, B., & Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16, 307-322.

I am not familiar with Laufer, but I most certainly have heard about Nation. He is the man behind the Academic Word List (AWL), and list of commonly used words in university writing. We use the AWL at our center to help our students improve their reading and writing of academic texts.

This study appears to be pre-AWL since Laufer and Nation make no reference to it, and instead refer to the general service list and other university word lists. I imagine that the AWL was created not long after this study, since the inklings of the AWL are emanating from this work.

The primary focus of the study is an argument in favor of a new kind of vocabulary measure: the Lexical Frequency Profile. Although I don't exactly buy the LFP bid (the explanation of its use was verbose and confusing), I did appreciate the succinct discussion of existing measures of vocabulary use. In a matter of about 2 pages, Laufer and Nation clearly explain the pros and cons to several commonly used lexical measures. Even if I didn't get much out of the rest of the paper, I enjoyed their assessment of these measures.

The question is raises for me is: what linguistic measures will I use in my study? Almost certainly I would like to have at least one lexical measure, and hopefully this study will help me to explain and justify my decision.

Research Proposal Draft 1

I finished a draft of my dissertation research proposal yesterday and took it by the department to show it to potential committee members. I was well received and everyone agreed to look it over with the promise that I would check back with them later in the week. And that started today.

I met with the potential chair of the committee, whose first comment was, “Do you remember what advice I give to anyone who attempts to do a validity study for a dissertation?”

Of course I knew. The answer is, “Don’t do it.”

“It’s not that I think that validity studies are a bad idea; they are very important and should be done,” he said. “Just not as dissertations.”

“Why exactly is that?” I asked.

“They are complicated and intensive. It’s hard to get it done.” Which I knew. I recognize it. In fact, as I explained to him, I have already done the lit review and collected the data of for a study that might work in place of this validity one. It would be an easy route. But I’ve already done it.

The integrated skills study needs to be done. It is complicated, but it’s real. I’d rather do something that is needed than something that it easy. So, yes, in other words, I am crazy and stupid.

In our conversation, this professor and I discussed my proposal and how I can limit/focus my efforts while still maintaining the multi-faceted approach of a good validation study.

The next step is to flesh out specific research questions, a solid literature review, and a clear methods section (including all proposed analyses). If I can get that done by January, I think that I will be in a good position to still do this complicated study while aiming for a realistic graduation timeline.

Research Progress

I just spoke with Sharon (our programmer) about how the integrated writing tasks are going (we are test piloting them with Levels 1, 3, and 5 today). It was good to see that Level 1 (the only level that handwrote) was able to summarize – and even synthesize – the listening and reading passages. We talked about how Level 3 might be especially difficult (given that it is not Level 3 adjusted and it TOEFL level with just a few vocabulary changes to make it easier).

We also discussed writing ratings and the evaluation study. She is very open to programming an interface for online writing rating feedback since she previously developed something similar to it in the past (for speaking rating). I explained that raters would want to see what others had rated a portfolio after they submitted their own rating to the system. She thought I meant all current raters, but I explained that it would be a database of previous ratings (collected from previous semesters and maybe including a recent set of ratings that I solicit from current teacher to be added to the database of benchmark portfolios).

I also think that the database should include more than just scores; information from the evaluation surveys indicates that raters want to see/hear the justifications for a rating. So, perhaps I could get current raters to write justifications (or maybe audio record them and then transcribe them) and add those to the database along with their scores.

Until this week, the evaluation had been progressing slowly. At the end of last semester I distributed surveys to all raters, but my research assistants never picked up the surveys before they both left for out of town jobs at the end of the semester. Frustrated, but not discouraged, I resent the surveys at the beginning of this semester, but half of the raters had left the ELC (due to graduation and full-time job offers) so I was only able to send out seven surveys. At the beginning of this week I had only received two of the seven, so I sent out another reminder and by this morning I had gotten two more. I will follow-up with the remaining raters in order to get the maximum number of responses possible.

The next step for the evaluation? I need to schedule interviews or a focus group with the raters to follow-up on the interviews. TREC had counseled that I not lead the interviews/focus group in an effort to minimize rater inhibition for free expression. I could a grad student to do it for me, but I want to use someone who is familiar with the rating system. I could also use a current teacher, who is one of my raters, but will her personal bias influence how she interprets feedback from others?

The next step for the dissertation study will involve sending a pilot survey to teachers and students based on the practice LAT is being piloting this week and next. The survey piloting will help me to revise the survey for actual use at the end of the semester during LATs. I have received IRB approval, but I am still awaiting final approval on my release form (the ORCA office wants me to update it).

Integrated Writing Task

My plans for a dissertation topic revolve around the use of integrated writing tasks. These exam items, used in ESL assessment, require students to write about a reading and listening passage. We are now piloting the use of integrated writing tasks at our center.

Reasons Why We Want to Implement the Integrated Task

• Every semester we inevitably deal with cases of unintentional plagiarism due to students whose summary, paraphrase, and quoting skills are weak
• Most of our students are planning to attend university in the USA and need to learn English for Academic Purposes (EAP) which involves writing from sources
• Integrated writing tasks are now part of the Test of English as a Foreign Language (TOEFL) which many of our students want to take and "pass"

Differences Between the TOEFL's and Our Integrated Tasks

• Our integrated tasks are used in connection with our Level Achievement Tests in order to assess students' readiness to move on the next class level whereas the TOEFL is used to assess readiness for university study which our program only assesses in an indirect form through level advancement
• Our implementation of the integrated tasks will assess student ability from novice to low advanced, namely our Levels 1 through 5; the TOEFL is designed to assess a narrower band of language ability, namely the ability of students in our Levels 4 and 5
  
• The TOEFL tasks are rated individually, but the integrated tasks will likely become part of our holistic portfolio score, or maybe even a subset portfolio score for fluency (non-draft) writing

In order to properly implement these tasks for our own center, I'm conducting a literature review of relevant studies. The first:

• Cumming, A., Kantor, R., Baba, K., Erdosy, U., Eouanzoui, K, & James, M. (2005). Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing, 10, 5-43.
  
• This study analyzed a variety of textual features of prototype integrated tasks versus the current independent writing tasks. It suggests that integrated tasks encourage the use of a greater variety of vocabulary and greater use of source ideas. The independent tasks are better at eliciting more language and at developing better arguments from examinees.
  
• I hope to be able to use this study as a model for the quantitative analysis of my dissertation research. Like the Cumming et al. study, I will examine a variety of discourse features and how they differ between responses to our independent and integrated tasks; I will also plan to look at these language features as they differ across our 5 proficiency levels.
  

I'll add more studies as I read them.