Vocabulary Measures

Because I plan to have some type of linguistic measurement as part of my validation study, I have been reading about types of language statistics. The most recent is a study by Batia Laufer and Paul Nation.

Laufer, B., & Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16, 307-322.

I am not familiar with Laufer, but I most certainly have heard about Nation. He is the man behind the Academic Word List (AWL), and list of commonly used words in university writing. We use the AWL at our center to help our students improve their reading and writing of academic texts.

This study appears to be pre-AWL since Laufer and Nation make no reference to it, and instead refer to the general service list and other university word lists. I imagine that the AWL was created not long after this study, since the inklings of the AWL are emanating from this work.

The primary focus of the study is an argument in favor of a new kind of vocabulary measure: the Lexical Frequency Profile. Although I don't exactly buy the LFP bid (the explanation of its use was verbose and confusing), I did appreciate the succinct discussion of existing measures of vocabulary use. In a matter of about 2 pages, Laufer and Nation clearly explain the pros and cons to several commonly used lexical measures. Even if I didn't get much out of the rest of the paper, I enjoyed their assessment of these measures.

The question is raises for me is: what linguistic measures will I use in my study? Almost certainly I would like to have at least one lexical measure, and hopefully this study will help me to explain and justify my decision.

Teacher Verification of iBT

While in line for tickets for the university's production of Little Women the Musical (we're going on my birthday for a pre-Thanksgiving holiday warm-up), I finished another integrated tasks evaluation study:

Cumming, A., Grant, L., Mulcahy-Ernt, P., & Powers, D. (2004). A teacher verfication study of speaking and writing prototype tasks for a new TOEFL. Language Testing, 21, 107-145.

You may notice that this is an article written by Alister Cumming who also wrote the iBT integrated tasks textual analysis article that I also read. I also referred to Cumming's other writing test related work for my MA thesis as well as my rater decision-making study (which I need to finish up and then develop into an article). Cumming seems like a very busy man, but he has been kind enough to respond to a couple of emails that I have sent him about his research and about the UToronto program.

Summary: This article attempts to provide content, context, and concurrent-related validity evidence in regards to the speaking and writing tasks for the new TOEFL (which is now known as iBT). As with the previous studies that I have reviewed, this one focuses on both integrated tasks (combining writing/speaking with listening or reading) and independent ones (in which examinees use their personal experience or opinions to complete the speaking/writing tasks).

Whereas the previous articles focused on the language content (Cumming, 2005) and the scoring procedures (Lee, 2006), this one focuses on how teachers of ESL students feel about the structure of the iBT tasks and their students' performance on these prototype items.

Primary questions are:

• Is the content domain of integrated tasks perceived to correspond to the demands of academic English requirements of university study?
• Is the performance of examinees on these tasks perceived to be consistent with their classroom performance?
• Are the tasks perceived to be adequate evidence for making decisions about examinee language ability?

The researchers found that the teachers felt that these tasks were fairly authentic and represented a variety of language skills required at university (so far as a test is able to recreate authentic situations). There were some concerns that the tasks were not fair for lower ability students who performed poorly on integrated tasks if they did not understand the input material; however, for the most part, teachers felt that student performance on the tasks were indicative of classroom ability. The greatest concerns that raters had with the evidence claims of the tasks were that some students may do poorly if they feel uncomfortable sharing personal opinion (for the independent tasks) or if they struggle with the stimulus content (for the integrated tasks). Cumming et al. conclude with some suggestions for improving the task format and content. They also suggest that students would benefit from exam preparation in order to understand the rhetorical and educational aims of the exam. Lastly, they suggest that more research be done into standardized ratings and on teacher perceptions of students' ability in the classroom versus an exam.

Critique: It's a shame that this article it stuck in the middle between an extensive evaluation (involving a variety of teachers and locations) and an intimate study (focusing on one or two individuals and their unique situation). As a result, we have neither the generalizablity of a large statistical study nor the valuable insight into a specific context. Instead we end up with a bunch of incomplete thoughts and ideas. It's as if Cumming et al. scratch the surface of several interesting ideas, but never uncover a single one. Still, for all its limitations, this study provided me with a good model for some qualitative research (including a survey that I adapted) and it is also likely to help me form some of my content and context validity questions. I also decided, as I read its account of teacher-based evaluation, that a student-based evaluation would be a great compliment to such a study. We'll see if I can make it work, or if I will have bitten off more than I can chew.

Connections: This study served to show me that I need to read a lot more about teacher evaluation study about tests. Was this a good example? How have others approached this avenue of validation? I did enjoy seeing how this person-focused (as opposed to text-focused or score-focused) study complimented other, more quantitative, studies on the iBT in order to give a more complete and well-rounded assessment of the exam.

Additional Reflections:

• I would really like to do an evaluation of the integrated writing tasks at the ELC that incorporates myriad sources of validty evidence including teacher-evaluation, student-evaluation, textual analysis, score-analysis, and possibly more. Am I crazy to want to do this much? Will TREC approve such a study? Will I find a dissertation committee who will approve such a multi-faceted approach?
• Who will I choose as teachers for my teacher evaluation? I think that I would prefer to use all of them (provided that they agree to participate in the study). I don't want to exclude someone because their reasons for not being eager to volunteer may actually be connected to their feelings about the exam and are therefore a real concern that I need to take into account.
  
• Will I have time to do this for speaking as well as writing? My tentative schedule, which I need to present to potential committee members next week, outlines an evaluation of writing for winter 2008 and speaking in summer 2008. I think I can be ready, but the issue is as much about analyzing data as it is about collecting it. Who will I get to help, especially when I am so busy around exam time with administrative duties?
• What kind of concurrent validity claims might I make? Will I ask teachers to rate their students and then compare performance with expectations, or will I use data that we already collect (classroom scores or rated evaluations)? Are teachers very good at predicting student ability? Lee's 2005 thesis of the L/S tests at the ELC say no (as did major portions of this study), but I think that both of those are problematic: Lee because the teacher ratings were probably based on Speaking when she was comparing those to Listening scores, and in Cumming's case, he admits that BICS/CALP may have a lot to do with false impressions. Of course BICS/CALP may be an issue for me too, but if students and teachers practice items in the classroom more than once, then teachers ought to be good at predicting success, shouldn't they?
  
• How will tasks be adjusted for lower levels (adhering the concerns of Sara in this study)? Laura (our Level 1 teacher) has already told me that she thinks Level 1 students need to be able to hear the integrated listening passage twice. As it is, they barely understand it before it's over. A second listening for Levels 1 and 2 might be appropriate and justified (given the findings of this study and Laura's experience).

Book Review: Academic Writing for Graduate Students

I have several ESL writing related books that I am in the process of reading. I figured that I would use this space to keep track of them and record my impressions of them. So here's the first:

Academic Writing for Graduate Students: Essential Tasks and Skills (Second Edition). By John M. Swales and Christine B. Feak. Published by University of Michigan Press (2004).

THE BACKSTORY

I first learned of this book in March while attending the 2007 TESOL convention in Seattle. I attended a forum on writing assistance for international students. Although I don't teach graduate students, many of my current students have already finished an undergraduate degree and have come to the USA to pursue graduate work. It's my job to help them improve their English so that they will be ready for graduate study at an English-speaking university. So for that reason, I figured that I would give the session a try.

As it was, I thoroughly enjoyed it. The panel included writing teachers from several different English-speaking universities who have large international (non-native English speaking) graduate students. They discussed the challenges and success to writing programs and what ESL teachers can do to facilitate more favourable university policy towards ESL learners, and better writing skills among international graduate students.

While they discussed their experiences, I wrote ferociously. I was immediately hit with the desire to take these suggestions back to our program and our students. Only there was a snag: I don't teach the writing course for international graduate students. Nevertheless, I figured the least I could do was pass on the material and suggestions. That night, at dinner, I sat at a table with my boss and discussed my suggestions for the graduate writing course and asked whether I would be able to discuss my ideas with the current instructor. "We no longer have an instructor for that course. She has retired, and we are looking for a new teacher," he explained. He suggested that I contact the professor overseeing that course, who just happens to be my former boss (I was her research assistant) and my thesis chair.

She wholeheartedly agreed to assign me to the course, and although it means extra work for me this fall, it is an opportunity I do not want to pass up since it will be a valuable experience for me. And although I have never used a textbook for any of the ESL writing courses that I have taught, I figured I would try out the text that was recommended in the TESOL forum. I ordered the text for my class, and the publisher sent me a desk copy that I have been reading over the past couple of week.

I frequently tell my students (and even my professors) that I don't like to read, that I got through an undergraduate degree in English literature without much reading, and that I finished a Master's degree in Teaching English with even less reading, and that I have nearly completed my PhD coursework without so much as reading a single textbook (I skim a lot). But despite all that, I actually read this book (well at least most of it - I'm not much for reading boring examples). But I mention this to say that I honestly did read through this text and I feel fairly good about its potential as a tool for me and the graduate students in the writing course I will teach this fall.

THE REVIEW

Part of what attracted me to Academic Writing for Graduate Students (AWG) was its reputation as a strong language guide. There is a current trend among second language (L2) writing texts to focus on rhetoric and writing process just as first language (L1) texts so. I fully support the need for rhetorical and process instruction (if fact, if this university had a PhD in rhetoric, I would be studying that rather than IP&T); however, L2 learners need more than just the rhetoric and process learning; they also need to learn the language of writing.

Imagine my surprise then, when I received my copy of AWG and read, in the introduction, "The general approach [of this book] is analytical and rhetorical" (p. 2). I was shocked, dismayed, and even felt a little betrayed. Here I had been duped into buying a text that was no different than the other L2 writing texts that I had seen in the past. It was going to ignore all the language needs (grammar, vocabulary, socio-linguistic appropriateness, and more) of my students and only focus on the types of writing that they could learn from a regular graduate research writing course.

I nearly stopped reading right there, but I held on and figured that regardless of my confused expectations, it had to have worth if enough people had recommended it to me. As it turns out, I misinterpreted the introduction. AWG is organized around rhetorical functions (i.e., writing general-to-specific, describing tables, summarizing), but each of those sections is chock full of language mini-lessons that describe how L2 writers can appropriately use English language to accomplish those tasks.

It is a dense text and contains a lot of language information, but its organization lends it to practical use. In fact, there are three aspects of its language focus that I really appreciate.

1. Much of its language discussion is supported by corpus research. This means that rather than state that many research papers use one grammatical form, or a particular vocabulary phrase, based on the authors personal anecdotes or impressions, instead this text cites research studies that have studied those very language questions. Corpus research is a hot new thing in applied linguistics, but for good reason. Rather than guess when we make claims about what kinds of words and structures we should teach language learners, corpus linguistics allows us to compile information from numerous real situations in order to help inform our recommendations and choices. It is an approach that L2, as well as L1, writers should greatly appreciate.
2. When the authors of this text discuss language use, they recognize that different disciplines have different standards and trends. They acknowledge the room for diversity, and in many cases, they do their best to cite corpus research in a variety of fields and compare them. Because I will be teaching students from a variety of graduate programs (and I am only familiar with writing from the Humanities and Education fields), I need a source that can help me to guide students from other fields.
   
3. Every time a new structural or vocabulary language topic is raised, the authors encourage writers to consult actual texts from their fields and discover how experts in their major use language to accomplish the particular task (or even if that particular task is relevant to that field of study). As a new graduate student, that is how I learned to write - by reading. Again, I joke about how I never read a book through graduate school, but I read numerous research articles. I paid attention to organization, patterns, vocabulary, structures and language. I still continue to do this, especially since I am now studying in the field of educational psychology and not just applied linguistics. I beleive in this method, and I am happy to see that the authors of this text agree; it will make it much easier for me to convince the students to adopt this approach.

It may seem premature for me to offer an evaluation of AWG. After all, I have yet to use it in the classroom. I do not know how students will receive it, and I do not know how I will feel about using a textbook for a writing course. However, at this point I am mildly optimistic (which is a huge leap for a realistic like me). I will be sure to revisit this topic in four month's time in order to compare my expectations with experience.