Vocabulary Measures

Because I plan to have some type of linguistic measurement as part of my validation study, I have been reading about types of language statistics. The most recent is a study by Batia Laufer and Paul Nation.

Laufer, B., & Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16, 307-322.

I am not familiar with Laufer, but I most certainly have heard about Nation. He is the man behind the Academic Word List (AWL), and list of commonly used words in university writing. We use the AWL at our center to help our students improve their reading and writing of academic texts.

This study appears to be pre-AWL since Laufer and Nation make no reference to it, and instead refer to the general service list and other university word lists. I imagine that the AWL was created not long after this study, since the inklings of the AWL are emanating from this work.

The primary focus of the study is an argument in favor of a new kind of vocabulary measure: the Lexical Frequency Profile. Although I don't exactly buy the LFP bid (the explanation of its use was verbose and confusing), I did appreciate the succinct discussion of existing measures of vocabulary use. In a matter of about 2 pages, Laufer and Nation clearly explain the pros and cons to several commonly used lexical measures. Even if I didn't get much out of the rest of the paper, I enjoyed their assessment of these measures.

The question is raises for me is: what linguistic measures will I use in my study? Almost certainly I would like to have at least one lexical measure, and hopefully this study will help me to explain and justify my decision.

Teacher Verification of iBT

While in line for tickets for the university's production of Little Women the Musical (we're going on my birthday for a pre-Thanksgiving holiday warm-up), I finished another integrated tasks evaluation study:

Cumming, A., Grant, L., Mulcahy-Ernt, P., & Powers, D. (2004). A teacher verfication study of speaking and writing prototype tasks for a new TOEFL. Language Testing, 21, 107-145.

You may notice that this is an article written by Alister Cumming who also wrote the iBT integrated tasks textual analysis article that I also read. I also referred to Cumming's other writing test related work for my MA thesis as well as my rater decision-making study (which I need to finish up and then develop into an article). Cumming seems like a very busy man, but he has been kind enough to respond to a couple of emails that I have sent him about his research and about the UToronto program.

Summary: This article attempts to provide content, context, and concurrent-related validity evidence in regards to the speaking and writing tasks for the new TOEFL (which is now known as iBT). As with the previous studies that I have reviewed, this one focuses on both integrated tasks (combining writing/speaking with listening or reading) and independent ones (in which examinees use their personal experience or opinions to complete the speaking/writing tasks).

Whereas the previous articles focused on the language content (Cumming, 2005) and the scoring procedures (Lee, 2006), this one focuses on how teachers of ESL students feel about the structure of the iBT tasks and their students' performance on these prototype items.

Primary questions are:

• Is the content domain of integrated tasks perceived to correspond to the demands of academic English requirements of university study?
• Is the performance of examinees on these tasks perceived to be consistent with their classroom performance?
• Are the tasks perceived to be adequate evidence for making decisions about examinee language ability?

The researchers found that the teachers felt that these tasks were fairly authentic and represented a variety of language skills required at university (so far as a test is able to recreate authentic situations). There were some concerns that the tasks were not fair for lower ability students who performed poorly on integrated tasks if they did not understand the input material; however, for the most part, teachers felt that student performance on the tasks were indicative of classroom ability. The greatest concerns that raters had with the evidence claims of the tasks were that some students may do poorly if they feel uncomfortable sharing personal opinion (for the independent tasks) or if they struggle with the stimulus content (for the integrated tasks). Cumming et al. conclude with some suggestions for improving the task format and content. They also suggest that students would benefit from exam preparation in order to understand the rhetorical and educational aims of the exam. Lastly, they suggest that more research be done into standardized ratings and on teacher perceptions of students' ability in the classroom versus an exam.

Critique: It's a shame that this article it stuck in the middle between an extensive evaluation (involving a variety of teachers and locations) and an intimate study (focusing on one or two individuals and their unique situation). As a result, we have neither the generalizablity of a large statistical study nor the valuable insight into a specific context. Instead we end up with a bunch of incomplete thoughts and ideas. It's as if Cumming et al. scratch the surface of several interesting ideas, but never uncover a single one. Still, for all its limitations, this study provided me with a good model for some qualitative research (including a survey that I adapted) and it is also likely to help me form some of my content and context validity questions. I also decided, as I read its account of teacher-based evaluation, that a student-based evaluation would be a great compliment to such a study. We'll see if I can make it work, or if I will have bitten off more than I can chew.

Connections: This study served to show me that I need to read a lot more about teacher evaluation study about tests. Was this a good example? How have others approached this avenue of validation? I did enjoy seeing how this person-focused (as opposed to text-focused or score-focused) study complimented other, more quantitative, studies on the iBT in order to give a more complete and well-rounded assessment of the exam.

Additional Reflections:

• I would really like to do an evaluation of the integrated writing tasks at the ELC that incorporates myriad sources of validty evidence including teacher-evaluation, student-evaluation, textual analysis, score-analysis, and possibly more. Am I crazy to want to do this much? Will TREC approve such a study? Will I find a dissertation committee who will approve such a multi-faceted approach?
• Who will I choose as teachers for my teacher evaluation? I think that I would prefer to use all of them (provided that they agree to participate in the study). I don't want to exclude someone because their reasons for not being eager to volunteer may actually be connected to their feelings about the exam and are therefore a real concern that I need to take into account.
  
• Will I have time to do this for speaking as well as writing? My tentative schedule, which I need to present to potential committee members next week, outlines an evaluation of writing for winter 2008 and speaking in summer 2008. I think I can be ready, but the issue is as much about analyzing data as it is about collecting it. Who will I get to help, especially when I am so busy around exam time with administrative duties?
• What kind of concurrent validity claims might I make? Will I ask teachers to rate their students and then compare performance with expectations, or will I use data that we already collect (classroom scores or rated evaluations)? Are teachers very good at predicting student ability? Lee's 2005 thesis of the L/S tests at the ELC say no (as did major portions of this study), but I think that both of those are problematic: Lee because the teacher ratings were probably based on Speaking when she was comparing those to Listening scores, and in Cumming's case, he admits that BICS/CALP may have a lot to do with false impressions. Of course BICS/CALP may be an issue for me too, but if students and teachers practice items in the classroom more than once, then teachers ought to be good at predicting success, shouldn't they?
  
• How will tasks be adjusted for lower levels (adhering the concerns of Sara in this study)? Laura (our Level 1 teacher) has already told me that she thinks Level 1 students need to be able to hear the integrated listening passage twice. As it is, they barely understand it before it's over. A second listening for Levels 1 and 2 might be appropriate and justified (given the findings of this study and Laura's experience).

Integrated Writing Task

My plans for a dissertation topic revolve around the use of integrated writing tasks. These exam items, used in ESL assessment, require students to write about a reading and listening passage. We are now piloting the use of integrated writing tasks at our center.

Reasons Why We Want to Implement the Integrated Task

• Every semester we inevitably deal with cases of unintentional plagiarism due to students whose summary, paraphrase, and quoting skills are weak
• Most of our students are planning to attend university in the USA and need to learn English for Academic Purposes (EAP) which involves writing from sources
• Integrated writing tasks are now part of the Test of English as a Foreign Language (TOEFL) which many of our students want to take and "pass"

Differences Between the TOEFL's and Our Integrated Tasks

• Our integrated tasks are used in connection with our Level Achievement Tests in order to assess students' readiness to move on the next class level whereas the TOEFL is used to assess readiness for university study which our program only assesses in an indirect form through level advancement
• Our implementation of the integrated tasks will assess student ability from novice to low advanced, namely our Levels 1 through 5; the TOEFL is designed to assess a narrower band of language ability, namely the ability of students in our Levels 4 and 5
  
• The TOEFL tasks are rated individually, but the integrated tasks will likely become part of our holistic portfolio score, or maybe even a subset portfolio score for fluency (non-draft) writing

In order to properly implement these tasks for our own center, I'm conducting a literature review of relevant studies. The first:

• Cumming, A., Kantor, R., Baba, K., Erdosy, U., Eouanzoui, K, & James, M. (2005). Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing, 10, 5-43.
  
• This study analyzed a variety of textual features of prototype integrated tasks versus the current independent writing tasks. It suggests that integrated tasks encourage the use of a greater variety of vocabulary and greater use of source ideas. The independent tasks are better at eliciting more language and at developing better arguments from examinees.
  
• I hope to be able to use this study as a model for the quantitative analysis of my dissertation research. Like the Cumming et al. study, I will examine a variety of discourse features and how they differ between responses to our independent and integrated tasks; I will also plan to look at these language features as they differ across our 5 proficiency levels.
  

I'll add more studies as I read them.

Does Guilt Fund Research?

Carbon credits - I first heard about them during a CBC Radio 3 podcast interview with the lead singer of Mother Mother. Host Grant Lawrence asked her if she knew what they were (coming from the environmentalist Islands of the BC Juan de Fuca Strait, you'd think that she would have), but she didn't. I had a good guess in mind, and later in the podcast, Grant explained how they work.

Carbon credits are intangible "vouchers" that business or individuals (such as independent musicians in the case of CBC Radio 3) can buy to help off-set the "carbon footprint" that their actions leave on the environment. In other words, if a rock band is going to drive their van from Ottawa to Victoria on a cross-country tour, and feels bad about the pollution that their travel will cause on the Canadian wilderness, then the band can buy a certain number of carbon credits to help compensate for any adverse effects on the environment.

I listened to that podcast a few weeks ago and hadn't heard anything more about carbon credits until today. This afternoon I was reading news headlines on CBC.ca and I came across Anne of Green Gables Goes Green. The article explained that the producer of a new PEI production of an Anne and Gilbert musical has purchased carbon credits to offset the environmental impact of their production (gasoline travel, stage lighting, or whatever).

The article further explained that the carbon credits were purchased from a Montreal non-profit that aims to reduce greenhouse emissions. And that made me wonder. What is this non-profit? And what do they doing to reduce their carbon footprint? Who died and made them environmental king? If it was the Government of Canada, then I can see some sense of authority, but independent firms selling carbon credits seems a bit like selling real estate on the moon or vending certificates of ownership for distant stars.

And then it gets worse. The lead actress explains her two-hour car commute every day by saying that the play's contribution to the non-profit "makes [her] feel better about [her]self." Um, doesn't she mean that the play's contribution is a responsible or wise decision? Or that it will have a significant impact on reducing pollution or raising public awareness? Nope. Her primary concern appears to be that this donation will make her feel better. Cause that's what the environmentalist movement is about: making people feel warm and fussy inside (and who cares about the human health concerns or long-term climate issues).

Is this where research funding comes from? Guilt? Is that how society advances: someone (i.e., special interest groups) makes us feel bad about something we are/aren't doing; we feel guilt; they inform us (entirely coincidentally) that researchers are currently looking into ways to reverse the negative impact that we have had on the world (and that they desperately need our help cough*money*cough); so we dish out funds in order to ease our conscience while maintaining the same practices as before; and researchers may or may not do anything useful with our donations (i.e., trip to Cancun to study something or other related to tropical resorts)?

It makes me wonder where my research money comes from. Will I be conducting research with guilt funds?

Carbon credits seemed like such a great idea when I first heard about it via Grant Lawrence, but now I'm suspicious and maybe even be already jaded. I'm such a sucker for celebrities.

Tight Fit

As a faculty member, I work on a 10-month contract. This means that, like a school teacher, I get two months off in the summer (or spring) from teaching. Most people would use this time to travel, relax, or just slack off. I, of course, am not doing any of those sensible things. Instead, I seemed to have managed to pack my time off with so many professional development opportunities that it will make my regular work feel like a vacation.

Load 1: Schoolwork
Since I was recently admitted to a PhD program, I decided to load up on the credits this spring term. The rationale was that the more credits I took now, the fewer credits I will need to take later on. In fact, by the end of this summer, it looks like I will have finished all the course work for my PhD. Also, even though I have tuition benefits through the university, I am normally limited to only sic credits a semester; however, because I am off during spring term, the benefits office agreed to pay for as many credit hours as I was willing to take. I called their bluff, and now I am taking double the normal credits that full-time student takes. It requires some time management, and efficient use of skim reading skills.

Load 2: Writing Seminar
Months ago I applied for a university week-long writing workshop. I hoped it would help me in my position as the Center's writing coordinator. My supervisor and dean nominated me for the opportunity, but since I never heard anything official back from the University Writing Office, I figured I never made the cut. Turns out, my official nomination form got lost in the "mail." I got a call from University Writing this morning while I was busy in the office doing homework. "We just had someone pull out at the last minute - are you still interested? And can you be here in 15 minutes?" Sure, it's going to be a little crazy this week balancing all day seminar participation with my full course schedule, but I'll make it work. Good thing two of my classes are evening courses. And the research grant can be a huge source of help for what we are trying to accomplish at the Center. This university keeps giving to me, and I am happy to keep giving back.

Load 3: Assessment Training
I also volunteered to become certified for the Center in oral proficiency assessment. It helps our center to have faculty members who are officially certified to conduct these sorts of assessments. Although the training will come at a difficult time (a week right in the middle of my classes and a family trip), it will work out. We'll change the family trip to a less crowded week and, as for classes, well, that's what the writing seminar week is for - practice for me to figure out how to balance a week of day-long meetings with full-time coursework. It will all work out. Besides, there's only five weeks left to the term. And then I head back to work, and the real vacation will begin.

How do people do this if they have children, busy church callings, or an supportive spouse? All the more reason to graduate before we have children. And all the more reason to stay teaching Primary. And... well, I married Kimberly, so I'm good for life.