Marking essay questions is a subjective practice. Two different people can see the exact same answer and give two different grades. In fact, the same person can see the exact answer a week apart and give a different grade. I don’t blame the ones who grade, but CFA Institute for allowing this practice to continue. The reasons to put L2 all MC must also apply for L3. No?
The whole CFA process has been devalued enough. If you can’t please the grader with your written comments, you should not pass L3. QED.
TheEconomist Wrote: ------------------------------------------------------- > The whole CFA process has been devalued enough. would you mind elaborating.
I work with educated and qualified people from around the world. They refer to MCQ’s as the “American system” and consider it a joke.
CFAI executives in charge of the program have published presentations stating that the essay questions have NO more meaningful ability to differentiate passing candidates from failing candidates than the multiple guess section. Other than graders enjoying each others company at the annual grading conference it’s hard to understand why the essay section endures.
BS, lig, and CFAI executives. Multiple choice is much easier to game than essay. You must really know the material to answer essay questions. And frankly, they’re not that bad overall. Yeah, some bad questions, or subjective stuff, but that doesn’t explain a no pass grade. I, as two time failure at LIII, agree with TheEconomist. If I had prepared correctly, would have passed. Dumbing down won’t help.
UAECFA Wrote: ------------------------------------------------------- > I work with educated and qualified people from > around the world. They refer to MCQ’s as the > “American system” and consider it a joke. Obviously none of them had to take the GRE.
former trader Wrote: ------------------------------------------------------- > Marking essay questions is a subjective practice. > Two different people can see the exact same answer > and give two different grades. In fact, the same > person can see the exact answer a week apart and > give a different grade. I don’t blame the ones > who grade, but CFA Institute for allowing this > practice to continue. > > The reasons to put L2 all MC must also apply for > L3. No? Check out my posts on this grading process. It is not nearly as subjective as you think and the reliability is very good. In fact, the inter and intra grader reliabilities are checked and tallied and re-tallied and scrutinized all over the place. Among other things the variability from the grading process is insignificant in the variability of the exam scores and insignificant compared to things like the test-retest validity of the test itself.
UAECFA Wrote: ------------------------------------------------------- > I work with educated and qualified people from > around the world. They refer to MCQ’s as the > “American system” and consider it a joke. Nothing like all the world’s people refering to the American educational system as a joke and then pouring in to our Universities.
lig Wrote: ------------------------------------------------------- > CFAI executives in charge of the program have > published presentations stating that the essay > questions have NO more meaningful ability to > differentiate passing candidates from failing > candidates than the multiple guess section. > huh? What does that mean? Passing and failing candidates are directly determined by those things. > > Other than graders enjoying each others company at > the annual grading conference it’s hard to > understand why the essay section endures. > Most graders feel like they are doing a service to the industry and the CFA designation by grading. Each grading room comtaining 20-30 people is whisper quiet so there isn’t a ton of socializing going on during the full 8 hr grading days.
jdv, iow, CFAI has done statistical analysis on the essay vs. multiple guess section. neither section is better than the other at differentiating passing vs failing candidates. Statistically, they are equally valuable at determining who passes and who fails. Some candidates think they pass or fail because of weak essay sections. But, in aggregate, neither the essay or multiple guess section is the cause of more failures than the other. since this is the factual case, as stated by a CFAI executive, I don’t know why they continue with essay questions at all considering the extra time and cost. one possibility is they just want a writing sample from candidates to help ensure general aptitude. they may also just want to see how candidates react to an essay question vs mg. the real world of analysis tends to throw curve balls - they probably want the test to replicate the real world to some extent.
I’ve noticed some people refer to L2 style questions as “Multiple Choice” and others call it “Multiple Guess.” I suppose that indicates what people think they are doing for the answer. Lig, do you have a citation for what you’ve said? I’d be interested in reading it. As for the essays, just because grading an essay may be less precise than grading a multiple choice/guess exam doesn’t mean that it is “purely subjective.” It is true that figuring out whether someone should score 95/100 points or just 92/100 points may be a subjective judgement call and depend on whether someone got to your essay after a fight with their spouse or a bad night of sleep, but I’ve graded essays in other contexts before it is is usually pretty clear to everyone as to who put together an A quality argument with clear understanding of the concepts and who has put together a C quality argument with a bunch of buzzwords strung together in the hopes of getting a few points. Maybe you’re not sure if something is an A or an A-, but you know it isn’t a B or C. It helps to have a rubric for what things are required in responses at different score levels, and I’m pretty sure the CFAI gives their graders one to use. In the essay portion, you are not required to get 100% to pass, so the fact that there are subjective factors is unlikely to affect any but very borderline cases, and borderline cases are exactly those - the ones that might need to demonstrate a stronger grasp of the material in any case. Also, I believe each essay is scored by at least two readers, and I think I heard somewhere that if the readers come up with dramatically different scores, it is read by a third. This reduces a good deal of the subjectivity (random factors averaging each other out, as with diversified portfolios) The essay section might give a slight benefit to native english speakers, but I can tell from experience that it is usually pretty easy to figure out if difficulties in understanding come from someone who is a non-native writer vs. someone who doesn’t have their thoughts connected logically. Borderline native speakers might have a small advantage here, but that makes CFA say even more (positive stuff) about non-native speakers who have the charter. Personally, given how much effort is required to grade an essay, I’m amazed that the CFAI still uses them. But I think at the final stage of the exam, it’s a good idea to figure out if someone can communicate the concepts in the curriculum and apply them to a situation, rather than simply game the questions by eliminating a few multiple choice/guess answers. I think it’s great that they have an essay section still - though I’m a bit nervous about it myself.
UAECFA Wrote: ------------------------------------------------------- > I work with educated and qualified people from > around the world. They refer to MCQ’s as the > “American system” and consider it a joke. and we care why?
oskigo Wrote: ------------------------------------------------------- > UAECFA Wrote: > -------------------------------------------------- > ----- > > I work with educated and qualified people from > > around the world. They refer to MCQ’s as the > > “American system” and consider it a joke. > > > and we care why? I never said you should care. The thread discusses making L3 all multiple choice and I offered my colleagues perspective. I don’t necessarily agree with them. I was educated in both Canada and the U.S. and proud of my education. They just feel MCQ is inferior to the European style of short/essay answers as well as a tradition of oral examinations even in the subjects of math, physics, and science.
lig Wrote: ------------------------------------------------------- > CFAI executives in charge of the program have > published presentations stating that the essay > questions have NO more meaningful ability to > differentiate passing candidates from failing > candidates than the multiple guess section. > > Other than graders enjoying each others company at > the annual grading conference it’s hard to > understand why the essay section endures. I’m just starting to study LIII, so I’m no expert on this, but doesn’t the CFAI use essays almost exclusively to test IPS and some other PM topics and use the MC questions to test the other sections? It seems like there is minimal intersection between subjects typically tested through MC and subjects typically tested through essays, so you wouldn’t expect essay scores or MC scores to predict Pass/Fail. Wouldn’t a claim like that be like saying that your Econ score is an accurate predictor of whether you pass or fail?
yes the essay section is so much tougher on non-native speakers like me but then global markets are handled in English and that is how it is, no exceptions made if you are afraid of the English bit then just take a local securities exam in your home country and native language, nobody forces you to go down the CFA route
I fully agree, it should all be MC. Subjective grading on essay answers has inevitable bias. For example, graders can be biased against candidates from certain backgrounds, I dont understand why candidate names are written on the answer book. Couldnt they just do with the candidate ID there? There can be other situations where bias can come in. The truth is that the whole process is very vague… no matter how many explanations people give here the exam grading process needs improvement, particularly morning session of L3.
Anybody who passed LI & LII have been able to read a zillion pages in English, hence the essay questions should not be a big deal. However, are the minimum lack of grammar consistency or the few thypos a non-native can make taken into account while grading? I think I am going to write down just bullet points because of this issue along with time pressure.
eastwest Wrote: ------------------------------------------------------- > I fully agree, it should all be MC. Subjective > grading on essay answers has inevitable bias. For > example, graders can be biased against candidates > from certain backgrounds, I dont understand why > candidate names are written on the answer book. > Couldnt they just do with the candidate ID there? > There can be other situations where bias can come > in. The truth is that the whole process is very > vague… no matter how many explanations people > give here the exam grading process needs > improvement, particularly morning session of L3. It seems that I keep posting on this but just the idea that there is human grading of these exams is offensive to some people. The grading is done with astounding care and consistency. There is no chance that graders are “biased” against some ethnic group and unless things have recently been changed there are no names on exam books (although its very easy to determine which sets of exam books come from foreign countries - e.g., Chinese write English in strokes as they write Chinese). I wish CFAI would do a little more PR on the process. There is no exam at any university in the world graded with as much care as those exams.
AAA Wrote: ------------------------------------------------------- > Anybody who passed LI & LII have been able to read > a zillion pages in English, hence the essay > questions should not be a big deal. > > However, are the minimum lack of grammar > consistency or the few thypos a non-native can > make taken into account while grading? I think I > am going to write down just bullet points because > of this issue along with time pressure. Graders don’t care about grammar at all as long as it’s not so opaque that you can’t tell what the person is saying.