In the past weeks I’ve been working on an exciting little project for the June 2012 results - building on some concepts that have been explored by others in the past here on AnalystForum. At the same time I’m trying to unify all these past efforts to really get a proper, organized approach towards building a good understanding of the CFA results.
Objective of the project is to understand the CFA results as much as possible and try to answer some of the ‘unanswerable’ questions that have been posed, such as ‘what is the MPS this year’ and ‘what is the best 40/60/80 formula’ and so on. After experimenting with several formats I think I’ve iterated to a version that’s easy to particpate in and generates maximum results.
Candidates would use a simple results calculator that also acts as a survey - candidates that receive their results can use this calculator/form to obtain their 40/60/80 score and max/min score. If enough results are collected this can further be analysed to much higher levels of detail (described below).
Data required
Obviously all individual scores will be private and analysis is done by aggregate only. Questions planned for the form (trying to keep it to as few as possible):
- Email address
- Score breakdown (i.e. results matrix)
- Pass/fail + band
- Study materials used
Immediate results
As mentioned, immediately after submitting the form each respondent will be sent their:
- 40/60/80 score estimate
- Minimum score
- Maximum score
Further results
This is where I think it gets a bit exciting. After a period of time (i.e. after enough submissions have been collected), the analysis can be expanded further to include:
- A better approximation of the 40/60/80 formula for that particular exam (and thus a more detailed recalculation of everyone’s xx/xx/xx score
- Approximate idea of what the MPS is for that exam
- Overviews and average scores topic-by-topic
- If pass, your approximate percentile in the passing band
- Breakdown of study materials used
Help needed
In order to pull this off properly, I would be super grateful if anyone could chip in on any of the points below:
- Send me latest weightings on results day. Currently I have based the Level I and Level II weightings and topics based on last year’s results - my understanding is that this isn’t necessarily applicable to this year’s results. Would appreciate it if on 24th July, someone sends me the weightings directly to me so that I can quickly update the script if there are changes. I will definitely need someone’s help for Level III as the weightings are different every year.
- More participation. The analysis only works if there is significant participation - if you are supportive please do send the form link to anyone you know that’s taking the exam.
- Help figure out if anything big has been missed. Am not trying to go for a lengthy survey but tryingt o maximize the analysis results - comments appreciated.
What do you guys think - would you be keen to participate and/or help out?
Thanks, X
Great idea. But what do you think CFAI will say about this?
There isn’t anything in the Code of Conduct (as far as I’m aware of) to say that you can’t have a further think about the results - it’s the exam content that is strictly forbidden to be shared. Also, many similar efforts have been done in the past - this is an effort to get more organized.
Obviously this is all approximate calculations and cannot be relied as a basis to challenge results or retabulation. It’s just further insight for candidates’ personal use.
Studying With
I agree I don’t think the CoC has anything about calculating MPS or other exam data, but don’t you think the end reuslt (assmuming successful MPS calulation) would be a new CoC Handbook banning this type of activity. After all the Institute is notoriously secretive about the MPS. Their idea being no one year is better than another, whilst published MPS scores could lead to exactly that kind of analysis, particularly when you could break down MPS further to see which areas were harder each year.
A person needs new experiences. They jar something deep inside, allowing him to grow. Without change something sleeps inside us, and seldom awakens. The sleeper must awaken.
At the end of the day, the MPS results calculations are simply approximations upon approximations. I plan to disclaim the crap out of this - I obviously can’t make any claim on the accuracy since the level of detail provided in the exam results simply isn’t enough.
This will never be an authoritative source so MPS results would still be speculative. But if this is deemed too sensitive anyway, the MPS analysis doesn’t have to be done. It’s just something that people do ponder about frequently and it would be interesting to try to calculate one.
People have been posting their anonymous L3 matrix for years and years and I’m sure CFAI knows about it since they monitor these forums. If it were a problem chad probably would have got a notice and told people to stop
The most telling score bands would probably be the guy who’s score matrix looks horrendous and still passes. Or the guy who has a good looking matrix and fails.
Everyone between with mediocre bands that fail or pass would be too hard to tell, becauset the variance in those bands would be too wide to guess accurately.
Hope. It is the quintessential human delusion, simultaneously the source of your greatest strength, and greatest weakness.
Which reminds me, I’ve also reached out to Chad for his opinion as he would have more experience into these things.
Worst-case is we can run a more watered-down version and not delve into controversial stuff like MPS. Less fun though. :(
USJ2, people post their results online, this whole post is unneccesary. Someone did this for last year’s Level 3 and made an extensive spreadsheet. Also I’ve taken Level 2 three times now and the weighting and order of sections has always been identical fyi.
Hi MissCleo
I know of the Level III spreadsheet, however my point is to make the process easy - i.e. easy for candidates to calculate their 40/60/80/max/min scores, and easy to aggregate data. The person that did the analysis for Level III took several full days to carry out the analysis and I hope to make this bigger.
I’m not suggesting that everyone post their scores here for collection - planning on sharing a link nearer to results day where you can calculate your results online instantly. Hope that clarifies.
Studying With
I think it’s a good idea. I thought about doing something similar myself.
How are you going to keep people from polluting the results pool with multiple entries, fake results, errors, etc?
Studying With
Nice initiative
That’s one of the reasons I’m including email addresses - I would exclude results that go to invalid addresses, use it to figure out multiple / errorneous entries etc. Obviously not immune if someone intently wants to skew things for some reason, but should weed out 95% of the errors.
If that’s the case, why do they only give us a range of topic weightings instead of exact percentages?
past weightings is no guarantee of future weightings.
Hope. It is the quintessential human delusion, simultaneously the source of your greatest strength, and greatest weakness.
Studying With
I would seriously try to dissuade you against this endeavour…
I am pretty sure that this is in breach of some ethics rule in some manner. No sense in offending the Institute if you want to continue with the programme.
Thanks for the feedback guys. I will keep it to a calculator (i.e. 40/60/80, min & max) and we’ll see if we even have enough data to do any further analysis.
Studying With
Good wishes to all! am sure everyone shall get results according to what they deserve.
Twitter handle - @imdnextbuffet
BBPIN: 236FB59A
To f*&$ with your head of course. They have used the same weightings since 2004, which are all of the data points I have. For the next 10 year barring some monumental restructruing I GUARANTEE for L 2 you will have 4 FRA and Equity vignettes, 2 FI CF Derivs and Ethics, and 1 PM Alts Quant Econ