Adding a decimal is basically allowing the voter to choose from 1 to 100, you know. Maybe it'd be less of an issue if we could rate something 0%-100%.
Anyway, moozooh, you're probably right about the difference between 8.7 and 8.8 not being significant, but the difference between 9.0 and 9.5 definitely is. I'd even say a 9.0 vs. a 9.3 is significantly different. And keep in mind, as you get closer to 10 ("perfection" presumably), the difference is more signifcant, so much so that 9.9 and 10.0 are still very different values. (Not that I'm sure anything should ever get a 10.0, but I'm also sure some people will vote 10.0).
Responding to Baxter's post from elsewhere. (Ha! Now try locating that post yourself!)
Baxter wrote:
It's indeed about significance... and I do think the difference between a 7.8 and 7.9 is significant. I don't see your problem, as you will still be able to vote in as big steps as you want.
Of course I will, but the main point is that it'll still instigate people towards making a different placement of movies (it will happen because it comes from human nature, and it will be imposible to counteract it), which totally defeats the purpose of assigning statistically counted indexes.
You know what would be the solution to your system? Replacing ratings with ranks. A list of n movies, where the top one is rank #1, and the bottom is rank #n. Then it all comes together perfectly, and what's more important, doesn't interfere with absolute systems. Because what your system is trying to do is bringing that relative component into what's indexed as absolute by the statistics engine.
Baxter wrote:
The reason why I think 0.1 is a significant difference, is since most of the movies tend to end up getting a rating between a 6 and a 9, while there still will be huge differences between them.
Now you're contradicting yourself. You say you don't need labels. Why do you not spread the marks evenly so that there's no huge spikes? Because you think 5 is way too low for most of the movies you rate? Maybe that is the problem?
Baxter wrote:
Rating for published movies, ratings below a 4 are really rare (I think... for most of the voters), so it's not really like you'll have 100 options.
Yes, that's pretty stupid. With nothing other than 0.1 decimals added to the current system, we'll see 2-3 10.0 movies, a dozen of 9.9, another dozen or two of 9.8… With nearly no decimals used below ~8.0, shifting the focus of the scale from 6-10 to 8-10. It's not that hard to predict because it almost directly protrudes from the currently observable rating inflation. EDIT: Also see Acheron86's post above, which can be used for illustrating this.
To conlude this discussion (I think I've already said everything I wanted to say on the matter), your proposition works well for a personal list (albeit in a different realization), but is worse for statistics than the currently used one.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
I personally would use a system divided by .5, but probably wouldn't find much use for any differentiation below that. I agree with moozooh that the degree of precision I'm able to give a rating simply doesn't need to be divided into tenths of a point. But perhaps some people do rate that precisely.
I'm one of those math nerds who would give something a 6.7 or a 4.5 or an 8.9 and feel like it was a more accurate assessment of my opinion than a single-digit number. That's not some quirk in human nature that only occurs because of the opportunity to change it (which seems to be what moozooh was concerned about), I already do this mentally when I vote.
While I'm on it, I think voting inflation doesn't really tie into the issue of decimals vs. no decimals. I'm convinced there's literally no cure to voting inflation, because the people voting are often anonymous viewers who will naturally feel like every TAS is a 9 or 10 because it's omg my favorite game being broken wowwsss. Actually, this is part of why I think a decimal or percentage system would be better, because we already know the votes will be top-heavy... why not just adjust our expectations accordingly? It hurts the mathematician in me, but I don't see any other solutions since they'd likely involve changing human nature. :|
Also, a snippet from our PRIVATE TREEHOUSE IRC channel. Some more food for thought.
<mmbossman> heh, you missed my arguement with Zurreco yesterday
<moozooh> i'm pretty sure i did.
<mmbossman> I was arguing that 5 is a good average, but only if you take into account all the submissions
<mmbossman> the gruefood would make up the lower half of the rating scale if they were rated
<mmbossman> (and it appears they will soon be)
<mmbossman> it just happens that most of our publications are "above average" submissions
<moozooh> well, that could work, and then 20 point scale would be justified.
<mmbossman> so they tend to get rated above 5
<moozooh> yes, exactly.
<moozooh> it's a problem rooted in site's positioning.
<moozooh> "we don't publish subpar movies"
<mmbossman> yeah
<moozooh> so technically pretty much all of them are above-par.
<moozooh> which cuts the scale in two from the beginning.
<mmbossman> yup
<moozooh> at least the technical.
<moozooh> leading to further inflation.
<moozooh> then there are people who rate only 10-15 movies, most of them with 8-10, contributing to the mess.
<mmbossman> yeah, those ones I can't argue with you against
<moozooh> there are dozens of them, more than the "normal" raters in fact.
<jimsfriend> well yeah, because they only rate movies they watch, and only watch movies they expect to like in advance
<moozooh> heh, yeah. that is understandable, but at the same time it's a huge blow to statistics.
<mmbossman> I've somewhat given up on trying to get great statistics
<moozooh> which makes it hard to tell if unified rating index was a good (or at least well-thought out) idea in the first place, if what most people need is in fact a personal list not influenced by others.
<mmbossman> but hopefully the new voting system will be workable
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
It's indeed about significance... and I do think the difference between a 7.8 and 7.9 is significant. I don't see your problem, as you will still be able to vote in as big steps as you want.
Of course I will, but the main point is that it'll still instigate people towards making a different placement of movies
I think this is currently fine. Each person rates at a different average, which is no problem, as long as people are rating consistently with their previous ratings. This is especially done if people will look at their list, and rate accordingly. I think this is a very good thing... not a bad thing.
So I wouldn't even say that people instigate towards different placement, I'd say with this, people will be able to rate consistently. Even if, by looking at their rating list, they come to the conclusion that they might want to switch a tenth or something... that's not nearly as much as someone who wants to give a 7.5 and needs to switch 5 tenths.
Joined: 5/25/2007
Posts: 399
Location: New England
My two cents:
1. I agree with having one number as a ranking for each movie; either decimal 0.0-10.0 or 0-100, whichever is easiest. For me personally, a game's "technical" rating has always been nebulous if I haven't TAS'd the game. How could anyone know how "close to perfect" it is? Just because there are no obvious mistakes? Runs with obvious mistakes don't really seem to get published, so I end up giving out a lot of 8's, 9's if I know the game and am really impressed. With one number, it's more 'how impressed was I watching this run,' and it's my little secret all the little components that go into that number.
2. This may have been suggested, but how about listing a user's previous ratings on the 'rate a movie' form. For users who've rated hundreds of movie's, a subset of movie's rated at various levels could be generated. It could even be driven by the tentative rating typed in, ie I put in 8.7 for the new LoZ 2nd quest, the following shows up below:
You rated "Awesome Run #1" 8.8
You rated "Lame Run #1", "Lame Run #2" 8.6
Rate "LoZ 2nd quest" 8.7? [submit]
This way your previous rating behavior is very transparent at the time you're rating without any effort or memory required.
You know what would be the solution to your system? Replacing ratings with ranks. A list of n movies, where the top one is rank #1, and the bottom is rank #n. Then it all comes together perfectly, and what's more important, doesn't interfere with absolute systems. Because what your system is trying to do is bringing that relative component into what's indexed as absolute by the statistics engine.
I mentioned in the other thread that lists of 'rankings' could generate overall ratings for all the movies.
I was just starting to work out decent formula, when I realised one serious drawback - when anyone rates one additional movie, all the ratings of anything they've ranked would change - I imagine that would be kinda a weighty server-load.
I'm just some random guy. Don't let my words get you riled - I have my opinions but they're only mine.
<mmbossman> the gruefood would make up the lower half of the rating scale if they were rated
Thats pretty much the way I've always viewed it. Anything which has a rating of 5 or below, probably shouldn't have been published in the first place. Looking through the ratings in general I find that people award 9's for technical and 8's far entertainment for too easilly.
For me personally, a game's "technical" rating has always been nebulous if I haven't TAS'd the game. How could anyone know how "close to perfect" it is?
I already wrote what I understand "technical rating" to mean in this thread (and also previously some years ago). Personally I don't think the technical rating is about how frame-optimal the movie is.
(Or, more precisely: Seeing clear mistakes in the movie, ie. the movie being clearly non-frame-optimal, should lower the technical rating. However, frame optimality should only be something which reduces, ie. penalizes the technical rating, not something which increases it. The actual score should be based on the other things I gave examples in my post about this subject.)
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Warp wrote:
I thought about the technical score to be an evaluation of the *techniques* used in the run. In other words, rather than being a pure optimal-frames/actual-frames-in-the-run score, it's an estimation of the quality and diversity of techniques used in the run. A bit like figure skating: The judges don't estimate how fast they are skating, but how well they perform their techniques (and how many there are).
Does it perform heavy luck manipulation? If so, does it do it to its great advantage? Is it "cool"? Does it zip through walls? Is the zipping performed with good style and technique? Does it "look good"? What kind of tools were used to make the run? Was lua scripting used to aid in making the run? Was a bot written to create part of the run? Was the game disassembled in order to understand how the rng works? That kind of things.
Even a frame-perfect run may deserve a low technical score if it doesn't show advanced and well-executed techniques. Perhaps the game in question just doesn't lend itself to awesome techniques, but then it's simply a poor game choice.
I have suggested this a couple of times in the past, but always shot down. For whatever reason I cannot comprehend, people don't want this. They want the technical score to be a pure optimal-frames/used-frames score, and nothing else. Interpreting it like that makes the whole technical score kind of moot and uninteresting. It has no value. It doesn't say anything.
I personally find your idea of technical rating being judged like ice skating to be one of the worse explanations given, due to the incredibly vague descriptions of criteria you use. Glitches like zipping and very game specific, and have become commonplace for the games that do utilize it.
Regarding technique used, I'm sure there are a large number of people who simply download the AVI and rate, without reading through the submission thread. How are they supposed to know that techniques like lua scripting and memory watch are used? Or if they disassembled part of the code? Many tools that used to be rare are commonly utilized today, such as RAM watching. Sure you can always make the point that "informed raters" should be reading through the submission thread, but often that information isn't even mentioned there. So then it's not only the raters fault, it's the authors.
Overall, I find very little I like about this view of "Technical" achievement. I may not be able to imagine a theoretically perfect run, but I have a much clearer idea of what the perfect run looks like than trying to imagine some of the things you've suggested.
(Or, more precisely: Seeing clear mistakes in the movie, ie. the movie being clearly non-frame-optimal, should lower the technical rating. However, frame optimality should only be something which reduces, ie. penalizes the technical rating, not something which increases it.
I don't see how this is any different from entertainment. Bear with me, but if you see a clear mistake, won't you already penalize its entertainment vote? And if you've TASed the game and you know something could've been better that others didn't know, wouldn't you still penalize the entertainment vote? I'd think any time someone noticed a mistake they'd feel less entertained.
Fine, let's just remove the technical rating since clearly people DON'T WANT any technical rating which has any rational meaning. They only want a technical rating which means "is it frame-optimal?", discarding any other possible subjective meaning, and will then complain that they can't know if it's frame optimal.
I really can't understand what's so detestable in the technical rating expressing the *subjective opinion* of the voter. What's so wrong with that? Why do people always have to complain about stupid things like this?
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
You're the one that has recently been complaining about how new suggestions should be listened to, and that discussion and argumentation are good things. So I pointed out my dislike of yours system. If that hurt your feelings, I apologize. But you can't have the site work both in favor of all discussion when you agree with it, and against discussion when you disagree with it.
EDIT: And I never said I wanted the technical scale to be eliminated. I think it serves a good purpose, and the fact that you have the opinion that my definition of technical isn't very good is fine with me.
EDIT2: For your reference:
Warp wrote:
We must be Politically Correct and avoid all constroversies and anything that might cause disagreement. This is, after all, the current PC trend here.
You're the one that has recently been complaining about how new suggestions should be listened to, and that discussion and argumentation are good things.
You know, you are rather good at twisting and reversing people's arguments against themselves (even if it doesn't work at all). That may be an effective online flamewar technique, but it's not really a good way to build amicable relationships.
You didn't present any new suggestion or new idea. You simply are shooting down my idea of what I consider should be a better definition of "technical rating". And, rather ingenuously, you are now reversing the whole situation and trying to make it look like *I* am the one who is opposing new ideas from you. Clever.
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Hmm... good timing with my edit to my above post. Here it is again:
Warp wrote:
We must be Politically Correct and avoid all constroversies and anything that might cause disagreement. This is, after all, the current PC trend here.
Obviously you were being sarcastic with the above quote, and think we need more conversation, and shouldn't be afraid of controversy or disagreement. I agree. Which is why I wrote why I disagree with your idea about technical ratings. If you disagree with me, that's fine too! I'm not trying to start a flamewar, I'm just was pointing out that your system makes very little sense to me. If you want to continue thinking in that particular way, feel free.
However, your reaction to my criticism makes me believe that you are very open to new discussion, as long as it's in agreement with how you think. And I simply think that's wrong.
EDIT: And I don't think I'm presenting new ideas, or new material, since I've always seen the "technical" rating as a way of saying how close to perfect I think the current movie is. And you apparently have been thinking your way for a long time also. So neither of us are presenting new material to the discussion.
EDIT: And I don't think I'm presenting new ideas, or new material, since I've always seen the "technical" rating as a way of saying how close to perfect I think the current movie is. And you apparently have been thinking your way for a long time also. So neither of us are presenting new material to the discussion.
I apologize if I took your earlier post unfairly harshly. It's just that when you started with "you're the one that has recently been complaining about how new suggestions should be listened to" it sounded like you were criticizing/attacking me using my own previous argument.
As a viewer, I enjoy watching movies, but usually hold off on voting, since being able to fairly judge the 'technical' score feels like I need to really think about things in a way I never do and - at the bare minimum - read and at least vaguely understand the author's explanation of techniques.
If there were only one rating to hand out, I'd probably rate a lot more stuff.
Maybe that's a problem with my way of thinking, but I'm sure other casual viewers are kinda the same.
Although maybe you actively want to discourage overly casual folk from voting, since I guess folk who only rate a few things skew averages.
I'm just some random guy. Don't let my words get you riled - I have my opinions but they're only mine.
If there were only one rating to hand out, I'd probably rate a lot more stuff.
Another idea which came to mind when I read that: What if voting on the technical rating was made optional? In other words, you could vote on the entertainment rating and leave the technical rating "blank". Basically it would mean "I have no opinion".
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Warp wrote:
Bezman wrote:
If there were only one rating to hand out, I'd probably rate a lot more stuff.
Another idea which came to mind when I read that: What if voting on the technical rating was made optional? In other words, you could vote on the entertainment rating and leave the technical rating "blank". Basically it would mean "I have no opinion".
It may cause some problems with the average score for each submission if a workaround isn't implemented, but I think this is a pretty good idea. But I don't think that the ratings should be reduced down to one category. I can see the possibility of having a technical category helping to initiate some more critical thinking when it comes for rating/voting that might not have been present before (for the voting system, at least). And having to have the viewer provide a little more thought into their vote could help to reduce the syndrome formerly known as "weak Yes-itis"
Another idea which came to mind when I read that: What if voting on the technical rating was made optional? In other words, you could vote on the entertainment rating and leave the technical rating "blank". Basically it would mean "I have no opinion".
There's some proactive thinking. I like this idea.
Maybe the description could read, "The technical quality of this movie (how technically impressive it is, techniques used, etc.)"
Warp wrote:
Bezman wrote:
What if voting on the technical rating was made optional?
That might encourage more rating, certainly.
I think - in my case though - I should maybe just kinda have confidence in my judgement more. Specially if going by Warp's definition, which I like, I often do have an opinion - I'm not sure if it's valid or not.
I changed my rating for 'Castle of Illusion' - whilst I like the game and kinda enjoyed watching it, nothing seemed that far off what might be humanly possible. So I gave it a 5.
Mario Land 2 just seemed really cool with the backwards-jumping showcasing, going through blocks, hitting enemies at the last possible moment with a fireball (took me a while to work out what was going on there) and managing to land on the sides of spikes. I gave it an 8.
I'm honestly not sure - is this way of rating movies OK? I have no TASing experience so whilst I imagine that frame-precision is easier than glitch-finding like in Mario 64 or the luck-management in King's Bounty, I can only imagine. So it's a lot easier for me to just base the rating on the techniques I can see, taking the guesswork and 2nd-hand information out of it.
I would tend to give puzzle games higher technical ratings than platformers, simply because the insanely fast speed of play - whilst maybe easy when using 'frame advance' - just seems more impressive and seems like the kind of 'broken-ness' I was hoping for when I first heard about this site.
I would sorta come from a viewpoint of 'how humanly IMPOSSIBLE does the run look?'. Thinking about it, that's probably the only way I COULD rate the 'technical' rating.
I guess if you only wanted ratings from TASers, you'd have hard-coded that into the site.
If my viewpoint is OK, maybe that could be part of the description - "How humanly IMPOSSIBLE does the run look?"
I'm just some random guy. Don't let my words get you riled - I have my opinions but they're only mine.
I'm honestly not sure - is this way of rating movies OK?
In my opinion there are no incorrect answers in the ratings, as long as you estimate the quality of the movie honestly and express your opinion. (Some people might use the rating system for malicious purposes, to try to undermine the system, and eg. always give 0 to everything or whatever, but those are exceptional cases.)
If you found a movie extremely boring and couldn't find anything enjoyable about it, then you should score lowly for entertainment. Some other person might enjoy the same movie immensely and give it a high entertainment rating. If both rate it honestly, both are correct. They are expressing their own opinion of the movie. I don't see the technical rating any different from this either.
