Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
From how feedback is worded in posts and judge's comments.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Perhaps you linked to the wrong post, because it doesn't answer my question in any way.
Mothrayas wrote:
The guideline in question states votes are only one of multiple factors to take into account when judging tiers. It does not say "80% = instant moons". It gives it a decent case, of course, but it is not conclusive or guaranteed.
Mothrayas wrote:
The idea of the voting question (as it is currently) is to ask about entertainment. Optimization or publishability are determined by the judge and not by votes. The problem is that, regardless of what you ask of the viewers, they will vote their own things which will on occasion not align with the actual question being asked. That is something that cannot be changed (we tried, and basically failed). People still vote "yes for Vault" nearly three years after tiers have been introduced and the voting question changed accordingly.
This also goes for movie ratings - no matter how exactly we want to define 'technical' rating etc., people will invent their own qualifiers and vote according to their own ideas. Even for entertainment rating, which everyone can get at least a decent idea of, their metrics can still be all over the place. Anyhow, there is another topic about this.
I don't see anything here that would answer the question of what's wrong with my suggestion of making the poll question to be:
Rate this run:
[ ] 0, [ ] 1, [ ] 2, [ ] 3, [ ] 4, [ ] 5
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Why did you completely skip the IRC quote?
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Joined: 4/17/2010
Posts: 11475
Location: Lake Chargoggagoggmanchauggagoggchaubunagungamaugg
Thing is, people like to misuse everything, and for any poll you present to them they will vote their own idea of it, as Moth describes in the post I linked. Just like they skew it with 3 options, they will skew it with 6. The former experiment showed that they eagerly tend to vote maximum just because they want it to be published, and it wouldn't correspond to how movies are rated at all. It's human nature and it can't be fought. I think what we have is the least of all evils.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
I don't think it's such a huge problem as you make it sound like.
Some people will tick that 5 if they want the run published on Moons, but I do believe that they are the minority of people who vote in these submission polls. I think the rest can be safely given the option to express their opinion with more nuance than just "yes/no/meh".
I think it would be valuable information for the judges. If a run has only 4's and 5's, it ought to be a clear indication that it's very well liked. If the majority of the votes are 2's and 3's, with perhaps a couple of outlying 5's, it may be an indication that this is a Vault publication.
I honestly don't see a problem with my suggestion.
I honestly don't see a problem with my suggestion.
Frankly I see your suggestion as less clear than the current situation, so I don't see how that would help.
Can you elaborate on that?
5-star rating systems are literally everywhere. You would need to be a complete hermit to have never encountered such a thing. If you encounter this:
Rate this run:
[ ] 0 [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5
what exactly is there to be confused about? Tick one of the boxes, and that's it. What's unclear about that?
what exactly is there to be confused about? Tick one of the boxes, and that's it. What's unclear about that?
What is unclear is how your unusual scale relates to the movie ratings (it doesn't) or to the decision of whether to publish and in which tier (it does, but less intuitively than the current system).
As someone who's historically had trouble figuring out how to rate things, there is a big problem with a 5-star rating system: It's arbitrary. Unless you specify and explain benchmarks and criteria to earn each of the stars, each individual rater is basically deciding on their own and independently what having a certain number of stars means.
A second problem is strategic voting. Say you only kind of like a new TAS, you'd rate it a 4, but you really really want it to make it to moons. So, you rate is 5! You can't disincentivize strategic voting since literally anyone can vote and you can't meaningfully account them for it, therefore you can't expect people to ever vote anything but 0 or 5.
In short, you're trying to solve the problem of 'the workbench poll isn't clear enough and people use it to vote strategically' by exacerbrating both problems.
As someone who's historically had trouble figuring out how to rate things, there is a big problem with a 5-star rating system: It's arbitrary.
So? All opinions are arbitrary. "Yes/no/meh" is arbitrary. Whether to publish something on Vault or Moons is ultimately arbitrary and up to opinion. You cannot remove arbitrariness from this. Arguing that it's "arbitrary" is like arguing that "it's on a web page". Of course it is. So what?
Unless you specify and explain benchmarks and criteria to earn each of the stars
You don't need to specify anything. It's a popularity vote. It's asking for people's opinion on how much they liked it. You cannot quantify that mathematically.
A second problem is strategic voting.
That can never be solved, and only few people engage in it. It doesn't make much sense to worry so much about it.
In short, you're trying to solve the problem of 'the workbench poll isn't clear enough and people use it to vote strategically' by exacerbrating both problems.
That's not the problem. Read the original post.
DrD2k9 wrote:
It has been brought up a few times recently (roughly in the last year or so) that the workbench entertainment polls can be (and in some cases are) essentially ignored by a judge when determining publication tier, especially if explanation/content in the thread itself is contrary to the poll results. There are even times that the poll may be essentially ignored simply because there is little to no discussion in the forum at all.
[...]
It is my understanding that the poll is treated this way, at least in part, due to the ambiguity of the results; this ambiguity stemming from the fact that some users answer the question regarding degree of entertainment, while others answer the question as if the movie in question should be published or not.
My suggestion is exactly a proposal for a solution to that. It makes it clear that this is a popularity vote, and gives nuance to the degree of likeability that can be expressed in the poll itself even without an additional post to the thread.
Whether to publish something on Vault or Moons is ultimately arbitrary and up to opinion. You cannot remove arbitrariness from this.
I think this is why nothing ever happens when discussions about this come up. It doesn't really matter what the poll asks or how many ways there are to answer. This is a system that runs on inertia, not utility. The perception of the poll will not change even if it's content changes.
It is my understanding that the poll is treated this way, at least in part, due to the ambiguity of the results; this ambiguity stemming from the fact that some users answer the question regarding degree of entertainment, while others answer the question as if the movie in question should be published or not.
My suggestion is exactly a proposal for a solution to that. It makes it clear that this is a popularity vote, and gives nuance to the degree of likeability that can be expressed in the poll itself even without an additional post to the thread.
To clarify what I mean: The current "yes/no" poll question may give people the wrong impression that this is a "should this be published?" poll. (It really doesn't matter that the poll question is "did you like this run?" People don't really read that, and instead read a "yes/no" answer and draw conclusions from that.)
If, however, the poll is "Rate this run", with a grade of 0 to 5, it becomes extremely clear that this isn't asking whether to publish the run or not. There is no binary option potentially causing confusion to the user as to what exactly it means. It becomes much clearer that the poll is asking how much they liked the run.
And I'm suggesting 0 to 5 because IMO it gives enough nuance for most people in expressing how much they liked it, but not too many possible values as to become overwhelming and meaningless. 0 to 5 is, IMO, in that sweet spot between too few and too many.
If, however, the poll is "Rate this run", with a grade of 0 to 5, it becomes extremely clear that this isn't asking whether to publish the run or not.
Why would that be the case? It's pretty easy to jump to the conclusion that 0-2 means don't publish, and 3-5 means do publish. Just like how e.g. a college test might map 0%-59% to "you fail" and 60%-100% to "you pass".
If, however, the poll is "Rate this run", with a grade of 0 to 5, it becomes extremely clear that this isn't asking whether to publish the run or not.
Why would that be the case? It's pretty easy to jump to the conclusion that 0-2 means don't publish, and 3-5 means do publish. Just like how e.g. a college test might map 0%-59% to "you fail" and 60%-100% to "you pass".
I honestly think you are reaching now.
Why would anybody think "0-2 means don't publish"?
If you are so worried about it, use literal star images for the poll answer (with zero stars being a possibility).
Emulator Coder, Site Developer, Site Owner, Expert player
(3573)
Joined: 11/3/2004
Posts: 4754
Location: Tennessee
Based on our failed attempt at replacing yes/no with the rating system, I think he might not be reaching enough.
Back then, ratings were hugely over-inflated if they were done pre-publication. It was clear people considered anything less than 10 (or 9 maybe) as hurting a movie's chance for publication.
I think a similar thing would happen in a 1-5 scale. 4-5 = published, anything less is a somehow a No.
I think warp is using logic and sensible reasoning. That won't be reflective of how it will end up being used.
Calling the 5 point system confusing though is silly. We all know a 5 star review system. It's a reasonable idea. I think though, we have historical evidence that suggests it won't be better than what we have now.
I think though, we have historical evidence that suggests it won't be better than what we have now.
How much work would it be to at least try it for a while and see how it pans out? If it turns out that everybody's just giving 5 stars to everything, I'll admit I was wrong.
Well, I'm all in favor of temporarily changing the forum to test things; but then we should really be testing DrD2k9's proposal that started the thread, rather than Warp's; because the former is more comprehensive.
Well, I'm all in favor of temporarily changing the forum to test things; but then we should really be testing DrD2k9's proposal that started the thread, rather than Warp's; because the former is more comprehensive.
Isn't his proposal to simply change the text of the answers?
I don't really know how you would measure the "success" of that test.
Well, I'm all in favor of temporarily changing the forum to test things; but then we should really be testing DrD2k9's proposal that started the thread, rather than Warp's; because the former is more comprehensive.
Isn't his proposal to simply change the text of the answers?
Isn't his proposal to simply change the text of the answers?
No; see the first post of this thread.
I did.
DrD2k9 wrote:
Implementation (At least for the second suggestion)
If this would indeed be an approach desirable to the community...How difficult would it be to implement this on the site?
I recognize that the coding for the workbench polls would need changed. The submission list page shouldn't need much change as it could maintain the vote percentages (using much the same calculation as is currently utilized) by the following variable conversions:
(old question answers) = (new question answers)
Yes = Moons
No = Vault
Meh = I Don't Care
This would still offer 3 entertainment level choices in the poll while yielding a general % entertainment value that would theoretically be more accurate than results generated by the current polling method.
If that suggested solution is not simply changing the text of the answers, what exactly am I missing here?
