It was noticed in the Rockman2 publication thread that new superb movies are much more likely to get to the top of the highest-rated statistics list because with the old rating system you only had the choice of giving 9 or 10 to a superb movie, and most people preferred reserving 10 for the really awesome ones, so they usually rated 9.
However, now that we have one extra decimal of accuracy, new superb movies are getting votes between 9 and 10, and thus in some cases maybe getting higher than they really deserve, compared to older superb movies, for the simple reason that people couldn't rate the older movies with a 9.8 or whatever.
So maybe it would be a good idea if people revised all of their old ratings, at least (and especially) those with a rating in the range 8-10? There's probably no big need to fine-tune ratings lower than that, but in that range some fine-tuning could be beneficial overall.
Btw, if you want to revise your ratings, please watch the movies again, especially if it has been several months (or years) since you last watched them. I don't think it would be a good idea to re-rate a movie based only on faint memories of it.
If you only revise your older 8-10 ratings, it shouldn't be such a pain to watch those movies again. After all, you thought they were great. :)
I find that the ratings will be skewed regardless of the numbers. So many people use 10s nowadays that one cannot tell a truly great movie from a meh-ish movie.
I don't think it far-fetched that some may rate movie X a 10 for both tech and entertainment, simply because they want the movie published.
I find that the few people who used to vote on the entertainment/tech for published movies, when we still used the yes/no method, are vastly outnumbered by non-critical watchers; some of whome may feel they can easily give out a 10 without truly considering what the rating means.
Long story short: Regardless of re-rating, the numbers are going to be misleading. That is why this system of rating fails. Horribly.
adelikat wrote:
I very much agree with this post.
Bobmario511 wrote:
Forget party hats, Christmas tree hats all the way man.
That is why this system of rating fails. Horribly.
I think that what you described was not a failure of the rating system per se, but a failure of allowing people to rate on the submission voting process.
I find that the ratings will be skewed regardless of the numbers. So many people use 10s nowadays that one cannot tell a truly great movie from a meh-ish movie.
I think that using the 7-9 scale here is pretty much a necessity. A movie that truly deserves below a 6 probably won't get published anyway, it's just the nature of the movies on this site and the viewers' expectations.
I don't rate movies, by the way. I've never understood exactly what my vote is supposed to imply.
That is why this system of rating fails. Horribly.
I think that what you described was not a failure of the rating system per se, but a failure of allowing people to rate on the submission voting process.
In other words, you're saying the choice of having people vote on the submission process was a mistake? That is, the rating system didn't work as well as it could have/as was intended? Or put another way, the rating system was a failure?
Something like that?
In fact, yes, now with that submission rating I feel like I'm voting No when giving a movie less than 6. While it's pretty average mark for published movies, at Workbench such movies tend to be unaccepted.
That is why this system of rating fails. Horribly.
I think that what you described was not a failure of the rating system per se, but a failure of allowing people to rate on the submission voting process.
In other words, you're saying the choice of having people vote on the submission process was a mistake? That is, the rating system didn't work as well as it could have/as was intended? Or put another way, the rating system was a failure?
I didn't say anything. I was just pointing out that if it is indeed so that people are rating 10's because they want a movie to be published, that's not a problem with the rating system, but a problem with the submission public voting system. If that is indeed the problem, the fixing it doesn't mean modifying the rating system. The rating system can be kept as it currently is (ie. a 100-value scale). Just the submission voting ought to be changed.
But as I said, I'm not saying there's something wrong the the submission voting.
Since everyone should bow down to my will, I figure I will give a possible solution to this problem.
I think the submission voting and the ratings on the main site should stay seperate; i.e. not carry over from workbench to published/front page.
This way, the oldies but goodies which had 6/7 "better-than-average/low-great" scores will not look like shit when compared to the recent surge of the 9.99984 "perfect" games.
I feel that said change would better attract viewers to some excellent older runs, that may not be rated as high as their newer counterparts.
adelikat wrote:
I very much agree with this post.
Bobmario511 wrote:
Forget party hats, Christmas tree hats all the way man.
I would argue that '0' means 'do not publish this' and anything above that means 'publish this strength X' but what do I know? In fact that would make more sense vis-a-vis 'terrible game choice'; you could have a reasonable Technical score and an Entertainment of 0 and a judge could still publish it because god knows entertainment score is fairly useless on a multicultural multibackground multiopinion site like this
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
LagDotCom wrote:
you could have a reasonable Technical score and an Entertainment of 0 and a judge could still publish it because god knows entertainment score is fairly useless on a multicultural multibackground multiopinion site like this
Have you not been paying attention for the past 6 months or something? I can think of several movies that I have rejected because of poor entertainment value, and several people started bitching about how every game was a special snow flake and deserved a run.
As for the rerating, I think it's a good idea, but very fewpeople will take the time to do it. I agree that the rating system is somewhat flawed for submissions, but I still think it was the best solution to come from here, given that the old system was axed by BoltR/Bisqwit.
<klmz> it reminds me of that people used to keep quoting adelikat's IRC statements in the old good days
<adelikat> no doubt
<adelikat> klmz, they still do
Well, some of them are (former) players, so that wouldn't really help. Besides, the difference in voting power isn't very pronounced, otherwise people would revolt.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Uh. I was arguing in favour of non-entertaining runs being published. Just marked differently. Sorry, I guess I was laying it on a little thick.
Personally I also agree with this opinion.
IMO submission voting should be there to catch submissions which are *flawed* and/or have too low quality (iow. they could be much better), and not to reject submissions simply because "watching the game is not "entertaining" enough".
All games are entertaining to someone. Who are we to say that some game is less entertaining than another? The voting and judging process should be about the quality of the run rather than about the game itself.
Joined: 11/18/2006
Posts: 2426
Location: Back where I belong
Warp wrote:
LagDotCom wrote:
Uh. I was arguing in favour of non-entertaining runs being published. Just marked differently. Sorry, I guess I was laying it on a little thick.
Personally I also agree with this opinion.
IMO submission voting should be there to catch submissions which are *flawed* and/or have too low quality (iow. they could be much better), and not to reject submissions simply because "watching the game is not "entertaining" enough".
All games are entertaining to someone. Who are we to say that some game is less entertaining than another? The voting and judging process should be about the quality of the run rather than about the game itself.
However, when you couple this opinion with several of the opinions from this thread (i.e. we shouldn't be too harsh on quality for fear of scaring people away), we basically turn this site into one which accepts anything. Not entertaining? Sure, accept it, because someone, somewhere may find it fun to watch. Not well done? Sure, accept it, because the person who made it may be scared away if we don't.
And thus, tasvideos.org turns into youtube.
I think there's a difference between "being harsh on quality" and "encouraging people to make higher-quality videos". The former is basically saying "Your video sucks, for these reasons. Fix it and maybe we'll consider it." The latter is "Hey, nice effort! I think if you did this and this, though, it'd be even better. Why don't you give that a shot?"
In other words, it's all a matter of wording, and being encouraging to newcomers.
Pyrel - an open-source rewrite of the Angband roguelike game in Python.
Joined: 10/27/2004
Posts: 1978
Location: Making an escape
Derakon wrote:
I think there's a difference between "being harsh on quality" and "encouraging people to make higher-quality videos". The former is basically saying "Your video sucks, for these reasons. Fix it and maybe we'll consider it." The latter is "Hey, nice effort! I think if you did this and this, though, it'd be even better. Why don't you give that a shot?"
In other words, it's all a matter of wording, and being encouraging to newcomers.
Last time I tried that, he left and never came back. >_>
A hundred years from now, they will gaze upon my work and marvel at my skills but never know my name. And that will be good enough for me.
However, when you couple this opinion with several of the opinions from this thread (i.e. we shouldn't be too harsh on quality for fear of scaring people away), we basically turn this site into one which accepts anything. Not entertaining? Sure, accept it, because someone, somewhere may find it fun to watch. Not well done? Sure, accept it, because the person who made it may be scared away if we don't.
Well, I disagree with publishing videos of subpar quality.
If newcomers are scared because of high standards, then IMO that's their problem, not the problem of the tasvideos site.
Recent ratings seem to be even much higher than normal, mostly caused by a few raters who constantly gives very high ratings to new submissions, while they never seem to be intereseted in published movies and thus their ratings aren't rised as high as newer movies'.
<klmz> it reminds me of that people used to keep quoting adelikat's IRC statements in the old good days
<adelikat> no doubt
<adelikat> klmz, they still do
Recent ratings seem to be even much higher than normal, mostly caused by a few raters who constantly gives very high ratings to new submissions, while they never seem to be intereseted in published movies and thus their ratings aren't rised as high as newer movies'.
That's the thing. You have to go out of your way to vote for older movies, when voting for workbench movies is easy as hell, and required to get a movie published on this site.
adelikat wrote:
I very much agree with this post.
Bobmario511 wrote:
Forget party hats, Christmas tree hats all the way man.
