That is what I think after seeing this:
Look at the the individual ratings by those eight users and the average rating for entertainment. That means that Nach and Kyman gave a rating of less than 2.2 for entertainment, which is hard to understand. I would like to read an explanation from both. Nach, is this one of your methods to torture TASers? :P
AzumaK wrote: I swear my 1 year old daughter's favorite TASVideo is your R4MI run :3
xxNKxx wrote: ok thanks handsome feos :D
Help improving TASVideos!
Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria
I think we should be a bit more concerned that people are voting 10/10 whatsoever...
I'd have gone 9.9/9.8 myself, at least that provides some room for the future...
That means that Nach and Kyman gave a rating of less than 2.2 for entertainment, which is hard to understand. I would like to read an explanation from both.
Your votes:
The entertainment value of this movie: 7.2 . — Average rating: 8.0
The technical quality of this movie: 9.4 . — Average rating: 9.1
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
The site code doesn't weight all votes equally - the ratings of judges are weighted higher than that of users with small numbers of posts and no submissions, for example.
In that case you all should forget this topic...
Is that stated somewhere on the main site?
AzumaK wrote: I swear my 1 year old daughter's favorite TASVideo is your R4MI run :3
xxNKxx wrote: ok thanks handsome feos :D
Help improving TASVideos!
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
Zeupar: I find every run of a Sonic game utterly boring, with the exception of Sonic Advance 2. Hence I will give low ratings in entertainment for Sonic runs, especially the Genesis ones.
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
I've placed information on how the weighting of average rating works in the [wiki SystemMovieRatingGuidelines]System Movie Rating Guidelines[/wiki] page which is displayed as the footer for the rating page.
Does "users with published movies" include users who had published movies in the past, but all of them currently being obsoleted (iow. "former players")?
Emulator Coder, Site Developer, Site Owner, Expert player
(3573)
Joined: 11/3/2004
Posts: 4754
Location: Tennessee
The built in bias for "staff" and players is now removed. Everyone's votes are equal Except for rare exceptions like Neofix <_<
The built in weight was actually implemented by Bisqwit, but I have no idea when as it was never mentioned or documented. I found out about it after taking over the site. I personally would have never approved such a weight, thanks to grunt for taking the time to remove it from the site code.
Joined: 10/28/2007
Posts: 1360
Location: The dark horror in the back of your mind
Warp wrote:
Does "users with published movies" include users who had published movies in the past, but all of them currently being obsoleted (iow. "former players")?
Upon closer inspection, it's not exactly 'published movies' so much as 'has an entry in the players list'. In other words, former players are also counted.
The built in weight was actually implemented by Bisqwit, but I have no idea when as it was never mentioned or documented. I found out about it after taking over the site. I personally would have never approved such a weight, thanks to grunt for taking the time to remove it from the site code.
Very well.
I admit I was not particularly vocal about the rating weighting system, but I never kept it secret from anyone who asked. It's just that I was completely content with people never asking. I think I explained it a few times on the IRC channel; that is how you knew the bit about Neofix's special case, for instance. And of course, people who had to do work with it (such as Warp) had disclosure.
Basically, the weighting system existed since the beginning, that is in 2006 when the rating system was implemented.
To recap, here is how it worked when I handed things over:
To calculate authority coefficient, i.e. rating weight for the user:
Anyone with a player entry has a post_count_threshold of 9 and scale of 1.1. Anyone else has a post_count_threshold of 22 and a scale of 1.0.
Entrusted users (judges, publishers, admins, and any combination thereof) have the coefficient between 1.1 to 2.0, exact value whereof is lower or higher depending on the entrust level of the role.
For anyone else, the number of posts is calculated. The coefficient is a value between 0 to scale, directly proportional to postcount, that maxes out at post_count_threshold. It is always higher than 0, though.
Consequently, the highest coefficient was on admin users, who had 2.0. The lowest coefficient was on lurkers who had never posted or contributed in a movie, who had 1/22, that is 0.045. One had to get at least 22 posts to get a coefficient of 1.0. If the user has a player entry, their coefficient is always at least 1.1/9, that is, 0.12, even if they have never posted. If they've posted more than 8 posts, their coefficient is 1.1.
Ratings you provide now will retroactively grow more significant as your current coefficient grows. That means you don't need to refrain from rating until your post count has increased. Your already-cast ratings will retroactively benefit from your posting / publications.
Personal exceptions can be made. (Only one has been done so far, and it was a demoting type.)
Rationale:
The system of rating was similar to that of Eurovision song contest. In the Eurovision song contest, anyone can "televote" by sending a SMS/phonecall, but there is also a panel of judges (jury), consisting of selected professionals in the area of music and entertainment. The weight in 2010 contest was 50% televote and 50% for panel judges. Thus, the average juror had the same rating power as some 1000 to 100000 "regular" people combined.
On TASVideos, anyone with a player entry likely has a sense of entertainment matching the site's goals, since a movie they authored according to their own ideals, was published on the site. Therefore they can be said to belong into a "panel of judges" as far as rating is concerned. Similar rationale goes for entrusted contributors, but with stronger confidence.
The post count provides an additional confidence factor, and especially with the case of lurkers it shields against rating abuse. It is a high threshold for anyone to register 22 dummy accounts to match some seasoned member's vote.
Oh, and I am not a fan of pure democracy either. It is unbiblical. No dictatorship nor communism either, but the fact stands that "majority" is always stupid. In leadership positions you need people who have the right vision and wisdom. And they must never be outranked by a mob. Democracy is why Europe and USA are falling to islamization. Own dog bites.
The built in bias for "staff" and players is now removed. Everyone's votes are equal
So what current system is still there? There needs to be some kind of bias towards certain users. Compared to a newbie; an experienced player is more likely to have a better idea as to how entertaining and technically accurate a movie is. Purely because of the fact that they will have seen more movies and will have more hands on experience.
Joined: 12/27/2006
Posts: 532
Location: Göteborg, Sweden
Mitjitsu wrote:
adelikat wrote:
The built in bias for "staff" and players is now removed. Everyone's votes are equal
So what current system is still there? There needs to be some kind of bias towards certain users. Compared to a newbie; an experienced player is more likely to have a better idea as to how entertaining and technically accurate a movie is. Purely because of the fact that they will have seen more movies and will have more hands on experience.
I agree on that the techincal vote should be biased towards experienced players. However, you need no experience to judge how entertainment a movie is. If you found it fun to watch, then I guess you should give it a high vote.
My published movies
[03:45:05] <Naohiro19> Soulrivers: ...
[03:45:19] <Soulrivers> ?
[03:46:35] <Naohiro19> <Soulrivers> No! <Naohiro19> So? <Soulrivers> Yes!
[03:46:48] <Naohiro19> joke
Joined: 11/4/2007
Posts: 1772
Location: Australia, Victoria
I agree that people with more player score should have more bias for the technical vote... I wouldn't trust my technical vote to be anywhere as accurate as say... someone that hasn't had every single one of their four runs gruefooded.
I just stumbled upon this, while actually looking at the ratings of FuzZerd's Sonic run ([1401] Genesis Sonic the Hedgehog by FuzZerd, upthorn & carretero in 15:51.73). If there is a significant concern about outliers (in this case, votes widely divergent from all other votes) screwing up a movie's rating, you may consider using a more robustestimator such as the median. With the median, any single extreme vote is unlikely to affect the result (as it happens with the mean), particularly for a small number of votes.
Indeed, going by Kyman's reply above, Nach voted 0.1~1.0 for entertainment in the S3&K run, and probably voted the same way for FuzZerd's Sonic run. I seriously doubt that he finds that little entertainment value in Genesis Sonic runs...
Edit: Assumed the lowest bound for the rating due to ignoring the possibility of rounding errors; fixed.
The rules of the rating system could to be a little modified. I think 5.0 or 4.0 should to be the limit of the minimum in rating. Cause guys like Nach/Hitler of TASVideos (who have weight 1.0) can to fuck alone all the rating of a published/starred/hard worked/amazing TAS like these marzojr's.
Joined: 11/11/2006
Posts: 1235
Location: United Kingdom
marzojr wrote:
Indeed, going by Kyman's reply above, Nach voted 0.1 for entertainment in the S3&K run, and probably voted the same way for FuzZerd's Sonic run. I seriously doubt that he finds that little entertainment value in Genesis Sonic runs...
You don't know Nach :)
People are (should be) allowed to vote how they choose. Limits should not be placed.
<adelikat> I am annoyed at my irc statements ending up in forums & sigs
Joined: 3/9/2004
Posts: 4588
Location: In his lab studying psychology to find new ways to torture TASers and forumers
marzojr wrote:
Indeed, going by Kyman's reply above, Nach voted 0.1 for entertainment in the S3&K run, and probably voted the same way for FuzZerd's Sonic run. I seriously doubt that he finds that little entertainment value in Genesis Sonic runs...
I didn't vote THAT low.
Raiscan wrote:
People are (should be) allowed to vote how they choose. Limits should not be placed.
Indeed. The rating system is up to each person's personal opinion. No two people agree on what they like, and we should let everyone vote how they want. I think it absurd that we have a few users who go around rating all the amazing Metroid movies <4 in entertainment, but they're still entitled to their opinion.
If the rating system only allowed "Good" or "Very Good", it wouldn't be much of a rating system, and wouldn't truly cover our diverse demographic of users.
If a handful of users are skewing the rating on a particular movie, that either means not enough people rated (more people need to rate moves!), or that the movie in question is actually quite bad (we actually did publish some rather bad movies in the past that deserve their sub 4.0 rating).
Warning: Opinions expressed by Nach or others in this post do not necessarily reflect the views, opinions, or position of Nach himself on the matter(s) being discussed therein.
I agree with Raiscan that there should not be limits placed on voting. Which is why I actually suggested the use of something other than the average value of all votes to determine the final rating of a movie. The problem with the average is that it isn't robust -- any outliers will alter the average and make it stop being truly representative of any trends in the sample. The median, on the other hand, is very robust -- it is robust right up until the point where the "outliers" drowns the data so much that you should just throw the data away and start anew. It is great for filtering out "noise" and revealing any actual trends in a sample of data.
Indeed, going by Kyman's reply above, Nach voted 0.1 for entertainment in the S3&K run, and probably voted the same way for FuzZerd's Sonic run. I seriously doubt that he finds that little entertainment value in Genesis Sonic runs...
I didn't vote THAT low.
You are right, of course; I should have checked that there was some leeway due to rounding errors; I apologize for the mistake.
Does it really matter that much? People will vote how they feel, and no one can change that.
It astounds me that Zeupar feels the need to call out two vested and extremely active members of this site; there is no realistic reason for why either of these members would not know and understand the rating guidelines.
adelikat wrote:
I very much agree with this post.
Bobmario511 wrote:
Forget party hats, Christmas tree hats all the way man.
This raises the interesting question of whether one should vote purely with regards to one's personal feelings about the game/run, regardless of whatever possible overwhelming popularity the game/run otherwise has, or whether the rating should reflect how much the voter thinks the average person would enjoy the movie.
There are many runs which I personally don't find extremely entertaining, but which I understand very well how the average gamer would find quite interesting. This poses a dilemma when rating: Should I rate it according to my feelings, or should I rate it according to what I know about the sentiments the average gamers and TASers have towards the game/run?
Of course the dilemma is even harder in the opposite case: In other words, when I personally like a run a lot, but I know that it's not very popular in average.
Look at the the individual ratings by those eight users and the average rating for entertainment. That means that Nach and Kyman gave a rating of less than 2.2 for entertainment, which is hard to understand. I would like to read an explanation from both. Nach, is this one of your methods to torture TASers? :P
