1 2 3 4 5 6
Posted: 9/14/2015 8:12 PM
Quote
dunnius
Active player (428)
Joined: 9/7/2007
Posts: 329
Computer TASer of 2015 Computer TAS of 2015
I too feel the ratings are unfair, especially the technical ratings, which I feel are completely bogus. The bigger problem is that I think nobody knows what the rating values mean, or at least not very consistently. This is not helped by having so many numbers. Since I don't know, I do not rate movies. The system needs an overhaul to make it more consistent, and more understandable.
Posted: 9/14/2015 8:23 PM
Quote
scrimpeh
Editor, Skilled player (1438)
Joined: 3/31/2010
Posts: 2106
TAS of 2017 First edition TAS of 2017 PSX TAS of 2017
I don't rate movies because I don't believe I can accurately express a complex opinion influenced by subjective taste with two numbers. Picking a number, any number, just feels arbitrary. While I probably shouldn't complain about ratings I got myself, I do must say it is a fair bit of a bummer when movies you worked harder on get worse ratings than movies you worked less on. In a similar vein, movies being rated worse than the ones they improve. Makes it feel like the whole effort was for nothing.
Posted: 9/14/2015 9:03 PM
Quote
Gay
Former player
Joined: 6/21/2015
Posts: 58
I do not even know the technical quality is. I searched the forums, but there is no clear definition. I found people who say the definition is not even related to optimizing. So what is it?
Gay Dominguez Jr.
Posted: 9/14/2015 9:05 PM
Quote
Masterjun
He/Him
Site Developer, Skilled player (1986)
Joined: 10/12/2010
Posts: 1185
Location: Germany
Exotic platforms TAS of 2020 Speedy TAS of 2018 PC TAS of 2018 Glitchy TAS of 2015 TASer of 2014 SNES TASer of 2014 TAS of 2014 SNES TAS of 2014 Funny TAS of 2014 Glitchy TAS of 2014 TASer of 2013 SNES TASer of 2013 NES TAS of 2013 SNES TAS of 2012 Funny TAS of 2012
Voting Guidelines - Ratings - Technical
Warning: Might glitch to credits I will finish this ACE soon as possible (or will I?)
Posted: 9/14/2015 9:23 PM
Quote
dunnius
Active player (428)
Joined: 9/7/2007
Posts: 329
Computer TASer of 2015 Computer TAS of 2015
For technical, how am I supposed know how much work went into the TAS? It is very hard to tell, even from reading submission comments. It doesn't make sense. And it doesn't make it clear how to rate, which leads to inconsistent ratings.
Posted: 9/14/2015 9:25 PM
Quote
Grincevent
Active player (309)
Joined: 8/21/2012
Posts: 429
Location: France
The problem with the technical rating is that I find the notion vague, and almost impossible to give a fair vote. Vague because it could mean how optimized the TAS is, how "perfect" it is, how hard it was to make, how many "techniques" it uses or requires (luck manipulation, lag reduction, botting, searching RAM/even making RAM maps...), or, interestingly, how technical the TAS looks. As we all know (I hope ^^), the hardest parts to optimize in a TAS are not necessarily the most impressive to watch, and are sometimes near impossible to notice without being told that "yes, that thing was extremely complex to do, I spent hours on this 5 seconds segment to make sure that the next 4 items are the ones I needed." (random invented example) Let's say that some game is extremely easy to TAS, so the result will be a perfect movie (until some new glitch or ACE is discovered... Or maybe after it is discovered and implemented ^^); does it warrant a very high rating? There is no definitive answer, I think. But I personally tend to prefer "rewarding" TASes that were a pain to create technically. That doesn't mean I would close my eyes to big optimization flaws either. Anyway, I don't vote that often... The next problem is: "how do I know what went into the making of the movie?". There are obviously the submission notes, but not always; and they usually don't speak about what was actually technically difficult to do, rather explaining how things work and what tricks were used. I'm sure we can find counter examples, of course. The thing is, if a TAS appeared and was a technical masterpiece, but without a detailed submission text, the ratings would be influenced. I'll stop before losing track about what I tried to explain at the beginning. I didn't even write about the entertainment rating. It's less of a problem because it is completely subjective. In the end, I don't really care about the ratings, I think I've explained why they are not that important to me.
Posted: 9/14/2015 9:28 PM
Quote
Gay
Former player
Joined: 6/21/2015
Posts: 58
So what MUGG is saying seems every movie that is well optimized deserves a rating like 9/10. And since each film must be well optimized, does not that mean that every technical evaluation would be really high?
Gay Dominguez Jr.
Posted: 9/14/2015 9:38 PM
Quote
MUGG
Editor, Expert player (2327)
Joined: 5/15/2007
Posts: 3927
Location: Germany
Gameboy TASer of 2015 NES TAS of 2013 Gameboy TASer of 2011 Gameboy TASer of 2010
dunnius wrote:
For technical, how am I supposed know how much work went into the TAS? It is very hard to tell, even from reading submission comments. It doesn't make sense. And it doesn't make it clear how to rate, which leads to inconsistent ratings.
Which is why I took a little time to rate my own movies, for fun, because I do know how much work went into my own TASes. But I think some people already agreed (on IRC or here) that the rating system could need a rework to make for more consistent ratings. This should probably be discussed here.
Gay wrote:
So what MUGG is saying seems every movie that is well optimized deserves a rating like 9/10. And since each film must be well optimized, does not that mean that every technical evaluation would be really high?
Yes, pretty much. But everyone seems to have their own definitions of what an "8", "9" or "10" is. For me, 8 = kind of improvable 9 = hardly improvable 10 = perfect I rated my own movies in the first post to defy some of the bad ratings I received - but I did it mostly for fun. Again, as said, I think we should use this topic to discuss a rework for the rating system, or maybe we can come up with a clearer definition of what an 8, 9 or 10 should be. For the record, I did not even take into account the "techniques" used in my movies. I only considered the frame-perfectness of the runs.
Youtube • Livestream • Progress 11th TASer of the Decade 2010-2019 !
Posted: 9/14/2015 10:25 PM
Quote
TehBerral
Active player (263)
Joined: 8/14/2014
Posts: 188
Location: North Kilttown
The issue I have is what exactly constitutes "technically difficult" differs depending on the person (and I try to keep this in mind when rating). Basically I'd be more impressed that a newbie TASer on their first submission is (effectively) using RAM search and reverse engineering methods than if, say, a veteran like Adelikat was. Maybe it's just me personally, but I just feel like no matter how complicated, if you've been doing it for years, any method would become commonplace, which only complicates the technical rating more. Regardless, a lot of that ends up being a moot point as I completely agree with Grincevent. Especially since I can say from experience that this below is true:
Grincevent wrote:
the hardest parts to optimize in a TAS are not necessarily the most impressive to watch
Which makes rating it even harder.
Somewhat damaged.
Posted: 9/15/2015 2:44 AM
Quote
Patashu
He/Him
Joined: 10/2/2005
Posts: 4042
Youtube used to have a 1-5 star rating system, and replaced it in favour of the like/dislike system, which was a lot simpler and achieved basically the same intended outcome. Why doesn't tasvideos use a like/dislike, or even just a like, system? (Previously existing ratings could be auto-converted into like/dislike based on whether they're higher than 7 or lower than 5, for example.)
My Chiptune music, made in Famitracker: http://soundcloud.com/patashu My twitch. I stream mostly shmups & rhythm games http://twitch.tv/patashu My youtube, again shmups and rhythm games and misc stuff: http://youtube.com/user/patashu
Posted: 9/15/2015 3:18 AM
Quote
TehBerral
Active player (263)
Joined: 8/14/2014
Posts: 188
Location: North Kilttown
Actually, Patashu, that would solve a lot of issues raised in this thread. If there was some sort of "liked TASes" list that they went to that would solve the issue of people wanting to go back to TASes that they liked easily, that'd be great. That would narrow rating down to one-click. Plus it gets rid of a lot of subjective nonsense - do you like it? Yes? Then click it! Simple as that. It also wouldn't need to send you to a different screen to vote. I have a question, though: what is the point of the 1-10 system we have now? The ratings don't decide the tier - that's already decided before publication. And they don't determine Stars being added - as mentioned in a different thread the considerations go farther than simple popularity. So is there any real need to have something as specific as the 1-10 system that I'm not aware of?
Somewhat damaged.
Posted: 9/15/2015 4:23 AM
Quote
ars4326
He/Him
Experienced player (775)
Joined: 12/8/2012
Posts: 706
Location: Missouri, USA
As with all these site change discussions, I have no idea how much coding work would have to go in to implement changes. With that said, one idea I would be in favor of is narrowing down the voting numbers to half increments (5.5, 6.0, 6.5, etc.). I believe that would simplify the process a little better, and give slightly more incentive for members to vote on movies. A while back, Mothrayas also mentioned in another thread the idea of simplifying voting by narrowing the process to a movie's publication page. Similarly, I'd also be favor of being able to rate movies while they're still on the Workbench (I know there was another thread discussing that somewhere, too...).
"But as it is written, Eye hath not seen, nor ear heard, neither have entered into the heart of man, the things which God hath prepared for them that love him." - 1 Corinthians 2:9
Posted: 9/15/2015 11:42 AM
Quote
NitroGenesis
He/Him
Editor, Experienced player (556)
Joined: 12/24/2009
Posts: 1873
TehBerral wrote:
The ratings don't decide the tier - that's already decided before publication.
They do though. If a Moon dips below 5.6 in entertainment value it generally heads to Vault. A TAS being sent to a tier by a judge isn't the last time it gets changed. Oh, but even though adelikat said that authors rating own movies doesn't affect the overall, it DOES affect the average entertainment/tech values shown on the ratings page, so this could be troubling.
YoungJ1997lol wrote:
Normally i would say Yes, but thennI thought "its not the same hack" so ill stick with meh.
Posted: 9/15/2015 12:39 PM
Quote
Warp
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
Gay wrote:
So what MUGG is saying seems every movie that is well optimized deserves a rating like 9/10.
As the guidelines say, "note that not all games are suitable for a TAS with perfect technical rating, similarly to how not all games are suitable for a perfect entertainment rating. Some games simply don't lend themselves for extensive technical achievements." If a game is such that not many technical feats can be performed on it, then it ought to get a lower technical rating. Sure, that might feel a bit "unfair" from the author's perspective, but it's no different from the entertainment value: If you chose a boring game, don't expect a high entertainment score. Likewise if you chose a straightforward game that doesn't lend itself to awesome technical feats, don't expect a high technical rating. But yeah, both rating categories are highly subjective. The technical one ought to be more objective, but it's unrealistic to expect everybody to be so objective about it. (And also there will always be cross-contamination of the entertainment rating into the technical rating.) Anyway, if you don't know what to rate for technical, you can leave it without value.
Posted: 9/15/2015 4:07 PM
Quote
feos
Site Admin, Skilled player (1250)
Joined: 4/17/2010
Posts: 11469
Location: Lake Char­gogg­a­gogg­man­chaugg­a­gogg­chau­bun­a­gung­a­maugg
NES TAS of 2011
I don't think it's even possible to take ratings seriously, since indeed they make little sense, when only the people who actually TAS a game know exactly how technical its runs are, and people who see a game for the first time might be not entertained at all and give 3s for a well done movie (I use to do that a lot lately). These 2 don't seem to be improvable, so no matter how you change the system, it'd still make that little sense.
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 10/11/2015 5:28 AM
(Edited: 12/27/2017 10:35 PM)
Quote
BrunoVisnadi
Editor, Skilled player (1343)
Joined: 12/28/2013
Posts: 396
Location: Rio de Janeiro, Brasil
SNES TAS of 2020 TAS of 2015 SNES TAS of 2015
It seems it is almost impossible to ratings to be fair, indeed. http://tasvideos.org/rating.exe/2926/details Here, someone voted 0 for both tech and entertainment, which I consider very unfair. (And this same person didn't give a 0 to the tech quality of the previous, obsoleted and less optimal movie, for some reason. People rate inconsistently). But also, someone voted 10 to entertainment, which is also really unfair. It's just Mario running and eating something for 20 secs. So, this 2 people have completely different opinions about the movie. Who will say which of them is right? Well, I would say both are wrong. But I'm sure some of my own ratings are probably unfair as well, and that should apply to everyone. Maybe a solution to this would be giving 5 rating options (1, 2, 3, 4, 5?) instead of 101, and describing them. Something like what adelikat did here (http://tasvideos.org/Adelikat/Ratings.html), but with all of this explicitly visible below the rating table, so that everyone will see this before voting. Maybe, this way it will be slightly less arbitrary what is a 1 and what is a 5.
My YouTube channel: https://www.youtube.com/channel/UCVoUfT49xN9TU-gDMHv57sw Projects: SMW 96 exit. SDW any%, with Amaraticando. SMA2 SMW small only Kaizo Mario World 3
Posted: 10/11/2015 5:14 PM
Quote
Amaraticando
It/Its
Editor, Player (159)
Joined: 1/10/2012
Posts: 673
Location: Brazil
TAS of 2015 SNES TAS of 2015
Ah, the revenge rating!
Posted: 10/11/2015 6:08 PM
Quote
Warp
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
I have an idea: What if the ratings were weighted according to player points? Or at least the technical rating. With this I don't mean that the weighted rating would replace the current way of calculating it (which is just a simple average). It could be shown in addition to the normal rating, in some manner. Also, the weighting wouldn't necessarily be linear. It could use some kind of cumulative normal distribution (so that people with very low player points would weight about the same between themselves, and people with very high player points would weight about the same between themselves.) The reasoning is that experienced TASers have more experience in evaluating the technical quality of a run, and thus their rating is (hopefully) more informed and less biased. (Also, the minimum weight, ie. people with 0 player points, wouldn't be 0, but something larger than 0, so that everybody's rating contributes, even if by a little.) Of course this will give good results only if there's a significant amount of ratings by people with lots of player points.
Posted: 10/11/2015 9:12 PM
Quote
jlun2
Skilled player (1737)
Joined: 9/17/2009
Posts: 4979
Location: ̶C̶a̶n̶a̶d̶a̶ "Kanatah"
GBA TASer of 2010
ars4326 wrote:
As with all these site change discussions, I have no idea how much coding work would have to go in to implement changes. With that said, one idea I would be in favor of is narrowing down the voting numbers to half increments (5.5, 6.0, 6.5, etc.). I believe that would simplify the process a little better, and give slightly more incentive for members to vote on movies. A while back, Mothrayas also mentioned in another thread the idea of simplifying voting by narrowing the process to a movie's publication page. Similarly, I'd also be favor of being able to rate movies while they're still on the Workbench (I know there was another thread discussing that somewhere, too...).
Post #414850
Mothrayas wrote:
I haven't read most of this topic yet, so sorry if some things were already mentioned. I had some talks with Nach a few months back about improving the rating system to be easier. The idea was to replace the ratings link/page with a simple form on the publication module itself with text input and/or a slider to input the entertainment and tech ratings. Combined with a confirm button, that'd bring the number of clicks needed to put in any rating down to 3 (or 2 if you only fill in one rating) and no need to go to different pages. We never got around to doing it because of lack of coding manpower, and lack of people who know enough site code/ratings code/publication module code to dare touch it or do anything with it. (In other words, we'd probably need Nach to get on it).
So unfortunately, unless someone with amazing spaghetti code untangling powers comes along then it'll take a long time for change(s) to be implemented.
LLCoolDave wrote:
This seems to be competing for the "Worst looking TAS of 2011" Award.
Posted: 10/11/2015 11:10 PM
Quote
Warp
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
I don't think what I suggested would require changing the existing code, just adding a new functionality somewhere that would calculate the ratings a bit differently. It ought to be a relatively simple database query and some simple math, to display some field somewhere.
Posted: 10/12/2015 9:21 AM
Quote
thatguy
Editor
Joined: 11/3/2013
Posts: 506
Warp, your suggestion is probably more trouble than it's worth, given that even a highly experienced and skilled TASer like MUGG has a dubious interpretation of the what technical ratings mean. (I always understood the technical rating to be "how much effort and/or skill was used in this movie?" whereas MUGG interprets it as "how frame-perfect is the movie?") We don't need funny algorithms, we just need to establish consensus on the meaning of the technical rating, or even whether it is necessary. If that requires a community-wide discussion, so be it. EDIT: And I quote directly from the Rating Guidelines wiki page: "A common misconception is to think that "technical quality" is a synonym for "frame perfection" (in other words, how optimal the run is). This is not so. Frame perfection is part of what constitutes the technical quality of a run, but only a part."
Posted: 10/12/2015 1:34 PM
Quote
moozooh
Senior Moderator
Joined: 8/4/2005
Posts: 5777
Location: Away
I raised this concern way way back (2007?) when people were liberally flinging 10s in tech quality at a run that contained numerous known improvements at the time (Fabian's SMW all-exit). The argument at the time was that, no matter how you interpret technical rating or its scale, having known improvements shouldn't have earned a perfect mark just because you liked the run/the game and/or were excited that a long wait was over and/or you wanted to congratulate the author's achievement that way. While I can't say the situation has changed significantly "for the better" since then, nor have I changed my own frame of reference, I've started treating tech ratings I see on publication pages differently. I now see it as a gauge of the audience's expectation of improvement, which is a quasi-integral parameter that combines optimality (knowledge-based or otherwise) and the perceived skill/effort/etc.. I'll elaborate. When a movie gets low votes, it means the audience wanted it to be better. It means the author either didn't optimize it well, or didn't find enough game-breaking stuff—no matter whether it's even possible in the game. On the other hand, if it gets high votes, it means the audience is satisfied with this movie's optimization and doesn't necessarily want more game-breaking stuff (as, indeed, it can hurt entertainment value). This new understanding actually alleviated a lot of the confusion and disturbance previously induced by tech ratings, and helped me see better what movies are worth trying to improve, if only with regards to the potential increase in ratings (as pointless as that can be). Hope this perspective was useful for the discussion.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Posted: 10/12/2015 2:45 PM
Quote
feos
Site Admin, Skilled player (1250)
Joined: 4/17/2010
Posts: 11469
Location: Lake Char­gogg­a­gogg­man­chaugg­a­gogg­chau­bun­a­gung­a­maugg
NES TAS of 2011
http://tasvideos.org/Feos/Ideas.html#MovieRating
Warning: When making decisions, I try to collect as much data as possible before actually deciding. I try to abstract away and see the principles behind real world events and people's opinions. I try to generalize them and turn into something clear and reusable. I hate depending on unpredictable and having to make lottery guesses. Any problem can be solved by systems thinking and acting.
Posted: 10/12/2015 2:49 PM
Quote
Niamek
Joined: 10/23/2009
Posts: 545
Location: Where?
moozooh wrote:
I'll elaborate. When a movie gets low votes, it means the audience wanted it to be better. It means the author either didn't optimize it well, or didn't find enough game-breaking stuff—no matter whether it's even possible in the game. On the other hand, if it gets high votes, it means the audience is satisfied with this movie's optimization and doesn't necessarily want more game-breaking stuff (as, indeed, it can hurt entertainment value). This new understanding actually alleviated a lot of the confusion and disturbance previously induced by tech ratings, and helped me see better what movies are worth trying to improve, if only with regards to the potential increase in ratings (as pointless as that can be).
Quoting that, since I feel that's important. But I think there is still an another dimension: personality. I for example, dislike almost every fighting TAS movies there is on this site(I dislike strongly movies like Marvel vs. Capcom for example, but some I like some specific fighting movies such as Tekkens). If I were to rate Marvel vs. Capcom, even if I feel they are very high technically (and entertainingly, according to the tier it is), I would give a very low rating to that on the entertaining side. I don't vote because I feel it's unfair since I just dislike strongly these kind of games. (I'd rather Combatribes for example. yeah, i dislike the fighting games like Marvel vs. Capcom that bad) For the technical side, it's the perfectionism. For example, Kirby and the amazing mirror, 100%, there was a trick that wasn't used because it was noticed too late. Instead of starting over, he just carried on without implementing it just to keep consistency. Other would vote low just because of that (The extreme would be the perfectionists), other would vote accordingly to that, other wouldn't even take account of that. It's the same for the submission votes.After a sub-optimal submission Some threw that the submission were blatantly bad simply because of sub-optimally others were like: "Okay, he did some mistakes, but hey it wasn't that bad." or "It wasn't that bad, but I feel you can improve further by doing X, Y, etc.". Some are just too perfectionist, others are more open. Note that I'm not targeting someone.
Posted: 10/12/2015 8:44 PM
Quote
Warp
Banned User
Joined: 3/10/2004
Posts: 7698
Location: Finland
feos wrote:
http://tasvideos.org/Feos/Ideas.html#MovieRating
Perhaps the simple act of changing the word "technical" to "effort" might in fact be enough. The latter gives a much better idea of what the rating is about than the former. And using one single word. I like it.
1 2 3 4 5 6