Since there seems to be some confusion about what the technical rating means, perhaps it would be a good idea to create a description of that rating which would help people to vote. (A description of the entertainment rating could be created too, but I suppose that's less confusing to begin with.)
Of course the exact meaning of the technical rating may be seen as a question of opinion. However, the problem with this is that if different voters have a different notion of what it means, the votes will get biased and the error margin will get larger, making the rating less useful. As a co-developer of the rating system I had my own idea of what the technical rating would mean, but I didn't think back then about actually describing it. (In fact, putting it into words back then would have been much more difficult than now that I have had time to think about it.)
So here's my proposal for the description of the technical rating:
When judging the "technical quality" rating, two elements of the video should be considered: How close to optimal the run length is (ie. if it's as fast as possible), and the techniques used to make the run.
Judging the techniques used to make the run relies, naturally, quite a lot on the submission description of the video, which is why submitters should describe these techniques in detail. Techniques usually have one goal: To make the run faster. These are things to consider about the techniques:
How much work was put into studying the game and background research (eg. searching for key values, decompiling, searching for glitches, routes...)? What kind of tools were used to make the run? How innovatively and ingenuously the tools and techniques were used? Was perhaps a new tool developed to make the run possible? How much work overall was put into making the run (while the number of re-records can be used as an indicator for this, it shouldn't be used as the only indicator)?
Other minor things can be considered too, although their weight on the rating should perhaps not be as accentuated. These things may be game-specific, such as how innovatively glitches were used to save time, how damage was used for shortcuts and so on.
It's not a question of discussing the meaning of a word. It's a question of defining better what the ratings mean, giving voters a clearer picture of what they should be looking at.
Currently the ratings are described by a couple of words and people are free to interpret these words as they like. Of course one could argue that people are free to do whatever they want anyways, but this just poses the quesiton of why have any rating at all if people are going to vote arbitrarily anyways? We may as well just put a rand() value there.
Rating makes sense when people rate something specific. However, if that something specific is badly defined, the value of the ratings diminishes.
I'd say, remove the technical rating option. If you don't know the game well you can't say if it's optimal or not. It's very hard to rate movies because of this.
Voting 0-10 seems a bit odd, too, given that all movies that would score below 4 get rejected or obsoleted.
http://tasvideos.org/274M.html has an average entertainment rating of 3.2. I suppose the lowest rating someone has given to it must be pretty low.
There are probably other examples. That's the one I found first.
Joined: 5/2/2006
Posts: 1020
Location: Boulder, CO
Only 0 I have ever givin. =X
I dont know if this site has ever delt with un-publishign a run, but honestly, words like "god-awful" and "boring as shit" just dont quite cut it sometimes.
As a converse to Warp's take on technical rating, I look at this situation:
Shinryuu's new Rockman 2 run obsoleted the old one by 7 seconds. In the submission text he explicity states that "Bisqbot/Bots were not used". The previous, now obsolete, submission did use them extensively.
I think rating the new movie lower on the technical scale simply because it didn't use bots is counter-intuitive.
Thoughts?
<Swordless> Go hug a tree, you vegetarian (I bet you really are one)
Can it be seen as a technical merit that a bot was not used but all optimizations were done by hand? If yes, then a high technical score is justified. I suppose it's up to the voter to decide whether it's technically more praiseworthy or not.
Anyways, my proposition was for a generic guideline, not a strict rule. Well-justified exceptions are always acceptable.
Large bump (one month), oh noes.
I would rather see the "technical" rating moved to be more of "how well does this movie accomplish its goals?" This would take more of the focus off of "one frame optimizations" and more of actually meeting the goals of the movie, since some movies do not have speed as their primary goal (SMB1 top score, for example)
This way, it wouldn't require a lot of technical knowledge, and could be better answered by "Does this run accomplish the goal well?", which is much more easily answered than "Is it perfect?"
For example, while technically a max-score SMB1 can be frame-perfect, it may fail to accomplish the goals (by skipping something worth points, for example). I'm not entirely sure people who only go by "frame perfection" will judge it right on the technical side, since it may have no mistakes in terms of speed.
But it'd also start going back to the "superplay, not speedrun" methodology of this site long ago, and I'm not entirely sure people are willing to go backwards however much I prefered it then.
I think that with Warp's idea of what the rating should be, that "technical rating" is a misleading name. Though, I can't think of a very fitting one. Something like "How thorough was the author?" Bleh, that doesn't sound very good either.
<Swordless> Go hug a tree, you vegetarian (I bet you really are one)
When judging the "technical quality" rating, two elements of the video should be considered: How close to optimal the run length is (ie. if it's as fast as possible), and the techniques used to make the run.
How should a watcher know how close it is to optimal length?
A impressed one would think, "Wow, I didn't think you can do this glitch and I never thought of using this at that place! This HAS to be optimal!" or something.
Most of the watcher aren't familiar with the in-game mechanism, glitches, sequence breaking and whatever. How should such a person know if this is perfect?
And quite nobody does frame-per-frame analysis to find out, "Hey! This can be improved by 1 frame!".
Or am I mistaken?
You're not mistaken.
Actually, all of the TAS movies here except a dozen or two aren't perfect and could be improved at least by 1 frame, so it's safer to assume that none are perfect until proven otherwise. :)
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
That's mostly a guesstimate. So far I can name SMB-warp, SMW-11 exit, Kung Fu, Circus Charlie, MTPO and most probably SPO, probably Legend of Kage, probably Dark Castle, 4-cpu Monopoly, Donkey Kong, Paperboy and probably some other autoscrollers. I guess there's more, I'm just not entirely sure of their perfection yet.
Warp wrote:
Edit: I think I understand now: It's my avatar, isn't it? It makes me look angry.
Cool. SMW is not perfect btw, I'm guessing at least a few of the others (haven't seen any of them) you've mentioned are only perfect-looking but actually aren't as well.
Still more than I expected. Neato.
Floor 99 can be improved by 1 frame. Then Floor 100's shots gets re-arranged making it slower. so it can not be improved. Also, I do not recall saying that.
moozooh,
It is very likely there is a better lag pattern avaliable somewhere which we (JXQ, I did nothing useful) were unable to find. After a few of the levels we lost a frame compared to the last version in lag on the worldmap, or the star road animation or bla bla. Point is, there still are 7 (if I recall correctly) "unncessary" lag frames, and while I doubt we will ever be able to find a way to improve it, I believe it can be improved.
In addition to this a 2 frame improvement in Star World 3 has been discovered. It's still not a confirmed improvement overall, because of similiar reasons Vatchern mentioned. It would mess with lag later in the run and it's impossible to say if it would save time or not.
So yeah. Nothing confirmed but I would be surprised if it's not theoretically improveable.
Vatchern, is the improvement you are referring to the same one that Brushy said he found after you submitted your latest version? Or not sure, maybe.
PS I don't understand that game at all, even after several watches, so congratulations on that run :D
<Swordless> Go hug a tree, you vegetarian (I bet you really are one)
I'd guess there isn't a perfect movie, each can be improved by at least 1 frame.
I might be wrong, but a perfect movie? I can't imagine.
Besides that, I really think only brute force, or AI which completely understand the game's algorithm, is able to perform a perfect run.
But who knows?
