Since I’m doing this anyway, I’ll report any responses I make on PBS Game/Show episodes. Here’s my latest on their video on game reviews:
Great stuff, Jamin. I really did like your passionate yelling, though. It kind of became the feel for the show. I’d yell because I’m passionate about this stuff too. If you ever get back to yelling, you have my support!
I think you’re right on many levels with reviews, that its methodology is broken, but also that they are still important on a few levels. Here are some ways I think can improve the process:
1) Since games are experiences, why not have multiple reviewers take a stab at reviewing the game? Journalists become a trusted voice, a personality with their own likes and dislikes. Bob, Jane, and John all review the same game and share their thoughts. Bob likes RPGs, Jane hates RTS, and John loves casual. How will each person react to the game they were just assigned to review. In a sense, it is sort of like user reviews… but this time you build a relationship with each journalist and understand their thought process. I like to review restaurants on Yelp a lot (and get Elite perks) but I always start off with what I tend to look for and the expectations I have before writing. E.g. I will always bash NYC restaurants for trying to cram way too many tables in a way too small of a space. A restaurant that has ample room to walk around and no butts in my face immediately gets more stars.
2) While games are experiences, those experiences are created from a static system. Reviews could go the academic route and try to examine the systems themselves. As a game designer, I do this on my own blog and try to reverse engineer the experiences I have and see what caused them. This kind of writing seems to be much more objective. The system is then compared to the current trends and historic systems. It’s a much harder review to write since you’ll need to be extremely knowledgeable in many game development related fields from programming, design, to art.
On a closing note, I really need to call out MetaCritic for being terrible. Their review system is completely opaque (the non-user side). Some sites are arbitrarily weighted higher than others. They even accept reviews during beta. And as you said Jamin, at what stage of the game should a review be done? However, MetaCritic’s user reviews seems pretty honest. Although the x/100 rating is fairly arbitrary still. Like/Dislike ratio seems to make a lot more sense.