Entropy

Below follow parts 1 and 2 of the documentary Stardock shot while performing open heart surgery on the networking code of Demigod in an attempt to fix the multiplayer issues that prevented the game from being the launch day success it should have been.

Part 1 shows the various meetings that were held discussing the problem and deciding who to pull in on the team. Stardock pulled resources from everywhere in the company, including the Elemental (the upcoming Stardock strategy title), WinCustomize, and Impulse teams to try and make Demigod work.


Part 2 shows a batch of interviews conducted by Frogboy/Brad Wardell during the wee hours of the morning with some of the Stardock staff that were working on the new Demigod multiplayer system.


Most of the stuff said in the videos has already been covered in my previous posts on the topic, Piracy of Demigod: The Real Story and From exemplary PR to colossal disaster (and back again) - What happened to Demigod? It is interesting, and I'm sure in many cases it'll be enlightening, to see what Stardock has had to do to recover from some poor initial design and architecture decisions.

I've been waiting for this 'documentary' since Brad Wardell first mentioned it in one of his Demigod Journals. I didn't notice these videos until now since I naturally assumed he would be the one posting them. It turns out Frogboy isn't the only one posting Demigod Journals.

EDIT v0.5: Here's part 3 so long. It shows the Demigod/Impulse team trying to track down a bug that's deleting records from the database it's not supposed to. More interestingly it shows some early work on Elemental, Stardock's next game.

Just some quick news while I'm in-between articles.

Frogboy/Draginol/Brad Wardell posted a log of additional fixes Gas Powered Games have submitted for Quality Assurance (QA) testing on Wednesday.

• Fixed chat history and announcement text overlays interference with mouse clicks.
• Switched order of data calls after hosting.
• Teleportation effect failure handling.
• AI
  1. Adjustments to caution and hazard detection and avoidance.
  2. Adjusted health level flee percentage.
  3. Demigods do not use minions when fleeing, may summon minions in combat.
  4. Improved teleportation.
  5. Increased Demigod planning frequency.
  6. Improved item usage when endangered.
  7. Tower tag improved.
  8. Moving targets in Ability task now properly handled (the famous “Sedna bug”)

Apparently the "Sedna" bug (more commonly known as the "Skill-activation-but-my-DG-won't-do-anything" bug) was isolated by a forumer/forumite and fan bman654. He posted his own fix on 24 May, it was picked up by GPG and rolled into the new patch. Servo, a developer from GPG, tried to explain how it was that they were unable to fix this bug since it was reported in the beta but gave credit where it was due.

Needless to say, if bman654 were to run for any office he would have the unanimous vote of the Demigod player community.

Fans got a preview of what lay ahead in the v1.01 patch on 22 May (last week Friday) with the following promised fixes. These will be included in addition to the new batch of fixes listed above.

• Bandwidth used reduced by 35% (more people with marginal connections able to play). This should reduce some of the “stutter” some people run into when they play with someone with a marginal Internet connection.
• Updated the way stats are submitted to make stat reporting significantly more robust.
• If a player loses connection with another player (or one player with a group of players but not all) only 1 user is disconnected instead of multiple.
• Demigod is better at detecting the difference between quitting and a disconnect (no connection dialogs when someone leaves).
• Significantly better handling of users who have shaky connections to keep them in game without adversely affecting in game performance.
• Favor Points, Achievements, Items stored on server more reliably (thanks to the updated stats system).

The skill activation fix and bandwidth use fix have me really excited. Besides the fact that the skill activation fix was really irritating, the bandwidth use optimisation will dramatically improve my game experience as there's a bunch of us with 128kbps upstream speeds. For those not technically inclined - upload speeds that slow are not even covered by the slowest ADSL spec on Wikipedia and Demigod is very dependent on upload speeds since it uses a peer-to-peer networking system.

Mr Wardell didn't want to make any promises as to when the patch will be made available but did state that it would be put through its paces this week. After releasing the details of the new fixes patch v1.01 will introduce Frogboy said they would test the patch today (Thursday, 28 May) but couldn't say for certain that it would be released today.

He did say that if it's not released on Thursday we shouldn't expect to see it this week. It would seem that releasing a patch on a Friday is hazardous to a game developer and publisher's weekends.

EDITS: Justified text. Typos and some improvements on sentences that seemed to make sense at 1 in the morning.

Those that have followed the development of Demigod or of previous Stardock titles may find this as no surprise: Stardock allows the legal piracy of Demigod.

A poorly advertised feature, though I don't know why. The magnitude of the hype you could generate is immense! Imagine telling the writhing masses that you only need one Demigod license for everyone on a LAN to play. They will be queueing up to buy copies! Or more realistically, one dude (or dudette for those that don't recognise the unisex connotation of 'dude') will buy a copy and that person's friends will just mooch off them every LAN. But we all knew that was going to happen anyway. In fact, without this liberating clause in Demigod's user manual I reckon it was more likely that everyone at said hypothetical LAN would simply play a warez copy and no-one would own a legal one.

Of course there's no such thing as legal piracy since software piracy is defined as the breaching of the clauses governing copying and distribution in End User License Agreements. But I got you reading didn't I?

Frogboy mentioned the ability to run a LAN off a single license in one of his daily updates during Demigod's multiplayer teething problems. By 'teething' I mean huge and debilitating, of course. This was on 29 April, 3 weeks after the official launch of Demigod, and in the same post where Frogboy/Draginol/Brad "declared victory in spite of (t3h) Piracy" as misreported by our more illustrious newsmongers.



Unfortunately Stardock doesn't make it particularly easy to make use of this feature. From my experimentation and reading through various threads on the Stardock forum it seems the only truly feasible way to install the game on different machines is using ImpulseAnywhere. This involves downloading a full build of the game at around 2.8GB. And you'll have to do that everytime you install a new patch in-between LANs. That's probably why Demigod's fantastic LAN licensing terms aren't advertised far and wide.

There are many that argue that piracy helps provide exposure for games (a similar argument is made for music as well) and that the PC gaming scene would see even less action than it does now if it weren't for the illegal copying and distribution of games. Here, finally, is a company that recognises that.

Rather than criminalising their customers they listen to them. They enable them (that would be you and I, the common game consumer) to enjoy their game license with their friends over LAN, and thus let their existing customer/fanbase and the game itself do marketing for them.

Now if only they had made it easier to do.

It began with something game fans hadn't really experienced before. Here was a AAA title that had delivered a poor launch-day performance and the publisher stepped up to the plate, took responsibility for the problems and set about fixing them. On top of that the CEO of the publisher provided daily updates on the status of the issue(s).

The sheer excellence of the service kept the fans appeased for awhile. For quite a while, in fact. All the transparency, honesty and accountability in the world can't keep unhappy customers placated forever though. So what had to go wrong with Demigod for possibly the best PR effort in gaming history to run out of juice?

On Monday morning (EDT) Brad Wardell (CEO of Stardock) published a massive statement on his blog entitled, Demigod: So what the hell happened?. Kotaku picked up on it earlier yesterday (20 May 2009).

For those not in the know, a summary of the woes that befell Demigod.

• Gamestop "breaks the street date" (Frogboy lingo) of Demigod by selling it on 9 April already instead of waiting until 14 April. The release-day server infrastructure isn't ready and it's Easter weekend. Staff are called back from their Easter break to help deal with the problem.
• Demigod torrents are released onto trackers and by release day around 85% of players are using a warez copy (i.e. 85% of the users of Demigod pirated the game).
• An innocuous HTTP request that checks for updates brings Stardock's server infrastructure to it's knees. It's so bad some people struggle to even get in the game because it sits and waits for a response from the server. Launch day is a disaster.
• A patch shuffles (most) licensed users over onto a separate server infrastructure as well as disables the HTTP request at launch. This improves the situation dramatically.

It turns out, however, that there were some fundamental problems with the multiplayer matchmaking system provided by ImpulseReactor. This was compounded by bugs and perceived imbalances in Demigod itself. A particularly debilitating bug worth mentioning is the one where your Demigod will stop whatever it was doing when you activate an ability and then not return to the task when it's done casting the ability. Together with a bug that leaves a Demigod totally unresponsive until it moves, this allows another Demigod that you are busy ganking to get away while you scream at your monitor to make your mouse clicks do something.

All this started receiving mention on the Demigod forums in the first few weeks after launch, not to mention features like a replay system, a friends system that integrates better into the game, a better in-game chat system, a bettter lobby. Ke5trel on the Demigod forums put together a post that'll put any effort I make to summarise all the feature requests to shame. For four weeks (Frogboy counts three, but I count four) the multiplayer issues got all the attention.

I could find only one or two things Frogboy might have done wrong as ad-hoc community manager for the Demigod multiplayer debacle. The biggest by far is that he gave false hope by saying that the problem would be resolved by a certain date. His assessment was made with the information at hand and with the understanding of the problem that Stardock had at the time. As Stardock came to better understand the problem, what became known as the "Be-all End-all Multiplayer Fix" on the forum was delayed more and more, jading customers that had hoped on a fix by the second week after launch.

During the week leading up to the eventual major multiplayer patch which was released late on Thursday 15 May (EDT), it became evident that players were growing increasingly impatient. It felt like posts were generally more negative and that there were far more and far more vocal unhappy players posting. The unhappiest were those whose games were working before a patch and who then couldn't get into an online match after a patch.

This just goes to show that no matter how good your customer service is or how transparent you are, when you keep your customers waiting too long or if patches degrade performance even the most tolerant start getting impatient. They get to a point where they no longer want to hear promises, they want to see results.

Stardock managed  to still the growing volume of the grumbles when they released the major patch in preperation of the European retail release (which didn't quite happen as advertised, as I wrote previously). There are still some very unhappy people that either still can't play the game, or were able to play online once upon a time and now no longer can. According to Wardell those players are in the 5% minority, however. If the general tone of the responses are anything to go by, Stardock has managed to turn around an increasingly negative sentiment into a positive one reminiscent of the first daily update blogs.

There's no doubt about it, Steam managed to change very entrenched game consumer paradigms. Rather than walking out of a store with a CD (yeah, games were still released on CD back then) or DVD in a crappy DVD case (the move from real boxes and manuals to plastic A5 cases and a brochure-sized sheet with the key on it was already in full swing), Valve had somehow convinced gamers that all they needed were the game files, delivered over the Internet.

Discs (and disks, once upon a time) in boxes are just a distribution mechanism after all, and people other than publishers were beginning to understand that. Or maybe Valve's relentless marketing made us understand that. It's difficult to tell who's idea it was. Was Valve responding to consumer demand or had they developed a solution and then tried to explain the problem to everyone? The fact that Valve blinded most consumers to the fact that Steam turned any normal game release into Software-as-a-Service is a topic for another rant.

Demigod's retail release date has been postponed another week in many parts of the world. The international retail release date of Demigod was set to be 15 May 2009 (this past Friday), but Atari seems to have told local distributors to only expect to release the game this coming Friday (22 May 2009). This without an official statement from either Atari or Stardock except for a brief mention in Frogboy/Draginol/Brad Wardell's Demigod Journals.

While it turned out that video didn't kill the radio star it's difficult to envision retail surviving in an age where consumers can buy straight from the producers. Perhaps a new supply chain will emerge? After all, even the largest publishers need distributors to handle local (and localised) advertising for them. But what will the effect be on the industry globally?

Oh, and my apologies to The Buggles (even though it was the Presidents of the United States of America version playing in my head as I wrote this).