Inside ArenaNet: Live Game Outage Analysis

[DaveInTheEU.9125]

As someone who gets called into a number of "Live issues" for a large service, I feel your pain (I was up at 0630 this morning by coincidence), and the eternal battle of Dev / Live not matching never ends :D

 

It's great to read open and honest writeups of these sorts of issues, I find it builds trust in the company. When Brian Harry was at Microsoft he used to provide them for Azure Dev Ops outages and they were always facinating reads.

 

Keep up the good work

[Vincent.3178]

Thanks for this well written, funny and informative Story. As Someone who was playing the Game on that particular Monday and experienced the "other Side" of these Events, I really enjoy this View from the Inside. Keep it up with these Posts😊

[Fragglerock.4965]

Very unrealistic write up...

 

as if having a database just gives 1% of cool... much closer to like 3%!

[nutri.1385]

I absolutely loved the whole post! I work in a live game team too and it's always great to learn more about other similar project, especially Guild Wars 2!

 

Please don't hesitate to write more such posts, I'd love to see more of them. ❤️

[zallesz.1650]

I really liked the post. Some of it was totally alien to me, but it's nice to read about how you guys work! Would love to see more of these!

[Frenzy.8730]

As someone who works in IT and has recently been exposed to AWS. I would love to know more about the environments and how they're setup. Obviously not actual details, but things like if any of the services run on a Kubernetes Cluster etc. 😁

[Snagglepuss.1278]

i was sooooo annoyed, always seemed to be halfway around a Griffon time trial, DC, able to log back in 10 minutes to find myself dead and miles away lol. As a player only interested in flying Griffon and socialising i never stopped to think WHAT actually goes on at Arenanet behind the scenes, for some reason i imagined some old guy sitting at a desk checking the game lights stayed green, your post was a real eye opener and although i,m not a techie i found the post EASY to read AND entertaining (congrats on that btw). So to any Anet people who's ear were burning from my cussin and temper tantrums, i APOLOGISE.   Thank you guys for your hard work and dedication. JB

[Xenorak.2154]

Ahhh!! I love reading about this stuff. Not only is the transparency extremely refreshing, it is nice to learn how different circumstances are managed. 

One of the things that I've always been consistently amazed by is the server tech that goes into GW2 and how the patch system works. The near zero downtime makes the game a lot more enjoyable to know it's most likely going to be there when I wake up or when I want to play, and it's nice to learn more about it. 

As someone aspiring to land a job in game programming, this just makes my inner nerd happy to read. 

Would love to see more back-end transparency posts like this! 

Thanks!

[Orisein.2058]

i dont understand

[Danikat.8537]

4 hours ago, Fragglerock.4965 said:

Very unrealistic write up...

 

as if having a database just gives 1% of cool... much closer to like 3%!

It wasn't a measure of how cool databases are, but how much of their functionality is being cool vs. storing data.

 

You'd need another measure for how cool they are, probably a chart, or a series of infographics.

 

Bonus cool factor: you could create another database to compile information on how comparatively cool different things are, and then you'll have another database. So like, 6% cool. 😸

[nicoper.3215]

On 7/14/2021 at 11:05 PM, Fire Attunement.9835 said:

Go behind the scenes with Platform Team Lead Robert Neckorcuk.

 

Very interesting read and a great look into the backend of a large, complex system like this.

Would love to see more posts like this in the future!

[Faridah.8431]

5 hours ago, Dave Shaw.2856 said:

and the eternal battle of Dev / Live not matching never ends 😄

As a release guy, I worked at this one place and the devs were notorious for always saying "it worked on my box" as if their work laptop was a perfect replica of the Production environment for which their code would never work for the first time out. Luckily we always caught that in Stage.

 

Glad to not be there anymore. They didn't give Release much authority which sort of negates a huge portion of what that role does. I work in an agile shop now and the culture is great. Don't play the blame game when stuff goes down, just work it out and fix it. And worked out our release process to not be a roadblock when you encounter these production outages/issues. Service Management telling you you have to get some high ups approval for an emergency which is essentially a rubber stamp is dumb.

Edited July 16, 2021 by Faridah.8431

[Mich.7480]

We wanted more communication from developers for a long time. I am glad that recently arenanet is a lot more transparent and is informing us about things they are doing. I really love it, thank you for blog post and keep it up arenanet this is great change i can't wait for future posts.

[Touchme.1097]

Well done, Robert, I like your attitude. I am proud of you 🙂

I really enjoyed your story, hope you are doing well. 

Take care

Edited July 16, 2021 by Touchme.1097

[Kaliba.4591]

I was always curious about what caused the problem back then and assumed different scenarios as a software engineer myself and  really hoped, that the damage done wasn't too big.

 

Glad to see, that it turned out well and you could rely on an obviously great team!

Thank you very much for sharing and your effort!

[bossmitch.8037]

for most, and as other mentioned, the zero-downtime patch is one of the most underrated physics behind the scene that arenanet can boast anywhere. kudos to robert and team. Its a wonderful insight.

[Chance.2346]

Thank you so much for the transparency behind this posting.  I think before ANYONE throws shade at Anet or AWS they should be required to read and understand this posting.  I've created and run large databases and networks (not Anet traffic levels) and your willingness to share with us what happened, what went right, what went wrong, and how the future will be better because of it was tremendous.

Thank you for everything you do to keep our play time fun and consistent.  I felt your pain with the "restoring is a different matter".  Thankfully, you don't have much practice at restoring.  That's a good thing!

 

 

[CJH.2879]

As a full time writer myself with well over 2 million words written to date, the blog post for me was kind of short! (LoL) However, that being said... it's good to see Robert giving some inside perspectives on things & all of the hard work that you & your team have been up to! I'm a long time fan of the Guild Wars franchise & I even make gw2 videos about it all on youtube with my wife. If at all interested (just to see what I get up to & hopefully get your tick of approval for fan made characters such as Queen Jennah), please feel free to check it out by looking up >> SweetMimiDoll Adventures on youtube. Fingers crossed you like them! And if you or anyone you know is interested, I have written a ton of fantasy books on Amazon, just look up >> Christopher John Hill << Okay well, I'll finish up by saying stay safe & I wish everyone at Arena Net all the best! I hope you all continue the great work you've been doing & am definitely looking forwards to any future developements within the game itself! I always buy the expansions & I have a feeling you've got some pretty interesting ideas on the table. Personally (and this my own opinion of course just for a great expanding storyline), I'd go with the whole >> Guild Wars 2 - War of the Gods! << idea, (but, that's me!). Alternative universes colliding in on one another might be another way to go as well! Or both idea's mixed together! Okay now I'm getting ahead of myself here! (It's the writer in me) LoL. Peace!

[TheGrimm.5624]

ArenaNet, +5 from me. Though I develop code for distribution systems more than game design, cheers. This article that is oh to familiar of a story made me smile and laugh. Thank you for sharing.

 

https://www.guildwars2.com/en/news/inside-arenanet-live-game-outage-analysis/

 

Oh the joy of working in IT, because as everyone knows IT doesn't mean Information Technology but instead stands for, 'Well it's an it, so that means its IT's issue yes!? Fix it.'

 

lol,

Grimm

[Finding Sanity.1283]

read it all. interesting what goes on most ppl dont know about. 

[jonnoy.1748]

As one of those DevOps employees down the road who also gets anxiety-inducing calls at 6 a.m., all of us at Amazon, Microsoft, et al. can absolutely relate to the hard work that goes into collecting as many 9's as possible in your availability numbers.

 

It's great to read this perspective about how things operate on the flip-side, and as both an engineer and a player, I appreciate and admire all the hard work of everyone involved.

[Gilthas.6523]

Thanks for this post! Read it all, nice insights into the things you normally don't see about the game.

One thought came to mind... at my previous job (yes, I'm in IT 🙃), we never did any major changes on a Friday.

It was usually either Tuesday or Thursday evening (most of the time, and most of our workforce and customers needed access during business hours), so if anything went wrong, we wouldn't have a weekend with less-than-optimal support (yes, you can get or arrange support, but most people will more easily be accessible on Mon-Fri working hours). Now I get that a game isn't business software (like ours), and so what's the best time window will vary.

Keep at it, and keep learning!

[HnRkLnXqZ.1870]

That sure is a lot of pressure on your shoulders. The best moment is definitely when the root of evil was discovered, fixed and everything just works. I am still surprised, how often complex and frustrating problems are easily solved by the most basic actions. It is almost like we expect the reasons to fit to the complexity of our work and skip the obvious ones right away. Guilty of that crime myself. 

 

It was quite entertaining and enjoyable to read after a rough and long week. Big thanks for sharing!

 

I would like to read more stories like this, probably from other departments as well. I doubt there is any job in your company, that would be boring to read about. A lot of big and small sprockets keep this place alive and running smoothly. 

Edited July 17, 2021 by HnRkLnXqZ.1870
removed some extra lines with no content

[AricanaRainbow.7145]

LOVED the nerdy techy stuff. Of course I understand you can't disclose certain things, but I'm pretty sure you're running Linux servers 😉 (AWS has a choice of OSes).

[Hydruka.5716]

I'm sure to be echoing many voices in this thread, but I'm really digging this type of communication from anet to the players. I can't stress how much I've rewatched your videos on YT on behind the scenes of developing the game and the guild chats where we get to meet the developers who dedicate their talent to the world of Tyria. I would definitely encourage more of these blog posts (or videos hint hint) about the game. I'm aware this is an extra effort to the team and with the past few years being quite the hefty blow to your resources i certainly understand that it's not really a priority to the team, but it does really help with rebuilding the connection with the players and developers. As a certain Djinn would be sure to say: "MOAR PLS" is an understatement.