Jump to content
  • Sign Up

Inside ArenaNet: Live Game Outage Analysis


Recommended Posts

As someone who gets called into a number of "Live issues" for a large service, I feel your pain (I was up at 0630 this morning by coincidence), and the eternal battle of Dev / Live not matching never ends :D

 

It's great to read open and honest writeups of these sorts of issues, I find it builds trust in the company. When Brian Harry was at Microsoft he used to provide them for Azure Dev Ops outages and they were always facinating reads.

 

Keep up the good work

Link to post
Share on other sites
  • Replies 113
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Go behind the scenes with Platform Team Lead Robert Neckorcuk.  

Hi All,   Many thanks for the feedback about this post and communication in general! I really enjoyed preparing this content - I hope my passion for this game and community showed through! T

I saw the post and thought "this is new?!". Read the entire post and loved it, thanks for the insight Robert!   Is it just me or is something.. happening/different at ANET the past few month

Posted Images

I absolutely loved the whole post! I work in a live game team too and it's always great to learn more about other similar project, especially Guild Wars 2!

 

Please don't hesitate to write more such posts, I'd love to see more of them. ❤️

Link to post
Share on other sites

As someone who works in IT and has recently been exposed to AWS. I would love to know more about the environments and how they're setup. Obviously not actual details, but things like if any of the services run on a Kubernetes Cluster etc. 😁

Link to post
Share on other sites

i was sooooo annoyed, always seemed to be halfway around a Griffon time trial, DC, able to log back in 10 minutes to find myself dead and miles away lol. As a player only interested in flying Griffon and socialising i never stopped to think WHAT actually goes on at Arenanet behind the scenes, for some reason i imagined some old guy sitting at a desk checking the game lights stayed green, your post was a real eye opener and although i,m not a techie i found the post EASY to read AND entertaining (congrats on that btw). So to any Anet people who's ear were burning from my cussin and temper tantrums, i APOLOGISE.   Thank you guys for your hard work and dedication. JB

Link to post
Share on other sites

Ahhh!! I love reading about this stuff. Not only is the transparency extremely refreshing, it is nice to learn how different circumstances are managed. 

One of the things that I've always been consistently amazed by is the server tech that goes into GW2 and how the patch system works. The near zero downtime makes the game a lot more enjoyable to know it's most likely going to be there when I wake up or when I want to play, and it's nice to learn more about it. 

As someone aspiring to land a job in game programming, this just makes my inner nerd happy to read. 

Would love to see more back-end transparency posts like this! 

Thanks!

Link to post
Share on other sites
4 hours ago, Fragglerock.4965 said:

Very unrealistic write up...

 

as if having a database just gives 1% of cool... much closer to like 3%!

It wasn't a measure of how cool databases are, but how much of their functionality is being cool vs. storing data.

 

You'd need another measure for how cool they are, probably a chart, or a series of infographics.

 

Bonus cool factor: you could create another database to compile information on how comparatively cool different things are, and then you'll have another database. So like, 6% cool. 😸

  • Like 1
  • Haha 2
Link to post
Share on other sites
5 hours ago, Dave Shaw.2856 said:

and the eternal battle of Dev / Live not matching never ends 😄

As a release guy, I worked at this one place and the devs were notorious for always saying "it worked on my box" as if their work laptop was a perfect replica of the Production environment for which their code would never work for the first time out. Luckily we always caught that in Stage.

 

Glad to not be there anymore. They didn't give Release much authority which sort of negates a huge portion of what that role does. I work in an agile shop now and the culture is great. Don't play the blame game when stuff goes down, just work it out and fix it. And worked out our release process to not be a roadblock when you encounter these production outages/issues. Service Management telling you you have to get some high ups approval for an emergency which is essentially a rubber stamp is dumb.

Edited by Faridah.8431
Link to post
Share on other sites

We wanted more communication from developers for a long time. I am glad that recently arenanet is a lot more transparent and is informing us about things they are doing. I really love it, thank you for blog post and keep it up arenanet this is great change i can't wait for future posts.

Link to post
Share on other sites

I was always curious about what caused the problem back then and assumed different scenarios as a software engineer myself and  really hoped, that the damage done wasn't too big.

 

Glad to see, that it turned out well and you could rely on an obviously great team!

Thank you very much for sharing and your effort!

Link to post
Share on other sites

Thank you so much for the transparency behind this posting.  I think before ANYONE throws shade at Anet or AWS they should be required to read and understand this posting.  I've created and run large databases and networks (not Anet traffic levels) and your willingness to share with us what happened, what went right, what went wrong, and how the future will be better because of it was tremendous.

Thank you for everything you do to keep our play time fun and consistent.  I felt your pain with the "restoring is a different matter".  Thankfully, you don't have much practice at restoring.  That's a good thing!

 

 

  • Like 1
Link to post
Share on other sites

As a full time writer myself with well over 2 million words written to date, the blog post for me was kind of short! (LoL) However, that being said... it's good to see Robert giving some inside perspectives on things & all of the hard work that you & your team have been up to! I'm a long time fan of the Guild Wars franchise & I even make gw2 videos about it all on youtube with my wife. If at all interested (just to see what I get up to & hopefully get your tick of approval for fan made characters such as Queen Jennah), please feel free to check it out by looking up >> SweetMimiDoll Adventures on youtube. Fingers crossed you like them! And if you or anyone you know is interested, I have written a ton of fantasy books on Amazon, just look up >> Christopher John Hill << Okay well, I'll finish up by saying stay safe & I wish everyone at Arena Net all the best! I hope you all continue the great work you've been doing & am definitely looking forwards to any future developements within the game itself! I always buy the expansions & I have a feeling you've got some pretty interesting ideas on the table. Personally (and this my own opinion of course just for a great expanding storyline), I'd go with the whole >> Guild Wars 2 - War of the Gods! << idea, (but, that's me!). Alternative universes colliding in on one another might be another way to go as well! Or both idea's mixed together! Okay now I'm getting ahead of myself here! (It's the writer in me) LoL. Peace!

Link to post
Share on other sites

ArenaNet, +5 from me. Though I develop code for distribution systems more than game design, cheers. This article that is oh to familiar of a story made me smile and laugh. Thank you for sharing.

 

https://www.guildwars2.com/en/news/inside-arenanet-live-game-outage-analysis/

 

Oh the joy of working in IT, because as everyone knows IT doesn't mean Information Technology but instead stands for, 'Well it's an it, so that means its IT's issue yes!? Fix it.'

 

lol,

Grimm

  • Like 1
  • Confused 2
Link to post
Share on other sites

As one of those DevOps employees down the road who also gets anxiety-inducing calls at 6 a.m., all of us at Amazon, Microsoft, et al. can absolutely relate to the hard work that goes into collecting as many 9's as possible in your availability numbers.

 

It's great to read this perspective about how things operate on the flip-side, and as both an engineer and a player, I appreciate and admire all the hard work of everyone involved.

Link to post
Share on other sites

Thanks for this post! Read it all, nice insights into the things you normally don't see about the game.

One thought came to mind... at my previous job (yes, I'm in IT 🙃), we never did any major changes on a Friday.

It was usually either Tuesday or Thursday evening (most of the time, and most of our workforce and customers needed access during business hours), so if anything went wrong, we wouldn't have a weekend with less-than-optimal support (yes, you can get or arrange support, but most people will more easily be accessible on Mon-Fri working hours). Now I get that a game isn't business software (like ours), and so what's the best time window will vary.

Keep at it, and keep learning!

Link to post
Share on other sites

That sure is a lot of pressure on your shoulders. The best moment is definitely when the root of evil was discovered, fixed and everything just works. I am still surprised, how often complex and frustrating problems are easily solved by the most basic actions. It is almost like we expect the reasons to fit to the complexity of our work and skip the obvious ones right away. Guilty of that crime myself. 

 

It was quite entertaining and enjoyable to read after a rough and long week. Big thanks for sharing!

 

I would like to read more stories like this, probably from other departments as well. I doubt there is any job in your company, that would be boring to read about. A lot of big and small sprockets keep this place alive and running smoothly. 

Edited by HnRkLnXqZ.1870
removed some extra lines with no content
  • Like 1
Link to post
Share on other sites

I'm sure to be echoing many voices in this thread, but I'm really digging this type of communication from anet to the players. I can't stress how much I've rewatched your videos on YT on behind the scenes of developing the game and the guild chats where we get to meet the developers who dedicate their talent to the world of Tyria. I would definitely encourage more of these blog posts (or videos hint hint) about the game. I'm aware this is an extra effort to the team and with the past few years being quite the hefty blow to your resources i certainly understand that it's not really a priority to the team, but it does really help with rebuilding the connection with the players and developers. As a certain Djinn would be sure to say: "MOAR PLS" is an understatement.  

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...