Jump to content
  • Sign Up

Inside ArenaNet: Live Game Outage Analysis


Recommended Posts

Really great read, and I can so relate.

 

After years of experience with a system, assumed goods start to become "known" goods, then you feel foolish when you finally find the problem.  I have my fair share of thinking that what is assumed to be good is most definitely known to be good..... oh skritt, guess it wasn't good after all...

 

4 hours ago, TheGrimm.5624 said:

Oh the joy of working in IT, because as everyone knows IT doesn't mean Information Technology but instead stands for, 'Well it's an it, so that means its IT's issue yes!? Fix it.'

 

Hear hear.  In the marines, everything that runs on batteries or plugs into a wall must be the comm guy's job.  I have literally gone under the hood of our vehicles because of this line of reasoning. 

"yeah yeah.... they technically do have batteries...I'll get 'em fixed"

 

Link to comment
Share on other sites

2 hours ago, HnRkLnXqZ.1870 said:

That sure is a lot of pressure on your shoulders. The best moment is definitely when the root of evil was discovered, fixed and everything just works. I am still surprised, how often complex and frustrating problems are easily solved by the most basic actions. It is almost like we expect the reasons to fit to the complexity of our work and skip the obvious ones right away. Guilty of that crime myself. 

 

It was quite entertaining and enjoyable to read after a rough and long week. Big thanks for sharing!

 

I would like to read more stories like this, probably from other departments as well. I doubt there is any job in your company, that would be boring to read about. A lot of big and small sprockets keep this place alive and running smoothly.

 

Guilty of this mistake as well. The stuff that rarely ever change are assumed to be good.  Then much later on they get thought of as known goods.  Then your wondering why things are exploding because evferything is seemingly fine when you check them and the pressure is really ramping up to get things done but I still can't figure this out and now I'm thinking in run-on sentences and....oh....I forgot those aren't actually known goods....

 

 

This was awesome Roert.  Hope the staff has the time and passion to write more stories about work.

Link to comment
Share on other sites

I really loved this blog post! As a player it always good to know that a problem is solved for good, and measure was done to make sure it doesn't happen again. But also loved as a programmer, because it's interesting (and useful!) to see what weird problems we can run into 🙂 Congrats for you and for your team!

Link to comment
Share on other sites

This is a great post to share to people who play other MMOs, where the developers aren't nearly as responsive or communicative as ArenaNet (and there are many like that). I think it could cause more people to try playing Guild Wars2.
In other games communities the players are accustomed to getting no timely bug fixes or explanations of problems and they constantly complain about networking problems, and about poor designs that make it difficult to identify and prevent modders from spoiling their game experience. Many that have never played Guild Wars don't seem to realise how good a service can be, and I think it's strange but true that telling them about a problem like this will make them realise how well ArenaNet manages things.
In fact I wish other game company devs would think about outsourcing their server and network design and management to ArenaNet, so that they can continue to focus only on their graphics and new content as they seem to now, but actually make their games playable and more reliable.

Link to comment
Share on other sites

I really appreciate this transparency and hope this will continue in the future.

 

Quote

In the last fourteen months, we’ve recorded an uptime of 99.98%, with only five minor service interruptions impacting user log-ins.

 

Is there any chance we could get a public status page from Arenanet in the future, where we as players could see downtimes and service/performance degradations of important game services?

 

Thanks a lot.

Link to comment
Share on other sites

  • ArenaNet Staff
On 7/16/2021 at 4:26 PM, HotDelirium.7984 said:

I'm more curious about why you only just now were allowed to let the community know what happened. Whatever leadership reorganization/act from divine allowed more transparency from ArenaNet I'm here for it!  Keep it up.

 

 

I can help answer this question! It wasn't a matter of allowed or not, but rather having the idea and making it a reality. As the anniversary of The Event rolled around, Robert thought it would be interesting to break down the events of the day. He checked with us on the Communications Team and we loved the idea!

Writing an extremely long (over three thousand words!), technical post like that takes time. Getting everything written, "translated" into layman's terms to make sure it was easy to follow, putting it through multiple rounds of feedback from other members of the team, polished, edited, scheduled, and posted–in four different languages–is a process

 

Big applause to Robert for making this happen, along with the people on the Platform Team who gave him feedback, our editing team, our localization team, and a lot of others who helped get this out the door to all of you! (Thanks for pitching in!)

  • Like 12
  • Thanks 5
Link to comment
Share on other sites

17 hours ago, Fire Attunement.9835 said:

I can help answer this question! It wasn't a matter of allowed or not, but rather having the idea and making it a reality. As the anniversary of The Event rolled around, Robert thought it would be interesting to break down the events of the day. He checked with us on the Communications Team and we loved the idea!

Writing an extremely long (over three thousand words!), technical post like that takes time. Getting everything written, "translated" into layman's terms to make sure it was easy to follow, putting it through multiple rounds of feedback from other members of the team, polished, edited, scheduled, and posted–in four different languages–is a process

 

Big applause to Robert for making this happen, along with the people on the Platform Team who gave him feedback, our editing team, our localization team, and a lot of others who helped get this out the door to all of you! (Thanks for pitching in!)

The post was fantastic and definitely going the extra mile, I would say though that for whoever would actually want to read the 3000 words, you would not need to translate it into layman's terms, the people who would be interested in reading this likely work in the same or a similar industry and would understand the terminology/technologies.

Link to comment
Share on other sites

On 7/17/2021 at 1:26 AM, HotDelirium.7984 said:

I'm more curious about why you only just now were allowed to let the community know what happened. Whatever leadership reorganization/act from divine allowed more transparency from ArenaNet I'm here for it!  Keep it up.

 

 

Maybe it has something to do with Colin Johanson return to the studio? I have been playing Guild Wars from the very beginning and one of the reasons was ArenaNet's transparency. They always talked to the players and most of all they listened to the players. Recently, something has broken down on this topic. So I hope they will go back to sharing more information with players about game development, not only the positive but also what was difficult for them.

 

PS. I remember the error from the first game when all the prices at the merchants were set to 100 gold (100 gold = kill 2-3 mobs in first game) and a rollback was announced. We had just finished the Thunderhead Keep missions (the most difficult mission in the Prophecy) and we thought we would lose progress because in those days experiences in other MMOs showed that the game was just going back to how it was several hours ago. But not at ArenaNet... They rollback only transactions with NPC and that was awesome.

Link to comment
Share on other sites

I love posts like this, while I've dabbled in some coding and technical stuff it is fun to read about how it's working in a game I love. 

In general I'm very proud of ANETs push to communicate, posts like this and others remind me of the ANET of years back. Thank you for the open communication, it helps us know that you have a good pulse on the community and that our concerns are heard and will be addressed. Status updates even if it's not the place we want it (take Vision, Aurora with the armory as an example) is refreshing and helps us manage expectations. Keep it up! 

Link to comment
Share on other sites

  • I loved the honesty of how you showed a vulnerability.
  •  
  • Understand the importance that this type of dialogue brings to the community. It's very frustrating to be seen only as a number or a dollar sign. When you allow us to enter your inner and see your weaknesses constructively, it strengthens ties with us (players), because we feel there is an exchange here, you enter our homes with the Guild Wars 2 universe and in return, we see a bit of how all gears rotate and work (even if we don't understand much of it).
  •  
  • I would particularly like this type of dialogue to be more frequent, in all areas of the game (one suggestion: I would love to know how the gears run behind the Gift of Aurene (container) and know if there are viable options to drop any more box than guaranteed in the 1000 chest). I feel like there's something good here that could be even bigger. Feel encouraged to come to us, we are ready to receive you.
  •  
  • Forgive me for scratched English 😅, it's not my native language, so I need the help of a translator to be able to communicate.
Link to comment
Share on other sites

This was an awesome write up.

 

I work in the IT infrastructure, cloud and security space, I am always looking for articles on how companies have built and maintain there environments. The more we share the better and more secure the industry becomes. 

 

I wish more posts where made about what happens behind scenes and how the game is architected. 

Really appreciate you sharing this with us all.

  • Like 1
  • Confused 1
Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...