ArenaNet Staff Popular Post Fire Attunement.9835 Posted July 14, 2021 ArenaNet Staff Popular Post Share Posted July 14, 2021 Go behind the scenes with Platform Team Lead Robert Neckorcuk. 50 37 Link to comment Share on other sites More sharing options...
Michram.6853 Posted July 15, 2021 Share Posted July 15, 2021 I loved it 🙂 Always cool to read behind-the-scenes stuff! 16 Link to comment Share on other sites More sharing options...
anduriell.6280 Posted July 15, 2021 Share Posted July 15, 2021 I really like the post! Really great story telling from behind-the-curtain and very relatable if you worked in any IT field. I would love to see more! 9 Link to comment Share on other sites More sharing options...
Shaft Master.9261 Posted July 15, 2021 Share Posted July 15, 2021 Very neat to be able to peak behind the scenes! Cool post 🙂 5 Link to comment Share on other sites More sharing options...
Qtto.9024 Posted July 15, 2021 Share Posted July 15, 2021 Very cool, I always love a post-mortem on these kinds of events! Quote The root cause was the drivers. Drivers... The root of all evil! 8 5 Link to comment Share on other sites More sharing options...
Avenrise.3592 Posted July 15, 2021 Share Posted July 15, 2021 I saw the post and thought "this is new?!". Read the entire post and loved it, thanks for the insight Robert! Is it just me or is something.. happening/different at ANET the past few months? Informative and fun peaks behind the scenes, a ramp up of communication via social media, content patch posts that are precise and to the point giving locations to travel to...even commitments to a long term strategy for GW2. I know EoD is coming up and I was on my 5 year 'sabbatical' during HoT and PoF launches but I am currently REALLY liking the communication right now. Add in some decent advertising and I think GW2 has a good future... which is the opposite of what I thought post-IBS. Oh, and just have to laugh that it was a driver issue! Even huge games companies struggle like us commoners 😄 17 Link to comment Share on other sites More sharing options...
ReaverKane.7598 Posted July 15, 2021 Share Posted July 15, 2021 Thanks for the explanation, it was actually very interesting, and its great to read these behind the scenes content. I still think the way Arena Net "compensated" players for the down time was a bit disrespectful of those that were mindful of your work and didn't want to add extra pressure on the servers by attempting to log in. But hopefully the team behind that decision might have learned something then. I've got to say i'm greatly enjoying this new Arena Net, keep up the good work you've been doing these last few months. 9 Link to comment Share on other sites More sharing options...
Arnox.5128 Posted July 15, 2021 Share Posted July 15, 2021 Love this communication! Please keep it up! ❤️ 8 Link to comment Share on other sites More sharing options...
slashouse.1374 Posted July 15, 2021 Share Posted July 15, 2021 Very interesting piece! I love getting to know the technical side of things, especially on AWS. My team went through that Auric Basin picnic situation with the DB space filling up suddenly, investigate, patch, and release. You guys actually make me want to check these driver matter with our DBs 😄 Anyway, keep these kind of behind-the-scene stories coming 6 Link to comment Share on other sites More sharing options...
cryorion.9532 Posted July 15, 2021 Share Posted July 15, 2021 True Gods of Tyria. 3 Link to comment Share on other sites More sharing options...
arravilo.9208 Posted July 15, 2021 Share Posted July 15, 2021 Extremely appreciated. I'm supper happy with Anet and all its team. From the day it happened I wish I could see a retrospective like this, but I could also understand that you'd keep this as internal. You've made my wish come true, and I love it. Thanks a lot for communicating this and being transparent. 4 Link to comment Share on other sites More sharing options...
Adzekul.3104 Posted July 15, 2021 Share Posted July 15, 2021 As an EU player who remembers (distantly now) the outage, I am thankful for the communication and the content of the post (yes, I read it all). Thank you. 3 Link to comment Share on other sites More sharing options...
Twin Haelix.8619 Posted July 15, 2021 Share Posted July 15, 2021 This is phenomenal, and I feel like it signals a desperately-wanted openness from the people who make the game. I feel like this was an excellent first iteration of what I hope becomes a regular series -- "Inside ArenaNet". This reminds me strongly of the blog posts from the creators of one of my other favorite games, Factorio. The developers have a long-running blog called Factorio Friday Facts (FFF) with hundreds of posts. They speak openly about the bugs and issues they've worked on in the game, and communicate clearly what they did to address it. And the FFF posts are *BELOVED* by the Factorio community. I would really, really love to see some sort of regular "Inside ArenaNet" post showing up, and I know others would be equally excited to have such an opportunity to learn more about what makes the game tick. Thank you so much for posting this! 6 4 Link to comment Share on other sites More sharing options...
Urud.4925 Posted July 15, 2021 Share Posted July 15, 2021 Hey, don't be too good with this swapping servers! Last time that we had a server down for half a day, we got a selectable mount skin :D Joking, you always do a very good job. Other videogame companies should take example from Arenanet, to learn how a game can be mantained without weekly disconnections. 3 Link to comment Share on other sites More sharing options...
Lan Deathrider.5910 Posted July 15, 2021 Share Posted July 15, 2021 Some unsung heroes right there. Great work! 4 Link to comment Share on other sites More sharing options...
Azulean.1728 Posted July 15, 2021 Share Posted July 15, 2021 I'd most definitely love to see more of these kinds of blogs. Also, very fitting image 😂 2 Link to comment Share on other sites More sharing options...
KazukoHimura.2314 Posted July 15, 2021 Share Posted July 15, 2021 Reading this sort of thing is incredibly interesting, especially for someone like me who has an IT background. 1 Link to comment Share on other sites More sharing options...
Araun.1638 Posted July 15, 2021 Share Posted July 15, 2021 (edited) Personally I consider such threads very interesting - it shows that there are also people on the another side who really do care about the game and the community. And trust me - also the community cares about the game and YOU. Additionally, as an apprentice to the "dark arts of IT" I can still learn something new - fantastic. I have you say, with your recent offensive regarding the posts, ideas and art of work you are getting me back to the franchise I love since...15 years - exact half of my life. After Champions flop I lost my hope, now you are giving it back to me that there is still place for me and many other people. Keep the good work up! We want to know more about all of you! Take care! Edited July 15, 2021 by Araun.1638 6 Link to comment Share on other sites More sharing options...
Nightshade.2570 Posted July 15, 2021 Share Posted July 15, 2021 20 hours ago, Fire Attunement.9835 said: Go behind the scenes with Platform Team Lead Robert Neckorcuk. Wow I mean wow! The first thing I said is this is amazing, the amount of info you shared with us was wow. I really appreciate this background look. I immediately consulted friends that worked in similar fields and passed this along. Why? Because what you learned in this episode will probably help them similarly. Thankyou for putting such time and passion into this! What a refreshing change and great amount of Knowledge. I laughed so hard when the root cause was *drivers* ahhahahahahha. 3 Link to comment Share on other sites More sharing options...
kokiman.2364 Posted July 15, 2021 Share Posted July 15, 2021 "We’d love to read your thoughts and feedback about this type of post, so we’ve set up a discussion thread on our official forums!" I absolutely love this type of content. The company I work for has a very open blameless culture, we have weekly "blameless portmortems" in which basically talk about our incidents in detail. I learn soooo much from these meetings. In this case, you guys obviously need to be a bit more vague about everything but I still enjoyed this openness that you usually only see from opensource communities. To the story itself: I feel you guys. Being involved in database incidents is never fun 😄 But you also learn a lot from it! For example seeing 2,147,483,647 in a database table as an sequence id is usually not good, especially if legacy software is involved 😛 4 Link to comment Share on other sites More sharing options...
Chaba.5410 Posted July 15, 2021 Share Posted July 15, 2021 (edited) Ah, the eternal struggle of trying to keep dev, staging, and prod all concurrent. Some things never change. Edited July 15, 2021 by Chaba.5410 1 Link to comment Share on other sites More sharing options...
Lascax.2163 Posted July 15, 2021 Share Posted July 15, 2021 I love this kind of posts! They're a very good insight on what happens behind online games and to remember how great is the always-online model of GW2 servers! Please, do more of these: I'll surely read them. 3 Link to comment Share on other sites More sharing options...
Funky Jah.4310 Posted July 15, 2021 Share Posted July 15, 2021 Great post! As a member of the "IT crowd" myself, I can very much relate to this story. Fun fact: during the outage, I recall discussing the potential cause of the database error with another IT specialist guildmate, using only the few clues provided by the error messages or blog posts that we had at hand. Reading this post-mortem made me very pleased to realize that the actual issue was no trivial matter, as we had feared at that time. It shows how good your team and your infrastructure are. Anyway, I would love to read more posts like this one! 2 Link to comment Share on other sites More sharing options...
Infusion.7149 Posted July 15, 2021 Share Posted July 15, 2021 You know of all the things you can possibly complain about , outages aren't really one of them. Even before the AWS migration, uptime has been stellar. 1 Link to comment Share on other sites More sharing options...
Vael Victus.2654 Posted July 15, 2021 Share Posted July 15, 2021 As someone who just about managed to get a Digital Ocean droplet running MySQL 8, I salute you. 1 1 Link to comment Share on other sites More sharing options...
Recommended Posts