ArenaNet Staff Daniel Snider.6241 Posted June 3, 2019 ArenaNet Staff Share Posted June 3, 2019 Hello,I know you all have noticed the API breaking frequently recently and I wanted to write a post about it to help describe exactly what parts are breaking for clarity.render.guildwars2.com is a piece of the API that converts game asset images into jpgs and pngs for the public to use in their apps and sites. It's backed by a pretty simple standalone piece of server software whose sole responsibility is to cache the assets, convert them, and serve them as images.This server has been getting stuck in a bad state about every other month or so for a while now (I think a little over a year). It has been on my backlog of things to fix, but since the solution is a gentle reboot now-and-then, I admit I flagged it as low priority.Over the last few weeks, for reasons I haven't yet figured out, it's been getting stuck very frequently, sometimes twice in a day and at least once every weekend. It's gotten so bad that due to server configuration reasons, it wiped the white list server which had to be recovered manually (which is why the item endpoint died recently). Thank you for your reports of failure on the forum. I have been using these reports to manually fix the server while we try to work on a real solution.The priority for fixing render.guildwars2.com has been bumped up and is being actively worked on. The current status is debugging and gathering info on why it's breaking. As soon as we know, we'll fix it as soon as possible.Thanks again for your reports, and thank you for your patience while we try to put out this fire for good.-Snider Link to comment Share on other sites More sharing options...
SpecterMAT.7306 Posted June 3, 2019 Share Posted June 3, 2019 You are a cool person. Thanks for the post and I wish you good luck to find that nasty thing! Link to comment Share on other sites More sharing options...
Pifil.5193 Posted June 3, 2019 Share Posted June 3, 2019 Thanks Daniel, we really appreciate it! Link to comment Share on other sites More sharing options...
Illconceived Was Na.9781 Posted June 3, 2019 Share Posted June 3, 2019 Thanks for the update. That's great background for those of curious about the details. Link to comment Share on other sites More sharing options...
WolfsFang.2301 Posted June 3, 2019 Share Posted June 3, 2019 Massive thumbs up for the transparency! Hope you can get this resolved. Link to comment Share on other sites More sharing options...
GHawkR.7231 Posted June 4, 2019 Share Posted June 4, 2019 Thanks for getting us informed! Indeed we wish you best luck on solving this! Link to comment Share on other sites More sharing options...
Curufindul.1286 Posted June 4, 2019 Share Posted June 4, 2019 That explains a lot. Thanks for the notice! I hope you can get that fixed soon. Link to comment Share on other sites More sharing options...
DragonHeart.7201 Posted June 4, 2019 Share Posted June 4, 2019 Thanks for the response, good luck fixing it. Link to comment Share on other sites More sharing options...
Vegeta.2563 Posted June 4, 2019 Share Posted June 4, 2019 I hope this is the cause of why map tiles aren't updating neither. Like any map that released after PoF, from Istan to Dragonfall. Link to comment Share on other sites More sharing options...
Cameryn.5310 Posted June 6, 2019 Share Posted June 6, 2019 I don't suppose there's any way to like, you know, keep rebooting it but leave it active while you're working out the details, right? It's become extremely hard to use Metabattle and almost impossible to use GW2Skills.net in the meanwhile. :( Link to comment Share on other sites More sharing options...
Tekoneiric.6817 Posted June 7, 2019 Share Posted June 7, 2019 Thanks for the update! Being a tech IRL I love hearing about the details behind the scenes. Link to comment Share on other sites More sharing options...
ArenaNet Staff Daniel Snider.6241 Posted June 10, 2019 Author ArenaNet Staff Share Posted June 10, 2019 Hello,I'd like to give a quick update on what's going on with render.guildwars2.com just so there's clarity into the progress being made.The Short Versionrender.guildwars2.com is still broken, but we're working on it. We'll understand the problem better after tomorrow's patch.The Longer VersionThe code behind render.guildwars2.com has two changes that will go out with the deploy tomorrow.First is better logging. Internally we send a signal to the server to have it "go to sleep" and "wake back up". It's going to sleep, but only parts of it are waking back up which puts us into a bad state. We've improved the logs so that we can see how we're getting into this state and why.While in this strange zombie state, the server is leaking system resources. An innocent victim of this is API white list server which figuratively explodes when it gets starved of the resource (there is an independent investigation to improve the robustness of the white list server). The second change is that the resource leak is fixed.What Does All This Mean?render.guildwars2.com will still be broken until we solve the root cause.render.guildwars2.com won't cause the API white list to lose its data at random times anymore.Thank you everyone for your patience while we solve this issue.-Snider Link to comment Share on other sites More sharing options...
Illconceived Was Na.9781 Posted June 10, 2019 Share Posted June 10, 2019 Very helpful update, @"Daniel Snider.6241". I like reading about the extra details.(Also :+1: for the term "strange zombie state.") Link to comment Share on other sites More sharing options...
VAHNeunzehnsechundsiebzig. Posted June 10, 2019 Share Posted June 10, 2019 usually a zombie is a process that has died but was not reaped by its parent process. Link to comment Share on other sites More sharing options...
Broadway.8925 Posted June 11, 2019 Share Posted June 11, 2019 OMG ty so much for the explanations and for working on this for us. It's been a bothersome pest for the past few weeks. I was afraid it was going unattended. I wish you the best of luck and hope to see everything figured out and fixed soon. GW2 is life Link to comment Share on other sites More sharing options...
Healy Unit.1023 Posted June 11, 2019 Share Posted June 11, 2019 @Daniel Snider.6241 Thanks for the update, and I hope it's not too much of a headache for everyone! Just out of curiosity, would a normal zombie state be something like the setting of Walking Dead, or Nebraska? Link to comment Share on other sites More sharing options...
Ashantara.8731 Posted June 11, 2019 Share Posted June 11, 2019 @"VAHNeunzehnsechundsiebzig.3618" said:usually a zombie is a process that has died but was not reaped by its parent process.Or by some evil necromancer. ;)@Daniel Snider.6241, thank you so much for keeping us up to date! Very interesting insights, too. :) Link to comment Share on other sites More sharing options...
VAHNeunzehnsechundsiebzig. Posted June 11, 2019 Share Posted June 11, 2019 @Ashantara.8731 said:@"VAHNeunzehnsechundsiebzig.3618" said:usually a zombie is a process that has died but was not reaped by its parent process.Or by some evil necromancer. ;)@Daniel Snider.6241, thank you so much for keeping us up to date! Very interesting insights, too. :)and now I feel like I should make a Reaper called Init or Pid1 Link to comment Share on other sites More sharing options...
Broadway.8925 Posted June 11, 2019 Share Posted June 11, 2019 Just curious. Was anything actually fixed with the patch? Link to comment Share on other sites More sharing options...
ArenaNet Staff Daniel Snider.6241 Posted June 11, 2019 Author ArenaNet Staff Share Posted June 11, 2019 @Broadway.8925 said:Just curious. Was anything actually fixed with the patch?Without promising anything, the outlook is good. :) Link to comment Share on other sites More sharing options...
Vegeta.2563 Posted June 12, 2019 Share Posted June 12, 2019 @Daniel Snider.6241 said:@Broadway.8925 said:Just curious. Was anything actually fixed with the patch?Without promising anything, the outlook is good. :)Just as long as the API doesn't break every single weekend and can remain stable. Link to comment Share on other sites More sharing options...
StevenL.3761 Posted June 12, 2019 Share Posted June 12, 2019 Thanks for the update!@"Daniel Snider.6241" said:render.guildwars2.com won't cause the API white list to lose its data at random times anymore.I noticed that some items that were previously in the API now return Not Found. Is that related? If it can be any help, here's a snapshot that I took a while ago. https://github.com/StevenLiekens/gw2sdk/blob/de79bd6dadb7d9c478abb20f62645c8f279cb93a/GW2SDK.Tests/Data/items.json Link to comment Share on other sites More sharing options...
xlion.3065 Posted June 16, 2019 Share Posted June 16, 2019 but since the solution is a gentle reboot now-and-thenThis is a very strange understanding of what a solution is and explains a lot. Link to comment Share on other sites More sharing options...
Broadway.8925 Posted June 19, 2019 Share Posted June 19, 2019 I've noticed the problem hasn't been fixed. The API is duck-taped from time-to-time, but it has yet to be fixed. It's actually kind of starting to drive me crazy. I'm pretty furious at this point, and it's been a week since we've heard anything and nearly a month since the problem started to arise and was brought up. I see patches being sent out... bug fixes... new content... There doesn't seem to be a problem putting things into the game that could cost the player more of their hard-earned dollars, and yet the render server/API problem seems to be unimportant to the dev team?... Link to comment Share on other sites More sharing options...
icy.9250 Posted June 20, 2019 Share Posted June 20, 2019 @Broadway, it affects community-driven sites like gw2armory, Snow Crows, metabattle. It's difficult to share builds with friends at this time, for sure, but at least the actual in-game experience is unaffected.Some progress is being made =) Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now