Render Server Outage — Guild Wars 2 Forums

Render Server Outage

Hello,

I know you all have noticed the API breaking frequently recently and I wanted to write a post about it to help describe exactly what parts are breaking for clarity.

render.guildwars2.com is a piece of the API that converts game asset images into jpgs and pngs for the public to use in their apps and sites. It's backed by a pretty simple standalone piece of server software whose sole responsibility is to cache the assets, convert them, and serve them as images.

This server has been getting stuck in a bad state about every other month or so for a while now (I think a little over a year). It has been on my backlog of things to fix, but since the solution is a gentle reboot now-and-then, I admit I flagged it as low priority.

Over the last few weeks, for reasons I haven't yet figured out, it's been getting stuck very frequently, sometimes twice in a day and at least once every weekend. It's gotten so bad that due to server configuration reasons, it wiped the white list server which had to be recovered manually (which is why the item endpoint died recently). Thank you for your reports of failure on the forum. I have been using these reports to manually fix the server while we try to work on a real solution.

The priority for fixing render.guildwars2.com has been bumped up and is being actively worked on. The current status is debugging and gathering info on why it's breaking. As soon as we know, we'll fix it as soon as possible.

Thanks again for your reports, and thank you for your patience while we try to put out this fire for good.

-Snider

Tagged:

Comments

  • You are a cool person. Thanks for the post and I wish you good luck to find that nasty thing!

  • Pifil.5193Pifil.5193 Member ✭✭✭✭

    Thanks Daniel, we really appreciate it!

  • Thanks for the update. That's great background for those of curious about the details.

    "Face the facts. Then act on them. It's ...the only doctrine I have to offer you, & it's harder than you'd think, because I swear humans seem hardwired to do anything but. Face the facts. Don't pray, don't wish, ...FACE THE FACTS. THEN act." — Quellcrist Falconer

  • Massive thumbs up for the transparency! Hope you can get this resolved.

  • Thanks for getting us informed! Indeed we wish you best luck on solving this!

  • That explains a lot. Thanks for the notice! I hope you can get that fixed soon.

  • Thanks for the response, good luck fixing it.

  • Vegeta.2563Vegeta.2563 Member ✭✭✭

    I hope this is the cause of why map tiles aren't updating neither. Like any map that released after PoF, from Istan to Dragonfall.

  • Cameryn.5310Cameryn.5310 Member ✭✭✭

    I don't suppose there's any way to like, you know, keep rebooting it but leave it active while you're working out the details, right? It's become extremely hard to use Metabattle and almost impossible to use GW2Skills.net in the meanwhile. :(

  • Tekoneiric.6817Tekoneiric.6817 Member ✭✭✭✭

    Thanks for the update! Being a tech IRL I love hearing about the details behind the scenes.

  • Very helpful update, @Daniel Snider.6241. I like reading about the extra details.
    (Also :+1: for the term "strange zombie state.")

    "Face the facts. Then act on them. It's ...the only doctrine I have to offer you, & it's harder than you'd think, because I swear humans seem hardwired to do anything but. Face the facts. Don't pray, don't wish, ...FACE THE FACTS. THEN act." — Quellcrist Falconer

  • usually a zombie is a process that has died but was not reaped by its parent process.

  • OMG ty so much for the explanations and for working on this for us. It's been a bothersome pest for the past few weeks. I was afraid it was going unattended. I wish you the best of luck and hope to see everything figured out and fixed soon. GW2 is life

  • @Daniel Snider.6241 Thanks for the update, and I hope it's not too much of a headache for everyone! Just out of curiosity, would a normal zombie state be something like the setting of Walking Dead, or Nebraska?

  • Ashantara.8731Ashantara.8731 Member ✭✭✭✭

    @VAHNeunzehnsechundsiebzig.3618 said:
    usually a zombie is a process that has died but was not reaped by its parent process.

    Or by some evil necromancer. ;)

    @Daniel Snider.6241, thank you so much for keeping us up to date! Very interesting insights, too. :)

  • @Ashantara.8731 said:

    @VAHNeunzehnsechundsiebzig.3618 said:
    usually a zombie is a process that has died but was not reaped by its parent process.

    Or by some evil necromancer. ;)

    @Daniel Snider.6241, thank you so much for keeping us up to date! Very interesting insights, too. :)

    and now I feel like I should make a Reaper called Init or Pid1

  • Just curious. Was anything actually fixed with the patch?

  • @Broadway.8925 said:
    Just curious. Was anything actually fixed with the patch?

    Without promising anything, the outlook is good. :)

  • Vegeta.2563Vegeta.2563 Member ✭✭✭

    @Daniel Snider.6241 said:

    @Broadway.8925 said:
    Just curious. Was anything actually fixed with the patch?

    Without promising anything, the outlook is good. :)

    Just as long as the API doesn't break every single weekend and can remain stable.

  • StevenL.3761StevenL.3761 Member ✭✭

    Thanks for the update!

    @Daniel Snider.6241 said:
    render.guildwars2.com won't cause the API white list to lose its data at random times anymore.

    I noticed that some items that were previously in the API now return Not Found. Is that related? If it can be any help, here's a snapshot that I took a while ago. https://github.com/StevenLiekens/gw2sdk/blob/de79bd6dadb7d9c478abb20f62645c8f279cb93a/GW2SDK.Tests/Data/items.json

  • xlion.3065xlion.3065 Member ✭✭

    but since the solution is a gentle reboot now-and-then

    This is a very strange understanding of what a solution is and explains a lot.

  • Broadway.8925Broadway.8925 Member ✭✭
    edited June 19, 2019

    I've noticed the problem hasn't been fixed. The API is duck-taped from time-to-time, but it has yet to be fixed. It's actually kind of starting to drive me crazy. I'm pretty furious at this point, and it's been a week since we've heard anything and nearly a month since the problem started to arise and was brought up. I see patches being sent out... bug fixes... new content... There doesn't seem to be a problem putting things into the game that could cost the player more of their hard-earned dollars, and yet the render server/API problem seems to be unimportant to the dev team?...

  • icy.9250icy.9250 Member ✭✭

    @Broadway, it affects community-driven sites like gw2armory, Snow Crows, metabattle. It's difficult to share builds with friends at this time, for sure, but at least the actual in-game experience is unaffected.

    Some progress is being made =)

©2010–2018 ArenaNet, LLC. All rights reserved. Guild Wars, Guild Wars 2, Heart of Thorns, Guild Wars 2: Path of Fire, ArenaNet, NCSOFT, the Interlocking NC Logo, and all associated logos and designs are trademarks or registered trademarks of NCSOFT Corporation. All other trademarks are the property of their respective owners.