Jump to content
  • Sign Up

Render Server Outage


Recommended Posts

  • ArenaNet Staff

Hello,

I know you all have noticed the API breaking frequently recently and I wanted to write a post about it to help describe exactly what parts are breaking for clarity.

render.guildwars2.com is a piece of the API that converts game asset images into jpgs and pngs for the public to use in their apps and sites. It's backed by a pretty simple standalone piece of server software whose sole responsibility is to cache the assets, convert them, and serve them as images.

This server has been getting stuck in a bad state about every other month or so for a while now (I think a little over a year). It has been on my backlog of things to fix, but since the solution is a gentle reboot now-and-then, I admit I flagged it as low priority.

Over the last few weeks, for reasons I haven't yet figured out, it's been getting stuck very frequently, sometimes twice in a day and at least once every weekend. It's gotten so bad that due to server configuration reasons, it wiped the white list server which had to be recovered manually (which is why the item endpoint died recently). Thank you for your reports of failure on the forum. I have been using these reports to manually fix the server while we try to work on a real solution.

The priority for fixing render.guildwars2.com has been bumped up and is being actively worked on. The current status is debugging and gathering info on why it's breaking. As soon as we know, we'll fix it as soon as possible.

Thanks again for your reports, and thank you for your patience while we try to put out this fire for good.

-Snider

Link to comment
Share on other sites

  • ArenaNet Staff

Hello,

I'd like to give a quick update on what's going on with render.guildwars2.com just so there's clarity into the progress being made.

The Short Version

render.guildwars2.com is still broken, but we're working on it. We'll understand the problem better after tomorrow's patch.

The Longer Version

The code behind render.guildwars2.com has two changes that will go out with the deploy tomorrow.

First is better logging. Internally we send a signal to the server to have it "go to sleep" and "wake back up". It's going to sleep, but only parts of it are waking back up which puts us into a bad state. We've improved the logs so that we can see how we're getting into this state and why.

While in this strange zombie state, the server is leaking system resources. An innocent victim of this is API white list server which figuratively explodes when it gets starved of the resource (there is an independent investigation to improve the robustness of the white list server). The second change is that the resource leak is fixed.

What Does All This Mean?

  • render.guildwars2.com will still be broken until we solve the root cause.
  • render.guildwars2.com won't cause the API white list to lose its data at random times anymore.

Thank you everyone for your patience while we solve this issue.

-Snider

Link to comment
Share on other sites

@Ashantara.8731 said:

@"VAHNeunzehnsechundsiebzig.3618" said:usually a zombie is a process that has died but was not reaped by its parent process.

Or by some evil necromancer. ;)

@Daniel Snider.6241, thank you so much for keeping us up to date! Very interesting insights, too. :)

and now I feel like I should make a Reaper called Init or Pid1

Link to comment
Share on other sites

Thanks for the update!

@"Daniel Snider.6241" said:render.guildwars2.com won't cause the API white list to lose its data at random times anymore.

I noticed that some items that were previously in the API now return Not Found. Is that related? If it can be any help, here's a snapshot that I took a while ago. https://github.com/StevenLiekens/gw2sdk/blob/de79bd6dadb7d9c478abb20f62645c8f279cb93a/GW2SDK.Tests/Data/items.json

Link to comment
Share on other sites

I've noticed the problem hasn't been fixed. The API is duck-taped from time-to-time, but it has yet to be fixed. It's actually kind of starting to drive me crazy. I'm pretty furious at this point, and it's been a week since we've heard anything and nearly a month since the problem started to arise and was brought up. I see patches being sent out... bug fixes... new content... There doesn't seem to be a problem putting things into the game that could cost the player more of their hard-earned dollars, and yet the render server/API problem seems to be unimportant to the dev team?...

Link to comment
Share on other sites

@Broadway, it affects community-driven sites like gw2armory, Snow Crows, metabattle. It's difficult to share builds with friends at this time, for sure, but at least the actual in-game experience is unaffected.

Some progress is being made =)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...