Response to 'performance issues'

Since we can't reply to announcement threads...

I work in the networking and computer industry. You all know you moved your servers to the cloud to, and I air quote, "save money". This is your problem. Move them back in house -- hire real IT devs to track performance, and infrastructure. Rent at at high capacity datacenters or rebuild your own. You don't control server -> client you will never solve the issue.

Just to add color our company also rents a lot of aws instances, and the trap here is the price keeps going up, but you've already move it! So now it's stuck there without a lot of work. For low - traffic stuff, this is ok, but mission critical, it's rarely worth it vs keeping it in house.

This is the announcement to which the thread refers: https://en-forum.guildwars2.com/discussion/111982/an-update-on-game-performance-issues for those who haven't seen it.

Since the move was several years ago, why weren't there issues at that time?

The fact is that aws provides a great monitoring infrastructure and sometimes good response, but sometimes they get infrastructure issues. Basically you have to wait for them to work through it. We all remember a few aws outages, that all we can do is sit back and watch.

Does this address the wings that are causing the huge hit in performance across the glove. They use an un-optimized fire effect that is only used on the new wings and the Skycale which is reaching havoc on FPS. This is incredibly noticeable in group events. The new area is not the problem..it's all the new wings killing performance in it that is causing the issue. This could easily be resolved by changing the flame effect to the Primeval effect or old flame effect.

Notes: Changed optimized to un-optimized

I have stuff turned down, so I don't notice any lag from effects. However there is numerous 'cannot connect to login servers' and just terrible skill connection and lag when a lot of objects are in the same instance at the same time. The trading post is on it's own instance and shares the same long sell times, errors, etc. These all indicate network/server performance issues.

@Josiah.2967 said:Does this address the wings that are causing the huge hit in performance across the glove. They use an optimized fire effect that is only used on the new wings and the Skycale which is reaching havoc on FPS. This is incredibly noticeable in group events. The new area is not the problem..it's all the new wings killing performance in it that is causing the issue. This could easily be resolved by changing the flame effect to the Primeval effect or old flame effect.

I bought those wings and took an immediate FPS hit such that I don't enable them. I thought it was my rig not optimized enough to handle the effect.

I was pushing through Derelict Delve last night on a server that kept asking me if I wanted to move to a more populated instance, and getting repeated skill lag with only the occasional sighting of another player in the area, all on best performance settings. I think there are quite a few issues at play here.

@Inculpatus cedo.9234 said:Since the move was several years ago, why weren't there issues at that time?

The lag was so bad it took several years to reach us. ?

@Just a flesh wound.3589 said:

@Inculpatus cedo.9234 said:Since the move was several years ago, why weren't there issues at that time?

??

Btw didnt @Friday.7864 make a 50€ bet about Anet making a statement about the lag issues?

I cant find it in the forum any longer, but its still available via google webcache.

That would mean another 4k gem purchase for them wouldnt it? :P

@Just a flesh wound.3589 said:

@Inculpatus cedo.9234 said:Since the move was several years ago, why weren't there issues at that time?

Hahaha. The irony. =)

Fire Attunement.9835

you said the following

"We're aware that some players have been impacted by ongoing gameplay performance issues due to latency. We know this makes for a frustrating gameplay experience, and we want to let you know what we're doing to find and mitigate those problems.

The reality is there's no single sweeping change that can fix latency issues for all players, or even most players. There are a large number of factors and situations that can cause lag, and so members of our Engineering, Infrastructure, Design, QA, and Production teams are working with a group of affected players to help identify underlying causes and find solutions.

We’ll keep you updated on their progress as we’re able."

Firstly thank you for FINALLY communicating players. This has been going on for months over here it Europe.

I know there are loads of variables, but.

How do you explain that I can be playing in Sandswept Isles late at night UK time and get horrendous lag one minute and then move to another zone in Central Tyria and get no lag at all. Same Computer, same ISP, very few people about on the map. No massive Zergs?

During the Dragonbash event, I took part in lots of the Hologram events, there were lots of players all taking part. No latency lag whatsoever with skills. I go back to a LS4>>>or above zone and there are lag spikes.

I've played with guildmates from all over Europe, from Finland to the Med and they all report the same thing.

There is something inherently wrong with your servers.

@Josiah.2967 said:Does this address the wings that are causing the huge hit in performance across the glove. They use an optimized fire effect that is only used on the new wings and the Skycale which is reaching havoc on FPS. This is incredibly noticeable in group events. The new area is not the problem..it's all the new wings killing performance in it that is causing the issue. This could easily be resolved by changing the flame effect to the Primeval effect or old flame effect.

This explains a lot. I was wondering why my FPS was so low and was beginning to think my GPU was dying or something. I played back in later 2018 / early 2019 and was getting decent FPS even in Lion's Arch and Divinity's Reach with (almost) maxed graphics, but now I come back and struggle to exceed 25fps in any somewhat populated area with all these glowy wings, armors, and mounts.

There's so much armchair silliness in this thread, people outside the company stating speculation as fact.

How do you know they only recently moved to the cloud? Maps have always been allowed to scale from zero or one instances and up. What reason is there to believe that they swapped their backend infrastructure from on-prem to AWS between S4E5 (one of the problem maps) and now? Even assuming this is true (which is a stretch), how do you know this is the problem? How have you divined the specific way in which ArenaNet is leasing hardware from AWS? There is more than one way and they are not the same. Please link the AWS service offering that ArenaNet is using to scale maps in Guild Wars 2.

How does anyone here know that the problem is specifically due to server infrastructure rather than game code, without having access to the game code? Have you managed to reverse-engineer the entire game? You should share these details. Please link us to a repository where you have reverse-engineered the Guild Wars 2 client and server code, thanks.

@Tulki.1458 said:There's so much armchair silliness in this thread, people outside the company stating speculation as fact.

How do you know they only recently moved to the cloud? Maps have always been allowed to scale from zero or one instances and up. What reason is there to believe that they swapped their backend infrastructure from on-prem to AWS between S4E5 (one of the problem maps) and now? Even assuming this is true (which is a stretch), how do you know this is the problem? How have you divined the specific way in which ArenaNet is leasing hardware from AWS? There is more than one way and they are not the same. Please link the AWS service offering that ArenaNet is using to scale maps in Guild Wars 2.

How does anyone here know that the problem is specifically due to server infrastructure rather than game code, without having access to the game code? Have you managed to reverse-engineer the entire game? You should share these details. Please link us to a repository where you have reverse-engineered the Guild Wars 2 client and server code, thanks.

It's funny because they started moving over to AWS servers when PoF launched.

Players should do an /ip on the map(s) that they're having issues with, run a traceroute/pingplotter, and send the data to Anet.

@"Tulki.1458" said:There's so much armchair silliness in this thread, people outside the company stating speculation as fact.

How do you know they only recently moved to the cloud? Maps have always been allowed to scale from zero or one instances and up. What reason is there to believe that they swapped their backend infrastructure from on-prem to AWS between S4E5 (one of the problem maps) and now? Even assuming this is true (which is a stretch), how do you know this is the problem? How have you divined the specific way in which ArenaNet is leasing hardware from AWS? There is more than one way and they are not the same. Please link the AWS service offering that ArenaNet is using to scale maps in Guild Wars 2.

How does anyone here know that the problem is specifically due to server infrastructure rather than game code, without having access to the game code? Have you managed to reverse-engineer the entire game? You should share these details. Please link us to a repository where you have reverse-engineered the Guild Wars 2 client and server code, thanks.

Here's one way we know it is not a recent move (the Devs posted about it at the time, as well), among other things: https://aws.amazon.com/blogs/gametech/arenanet-guild-wars-mmorpg-migration/

You're speaking in a language unknown to me.

@Jagblade.4627 said:

@Josiah.2967 said:Does this address the wings that are causing the huge hit in performance across the glove. They use an optimized fire effect that is only used on the new wings and the Skycale which is reaching havoc on FPS. This is incredibly noticeable in group events. The new area is not the problem..it's all the new wings killing performance in it that is causing the issue. This could easily be resolved by changing the flame effect to the Primeval effect or old flame effect.

This perhaps is one of the reasons why Anet need to implement a dropdown menu that hides all other players wings/legendary amulet and ring effects/reskinned mounts. Make it so that it can be set to 1) on 2) display Basic wing/standard mount model 3) off.

Then we can see if its making a difference with the lag in wvw/normal zones.

@Andy.5981 said:

@Jagblade.4627 said:

@Josiah.2967 said:Does this address the wings that are causing the huge hit in performance across the glove. They use an optimized fire effect that is only used on the new wings and the Skycale which is reaching havoc on FPS. This is incredibly noticeable in group events. The new area is not the problem..it's all the new wings killing performance in it that is causing the issue. This could easily be resolved by changing the flame effect to the Primeval effect or old flame effect.

How would one hide all Mount models? Or Glider models?

I mean, that would look rather odd to see characters floating through the air with no means visible, I think.

@"Tulki.1458" said:How do you know they only recently moved to the cloud? Maps have always been allowed to scale from zero or one instances and up. What reason is there to believe that they swapped their backend infrastructure from on-prem to AWS between S4E5 (one of the problem maps) and now?

They switched to AWS in 2017 (at the time of PoF launch). Anet said this.

Season 4 was released after Path of Fire. So they were already on AWS when the season 4 maps were released. I played a lot on all of those season 4 maps when they were released. The S4-maps on AWS made no noticible problems at the time. The maps were full and everything worked without lag. But when I recently revisited the season 4 maps, they had delays and lagged like hell. When I played on other maps, (Core, Season 3, Season 5) nearly at the same time for comparison there was not this big lag. So there is a specific problem with the season 4 maps (maybe also PoF maps, but I have not visited them recently).

How does anyone here know that the problem is specifically due to server infrastructure rather than game code, without having access to the game code? Have you managed to reverse-engineer the entire game?

Because the Season 4 maps worked well in the past when they were released, something has changed since then. If something has changed with the season-4 AWS instances, or with the code for the season 4 maps, we do not know. Both ways its a problem Anet has to solve because it is on Anets side, not on the players side (client PC, ISP of the player).

And because other (non-S4) maps (when I tested this) worked well at nearly the same time when the S4 maps had problems, its not a general problem with the "internet-traffic" (overloaded routers or overloaded internet exchanges) between the map-servers and the client-pc.

To manage a complex and distributed game service like GW2 on AWS instances is a very complicated task that requires a lot of skill and knowledge not only of AWS but also the internal systems workings of GW2. I have read a comment at the time of the big lay-offs last year, that also employees with deep knowledge of the internal systems left the company.

Maybe we experience now the consequences of that loss-of-knowledge (if it really happened).

@Inculpatus cedo.9234 said:

@Andy.5981 said:

@Jagblade.4627 said:

@"Josiah.2967" said:Does this address the wings that are causing the huge hit in performance across the glove. They use an optimized fire effect that is only used on the new wings and the Skycale which is reaching havoc on FPS. This is incredibly noticeable in group events. The new area is not the problem..it's all the new wings killing performance in it that is causing the issue. This could easily be resolved by changing the flame effect to the Primeval effect or old flame effect.

Not far off whats happening with all the lag at the moment then really :)

Fair point though, just have a setting for "Force standard model for mounts/wings" at the moment. See how that goes.

@"Feyd Rautha.7298" said:I work in the networking and computer industry. You all know you moved your servers to the cloud to, and I air quote, "save money". This is your problem. Move them back in house -- hire real IT devs to track performance, and infrastructure. Rent at at high capacity datacenters or rebuild your own. You don't control server -> client you will never solve the issue.

UDP traffic is not part of balancing a typical web server (AWS). Perhaps hire higher tier servers for maps with dailies and WvW. You know your traffic ...

As said .. hire an expert - money well-spent.

@Zok.4956 said:To manage a complex and distributed game service like GW2 on AWS instances is a very complicated task that requires a lot of skill and knowledge not only of AWS but also the internal systems workings of GW2. I have read a comment at the time of the big lay-offs last year, that also employees with deep knowledge of the internal systems left the company.

Maybe we experience now the consequences of that loss-of-knowledge (if it really happened).

Sad but could be true... that kind of issue needs really skilled and knowledgeable people with deep experience with the backend code and structure...

FYI: there are no relation with the wings FPS issue. What you see (the game) = frontend. What is causing lag issues is the server code = backend. Anet needs people who are skilled at AWS network optimization and they need to optimize their backend code.