Player Data Backups - [Merged]

[TheQuickFox.3826]

https://techraptor.net/gaming/news/25-rust-eu-servers-lost-due-to-ovh-data-center-fire

Rust@playrust

Update:We've confirmed a total loss of the affected EU servers during the OVH data centre fire. We're now exploring replacing the affected servers.

Data will be unable to be restored.

https://twitter.com/playrust/status/1369611688539009025

This makes me wonder if ArenaNet has proper off-site backups for the game and player data of Guild Wars 1 and Guild Wars 2. An incident like this would be devastating to the playerbase.

[Dawdler.8521]

Anet has already had to roll back a couple of times. One would assume those backups arent on the same servers...

[Khisanth.2948]

@Dawdler.8521 said:Anet has already had to roll back a couple of times. One would assume those backups arent on the same servers...

They could still be in the same data center. I don't see how rollback implies different servers. That is mostly a matter of loading different data sets.

[mercury ranique.2170]

@"TheQuickFox.3826" said:https://techraptor.net/gaming/news/25-rust-eu-servers-lost-due-to-ovh-data-center-fire

Rust@playrust

Update:We've confirmed a total loss of the affected EU servers during the OVH data centre fire. We're now exploring replacing the affected servers.

Data will be unable to be restored.

https://twitter.com/playrust/status/1369611688539009025

This makes me wonder if ArenaNet has proper off-site backups for the game and player data of Guild Wars 1 and Guild Wars 2. An incident like this would be devastating to the playerbase.

The GW2 data is stored in a cloud service. This means the datacenter owner is responsible for the backup. They make a promise of availability to the customer (arenanet). So the real question is if the cloud provider had proper backups. This is Amazon. I do believe they do it properly.

[ugrakarma.9416]

g2w use cloud service, normaly data centers are "spreaded", highly virtualized.

[TheQuickFox.3826]

@mercury ranique.2170 said:[...]

The GW2 data is stored in a cloud service. This means the datacenter owner is responsible for the backup. They make a promise of availability to the customer (arenanet). So the real question is if the cloud provider had proper backups. This is Amazon. I do believe they do it properly.

This usually depends on the terms and contract (service level agreement) with the cloud service provider. Sometimes you just rent server capacity and have supply the software yourself, do your own installation and make your own backups. And other cloud hosting contracts are full-service, all-in contracts where the client only need to provide the content.

[Dawdler.8521]

@TheQuickFox.3826 said:

@"mercury ranique.2170" said:[...]

The GW2 data is stored in a cloud service. This means the datacenter owner is responsible for the backup. They make a promise of availability to the customer (arenanet). So the real question is if the cloud provider had proper backups. This is Amazon. I do believe they do it properly.

This usually depends on the terms and contract (service level agreement) with the cloud service provider. Sometimes you just rent server capacity and have supply the software yourself, do your own installation and make your own backups. And other cloud hosting contracts are full-service, all-in contracts where the client only need to provide the content.Yeah, I'm guessing they have some pretty heavy backups of the account/TP databases, that's the important stuff. AWS has built in backup services, wouldnt surprise me if Anet do it across regions too - NA holding EU backups and EU holding NA backups in case of catastrophic failure of either datacenter - and have longer term backups inhouse on regular basis.

For games like Rust, I think each persistent server also hold all the player data so if that server explode without a backup (or backups explode too)... everything is gone. For GW2 most of the "servers" (ie instances) is irrelevant for backup purposes. If they crash you just boot up another.

[Danikat.8537]

I assume they do have appropriate backups, but then I would have assumed any game does, so it would be nice if someone from Anet could confirm it. No details of course, I don't expect to know where they are or anything, just that situations like this are covered.

[Khisanth.2948]

Well there is backups then there are backups in different physical locations

[TheQuickFox.3826]

@Khisanth.2948 said:Well there is backups then there are backups in different physical locations

Yes, that is an off-site backup. Some data centers probably have the option of this, of making backups to a server/storage in a different location in the world even. But not sure if ArenaNet has any of this.

[Pacificterror.7805]

AFAIK Amazon is the current provider for GW2. I can only imagine that they use offsite backups (it's hard to imagine a company that doesn't....having an onsite backup is only cool if you also have a secure one offsite) due to how large their network would be ....but then again I'd also expect their facilities to have some pretty heavy fire suppression making data loss due to fire an unlikely event.

It's a great question to raise but upon pondering an answer, it probably actually gives me 1 reason to be happy they migrated over to Amazon's cloud service. Performance in some game environments have suffered as a result but I guess at least the likelihood of a loss like this is much lower with them as a provider.

[Khisanth.2948]

@TheQuickFox.3826 said:

@Khisanth.2948 said:Well there is backups then there are backups in different physical locations

Yes, that is an off-site backup. Some data centers probably have the option of this, of making backups to a server/storage in a different location in the world even. But not sure if ArenaNet has any of this.

AWS has that by default for some services and an option for others. Unless someone from sysops or maybe accounting chimes in I doubt there are much other people who would know if this is being done.

[Danikat.8537]

@Khisanth.2948 said:

@TheQuickFox.3826 said:

@Khisanth.2948 said:Well there is backups then there are backups in different physical locations

Yes, that is an off-site backup. Some data centers probably have the option of this, of making backups to a server/storage in a different location in the world even. But not sure if ArenaNet has any of this.

AWS has that by default for some services and an option for others. Unless someone from sysops or maybe accounting chimes in I doubt there are much other people who would know if this is being done.

That's a good point. In my old job (a much smaller place than Anet) we actually ended up with two off-site backups of our database because the sysop was making one and then a new head of department started, just assumed no one had thought to do it and started making his own. Which isn't a bad thing exactly but it was almost certainly unnecessary. It's the kind of thing most people don't think about until they need it so it wouldn't surprise me if a lot of people at Anet just assume they have one but don't know for sure.

[kharmin.7683]

Nothing in the GW2 User Agreement about Anet's responsibility or liability (or lack thereof) for backups and such. Just parts about interruptions of service, which I don't think mean a complete data center loss.

[Khisanth.2948]

@"kharmin.7683" said:Nothing in the GW2 User Agreement about Anet's responsibility or liability (or lack thereof) for backups and such. Just parts about interruptions of service, which I don't think mean a complete data center loss.

I didn't go read it but that sounds like it only covers availability but not durability.

[kharmin.7683]

@Khisanth.2948 said:

@"kharmin.7683" said:Nothing in the

GW2 User Agreement

about Anet's responsibility or liability (or lack thereof) for backups and such. Just parts about interruptions of service, which I don't think mean a complete data center loss.

I didn't go read it but that sounds like it only covers availability but not durability.

Quite right.

[Astralporing.1957]

Remember, that GW2 had, in fact, already been hit once by a massive fire in the datacenter, and went through it without losing any data. If anything, i'd assume they would be even more prone to being cautious now (unless, of course, the lessons from the past have already been forgotten, which does happen a lot).

[Dawdler.8521]

@kharmin.7683 said:

@Khisanth.2948 said:

@kharmin.7683 said:Nothing in the

GW2 User Agreement

about Anet's responsibility or liability (or lack thereof) for backups and such. Just parts about interruptions of service, which I don't think mean a complete data center loss.

I didn't go read it but that sounds like it only covers availability but not durability.

Quite right.It wouldnt cover any such responsibility with users anyway. Anet could shut down the game servers tonight if they wanted and delete the account and TP databases. Thats how games work. We dont "own" any data, we only lease access to it.

[TheQuickFox.3826]

@"kharmin.7683" said:Nothing in the GW2 User Agreement about Anet's responsibility or liability (or lack thereof) for backups and such. Just parts about interruptions of service, which I don't think mean a complete data center loss.

You won't find any answers in ArenaNet's user agreements or legal terms. These are just to protect them against legal claims.

[Xanadrine.4352]

Whoa that would be terrible. If you lost years of advancement in the game I think I would just quit. Last game I played 14 years so many titles achievements boss kills, thousands of mounts and pets and weapons and armor. Yeah I wouldnt start over

[TheQuickFox.3826]

@Zuldari.3940 said:Whoa that would be terrible. If you lost years of advancement in the game I think I would just quit. Last game I played 14 years so many titles achievements boss kills, thousands of mounts and pets and weapons and armor. Yeah I wouldnt start over

Yeah, I got some form of autism and this franchise has literally been of therapy to me. It is wonderful to lose myself in this virtual world on a regular basis. A loss like what happened to Rust would be devastating. But probably not just to me or the entire player base, probably to ArenaNet as well. So having a decent backup strategy that includes scenarios like ransomware and fires is quite important.

[Hyomea.9702]

I've just read a newspaper article about a data center in Europe that burned down. Most of the data stored on the burnt servers is lost now, since many clients of the cloud provider OVH, which was running the center, did not make sure to have a backup of the data outside of said center. There's a french game studio, called Facepunch, which is famous for it's game "Rust". They had all of their players' data stored there, while not having a backup somewhere else. All players' data is now gone, forever ...

Obviously, as a long time, avid GW2 player, this made me think about the situation of GW2 players' data. Imagine, what happened to the data center of OVH, happening to a center, where GW2 data is stored: a huge amount of relevant servers burnt to a crisp. Imagine its implication for the game and even for Anet as a whole, if there's no backup of the burnt data. I can only speak for me, but I surely wouldn't play GW2 anymore, if I had to start from zero again. Not to think of, what it would mean to Anet as a game studio!

Anyway, OVH is (maybe was?) Europe's cloud computing market leader, a highly successful and professional enterprise only recently announcing their initial public offering expecting to get a rating of 1 billion euro in value! What happened to OVH can most definitely and will eventually happen to others in the industry.

Now, does Anet have a backup of GW2 players' data stored somewhere else, than where the original data is being kept?What I've written might just sound alarmist to some, since such incidents rarely happen, but if they do, then with great, often fatal impact!

Newspaper article source:https://faz.net/aktuell/feuilleton/medien/groesstes-rechenzentrum-europas-brennt-komplett-nieder-17241629-p2.html

[Astyrah.4015]

see the recent discussion about it here:

https://en-forum.guildwars2.com/discussion/124936/survival-game-rust-loses-data-in-dc-fire-does-anet-has-proper-off-site-backups

[Danikat.8537]

There's already a topic about it here: https://en-forum.guildwars2.com/discussion/124936/survival-game-rust-loses-data-in-dc-fire-does-anet-has-proper-off-site-backups#latest

The short version is that we don't know any of the specifics, but apparently there has been a fire at their data centre before and they didn't lose any data, so presumably they do have off-site backups.

[Hyomea.9702]

Oh, didn't know that. Thx for letting me know!