Jump to content
  • Sign Up

Items list and timestamp


Recommended Posts

Hello everyone,I'm in the process of writing an /items scraper, and I'm wondering if there's a more efficient way to keep the data up-to-date than running a full crawl every X hours. Can some HTTP headers like Last-Modified or If-Modified-Since be used in order to know if there are new items to download, or whether some items have been changed?Maybe the question has already been answered, but if so, I can't find it.

Thanks everyone!

Link to comment
Share on other sites

  • 1 month later...

@"StevenL.3761" said:I don't know about any headers or how to use them. What I do is get the entire index of items (/v2/items), discard the ids that I already have in my storage and then fetch the ids that remain. If the build number changes (/v2/build), I wipe my storage.

I use the same strategy for https://www.gw2gh.com/ with one exception: If the build number changes, I do not delete my storage but mark all items in my storage as "dirty".

"dirty" items will be replaced asap with actual ones. But if the API fails and (some) items are not available over the API anymore (which did happen a few times in the past) I could still use the "dirty" items from my storage.

Of couse, this means that old items, that were removed from the game, will not be automatically deleted in my storage. This is acceptable for me.

Link to comment
Share on other sites

One strategy I have been considering is keeping all item records with a timestamp for creation and last check, and when an item actually changes, creating a new record, in order to keep track of the evolution of the items. I am a bit worried this might be against the ToS though, what do you think?

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...