Items list and timestamp — Guild Wars 2 Forums

Items list and timestamp

Hello everyone,
I'm in the process of writing an /items scraper, and I'm wondering if there's a more efficient way to keep the data up-to-date than running a full crawl every X hours. Can some HTTP headers like Last-Modified or If-Modified-Since be used in order to know if there are new items to download, or whether some items have been changed?
Maybe the question has already been answered, but if so, I can't find it.

Thanks everyone!

Comments

  • StevenL.3761StevenL.3761 Member ✭✭

    I don't know about any headers or how to use them. What I do is get the entire index of items (/v2/items), discard the ids that I already have in my storage and then fetch the ids that remain. If the build number changes (/v2/build), I wipe my storage.

  • Leo.3428Leo.3428 Member ✭✭

    I was about to ask the same question and was hoping for an endpoint for diff listing. Thank you @StevenL.3761, refreshing the whole data on a new build is a good compromise.

  • Zok.4956Zok.4956 Member ✭✭✭
    edited July 21, 2019

    @StevenL.3761 said:
    I don't know about any headers or how to use them. What I do is get the entire index of items (/v2/items), discard the ids that I already have in my storage and then fetch the ids that remain. If the build number changes (/v2/build), I wipe my storage.

    I use the same strategy for https://www.gw2gh.com/ with one exception: If the build number changes, I do not delete my storage but mark all items in my storage as "dirty".

    "dirty" items will be replaced asap with actual ones. But if the API fails and (some) items are not available over the API anymore (which did happen a few times in the past) I could still use the "dirty" items from my storage.

    Of couse, this means that old items, that were removed from the game, will not be automatically deleted in my storage. This is acceptable for me.

    https://www.gw2gh.com/ - A GW2-Guild-Hall.
    Register and check your guild leaderboard to see who is the best in your guild and who finished achievements first.

  • StevenL.3761StevenL.3761 Member ✭✭

    That's undeniably better than deleting data. Minor suggestion: "stale" is a more appropriate term than "dirty" (each item doesn't necessarily change between builds).

  • Leo.3428Leo.3428 Member ✭✭

    One strategy I have been considering is keeping all item records with a timestamp for creation and last check, and when an item actually changes, creating a new record, in order to keep track of the evolution of the items. I am a bit worried this might be against the ToS though, what do you think?

©2010–2018 ArenaNet, LLC. All rights reserved. Guild Wars, Guild Wars 2, Heart of Thorns, Guild Wars 2: Path of Fire, ArenaNet, NCSOFT, the Interlocking NC Logo, and all associated logos and designs are trademarks or registered trademarks of NCSOFT Corporation. All other trademarks are the property of their respective owners.