Adventure Rankings show removed Friends / apparently random players

I've always wondered where the apparently random people listed in the game adventure time rankings window come from.

Now I think I realized at least one issue with it: I recently added someone as a friend to see his time, but then removed him again from my buddy list. However, he still appears in my rankings. He's also not in my followers list or anything anymore.

Why isn't the panel simply taking the Friends list to display times for all my actual friends? In case it should be already doing that, it isn't for me.

It also includes guild members.