I came across an issue where the downed penalty will reappear on you after you waypoint from a death. It looks to be a case where you get the penalty and eventually die from rallying too many times, waypoint, and then enter another zone (in this case it was leaving my keep and entering the zone for ogre watch). Here is the footage of it happening. You can see at 33 seconds the penalty isn't there but it appears (and is actively working as the footage shows) at the 51 second mark. After death and waypointing, it does it again at the 1:36 mark.