Not sure I count as a casual player, but I couldn't agree more. I personally find jumping puzzles tedious and would rather be doing other things in game, but the good loot (legendary trinkets etc.) gates behind them makes them kind of a necessity.
Would be great if AN just treated them as their own thing with decent loot and AP for people that want to do them but let those of us who dislike them ignore them.