After reading all the horror stories here I decided to try more fishing and I'm still not sure what gives people so much trouble?
So far I've seen three difficulties (hook icon on the left of the fishing bar):
Green - an actual snooze fest, very slow fish, I'm really not sure how you can fail that one (seitung)
Yellow - faster, you actually need to predict it's movement instead of blindly following it (queensdale)
Red - fast, erratic, very hard to predict, almost a guaranteed failure if you lose it a couple times (frostgorge)
I only have +40 from 2 masteries, +25 from striped rod you can buy and +50 from bait you get from the story, but if fishing power is the problem just level up and get better rod and bait. GW2 is still an RPG and stats are important.