I don't see karma as something external to us. If we treat someone like crap and have a conscience, we feel shit as a result - that's karma coming back to bite us, not some external force sending a random someone to come and be mean to us. We can always go and apologise and repent some of our awful behaviour. That good action will make us feel better - again the karma of making a positive decision.
If we don't have a conscience and we habitually treat people like crap, then yes, eventually we will suffer some consequences by someone else doing something about our behaviour.
I don't think that if you hurt someone thirty years ago, karma is about waiting for the punishment from that to come to us. As long as we feel the guilt from it, we are being punished. And we can always choose to do something about it.
In other words. I don't think karma is anything to be afraid of. It's simply nature.