Can our mortgage company take away our mortgage?

We are thinking of refinancing our mortgage. But our income and credit is A LOT worse than when we were originally loaned the money 3 years ago. Can the mortgage company now penalize us on our current mortgage/decide not to lend us money based on our greater risk?