Pembrokeshire and Derwen NHS Trust, which is around �11m in the red, has to pay back the Welsh Assembly Government �2.4m a year until 2009.
But, because of deficits going back to 2004/5, the board has been told to save about �4m by next April.
The trust says service changes are "a temporary measure" to achieve balance.
No more use of bank and agency staff In a statement on Monday, Mary Hodgeon, active chief executive of the trust, said: "I would like to express our disappointment at the anxiety that has been generated about the future of services at Withybush Hospital. We are sorry these misunderstandings have arisen.
"The temporary savings plans agreed were those which would cause minimum disruption to patients.
"They have been approved by the trust and LHB boards as well as Pembrokeshire Community Health Council and more work is being undertaken to address long term solutions in partnership with Welsh Assembly Government."
She added: "The people of Pembrokeshire can be reassured that there are no plans to downgrade Withybush Hospital.
"We do need to change the way we provide services to ensure the very best healthcare for people but we will only introduce any changes after the full involvement of staff and local people."
The trust recently had to assure the public that Tenby Cottage Hospital would not be closing to save money, and would remain open until its replacement was finished in 2007.
The local community health council and local politicians have expressed their concern over the proposals.
Pembrokeshire AMs Christine Gwyther and Tamsin Dunwoody issued a joint statement voicing their disappointment that efficiency savings did not appear to have been identified before front line services were cut.
"We all understand that difficult decisions sometimes have to be made," they said.
'Cuts not supported'
"But these decisions should not be at the expense of front line services for Pembrokeshire patients.
"The community health council has said it will not support the cuts. Neither will we. It is time that the trust looked again at its proposed cuts and made more effort to protect patient services."
Stephen Crabb, Conservative MP for Preseli Pembrokeshire, added: "There is hardly a family in Pembrokeshire that does not have reason to be thankful to the staff and facilities at Withybush.
"It is vital that all politicians - whatever their party - now pull together to preserve the status of Withybush Hospital."
Mike Pelowe from Jameston is one of the patients who joined Monday's protest.
He suffered a stroke almost a year ago which left him with mobility problems. He is currently given physiotherapy at a rehabilitation centre in Pembroke Dock twice a week, but is worried about plans to move his treatment to Haverfordwest.
"At the moment it takes about half an hour to 40 minutes," he said.
"But if we get moved to Haverfordwest, it will take and hour to get there and an hour to come back.
"Unfortunately, due to the fact that I get a bit weaker at the end of the day, I will get very tired on the way back and more liable to fall over."