On-loan Celtic youngsters Aiden Nesbitt and Luke Donnelly both missed clear-cut opportunities for the visitors after the interval.
Nesbitt blazed wildly over the bar eight minutes before the home side got the winner. He had a good look at goal, but lost his composure at the crucial moment.
And Donnelly, who agreed to a loan deal with the Greenock men only a week ago, lobbed the ball off target with keeper Wes Foderingham off his line with just 30 seconds to play.
A disappointed Duffy admitted his battling team were severely punished.
He said: “We had two fantastic chances. I thought Aiden had a terrific match. He puts in a lot of work – a lot of times he’s isolated – but I thought we scored an outstanding goal through Michael Tidser.
“And then we had those two great chances. If you get opportunities like that at Ibrox, you’ve got to take one of them.
“That’s the most disappointing thing, that we weren’t able to do that. Overall, I can’t complain about the performance – I thought we played some good stuff at times.
“We’ve got a young side, but they’ve got great belief in each, they work hard for each other, were organised and we know no matter who we play, we’re always going to give them a game.
“We put an awful lot into the game. You can ask to make the game uncomfortable and difficult for Rangers and I think we did do that, but ultimately it’s about winning Cup-ties.”