I'd use care with pry bars. If you get too far underneath you'll engage the edge of the black speaker body, and you likely don't want to be prying down on that. Yet the edge of the white ring escutcheon is plastic, and if forced it might crack or be marred if your bar slips; the ring is maybe only 1/8 in. thick.
I went out and tried removing mine again and it wasn't so easy. A gentle screwdriver pry got one edge started, and my fingers worked around the edge to finish getting it off. A bit perplexed at it being more difficult, I examined the escucheon more closely. The speaker body rim has four mildly raised nubs equally spaced around it. The white ring has four matching slight rectangular depressions around the inside of its rim. It was obvious that after taking the photo for you, I'd pressed the ring back on without noting the nubs and depressions, so it was harder to remove the second time. That may be your problem; someone's put the ring on without aligning the nubs with the depressions.
So hopefully the photo below will be of some help when you do get the ring pryed off, and you can replace it properly. Note the rectangular depressions upper left and barely discernible right inside the ring edge. There are four, 90 degrees apart that should align with the four nubs around the outside of the speaker body rim.
Joel