Chris,
The only advantage to leaving the inverter section of your inverter/converter on when hooked up to power is that if you have an intermittent lose of shore power like the four or five minute interruptions that happen occasionally in a thunder storm, the inverter will take over automatically and continually produce 120 volt power from your batteries. The only downside to this is that if the power is off too long, the switch can be so seamless that the inverter will completely discharge your batteries without you noticing unless you are using an auto gen start.
So there is no "correct or absolute" answer to your question unless you have a residential refrigerator and an automatic gen start. It just depends on how you feel the most comfortable. However, if you have a residential refrigerator and an automatic gen start, you should probably leave the inverter on all the time.
Gerald