Sounds like the transfer switch is not closing both legs. We’ve run our coach on as little as a 15 amp source (think Prineville fairgrounds 2007), just with highly discretionary device use.
On ours the inverter switch over the door does not need to be "on" unless we are not on shore power and need some 110v device. The fridge on most rigs is not run off the inverter; the dedicated outlet it plugs into (seen from the outside rear fridge access) comes off a breaker in the 50amp Main breaker box. The last breaker in there feeds the adjoining 30amp Main breaker box, aka the "Inverter Panel" because everything in it goes through the inverter and can be run off it when you aren't plugged in to shore power. That said, on ours the inverter panel is on the same leg as the fridge and the front air conditioner among other hardware.
It's possible the inverter's charger is trying to charge up low batteries at the same time the fridge is first powered up and maybe something else is on besides, maxing out amps; but that should blow a house breaker.
Did you try plugging into a house outlet on a different home circuit? Did you run an extension directly to the fridge power cord in the back? Just trying to isolate the problem a bit more.
I will leave any in-depth diagnosis to others more versed here, but need to say that the entry door “salesman’s switch” is for 12v only, and you should leave it on all the time. It and it’s latching solenoid can wear and weren’t meant for regular use, just dealer convenience on the lot. To store the coach, shut off your 12v Mains in the battery compartment area, NOT the door switch... leave it in the ON position always. Some have even rewired the back of the switch so it can’t be turned off.
Joel