I'm not familiar with Xantrex, Rick, but our Magnum appreciates knowing what type of batteries it's asked to charge; so be sure that part is correct - wet cell or glass mat. Also your AGS (automatic generator start) won't work if there isn't even enough battery voltage to start the genset in the first place. Your batteries are like half discharged at 12 volts... 5 v isn't anywhere near "competent".
Your AGS Needs to be programmed to kick on at 12v. As a new-be to AGS I set mine at 11v. Not. The AGS programming gets confusing, requiring coordination between a remote programmer in the electrical bay and the inverter controller panel. I don't rely on it anymore, but that's likely more my lack of programming adeptness than the system's ability.
And Dave's right... If you finally get your inverter back up, but your battery banks are very discharged, a lot of juice is going to go to charging them ultimately back to ideally 13.2-13.8v if it's set to max rate; the initial bulk charge rate will show battery voltage well over 14v. for some time until it settles down. If you put much else online, like the electric side of the AquaHot, fridge/ice maker, A/C, or other power hog, the system is stressed. Your inverter or its charger circuit may indeed need checking by techs.
If your house batteries are wet cells, how long since someone checked their water level?
Joel