Well, I learned more on this and came to a resolution with the broker.
Toterhome financing is like a thinly-traded stock. Not all lending sources with finance it as most do not understand what the vehicle is. So this company, which arranges financing with banks that know these loans, has a mini-monopoly (not using this term derisively), and hence his fee is higher than a standard RV loan (which is typically 1%). The banks are not "rebating" any fee to him, so his fee is his compensation.
We talked and came to a resolution that is acceptable to me. His company provided value to me, so I didn't want to to just walk away, but the initial fees were higher than I thought reasonable.