I always fire the bikes up then take a few minutes to finish kitting up and have a walk around check before heading off. The exhausts generally point at the house opposite when wheeled out of the garage, and I don't care what that selfish old arse thinks, but nobody has ever mentioned it in a negative way anyway. The KTM isn't subtle when idling either. If I was doing a very early start though, I'd do the warm up in the garage as it's reasonably well sound insulated, then open the door when ready to leave. Powered door means I can pretty much just ride straight out & away without stopping though.
As yours is standard, I think your neighbour is being ta bit of an arse, not you. As has been suggested, move the bike a bit, or just turn it away from him.
Bike engines are more highly stressed than your average car so I feel benefit from allowing the oil to warm at least a little before loading them. We used to go with letting the Maicos sit until you felt some warmth on the cooling fins before trying to coax them into action, but there's no choke on a Bing carb, so they wouldn't run worth a damn cold anyway. I still apply the same principle to the air cooled bikes I have.