Since you stated that you do not mind having one or two over, why not start with that? Invite them individually, over time.