Immigrants vs Visitors

When Lamoni asks if Ammon wanted to dwell in the land I get the impression that he was generally harder on any Nephites who wandered into his land than on any who we intent on leaving the Nephites. (Based on his question I imagine that there were some few who did defect from the Nephites.)