Residents worry about late-night knocks on doors