Quit AP goroutine early when there's no "to"
Previously, we'd sleep for 2 seconds and then return for no reason. This fixes that.