Visa concerns following last week's immigration raid on LG Energy Solution's joint facility with Hyundai Motor have led South Korean workers to head home.