From January to June, more than 20,000 employees out of more than 110,000 left the agency, according a report by the agency’s inspector general.