The overall responsibility for ensuring that compliance audits are performed generally falls under the Quality Assurance department. Auditors however can be quality assurance (QA) personnel, personnel from other areas within the company, or consultants (Note: The auditors themselves should have the knowledge and experience/training to perform the task adequately. You would not expect the office cleaner to audit packaging and labelling systems, for example.)
Some small companies use consultants to address concern about independence, while larger manufacturers use consultants to obtain an outside verification of adequate compliance.