That’s one of those business risks. If you subsequently find out that there is a problem with a raw material, you will have to evaluate every batch that raw material could have affected regardless of how small an amount of raw material went into the batch.
The risk is real, but the chance of a recall caused by a raw material is far less a risk in API manufacturing than it would be in dosage form manufacturing.