If you fail to file your tax returns on time you could be charged with a crime. The IRS recognizes several crimes related to evading the assessment and payment of taxes. Penalties can be as high as five years in prison and $250,000 in fines. However, the government has a time limit to file criminal charges against you.