No. An IRS employee may not enter your home without an express invitation from you or another rightful occupant. The only exception is if the IRS has a court order, which is very rare. A field auditor may ask to come in to verify your home office deduction, but you don't have to let him in. Of course, if you don't, he'll no doubt disallow the deduction. Your choice.
For more on handling a tax audit, see Surviving an IRS Tax Audit, by Frederick Daily (Nolo).