Yes, but the Taxpayers' Bill of Rights discourages the IRS from seizing primary residences (however, your vacation home or rental property is fair game).
IRS collectors, however, cannot act on their own to seize your house. The IRS must obtain a court order, which you can contest. And you can request help to stop the seizure from the Taxpayer Advocate Service.
The IRS doesn't like publicity about taxpayers losing their homes, so a call to the newspaper might help. Typically, the IRS seizes homes only if you have totally failed to communicate or cooperate with the IRS collectors.
For detailed information on protecting your assets from the IRS, see Stand Up to the IRS, by Frederick Daily (Nolo).