The implications of having your application for an F-1 or M-1 visa (or a change of status to F-1 or M-1) denied depend in part upon where you applied; either at a U.S. consulate abroad, or to an office of U.S. Citizenship and Immigration services (USCIS) within the United States.
If the U.S. consulate denies your F-1 or M-1 student visa application, that's final. There is no appeal process. But make sure that you ask the consular officer for the exact reason underlying the denial. If you can correct the problem, you can start over and submit a new application.
In rare cases, the consular officer might have actually misapplied the law to your case. If the officer mentions legal grounds for the denial, ask for the exact section number of the law under which you're being denied, in order to assist any lawyer whom you later consult.
Also ask for the rules on reapplying. Some consulates have special procedures to reduce your paperwork if you are reapplying within one year.
There is no appeal from the denial of a Change of Status application to student (or other) status, which you would have made to USCIS if you were already in the U.S. in valid immigration status. However, a lawyer can help you request that your application be reconsidered. The lawyer will need to argue that USCIS incorrectly applied the law to the facts in your original application—not an easy argument to make, unless an obvious error occurred.
If you decide to consult a lawyer or to apply for reconsideration, act quickly. If your original visa status has run out, USCIS will follow its denial of your application with an NTA notice placing you into removal (deportation) proceedings. If you can't show that you have some separate basis for being allowed to stay in the United States (for example, asylum or marriage to a U.S. citizen), you will be asked to voluntarily depart by a certain date or you will be ordered deported.
Most people who are given a choice depart voluntarily, because once you have been deported, you cannot return to the United States legally for five years without special permission (a "waiver") from the U.S. government.
If your Change of Status application in the United States is denied and you leave within the time required (or voluntarily), you can always try applying for a visa at an overseas consulate. This could be tricky—by this time you might have spent more than six months in the United States unlawfully, which would make you unable to return for three or ten years. (See Consequences of Unlawful Presence in the U.S.—Three- and Ten-Year Time Bars.) Consult a lawyer if you're facing this issue.
If you decide to reapply for a student visa, the main question will be, can you correct the underlying problem that led to the first denial?
If the problem had to do with your basic eligibility, such as whether you have enough savings or a source of funds with which to cover your stay or whether you will return home afterwards, you might be able to fix it yourself. Read about how to establish your eligibility in the first place, including F-1 Student Visa to the U.S.: Do You Qualify? and M-1 Vocational Student Visa to the U.S.: Do You Qualify?
Think about what additional forms of evidence you can come up with to tip the balance in your favor. The consulate will want to see something new, such as a source of financial support that you did not mention before, your recent wedding to someone who will stay behind in your home country during your studies (giving you a reason to return), or a home purchase indicating your plans to live permanently in your home country.
The more difficult problems to correct are those having to do with fraud and inadmissibility, especially for security or health reasons. These are problems that would stop you from getting any kind of visa, not just the one you're applying for now. See Inadmissibility: When the U.S. Can Keep You Out for more information. If fraud or inadmissibility is the reason for a visa denial, consult with an immigration lawyer in the United States to discuss whether you can overcome this ground of inadmissibility or whether a waiver is available.
Having a visa or Change of Status application denied could have long-term consequences for your visits or eventual immigration to the United States. The U.S. government will keep a record of your visa denial. If you used fraud or they discover that you were inadmissible for some other reason, this could affect your future visa applications.
At the least, having one application denied will mean that the U.S. government will look hard at any other immigration application that you submit. Immigration authorities can consider this denial when making other visa decisions for you, no matter how long ago the denial happened. If you later try and hide the denial and are caught, it will probably destroy your chances of ever getting a U.S. visa or other immigration benefit.