Defendants cannot represent themselves unless a judge determines that they are competent to do so. The community as a whole has an interest in achieving justice, and a trial in which an incompetent defendant self-represents isn't a fair one.
Competency to Represent Yourself
The case that established that defendants have a right to represent themselves was Faretta v. California, U.S. Sup. Ct. 1975. The Faretta case said that a judge must allow self-representation if a defendant is competent to understand and participate in the court proceedings.
To determine competence, the judge often weighs factors such as:
the defendant’s age
the defendant’s level of education
the defendant’s familiarity with English, and
the seriousness of the crime with which the defendant is charged.
No single factor determines the result, and a defendant doesn’t need the legal skills of a professional lawyer to qualify for self-representation. As long as a defendant is competent, knowingly gives up the right to an attorney, and understands court proceedings, the defendant is entitled to self-represent.
It's critical to note, though, that the fact that one can self-represent doesn't mean that one should. Almost everyone who works in the criminal justice system agrees that self-representation is normally a bad idea, in no small part because the defendant usually lacks anything near the necessary training and experience.
Competency to Stand Trial
Whether a defendant is mentally competent to stand trial is a different issue. A time-honored principle in American law holds that we should not subject someone to trial who lacks the capacity to understand the nature and purpose of the legal proceedings against him, to consult with his lawyer, or to help in the preparation of his defense. Someone who is competent to stand trial is oriented as to time and place, and has a reasonable degree of rational understanding. (18 U.S.C.A. § 4241.)
A judge has the power to decide that a defendant is mentally competent to stand trial, yet not competent enough to represent himself (Indiana v. Edwards, U.S. Sup. Ct. 2008).