The Fifth Amendment’s self-incrimination provision is a privilege that people can invoke when their responses could get them into trouble with the law. When testifying at someone else’s trial, for example, a witness must claim the privilege at the time of questioning. Otherwise, the government can use whatever the witness says in a later criminal prosecution of him or her. The witness must invoke the privilege to, on the one hand, guard against self-incrimination and, on the other, avoid contempt for failing to answer questions.
But someone who is in law enforcement custody and being interrogated doesn’t have to begin the interrogation by claiming the right against self-incrimination. Instead, the right is automatically asserted on the suspect’s behalf. That means that the suspect has to waive the privilege—rather than fail to assert it—in order for his or her statements to be admissible at trial. Officers have to provide the Miranda warning, and then the defendant has to indicate a willingness to talk. (But see Miranda Rights: Cutting Off Police Questioning. And note that suspects' silence can sometimes be used against them; see, for example, Can the government use suspects' post-arrest, pre-Miranda silence against them?)
Police officers can be quite crafty and work around Miranda. By making it clear that the suspect isn’t under arrest and is free to leave, they can avoid the requirement that they provide the Miranda warnings. In that instance, it’s up to the suspect to claim (and, in the first place, know about) the Fifth Amendment privilege. (See Invoking Your Right to Remain Silent.)