In general, whatever a private citizen—rather than a police officer—uncovers through an illegal search is admissible in court. But if the private citizen acted on behalf of the government, a court will likely suppress the evidence just as if the police had found it. That's because the "exclusionary rule," providing that evidence found as a result of an illegal search is inadmissible, is designed to deter government agents—not private citizens—from unlawful snooping.
(For related information, see Can a private business search your stuff and then turn it over to the police? This article mentions the fact that the law in some states may provide greater protection against private-party searches.)
Many cases over the years have defined what constitutes an illegal search by law enforcement. But, in some ways, there's really no such thing as an "illegal" search by a private citizen, at least in the sense that police searches can be illegal: Regardless of issues like lack of probable cause, evidence found by private citizens acting on their own is usually admissible in court. That's true even if the private citizen committed a crime like trespass or theft to accomplish the search. (Regardless of the admissibility of evidence, though, the citizen will be liable for any crime committed in the course of the search.)
Because the exclusionary rule applies to government conduct, evidence from a search by a private citizen who acted on behalf of law enforcement, but without probable cause, is usually inadmissible. For example, if the police don't have probable cause but employ an informant to look through someone's bag, a judge will likely suppress any evidence of the contents.
Although police officers sometimes seek the help of private citizens, many professions that appear to be "law enforcement" aren't. For example, courts typically don't consider security guards employed by private businesses to be government agents. The same goes for private investigators and bounty hunters, who operate from profit motive rather than to assist a government investigation. Even off-duty police officers employed as private security aren't government agents for Fourth Amendment purposes.
The admissibility of evidence found by a private citizen usually turns on the government's "share" of the search. In other words, how involved was the government? While cases where the government ordered or paid a citizen to conduct the search are fairly straightforward, others aren't. In determining whether to admit the evidence in question, courts consider questions like:
The defendant seeking to exclude the evidence bears the burden of proving by a preponderance of the evidence (a more-likely-than-not standard) that the private citizen acted as an agent of the government. (U.S. v. Feffer, 831 F.2d 734 (7th Cir. 1987).)
Example: Paul, a security guard at the local mall, thinks Mandy, a shopper, looks a little suspicious. When she sets her purse down and starts talking on her cellphone, he takes the opportunity to search it. He notices a bag of heroin. Paul flags down a police officer, who arrests Mandy and takes the heroin as evidence. Before her trial, Mandy's attorney files a motion to suppress the evidence, but because Paul found the heroin on his own initiative, the court denies it. If the police officer had told Paul to search Mandy's purse, the result would have been different.
If you've been arrested or prosecuted due to evidence found by a private citizen, seek the help of an experienced criminal defense attorney. A knowledgeable attorney will be able to advise you of the applicable law and protect your rights.