Let's say your neighbor has built a shed that is on your property; in fact, that was put about 20 feet over the line. You might not care so much about the shed itself, but there's a legal consideration here: Your neighbor could eventually claim adverse possession over the shed and the path leading up to it, as discussed here.
To establish a legal claim of adverse possession, your neighbor would need, under the law of most states, need to use the property in a manner that is hostile, actual, open and notorious, and exclusive for a continuous period as prescribed by your state's statute. Although the statutory period varies from one jurisdiction to another, it is usually at least seven years. (Read more about adverse possession claims.)
This means that if your neighbor built and uses the shed for the statutory period, and you don't say anything to stop this or assert your ownership of the land, the neighbor might be able to establish ownership through adverse possession.
More concerning, perhaps, than "ownership" of the shed and the land underneath it, would be the claim to the 20-feet pathway the trespassing neighbor created leading to the shed. The neighbor could claim having used that pathway without any intervention on your part. This trims the size of your property significantly, and could impact the value of your land.
There are a number of steps you might take.
First, having become aware of the issue, you should put your neighbor on written notice. Send a letter or, if you feel more comfortable, have your attorney do so. That letter should make reference to specific property boundaries as described in publicly filed deeds. It should clearly state that you do not consent to the use of your property, and that you revoke any perceived license that the neighbor believes were given.
Second, you might ask the neighbor to sign a written agreement stating that they acknowledges that the land is yours, and that your permission to use the land is freely revocable at any time. Or, you might present the neighbor with a rental agreement. Rental agreements are a useful method of preventing trespassers from establishing adverse possession, because they establish a different understanding of the ownership and usage arrangement.
A neighbor who is friendly and didn't realize the mistake would probably be apologetic. They would also understand your perspective: they built a structure on your land! You might offer that the neighbor can maintain the structure, but in exchange for a nominal rent and a written acknowledgement that the property is yours. This way, the neighbor can have his shed, and you can have peace of mind, knowing that it won't impact your property rights.
Need a lawyer? Start here.