This is a common question, and there's a lot of misinformation surrounding it, so hopefully I can shed some light on it... Bear in mind that I am not a legal advisor and that what experience I do have revolves around United States laws. My answers here may not apply in your jurisdiction.
First things first, Nintendo is often a subject of ridicule with regards to emulation because their official stance on it
is misleading and incorrect. They explicitly state that a device that can be used to copy games is in itself illegal, which even someone with only a passing familiarity with the law can take issue with. I don't want to touch on Nintendo's claims in this post--rather, focus on what they can legally do in a court of law.
The most common claim is that fan games are violations of copyright. In order for something to be considered an infringement of copyright as defined by title 17, it must be used "for sale or trade". If there's no exchange of goods going on, then it's not a matter of copyright at all. That doesn't mean Nintendo won't go after it, it just means that copyright isn't a magical umbrella that makes or breaks every problem. Similarly, it's important to remember that even if your own project isn't technically an infringement of copyright, you're not necessarily in the clear.
Frankly, Nintendo is usually hands-off when it comes to fan projects, but have on occasion issued cease and desist orders to a select few. This article on Gaming Reinvented
lists a good few cases where this happened. Take a look at that list and see if you can identify what all of those projects have in common. They're all pretty good, right? But it's not that Nintendo is gunning for the good works in some huffy retaliation for being shown up by fans...
What it boils down to is whether a fan project legitimately causes a problem for Nintendo. Remember, Nintendo is a big corporation with a lot of investors who have a keen interest in seeing those investments succeed. So when something comes around that uses Nintendo's intellectual properties or creative works in such a way that customers are distracted by it, that's when it becomes a problem and Nintendo is obligated to respond. This almost exclusively happens with good
fan projects, so as far as I'm concerned, a cease and desist from Nintendo is tantamount to receiving their stamp of approval.
I guess what I want everyone to take away from this is that while there's nothing intrinsically wrong with mods or ROM hacks from a legal perspective, we still need to be mindful that even something as innocuous as a fan game can have genuine, measurable effects on the company that inspired them. If that company should deem it necessary to take legal action as a responsibility to their stakeholders, it should come as no surprise and we really ought to accept it and move on. If we think we've got something good on our hands, we could always put some effort into making a brand new intellectual property and sidestep the issue completely...