The NES and SNES were unquestioned successes though. Outside of those with a fanboy streak in North America the N64 was not, not to mention it would be missing quite a few games that have since been remade in superior forms (give or take your opinions of the censorship of Conker's bad fur day). Going further many N64 games did not look pretty at the time, never mind today.
To that end I would seriously question the logic of a N64 mini.
Anyway I don't really know any times commercial emulators were reverse engineered to improve a scene as a whole, only for individual systems where things were injected back in or maybe if said emulators had a really cool trick (and I doubt they will have anything here as the only things they might have will be things we have already seen in fceux or maybe a minor scaling/video filter a la http://bogost.com/games/a_television_simulator/
), possible slim exception if something is unknown about a system and the official method (something like the GBC colourisation schemes for GB games that did not officially get dumped for a while) and another if the would be N64 mini is similar enough to a more common device to allow a port of sorts.
The main thing that would improve the N64 emulation world is the same thing that would improve most others in that it needs a really good, open source, debugging capable emulator which works out of the box (no futzing with config or plugins) for most games, and with scope for things to improve the graphics in various ways. You may or may not want to include support for all those multi gig texture replacement hacks as well.