Other people told me the GBA/GBC mode is toggled by a tiny switch that gets physically pushed by the weight of the pins on the appropriate cart.
That would explain why I've heard of worn consoles being unable to boot one or the other.
(wish I could remember the exact details)
Not pushed by the weight of the pins, but pushed by the different shape of the cartridge shell. Notice that GBA cartridges have notches on the sides, but GB/GBC cartridges do not. So the GB/GBC cartridges push a tiny button, and the GBA cartridges do not. This also let Nintendo design a different GBA slot where GB/GBC carts won't fit inside.
Edit: just re-read the thread, Drakon already posted this.
Also, the GBA BIOS does try to detect GBC mode (by reading bit 15 of 0x04000204). If it detects a GBC cartridge, it will fade to an image of the black border, then it performs a software switch to GBC mode (set bit 3 of 0x04000000). However, software switches to GBC mode don't actually work, you can't read the cartridge after this happens. Also, the hardware switch immediately makes it change to GBC mode, so the BIOS code to check for GBC cartridges will never execute. You can only see what would happen on emulators if you patch the code in the BIOS that detects GBC mode.