I haven't encountered an Iron Giant yet, but I do feel that monsters flee too much in general. Dividing the HP difference by 64 (minimum 14444 party HP required for a 1% chance for a lone Iron Giant to run) or even 128 (25388 HP) instead of 32 (8972 HP) would be an easy change to make. That said, I've had an average over my recent playthroughs of about 7000 HP when reaching the final boss and haven't had any serious difficulties, so 12000 HP seems like a lot... maybe I've been abusing save states too much?
I've done a bit of farming for rare items early on, and I've avoided save stating at all except when I need to walk away for long periods of time, but I've more or less played the game as an end user in my testing. My HP might be a bit higher, but not by too much. Enemies fleeing right now feels fine given my HP, I just think we should set Iron Giant's formation flag as inescapable, honestly, tempting to set it on both. It doesn't seem like something that should run away, it has stats and even a Fear Score comparable to bosses, but the bosses won't run because they have the flag. As far as I can tell, he's meant to be a rare boss.
Based on your explanation of the math being done for fear in the original, it sounds like current behavior is pretty close to what the devs intended, and I think that's alright. The issue is almost entirely with the Iron Giant.