I feel like I'm missing something, because when I load the ROM up in a hexeditor everything is a garbled mess.
This is pretty normal - NES ROMs contain a mixture of code and various kinds of data (graphics, sounds, text, etc., etc.) intermingled in whatever order the original developers felt like putting it in, so just looking at the raw bytes is usually not enough to make much sense.
I used a table program to at least make the letters and numbers appear, but I'm not sure what I need to do to find where the values for the monsters experience/gold are.
There's actually a disassembly available on the wiki that includes this information and a whole lot more: https://datacrystal.romhacking.net/wiki/Dragon_Warrior_II::ROM_map/ASM
Monster stats are located starting at $04:$B7F5 and are each 15 bytes long. Gold is the third byte, experience is split between the fourth byte (containing bits 0-7 of the experience value), the high 2 bits of the ninth byte (containing bits 8-9 of the experience value), and the high 2 bits of the tenth byte (containing bits 10-11 of the experience value).
Taking a Metal Babble as an example:
; Enemy Stats (Max HP, [4-bit evade chance / 64, 4-bit unused], Max Gold dropped, EXP low byte, AGI, Attack Power, Defense Power, [2-bit attack probability list, 3-bit Sleep res., 3-bit spell damage res.], [2-bit EXP * 256, 3-bit Defeat res., 3-bit Stopspell res.], [2-bit EXP * 1024, 3-bit Defense res., 3-bit Surround res.], [4-bit Attack command 1, 4-bit Attack command 2], [4-bit Attack command 3, 4-bit Attack command 4], [4-bit Attack command 5, 4-bit Attack command 6], [4-bit Attack command 7, 4-bit Attack command 8], 8*1-bit use alternate attack command)
0x013BD4|$04:$BBC4:23 00 FF 1A C8 4B FF 3F 3F 7F 77 74 55 55 08 ; Monster ID #$42: Metal Babble
For gold, the third byte is 0xFF, so they can drop a maximum of 255 gold, and then for experience, the fourth byte is 0x1A, the ninth byte is 0x3F (high 2 bits are 0b00), and the tenth byte is 0x7F (high 2 bits are 0b01), so if we put them together we get 0x41A = 1050 EXP.