[C64, 6502] Incorrect behavior for "undocumented" CPU functions

[SaxxonPike]

Summary

The test suite by Wolfgang Lorenz was used to verify the proper operation of undocumented 6502 functions in the C64 core, which uses the M6502X core. Some of the 6502 tests (for functionality common to all systems that use this core) did not pass:

• Undocumented opcodes ANE, ARR, SHX, SHY, SHA
• Decimal mode in SBC for input values that are not BCD representable (e.g. $9A)

There are some separate issues specific to the 6510's I/O port emulation but that will be considered a separate issue as it will only affect C64 (and not other emulator cores)

Repro

1. Download tsuit215_d64.zip from the PC64 test suite from this page: https://www.zimmers.net/anonftp/pub/cbm/crossplatform/emulators/msdos/pc64/
2. Load disk1.d64 or disk2.d64 into Bizhawk
3. Run the commands LOAD"*",8 and then RUN
4. Wait until the test halts with a register readout during tests such as "sbcb", "aneb" or "shaay"

Output

An example of some of the failed tests:

What it is supposed to look like (this demonstrates some of the fixes in an upcoming related PR):

Host env.

• BizHawk dev 2025-01-07

Considerations

The behavior of some opcodes (certainly SHA/SHX/SHY and ANE) are not consistent between systems of different models, but are more or less consistent among the same model - enough that they can be reasonably emulated with some confidence. On the C64, these opcodes are seen "in the wild" from commercial releases in the late 80s, for certain. (Wizball and Turrican are cited in the "No More Secrets" document which can be used as the authoritative source on how to properly emulate these for the C64 at least.)

We will need to know what to look for in the other systems that use the M6502X core - we have A7800 and NES at least, probably Lynx and possibly others. It is unlikely that this is a one-size-fits-all situation, but we do have concrete evidence of the behavior of these functions on all first-party and licensed 6502 CPUs.

I'd like to hear from other core devs before moving forward with this one.