Embedded data in EXE

[RedMike]

There are a number of things embedded into the EXEs themselves, which means both reading and modifying them is much more complex than normal data. An initial implementation would likely just be to read this data as read-only in order to allow the editor to display it usefully. A full editing implementation might start being as complex as just reimplementing the game engine to some extent, so it's unclear how useful that would be; but a small editing implementation might let strings be changed while locking them into the current length, etc.

Known data embedded and fairly easy to find visually (but not programmatically):

• Mission Sets - the combination of missions and strings to identify VICTIM/etc, arbitrary number of crimes in each mission set but maximum of 6 (otherwise there's an overflow that leads to some placeholder data being loaded, specifically Max Remington's apartment as a location and a mattress full of cash as the object); modifying this data is non-trivial because changing the lengths of the strings or the count of crimes in a mission set would require updating the relocation table and potentially direct instructions if offsets are encoded directly somewhere. Additionally there are many bits of data in the mission set that seem to be pointers to various other bits of data.
• Organisation Names - despite the WORLD files containing a list of organisations, this also exists in the EXE files, probably to allow things like the save files or hall of fame screen to correctly show them?
• References to animation files as well as prose/text/dialogues - this also includes an apparently placeholder of animation.pan which doesn't seem to match a file?
• Various in-game strings - menu options, prompts, difficulty levels, skill names, as well as some dialogue like the crime end briefing dialogue and others; also prefixes/suffixes/infixes for various things like describing actions, time, options, etc. All are null-terminated strings so changing the length would be a problem because it would require updating the relocation table and potentially direct instructions if offsets are encoded directly somewhere.

The EXE files themselves are reasonably simple to decompress/recompress (EXEPACK).

However the big problem is that there is no simple way to identify the correct places to modify, short of assuming explicit positions in the file; and even then, that won't allow changing the lengths of strings/counts of arrays/etc without also understanding the format of the instructions/etc.

[RedMike]

Have begun breaking this down gradually with LLM assistance for writing scripts quickly to test theories (otherwise I would have definitely given up by now). A lot of progress made already and will progress it further. EXEPACK implementation seems a bit odd (re-compressing fails to then run).

That said, successfully managed to uncover some previously-unknown functionality (Kitchen and Bed Room room types) and re-enabled them, which instantly worked. The sprites look terrible and the bedroom sprite locations were reused for icons, but still, it works.

[RedMike]

Also confirmed EXE includes some "RastPort" structures that are based off of the Amiga graphics API (with some fields removed); not sure if this is useful but the values map directly. Unfortunately the MicroProse DOS graphics library is proprietary and closed-source, even though they included some of the documentation in the binaries by accident.

[RedMike]

Successfully gotten decompression and recompression working as a script with LLM assistance, this should make investigation easier and also then converting the code to C# to allow project to read/modify the data.

[RedMike]

Initial EXE parsing has been begun in develop, with successful decompression and recompression, and also changing room names to confirm it works.

[RedMike]

Also uncovered the un-released Kitchen and Bed Room rooms, and re-enabled them successfully, with Kitchen having some (awful-looking) sprites for objects, and Bed Room being clearly only half-done. Will be fully implementing EXE editing if at all possible.

[RedMike]

Fun discoveries today:

• The EXE files have a literal animation.pan which obviously doesn't match any of the animation filenames...that's because it's a buffer into which the filename is placed before reading. This then means that you can't adjust an animation's filename to be longer than animation otherwise it'll splat the next data next to the buffer and corrupt the memory
• The PLOT/etc text files do have dead data in them because they have duplicate entries and the game silently stops after the first entry.
• One single screen in the entire game seems to use mid-string colour codes to make strings change colour halfway through, and it's the scoring screen (specifically the EP:X/Y strings) in the attached image. It took me a while to figure out why: there can be enough individual strings being drawn on-screen at once that most likely this could make CPUs of the time fail to cope. However interestingly these codes are always technically valid which means colour changing strings could be added anywhere.
• Additionally, when doing file IO, there's a recurring env.sve file that doesn't correspond to a real file I could see...that's because it's a special value used to signal that the write should go into the PSP memory area the EXE files use to communicate to each other

[RedMike]

Confirmed there's actually more mid-string colour codes in-game although all used for screens like that with many different strings. However the handling is present in ALL string handling which means it's possible to add the formatting to any of them. It's specifically just colour-formatting though; it's possible the two special TAC map colours (enemy shirt colour, player shirt colour) could get replaced in text there too but not on briefing screen at least.
edit: Confirmed that in TAC screens, enemy shirt colour is NOT properly replaced (stays yellow as default), but player shirt colour is weirdly replaced (instead of the magenta it is, it's replaced with black which is the default non-disguised player colour, even once the player gets a disguise).

[RedMike]

Turns out making all the fields editable will take quite a while due to pointer recalculation being a nightmare. That said, changing strings in-place is feasible. However all the LLM-generated code for it so far in the branch needs a heavy refactor to make it usable and extensible, now that I understand the problem space better.

[RedMike]

I continued on this work for a while but as time went on I realised that trying to map the entire EXE files to actual data would take as long as it'd take as just disassembling the entire binary/mapping the logic/effectively REing the game logic itself; particularly because of the lack of section separators and the binary data blobs making it really hard to figure out what data is what and how it works.

Because of this I'm going to reconsider the approach and probably do this:

1. Identify the data that would actually be useful to modify first, just based off of knowledge of the game
2. Run through and isolate where these sections of data are and implement only those, with the rest of the data being opaque blobs
3. Due to the lack of knowledge of the rest of the data, there is then no option to allow pointer recalculation, so data modification will be fairly limited (no adding extra values, no extending strings, etc)

Unfortunately, this does mean that some things that would have been nice (allowing mission sets with more than 3 crimes) may not be possible at this point.

[RedMike]

I've returned to this somewhat, but the point stands: there is no way to make this tooling actually modify the embedded data usefully; in particular for TAC.EXE it's impossible even with pointer relocation because the segments holding this data are near the limit (<10 bytes to the max size) so it would be less than useful.

In lieu of this I have begun a full DOS disassembly to arrive to usable C source code instead, which might be a way forwards long-term.