Rust on DOS

Rabbit Hole - 8月2022年 (Written 7月2023年)

why

so, in 2022 i was really into DOS stuff, and had learned rust not too long before. and i was thinking 'wow itd be cool to use rust on DOS' a few projects gave me some hope! specifically Serentty/rusty-dos and o8vm/rust_dos. but these both had the same issue - they're limited to 16-bit code. this might not seem like an issue do to DOS being a 16-bit platform, but rust does NOT like doing 16-bit when i tried to use it for a bit, and we don't actually have to be limited by it! DOS extenders were very common! i should have thought about that as just 'neat idea' but no. i did it.

what's a DOS extender?

so you might not know what the hell i'm talking about, i'll try to explain. since it was designed for early 16-bit PCs, DOS is a 16-bit operating system, which means you only get 64KiB to work with at a time, like for executable size. you can actually use 20-bit addresses through segmentation - 1MiB, although really you'd only use the first 640KiB... which is where that DOS limit comes from ...you can read my VGA bootloader tutorial if you really want to know more about 16-bit x86 memory.

anyway! DOS extenders came about once 32-bit computing hit, since people didn't want to not use DOS. instead there was a nice solution, x86 lets you have a 32-bit thing on top of 16-bit stuff! with an application managing the 32-bit side of the computer, 16-bit stuff can work alongside 32-bit stuff, and 32-bit stuff can run on your DOS PC instead of like, windows. but you can't just make a .COM or .EXE file and expect it to work, you have to provide a binary in a format your DOS extender uses. you don't have to worry about this if you're using like DJGPP or some other compiler which is meant for a DOS extender, and i figured i could just link with one of those, or something, and it'd be fine.

after struggling to get a COFF or convert with tools or really get anywhere with using existing tools like DJGPP and Open Watcom, i decided i would just do it myself. i bet i just missed something or could have done it easier by getting them to work, but hey, it was fun kinda.

i saw that the DOS/32A extender had good documentation, so i decided to use that. it uses LE executables, so it was time to get rust into one of those.

the LE executable

used by OS/2 and some DOS extenders, the LX/LE executable has not remained popular a few resources ended up being invaluable for figuring out this format:

• this summary of some section layouts (duplicated here)
• this text file explaining a lot of the format (duplicated here)
• the IBM OS/2 specification for LX/LE (duplicated here)
• DOOM.EXE - which is an embedded LE, extracted for ease here

here's a NASM assembly file with an LE stub as i figured it out: lestub.nasm

importantly, the LE executable uses relocations, and luckily the ELF executable, which everything good supports, uses them too for static libraries/unlinked objects.

then it was as simple as making an ELF to LE converter, getting rust to output a nice object file, and writing code!

bringing it together

first step: the ELF to LE converter. i did this in an incredibly hackish way after figuring out the format enough to manually make LE executables. i just got some rust library which can read ELF and in extremely brute and awful manner took what i needed and shoved it in LE. it's terrible. there are symbol problems. it doesn't support things that both formats support. the code is absolute awful spaghetti. it uses a weird intermediate file new.elf and always outputs as a.exe. it spams a ton of output to the console so that i could debug it easier. it's horrible "just make it work" code i intended on doing something better or even just improving it but never did may the code gods have mercy on me for releasing this with all that said, here's the code. feel free to take it and do whatever you want with it, if for some reason you must.

second step: making Rust do what we need. this is much less ugly, we just have to tell rust about the platform, and tell it to build some built-in libraries. here's a .cargo/config.toml which will do that, along with a platform target file. and here's a demo project, which shows how to set it up. usage is elf2le target/dos/release/file.elf then DOS32A A.EXE to run under DOS. you can do sc /bs A.EXE or sc /b A.EXE to create versions that you can run as A.EXE then it's just coding something. i did some stuff but nothing too interesting.

thanks for reading!

contact me with any questions or if you actually do something with this or do something better