At the beginning of the 90’s, the PC platform was often mocked by its rivals. PCs of that era had become much more powerful than, say, an Amiga 500. But the Amiga offered a flat memory address, while a DOS program could only access memory using cumbersome 64 KiB segments. And to add insult to injury, there was this strange 640 KiB memory limitation. No matter how much physical memory you had in your box, the utter most important Conventional Memory was limited to 640 KiB!

The Legend teaches us that Bill Gates once declared that “640 KB ought to be enough for anybody“, then designed MS-DOS to enforce this limitation.

The truth is of course a little more complicated than that.

An IBM PC legacy

Segmented memory

The first “IBM PC”, numbered 5150, was designed around the Intel 8088, a castrated version of the 8086. As a 16 bit CPU, it can only manipulate addresses up to 64 KiB in its registers. Thus, in order to access more memory, a bigger address space is split into multiple segments.

The way the 8086 generates addresses from segments is really peculiar. Instead of using a register to identify a segment and another one to indicate an address inside it, a new segment begins every 16 bytes. As segments are 64 KiB long and there is no memory protection, it means that segments overlap. Any physical address can be accessed by up to 4096 segment+offset combinations! This also caps the maximum addressable memory to around 1 MiB, corresponding to the 20 bit address bus of the 8086/8088.

Segmented memory The x86 segmented memory model is quite peculiar: segments can overlap!

The 640K ceiling

With its 8088, the IBM PC could address up to 1 MiB of memory. A huge amount in 1981.

This address space had to be shared between the RAM, the video memory, and the various peripherals. IBM decided that the lower 640 KiB would be reserved for the (user accessible) RAM, while the upper 384 KiB would be used to address the video memory, the BIOS and the peripherals. These two zones should be to be big enough to fulfill their purpose…

And Microsoft? It never imposed any limitation!
Take for instance the almost forgotten Apricot PC, which was another MS-DOS machine of the time, albeit incompatible with the IBM PC. It puts the frontier elsewhere and the user accessible RAM could be sized up to 768 KiB.

So the infamous 640K memory ceiling was due to IBM! And also a bit to Intel…

The 286, a “brain dead chip”

In 1984, IBM commercialized the IBM PC/AT (“Advanced Technology”!) based on an Intel 80286. The 80286 is of course fully compatible with the 8086. It is still a 16 bit CPU relying on segmented memory, but it can access to up to 16 MiB of memory. And it provides some form of memory protection.

But…

In order to retain compatibility with the programs and OSes written for the 8086, all these shiny features are only accessible when running in the newly introduced Protected Mode. When operating in Real Mode (a.k.a “Legacy Mode”), the 286 manages memory just like a (faster) 8086. No memory protection and no access to more than 1 MiB of memory!

The Real Mode is the default operating mode of the 286 when starting up. Once loaded, the OS can switch to Protected Mode and let the user reap the benefits. Alas MS-DOS, along with its drivers and its existing applications, had been written when no PC came with anything else than a 8086 or a 8088. Unfortunately, their developers made some bold assumptions on that matter. Modifying MS-DOS to operate fully in Protected Mode would have been an extensive work and would have broken the compatibility with most applications and drivers!

A solution might have been to switch back and forth from Protected Mode. But entering Protected Mode on the 286 is a one way ticket: if you want to switch back to Real Mode you have to reset the CPU! Bill Gates, who understood very early all the implications, is said to have called the 286 a “brain dead chip” for that reason.

Hacking the way to more memory

Fortunately, some (rather hacky) schemes were devised to allow DOS programs accessing more memory. Otherwise, the PC as a computing platform would have failed…

Understanding these schemes requires some naming conventions.

  • Conventional Memory is the precious first 640 KiB of addressable memory. In MS-DOS the instructions of a program shall reside in it. That partially explains why some latter programs / games required most of conventional memory to be at their disposal. And that is why freeing a mere KiB of it was a worthy battle.
  • Upper Memory Area refers to the area that was reserved by IBM for the video adapter and the various peripherals. Note that this area is only reserved and is not always entirely consumed!
  • Extended Memory is the memory out of reach of for a 8086/8088 (past the first MiB). It is only naturally accessible on latter Intel CPUs. This is the memory pool which access is required for more advanced computing purposes.

Memory map IBM PC Memory map (by Wtshymanski, CC BY-SA 3.0

HMA (High Memory Area)

In Real Mode, memory segments start every 16 bytes and are 64KiB long. So the last segment exceeds the 1 MiB memory limit by almost 64 KiB!

On a 8086 this is a non-problem. Indeed, its 20 pin address bus makes impossible to access more than 1 MiB of memory. All generated addresses that exceed 1 MiB are aliased to the very first addresses.
But the 286 is equipped wit a 24 pin address bus. So it can access this memory block, even in Real Mode!

This very last segment is called HMA (High Memory Area). This caused compatibility issues with some 8086 programs that used the last segment, expecting it to loop back to the first addresses. But it also allowed HIMEM.SYS to load some parts of MS-DOS 5 to HMA on a 286/386/486, freeing up to 46 KiB of precious conventional memory.

EMS (Expanded Memory Specification)

In 1985 Lotus, Intel and Microsoft announced the EMS standard, allowing a standardized access to extended memory.

A 64 KiB unused and contiguous portion of Upper Memory is identified and split into four 16 KiB segments. A hardware assisted bank switching mechanism and an appropriate DOS driver are then used to access any location in extended memory when requiring access to those four segments.

It potentially gives the applications access to the whole 16 MiB address space, even in Real Mode.  Although, programmers need to manually configure the four 16 KiB “views” via the driver.

EMS EMS Illustrated: a free 64K block from Upper Memory gives a “view” to four 16K pages from Extended Memory.

Great! Is there any drawback? It requires dedicated hardware support on the motherboard and the use of a driver consuming a few KiB of previous conventional memory. And programmers cannot easily manipulate data structures bigger than 16 KiB.

Quoting Bill Gates himself:
“It’s garbage! It’s a kludge! … But we’re going to do it!”

The dedicated hardware often takes the form of an expansion board that can also by used by older 8086/8088 machines.

XMS (Extended Memory Specification)

Another scheme exists to access extended memory from a Real Mode DOS program: XMS. XMS is a specification that defines an interface to all necessary functions to allocate, free, and copy continuous blocs in extended memory. Of course, only a Protected Mode program has access to these blocks. But for DOS Real Mode programs, there is a hack!
The idea: switch the CPU into Protected Mode and copy segments of data from/to extended memory to/from unused segments of upper memory, then to switch back to Real Mode to allow a DOS program to access it.

The advantages? It does not require any dedicated hardware. It also frees the programmer from being caped to 16 KiB of continuous memory.
The drawback? There are a lot of copies involved. Furthermore, the 286 requires a reset to switch back to Real Mode, making the use of XMS on a 286 extremely slow! Thus it was rarely used, until later…

EMS and XMS both require free blocks in upper memory. If there are not enough free blocks available for both to be available, the user has to choose which standard he will enable in the boot configuration files. His judgment will be based on the programs he intends to launch. Configuring MS-DOS on a 286 was not for the faint hearted!

HIMEM.SYS, which manages the access to HMA, is also in charge to provide an access to extended memory.

To be fair with the 286, a clever method was devised in latter revisions of HIMEM. It allows accessing XMS without requiring a switch to Protected Mode. This method is built around LOADALL, an undocumented 286 instruction. This instruction manipulates the entire processor state, including “hidden” registers that are out of reach of normal instructions. A hidden segment register exists among them, which can be modified to access memory addresses beyond 1 MB, even in Real Mode.

This trick is only used on the 286 as later CPUs do not require it.

Enter the 386

When it designed the 80386, Intel now understood well that 8086 and DOS compatibility would be its main asset. Neat features were also added.

  • A MMU (Memory Management Unit) which performs “Virtual Memory” management, mapping virtual addresses into physical space. This opens the way for modern OSes such as Linux.
  • The Virtual 8086 mode, a sub-mode of Protected Mode, in which Real Mode program runs in a kind of virtual machine. The monitor itself operates in Protected Mode so advanced memory usage such as paging and isolation are possible.
  • An instruction to quickly switch between Protected Mode and Real Mode on the fly.

XMS is now easily usable on a 386!

The 80386 still relies on a segmented memory model. But being a 32 bit CPU, the segment can be 4 GiB long, giving the same facilities as a flat memory model.

EMM386.EXE, the memory manager introduced with MS-DOS 5 in 1991, locks DOS itself into a Virtual 8086 task. It also provides DOS with a driver emulating EMS, using extended memory from Protected Mode. This driver makes use of the 386’s MMU to map blocks into the “EMS Window”: no memory copy occurs. Thus, EMS memory is faster than XMS. But it is still awkward to use.
EMM368 is also able to load device drivers and TSRs into Upper Memory Area if enough free segments are available. By using LOADHIGH and DEVICEHIGH directives in autoexec.bat and config.sys, instead of their older counterparts LOAD and DEVICE, it is possible to free dozen KiBs from the conventional memory!

To be fair, those techniques were first introduced by Quaterdeck in its QEMM memory manager, then by Digital Research in its DR-DOS 5, before being mimicked by Microsoft in MS-DOS 5.

DOS, extended!

With EMS, XMS and ways to free most of conventional memory, PCs with MS-DOS became serious contenders in the video gaming platforms arena. However, accessing EMS and XMS is a slow process. And, even on a 386, they don’t allow to get rid of the 16 bit memory segmentation schemes.
Amiga users still had many arguments to mock PC users… until the generalization of DOS Extenders, which finally allowed the PCs to fly. One of the first games making use of a DOS Extender was no-less than DOOM!

DOS4GW When you saw DOS4/GW, you knew you were in for some serious gaming. If you have ever wondered what that DOS4/GW is, it’s a DOS Extender!

DPMI (DOS Protected Mode Interface)

One major software editor impeded by the 640K memory limit was Microsoft itself! Windows 3.0 was an ambitious project, but it required more memory and some facilities that Real Mode was unable to provide. Thus Microsoft introduced DPMI in 1989, before giving its control to an industry committee.

DPMI describes a collection of services to be offered by a DPMI host, which allows client applications, called DMPI clients, to run in Protected Mode while still being able to make use of DOS services in a well-behaved and hardware-independent way.
In order to fulfill its role, a DPMI host has to perform mode switching, allocate extended memory, allocate conventional memory, control the interrupt subsystem, communicate with Real Mode programs, and read or write certain CPU control registers.
Running in Protected Mode, some form of multitasking between DPMI clients is possible while, for compatibility sake, Real Mode TSRs are still present alongside!

Although it is quite possible to code a DPMI host on a 286, most of them require a 386. Thanks again to its fast mode switching and its Virtual 8086 mode!

DOS Extenders

The DPMI specifications allowed the apparition of many DOS Extenders, which are DPMI clients that load and take control of an application. The purpose is to service them with an extended version of the APIs that DOS and the BIOS provide via software interrupts (respectively 0x21 and 0x10). In order to avoid switching back to Real Mode too often and save performance, DOS Extenders also handle frequent hardware interrupts, such as those generated by a mouse.

With a 32 bit DOS Extender, a 386 is able to run true 32 bit programs and to allocate huge blocks of memory. At last!
This also ends the time when the conventional memory pool was the fastest available. Applications built on DOS Extender thus require far less free Conventional Memory, easing the painful process of crafting custom autoexec.bat and config.sys in order to run ones favorite games.

DOS4GW games DOS Extenders finally allowed games to run in 32 bit Protected Mode, making at last good use of the powerful Intel CPUs.

While still a little bit hacky, DOS extenders finally allowed PCs to make good use of their memory and powerful CPUs. A few years later, Windows 95 made those memory headaches a thing of the past as MS-DOS became progressively irrelevant.

On the same subject