I need info on flat binaries =)

[SunSpyda]

Hi, I've been looking round the net for info regarding binary formats.

Apparently, the only format a BIOS can boot from is a flat (raw) binary that contains no extra OS specific info. I have found the option on my assembler to do this, but I just want to know a few things about flat binaries first....

Firstly, is there anyway I can execute flat binaries within a OS, without using emulators or VMs? I heard that you can only run OS (Win32, ELF etc.) binaries on OSes, without extra tools. I don't have a extra test machine, so I need a way to test flat binaries within my OS (Which is OpenBSD), before trying to boot off anything.

Secondly, why do OSes use OS binaries and not flat binaries? This is a question I have Googled, but I haven't found a real answer. Is it because OS binaries are more convenient for the OS over flat binaries?

Thirdly, if I'm writing large bootable applications, should the entire thing be written in flat binary, or should I load some sort of program that executes before anything else and runs the rest of the code in a OS binary format?

Fourthly, Can I just write a ISO/floppy image with just the flat binary on it as a file, and then just tell the VM BIOS to boot off it and it will work, or does it require more work than this?

Sorry for bombarding questions, I have been Googling quite a bit, but there seems to be varying (And sometimes completely opposing) answers scattered across the net, so I decided to ask here, to get some solid answers....

Thanks in advance   :bg

[dedndave]

well, in 16-bit DOS world, a .COM file is as close as it gets to raw binary under an OS
even then, the kernel provides many functions that would otherwise have to be written by you

a good place to start, in learning what you want to do, is the BIOS, itself
it is good to understand what happens when, say, a system boots from a floppy disk
it is fairly easy to disassemble the boot sector from a disk to see what happens

about 20 years ago, i wrote a specialized application for
public schools to run their video/security/fire alarm systems
i used DOS 3.3 (they had to pay for the entire DOS)
i removed the command.com program, and replaced it with one i wrote
that way, i had the kernel available without re-inventing the wheel

[FORTRANS]

Hi,

   Go to Google Groups and look at alt.os.development.  The FAQ
and past discussions should help.

   1)  You probably cannot execute a boot image in a normal OS
environment.  You could walk throught parts of the code in a
debugger.

   2)  Right.  The OS needs some control of the load process.
Except for the DOS *.COM programs, you must adjust things
at load time before execution.

   3)  That is up to you.  But that said, most seem to write a smal
boot-loader that loads something larger, that then loads the kernal.

   4)  Should work like that, but then why is there a bunch of usenet
groups, forums, books, and computer science courses to address
writing an OS?

$0.02

Steve N.

[vanjast]

I haven't looked at this since DOS days...

1) I'm not sure whether the cpu still starts in 16 bit (Real) mode, and you have to do the 32bit (64bit) switch into protected mode. I think it's more likely that you have to switch to Real mode.

2) I don't think there is much support for Real mode any more except from people who have kept that part of DOS alive.

3) There are still a lot of 'embedded controller' mobos (a smaller compressed motherboard) that have there own BIOS's and probably still use DOS in  REAL Mode, as this has always proved more stable for industry applications where reliability is more important than fancy visuals.

For information you'd want to get hold of the:
- Intel CPU books and datasheets.
- The Mobo manufacturers cct diags and BIOS lisitngs (This is most likely not possible without serious spadework)
- The individual chip (on the mobo) datasheets
- Get hold of the DOS Interrupt List and OS books (a mine of info in these books)

An erratic reply, but hope the general direction helps.
:dazzled:

[sinsi]

Go over to FASM's OS Construction, get yourself Bochs (or Virtual PC, or VMWare) and FASM. Forget about MASM for that kind of stuff - it's too hard to do absolute address coding, and MASM32's license won't let you...

[SunSpyda]

Go over to FASM's OS Construction, get yourself Bochs (or Virtual PC, or VMWare) and FASM. Forget about MASM for that kind of stuff - it's too hard to do absolute address coding, and MASM32's license won't let you...

Yeah, I think I will just end up installing VMWare Workstation. That's a honest bit of advice, coming from a MASM forum  :bg

[BogdanOntanu]

I have split the slightly "hot" and off topic posts to this thread: http://www.masm32.com/board/index.php?topic=11261.msg83177#msg83177

[mitchi]

Go over to FASM's OS Construction, get yourself Bochs (or Virtual PC, or VMWare) and FASM. Forget about MASM for that kind of stuff - it's too hard to do absolute address coding, and MASM32's license won't let you...

Yeah, I think I will just end up installing VMWare Workstation. That's a honest bit of advice, coming from a MASM forum  :bg

Tip :

You can install VMWARE workstation trial version. Make yourself a Virtual machine with it. Use it until the end of the trial. Then you download VMWARE player and you can keep using your Virtual Machine ;)

[Shantanu Gadgil]

I personally would recommend VirtualBox (http://virtualbox.org) as a VM solution.

I think what 'sinsi' meant about the license thing is that MASM doesn't allow creating binaries (of any sort) for non Microsoft OSes
The above is a very simplified version of the "interpretation" of the MASM32 license, reading the entire license yourself always helps!!!  :bg

(I think a better "expert" on licensing can elaborate this, if required!  :bg )

Regards,
Shantanu

[SunSpyda]

My VM solution is sorted, I got a enterprise copy of VMWare Workstation off a torrent site. I have been using VMs for quite a long time on many host OSes, and VMware has seemed the best out of the *many* I have tried.

VirtualBox is decent, but it lacks Guest OS support, then again I do test a *lot* of wacky OSes like Plan 9, Hurd and Darwin.

Anyway....

So, am I right in saying that the BIOS is in 16bit real mode and then the OS gets booted by the BIOS (By a 16bit real mode flat binary), which in turn switches the system to 32/64bit protected mode?

So, if I were to build a bootable application, it would need to be a flat binary that supports 16bit real mode. I could remain in this mode, or I could get the running binary to 'upgrade' the system into 32/64bit protected mode?

Exactly what limits are there to running a app in real mode? I assume protected mode has advantages, else it wouldn't be used.

Thanks for the help everyone =)

[FORTRANS]

Anyway....

So, am I right in saying that the BIOS is in 16bit real mode and then the OS gets booted by the BIOS (By a 16bit real mode flat binary), which in turn switches the system to 32/64bit protected mode?

   That's pretty much it.  The BIOS loads a boot sector and starts it in real mode.
The BIOS need not run in real mode, but the boot code loaded starts that way.

So, if I were to build a bootable application, it would need to be a flat binary that supports 16bit real mode. I could remain in this mode, or I could get the running binary to 'upgrade' the system into 32/64bit protected mode?

   Right.

Exactly what limits are there to running a app in real mode? I assume protected mode has advantages, else it wouldn't be used.

   You can use 32 bit instructions in 16-bit real mode, so the biggest limit is
the 64 kilobyte segments.  16-bit protected mode is also limited, the big
advantage to 32-bit protected mode is the 4 gig flat addressing.

HTH,

Steve N.

[redskull]

Just so I know what I'm talking about: technically the BIOS only loads the MBR (at the first sector of the disk) which contains code for only  searching out the one 'active' partition from the table, and then loads the Volume Boot Record (from the first sector of that partition), which is the O/S specific code to start the show (load files, switch to protected mode, etc); right?  Basically, there's generally no system code in the MBR, only the VBR?

-ac

[BogdanOntanu]

Just so I know what I'm talking about: technically the BIOS only loads the MBR (at the first sector of the disk) which contains code for only  searching out the one 'active' partition from the table, and then loads the Volume Boot Record (from the first sector of that partition), which is the O/S specific code to start the show (load files, switch to protected mode, etc); right?

Yes, with the small detail that the MBR will have to relocate itself away to another address because the VBR expects to be loaded to the exact same address "as if" the BIOS was loading it. 

Also it is usually not enough code space in the 512 bytes of the VBR in order to load the whole OS and switch to protected mode, etc. Hence most of the times there is another "stage two" loader that is loaded by the VBR and this one handles the OS file loading and start-up.

And for floppy disks the MBR is the VBR ;)

  Basically, there's generally no system code in the MBR, only the VBR?
-ac

Yes. But this is by convention only. As far as BIOS is concerned it's loading task was done when it loaded the MBR. The rest of this "chain loading" is done by MBR in order to facilitate multiple bootable partitions on the same disk.

[BogdanOntanu]

So, am I right in saying that the BIOS is in 16bit real mode and then the OS gets booted by the BIOS (By a 16bit real mode flat binary), which in turn switches the system to 32/64bit protected mode?

The BIOS loads a 512 bytes sector from a floppy or a HDD or a CD in emulation mode or uses El Torrito standard for CD booting or loads a 512 byte sector from a USB stick or uses network to load a boot file.

However for most of your cases you can consider that it loads a 512 bytes "flat binary". It is the task of this 512 bytes code to use the BIOS for loading another "flat binary" that contains your code and/or to switch to protected  or long mode.

So, if I were to build a bootable application, it would need to be a flat binary that supports 16bit real mode. I could remain in this mode, or I could get the running binary to 'upgrade' the system into 32/64bit protected mode?

Your 512 bytes MBR or VBR has to be "flat binary" in 16 bits real mode beacuse the BIOS loads it like that at address 0000:7C00h.

However the next file that your code loads could be in whatever format you prefer or you know, either binary or PE or ELF or your own format.

Exactly what limits are there to running a app in real mode? I assume protected mode has advantages, else it wouldn't be used.

There are many limitations:
1) You can not access memory above 1M (well 1M+64k if you use HMA).
2) Your code has to use segments.
3) You can not access hardware that is memory mapped above 1M.
4) 32bit instructions are longer/slower because of prefix

Advantages:
1) BIOS has a lot of functions already done for you and it does support your hardware.


There are many advantages of using protected mode:
1) Use of full memory 0...4G
2) Access to memory mapped devices
3) 32bits instructions need no extra prefix
4) No need to use segments (on a flat memory model)

Disadvantage of protected or long mode:
1) You have no support from BIOS and hence you must write a driver for each and every hardware available out there :D

This is a huge problem for a beginner or a hobby OS or bootable application. It took Windows a lot of years and thousands of programmers to write those.

Hence without protected mode your application will be very limited / crippled on today hardware and with 32/64bit protected/long mode you will have a huge task at hand: hardwire drivers.

That is why I consider that it is not worthy to write such an application unless you are very very experienced.

It will take a lot of time and it will fail on many hardware and a lot of experience structure and knowledge is required. You must be a very experienced programmer that has written at least a few huge HLL and ASM functional applications before you can start such a project.

I can also be used as a learning toll but the frustrations will be major because hardware programming is not easy and the information is usually secret or not available.

Hence you will learn better if you stick with an already done OS like Windows and learn how to use it's API. Then maybe one day when you have learned everything and gained a lot of experience THEN you can start this task also.

[mitchi]

I think that creating your own operating system is probably something you can do alone if you are a very experienced programmer with lots of time to read through complicated documentation and learn specialized knowledge.

I had an Operating system class this semester and frankly I really liked it. The algorithms for the file system and memory management are all so interesting, I really enjoyed every class of it  :bg. Bodgan, I see that you've created one :Solar_OS, which part of the coding did you enjoy most? And which part did you dislike most ?  :wink