Writing an OS

De Ensiwiki
Aller à : navigation, rechercher
Subject: Writing an OS - questions !!
Date: 5 May 92 01:57:05 GMT
Reply-To: nani@td2cad.intel.com (V. Narayanan)


Hi folks,
        For quite some time this "novice" has been wondering as to how one goes
about the task of writing an OS from "scratch".  So here are some questions,
and I would appreciate if you could take time to answer 'em.

1) How would you typically debug the kernel during the development phase?
2) Can you test the kernel functionality by running it as a process on a
   different OS?  Wouldn't the OS(the development environment) generate
   exceptions in cases when the kernel (of the new OS) tries to modify
   'priviledged' registers?
3) Would new linkers and loaders have to be written before you get a basic
   kernel running?
4) Was Linux developed in a DOS environment?  Maybe Linus could shed some light
   on this subject.

Thanks in anticipation.


N. Venkat


From: bazyar@mrcnext.cso.uiuc.edu (Jawaid Bazyar)
Newsgroups: comp.os.linux
Subject: Re: Writing an OS - questions !!
Date: 5 May 92 03:33:10 GMT
Organization: University of Illinois at Urbana

nani@td2cad.intel.com (V. Narayanan) writes:


   Having just written a Multitasking OS, I couldn't resist...

>Hi folks,
>        For quite some time this "novice" has been wondering as to how one goes
>about the task of writing an OS from "scratch".  So here are some questions,
>and I would appreciate if you could take time to answer 'em.

>1) How would you typically debug the kernel during the development phase?

   On my machine (Apple IIgs), we have a machine-language debugger which
has a number of nifty features, the most important here being that it
'turns-on' whenever certain reserved software traps are activated.  You
can intersperse these in your code to trace certain locations.
   And, there's always the infamous "printf" method, printing out every
value you deem relevant to the problem.
   You probably won't be able to use existing source-level debuggers (see
below), but this is just conjecture on my part and indeed depends on how
you initially implement the system.

>2) Can you test the kernel functionality by running it as a process on a
>   different OS?  Wouldn't the OS(the development environment) generate
>   exceptions in cases when the kernel (of the new OS) tries to modify
>   'priviledged' registers?

   Yes, and in fact many classes in OS design are taught with this principle.
"Operating System Design" by Douglas Comer implements a Unix-like OS
called XINU.  There is source code publically available for a version
of XINU which runs under UNIX systems.  This system works by basically
implementing threads (multiple processes inside one real Unix process).
   Your mention of the new OS accessing privileged registers is a good question,
and as far as I know there's no good answer except one: you have to get
the basic multitasking and low-level OS features working by sweat alone.
Usually, you have to boot into your new OS, instead of running it from
your development system.  This means that you have a really tedious
test, reboot, compile, reboot, test cycle.

>3) Would new linkers and loaders have to be written before you get a basic
>   kernel running?

   No, the standard technique is to use a cross compiler of some sort.
If the OS you're writing is for the same processor your development system
is on, then most of the problem is solved.  All you need to do is avoid
using any library routines that use the development system's OS.  Now,
when you get to the point where you want to start using the C compiler
under your new system, you have a bit of a problem (unless your new OS
can run other OS's software, kinda like OS2->Windows, Windows->DOS, etc).
   As far as loaders, you probably would want to for later user programming, 
but most OS's are written to be loaded into a pre-defined area of physical 
memory, and thus you won't need a loader to boot your system.  In any
event, loaders are in general fairly simple, and you could probably
emulate existing ones (so as to be able to use existing compilers, etc).

   Writing an OS is probably one of the most satisfying things I've done;
I wouldn't recommend it for the faint of heart, however. :)

--
 Jawaid Bazyar              |   Ask me about the GNO Multitasking Environment
 Procyon, Inc.              |   for the Apple IIgs!   (314) 334-7078
 bazyar@cs.uiuc.edu         |   1005 N. Kingshighway, Suite 309 
 --Apple II Forever!--      |   Cape Girardeau, MO 63701


From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroups: comp.os.linux
Subject: Re: Writing an OS - questions !!
Date: 5 May 92 07:58:17 GMT

In article <10685@inews.intel.com> nani@td2cad.intel.com (V. Narayanan) writes:
>
>Hi folks,
>        For quite some time this "novice" has been wondering as to how one goes
>about the task of writing an OS from "scratch".  So here are some questions,
>and I would appreciate if you could take time to answer 'em.

Well, I see someone else already answered, but I thought I'd take on the
linux-specific parts.  Just my personal experiences, and I don't know
how normal those are.

>1) How would you typically debug the kernel during the development phase?

Depends on both the machine and how far you have gotten on the kernel:
on more simple systems it's generally easier to set up. Here's what I
had to do on a 386 in protected mode.

The worst part is starting off: after you have even a minimal system you
can use printf etc, but moving to protected mode on a 386 isn't fun,
especially if you at first don't know the architecture very well.  It's
distressingly easy to reboot the system at this stage: if the 386
notices something is wrong, it shuts down and reboots - you don't even
get a chance to see what's wrong.

Printf() isn't very useful - a reboot also clears the screen, and
anyway, you have to have access to video-mem, which might fail if your
segments are incorrect etc.  Don't even think about debuggers: no
debugger I know of can follow a 386 into protected mode.  A 386 emulator
might do the job, or some heavy hardware, but that isn't usually
feasible.

What I used was a simple killing-loop: I put in statements like

die:
	jmp die

at strategic places.  If it locked up, you were ok, if it rebooted, you
knew at least it happened before the die-loop.  Alternatively, you might
use the sound io ports for some sound-clues, but as I had no experience
with PC hardware, I didn't even use that.  I'm not saying this is the
only way: I didn't start off to write a kernel, I just wanted to explore
the 386 task-switching primitives etc, and that's how I started off (in
about April-91).

After you have a minimal system up and can use the screen for output, it
gets a bit easier, but that's when you have to enable interrupts. Bang,
instant reboot, and back to the old way. All in all, it took about 2
months for me to get all the 386 things pretty well sorted out so that I
no longer had to count on avoiding rebooting at once, and having the
basic things set up (paging, timer-interrupt and a simple task-switcher
to test out the segments etc).

>2) Can you test the kernel functionality by running it as a process on a
>   different OS?  Wouldn't the OS(the development environment) generate
>   exceptions in cases when the kernel (of the new OS) tries to modify
>   'priviledged' registers?

Yes, it's generally possible for some things, but eg device drivers
usually have to be tested out on the bare machine.  I used minix to
develop linux, so I had no access to IO registers, interrupts etc. 
Under DOS it would have been possible to get access to all these, but
then you don't have 32-bit mode.  Intel isn't that great - it would
probably have been much easier on a 68040 or similar. 

So after getting a simple task-switcher (it switched between two
processes that printed AAAA...  and BBBB...  respectively by using the
timer-interrupt - Gods I was proud over that), I still had to continue
debugging basically by using printf.  The first thing written was the
keyboard driver: that's the reason it's still written completely in
assembler (I didn't dare move to C yet - I was still debugging at
about instruction-level). 

After that I wrote the serial drivers, and voila, I had a simple
terminal program running (well, not that simple actually).  It was still
the same two processes (AAA..), but now they read and wrote to the
console/serial lines instead.  I had to reboot to get out of it all, but
it was a simple kernel.

After that is was plain sailing: hairy coding still, but I had some
devices, and debugging was easier.  I started using C at this stage, and
it certainly speeds up developement.  This is also when I start to get
serious about my megalomaniac ideas to make "a better minix that minix". 
I was hoping I'd be able to recompile gcc under linux some day... 

The harddisk driver was more of the same: this time the problems with
bad documentation started to crop up.  The PC may be the most used
architecture in the world right now, but that doesn't mean the docs are
any better: in fact I haven't seen /any/ book even mentioning the weird
386-387 coupling in an AT etc (Thanks Bruce). 

After that, a small filesystem, and voila, you have a minimal unix.  Two
months for basic setups, but then only slightly longer until I had a
disk-driver (seriously buggy, but it happened to work on my machine) and
a small filesystem.  That was about when I made 0.01 available (late
august-91? Something like that): it wasn't pretty, it had no floppy
driver, and it couldn't do much anything.  I don't think anybody ever
compiled that version.  But by then I was hooked, and didn't want to
stop until I could chuck out minix.

>3) Would new linkers and loaders have to be written before you get a basic
>   kernel running?

All versions up to about 0.11 were crosscompiled under minix386 - as
were the user programs.  I got bash and gcc eventually working under
0.02, and while a race-condition in the buffer-cache code prevented me
from recompiling gcc with itself, I was able to tackle smaller compiles. 
0.03 (October?) was able to recompile gcc under itself, and I think
that's the first version that anybody else actually used.  Still no
floppies, but most of the basic things worked.

Afetr 0.03 I decided that the next version was actually useable (it was,
kind of, but boy is X under 0.96 more impressive), and I called the next
version 0.10 (November?).  It still had a rather serious bug in the
buffer-cache handling code, but after patching that, it was pretty ok. 
0.11 (December) had the first floppy driver, and was the point where I
started doing linux developement under itself.  Quite as well, as I
trashed my minix386 partition by mistake when trying to autodial
/dev/hd2.

By that time others were actually using linux, and running out of
memory.  Especially sad was the fact that gcc wouldn't work on a 2MB
machine, and although c386 was ported, it didn't do everything gcc did,
and couldn't recompile the kernel.  So I had to implement disk-paging:
0.12 came out in January (?) and had paging by me as well as job control
by tytso (and other patches: pmacdona had started on VC's etc).  It was
the first release that started to have "non-essential" features, and
being partly written by others.  It was also the first release that
actually did many things better than minix, and by now people started to
really get interested. 

Then it was 0.95 in March, bugfixes in April, and soon 0.96. It's
certainly been fun (and I trust will continue to be so) - reactions have
been mostly very positive, and you do learn a lot doing this type of
thing (on the other hand, your studies suffer in other respects :)

		Linus


From: sinster@scintilla.capitola.ca.us (Darren Senn)
Newsgroups: comp.os.linux
Subject: Re: Writing an OS - questions !!
Date: 5 May 92 18:29:42 GMT

I started an OS called Syrinx in Summer of 1990.  It started off being
286 specific (that was what I had), and is now 486 specifc (thank god
for upgrades)!  Feeling an overwhelming desire to continue this thread,
here's my 2 cents.

Syrinx was largely inspired by BSD 4.3 and some little hardware experiments
that I did in my Computer Engineering studies.

I started off from scratch, by writing some utilities under Borland C that
allowed me to format a partition with the Syrinx filesystem, and then copy
files to the filesystem, etc.

Once I did that, I pulled up TASM, and started with my bootsector.  I wrote
a little bootsector that could multi-boot between DOS and Syrinx.  If you
hit return, it would boot Syrinx, but if you typed "DOS", it would go to
the first primary DOS partition and boot that.  In order to boot Syrinx,
I loaded a file called "/init" and executed it.

The first version of the boot sector took about a month to write and debug,
(I HATE DEBUGGING BOOT CODE!) and had the location of /init hardwired into
it.  The second version actually searched the filesystem for /init by looking
through directories and the inodes, and took another two months.

Linus mentioned his "die loops".  I did something very similar to that, though
my die loops started off by creating an exploding window in the middle of the
screen (ok, so I was bored :)  ), and putting the values of all the registers
into that window, then it went into a pretty solid loop:
	@@die:	sti		; Disable interrupts
		hlt		; Halt the computer
		jmp @@die	; If I get out of the halt, repeat!
So this thing pretty solidly stopped the microprocessor, and I didn't have
any problems with reboots clearing the screen.  Very helpful.

I had to deal with the debug/compile/write/reboot/... loop that Jawaid
Bazyar mentioned.  It was slow, but it worked...

There were two big things that slowed me down:
	a) My filesystem was fully formed from the start, and
	b) I switched to protected mode through the BIOS.

I didn't (at that time) know how protected-mode friendly BIOS was, so I
went ahead and used the BIOS function call to switch into protected mode.
That gave me the bonus of getting a descriptor for the BIOS data segment
in my GDT for free.  Very helpful.  Unfortunately, I really had to go
through loops to keep the BIOS from using my hardware interrupts.  Very
ugly.

I never did get as far as Linux has come, so when I found Linux last month,
I jumped in with both feet.  Some of my Syrinx stuff is better than Linux's,
but most of Linux is far more advanced than Syrinx ever got.

In general, I think Linus' tactic of getting a bare system up first is a
better bet than trying to make everything fully-formed as you go.  I
never did get a C compiler, but I did write an assembler that worked under
Syrinx (I called it Sinas).

One thing that helped my Syrinx development was DOS EXE compatibility.  I
checked the header of any files that you tried to execute, and if it was
a DOS EXE header, then I ran it in V86 mode.  I could get a number of things
to run, but anything that tried to switch to protected mode would die in
a rather horrible way, so there were a lot of things I couldn't use.  I
also didn't support EMS or XMS or DPMI, so those programs were all out.
Basically, what it came down to was that I couldn't run any commercial
programs.  :(

Syrinx has now been shoved way back on the burner, however, and I've
gone whole-hog into Linux.

One of the first things I did when I got Linux was to look at the kernel
code (the equivalent to my /init).  I was rather surprised to see that
Linus switched to protected mode all on his own, without going through
the BIOS.  I also like his little comment in the sources:
	! This is how real programmers do it
or something to that effect.  :)


From: wolff@neuron.et.tudelft.nl (Rogier Wolff)
Newsgroups: comp.os.linux
Subject: Re: Writing an OS - questions !!
Date: 6 May 92 09:51:12 GMT

rhys@cs.uq.oz.au (Rhys Weatherley) writes:

>Thanks for posting that Linus - it's very good to read how it all began.  I
>hope that the FTP site administrators save away your message as a piece of
>Linux folklore.  I especially liked the bit about you being proud of your
>two AAAA... and BBBB... processes. :-)

>Cheers,

I once "ported" XINU to a PDP 11. (It was written on one, but you don't 
get some, necessary software with the book, and you'd need a VAX that we
didn't have to run it. Also some PDP11s have hardware MUL and DIV while
others don't)

Anyway we ran the same test: One process printing "process A" and the
other "process B". This resulted in the following output:

process A
process A
process A
prpocess A
process A
process A
procress A
process A

And we where thinking of bugs in the tty code, when it struck that the
output buffer was filling, and process B was mad runnable untill it
printed exactly one character. This blocked process B, and would wait 
until the buffer cleared. Next process A would be made runnable again,
and be allowed to fill most of the buffer again.....

Yes, we where also proud that this worked!

					Roger
-- 
If the opposite of "pro" is "con", what is the opposite of "progress"? 
	(stolen from  kadokev@iitvax.iit.edu)
EMail:  wolff@duteca.et.tudelft.nl   ** Tel  +31-15-783644 or +31-15-142371


From: ted@telematics.com (Ted Goldblatt)
Newsgroups: comp.os.linux
Subject: Re: Writing an OS - questions !!
Date: 6 May 92 22:46:52 GMT

In article <10685@inews.intel.com> nani@td2cad.intel.com (V. Narayanan) writes:
>
>Hi folks,
>        For quite some time this "novice" has been wondering as to how one goes
>about the task of writing an OS from "scratch".  So here are some questions,
>and I would appreciate if you could take time to answer 'em.
>
>1) How would you typically debug the kernel during the development phase?

I can only speak from the point of view of building a system on non-PC
type hardware.  I don't know how relevent this is to someone who wishes
to build a system on a PC type platform as it normally exists, especially
if it is for hobby type use as opposed to "for real" (i.e., if you don't
have _lots_ of time and _lots_ of money to spend).

The first step was (would be) to build (if not already present) a
"debug monitor".  (Actually, I once worked on a system without this, 
but that was on a machine that had a "real" front panel :-), so it's
too far in the past to think about.)

The debug monitor is actually a collection of software, normally 
resident in a ROM/EPROM.  On the system I currently work on, this
includes automatic hardware health checks, stand-alone I/O support
for the console, disk, and network (the equivalent of a PC's BIOS,
I believe), assorted initialization functions, and (most important
for this discussion) a command interpreter.

The command interpreter's commands allow such things as examining and
modifying memory and registers, planting breakpoints, and loading and
running programs (e.g., off of disk).  Using the breakpoint capability,
breakpoints can be planted in the O/S (which is just another program
to the monitor), and the examine and modify capabilities can be used
when a breakpoint is hit to see what is going on.

Now (you ask), how do you debug the debug monitor?  For that, we use
hardware debugging facilities, principally emulators and logic
analyzers.  An emulator gives you capabilities similar to those of
the debug monitor, but totally external to the system (they don't
depend on any of the system's facilities to work).  It generally
plugs into the microprocessor socket, replacing the micro.  

The tool I use much more often is a logic analyzer, which also plugs
into the micro socket.  However, rather than giving you monitor-like
capabilities, the analyzer can just look at what is happening on the
chip's pins.  It provides you a way to capture the micro's
interactions with the "outside world" (mostly memory).  Its benefits
are that (most) logic analyzers have very elaborate facilities for
conditionally choosing _what_ is captured (triggering and selection)
as well as the ability to watch other things in parallel (bus
activity, I/O chips, etc.) with cross correlation and time stamping.
These gadgets are _very_ useful, especially for watching interactions
of multiprocessors and other concurrent activities.  Even with a good
debug monitor available on our system, I still make heavy use of a
logic analyzer for O/S debugging.  The major downsides are that they
tend to be low-level debugging tools (assembly level, maybe with
symbols, sometimes with limited support for C), and they are expensive
(the one I generally use is a good one, and supports 3 processors and
a 2 channel digital scope simultaneously, but it cost $45K).

>2) Can you test the kernel functionality by running it as a process on a
>   different OS?  Wouldn't the OS(the development environment) generate
>   exceptions in cases when the kernel (of the new OS) tries to modify
>   'priviledged' registers?

This is very dependent on the underlying H/W (it assumes all necessary
requests would be trapped cleanly, and that the machine model captures
sufficient information).  In many cases, for this to work, the
underlying O/S would have to be changed to emulate the required
facilities transparently.  This may or may not be possible.  Also,
timing would be completely different, so this approach would have
limited benefit in finding timing related problems, and might not work
at all if the system in question was real-time or had timing
constraints of its own.  However, this idea _is_ the basis for virtual
machine systems (e.g., VM/370), and (if it can be employed) is very
useful.  Note that true virtual machine operation (as opposed to S/W
emulation) cannot be done on all processors.  The Moto 68010, 68020,
and 68030 do support this, but the 68040 _does not_.  I don't know the
Intel line well enough to tell, but I have heard that the chips can
virtualize lower members of the family but not themselves.

>3) Would new linkers and loaders have to be written before you get a basic
>   kernel running?

Certainly not before.  Whether they were needed before the system was
useful depends in part on what the system's intended facilities are
and what object module and load module formats were chosen.  Linkers
need not be O/S dependent (though they normally are at least
somewhat), and loaders need not be complex (especially in a system
that uses demand paged loading - of course in this case, you just
pushed the work into your virtual memory system :-)).

>4) Was Linux developed in a DOS environment?  Maybe Linus could shed some light
>   on this subject.

Left for Linus :-)





-- 
Ted Goldblatt    ted@telematics.com     (305) 351-4367
    Telematics Intl., Inc.   Ft. Lauderdale, FL



From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Subject: Re: To Linus, Lions book?
Date: 3 Jul 92 11:35:06 GMT

In article <1992Jul3.090424.15207@athena.mit.edu> J.Jagger@sheffield-city-poly.ac.uk writes:
>
>How did you get all your now how to create Linux?
>Was hands on learning, which books/mags/articles/code
>did you read? Go on Linus, give us a potted life history :-)

Heh. I can't let these ego-building posts go by, so I'll have to answer
it.

Linux started for several different reasons: originally just to learn
the 386, and for this the book "Programming the 80386" by Crawford and
Gelsinger is a must.  At least I haven't found anything better (although
admittedly bookstores in Finland probably don't have the plethora of
books available over in the US etc). 

The 386 book info has been used for all the 386 details: the lowest
level of the task-switching code, memory manager, floating point
emulations etc.  It doesn't even mention the AT hardware, so for device
drivers you'll need more info, but it does contain everything on the
386/387 chips (even if it's a bit hard to read: read it through several
times before starting to look into details). 

The device drivers have been written mostly by trial and error: I
haven't found a single good source of info on the standard AT hardware. 
Helpful sources have been (a) the AT Technical Reference Manual (notably
the BIOS listing) (b) actual data-sheets for the 8250 etc chips used in
the AT hardware (c) other sources (mach drivers and to a lesser extent
minix drivers) (d) various books like "IBM Microcomputers: a programmers
manual", "The PC Sourcebook" etc. 

Even though there are a lot of books on the AT hardware, none of them
seem to approach the information content of the 80386 books.  I can't
recommend any of the sources I've used, and the best bet is probably to
have a lot of books and hope they together cover most of it.

Then there is naturally the hardware-independent parts of the kernel:
general filesystem/process/memory management.  The two books I
originally used were "The Design of the Unix Operating System" by Bach,
and "OS Design and Implementation" by Tanenbaum.  Tanenbaum's book
should be read a couple of times: ignore the details (especially about
minix), but get the general picture clear.  The Bach book has a few nice
algorithms in it, and I'd suggest reading them too: many of the actual
ideas on how something could be implemented came from that. 

Still, OS design is mostly common sense: it doesn't need complicated
algorithms like a compiler does or similar.  A lot of thought, careful
programming and some good decisions.  And unix is a very simple OS
really: I first came in contact with it 2 years ago when I did the
"C-kieli ja unix ohjelmointiymp{rist|" course in the fall of -90.  By
early -91 I had bought a PC, and linux v0.01 was ready in August -91. 
That wouldn't have been possible with some other systems I could mention
(VMS..  arghh). 

The most difficult parts are:

 - device drivers: you can go crazy trying to find why something doesn't
work on some machines (especially if you can't even test it out).

 - filesystem: trying to make it clean and well designed, as well as
getting rid of all race conditions.  The current linux vfs layer seems
to work well enough, but my original design wasn't too good.  Rewriting
big parts isn't fun when something works. 

The kernel proper is pretty simple in fact, although you have to keep
track of a lot of things.  The memory manager has also survived pretty
well all the changes, and although I'll have to change it to use
different page directories for different processes some day it will
probably look pretty similar even after that. 

>Also could you mail me a few lines, grouping the source code
>files into chunks. I.e., which files make up the lowest most
>basic layer of the OS 'onion', which ones make up the next layer?
>This would make it lot easirt to peruse the code in an orderly fashion
>so I can try to understand it.

Hmm.  To get a good general idea of the different pieces, I'd suggest
reading linux/*/*.c - all the memory management sources, the normal
kernel sources and the vfs routines.  They are pretty modular: you can
generally understant linux/fs/*.c without having to understand the
kernel and vice versa (for a general picture, that is.  For the details
you usually have to read it all). 

The linux device drivers (linux/kernel/chr_drv and linux/kernel/blk_drv)
are a pain to read: they don't actually give you any more info on the
kernel itself, so reading them is only needed if you really want to know
how a block is loaded into memory etc.  Similarly for the math-emulator
(linux/kernel/math). 

The special filesystems (linux/fs/minix and now linux/fs/ext) can be
read if you are interested in the actual lay-out of the disk, or if you
want to see how some of the fs races are handled.  Note that most
race-conditions are handled directly by the buffer-cache routines or the
vfs layer, but some races are inherent to the actual fs implementation
(renaming a file etc). 

                Linus