Debugging binutils under my OS?

[mariuszp]

I recently ported binutils to my OS. I've already been using cross-compiling binutils running under Linux and targetting Glidix (x86_64-glidix), and they were working fine. So recently I used my cross-compiler to build a native binutils. Configuration line:

Code: Select all

../glidix-binutils/configure --prefix=/usr --exec-prefix=/usr --host=x86_64-glidix --target=x86_64-glidix --disable-werror --disable-nls

I installed it under a destination directory (DESTDIR=/glidix-mipdir make install) then used that directory as the root of an installable package that my OS recognises. There were some symbolic links in the directory, and the package creator copied the files instead of creating links (I'm not sure how to handle symlinks in an archive creator). The symlinks were just /usr/x86_64-glidix/bin stuff, I don't think that is needed for the simple test I attempted.

Anyway, I installed the binaries under my OS, and I attempted to assemble this simple program which basically does:

Code: Select all

write(1, "hello world\n", 12);
exit(0);

In assembly:

Code: Select all

	.text
	.globl _start
_start:
	mov $1,			%rdi
	mov $hello_world,	%rsi
	mov $12,		%rdx
	ud2
	.hword 1
	
	mov $0,			%rdi
	ud2
	.hword 0

hello_world:
	.ascii "hello world\n"

I tried assembling under Glidix with:

Code: Select all

as -c hello.s -o hello.o

At first it saw an instruction "text" instead of ".text" for some reason (this could be due to the fact there was no tab at the beginning of the line? I know it shouldn't happen). I then changed the configuration to the one specified above. Now it assembles without errors. But, when I look at the "hello.o" file, it is far too small (about 200 bytes smaller than the one produced by the cross-assembler under Linux) and horribly corrupt.

OK, so I attempt to skip this step to see what else is wrong. I assemble under Linux (using x86_64-glidix-as) and copy the hello.o into Glidix. Then I try linking it:

Code: Select all

ld hello.o -o hello

Again, "hello" is too small and horribly corrupt! By horribly corrupt I mean:

Code: Select all

Elf file type is EXEC (Executable file)
Entry point 0x400078
There are 1 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  <unknown>: 9cc 0x010b0f0000000cc2 0x00000000c7c74800 0x6c6c656800000b0f
                 0x0a646c726f77206f 0x0000000000000000         0

 Section to Segment mapping:
  Segment Sections...
   00     

But if I produce "hello" under Linux, it runs perfectly under Glidix and prints "hello world".

The problem appears to be similar in both cases: smaller and corrupt files. So I decided to trace all fopen(), fwrite(), fseek(), fclose() calls, and here's what I got (from running ld):

Code: Select all

{fopen 'hello', mode 'w+'}
{fopen '/media/cdrom/memes/hello.o', mode 'r'}
{fseek 168:0 in 6}
{fseek 616:0 in 6}
{fseek 232:0 in 6}
{fseek 112:0 in 6}
{fseek 800:0 in 6}
{fseek 848:0 in 6}
{fseek 704:0 in 6}
{fseek 64:0 in 5}
{fwrite 0+56 -> 5}
{fseek 680:0 in 6}
{fseek 64:0 in 6}
{fwrite 56+48 -> 5}
{fseek 528:0 in 5}
{fwrite 104+216 -> 5}
{fwrite 320+1 -> 5}
{fwrite 321+27 -> 5}
{fwrite 348+12 -> 5}
{fwrite 360+7 -> 5}
{fwrite 367+12 -> 5}
{fwrite 379+7 -> 5}
{fwrite 386+5 -> 5}
{fseek 168:0 in 5}
{fwrite 168+1 -> 5}
{fwrite 169+8 -> 5}
{fwrite 177+8 -> 5}
{fwrite 185+10 -> 5}
{fwrite 195+6 -> 5}
{fseek 0:0 in 5}
{fwrite 0+64 -> 5}
{fseek 208:0 in 5}
{fwrite 208+320 -> 5}
{fclose}
{fclose}

So indeed with this series of calls, we expect the size to be 528 bytes, which is what it is - but it is this series of calls that produces the corrupt binary! When linking under Linux, the binary is 796 bytes and not corrupt.

I'm not sure exactly what could be the problem and how to debug it. Any clues?

[gerryg400]

mariuszp, this is a very difficult problem. I had many similar problems with binutils and gcc. Both were producing output without showing any error but the object files were rubbish. sortie guessed that my problems were due to libc bugs. Small errors in my stdio implementation mainly. I do remember that ungetc was one problem. Which libc are you using ?

[mariuszp]

mariuszp, this is a very difficult problem. I had many similar problems with binutils and gcc. Both were producing output without showing any error but the object files were rubbish. sortie guessed that my problems were due to libc bugs. Small errors in my stdio implementation mainly. I do remember that ungetc was one problem. Which libc are you using ?

Custom libc.

The stdio implementation is quite simple:
https://github.com/madd-games/glidix/tr ... /src/stdio

Perhaps a second pair of eyes will notice something. Sorry for the mess.
(Now that I think of it... should fread() return the character placed by ungetc() ???)

[gerryg400]

I'm not at my desk and can't remember exactly what happens with fread. After a lecture from sortie I went away and wrote unit tests for my stdio that covered all the weird corner cases in the spec. Eventually it all started to work. Getting a conforming stdio is not a small task. Mine is not there yet. I also had some subtle bugs in my scanf driver that broke things. I guess I'd look at ungetc and scanf first.

[edit] I just checked and indeed all the read functions (including fread) must use ungetc chars.

[mariuszp]

I'm not at my desk and can't remember exactly what happens with fread. After a lecture from sortie I went away and wrote unit tests for my stdio that covered all the weird corner cases in the spec. Eventually it all started to work. Getting a conforming stdio is not a small task. Mine is not there yet. I also had some subtle bugs in my scanf driver that broke things. I guess I'd look at ungetc and scanf first.

Hmm.. actually, my formatting functions are terrible. I remember scanf() failing to parse an IPv4 address ("%d.%d.%d.%d"), and printf() does not support qualifiers (it only understands single characers after '%', e.g. "%s", "%d", etc). I kept postponing proper implementation because it worked for the stuff I was working on.

I guess it is now the time to implement it properly and rigirously.

[gerryg400]

Yep, it's time. It's actually a task I enjoyed. It did take a considerable time though. Perhaps a 100 hours or more to tidy up my 'working' stdio. Make certain to write unit tests or you will keep breaking stuff since lot's of the stuff is intertwined and messy.

[mariuszp]

Yep, it's time. It's actually a task I enjoyed. It did take a considerable time though. Perhaps a 100 hours or more to tidy up my 'working' stdio. Make certain to write unit tests or you will keep breaking stuff since lot's of the stuff is intertwined and messy.

How do you handle buffering?

I mean currently my fwrite() writes data to a buffer in RAM which is then forwarded to a write() call when calling fflush() or fclose() (but since that doesn't work when for example someone forgets to call fclose(), I had to add an fflush() call at the end of fwrite() which is just stupid).

[gerryg400]

It's the programs responsibility to cal fflush or fclose. If it's not called bad luck.

[mariuszp]

It's the programs responsibility to cal fflush or fclose. If it's not called bad luck.

So the C standard allows me not to care about lost data if fflush() or fclose() are not called?

[gerryg400]

Actually there are other ways to flush the data including reading, seeking etc. but generally yes.

[Roman]

AFAIK, on a "normal" exit (the exit function) all streams must be flushed implicitly by the implementation, they may not be flushed on an "abnormal" exit (abort, _Exit, quick_exit).

[gerryg400]

Roman, I think you're right.

[mariuszp]

I guess I could just map fwrite() etc directly to write() on a file descriptor (in this case close() is implicitly called on all file descriptors when the process exits). Is there any reason this should not happen?

[mariuszp]

Can I just ask, does anyone have rigorous unit tests they've used for their library, that I could also use?

I mean if I write unit tests myself, I'll end up just using the same assumptions I had in the first place that caused me to write buggy code. Googling "stdio unit tests" or "c library unit tests" yields nothing useful.

[gerryg400]

Hi, please see the attached file. I wouldn't describe these as rigorous. Most of these failed when I first tried but they all pass now