Spurious Imports

[gray]

In the process of debugging an inexplicable error during the system start-up sequence I started to look into 'rsc' files (yes, I know), namely the list of imported modules. I have realised that the list of imports in the 'rsc' files sometimes contains additional modules that are not in the IMPORT list of the source code.

Here's one case:

Code: Select all

MODULE Coroutines;
  IMPORT SYSTEM, StackMonitor, Calltrace;
  (*..*)
END Coroutines;

The top of the Coroutines.rsc file:

Code: Select all

00000000   43 6F 72 6F 75 74 69 6E 65 73 00 9D C9 F6 8B 01  Coroutines.?Éö?.
00000010   70 02 00 00 53 74 61 63 6B 4D 6F 6E 69 74 6F 72  p...StackMonitor
00000020   00 B0 71 76 B3 43 61 6C 6C 74 72 61 63 65 00 23  .°qv³Calltrace.#
00000030   51 86 32 54 65 78 74 73 00 EE 01 CE F1 00 14 00  Q?2Texts.î.Îñ...
00000040   00 00 40 00 00 00 FF FF FF FF FF FF FF FF FF FF  ..@.............

Modules.Load will import module Texts when loading Coroutines. To debug the aforementioned error, I have modified Modules.Load to print the recursive loading process to the Astrobe terminal. For Coroutines this looks as follows (with some comments added here):

Code: Select all

ld  Coroutines
imp Coroutines StackMonitor   (* first import for Coroutines *)
ld  StackMonitor   (* a ld following an imp means not loaded yet *)
rd  StackMonitor   (* no IMPORTs, read and init StackMonitor , then back to Coroutines *) 
imp Coroutines Calltrace (* second IMPORT for Coroutines *)
ld  Calltrace    (* start loading *)
imp Calltrace Modules   (* already loaded, no further action *)
imp Calltrace Texts   (* correct import, see below *)
ld  Texts
rd  Texts
rd  Calltrace   (* read and init Calltrace, then back to Coroutines *)
imp Coroutines Texts   (* that's the spurious one *)
rd  Coroutines (* read and init Coroutines *)

where:
'ld' = starting to load a module (recursive Module.Load)
'imp' = importing a module (iteration within Module.Load), first module printed is the importer, the second the imported
'rd' = reading and initialising the module after all imports have been processed

Hence 'ld' and 'rd' appear as pairs, bracketing one module load, with 'imp', 'ld', and 'rd' in-between recursively for the module's imports, if any.

Module Calltrace has this IMPORT list:

Code: Select all

MODULE Calltrace;
  IMPORT SYSTEM, Modules, Texts;
  (* .. *)
END Calltrace;

Modules.Load imports and reads module Texts when loading Calltrace, and then again imports it for Coroutines, finding it already loaded and initialised.

Where does this "hidden" import come from?

PS: FWIW, my start-up problem turned out to be caused by on older version of a module that created a circular import and thus an infinite loading loop. The compiler had correctly flagged the cyclic reference, which I corrected, and tested outside the system start-up sequence. I don't know how this faulty version had survived on the SD card. I am usually diligent with building the whole system and uploading the changed modules. But when working with system files, I do a lot of backups, and restores on error. So I might have missed an upload after restoring the system from backup. For now I am confident that this was a layer 8 error. On the positive side, the extended Module.Load turned out to be a useful tool for debugging the system load process.

Here's what happened, and the error was easy to spot:

Code: Select all

ld  Oberon 
imp Oberon Modules
imp Oberon Files
imp Oberon Kernel
imp Oberon Processes
ld  Processes 
imp Processes Kernel
imp Processes Coroutines
ld  Coroutines 
imp Coroutines StackMonitor
ld  StackMonitor 
rd  StackMonitor 
imp Coroutines Calltrace
ld  Calltrace 
imp Calltrace Modules
imp Calltrace Texts
ld  Texts 
rd  Texts 
imp Calltrace Console (* but Console is not on Calltrace's IMPORT list! *)
ld  Console 
imp Console Texts
imp Console RS232dev
ld  RS232dev 
imp RS232dev Texts
rd  RS232dev 
imp Console RS232b
ld  RS232b 
imp RS232b Texts
imp RS232b RS232dev
rd  RS232b 
imp Console RS232sig
ld  RS232sig 
imp RS232sig Texts
imp RS232sig RS232dev
imp RS232sig Processes (* RS232sig does IMPORT Processes *)
ld  Processes  (* loading of Processes starts again, before 'rd Processes' => loop *)
imp Processes Kernel
imp Processes Coroutines
ld  Coroutines 
...

Console had been on Calltrace's IMPORT list before correcting the circular reference problem.

[cfbsoftware]

Search for the word ''índirect'' in the Project Oberon book and you will find references to indirect module imports which should help in understanding when they occur and why they are needed.

[gray]

OK, thanks, I have read up.

Does the Astrobe compiler more or less work like the Project Oberon one in that area? In particular the import and export of symbols (reading and writing of symbol files, ORB. Import and ORB.Export) and creating object files (ORG.Close)?

The way the symbol table is built with respect to modules, I think, with my yet limited understanding, means that the indirect imports count regarding the max. 15 allowed imported modules. Correct?:

[cfbsoftware]

The v8.0 Astrobe compiler includes some modifications to the way symbol files are handled to as mentioned in the What's New notes:

The items in an IMPORT list may now be included in any order. The sequence is not affected by dependencies or indirect imports.

I'm on leave right now - I'll check sometime next week when I return to the office if and how this has affected indirect imports. ORG.Close should not have been directly affected.

My understanding is that indirect imports do count to the 15 allowed limit. Andreas Pirklbauer has documented and published various modifications to the way symbol files are handled (including increasing the limit). You can find his work here:

https://github.com/andreaspirklbauer?tab=repositories

[cfbsoftware]

I have checked and as far as I can see the Astrobe compiler more or less works like the Project Oberon compiler as far as indirect imports go. As an example, if module M1 imports Texts and module M1 exports an item that references Texts then a module that imports M1 will indirectly import Texts as well.

You can see this in Project Oberon: Module Oberon exports the variable Par which is declared as:

Code: Select all

    Par*: RECORD
      vwr*: Viewers.Viewer;
      frame*: Display.Frame;
      text*: Texts.Text;
      pos*: LONGINT
    END;

Hence, any Project Oberon module that imports Oberon also indirectly imports Viewers, Display and Texts.

[gray]

Thanks! The indirectly imported modules do count as regards the max allowed 15 imported modules, which we can check in the BL opcode of an external procedure call, where the module number is encoded in a 4 bit value (hence the restriction of 15, zero is not a valid module number for BL instructions).

If we take the example of module Oberon, which in EO exports Par like so:

Code: Select all

MODULE Oberon;
  (* ... *)
  TYPE
    ParDesc* = RECORD
      text*: Texts.Text;
      pos*: LONGINT
    END;
I
  VAR Par*: ParDesc;
  (* ... *)
END Oberon.

That is, Texts will be an indirect import.

Test modules M1 and M2:

Code: Select all

MODULE M1;
  IMPORT Oberon, M2;
  PROCEDURE Run*;
  BEGIN
    M2.Do
  END Run;
END M1;

MODULE M2;
  PROCEDURE Do*;
  END Do;
END M2.

The call 'M2.Do' in M1 gets encoded as:

Code: Select all

F7301002H

where:
F7 = BL unconditional
3 = module number
01 = procedure number
002 = fixup value

That is, Oberon is module number 1, the indirect import of Texts via Oberon is number 2, and M2 is number 3.

If module M1 has 16 imports, direct and indirect, with M2 the last in the list, the module still compiles, but the module number rolls over to zero, ie. the opcode for 'M2.Do' in M1 becomes

Code: Select all

F7001002H

And so on for more imports.

Modules.Load will now fail:

1) With more than 16 imports, this assignment will result in a trap 1:

Code: Select all

import[nofimps] := impmod

2) With exactly 16 imports, the fixups will fail. For example, the BL fixups, since 'mno' for module 16 will be zero:

Code: Select all

WHILE adr # mod.code DO
  SYSTEM.GET(adr, inst);
  mno := inst DIV 100000H MOD 10H;
  (* ... *)
  SYSTEM.GET(mod.imp + (mno-1)*4, impmod); (* !!! *)
  SYSTEM.GET(impmod.ent + pno*4, dest); dest := dest + impmod.code;
  (* ... *)
END ;

As far as I understand this, both cases will leave the system in an inconsistent state, as the ref count increments will not be reversed. With exactly 16 imports, also the memory will be corrupted, since the allocated slot is not freed.

I have experimentally extended and tested Modules.Load as follows, using error code 6:

Code: Select all

  (* ... *)
  IF F # NIL THEN
    Files.Set(R, F, 0); Files.ReadString(R, name1); Files.ReadInt(R, key); Files.Read(R, ch);
    Files.ReadInt(R, size); importing := name1;
    IF ch = versionkey THEN
      Files.ReadString(R, impname);   (*imports*)
      WHILE (impname[0] # 0X) & (res = 0) DO
        IF nofimps < 15 THEN                  (* here *)
          Files.ReadInt(R, impkey);
          Load(impname, impmod); import[nofimps] := impmod; importing := name1;
          IF res = 0 THEN
            IF impmod.key = impkey THEN INC(impmod.refcnt); INC(nofimps)
            ELSE error(3, impname); imported := impname
            END
          END ;
          Files.ReadString(R, impname)
        ELSE                                 (* and here *)
          error(6, impname)
        END
      END
    ELSE error(2, name1)
    END
  ELSE error(1, name)
  END ;
  (* ... *)

With too many imports, no module memory gets allocated, and the ref counts will be corrected, so the system is in a consistent state. Unless I miss something here, which is always a possibility -- see my original question.

For a meaningful error message, Oberon.WriteError needs to be extended accordingly (in EO).

I now get:

Code: Select all

Command: M1.Run
Importing: M2
Error 6: too many imports