estdlib: add gen:start/5,6

[pguyot]

The Elixir GenServer implementation calls them. The monitor linkage folds proc_lib:start_monitor/3's {Result, MonitorRef} into OTP's {ok, {Pid, Mon}}.

These changes are made under both the "Apache 2.0" and the "GNU Lesser General
Public License 2.1 or later" license terms (dual license).

SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later

[petermm]

PR re-review: cd8cac140 estdlib: add gen:start/5,6

Recommendation: request changes

The contributor fixed the previous blocker: timeout and failure cleanup now handles links from
both start_link and spawn_opt, no path waits for a second DOWN, ignore types are correct,
and both monitor spawn-option forms are rejected. One monitor-ownership mismatch remains:
failed proc_lib:start_monitor/* calls consume the terminal DOWN before returning its monitor
reference, contrary to the documented OTP contract. This should be fixed before merge.

This re-review compared abda2ef22..cd8cac140 with Erlang/OTP's stdlib/src/proc_lib.erl and
gen.erl in /Users/petermm/OSScontrib/otp, and included another independent Oracle review.

Status of previous findings

• Stale link tracking and unsafe timeout cleanup: fixed. Failure paths now unlink
  unconditionally, selectively flush EXIT, and await exactly one DOWN. A caller-supplied
  {spawn_opt, [link]} is handled the same way as start_link.
• Pre-ack start_link/start_monitor hang: fixed. The branch no longer consumes one DOWN
  and waits for another.
• Missing ignore callback and return types: fixed. start_ret/0, start_mon_ret/0, and
  init_result/1 now include ignore (libs/estdlib/src/gen_server.erl:70-81).
• Tuple monitor option validation: fixed. Both monitor and {monitor, _} are rejected
  before spawning (libs/estdlib/src/proc_lib.erl:194-198), with a regression test.

Finding

High — failed proc_lib:start_monitor/* calls lose the returned monitor's DOWN

The Monitor = true failure branches return {Result, MonitorRef} after consuming the only
terminal DOWN:

• init_fail/nack: libs/estdlib/src/proc_lib.erl:213-218
• pre-ack child death: libs/estdlib/src/proc_lib.erl:225-227
• timeout: libs/estdlib/src/proc_lib.erl:231-243

This violates the direct proc_lib:start_monitor/* contract. OTP explicitly documents that if
the function returns an error, a DOWN is still delivered to the caller, including after a
timeout. Its sync_start_monitor/2 re-enqueues the exact consumed DOWN before returning the
reference (/Users/petermm/OSScontrib/otp/lib/stdlib/src/proc_lib.erl:582-598). A direct AtomVM
caller that waits on the returned monitor after a failed start will instead block forever.

The higher-level APIs have the opposite ownership rule: gen:start(..., monitor, ...) and
gen_server:start_monitor(...) hide the reference on failure, so they must consume the
preserved DOWN before returning. OTP does this in gen:monitor_return/1. AtomVM currently gets
a clean mailbox only because proc_lib swallows the message too early.

Preserve DOWN at the proc_lib boundary:

diff --git a/libs/estdlib/src/proc_lib.erl b/libs/estdlib/src/proc_lib.erl
@@
         {nack, Pid, Result} when Monitor ->
             flush_exit(Pid),
-            receive
-                {'DOWN', MonitorRef, process, Pid, _} -> ok
-            end,
+            Down = receive
+                {'DOWN', MonitorRef, process, Pid, _} = D -> D
+            end,
+            self() ! Down,
             {Result, MonitorRef};
@@
-        {'DOWN', MonitorRef, process, Pid, Reason} when Monitor ->
+        {'DOWN', MonitorRef, process, Pid, Reason} = Down when Monitor ->
             flush_exit(Pid),
+            self() ! Down,
             {{error, Reason}, MonitorRef};
@@
-        receive
-            {'DOWN', MonitorRef, process, Pid, _} -> ok
-        end,
+        Down = receive
+            {'DOWN', MonitorRef, process, Pid, _} = D -> D
+        end,
         case Monitor of
             true ->
+                self() ! Down,
                 {{error, timeout}, MonitorRef};

Then consume that DOWN only where the monitor reference is hidden. For gen:

diff --git a/libs/estdlib/src/gen.erl b/libs/estdlib/src/gen.erl
@@
     case proc_lib:start_monitor(GenMod, init_it, InitArgs, Timeout, SpawnOpts) of
         {{ok, Pid}, Mon} -> {ok, {Pid, Mon}};
-        {Error, _Mon} -> Error
+        {Error, Mon} ->
+            receive
+                {'DOWN', Mon, process, _Pid, _Reason} -> Error
+            end
     end.

Apply the same failure receive to both public gen_server:start_monitor overloads. A small local
helper can avoid duplicating the success/error folding:

diff --git a/libs/estdlib/src/gen_server.erl b/libs/estdlib/src/gen_server.erl
@@
 start_monitor(Module, Args, Options) ->
-    {Result, Monitor} = proc_lib:start_monitor(?MODULE, init_it, [self(), Module, Args, Options]),
-    case Result of
-        {ok, Pid} ->
-            {ok, {Pid, Monitor}};
-        _ ->
-            Result
-    end.
+    monitor_start_result(
+        proc_lib:start_monitor(?MODULE, init_it, [self(), Module, Args, Options])
+    ).
+
+monitor_start_result({{ok, Pid}, Monitor}) ->
+    {ok, {Pid, Monitor}};
+monitor_start_result({Result, Monitor}) ->
+    receive
+        {'DOWN', Monitor, process, _Pid, _Reason} -> Result
+    end.

Use monitor_start_result/1 in the named overload after its whereis/1 precheck as well.

Remaining non-blocking compatibility note

If register/2 loses a race and the winner exits before the subsequent whereis/1,
gen_server:init_it/5 returns {error, badarg}. OTP produces
{error, {already_started, undefined}}. This rare error-shape difference has no lifecycle or
resource leak and may reasonably remain an AtomVM subset difference.

Test gaps

• Add direct failed-proc_lib:start_monitor tests for init_fail, pre-ack crash, and timeout.
  Each must assert that the returned reference receives its terminal DOWN.
• Add corresponding gen:start(..., monitor, ...) and public gen_server:start_monitor failure
  tests asserting that those higher-level APIs leave no hidden-monitor DOWN behind.
• test_proc_lib:test_start_crash/0 still covers only unlinked proc_lib:start; add immediate
  crash cases for trapped start_link, start_monitor, and start_monitor(..., [link]) to
  directly guard the former second-DOWN hang.
• The occupied-name test still covers only the initial whereis/1 fast path, not a registration
  race between the precheck and child register/2.
• isolated_start/2 demonitor-flushes as soon as it receives {done, ...}. A delayed fatal exit
  after that send could be concealed. Keep the monitor and require a subsequent normal worker
  DOWN; on the eight-second timeout, kill and reap the worker instead of leaving it alive.
• The shared drain_mailbox/0 remains broad and timing-based. Prefer isolated workers and
  selective matching by known PID/reference for lifecycle assertions.

Verification

• git show --check HEAD — passed.
• cmake --build build --target test_estdlib -j4 — passed (one existing deprecated-catch
  warning).
• ./build/src/AtomVM ./build/tests/libs/estdlib/test_estdlib.avm — passed; every listed module,
  including test_gen_server and test_proc_lib, returned ok. The direct failed-monitor
  contract and pre-ack crash paths described above remain uncovered.