Set error code on failure via implicit top-level task

[elliottslaughter]

Rolled this back temporarily because it causes multiple top-level tasks to be launched in the multi-node case.

[elliottslaughter]

I realized that this approach isn't going to work, because it introduces control divergence (potentially) into the top-level task. If the data goes bad, this lifts a data corruption issue into a control corruption issue. Ironically, the current abort either freezes or kills the process right away so we don't need to deal with the downstream consequences of a potential control divergence.

For now it seems safer to comment out the abort on a case by case basis when we need clean exits; the alternative is to put validation inside of a task, but then that risks altering the data flow of the program.

[elliottslaughter]

I thought about this some more and realized the control divergence can be avoided by carefully coding to avoid branches in the error path. Check everything, then set a return value at the end. Never skip a check.

Assuming it passes, it would make it a lot easier to track down bad region output cases like StanfordLegion/legion#1946.

[elliottslaughter]

There is still the issue of shards agreeing on a return value. I think this is what Runtime::consensus_match is for, but I don't yet understand the API well enough to use it.

[elliottslaughter]

I added the consensus match algorithm, but now I see that we cannot run non-control-replicated top-level tasks with the new implicit task option. If you pass replicable, it will be replicated. Full stop. I need to figure out how to expose this to the mapper or else control it in our invocation to begin_implicit_task.

[lightsighter]

I added the consensus match algorithm, but now I see that we cannot run non-control-replicated top-level tasks with the new implicit task option. If you pass replicable, it will be replicated. Full stop. I need to figure out how to expose this to the mapper or else control it in our invocation to begin_implicit_task.

Just to be clear, you want to run a non-control-replicated implicit top-level task from one node? That should already be possible today: you just have to pick which node is going to do the begin_implicit_task call (only one should to produce one top-level task). Just as an example, you can configure the machine with the Realm API and then consult the machine model to see which node is node 0 and then have that one do the begin_implicit_call while everyone else calls Runtime::shutdown right away.

[elliottslaughter]

That's one half, but then I need to broadcast the error code out of that one task so that all of the ranks can return it from their respective mains.

[lightsighter]

That's one half, but then I need to broadcast the error code out of that one task so that all of the ranks can return it from their respective mains.

Yeah, we're in the area outside of Legion here. I'm not sure we want to be using Legion to help with stuff nominally outside of its programming model. It's up to the "user" to figure out how to decide which nodes do the launching of the top-level task and then how to broadcast any results that come out of it. Having a Realm collective library might be one option.

[elliottslaughter]

I understand, but given how much work we're talking about and how half-baked the support is currently, I'd prefer to go back to Runtime::set_return_code and use normal top-level tasks for this.

[lightsighter]

I understand, but given how much work we're talking about and how half-baked the support is currently, I'd prefer to go back to Runtime::set_return_code and use normal top-level tasks for this.

I'm confused then. What is stopping you from doing that right now?

[elliottslaughter]

Nothing is stopping me, and in fact I have a PR at #7 which does exactly that. The reason it took me so long to try this is because I was originally discouraged from using Runtime::set_return_code and until I actually tried the implicit top-level task route, it wasn't obvious why it wouldn't work.

[lightsighter]

Catching up on this...

The reason it took me so long to try this is because I was originally discouraged from using Runtime::set_return_code and until I actually tried the implicit top-level task route, it wasn't obvious why it wouldn't work.

I was thinking about this some more. While I think it's ok to use set_return_code in this way here, in general I think I probably will still discourage it being used this way. Most people use Legion as a one-to-one execution of a top-level task by a particular instance of the Legion runtime, but the runtime is actually more flexible than that and supports an arbitrary number of top-level tasks executing at the same time. They're all effectively isolated from each other (like processes) even though they are executing on the same Realm (you can almost think of Legion in this way as a kind of cluster-scale OS that doesn't require strict resource partitioning the way that slurm does). Setting an error code then is not something that should really apply to any one top-level task, it's more a property of the runtime. I think my general recommendation for users would be to return error codes along the with the return value of their top-level task that gets set in the external future that Legion returns from each top-level task launch. That said, obviously most people do just launch one top-level task and therefore you can set the return code if you're willing to couple these two concepts.

[elliottslaughter]

Just to be clear (since I also had to re-read this issue to remind myself what's going on):

The method used to set a return code is mostly a distraction, because you can use consensus match to coordinate the return code in the implicit top-level task case. However, the two key issues with implicit top-level tasks are:

1. When you launch implicit top-level tasks, you lose the ability to control them from the mapper. This is obvious in retrospect because the user pre-selects the thread(s) to run them on and obviously Legion can't do anything about it after the fact. But the consequence is that you no longer have a single switch that controls control replication and task placement, which increases the complexity of the implementation.
2. Doing basically anything outside of Legion is really painful due to the lack of a coherent programming model in the primordial (pre-Legion) environment. This means that, for example, coordinating the execution and return value for a non-control replicated explicit top-level task is a prohibitive level of complexity for typical users.

It's a nice idea in theory to say that you can launch top-level tasks into Legion and they'll execute independently and then return, but the practical reality isn't really there unless the level of coordination that you require outside of Legion is essentially nil (i.e., you never actually need to read the result of the top-level task and you assume a fixed mapping strategy for the top-level task assigned up front).

[lightsighter]

When you launch implicit top-level tasks, you lose the ability to control them from the mapper. This is obvious in retrospect because the user pre-selects the thread(s) to run them on and obviously Legion can't do anything about it after the fact. But the consequence is that you no longer have a single switch that controls control replication and task placement, which increases the complexity of the implementation.

Right, when you're creating an implicit top-level task you're effectively making those decisions yourself ahead of time so you've got to coordinate it all yourself.

Doing basically anything outside of Legion is really painful due to the lack of a coherent programming model in the primordial (pre-Legion) environment.

I wouldn't say it's completely incoherent. Nominally you're likely in a bulk-synchronous execution mode so you can use GASNet or MPI to do pretty standard coordination if you want.