Allow depending on tasks from an earlier submit in the jobfile

[Kobzol]

We should be checking the validity of the tasks on the server too, as doing it on the client can cause race conditions, in theory. But even if that happens, the server will reject the submit anyway, I suppose.

[spirali]

We should be checking the validity of the tasks on the server too, as doing it on the client can cause race conditions, in theory. But even if that happens, the server will reject the submit anyway, I suppose.

Yes, Tako checks this, so in the case race condition. the second submit gets an error.

I will do a proper code review tomorrow.

[spirali]

I think that there is no need to construct "deps_from_previous_submit", you just need to check that tasks already exists (ID exits) and then forget it. You just need to modify the check for emptiness of deps. When all deps depends only on previously existing tasks, then you can put it immediately into new_tasks. Then there is no need to process anything like "deps_from_previous_submit" with handle_consumers.

(This is written from a hospital bed, so I may be wrong:)

[Kobzol]

Good point, it's a nice simplification. It needed a few more changes, but it's better this way, I think.

[Kobzol]

Note: this could be

            .iter()
            .filter(|&task| !existing_tasks.contains(task))
            .copied()
            .collect();

But I think that pre-cloning might be better to avoid partial reallocations of the Vec, since the usual case is that all/most deps will be from the same submit.

[spirali]

Maybe we can avoid creating task_deps_from_this_submit at all. In the first run just to count the elements and then do the filtering again. But without benchmark it is just guessing. So LGTM for me.