9  Controlling a running job

Graceful stop, scheduled snapshots, and gRPC runtime control

A running solver accepts intervention through three channels. All of them end in the same graceful path: the current iteration is finished and the final output and restart file are written before the process exits — nothing is lost.

Channel Intent How
Signals stop now Ctrl+C, or kill <pid> / kill -INT <pid> from another shell
wake.control.json stop or dump at a chosen iteration drop a file into the run directory (below)
gRPC remote low-latency control/monitoring start the run with --grpc; used by the GUI, daemon, and SDK

9.1 Stopping with a signal

Ctrl+C (or kill/kill -TERM/kill -INT with the solver pid) requests a graceful stop; a second signal forces immediate exit. This works even when you forgot --grpc.

For MPI jobs, send the signal to a rank process pid (any rank works — the ranks agree on the stop collectively). Signalling the mpirun process itself also works but Open MPI escalates to a hard kill after a short timeout, so a heavy case may not finish writing its final output in time.

9.2 wake.control.json — scheduled stop and snapshots

For “stop at iteration N” or “give me a snapshot at iteration N without stopping”, write a wake.control.json file into the run output directory (the one that already contains wake.status.json):

// <case>_001/wake.control.json — comments are allowed
{
  "stop_at": 6000,                    // stop gracefully at iteration 6000
  "dump_solution_at": [3000, 4000],   // force a volume/surface output
  "dump_restart_at": 2500             // force a restart (.wsol) snapshot
                                      // (single value or array both work)
}

Rules:

  • Scheduled only. There is intentionally no “stop immediately” key — GPU runs advance too fast for a file drop to mean a well-defined “now”. Use a signal for that.
  • Past iterations are ignored, not fired late. If the run is already at iteration 5200 when it reads "stop_at": 5000, the entry is dropped (and reported), so you never get a surprise stop.
  • The file is consumed (deleted) as soon as it is read, and the solver answers with wake.control.ack.json in the same directory listing what was accepted, what was ignored, and any parse error. Check the ack file to confirm your command landed.
  • Dropping a new file replaces any pending schedule from a previous one.
  • A malformed file never disturbs the run — it is rejected with the reason in the ack file.

A stopped run pairs naturally with run_control.resume: restart from the .wsol it wrote with run_control.resume: true and the iteration counter, convergence-reduction origin, and adaptive-CFL state all continue where they left off.

9.3 gRPC

Starting a run with --grpc <port|auto> opens a low-latency control and monitoring channel used by the GUI, the daemon, and the Python SDK’s run.runtime() client — see Python SDK: Results and runtime control. A standalone protocol reference for direct gRPC integration is planned but not yet published here.