Multi Step Processes hard

In a workflow engine (like Airflow or Temporal), why should the 'Scheduler' (Control Plane) never run the actual job code (Data Plane)?

A Isolation. If a user's job crashes (OOM) or hangs, it shouldn't take down the Scheduler that manages thousands of other jobs.
B Security.
C Schedulers are written in Python.
D Data Planes are slower.

One system design question a day, with spaced repetition so the ones you miss come back.

Related questions

A user registration process involves two steps: inserting a row into the 'Users' database table and inserting a row into the 'Profiles' table. If the first insert succeeds but the second fails, the database is left in an inconsistent state. What mechanism prevents this?
In a multi-step workflow across different services (Order Service -> Payment Service), you cannot use a single database transaction. If the Payment fails after the Order was created, how do you fix the Order state?
You are writing a script to process 10,000 images. The script runs on a server that might restart unexpectedly. To ensure you don't have to start over from image #1 after a crash, what should you implement?
A client app sends a 'Pay $50' request to your API. The server processes the payment but the network response is lost. The client automatically retries the request. What prevents the user from being charged $100?
A user uploads a large video file that needs to be transcoded (converted), which takes 5 minutes. Why is it bad practice to keep the HTTP connection open while the server processes the video?
How does CDC (reading the database transaction log) solve the 'Dual Write' problem better than the Application-level Outbox pattern?