How Can I Launch A Blocking Task From Asyncio Asynchronously?

I am new to asyncio and am attempting to make a simple webserver that, upon receiving a request, calculates a chess move and returns the result as a response. The problem is, the p

Solution 1:

Overall your code looks fine. The issue you are running into is Pythons limited threading support.

Try swapping the ThreadPoolExecutor for a ProcessPoolExecutor (from concurrent.futures import ProcessPoolExecutor) this will run your engine task in a separate process allowing the task to actually run in parallel.

To expand on the threading issue, the Python interpreter has a lock (known as the GIL) that only allows Python code to be executed on one thread at a time. This greatly simplifies the majority of development tasks. The lock is released under certain conditions, calls to the Kernel to perform IO (either reading from a file or network) release the GIL, making IO a good use-case for threading. The ProcessPoolExecutor executes your worker task in a separate process with its own separate interpreter allowing your code to run in parallel, this approach comes with some overhead but in the context of modern computers, this is fairly minimal. The ProcessPoolExecutor will automatically handle the process of moving data between each process for you.