Querying in Subprocess Hangs when CHDB Imported in Parent Process

[Firbydude]

Describe what's wrong

Importing CHDB in a parent process will cause queries in a subprocess to hang indefinitely. Importing in the subprocess works as expected. Importing in both locations will hang.

Does it reproduce on the most recent release?

chdb.__version__='3.1.2'
sys.version='3.12.10 (main, Apr  9 2025, 04:03:51) [Clang 20.1.0 ]'

How to reproduce

#!/usr/bin/env python3
"""
Minimal reproduction of chdb ProcessPoolExecutor issue.
"""
import asyncio
import sys
from concurrent.futures import ProcessPoolExecutor
import chdb  # This import in the parent process causes hanging in subprocesses

def worker_function(worker_id):
    """Worker function that runs in a subprocess"""
    print(f"Worker {worker_id} starting...")
    
    # IMPORTANT: Moving the import here does not hang:
    # import chdb
    
    try:
        # This will hang because chdb was imported in the parent process
        result = chdb.query("SELECT sleep(2), count() FROM numbers(10)", "DataFrame")
        print(f"Worker {worker_id} completed query successfully")
        return True
    except Exception as e:
        print(f"Worker {worker_id} error: {str(e)}")
        return False

async def run_test():
    """Run test with problematic global chdb import (will hang)"""
    print("Testing with global chdb import (this will hang)")
    with ProcessPoolExecutor(max_workers=3) as executor:
        loop = asyncio.get_event_loop()
        tasks = [
            loop.run_in_executor(executor, worker_function, i)
            for i in range(3)
        ]
        
        try:
            # This will hang indefinitely
            results = await asyncio.gather(*tasks)
            print(f"Results (you shouldn't see this): {results}")
        except KeyboardInterrupt:
            print("Test interrupted - expected behavior if hanging")

if __name__ == "__main__":
    print("Starting chdb ProcessPoolExecutor contention test")
    print("This test demonstrates that importing chdb globally causes hanging")
    print("Press Ctrl+C after ~10 seconds when processes appear stuck")
    print(f"{chdb.__version__=}")
    print(f"{sys.version=}")
    
    try:
        asyncio.run(run_test())
    except KeyboardInterrupt:
        print("\nTest interrupted by user")
        print("To fix: Move the chdb import inside the worker function")

$ uv run python repro.py 
Starting chdb ProcessPoolExecutor contention test
This test demonstrates that importing chdb globally causes hanging
Press Ctrl+C after ~10 seconds when processes appear stuck
chdb.__version__='3.1.2'
sys.version='3.12.10 (main, Apr  9 2025, 04:03:51) [Clang 20.1.0 ]'
Testing with global chdb import (this will hang)
Worker 0 starting...
Worker 1 starting...
Worker 2 starting...
^C^C
Test interrupted by user
To fix: Move the chdb import inside the worker function

Expected behavior

CHDB should work in subprocesses regardless of import location.

[sirfz]

Passing mp_context=multiprocessing.get_context('spawn') in the executor init should solve your problem.

[Firbydude]

@sirfz is correct. This only occurs when subprocesses are forked.