Image encoding and map loading hold the Python GIL #300

Formatted Image.to_string() encoding holds the Python GIL throughout mapnik::save_to_string(). Moving encoding to a Python worker thread therefore still blocks other Python threads, including an asyncio event loop.

Reproduced with python-mapnik v4.3.2 (upstream master at 288396110300f5f6892657207607609df82f745dae9) on GIL-enabled CPython. The formatted ImageView.to_string() overloads use the same binding pattern, including the explicit-palette overloads.

Minimal reproducer

import sys
import threading

import mapnik

image = mapnik.Image(2048, 2048)
image.fill(mapnik.Color('red'))
start = threading.Event()
progressed = threading.Event()


def other_thread():
    start.wait()
    progressed.set()


thread = threading.Thread(target=other_thread)
thread.start()
interval = sys.getswitchinterval()
try:
    # Exclude ordinary Python bytecode switching during the short encoding call.
    sys.setswitchinterval(60)
    start.set()
    encoded = image.to_string('png')
    ran_during_encoding = progressed.is_set()
finally:
    sys.setswitchinterval(interval)
    start.set()
    thread.join()

print(ran_during_encoding)

The original bindings print False. With the GIL released around native encoding, this prints True.

Proposed fix

Use a scoped py::gil_scoped_release only around the native save_to_string() calls in the formatted Image/ImageView wrappers. Reacquire the GIL before constructing the returned Python bytes. This preserves the encoding algorithms and parameters; mapnik.render() already releases the GIL during native rendering.

A fix and regression tests for all four formatted overloads are prepared.

Map loading has the same issue

Both load_map() and load_map_from_string() also hold the GIL throughout native map loading. Loading initializes datasources, which may execute database queries, so database waits can also prevent other Python threads from running even when map loading is offloaded to a worker thread.

The regression test uses XML with 2,000 empty layers and the same thread-progress check shown above, parametrized over file and string loading. Both cases fail with the original v4.3.2 bindings because the other Python thread cannot run during loading. No database is needed for this reproducer.

The additional fix in #301 releases the GIL during both native load calls using py::call_guard<py::gil_scoped_release>(). Existing arguments and loading behavior are preserved.

Generated with Codex.