Repository navigation
Decode date, timestamp and timestamptz without Python-level arithmetic - #34
ivlevdenis wants to merge 2 commits into
Conversation
date_decode() called datetime.date.fromordinal(), and both timestamp codecs built every value as pg_epoch_datetime[_utc] + timedelta(...), creating an intermediate timedelta and performing a Python-level datetime.__add__ call for every value. Build the value directly through the C API instead: convert the PostgreSQL day/microsecond count to a civil date with Hinnant's algorithm and call PyDateTimeAPI.Date_FromDate()/DateTime_FromDateAndTime(). Out-of-range values keep raising OverflowError, as before. Measured on 100 000 rows (PostgreSQL 18.6, unix socket, CPython 3.14.4, median of 7 runs, an int8 column as control, unchanged): fetching a timestamptz column 57.6 -> 33.9 ms, timestamp 56.8 -> 30.4 ms, date 21.4 -> 14.4 ms.
The hour split used a literal that does not fit in a C int:
h = <int>(rem // 3600000000)
Cython therefore types the literal as a Python int and generates
PyNumber_FloorDivide plus several PyLong round-trips for every decoded
value, so the time-of-day split still re-entered the interpreter.
Minutes and seconds escaped this only because their literals (60000000,
1000000) happen to fit in int32.
Hoist the per-unit microsecond counts into cdef int64_t constants and use
them throughout, keeping the whole split on the C level.
Fetching 100 000 rows of one column (PostgreSQL 18.6 over a unix socket,
CPython 3.14.4, median of 7 runs, master as the baseline): timestamptz
69.6 -> 18.7 ms, timestamp 71.1 -> 16.6 ms; date is unchanged by this
commit.
|
Follow-up: the hour split still used Python-level arithmetic One more place where the "no Python-level arithmetic" goal isn't fully met.
h = <int>(rem // 3600000000)
rem -= <int64_t>h * 3600000000
Fix — hoist the per-unit counts into cdef int64_t us_per_hour = 3600000000
cdef int64_t us_per_minute = 60000000
cdef int64_t us_per_second = 1000000After this, Numbers (before =
master → fix is ≈ 4.3× ( Correctness: unchanged — decoded values match the server's own |
What
date_decode()callsdatetime.date.fromordinal(), and both timestamp codecsbuild every value as
which allocates an intermediate
timedeltaand performs a Python-leveldatetime.__add__call for every single value.This replaces that with the datetime C API: the PostgreSQL day/microsecond
count is converted to a civil date (Hinnant's
civil_from_days) and the valueis built with
PyDateTimeAPI.Date_FromDate()/DateTime_FromDateAndTime(), passing the existingtimezone.utcobject fortimestamptzandNULLtzinfo fortimestamp.Numbers
Fetching 100 000 rows of one column (PostgreSQL 18.6 over a unix socket,
CPython 3.14.4, median of 7 runs; an
int8column fetched alongside as acontrol, unchanged):
timestamptztimestampdateThat is ~240 ns removed per
timestamptz/timestampvalue and ~70 ns perdatevalue — the codecs no longer allocate a second object nor re-enter theinterpreter.
Behaviour and compatibility
to_char()output for all 100 000 rows, for 98 705 dates spanning
0001-01-01..9999-12-31, and for 20 000 random timestamps across the whole
supported range, including leap days (1600, 1900, 2000, 2024-02-29),
microsecond precision and
infinity/-infinity(238 724 comparisons, 0 mismatches).
OverflowError('date value out of range'),exactly as the previous
timedeltaoverflow did.Date_FromDatesince 2.6,DateTime_FromDateAndTimesince 3.0);import_datetime()is already calledat module import, so
PyDateTimeAPIis initialised. No version guards, nonew dependencies.
(free-threaded); asyncpg's test suite passes on all of them.