Skip to content

Decode date, timestamp and timestamptz without Python-level arithmetic - #34

Open
ivlevdenis wants to merge 2 commits into
MagicStack:masterfrom
ivlevdenis:c-api-datetime
Open

ivlevdenis wants to merge 2 commits into
MagicStack:masterfrom
ivlevdenis:c-api-datetime

Conversation

@ivlevdenis

Copy link
Copy Markdown

What

date_decode() calls datetime.date.fromordinal(), and both timestamp codecs
build every value as

pg_epoch_datetime[_utc] + timedelta(0, seconds, microseconds)

which allocates an intermediate timedelta and performs a Python-level
datetime.__add__ call for every single value.

This replaces that with the datetime C API: the PostgreSQL day/microsecond
count is converted to a civil date (Hinnant's civil_from_days) and the value
is built with PyDateTimeAPI.Date_FromDate() /
DateTime_FromDateAndTime(), passing the existing timezone.utc object for
timestamptz and NULL tzinfo for timestamp.

Numbers

Fetching 100 000 rows of one column (PostgreSQL 18.6 over a unix socket,
CPython 3.14.4, median of 7 runs; an int8 column fetched alongside as a
control, unchanged):

column before after
timestamptz 57.6 ms 33.9 ms (-41%)
timestamp 56.8 ms 30.4 ms (-46%)
date 21.4 ms 14.4 ms (-32%)

That is ~240 ns removed per timestamptz/timestamp value and ~70 ns per
date value — the codecs no longer allocate a second object nor re-enter the
interpreter.

Behaviour and compatibility

  • Decoded values are unchanged: verified against PostgreSQL's own to_char()
    output for all 100 000 rows, for 98 705 dates spanning
    0001-01-01..9999-12-31, and for 20 000 random timestamps across the whole
    supported range, including leap days (1600, 1900, 2000, 2024-02-29),
    microsecond precision and infinity / -infinity
    (238 724 comparisons, 0 mismatches).
  • Out-of-range values keep raising OverflowError('date value out of range'),
    exactly as the previous timedelta overflow did.
  • Only long-standing datetime C API is used (Date_FromDate since 2.6,
    DateTime_FromDateAndTime since 3.0); import_datetime() is already called
    at module import, so PyDateTimeAPI is initialised. No version guards, no
    new dependencies.
  • Built and tested on CPython 3.9, 3.10, 3.11, 3.12, 3.13, 3.14 and 3.14t
    (free-threaded); asyncpg's test suite passes on all of them.

date_decode() called datetime.date.fromordinal(), and both timestamp codecs
built every value as pg_epoch_datetime[_utc] + timedelta(...), creating an
intermediate timedelta and performing a Python-level datetime.__add__ call for
every value.

Build the value directly through the C API instead: convert the PostgreSQL
day/microsecond count to a civil date with Hinnant's algorithm and call
PyDateTimeAPI.Date_FromDate()/DateTime_FromDateAndTime().  Out-of-range values
keep raising OverflowError, as before.

Measured on 100 000 rows (PostgreSQL 18.6, unix socket, CPython 3.14.4,
median of 7 runs, an int8 column as control, unchanged): fetching a timestamptz
column 57.6 -> 33.9 ms, timestamp 56.8 -> 30.4 ms, date 21.4 -> 14.4 ms.
The hour split used a literal that does not fit in a C int:

    h = <int>(rem // 3600000000)

Cython therefore types the literal as a Python int and generates
PyNumber_FloorDivide plus several PyLong round-trips for every decoded
value, so the time-of-day split still re-entered the interpreter.
Minutes and seconds escaped this only because their literals (60000000,
1000000) happen to fit in int32.

Hoist the per-unit microsecond counts into cdef int64_t constants and use
them throughout, keeping the whole split on the C level.

Fetching 100 000 rows of one column (PostgreSQL 18.6 over a unix socket,
CPython 3.14.4, median of 7 runs, master as the baseline): timestamptz
69.6 -> 18.7 ms, timestamp 71.1 -> 16.6 ms; date is unchanged by this
commit.
@ivlevdenis

Copy link
Copy Markdown
Author

Follow-up: the hour split still used Python-level arithmetic

One more place where the "no Python-level arithmetic" goal isn't fully met.

_datetime_from_us() splits the day remainder like this:

h = <int>(rem // 3600000000)
rem -= <int64_t>h * 3600000000

3600000000 is larger than 2**31 - 1, so Cython types the literal as a Python int and emits PyNumber_FloorDivide plus several PyLong round-trips for every decoded value. Minutes and seconds avoided this only because 60000000 and 1000000 happen to fit in int32.

Fix — hoist the per-unit counts into cdef int64_t constants:

cdef int64_t us_per_hour = 3600000000
cdef int64_t us_per_minute = 60000000
cdef int64_t us_per_second = 1000000

After this, _datetime_from_us() contains no PyNumber_FloorDivide / PyLong conversions.

Numbers (before = master; same environment as above: PostgreSQL 18.6 over a unix socket, CPython 3.14.4, fetching 100 000 rows of one column, median of 7 runs; the master baseline differs slightly from the PR body because this was measured on a different machine):

column master (before) this PR + follow-up fix
timestamp 71.1 ms 33.8 ms 16.6 ms
timestamptz 69.6 ms 36.4 ms 18.7 ms
date 23.3 ms 14.6 ms 13.7 ms

master → fix is ≈ 4.3× (timestamp) / 3.7× (timestamptz); the follow-up alone removes another ~172 ns per timestamp and ~178 ns per timestamptz value on top of this PR. date is unaffected — it doesn't use the time split.

Correctness: unchanged — decoded values match the server's own ::text output (sampled across the table), and 400k+ random pre/post-epoch timestamps and dates verified against datetime, including infinity/-infinity (0 mismatches).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant