Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,16 @@ If you need to process dates far in the future with accurate timezone handling,
- Using timezone-naive types (`timestamp_ntz`) when timezone conversion is not required
- Falling back to Spark for these specific operations

### Timezone Database Versions

Comet's native code converts between instants and local time with the IANA timezone database that
chrono-tz compiles into the Comet library. Spark uses the JVM's timezone database (`tzdb.dat`), whose
version depends on the JDK build and on whether its timezone data has been updated. When the two versions
have different rules for a timezone, local times that Comet computes natively for that timezone can differ
from Spark's for the affected dates. That covers `hour`, casts between timestamps and strings or dates,
`date_trunc`, and parsing strings as timestamps. Comet logs a warning at startup when the two versions

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the date_trunc bullet at the top of this page says non-UTC sessions go through the JVM codegen dispatcher by default. So I'd expect native date_trunc to be affected only with spark.comet.expression.TruncTimestamp.allowIncompatible=true. Also, "parsing strings as timestamps" looks covered by the casts already. Would it make sense to tighten this list?

differ.

<!--BEGIN:EXPR_COMPAT[datetime]-->

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions native/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions native/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ datafusion-comet-jni-bridge = { path = "jni-bridge" }
datafusion-comet-proto = { path = "proto" }
datafusion-comet-shuffle = { path = "shuffle" }
chrono = { version = "0.4", default-features = false, features = ["clock"] }
# Already in the tree through arrow's `chrono-tz` feature. Comet uses it directly only to report the
# IANA database version it bundles, which can differ from the JVM's (see NativeBase.getTzdataVersion).
chrono-tz = "0.10"
# Pinned exactly rather than left to the default caret range. `hll_sketch.rs` carries a
# workaround for a compact-decoding bug in 0.3.0 and a guard against a mis-read aux layout,
# both written against that release's `Array4`/`Array6`/`Array8` code. A silent bump to a
Expand Down
1 change: 1 addition & 0 deletions native/core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ publish = false
[dependencies]
arrow = { workspace = true }
base64 = "0.23.0"
chrono-tz = { workspace = true }
bytes = { workspace = true }
parquet = { workspace = true, default-features = false, features = ["experimental", "arrow", "arrow_canonical_extension_types", "snap", "lz4", "zstd", "flate2-zlib-rs"] }
futures = { workspace = true }
Expand Down
13 changes: 13 additions & 0 deletions native/core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,19 @@ pub extern "system" fn Java_org_apache_comet_NativeBase_isObjectStoreSchemeSuppo
})
}

/// JNI: the version of the IANA timezone database native code uses, such as `2025b`. chrono-tz
/// compiles the database into libcomet, so it can differ from the JVM's `tzdb.dat`, and then local
/// times computed natively can differ from Spark's.
#[no_mangle]
pub extern "system" fn Java_org_apache_comet_NativeBase_getTzdataVersion(
env: EnvUnowned,
_: JClass,
) -> jni::sys::jstring {
try_unwrap_or_throw(&env, |env| {
Ok(env.new_string(chrono_tz::IANA_TZDB_VERSION)?.into_raw())
})
}

// Creates a default log4rs config, which logs to console with log level.
fn default_logger_config(log_level: &str) -> CometResult<Config> {
let console_append = ConsoleAppender::builder()
Expand Down
38 changes: 38 additions & 0 deletions spark/src/main/java/org/apache/comet/NativeBase.java
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import java.io.InputStreamReader;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.time.zone.ZoneRulesProvider;
import java.util.concurrent.atomic.AtomicBoolean;

import org.slf4j.Logger;
Expand Down Expand Up @@ -101,6 +102,7 @@ static synchronized void load() {
}

initWithLogConf();
warnOnTzdataMismatch();
// Only set the Arrow properties when debugging mode is off
if (!(boolean) CometConf.COMET_DEBUG_ENABLED().get()) {
setArrowProperties();
Expand Down Expand Up @@ -177,6 +179,35 @@ private static void initWithLogConf() {
init(logConfPath, logLevel);
}

/**
* Native code converts between instants and local time with the IANA timezone database that
* chrono-tz compiles into libcomet, while Spark uses the JVM's. When the two versions differ,
* local times in timezones whose rules changed between them can differ from Spark's.
*/
private static void warnOnTzdataMismatch() {
try {
String warning =
tzdataMismatchWarning(getTzdataVersion(), ZoneRulesProvider.getVersions("UTC").lastKey());
if (warning != null) {
LOG.warn(warning);
}
} catch (Throwable t) {
LOG.debug("Could not compare timezone database versions", t);
}
}

/** The warning to log when native code and the JVM use different tzdata versions, or null. */
static String tzdataMismatchWarning(String nativeVersion, String jvmVersion) {
if (nativeVersion.equals(jvmVersion)) {
return null;
}
return String.format(
"Comet's native library uses timezone database %s, but the JVM uses %s. Local times that "
+ "Comet computes natively can differ from Spark's in timezones whose rules changed "
+ "between these versions.",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JDK builds ship different tzdata versions. For example, the tzdb.dat headers of the JDKs I have installed say 2024b and 2026a, so I'd expect this warning on most clusters (I haven't run it). Would it make sense to end the message with a pointer to the "Timezone Database Versions" docs section, and let that section say what users can do about it?

nativeVersion, jvmVersion);
}

private static void cleanupOldTempLibs() {
String tempFolder = new File(System.getProperty("java.io.tmpdir")).getAbsolutePath();
File dir = new File(tempFolder);
Expand Down Expand Up @@ -331,4 +362,11 @@ public static void releaseNative() throws Throwable {
* @return true if object_store can construct both a store and an object key for this URL
*/
public static native boolean isObjectStoreSchemeSupported(String url);

/**
* The version of the IANA timezone database that native code uses, such as "2025b".
*
* @return the version compiled into libcomet
*/
public static native String getTzdataVersion();
}
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,13 @@ class CometTemporalExpressionSuite extends CometTestBase with AdaptiveSparkPlanH
assert(CometTimeZone.nativeId(None).contains("UTC"))
}

test("native tzdata version is reported") {
assert(NativeBase.getTzdataVersion.matches("[0-9]{4}[a-z]"), NativeBase.getTzdataVersion)
assert(NativeBase.tzdataMismatchWarning("2025b", "2025b") == null)
val warning = NativeBase.tzdataMismatchWarning("2025b", "2023c")
assert(warning.contains("2025b") && warning.contains("2023c"), warning)
}

test("date_trunc (TruncTimestamp) - reading from DataFrame") {
val supportedFormats = CometTruncTimestamp.supportedFormats
val unsupportedFormats = Seq("invalid")
Expand Down
Loading