Repository navigation
docs: improve crate API docs for docs.rs - #57
Conversation
Reworks the documentation entry points: - Top-level README: usage pointer, layout, contributing checklist. - Crate README: user-facing content for crates.io (overview, quick start, CLI, caveats). - New `CONTRIBUTING.md`: implementation and testing guidelines moved out of the crate README. Rustdoc/Cargo metadata changes are in #57.
comphead
left a comment
There was a problem hiding this comment.
Thanks @andygrove. The description asks CI to check the doc build, but no workflow runs cargo doc or cargo test -p datafusion-spark. lint.yml runs cargo fmt and cargo clippy, and sqllogictest.yml runs only cargo test -p datafusion-spark-sqllogictest. So neither the new intra-doc links nor the crate's doctests are checked by CI. I read the link targets (function::string, register_all, planner::SparkFunctionPlanner, Expr) and they resolve by name, but I did not build the docs. Running RUSTDOCFLAGS="-D warnings" cargo doc -p datafusion-spark --no-deps --all-features locally would settle it. The missing cargo test -p datafusion-spark job also means the Rust unit tests in the open function PRs are not run by CI, which might be worth its own issue.
| //! [dependencies] | ||
| //! datafusion = "X" | ||
| //! # `core` enables `SessionStateBuilderSpark`, the easiest way to turn Spark mode on | ||
| //! datafusion-spark = { version = "X", features = ["core"] } |
There was a problem hiding this comment.
This core snippet is repeated under "Example: enabling Apache Spark features with SessionStateBuilder" (lines 155-157). Could we keep one copy and link to it from the other section, so the two do not drift?
There was a problem hiding this comment.
Kept the copy in Getting started and linked to it from the SessionStateBuilder example. Done in 8961c84.
| //! * **Registration overrides built-ins.** Spark functions are registered under | ||
| //! their Spark names. If DataFusion already has a function with that name, | ||
| //! the Spark one replaces it for the session. Register Spark functions after | ||
| //! DataFusion's defaults, and only when you want Spark behavior. |
There was a problem hiding this comment.
This bullet says the same thing as the register_all paragraph below ("Any existing functions will be overwritten, with these Spark functions taking priority"). Could we keep it in one place? This section is a good home, and the example can then link to it.
There was a problem hiding this comment.
Kept it in "Semantics and caveats"; the register_all paragraph now links there. Done in 8961c84.
| // SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership | ||
| // SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements | ||
|
|
||
| //! Spark predicate functions. No functions are implemented in this package yet. |
There was a problem hiding this comment.
"No functions are implemented in this package yet" appears in seven packages (csv, lambda, predicate, struct, table, window, xml), and nothing prompts an update when the first function lands. For example apache/datafusion#27078 adds at_least_n_non_nulls to predicate. Could these say only what the package is for, for example //! Spark predicate functions.?
There was a problem hiding this comment.
All seven packages now say only what they're for, e.g. //! Spark predicate functions. Done in 8961c84.
4b73447 to
8961c84
Compare
|
Thanks @comphead. I ran |
There was a problem hiding this comment.
Thanks for improving the crate documentation. The added getting-started guidance, compatibility caveats, and function-package descriptions make the Spark integration easier to discover and use. The approach looks appropriate, and I found no blocking issues.
The documentation correctly keeps Spark-specific function registration and planning separate from broader SQL compatibility. This distinction matters because enabling Spark features does not automatically reproduce every aspect of Spark SQL behavior, including parser dialect selection.
Approving.
I have two optional suggestions to improve navigation and clarify dialect configuration. Neither affects correctness or needs to block this PR.
| //! | ||
| //! Functions are grouped into packages by category (for example [`function::string`], | ||
| //! [`function::datetime`], [`function::math`], [`function::hash`]). See the list of | ||
| //! [modules](#modules) in this crate for the available packages. Each package |
There was a problem hiding this comment.
Consider linking "available packages" directly to [function] instead of #modules. The package descriptions are under function/index.html, so this would save readers a navigation step. The existing link works; this is only a discoverability suggestion.
There was a problem hiding this comment.
Good idea, done in a0506c7. "Available packages" now links to [function].
| //! * **ANSI mode.** Functions whose Spark behavior depends on ANSI mode read | ||
| //! `datafusion.execution.enable_ansi_mode` from the session configuration. | ||
| //! * **Type inference.** DataFusion and Spark do not infer literal types in the | ||
| //! same way, so add explicit casts (`CAST(1 AS INT)`) when exact types matter. |
There was a problem hiding this comment.
Consider clarifying that with_spark_features() does not select the Spark SQL parser dialect. Spark-specific syntax such as CAST(1 AS LONG) requires setting config.options_mut().sql_parser.dialect = Dialect::Spark and passing config to SessionStateBuilder::with_config(config); the generic dialect rejects LONG. Not all supported queries require the Spark dialect.
There was a problem hiding this comment.
Added in a0506c7. "Semantics and caveats" has a new SQL dialect bullet saying that registering Spark functions does not change the parser, so CAST(1 AS LONG) needs the Spark dialect. SessionStateBuilderSpark now has a doctest that sets sql_parser.dialect = Dialect::Spark, passes the config to with_config, and runs CAST(1 AS LONG). The existing test_spark_dialect_with_spark_functions unit test already checks that the generic dialect rejects LONG.
|
Thanks for the review @kosiew! I've addressed the suggestions (see the replies inline) and will merge once CI is green. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the SQL dialect Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a0506c7 to
ec80d24
Compare
## Which issue does this PR close? Part of #56: the "Build the docs with `RUSTDOCFLAGS="-D warnings"`" item. ## Rationale for this change Nothing in CI builds the API docs. A broken intra-doc link, invalid HTML in a doc comment or another rustdoc warning only shows up on docs.rs after a release. apache/datafusion checks this in `ci/scripts/rust_docs.sh`, and nothing replaced that check when the crate moved here. ## What changes are included in this PR? A new `.github/workflows/docs.yml` that runs `cargo doc --document-private-items --no-deps --workspace --all-features` with `RUSTDOCFLAGS: -D warnings`. - The flags are the same as DataFusion's, plus `--all-features`. That flag documents the items behind the `core` feature, as docs.rs does (`[package.metadata.docs.rs] all-features = true`). - It is a separate workflow file, so it doesn't conflict with #318, which adds `test.yml`, or #320, which adds a job to `lint.yml`. ## Are these changes tested? The command passes locally on current `main` and on the branch of #57, which improves the crate's API docs. So this check won't block #57. This PR can be rebased after #57 merges. `actionlint` reports no problems, and the workflow runs on this PR. ## Are there any user-facing changes? No. _This PR was written by an LLM (Claude) and reviewed before opening._ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Part of #55
Rationale
The published crate page on crates.io is generic, and docs.rs shows a bare list of function modules with no descriptions.
Changes
lib.rs(motivation, getting started, packages, semantics and caveats).//!module docs to every function package. Packages with no functions yet say so.SparkFunctionPlanner.keywords,categoriesanddocumentationtoCargo.toml.Testing
Docs only. I could not run
cargo docin my checkout (no workspace root), so please check that CI's doc build is clean. README changes are in #60.Per-function documentation is tracked separately in #55.