Skip to content

docs: improve crate API docs for docs.rs - #57

Merged
andygrove merged 2 commits into
apache:mainfrom
andygrove:docs-crate-docs
Nov 16, 2026
Merged

andygrove merged 2 commits into
apache:mainfrom
andygrove:docs-crate-docs

Conversation

@andygrove

@andygrove andygrove commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Part of #55

Rationale

The published crate page on crates.io is generic, and docs.rs shows a bare list of function modules with no descriptions.

Changes

  • Expand the crate-level docs in lib.rs (motivation, getting started, packages, semantics and caveats).
  • Add //! module docs to every function package. Packages with no functions yet say so.
  • Document SparkFunctionPlanner.
  • Add keywords, categories and documentation to Cargo.toml.

Testing

Docs only. I could not run cargo doc in my checkout (no workspace root), so please check that CI's doc build is clean. README changes are in #60.

Per-function documentation is tracked separately in #55.

@andygrove
andygrove marked this pull request as draft November 11, 2026 21:38
@andygrove andygrove changed the title docs: improve crate README and API docs for crates.io and docs.rs docs: improve crate API docs for docs.rs Nov 12, 2026
andygrove added a commit that referenced this pull request Nov 12, 2026
Reworks the documentation entry points:
- Top-level README: usage pointer, layout, contributing checklist.
- Crate README: user-facing content for crates.io (overview, quick
start, CLI, caveats).
- New `CONTRIBUTING.md`: implementation and testing guidelines moved out
of the crate README.

Rustdoc/Cargo metadata changes are in #57.
@andygrove
andygrove marked this pull request as ready for review November 13, 2026 13:56

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. The description asks CI to check the doc build, but no workflow runs cargo doc or cargo test -p datafusion-spark. lint.yml runs cargo fmt and cargo clippy, and sqllogictest.yml runs only cargo test -p datafusion-spark-sqllogictest. So neither the new intra-doc links nor the crate's doctests are checked by CI. I read the link targets (function::string, register_all, planner::SparkFunctionPlanner, Expr) and they resolve by name, but I did not build the docs. Running RUSTDOCFLAGS="-D warnings" cargo doc -p datafusion-spark --no-deps --all-features locally would settle it. The missing cargo test -p datafusion-spark job also means the Rust unit tests in the open function PRs are not run by CI, which might be worth its own issue.

Comment thread crates/spark/src/lib.rs
//! [dependencies]
//! datafusion = "X"
//! # `core` enables `SessionStateBuilderSpark`, the easiest way to turn Spark mode on
//! datafusion-spark = { version = "X", features = ["core"] }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This core snippet is repeated under "Example: enabling Apache Spark features with SessionStateBuilder" (lines 155-157). Could we keep one copy and link to it from the other section, so the two do not drift?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept the copy in Getting started and linked to it from the SessionStateBuilder example. Done in 8961c84.

Comment thread crates/spark/src/lib.rs
Comment on lines +54 to +57
//! * **Registration overrides built-ins.** Spark functions are registered under
//! their Spark names. If DataFusion already has a function with that name,
//! the Spark one replaces it for the session. Register Spark functions after
//! DataFusion's defaults, and only when you want Spark behavior.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bullet says the same thing as the register_all paragraph below ("Any existing functions will be overwritten, with these Spark functions taking priority"). Could we keep it in one place? This section is a good home, and the example can then link to it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept it in "Semantics and caveats"; the register_all paragraph now links there. Done in 8961c84.

// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements

//! Spark predicate functions. No functions are implemented in this package yet.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"No functions are implemented in this package yet" appears in seven packages (csv, lambda, predicate, struct, table, window, xml), and nothing prompts an update when the first function lands. For example apache/datafusion#27078 adds at_least_n_non_nulls to predicate. Could these say only what the package is for, for example //! Spark predicate functions.?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All seven packages now say only what they're for, e.g. //! Spark predicate functions. Done in 8961c84.

@andygrove

Copy link
Copy Markdown
Member Author

Thanks @comphead. I ran RUSTDOCFLAGS="-D warnings" cargo doc -p datafusion-spark --no-deps --all-features locally and it builds with no warnings, so the intra-doc links resolve. cargo test -p datafusion-spark passes, including the doctests. I also rebased onto main and squashed to one commit, because the old history added and then reverted the README and CONTRIBUTING changes that #60 has since landed. I agree the missing cargo test -p datafusion-spark / cargo doc CI job deserves its own issue.

@kosiew kosiew left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove,

Thanks for improving the crate documentation. The added getting-started guidance, compatibility caveats, and function-package descriptions make the Spark integration easier to discover and use. The approach looks appropriate, and I found no blocking issues.

The documentation correctly keeps Spark-specific function registration and planning separate from broader SQL compatibility. This distinction matters because enabling Spark features does not automatically reproduce every aspect of Spark SQL behavior, including parser dialect selection.

Approving.
I have two optional suggestions to improve navigation and clarify dialect configuration. Neither affects correctness or needs to block this PR.

Comment thread crates/spark/src/lib.rs Outdated
//!
//! Functions are grouped into packages by category (for example [`function::string`],
//! [`function::datetime`], [`function::math`], [`function::hash`]). See the list of
//! [modules](#modules) in this crate for the available packages. Each package

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider linking "available packages" directly to [function] instead of #modules. The package descriptions are under function/index.html, so this would save readers a navigation step. The existing link works; this is only a discoverability suggestion.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, done in a0506c7. "Available packages" now links to [function].

Comment thread crates/spark/src/lib.rs
//! * **ANSI mode.** Functions whose Spark behavior depends on ANSI mode read
//! `datafusion.execution.enable_ansi_mode` from the session configuration.
//! * **Type inference.** DataFusion and Spark do not infer literal types in the
//! same way, so add explicit casts (`CAST(1 AS INT)`) when exact types matter.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider clarifying that with_spark_features() does not select the Spark SQL parser dialect. Spark-specific syntax such as CAST(1 AS LONG) requires setting config.options_mut().sql_parser.dialect = Dialect::Spark and passing config to SessionStateBuilder::with_config(config); the generic dialect rejects LONG. Not all supported queries require the Spark dialect.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in a0506c7. "Semantics and caveats" has a new SQL dialect bullet saying that registering Spark functions does not change the parser, so CAST(1 AS LONG) needs the Spark dialect. SessionStateBuilderSpark now has a doctest that sets sql_parser.dialect = Dialect::Spark, passes the config to with_config, and runs CAST(1 AS LONG). The existing test_spark_dialect_with_spark_functions unit test already checks that the generic dialect rejects LONG.

@andygrove

Copy link
Copy Markdown
Member Author

Thanks for the review @kosiew! I've addressed the suggestions (see the replies inline) and will merge once CI is green.

andygrove and others added 2 commits November 16, 2026 05:32
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the SQL dialect

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove
andygrove merged commit eb9275f into apache:main Nov 16, 2026
4 checks passed
mbutrovich pushed a commit that referenced this pull request Nov 16, 2026
## Which issue does this PR close?

Part of #56: the "Build the docs with `RUSTDOCFLAGS="-D warnings"`"
item.

## Rationale for this change

Nothing in CI builds the API docs. A broken intra-doc link, invalid HTML
in a doc comment or another rustdoc warning only shows up on docs.rs
after a release. apache/datafusion checks this in
`ci/scripts/rust_docs.sh`, and nothing replaced that check when the
crate moved here.

## What changes are included in this PR?

A new `.github/workflows/docs.yml` that runs `cargo doc
--document-private-items --no-deps --workspace --all-features` with
`RUSTDOCFLAGS: -D warnings`.

- The flags are the same as DataFusion's, plus `--all-features`. That
flag documents the items behind the `core` feature, as docs.rs does
(`[package.metadata.docs.rs] all-features = true`).
- It is a separate workflow file, so it doesn't conflict with #318,
which adds `test.yml`, or #320, which adds a job to `lint.yml`.

## Are these changes tested?

The command passes locally on current `main` and on the branch of #57,
which improves the crate's API docs. So this check won't block #57. This
PR can be rebased after #57 merges. `actionlint` reports no problems,
and the workflow runs on this PR.

## Are there any user-facing changes?

No.

_This PR was written by an LLM (Claude) and reviewed before opening._

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants