Skip to content

Series cumSum/cumProd/cumMax/cumMin, abs, std/var and nUnique mishandle missing values #708

Open

Description

@trackoor

Describe the bug
Danfo follows the pandas API and skips missing values in Series.sum() and Series.mean(), but cumSum, cumProd, cumMax, cumMin, abs, std, var, and nUnique mishandle NaN and null:

  • cumSum / cumMax / cumMin let NaN propagate to later entries instead of skipping missing values, and cumProd treats null as 0 (2.5 * null === 0).
  • abs passes null to Math.abs(null), turning a missing value into 0.
  • std and var return 0 instead of NaN on a single-element Series (unbiased sample variance with ddof=1 divides by N - 1 = 0).
  • nUnique counts NaN as a distinct value.

To Reproduce
Run with node after npm i danfojs-node@1.2.0:

const { Series } = require('danfojs-node')

console.log('cumSum:', new Series([3, NaN, 7, -1.5]).cumSum().values)
console.log('cumProd:', new Series([1, 2.5, null, 4]).cumProd().values)
console.log('cumMax:', new Series([NaN, 1, 2]).cumMax().values)
console.log('cumMin:', new Series([NaN, 1, 2]).cumMin().values)
console.log('abs:', new Series([-1, null, -3]).abs().values)
console.log('std:', new Series([1]).std())
console.log('var:', new Series([1]).var())
console.log('nUnique:', new Series([3, NaN, 7, NaN]).nUnique())

Actual output:

cumSum: [ 3, NaN, NaN, NaN ]
cumProd: [ 1, 2.5, 0, 0 ]
cumMax: [ NaN, NaN, NaN ]
cumMin: [ NaN, NaN, NaN ]
abs: [ 1, 0, 3 ]
std: 0
var: 0
nUnique: 3

Expected behavior
Matching pandas defaults (skipna=True, ddof=1, dropna=True):

cumSum: [ 3, NaN, 10, 8.5 ]
cumProd: [ 1, 2.5, NaN, 10 ]
cumMax: [ NaN, 1, 2 ]
cumMin: [ NaN, 1, 2 ]
abs: [ 1, NaN, 3 ]
std: NaN
var: NaN
nUnique: 2

Desktop (please complete the following information):

  • OS: Linux x86_64
  • Node.js: v26.13.0
  • Version: danfojs-node@1.2.0 (also on dev at 21d8c55)

Additional context
In src/danfojs-base/core/series.ts, cumOps combines tempval and currVal directly without checking utils.isEmpty, and abs runs this.values.map(val => Math.abs(val as number)) where Math.abs(null) evaluates to 0.

Credit: https://ucsbplse.github.io/

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions