Skip to content

Conversation

xinrong-meng
Copy link
Member

What changes were proposed in this pull request?

Bools |/&/^ None should fail under ANSI, following native pandas

For example,

>>> pd.Series([True, False]) | None
Traceback (most recent call last):
...
TypeError: unsupported operand type(s) for |: 'bool' and 'NoneType'

but under ANSI

>>> ps.Series([True, False]) | None
0    False                                                                      
1    False
dtype: bool

Why are the changes needed?

Part of https://issues.apache.org/jira/browse/SPARK-53389

Does this PR introduce any user-facing change?

No, the feature hasn't been released yet.

Now bools |/&/^ None fails under ANSI, e.g.

>>> ps.Series([True, False]) | None
Traceback (most recent call last):
...
TypeError: OR can not be applied to given types.

How was this patch tested?

Unit tests

Commands below passed:

 1027  SPARK_ANSI_SQL_MODE=true ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests"
 1028  SPARK_ANSI_SQL_MODE=false ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests"

Was this patch authored or co-authored using generative AI tooling?

No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant