Skip to content

BUG: Raise promotion error if a DType was provided in array coercion#17706

Merged
seberg merged 6 commits into
numpy:masterfrom
seberg:fix-void-coercion
Nov 10, 2020
Merged

BUG: Raise promotion error if a DType was provided in array coercion#17706
seberg merged 6 commits into
numpy:masterfrom
seberg:fix-void-coercion

Conversation

@seberg

@seberg seberg commented Nov 3, 2020

Copy link
Copy Markdown
Member

This fixes mainly the creation of unstructured void arrays from
scalars, by allowing promotion two unstructured void dtypes.

Structured voids will fail promoting. This promotion error is
now correctly floated out instead of using object dtype which
is incorrect when a dtype was passed in.

This fixes mainly the creation of unstructured void arrays from
scalars, by allowing promotion two unstructured void dtypes.

Structured voids will fail promoting. This promotion error is
now correctly floated out instead of using `object` dtype which
is incorrect when a dtype was passed in.
@seberg seberg force-pushed the fix-void-coercion branch from 34800ec to 937e624 Compare November 3, 2020 19:37

@mattip mattip left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code itself looks good, but could use some documentation cleanup

Comment thread numpy/core/src/multiarray/array_coercion.c
Comment thread numpy/core/src/multiarray/array_coercion.c
Comment thread numpy/core/src/multiarray/dtypemeta.c Outdated
mattip
mattip previously approved these changes Nov 4, 2020
Comment thread numpy/core/tests/test_multiarray.py Outdated
arr = np.array([b"12345", b"1234"], dtype="V")
assert arr.dtype == 'V5'
# Check the same for the casting path:
arr = np.array([b"1234", b"12345"], dtype="O").astype("V")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this do? Zero-pad? Undefined-data-pad?

Can you add a test for that?

@eric-wieser

Copy link
Copy Markdown
Member

I'm not convinced that allowing promotion of void buffers is sensible - the void type is not null-terminated, so void("\0") and void("\0\0") are different things. If the user wants to null-pad their void data, they can use np.bytes_, either directly or by casting to void through `bytes..

@seberg

seberg commented Nov 4, 2020

Copy link
Copy Markdown
Member Author

Hmmpf, that is a good point. The reason for that change was to make the np.array call work because it currently (inexplicitly) does.

I guess I can make the coercion with np.array([b"a", b"ab"], dtype="V") just fail and we can probably get away with it. That sounds actually better to me from that perspective. Does that sound fair to you guys?

@mattip mattip dismissed their stale review November 4, 2020 17:05

not ready yet

@mattip mattip added the triage review Issue/PR to be discussed at the next triage meeting label Nov 4, 2020
This means that we effectively remove certain (strange) array coercions
for unstructured void datatypes.
@seberg seberg changed the title BUG: Fix unstructured Void array creation and promotion BUG: Raise promotion error if a DType was provided in array coercion Nov 4, 2020
@seberg

seberg commented Nov 4, 2020

Copy link
Copy Markdown
Member Author

OK, I change it to that. We can briefly discuss this, but I think this is the reasonable thing to do. A bare dtype="V" should be very strange, and in fact there was a bug here that it went to object even though "V" was provided and nobody noticed yet, so it doesn't seem likely it affects any important downstream package (and I don't see how they could employ dtype="V" reasonably).

@eric-wieser

Copy link
Copy Markdown
Member

Can we raise an error message that indicates that null padding can be achieved by using bytes instead?

@seberg

seberg commented Nov 4, 2020

Copy link
Copy Markdown
Member Author

Not really, since this is a promotion error message you would need special handling just for void somewhere...

@eric-wieser

Copy link
Copy Markdown
Member

The code right now seems to give the error "invalid type promotion with structured or void datatype(s)". I'm suggesting we tailor that a bit for different types of void dtypes.

Or is your point that the error is swallowed?

@seberg

seberg commented Nov 5, 2020

Copy link
Copy Markdown
Member Author

Sorry, too much casting :(. For np.can_cast I can't raise a custom error, promotion currently indeed raises its own error and that seems fine (we are not likely to write if can promote_types(...)).

I added the error, it will also show up for other things (although probably not realistically).

I guess one thing we could think about is creating a PromotionTypeError if we ever run into the issue of wanting to be careful about it. In theory our comparison code does run into it. arr1 == arr2 has (or should have) a try/except around promotion and return an array of False if it fails.

Comment thread numpy/core/src/multiarray/dtypemeta.c Outdated
Co-authored-by: Eric Wieser <wieser.eric@gmail.com>
@seberg seberg added triaged Issue/PR that was discussed in a triage meeting and removed triage review Issue/PR to be discussed at the next triage meeting labels Nov 5, 2020
@seberg

seberg commented Nov 9, 2020

Copy link
Copy Markdown
Member Author

Gentle ping, can we put this in?

@mattip mattip left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me

Comment thread numpy/core/tests/test_numeric.py Outdated
@seberg

seberg commented Nov 9, 2020

Copy link
Copy Markdown
Member Author

Thanks @mattip, its not a big change and it is holding me up a bit, so I will squash merge it in a bit when the test ran through (undid a leftover change from the first version).

@seberg seberg merged commit 17cd07f into numpy:master Nov 10, 2020
@seberg seberg deleted the fix-void-coercion branch November 10, 2020 00:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

00 - Bug 06 - Regression triaged Issue/PR that was discussed in a triage meeting

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants