Conversation
abstrackt
changed the title
abstrackt
commented
May 18, 2023
| template <class NumberT> inline __device__ Num operator()(NumberT c) const { | ||
| Num val = static_cast<Num>(c); | ||
| Num div = static_cast<Num>(pow(10.0, Dec)); | ||
| val = trunc(val*div)/div; |
Author
There was a problem hiding this comment.
Multiply by a power of 10, truncate to the nearest integer to cut the remaining decimal places and divide to revert to the original value.
abstrackt
commented
May 18, 2023
| constexpr static In Zero = static_cast<In>(0); | ||
|
|
||
| template <class NumberT> inline __device__ bool operator()(NumberT c) const { | ||
| In val = static_cast<In>(c); |
Author
There was a problem hiding this comment.
Simple cast depending on the sign of the input value
|
this fixes issue #10 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Number test context contained invalid values for the m_max_precision variable. This limited test data to three digits, meaning for example that any floating point numbers above 10 would get truncated to 10.x, instead of 10.xxx, etc. This created issues with some tests which deal with truncating numbers.
Also added simple rounding and bool cast functors with tests as described below.