tensorflow / java-ndarray Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
Hi,
I am seeing this behavior when creating a Tensor using a buffer that is created by wrapping an array.
The example below can reproduce the issue:
// create an buffer by wrapping an array but buffer start from position 3 and length is 4
final IntBuffer src = IntBuffer.wrap(new int[]{0, 1, 2, 3, 4, 5, 6, 7, 8}, 3, 4).asReadOnlyBuffer();
final org.tensorflow.Tensor tensor = TInt32.tensorOf(Shape.of(4), DataBuffers.of(src));
final int[] value = new int[4];
tensor.asRawTensor().data().asInts().read(value);
assertArrayEquals(new int[]{0, 1, 2, 3}, value); // This is not really a correct behavior, it is not honoring the buffer offset when wrapping the array, it just takes the first 4 elements
//assertArrayEquals(new int[]{3, 4, 5, 6}, value); // This should be the desired result
System info:
Java 8
tensorflow-java (tensorflow-core-api) 0.4.1
How do I multiply two matricies (2d arrays)?
Have the release in the README up to date
Latest version is 0.3.3 but README shows 0.3.2
See https://mvnrepository.com/artifact/org.tensorflow/ndarray/0.3.3
Note: title changed - Strange error interfacing with Scala when doing a StdArrays.copyFrom
with a matrix using JDK 17
While exploring and documenting the Index API, I get errors when using Scala. I get a java.nio.BufferOverflowException
exception. However, what seems like an equivalent Java test, has no problem. I expect the Scala code to work just as it does in Java. I am trying to replicate the error in Java.
I understand that Scala is not a supported platform, but will appreciate any help. I have been at this for a week.
I have created a test that succeeds here . The relevant code is:
String[][] indexData = new String[5][4];
for (int i=0 ; i < 5; i++)
for (int j=0 ; j < 4; j++)
indexData[i][j] = "("+j+", "+i+")";
NdArray<String> matrix2d = StdArrays.ndCopyOf(indexData);
assertEquals(2, matrix2d.rank());
/*
|(0, 0), (1, 0), (2, 0), (3, 0)|
|(0, 1), (1, 1), (2, 1), (3, 1)|
|(0, 2), (1, 2), (2, 2), (3, 2)|
|(0, 3), (1, 3), (2, 3), (3, 3)|
|(0, 4), (1, 4), (2, 4), (3, 4)|
*/
// all rows, columns 1 to 2
NdArray<String> same7 = matrix2d.slice(Indices.all(), Indices.slice(1,3));
assertEquals(2, same7.rank());
assertEquals(Shape.of(5,2), same7.shape());
assertEquals(10, same7.size());
String[][] expectedr7 = new String[][]
{
{"(1, 0)", "(2, 0)"},
{"(1, 1)", "(2, 1)"},
{"(1, 2)", "(2, 2)"},
{"(1, 3)", "(2, 3)"},
{"(1, 4)", "(2, 4)"}
};
String[][] lArray = new String[5][2];
StdArrays.copyFrom(same7, lArray);
The equivalent Scala code is:
val indexes =
for
i <- (0 until 5)
yield
val row = for { j <- (0 until 4) } yield s"($j, $i)"
row.toArray
val indexData = indexes.toArray
val matrix2d = StdArrays.ndCopyOf(indexData)
// all rows, columns 1 to 2
val same7 = matrix2d.slice(Indices.all(), Indices.slice(1,3))
assert(same7.rank() == 2)
assert(same7.shape() == Shape.of(5,2))
assert(same7.size() == 10)
val expected_r7 = Array
(
Array("(1, 0)", "(2, 0)"),
Array("(1, 1)", "(2, 1)"),
Array("(1, 2)", "(2, 2)"),
Array("(1, 3)", "(2, 3)"),
Array("(1, 4)", "(2, 4)")
);
val lArray = Array.ofDim[String](5,2)
assert(lArray.size == 5)
assert(lArray(0).size == 2)
StdArrays.copyFrom(same7, lArray)
Exception in thread "main" java.nio.BufferOverflowException
at org.tensorflow.ndarray.impl.buffer.Validator.copyToArgs(Validator.java:61)
at org.tensorflow.ndarray.impl.buffer.misc.ArrayDataBuffer.copyTo(ArrayDataBuffer.java:52)
at org.tensorflow.ndarray.impl.dense.DataTransfer.execute(DataTransfer.java:114)
at org.tensorflow.ndarray.impl.dense.AbstractDenseNdArray.read(AbstractDenseNdArray.java:94)
at org.tensorflow.ndarray.StdArrays.copyFrom(StdArrays.java:2735)
at org.tensorflow.ndarray.StdArrays.lambda$copyFrom$75(StdArrays.java:2752)
at org.tensorflow.ndarray.impl.sequence.SlicingElementSequence.lambda$forEachIndexed$0(SlicingElementSequence.java:65)
at org.tensorflow.ndarray.impl.sequence.NdPositionIterator.forEachIndexed(NdPositionIterator.java:43)
at org.tensorflow.ndarray.impl.sequence.SlicingElementSequence.forEachIndexed(SlicingElementSequence.java:64)
at org.tensorflow.ndarray.StdArrays.copyFrom(StdArrays.java:2751)
at core.TensorExamples$.ndArraySlicing(TensorExamples.scala:751)
at core.TensorExamples$.main(TensorExamples.scala:1176)
at core.TensorExamples.main(TensorExamples.scala)
It is difficult for me get a failing example in Scala that anyone here could easily run and check. So my goal is to try and replicate this in Java. In particular I have found that the Java version does not follow the same path as the Scala code. When debugging the Java version I manually get this trace:
/workspaces/java-ndarray/ndarray/src/main/java/org/tensorflow/ndarray/StdArrays.java [2747]
/workspaces/java-ndarray/ndarray/src/main/java/org/tensorflow/ndarray/StdArrays.java [2751]
/ndarray/src/main/java/org/tensorflow/ndarray/impl/dense/AbstractDenseNdArray.java [50]
/workspaces/java-ndarray/ndarray/src/main/java/org/tensorflow/ndarray/impl/sequence/SlicingElementSequence.java []
/workspaces/java-ndarray/ndarray/src/main/java/org/tensorflow/ndarray/StdArrays.java [2752]
/workspaces/java-ndarray/ndarray/src/main/java/org/tensorflow/ndarray/StdArrays.java [2735]
/workspaces/java-ndarray/ndarray/src/main/java/org/tensorflow/ndarray/impl/dense/AbstractDenseNdArray.java [94]
dst: ArrayDataBuffer length = 2, offset = 0, readOnly = false
/workspaces/java-ndarray/ndarray/src/main/java/org/tensorflow/ndarray/impl/dense/DataTransfer.java [104] <-----> [114]
Note that with the Java version the DataTransfer
class executes the copyByElement [104]
method. The Scala code does a srcBuffer.copyTo [114]
. The test srcDimensions.isSegmented()
seems to be different. The relevant code is shown below:
static <T, B extends DataBuffer<T>> void execute(B srcBuffer, DimensionalSpace srcDimensions, B dstBuffer, OfValue<B> valueTransfer) {
if (srcDimensions.isSegmented()) {
long elementSize = srcDimensions.get(srcDimensions.segmentationIdx()).elementSize();
copyByElement(
srcBuffer,
PositionIterator.create(srcDimensions, srcDimensions.segmentationIdx()),
dstBuffer,
PositionIterator.sequence(elementSize, dstBuffer.size()),
elementSize,
valueTransfer
);
} else {
srcBuffer.copyTo(dstBuffer, srcDimensions.physicalSize());
}
}
So my question is, how can one change the Java test code so that a srcDimensions.isSegmented()
is false and I srcBuffer.copyTo
method is used.
On another note, would it be interesting to provide additional tests as a PR, including the one above?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.