Lab 1 / Exercise 4 / Task 2 / Step 19 about azure-synapse-analytics-workshop-400 HOT 5 CLOSED

For some reason, on my first try, the TransactionDate column was not shown when I went to "Mappings" on my copy task and tried to map it from source to destination. I had to import the schema again from the parquet file so that it was shown. Maybe a bug, maybe a mistake on my part... just a heads up in case you can repro.

from azure-synapse-analytics-workshop-400.

after having corrected the mapping, I am getting the following error message when I trigger the pipeline execution:

{
"errorCode": "2200",
"message": "ErrorCode=ParquetJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.OutOfMemoryError:Direct buffer memory\ntotal entry:32\r\njava.nio.Bits.reserveMemory(Bits.java:658)\r\njava.nio.DirectByteBuffer.(DirectByteBuffer.java:123)\r\njava.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)\r\norg.apache.parquet.hadoop.codec.SnappyDecompressor.setInput(SnappyDecompressor.java:102)\r\norg.apache.parquet.hadoop.codec.NonBlockedDecompressorStream.read(NonBlockedDecompressorStream.java:46)\r\njava.io.DataInputStream.readFully(DataInputStream.java:195)\r\njava.io.DataInputStream.readFully(DataInputStream.java:169)\r\norg.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:251)\r\norg.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:202)\r\norg.apache.parquet.column.impl.ColumnReaderImpl.readPageV1(ColumnReaderImpl.java:592)\r\norg.apache.parquet.column.impl.ColumnReaderImpl.access$300(ColumnReaderImpl.java:61)\r\norg.apache.parquet.column.impl.ColumnReaderImpl$3.visit(ColumnReaderImpl.java:541)\r\norg.apache.parquet.column.impl.ColumnReaderImpl$3.visit(ColumnReaderImpl.java:538)\r\norg.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:96)\r\norg.apache.parquet.column.impl.ColumnReaderImpl.readPage(ColumnReaderImpl.java:538)\r\norg.apache.parquet.column.impl.ColumnReaderImpl.checkRead(ColumnReaderImpl.java:530)\r\norg.apache.parquet.column.impl.ColumnReaderImpl.consume(ColumnReaderImpl.java:642)\r\norg.apache.parquet.column.impl.ColumnReaderImpl.(ColumnReaderImpl.java:358)\r\norg.apache.parquet.column.impl.ColumnReadStoreImpl.newMemColumnReader(ColumnReadStoreImpl.java:82)\r\norg.apache.parquet.column.impl.ColumnReadStoreImpl.getColumnReader(ColumnReadStoreImpl.java:77)\r\norg.apache.parquet.io.RecordReaderImplementation.(RecordReaderImplementation.java:270)\r\norg.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:140)\r\norg.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:106)\r\norg.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:154)\r\norg.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:106)\r\norg.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:136)\r\norg.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:194)\r\norg.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:122)\r\norg.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:126)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetBatchReaderBridge.(ParquetBatchReaderBridge.java:68)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetBatchReaderBridge.open(ParquetBatchReaderBridge.java:63)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetFileBridge.createReader(ParquetFileBridge.java:22)\r\n.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'",
"failureType": "UserError",
"target": "Copy sales",
"details": []
}

from azure-synapse-analytics-workshop-400.

On my third attempt, I changed the "Data integration unit" of the copy data activity to "AUTO" and it completed the job successfully

from azure-synapse-analytics-workshop-400.

We have been seeing the out of memory error intermittently quite a few times. The best guess currently is that it has something to do with Parquet files that are larger in size.

from azure-synapse-analytics-workshop-400.

I have run through this a few times now and have not seen the issue popup again. Closing.

from azure-synapse-analytics-workshop-400.