Parquet Viewer Version
Version 2.6.0 and later. Works just fine on earlier versions.

Where was the parquet file created?
Locally using C#

Sample File
Contains PHI. I can make a sanitized one if you think it is needed. Let me know if you have trouble reproducing with a parquet file containing a byte[] column.

Describe the bug
When the window is widened to the column in question, the application repeatedly throws the exception in the attachment

Screenshots

Additional context
I suspect the app is having a problem with a column of type byte[]

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

Hello, I'm getting below error while building project. Please suggest what to do in this issue.

**Severity Code Description Project File Line Suppression State
Error The expression "[System.Version]::Parse('')" cannot be evaluated. Version string portion was too short or too long. ParquetFileViewer

Severity Code Description Project File Line Suppression State
Error CS0012 The type 'Object' is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. ParquetFileViewer C:\Users\ssharma1\ParquetViewer\src\ParquetFileViewer\UtilityMethods.cs 16 Active**

When opening a parquet file, ParquetViewer first launches a popup "Select fields to load", where you either can confirm to load all fields, or select the fields you want.

In all use cases relevant for me, I want to display all fields. Hence I'm wondering if it would be possible to skip this popup all together? It's just inconvenient to always confirm the "All fields...", before you see any data.

Maybe skipping this popup can become a configuration option somewhere?

Describe the feature you'd like to be added to Parquet Viewer

I would like the metadata viewer to show the custom key_value_metadata added to each column of the schema. PyArrow's API seems to allow this to be added at the schema level, while Parquet.Net's API adds it per row group, which is more inline with the actual file structure.

Share why this feature would be a good addition to the utility

I want to validate that I'm building Parquet files correctly with the data I expect. I would like to use metadata for per-column information like units and description.

Parquet Viewer Version
v2.3.6

Describe the bug
The window's close button is not exactly at the top right of the page.
Clicking exactly at the right close windows behind.

Screenshots

Additional context
You must be exactly at the very very top right.
It must be only a 1px gap.

Describe the feature you'd like to be added to Parquet Viewer
Support for the ZSTD compression format (and possibly other additional compression formats).

Share why this feature would be a good addition to the utility
I have some Parquet files compressed with ZSTD, I get the error in the screenshot below,

Screenshots

Would it be possible to add an element to view the custom file metadata?

This would be useful for accessing additional fields that may be stored as metadata in the file.

Parquet Viewer Version
What version of Parquet Viewer are you experiencing the issue with?
2.3.6.22567

Where was the parquet file created?
Apache Spark, Hive, Java, C#, pyarrow, etc.
Python through pandas with pyarrow backend

Sample File
Upload a sample file so the issue can be debugged!
DLIx12_test.zip

Describe the bug
A clear and concise description of what the bug is.
Opening file in ParquetViewer gives this as output. No idea what is wrong.
Can it be linked to pyarrow version 9? I have no problem opening same sized file for version 8.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

Thanks a lot for providing this Viewer!

I'm wondering if it should be possible to open a .parquet file in Windows Explorer directly with a mouse click. Currently, when I do this, the ParquetViewer starts up, but with an empty window. I then need to open the file manually by pressing main menu->Open, and browsing through the directory structure.

Maybe it's just a configuration that's missing on my side?
Otherwise, this is a kind feature request :)

Tobias

Parquet Viewer Version
2.4.2

Where was the parquet file created?
Not parquet related issue

Sample File
Not parquet related issue

Describe the bug
Launching latest pre-compiled binary fails silently.
Launching project using dotnet run fails silently
Launching project using Jetbrains Rider produces error Appx recipe file[] does not exist
Launching project using VS2022 works fine and then later launching produced binary works fine as well

Screenshots
I don't think they're helpful here

Additional context
Windows 10 21H2
dotnet cli version - 7.0.101
dotnet sdks installed - 7.0.101, 6.0.100
.NET 4.6 installed
.NET 4.7.2 or later is installed, or at least that's what 4.7.2 installer tells me

I'm not too familiar with windows GUI app development, so don't really know what to look at. If anything the error from running it in Rider seems to be the biggest hint, but can't really find much information on that specific error.

In the metadata viewer we could have a tab with row group info. Row group count, the size and number of rows of each of them

Is there a possibility that this could be implemented?
My files are compressed using pandas and brotli compression. Compression gzip works.
df.to_parquet(path=saveurl, compression='brotli')

Parquet Viewer Version
2.2.

https://github.com/elastacloud/parquet-dotnet/issues/314

It is already fixed in Parquet.Net: aloneguid/parquet-dotnet#91

I'm sending a PR to you to upgrade Parquet.Net version. I tested it with the new version and it fixes the issue

I created a parquet file with two columns - nullable int and non nullable int.
when I try to view it in ParquetViewer, it crashes when I select the non nullable int.
loading the same file with:

http://parquet-viewer-online.com/

is fine

test.zip

attached is the problematic file.

crash image:

I am exporting a table from SQL Server to parquet and back. I am using Parquet.NET v 3.0 to perform the export and it seems to run normally.
When opening one file (~266KB) with the viewer it just works fine. Another "larger" file (1,6MB) would throw the following exception:

after the column selection dialog.

I investigated further and this only happens if I select String columns that had been of type "varchar" on SQL server. Any "nvarchar" column works just fine.

However when passed to the writer they are converted to regular C# strings before, so encoding should not normally be an issue.

Parquet Viewer Version
What version of Parquet Viewer are you experiencing the issue with?
2.4.2.0

Where was the parquet file created?
pyarrow

Sample File
Example.zip

Describe the bug
I believe the bug comes from having two column names that are equal when viewed as lowercase.
I can open the file in pyarrow/python, not in ParquetViewer.

Screenshots

Additional context
The similar column names is a bug in my code, but should not make the program crash.

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

Parquet Viewer Version: 2.7.1.0

Where was the parquet file created?
python, using pandas and fastparquet library

Sample File
Sample file attached.

Example.zip

Describe the bug
Try to open file; if you select only first column, if will open fine. If you select all, second will cause problem and no data will be displayed.

Screenshots
Attached screenshot.

Additional context
Original column definition from Oracle database:

Limit type        NOT NULL VARCHAR2(20)
Limit period in days                NUMBER

Latest version(v2.3.5) no support for sort data by columns. Can it be supported?

Parquet Viewer Version
2.2

Where was the parquet file created?
C#

Sample File
Test.zip

Describe the bug
In UtilityMethods class this line contains a bug after the first call:

For example, if I have two row groups with 2 lines each.
At first call the if will be if (rowIndex=0 >= readRecords=2) - OK
But the next calls will be if (rowIndex=2 >= readRecords=2) and it will break. Unless the second row group is bigger than the first, but it is buggy anyway, since it will skip rows.

After fixing this issue, I also found another problem, where the row count is not respected after the first row group:

Screenshots

Additional context
Add any other context about the problem here.

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

Larger .parquet files give ArgumentOutOfRangeException" error for me, just like mentioned in closed ticket #5.

For me, it isn't an urgent issue, as I transitioned to .sqlite files for some other technical reasons, but as it might be usefull for you or other users... but as you asked for example files in the now closed tickets.

The .parquet file was created using pyarrow in python, and I tested in ParquetViewer 1.1:
https://drive.google.com/open?id=12vIw3f5tMURfzhI6hO3clHtPafPwhqnq

Hi,

i wanted to view a parquet file that contains a datetime field. However it caused this error:

The file is not broken, as it worked within AWS Glue/Athena.

First of all: I'm not sure if my problem is a problem of ParquetViewer or parquet-dotnet please let me know if I'm wrong here...

Parquet Viewer Version
v2.3.6

Where was the parquet file created?
Apache Spark 3.1.2

Sample File
PySpark code for creating the file

import datetime

df = sc.parallelize([
    [
        1,
        datetime.date(1985, 12, 31),
        datetime.date(   1,  1,  2),
        datetime.date(9999, 12, 31),
        datetime.date.max,
        datetime.date.min,
        datetime.datetime(1985,  4, 13, 13,  5),
        datetime.datetime(   1,  1,  2,  0,  0),
        datetime.datetime(9999, 12, 31, 23, 59, 59),
        datetime.datetime.max,
        datetime.datetime.min
    ]
]).toDF((
    "ID",
    "Date_Normal",
    "Date_Low",
    "Date_High",
    "Date_Max",
    "Date_Min",
    "Timestamp_Normal",
    "Timestamp_Low",
    "Timestamp_High",
    "Timestamp_Max",
    "Timestamp_Min"
))

display( df )


spark.conf.set('spark.sql.legacy.parquet.int96RebaseModeInWrite', 'CORRECTED')
spark.conf.set('spark.sql.legacy.parquet.datetimeRebaseModeInWrite', 'CORRECTED')

(df.coalesce(1)
  .write
  .mode('overwrite')
  .format('parquet')
  .save('tmp/spark_datetime/')
)

part-00000-f85e122f-806f-4375-91da-04de38bc0c9c-c000.snappy.parquet.zip

Describe the bug
When a Parquet file contains very low or very high date and timestamp values, this causes trouble:

• Date 0001-01-01 is displayed as blank cell (column "Date_Min)
• Timestamp 0001-01-01 00:00:00.000000 is displayed as blank cell (column "Timestamp_Min)
• Timestamp 9999-12-31 23:59:59.999999 causes an exception (column "Timestamp_Max)
  • System.AggregateException, SystemArgumentOutOfRangeException

Screenshots

Additional context
The problem might be related to https://issues.apache.org/jira/browse/SPARK-31404.
Spark changed calendar between Spark 2.4 and 3.0.

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

Describe the feature you'd like to be added to Parquet Viewer
It would be great to be able to adjust automatically the size of all columns to the size of the data and/or the column name. When opening parquet files with lots of double type columns, you have to adjust individually each column to be able to see the whole number (with the E-# at the end). That way you can easily view column values and compare them between columns.

Adjusting by column contents will make all column sizes to match maximum length of a column value
Adjusting by column names will make all column sizes to match the length of the column name
Adjusting by column names and contents will make all column sizes to match the biggest length of either the column name or the column values

Share why this feature would be a good addition to the utility
It will improve usability and data readability

Screenshots
I will try to put here a proposal for the menu items:

Tools
|  Adjust columns > | Column contents
                    | Column names
                    | Column names and contents

I just found your tool and I think it's great!!

Thank you for the effort!

Best regards,

Carlos

Using Release 2.1 Binary download.

Problem:
I have a parquet file with 30 rows (attached as a file with .parquet extension then zipped
Rows_5036221_5036251.zip
). When I load it in to ParquetViewer, every 2nd row is incorrect. The first column is titled i (columns i,j,k are coordinates). Every 2nd row of the i column displays a 0 instead of the value it's meant to have (j,k are meant to all have the same value). The value it's meant to have in i, is then pushed to the next row. So for example:
I have column of
156,
157,
158,
159,
160

in parquet viewer i get:
156,
0,
157,
0,
158,

For the rest of the columns , some of them have correct values for their row position and some don't
e.g.
row 20 and 21 (count beginning at 1) contain "LMS1" for column V1. This is correct.
but in the same rows S6 contains 0 when it should be "5".

In this file the S columns are a status for a null value replacement string. If the S column is 0, then the V column of the same number should contain a value, but if the S column contains a non 0 value, the V column should be blank.

I've compared the results with the similar tool "BiddataFileViewer" at https://github.com/Eugene-Mark/bigdata-file-viewer and also with my dev's while debugging the parquet file creation (they are using pyarrow to import a csv into a parquet file). Screen shots of results attached.

*Note that BigFileDataViewer has other issues with this file. It errors on the first attempt to load (complaining about incorrect magic numbers), but then loads correctly on a subsequent attempt.

ParquetViewer

BigFileDataViewer

Parquet Viewer Version
What version of Parquet Viewer are you experiencing the issue with?

Where was the parquet file created?
Apache Spark, Hive, Java, C#, pyarrow, etc.

Sample File
Upload a sample file so the issue can be debugged!

Describe the bug
A clear and concise description of what the bug is.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

Currently there is no support for data structures like Map, Lists etc. and an error message is thrown when such files are opened. Can Map datatype be supported?

Parquet Viewer Version
2.3.1

Where was the parquet file created?
no Parquet file involved

Sample File
no Parquet file involved

Describe the bug
Scanned the ParquetViewer.exe with Virustotal and it gave the following issues, see screenshot.
Please check and advise whether this is really an issue or wrong positive.

Screenshots

Additional context

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

Possibly also other metadata. Thanks!

Parquet Viewer Version
What version of Parquet Viewer are you experiencing the issue with?
2.5.1.0

Where was the parquet file created?
Pandas Python - 1.5.3

Sample File
Example.zip

Describe the bug
Cannot open the file, but file can be opened in Python as the column flagged as missing is present. Likely because of a trailing "." (as that is the only mismatch between the file and the bugreport)

Screenshots
If applicable, add screenshots to help explain your problem.

Hello
Is this possible in the new version
Include the ability to search for the same text in multiple Parquet files
Currently, all files must be opened one by one to search for a text
And it will take a lot of time
The ability to search for a text in several Parquet files inside a folder can be really useful
Now I have to open files one by one and wait for loading and searching and then the next file
If you add this feature, it will be really great
With respect

Parquet Viewer Version
ParquetViewer_SelfContained.exe, 2.7.1.0

Where was the parquet file created?
unknown

Sample File
https://huggingface.co/datasets/Gaivoronsky/hh-rlhf-ru-rl/tree/main/data

Describe the bug
When user opens file error is emitted. No stack trace given.

Screenshots
Standard windows dialog saying "Application stopped working"

Additional context
Error emitted when trying to open file:
https://huggingface.co/datasets/Gaivoronsky/hh-rlhf-ru-rl/tree/main/data

[[Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.]

Parquet Viewer Version
Version 2.7.0.0.

Where was the parquet file created?
My own tool

Describe the bug
In the parquet viewer SByte and Bye are mixed up:

your code:

    private static Type ParquetNetTypeToCSharpType(Parquet.Thrift.SchemaElement thriftSchema, Parquet.Schema.DataType type)
        {
            Type columnType;
            switch (type)
            {
---removed some lines---
                case Parquet.Schema.DataType.Byte:
                    columnType = typeof(sbyte);    --> should be typeof(byte) as a .Net byte is unsigned
                    break;
---removed some lines---
                case Parquet.Schema.DataType.SignedByte:
                    columnType = typeof(byte);   --> should be typeof(sbyte) as a .Net byte is unsigned
                    break;
---removed some lines---            }

            return columnType;
        }

atleast fixed solved the problem for my test.

Kind regards,
Maurice,

Parquet Viewer Version
2.5.1.0

Where was the parquet file created?
pyarrow

Describe the bug
I am working with parquet files with many columns (e.g. 30'000).
There is seemingly no efficient way to limit the columns - a deselect all option would be very helpful.
Even for small parquet files with many columns (e.g. 3 MB, 239 rows, 3643 columns), the data load takes long (3-5 min, massive CPU usage, low memory consumption).
For larger files (e.g. 12 MB, 239 rows, about 10'000 columns), the whole system freezes already during the file analysis stage.

Is there any way to work with files with many columns?

Describe the feature you'd like to be added to Parquet Viewer
When the value is null, display the word NULL in italic

Share why this feature would be a good addition to the utility
Currently it is impossible to distinguish between empty string and NULL value

Parquet Viewer Version

2.3.1.41849

Where was the parquet file created?

Pandas -> pyarrow

dfStore.to_parquet(BUILDINGS_OUTPUT_FILE, partition_cols= ["type"])

Sample File

pv_bugdemo.parquet.zip

Describe the bug

A partitioned file that is actually a folder with several subfiles should be supported. This probably involves checking if the "file" is actually a directory then traversing the tree to read the individual constituent files.

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

ParquetViewer is the best parquet file viewer. However, it shows 'timestamp' fields in Unix epoch (or Unix time or POSIX time or Unix timestamp), i.e., the number of seconds that have elapsed since January 1, 1970 (midnight UTC/GMT). For example,

The 'date' field is displayed just fine. It is requested that 'date-time' field be made viewable in human-intelligible format.

Meanwhile, I've posted a request at https://stackoverflow.com/questions/68741520/date-time-timestamp-field-in-parquet-file-shown-as-numbers-in-parquet-file-vie to see if something (like formatting change) could be done in R environment before exporting to parquet format so that it displays 'timestamp' field in human-intelligible format in ParquetViewer.

Describe the feature you'd like to be added to Parquet Viewer
When associating ParquetViewer as default app for parquet files, double clicking the parquet file should open ParquetViewer with the contents of file. Currently, it just opens ParquetViewer with no file contents.

Note: There are no guarantees your feature will be implemented.

Hi,
I am facing exception when trying to open some of parquet files. The exception is:

Hi,

Thanks for making the tool. The time data being displayed appears to omit data finer than minute.
Is it possible to add the higher resolution to the data viewer?

when opening a file, it shows the Time format to many decimal points
but when exporting the same data, it does not "honour" the time format and only saves down to a second,

Would you please kindly maybe one day make the export function also put in the extra decimal figures in the CSV?

hugs and kisses
Amin

Describe the feature you'd like to be added to Parquet Viewer
Provide as much detail as possible.

Share why this feature would be a good addition to the utility
Will it improve usability, reliability or some other aspect?

Screenshots
Any screenshots describing how the feature would look is a plus.

Note: There are no guarantees your feature will be implemented.

Parquet files are nowadays often saved with ZSTD compression since its decompression speed and hence reading speed is very significantly faster than GZIP standard which is currently supported by ParquetViewer.

We regularly use Apache Arrow and plan to use ZSTD compression instead of GZIP as documented in the webpage https://arrow.apache.org/docs/r/reference/write_parquet.html. It would be great if we could continue to use ParquetViewer.

Currently attempting to open ZSTD compressed parquet file results in an error as shown in this screenshot:

It is requested that support for ZSTD compression be incorporated in future versions of ParquetViewer.

Version: 2.2.7625.41329
Parquet file created by pyarrow

Sample File (Note that the data in this file is junk data so in this case the values are mostly meaningless) :
NewRows_1_30_2009cols.zip

Problem:
When I try to load a parquet file that contains > 4000 columns I receive an error: " Sum of the columns FillWeight values cannot exceed 65535" when it tries to add columns to the DataGridView forms object.
The Grid view seems to load ~656 of my 4666 columns into the header row within the grid, before it throws the error and then ceases to load anything else into the grid.

To replicate:

1. Try to open the attached parquet file.

I have successfully been able to open the file with another parquet viewer (within an IDE) so it seems that the parquet file is valid.

Hi,

I would like to understand what are the special characters used for path separator as I am getting the below error? Attaching the column list for your reference. Kindly assist.

Lead_Test1.zip

Parquet Viewer Version
2.7.0.3

Where was the parquet file created?
C#

Sample File
NullableGuid.zip

Describe the bug
Program is unable to open the file if it contains a nullable guid column.

Screenshots

Additional context
The nullable guid issue was resolved here https://github.com/aloneguid/parquet-dotnet/issues so everything needed is probably just the Parquet.Net library update.

Describe the feature you'd like to be added to Parquet Viewer
Support Decimal format so that it shows proper text instead

For a field of decimal like below
{
"Field_id": 0,
"Name": "CD_ID",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": "LogicalType(DECIMAL: DecimalType(, Scale: 12, Precision: 38))",
"Scale": 12,
"Precision": 38,
"Repetition_type": "OPTIONAL",
"Converted_type": "DECIMAL"
},

GUI shows:
Weird characters

For these characters even copy-paste is not possible

Additional Feature/Bug Requerst:
Integer adds thousand seperator by default, it should not add it by default as settings do not persist and need to uncheck each time open a file.

Thanks.

I've found some data named like XXXXX_UserData_0001.parquet up to XXXXX_UserData_0005.parquet
When opening any of the files I get this error:

Is it possible to open any of these files?

Parquet Viewer Version
2.3.0.40676

Where was the parquet file created?
AWS EMR > Apache Spark

Sample File
cant do, its 22MB with sensitive information and i dont have control of producer.

Describe the bug
Parquet contains a field which is a nested table (array of strings).. so I guess ParquetViewer is unable to open these type of parquets.
I cant share parquet data file, but im attaching detailed error popup screenshot - basically its refering to the nested field.

Screenshots

Additional context

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

Parquet Viewer Version
2.5.1

Where was the parquet file created?
pyarrow

Sample File
test_file_20230120_common_ts.zip

Describe the bug
The Parquet Viewer is not able to open a file that contains columns of LIST type. In the attached file "object_data__obj_angles" is a column of LIST type but as you see in the screenshot it complains that this column doesn't exist (don't know why). However, v2.4 mentioned at #33 (comment) is able to load this file without issues.

Screenshots
v2.5.1

v2.4

Runs out of memory when opening files larger than 10-20 MB. It uses massive amounts of memory and becomes unusable with anything but the smallest of files.