githubocto / flat-viewer Goto Github PK

Would it be possible to establish some way to configure the file location for dates and values?

Hi Team,

Hi, I have created Open In flat viewer Extension. when clicked on GitHub repo, will open that Flat Viewer. I will be publishing soon on the Chrome Web store. The extension simply provides a URL redirect: it reads the current Github repo URL and opens a new tab at https://flatgithub.com/ with the user, repository.

Link: https://github.com/misalraj/open_in_flatViewer.

Do check this. Feedback needed.

[cross-posting https://github.com/githubocto/flat/discussions/15 in case this is a better place to discuss this]

🧙 The killer feature for me would be to be able to graph the evolution of values over time (i.e. across commits)

i.e. I'm using a Flat action to scrape the number of "things" from an API. I want to graph number of "things" over time 📈

Let me know if I can help!

I have set up a repo to collect holdings of ARK funds using flat

When I sort by market value column, I notice that the sorting is done in pure alphabetical sort order and the data is not sorted by the actual value.

Link to the data

d3-dsv supports both CSV and TSV file formats. I'd love to see support for .tsv files in addition to .csv.

It would be handy to extend this:

since Get repository content would use base64-encoded when size less than 1MB, but when the size exceed 1MB, the encoded payload became empty ''

{
  content: ""
  download_url: "https://raw.githubusercontent.com/zhang699/translations/b11706c2aff98a97988faf9d38a873565053d4c7/xxx.json?token=ABIL4IH7KXMQWJ2IS5LK7TTC4O3FG"
  encoding: "none"
}

one of solution I think we can use download_url to fetch the raw

@duckdb provides an in-process database query engine for common tabular file formats, including csv and JSON. A wasm distribution is available on npm, and can be used in JSONDetail to provide the user with an input field to run queries against a selected file and commit.

Richer support (eg, joins between multiple files and/or commits) could be addressed in subsequent issues.

I'm experimenting with a large CSV file (> 100 MB) so I had to use LFS. Everything went well during the Post Fetch Data step but the flat viewer is not able to visualize files stored with Git LFS.

url: https://flatgithub.com/rmaceissoft/flat-vaccination-peru-example?filename=registro_vacunacion.csv

Is there something I'm missing here? Thanks in advance

Sorting by numbers seems to get confused if there are blank values as well.

https://flatgithub.com/labordata/nlrb-voluntary-recognitions?filename=voluntary_recognitions.csv&filters=&sha=d9d936a07f6d8084063e1321202bec3fb087a07f&sort=Number%20of%20Employees%2Cdesc&stickyColumnName=VR%20Case%20Number

Use case: It is nice to be able to visualise some files in a development branch

Example: this is what I get when hitting one of my repos:

Expected: you skip .json configs you can't display instead of being greeted with an error when viewing web app/vscode repos.

For non-technical users, Flat Viewer as hosted on flatgithub.com has been effective for exploring tabular data hosted on GitHub repositories. A team could create a GitHub organization containing Flat Data repositories, each one set with the corresponding flatgithub.com URL, and non-technical users can then be directed to explore these repositories on github.com/<org-name>.

However, given how GitHub's UI is tailored mostly for developers, non-technical users may be unfamiliar (and possibly unwilling to be familiar) with reading through a list of repositories on a GitHub org page, and would hence be intimidated from exploring further.

To help such users, Flat Viewer could implement the listing of repositories that contain Flat Data (identified by eg being tagged with flat-data) within a given GitHub organization. Users can then look at a GitHub org's flat-data repos, through an interface specifically for exploring Flat Data, via a URL like flatgithub.com/<org-name>.

Hi!

I'm trying to use flat viewer on a private repo (via local storage token).
Flat viewer can access files/commits, but I get an error in the data viewer pane: "Oh no, we couldn't load x/y.csv for some reason".

A 404 is logged to console because looking up the pattern:
https://raw.githubusercontent.com/${owner}/${name}/${sha}/${filename}
gives a 404 for my private repo.

Not sure whats wrong - should there be a token included in the URL (for private repos) for instance? (github links to raw data in a private repo using ?token=...)

Files downloaded from Flat Viewer have the name flat-ui__data <timestamp>.extension. . Would it be possible to use flat-ui_<timestamp>_<filename> or similar naming schemes to make it easy to identify them?

The current flat viewer allows for easy filtering of single values.

However, I would like to know how to filter for multiple values in the same column.

For example, you have a column animals with values: cat, dog, penguin, etc.

How to filter for both cat AND dog?

@Wattenberger can you please tag your releases and list what's in those release bumps?

https://github.com/githubocto/flat-viewer/releases

Thanks!

Hi!

Great tool!

I know it would require new auth plumbing but I think it would be even better if I the viewer could work for private repos!

I think this is worth considering 😃

UI/homepage leaves lots of space for a sticker, and repo gives a lot of helpful context about the tool which the UI/homepage doesn't (being nice and concise themselves)

I love this project! We used a flat-like data structure for @Karuna2020, our COVID-19 nonprofit, and you can for example see a list of all our donors and the amounts using Flat Viewer: https://flatgithub.com/Karuna2020/open-data?filename=donations.json

I'd also love to see YAML support, for example for this file: https://raw.githubusercontent.com/AnandChowdhary/life-data/master/heart-rate.yml (currently, YAML is not supported: https://flatgithub.com/AnandChowdhary/life-data?filename=heart-rate.yml).

Going to https://flatgithub.com/smittyvb/smitop.com gives a 404, while going to the main page and entering Smittyvb and smitop.com works as expected.

Hi,
when I apply a filter I see something like:

?filename=dati%2Fsomministrazioni-vaccini-latest.csv
&filters=fornitore%3DJanssen
&sha=9e6a9f4a150e6b96ef52827e0e2692274d68701e
&sort=data_somministrazione%2Cdesc
&stickyColumnName=data_somministrazione

For some I can deduce the meaning. Is there a page with documentation?
What's a stickyColumnName, what is it for?
What's the date format I must use for date fields? etc.

Thank you

Thank you for Flat Viewer! I love this project.

This is a feature request, and I'm not sure how feasible it would be to implement. But we are using Flat Viewer to host and share datasets for the Post45 Data Collective, a repository for literary and cultural data from 1945 to the present. We would love to have the option of adding a title or perhaps some brief documentation to the top of the Flat Viewer page — perhaps this could be customized by including information in a certain way in a README file in each folder in the repository?

Just to give a concrete example, here's what our current framework for hosting the datasets looks like:

vs

We would basically like to add the title, author, and contextual information.

Using the flat data repo linked from the documentation (https://githubnext.com/projects/flat-data#part-iii-visualizing-our-data-for-easy-sharing-optional)
Sample filter:
https://flatgithub.com/the-pudding/data?filename=boybands%2Fboys.csv&filters=name%3DSlim&sha=9b4e89ed178e11bf70a9b3d8ffa57ac2ea181b96&sort=band%2Casc&stickyColumnName=band
The filter text is "slim": 3 records are shown but only the first record contains this value

This would make many GIS devs and experts very happy.

See this data wrangling ticket for more context info: RandomFractals/vscode-leaflet#9

and sensible defaults and conventions that could be used for converting geo data to flat data files: https://github.com/keplergl/kepler.gl/blob/master/docs/user-guides/b-kepler-gl-workflow/a-add-data-to-the-map.md#supported-file-formats

It looks like we can't have \r\n as a new line character in CSV and should use \n.

Looking at data fetcher in this repo, I found https://github.com/githubocto/flat-viewer/blob/main/src/api/index.ts#L142 which parses CSV using https://github.com/githubocto/flat-viewer/blob/main/src/api/index.ts#L7

So it looks like that parser is here https://github.com/d3/d3-dsv/blob/main/src/dsv.js

Fantastic work on Flat, GitHub OCTO! 🐙

I've used Flat Data to grab the 7-day asteroid forecast from NASA's Near Earth Object API.

Opening Flat Viewer on nasa-asteroids-postprocessed.csv on desktop, everything looks great: https://flatgithub.com/nsgrantham/nasa-asteroids

However, when opened on mobile (iOS 14.4.2 using Safari and Chrome), there appears to be a problem with the datetime columns close_approach_datetime and orbit_determination_datetime.

These columns are of the form yyyy-MM-dd HH:mm and yyyy-MM-dd HH:mm:ss respectively.

There is no problem with the regular date columns of the form yyyy-MM-dd.

Example: https://github.com/RandomFractals/vscode-data-preview/tree/master/data/arrow

Apache Arrow ecosystem page: https://arrow.apache.org/powered_by/

Sample preview of that arrow data file:

would be nice to view it via flatgithub url.

Thanks!

thanks for the great tool!

flatgithub.com is not recognizing a field that contains only integers and empty value as a number:

https://flatgithub.com/labordata/nlrb-voluntary-recognitions?filename=voluntary_recognitions.csv&sha=366cf624c4bc0f870e1aa41fcd6a3425214843e2

I've verified that there are no sneaky thousand separators or anything else.

The column in question is the "Number of employees" field