Comments (5)
Relevant code comment:
datasette-upload-csvs/datasette_upload_csvs/app.py
Lines 25 to 34 in 54aa31f
from datasette-upload-csvs.
I think the easiest way to do this is with the new Datasette write queue mechanism. It's not ideal for long operations but I think it's OK in this particular case to lock the database write connection for the duration of processing the CSV.
What is really like is to be able to poll for progress with the conversion and show another progress bar.
Problem there is that I don't know how many rows total are going to be imported - at least not without parking the CSV file through a first time just to count rows.
But... maybe I can use the file size trick? If I know the size of the file in bytes I can have the CSV reader work against a wrapper class which counts the number of bytes that have been read.
I would still need a mechanism for the write thread to make the progress available to be polled. How about it writes to a temporary table every few thousand bytes or so?
from datasette-upload-csvs.
I would have to be committing as I go, but I want to do that with the rows I'm inserting anyway.
from datasette-upload-csvs.
I'm feeling pretty good about this approach - opening a pull request.
from datasette-upload-csvs.
Fixed by #7
from datasette-upload-csvs.
Related Issues (20)
- Require upload-csvs permission HOT 4
- Upgrade for compatibility with Datasette 0.44+ HOT 5
- Rename /-/upload-csv to /-/upload-csvs and set up a redirect
- Switch CI to GitHub Actions
- Make upload-csvs more discoverable by adding a link to Datasette's top level nav HOT 4
- Error if CSV filename contains spaces HOT 1
- UI to select name of table at time of upload HOT 6
- If a database has no tables, add a prompt to upload a CSV
- Defaults to uploading to _internal which is wrong HOT 1
- Tests fail against latest dependencies
- URL on final confirmation page needs to use tilde encoding HOT 1
- Should use block=False HOT 1
- UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 105: invalid start byte HOT 2
- Don't hide errors in imports from the user HOT 3
- ascii codec Error uploading CSV HOT 4
- Detect column types in uploaded data automatically HOT 1
- Template extends extra_head without super()
- Allow selection of the database to add the table to HOT 4
- If table already exists, create a new table with _2
- Don't crash if there are no writable databases available HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datasette-upload-csvs.