Comments (13)
I've created a branch ( https://github.com/noyez/atuin/tree/noyez_zsh_histdb_import ). It compiles, and the tests run, but i haven't got it to run in practice b/c of async code. Specifically, the import traits are synchronous, but sqlx-rs is asynchronous, but i can't figure out how to call async code in sync function -- which itself is call from async. So the call stack is:
- async fn
- sync fn
- block_on ( async {} )
- ... and BOOM, this is not allowed.
The resulting error: Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.'
I do struggle w/ rust's async code, so if you have any ideas, i'm all ears. I'll revisit this soon, but i need to move on for now.
from atuin.
@norpol I'm just looking at the code, and i realize that i had protections against invalid strings only in the command text, but NOT in the directory -- which is the dir
error that you're seeing -- name NOR in the hostname. I'll submit a patch to fix that. Would you be able to test code from my fork if i push an update?
from atuin.
As a user of zsh-histdb, i'd love to see this (esp as i've found histdb to be a little slow w/ a large history). As a rust developer, i could probably submit a PR for this. Do you have any vision on how you'd like to see done? If not, i can just dive in.
from atuin.
If you know of a good way of detecting a regular ZSH install vs a ZSH + histdb setup, then we can have a histdb.rs file that manages the extraction
Although thinking about it, the current import code (using their Read
ers and Iterator
s is a bit obtuse and fugly) would be hard to interopt with sqlite... I might rewrite it to be considerably simpler. I first rewrote this code when I started getting confident with Rust so I felt like it was super smart... 😂
from atuin.
You're right that sqlx traits don't seem to lend itself well to the Read + Seek + Iterator
implementations, at least as far as i can tell, i've never used sqlx before. Either way, i'll get something working, i hope.
from atuin.
If needed, I can refactor our import code to be less sophisticated. Simple is better, I wish I realised that a year ago 🤦
from atuin.
There are already two PRs merged that relate to this ticket: #393 and #395.
I managed to successfully import, though there was an error message with some error dir with id 22 doesn't contain UTF-8
.
Interestingly checking the sqlite-file, ID 22 was a valid string.
I managed to find the faulty directory field by running this code in a Python console.
import sqlite3
con = sqlite3.connect("zsh-history.db")
cur = con.cursor()
for row in cur.execute("SELECT * FROM places ORDER BY id"):
print(row)
Which would raise an error, so I fixed that field value and afterwards import worked.
---------------------------------------------------------------------------
OperationalError Traceback (most recent call last)
Input In [1], in <cell line: 4>()
2 con = sqlite3.connect("zsh-history.db")
3 cur = con.cursor()
----> 4 for row in cur.execute("SELECT * FROM places ORDER BY id"):
5 print(row)
OperationalError: Could not decode to UTF-8 column 'dir' with text '/mnt/waid/pl/static/cd_rips/2012091712033700�'
I'm really a database noob, otherwise I'd provide some more help than that.
from atuin.
@norpol Do you have a sample sqlite db file with that non utf-8 directory that i could use as a test?
from atuin.
@noyez I'm trying to replicate the faulty database string, though if I recreate a database the error is slightly different.
Atuin
======================
🌍
🐘🐘🐘🐘
🐢
======================
Importing history...
Importing history from zsh_histdb
Error: error occurred while decoding column "dir": mismatched types; Rust type `alloc::string::String` (as SQL type `TEXT`) is not compatible with SQL type `BLOB`
Caused by:
mismatched types; Rust type `alloc::string::String` (as SQL type `TEXT`) is not compatible with SQL type `BLOB`
Location:
atuin-client/src/import/zsh_histdb.rs:92:40
is the new error, the previous one is
Atuin
======================
🌍
🐘🐘🐘🐘
🐢
======================
Importing history...
Importing history from zsh_histdb
Error: error occurred while decoding column "dir": incomplete utf-8 byte sequence from index 44
Caused by:
incomplete utf-8 byte sequence from index 44
Location:
atuin-client/src/import/zsh_histdb.rs:92:40
though index 44 is a regular string (could that be some sort of async problem?).
Attached is my recreated database, the main one is really big (1.5 Million records) and is a bit sensitive.
I have no idea how zsh-histdb ended up with that invalid entry.
from atuin.
@norpol If you are able, try the following branch : https://github.com/noyez/atuin/tree/noyez_fix_dir_hostname_utf8
I update the dir
and hostnames
to be more utf-8 friendly.
from atuin.
Awesome, works great! Thank you so much.
atuin search -c /mnt/waid/pl/static/cd_rips/2012091712033700�
atuin search -c /mnt/waid/pl/static/cd_rips/2012091712033700
# 2018-12-02 14:45:22 less extra_metadata.txt 0s
from atuin.
Should be addressed by PR #393 and PR #430
Can this be closed?
from atuin.
Yup it totally can! Thanks again for the work 🚀
from atuin.
Related Issues (20)
- Gitpod with Atuin Blog Post - Any chance for a review? 🤝 HOT 4
- Default OS X `db_path` does not match the comment in default `config.toml` HOT 2
- Self-hosting atuin with apache as reverse proxy
- Make install.sh check for existing configurations
- Make server database agnostic HOT 6
- Feature Request: customize up-arrow options based on current context (cwd, current buffer, etc)
- Windows support HOT 1
- Tracking tty in the database
- ability to add comments/descriptions to commands
- Raspberry Pi version of the self hosting server Docker image HOT 1
- atuin on ARM64 HOT 2
- docs: Commands page only lists the import command
- Unable to communicate with self-hosted Server
- Customize _atuin_search_widget
- make server sync async HOT 6
- Update install.sh script for aarch64 assets
- `atuin sync --force` does not download everything HOT 1
- Highlight `atuin search` query matches with color
- Latest nushell release breaks atuin init config generation HOT 2
- colors should be configurable HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from atuin.