Comments (5)
from wn.
from wn.
Thanks! I'm concerned how it would scale when we consider more than just hypernym relations, but it's worth considering.
from wn.
Some updates to this:
Here is a query that recursively finds relations within the same lexicon:
WITH RECURSIVE
rt(rowid) AS
(SELECT rowid FROM relation_types WHERE type IN (?,...)), -- hypernym, instance_hypernym, etc.
tgt(rowid, path, depth) AS
(SELECT ?, ">"||?||",", 0 -- synset rowid, twice
UNION ALL
SELECT rels.target_rowid,
tgt.path||rels.rowid||">"||rels.target_rowid||",",
tgt.depth+1
FROM synset_relations AS rels
JOIN tgt ON rels.source_rowid = tgt.rowid
WHERE rels.lexicon_rowid IN (?,...) -- target lexicons
AND rels.type_rowid IN rt
AND tgt.depth < ? -- depth limit
AND NOT instr(tgt.path, ">"||rels.target_rowid||",")
ORDER BY 3 DESC
LIMIT 1000000
)
SELECT rowid, path, depth FROM tgt;
Despite the string operations for avoiding cycles (see https://stackoverflow.com/q/66866542/1441112), it is fairly fast. And not only does it return all transitive relations, but also the path to get there and the depth, and the search can be limited by depth. However, it can easily become slow by broadening the query to, e.g., all relations, or even just hypernym
and hyponym
. For this reason, the (row) LIMIT
is important and also an appropriate depth limit.
More importantly, it does not do expanded (inter-lexicon) search. I tried to replicate the two-step process currently used (first intra-lexicon relations, then expanded ones) by adding this second recursive part:
UNION ALL
SELECT ss.rowid, tgt2.path||exp.rowid||">"||ss.rowid||",", tgt2.depth+1
FROM tgt AS tgt2
JOIN synsets AS ss1 ON tgt2.rowid = ss1.rowid
JOIN synsets AS ss2 ON ss2.ili_rowid = ss1.ili_rowid
JOIN synset_relations AS exp ON exp.source_rowid = ss2.rowid
JOIN synsets AS ss3 ON ss3.rowid = exp.target_rowid
JOIN synsets AS ss ON ss.ili_rowid = ss3.ili_rowid
WHERE exp.lexicon_rowid IN (?,...) -- expand lexicons
AND exp.type_rowid IN rt
AND tgt2.depth < ? -- depth limit, again
AND NOT instr(tgt2.path, ">"||ss.rowid||",")
AND ss.lexicon_rowid IN (?,...) -- target lexicons, again
ORDER BY 3 DESC
LIMIT 1000000
But recursive queries with more than one recursive part were not added to SQLite until version 3.34, and Python's version is at 3.31, so I'm not sure if the above even works. When I try something like the above as the only recursive query, it works but is very slow. I suspect that joining synsets
4 times at every recursive step is slowing things down. These 4 are necessary because we start with a synset rowid and relations are between synset rowids, but the expansion happens in ILI, so:
- Join to get ILI of source synset
- Join to get other synsets with that ILI
- Join to get ILIs of matching relations' target synsets
- Join to get synsets in target lexicon with those ILIs
This could perhaps be simpler if ILIs were required by all synsets. As it is, imagine if the PWN did not have ILIs: the relations still work between its synsets. Since they are optional we have to check, at every relation traversal, if there are relations within the current lexicon as well as the expanded ones available via ILI traversals.
from wn.
Changing the label to "maintenance" as this doesn't change functionality, only (potentially) performance.
from wn.
Related Issues (20)
- Document "default mode" queries
- Missing Spanish definitions HOT 3
- antonyms in languages other than German and English HOT 5
- Support for PTB and Universal POS tags HOT 6
- If you create an entry with an ILIDefinition, but ill.id='' you lose the definition HOT 2
- Tracing back 'inferred' synsets to their reference lexicons HOT 3
- Stumped by multilingual relation traversal HOT 7
- Synset.relations() for some lexicons uses synset id as relation name
- Update Python versions HOT 2
- Is there any mapping between different English wordnet? HOT 4
- synset.relations fails with a KeyError HOT 3
- Merged synsets are lost in translation HOT 8
- SQLite objects created in a thread can only be used in that same thread HOT 3
- Add OEWN 2022 to index HOT 2
- Addition to NLTK migration guide w.r.t. offsets HOT 4
- Add a `conda` install option for `wn` on conda-forge channel HOT 5
- Allow contributors to self-assign issues with GitHub workflow HOT 3
- pyproject.toml: Fix ruff rules in tool.ruff.ignore
- Update Python versions, 3.8 to 3.12
- Add OEWN 2023 to index
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wn.