Comments (3)
Had this discussion in slack, but putting here as well:
Looking. back at v 1.2, I think that what happens is that the first query does some sorting, but it doesn't really matter because the second query is the one that goes at gets all the names for the curies found in query 1:
7:41
{
"query": f"({curie_filter}) AND ({name_filters})",
"limit": 1000000,
"sort": "length ASC",
"fields": "curie,name",
}
7:42
So now you have this sorted by length of the name, but with the curie on mutliple rows and all the curies intermixed together.
So then the results get parsed out like this:
output = defaultdict(list)
for doc in response.json()["response"]["docs"]:
output[doc["curie"]].append(doc["name"])
(edited)
7:44
And this output dict is magically preserving the order. curie with the shortest doc gets added first. Now the sorting of the first query does actually matter, because you want to get the curies that have the shortest name first.
7:45
So for us, either we need to add another parameter on the doc which is the length of the shortest element in the synonym list, or we need a way to figure it out in the query
Chris Bizon
8:02 PM
I think we could do it on the query with https://solr.apache.org/guide/7_6/function-queries.html#sort-by-function, we just need to figure out the right function. I think it's the length of the first synonym
from nameresolution.
So for us, either we need to add another parameter on the doc which is the length of the shortest element in the synonym list, or we need a way to figure it out in the query
I’ve been looking in https://solr.apache.org/guide/7_6/function-queries.html and I don’t see a length function. I tried Googling for it and couldn’t find anything either (apart from ChatGPT which helpfully suggested the field_length() function that nobody else has heard of). I think you’re right that probably the best short-term solution is to reinstate the length field in Solr that we used to have previously. I’m going to call this shortest_name_length for now so it’s clear what it means, but I wonder if we should try to factor the information content value in here as well. I should be able to implement the shortest_name_length field in Babel pretty quickly, so I'll take back ownership of this ticket.
from nameresolution.
This has now been significantly improved with CURIE suffix sorting, and it's working well enough that is what is being used by Translator UI. Closing.
from nameresolution.
Related Issues (20)
- Duplicated results
- Rat v rats
- Rename to Name Resolver to be consistent
- Error on reverse lookup for PUBCHEM.COMPOUND:107526
- Searching for nothing actually searches for "OR"
- Figure out some way to document the dataUrl being used for a NameLookup instance
- Update NameLookup Jupyter Notebook and write some Translator-specific documentation
- We don't do a good job on depression
- Rename `/reverse_lookup?curies=X` to `/synonyms?preferred_curie=X`
- How do you get synonyms JSON file from the download TSV files? HOT 2
- Add LICENSE to NameRes HOT 1
- Lookup for "diabetes type ..." takes more than a minute, and gives a wrong result HOT 1
- Can we incorporate hierarchy information somehow?
- "Bone" works well when autocomplete=true but breaks when autocomplete=false
- Figure out how to reduce boosts on repeated terms
- Brand Names? ®
- Hyphen processing still a bit dodgy, possibly because of our choice of tokenizer
- Upgrade Solr to 9.5
- Searching for BRCA1 in autocomplete=true mode gives a lot of bad matches
- Ideal memory requirement for NameRes on Translator HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nameresolution.