Comments (2)
jayzeng
commented on July 28, 2024
@Feelthewind that's correct, it is used to construct a unique document id (see https://github.com/knockrentals/scrapy-elasticsearch/blob/master/scrapyelasticsearch/scrapyelasticsearch.py#L81-L85).
If you want to use url as the id, and assuming you have 'url' defined in your model in items.py, then it should should be simple as:
ELASTICSEARCH_UNIQ_KEY = 'url'
I have an example project https://github.com/jayzeng/dirbot illustrates how to use this plugin. Let me know if you have more questions.
from scrapy-elasticsearch.
jayzeng
commented on July 28, 2024
@Feelthewind feel free to re-open the issue if you are confused or need any help
from scrapy-elasticsearch.
Related Issues (20)
- Unable to post to ssl endpoint with custom CA HOT 7
- Content-Type required, Elasticsearch 6.x HOT 1
- Text field always gets ignore_above keyword HOT 1
- Unique key is tuple if using items HOT 1
- Removal of _type requirement
- BulkIndexError HOT 1
- Missing header information for ElasticSearch 6.2 HOT 2
- got an unexpected keyword argument 'headers' HOT 3
- is it possible update item if this item with id exists? HOT 2
- Suggest making item_id = hashlib.sha1(unique_key).hexdigest() optional
- Scrapy logging show UnicodeDecodeError HOT 1
- deleted
- Suggestion of setting '_index', '_source' and other parameters directly in parser
- ElasticsearchDeprecationWarning: [types removal] Specifying types in bulk requests is deprecated.
- Please support Scrapy 2.4.1 HOT 1
- I downloaded the package but I still get a module not found.
- Cluster name and Cluster node field not available
- elasticsearch.helpers.BulkIndexError after long time of scraping HOT 5
- Doesn't support scrapy dataclass item
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scrapy-elasticsearch.