sudheer-ranga / aliexpress-product-scraper Goto Github PK
View Code? Open in Web Editor NEWGet Aliexpress product details as a json response including feedbacks, variants, shipping info, description, images, etc.,
License: MIT License
Get Aliexpress product details as a json response including feedbacks, variants, shipping info, description, images, etc.,
License: MIT License
Hi,
i'm currently facing an issue due an outdated dependencie (agent-base) :
HTTPError: Response code 401 (Unauthorized)
at Request.<anonymous> (/home/adrien/Documents/dropshapp/node_modules/got/dist/source/as-promise/index.js:117:42)
at processTicksAndRejections (internal/process/task_queues.js:97:5) {
code: undefined,
timings: {
start: 1600086737470,
socket: 1600086737474,
lookup: 1600086737737,
connect: 1600086737745,
secureConnect: 1600086737764,
upload: 1600086737765,
response: 1600086738120,
end: 1600086738133,
error: undefined,
abort: undefined,
phases: {
wait: 4,
dns: 263,
tcp: 8,
tls: 19,
request: 1,
firstByte: 355,
download: 13,
total: 663
}
}
}
According to agent-base, the error has been fixed in version >= 5, could you please update your dependencies ?
(node:6408) [DEP0139] DeprecationWarning: Calling process.umask() with no arguments is prone to race conditions and is a potential security vulnerability.
└─┬ [email protected]
└─┬ [email protected]
└─┬ [email protected]
└── [email protected]
Can i get prices of item when ship is for example only from Poland?
Hello.
Sometimes this lib throws an exception:TypeError: Cannot read property 'descriptionUrl' of undefined
OS: Ubuntu 22.04
Node Version: v14.17.6
Lib version: ^1.0.6
Hi, runParams
variable is missed. Could you commit working version of the code?
const aliExpressData = await page.evaluate(() => runParams);
I was wondering if there's a way to force the currency type, for example USD. Right now the prices are in my local currency.
Hello!, it would be good to add the pageSize in the options object, because I see that it has a default value of 20, and it would be good to be able to change it from the options. Thanks!!
and checking the filterReviewsBy I see that these would be the valid options
{
reviewsCount: 20,
filterReviewsBy: 'all' | 1 | 2 | 3 | 4 | 5, //filterCode values
puppeteerOptions: {},
pageSizeReview: number //suggested
}
Seeing that there are more features added that need input parameters, I think we should refactor the scrape function as such:
type AliExpressScrapeOptions = {
puppeteerOptions?: LaunchOptions,
feedbackLimit?: Number,
shippingCountryNames?: String[] // should be typed with the list of country codes so it's clear which standard is being used
}
function scrape(productId: number, options: AliExpressScrapeOptions = {})
Hello, I don't know how to choose the default delivery address, because I sell in France and I want to follow prices only for delivery to France.
Is there any solution to this because I can't choose delevry to France in the url by get method.
Thanks for the script
actually not working
I needed some types for typescript and thought i would share my aliexpress-product-scraper.d.ts file here:
type Product = {
title: string;
categoryId: number;
productId: number;
quantity: {
total: number;
available: number;
};
description: string;
orders: number;
storeInfo: {
name: string;
logo: string;
companyId: number;
storeNumber: number;
isTopRated: boolean;
hasPayPalAccount: boolean;
ratingCount: number;
rating: float;
}
ratings: {
totalStar: number;
averageStar: number;
totalStartCount: number;
fiveStarCount: number;
fourStarCount: number;
threeStarCount: number;
twoStarCount: number;
oneStarCount: number;
};
images: string[];
reviews: [{
anonymous: boolean;
name: string;
displayName: string;
gender: string;
country: string;
rating: number;
info: string;
date: string;
content: string;
photos: string[];
thumbnails: string[];
}];
variants: {
options: [
{
id: number;
name: string;
values: [{
id: number;
name: string;
displayName: string;
image: string;
}]
}
]
}
prices : [
{
skuId: number;
optionValueIds: number[];
availableQuantity: number;
originalPrice: {
currency: string;
formatedAmount: string;
value: float;
}
salePrice: {
currency: string;
formatedAmount: string;
value: floar;
}
}];
specs: [
{
attrValue: string;
attrName: string;
}
];
currencyInfo: {
baseCurrencyCode: string;
enableTransaction: boolean;
currencySymbol: string;
symbolFront: false
currencyRate: float;
baseSymbolFront: Boolean;
currencyCode: string;
baseCurrencySymbol: string;
multiCurrency: boolean;
}
originalPrice: {
min : {
currency: string;
formatedAmount: string;
value: float;
}
max : {
currency: string;
formatedAmount: string;
value: float;
}
salePrice: {
min: {
currency: string;
formatedAmount: string;
value: float;
}
max: {
currency: string;
formatedAmount: string;
value: float;
}
}
}
shipping: [
{
deliveryProviderName: string;
tracking: string;
provider: string;
company: string;
deliveryInfo: {
min: number;
max: number;
}
shippingInfo: {
from: string;
fromCode: string;
to: string;
toCode: string;
fees: string;
displayAmount: string;
displayCurrency: string;
}
warehouseType: string;
}
]
};
enum filterReviewsBy {
1 = 1,
2 = 2,
3 = 3,
4 = 4,
5 = 5,
}
type Options = {
reviewsCount?: number;
filterReviewsBy?: filterReviewsBy
puppeteerOptions?: {}
}
declare module 'aliexpress-product-scraper' {
export default function scrape(id: string, options?: Options): Promise<Product>;
`}`
Why is there a limit for review fetching? like 10 pages or 100 reviews?
I'm experiencing this when I bundle it using Browserify and use it to HTML:
bundle.alisavetools.js:183238 Uncaught TypeError: Cannot read properties of undefined (reading 'bind')
at 1777.../common/Debug.js (bundle.alisavetools.js:183238:50)
at o (bundle.alisavetools.js:1:265)
at bundle.alisavetools.js:1:316
at 1782.../common/Puppeteer.js (bundle.alisavetools.js:184562:29)
at o (bundle.alisavetools.js:1:265)
at bundle.alisavetools.js:1:316
at Object. (bundle.alisavetools.js:183067:24)
at Object. (bundle.alisavetools.js:183091:4)
at 1775../node/Puppeteer.js (bundle.alisavetools.js:183091:17)
at o (bundle.alisavetools.js:1:265)
So we have a problem because data container which is called runParams unavailable anymore. Do you have any ideas from where we can get data? except scraping HTML of course)
When we run your example code, we get Json response with properties not covered using quote.
Which is invalid json.
{
title: 'Floral Print Dress Loose Casual Dress Women Summer Robe Boho Chic Loose Dress Fashion Long Sleeve Short mini Dress Elegant',
categoryId: 200000347,
productId: 3256803598798951,
totalAvailableQuantity: 524,
description: '
\n' +
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
'
\n' +
\n' +
Hello, I'm trying to run the sample code, with the npm package installed and downloading this repo, and I get the following error: "runParams is not defined".
The problem seems to be in this line: https://github.com/sudheer-ranga/aliexpress-product-scraper/blob/master/src/aliexpressProductScraper.js#L14
Any idea? Thanks in advance
Full log:
(node:32828) UnhandledPromiseRejectionWarning: Error: Evaluation failed: ReferenceError: runParams is not defined at __puppeteer_evaluation_script__:1:8 at ExecutionContext._evaluateInternal (C:\Users\iicc\Desktop\xxxx\xxxx\node_modules\puppeteer\lib\ExecutionContext.js:122:13) at process._tickCallback (internal/process/next_tick.js:68:7) -- ASYNC -- at ExecutionContext.<anonymous> (C:\Users\iicc\Desktop\xxxx\xxxx\node_modules\puppeteer\lib\helper.js:111:15) at DOMWorld.evaluate (C:\Users\iicc\Desktop\xxxx\xxxx\node_modules\puppeteer\lib\DOMWorld.js:112:20) at process._tickCallback (internal/process/next_tick.js:68:7) -- ASYNC -- at Frame.<anonymous> (C:\Users\iicc\Desktop\xxxx\xxxx\node_modules\puppeteer\lib\helper.js:111:15) at Page.evaluate (C:\Users\iicc\Desktop\xxxx\xxxx\node_modules\puppeteer\lib\Page.js:833:43) at Page.<anonymous> (C:\Users\iicc\Desktop\xxxx\xxxx\node_modules\puppeteer\lib\helper.js:112:23) at AliexpressProductScraper (C:\Users\iicc\Desktop\xxxx\xxxx\node_modules\aliexpress-product-scraper\src\aliexpressProductScraper.js:14:37) at process._tickCallback (internal/process/next_tick.js:68:7) (node:32828) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1) (node:32828) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Hi sudheer, can you please add a repository with closed environment and all the node modules, it keeps giving new errors for puppetteer 😭
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.