LCSC: Large categories only have the first 10k items scraped #2

Open
opened 3 years ago by joepie91 · 0 comments
Owner

Currently, the LCSC scraper uses the category search API to extract all products. This is fast and efficient. However, this API will not under any circumstances produce results beyond the first 10k items in the category.

Some categories, such as this one, have quite a few more products in them than that, and are not subdivided into smaller categories. We need a way to get at these other items as well, preferably in a way that doesn't require explicitly defining the problematic categories in the scraper.

Currently, the LCSC scraper uses the category search API to extract all products. This is fast and efficient. However, this API will not under any circumstances produce results beyond the first 10k items in the category. Some categories, such as [this one](https://lcsc.com/products/Multilayer-Ceramic-Capacitors-MLCC-SMD-SMT_313.html), have quite a few more products in them than that, and are not subdivided into smaller categories. We need a way to get at these other items as well, preferably in a way that *doesn't* require explicitly defining the problematic categories in the scraper.
joepie91 added the
help wanted
label 3 years ago
Sign in to join this conversation.
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: seekseek/scraper-config#2
Loading…
There is no content yet.