#2 LCSC: Large categories only have the first 10k items scraped

Open
opened 1 month ago by joepie91 · 0 comments
joepie91 commented 1 month ago
Owner

Currently, the LCSC scraper uses the category search API to extract all products. This is fast and efficient. However, this API will not under any circumstances produce results beyond the first 10k items in the category.

Some categories, such as this one, have quite a few more products in them than that, and are not subdivided into smaller categories. We need a way to get at these other items as well, preferably in a way that doesn't require explicitly defining the problematic categories in the scraper.

Currently, the LCSC scraper uses the category search API to extract all products. This is fast and efficient. However, this API will not under any circumstances produce results beyond the first 10k items in the category. Some categories, such as [this one](https://lcsc.com/products/Multilayer-Ceramic-Capacitors-MLCC-SMD-SMT_313.html), have quite a few more products in them than that, and are not subdivided into smaller categories. We need a way to get at these other items as well, preferably in a way that *doesn't* require explicitly defining the problematic categories in the scraper.
joepie91 added the
help wanted
label 1 month ago
Sign in to join this conversation.
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.