LCSC: Large categories only have the first 10k items scraped
#2
Open
opened 3 years ago by joepie91
·
0 comments
Loading…
Reference in New Issue
There is no content yet.
Delete Branch '%!s(<nil>)'
Deleting a branch is permanent. It CANNOT be undone. Continue?
Currently, the LCSC scraper uses the category search API to extract all products. This is fast and efficient. However, this API will not under any circumstances produce results beyond the first 10k items in the category.
Some categories, such as this one, have quite a few more products in them than that, and are not subdivided into smaller categories. We need a way to get at these other items as well, preferably in a way that doesn't require explicitly defining the problematic categories in the scraper.