There are ways of scraping data that just work. And then there are ways of scraping data that will scale and will be more reliable in the long run. Sometimes it's hard to tell which is which. So in this lesson, I'm gonna show you a few tips about writing scrapers that scale.
com and then within our scraper code, typed out books about Mars inside of the search box, hit the search button in order to navigate to the results page. This worked, but over in the browser, notice that the URL contains a query string K equal to our query books about Mars. com slash S, question mark K equals books plus about plus Mars. Why?
Because it's a lot less likely that this is ever going to change. If it were to change, everyone who shared this URL right here on their social media or on other websites, would then get a 404 page visiting this. So it's extremely unlikely that Amazon is going to change something that breaks this format. Not only that, it's one less step within our scraper.
So that's one less thing that's likely to break. So over here, what we're gonna do then is provide slash S question mark K equals books. about Mars. Then we can get rid of filling out the search input and hitting the search button.
Writing our scraper again, we do indeed see the same titles being printed to the console. Another best practice when writing scalable scrapers is to use stronger and less brittle query selectors. For example, whenever there is an ID available, they are guaranteed to be unique. So definitely prefer selecting elements with IDs over selecting elements with classes or just tag names.
You can also target elements based on their relationship to parent elements. We saw this when we were selecting the title, right? Just because it was an H2 doesn't mean it was necessarily a title, but because it was an H2 nested inside of where the search results then we could be barely confident that we're getting the right thing. However, do avoid overly long chains, as if one element in the chain is removed, the whole thing will break.
You should also prioritize data attribute selectors. These are often more stable than classes or structure-based selectors since they're frequently used in testing. When all else fails, you can identify some items by their consistent position in the DOM. But these selectors are quite brittle, so do be aware that you'll probably have to baby them a little bit.
Finally, it's also a good idea to break up your scrapers into functions when it makes sense. For example, our code here where we get the books could be wrapped up into a function called getBooks. and then reuse whenever we need to get those books again, or simply for better legibility. So here I'll call await getBooks, passing it the page for us to do the query selection on, and then down below at the bottom of my page, I'll just define an async function called getBooks.
It of course accepts the page. and then I'll paste in what we had before. Doesn't that read better? We visit the page, wait for the results, get the books, and then later on we'll parse them.
There are ways of scraping data that just work. And then there are ways of scraping data that will scale and will be more reliable in the long run. Sometimes it's hard to tell which is which. So in this lesson, I'm gonna show you a few tips about writing scrapers that scale.
com and then within our scraper code, typed out books about Mars inside of the search box, hit the search button in order to navigate to the results page. This worked, but over in the browser, notice that the URL contains a query string K equal to our query books about Mars. com slash S, question mark K equals books plus about plus Mars. Why?
Because it's a lot less likely that this is ever going to change. If it were to change, everyone who shared this URL right here on their social media or on other websites, would then get a 404 page visiting this. So it's extremely unlikely that Amazon is going to change something that breaks this format. Not only that, it's one less step within our scraper.
So that's one less thing that's likely to break. So over here, what we're gonna do then is provide slash S question mark K equals books. about Mars. Then we can get rid of filling out the search input and hitting the search button.
Writing our scraper again, we do indeed see the same titles being printed to the console. Another best practice when writing scalable scrapers is to use stronger and less brittle query selectors. For example, whenever there is an ID available, they are guaranteed to be unique. So definitely prefer selecting elements with IDs over selecting elements with classes or just tag names.
You can also target elements based on their relationship to parent elements. We saw this when we were selecting the title, right? Just because it was an H2 doesn't mean it was necessarily a title, but because it was an H2 nested inside of where the search results then we could be barely confident that we're getting the right thing. However, do avoid overly long chains, as if one element in the chain is removed, the whole thing will break.
You should also prioritize data attribute selectors. These are often more stable than classes or structure-based selectors since they're frequently used in testing. When all else fails, you can identify some items by their consistent position in the DOM. But these selectors are quite brittle, so do be aware that you'll probably have to baby them a little bit.
Finally, it's also a good idea to break up your scrapers into functions when it makes sense. For example, our code here where we get the books could be wrapped up into a function called getBooks. and then reuse whenever we need to get those books again, or simply for better legibility. So here I'll call await getBooks, passing it the page for us to do the query selection on, and then down below at the bottom of my page, I'll just define an async function called getBooks.
It of course accepts the page. and then I'll paste in what we had before. Doesn't that read better? We visit the page, wait for the results, get the books, and then later on we'll parse them.