So far, we've only gotten the first page of data for our search result on Amazon. There are still plenty of other pages of relevant data for us to scrape. Just like we submitted the search form earlier, we can also click on the pagination buttons now in order to move between the different pages of our search results. The relevant selector that I want to use for this trigger is the a tag combined with the class S pagination next, as this is the most specific thing I could find without having to update the area label and change this page number every time I click on a new page.
OK, so how do we click through the pages programmatically in the code? Back over in my IDE, let me paste in a new function to handle this pagination for us. And then let's walk through what it's doing step by step. So, notice that I have called it paginateResults.
This is another really good example of where breaking up our scraper into individual, focused functions makes things easier to understand. The paginateResults function takes the page, that is the page object provided by Playwright, and then it also takes a callback function called processPage. Essentially, we'll be able to pass it our getBooks function and get all the results for every single page. Now, we start off with the current page being assumed as one.
We also have a boolean called hasNextPage set to true. And then we simply loop while hasNextPage is true. We'll console log the current page, just kind of an indicator that this is the page we're currently scraping. Just like we did before, we will wait for a selector to indicate that the page has fully loaded, and then we will call that processPageCallback function.
As opposed to hardcoding and the getBooks function here, means that our paginate results function is more flexible and reusable, and can be used for more than just getting books. After we've scraped the current page, then we go and find that next button using the selector that we just talked about over in the browser. If the next button doesn't exist, then we've reached the last page and has next page becomes false, meaning our loop will end. Otherwise, if there is still a next button, we'll click it and go to the next page, starting the loop all over again.
So how do we put this to use? Up inside of our main function, I'll simply call awaitPaginateResults, pass it the page from Playwright, and then provide a callback function that calls getBooks. That means we can get rid of the individual call here of getBooks, instead preferring getBooks to happen for every page and not just the first one. And with this, our scraper should work.
However, just so we aren't sitting here and waiting for all the different pages of the books about Mars to be scraped, let's provide a more specific query that will give less results. To do that, let's create a new variable called booksSearch, and then I'll paste in this query that I want to search for, live on Mars books. I searched this a little bit earlier and know that it only has five pages of results. Then, we'll replace the hard-coded books about Mars right here with the variable booksSearch.
To do this, we'll need to use string interpolation though, so I should change the quotation marks to tick marks. Finally, let's use the encodeUriComponent function in order to make sure that booksSearch is properly encoded for use inside of a query string variable. Whoops, it looks like this should be booksSearch. instead of book search.
All right, moment of truth. Let's see if this works. All right, it looks like we have one page, two pages, three pages of data. From my manual search off camera, I was actually expecting five pages.
So what's going on here? Well, let me first just go ahead and confirm in the browser that I should be expecting five pages. Right down here. Yeah, we should have five pages Well, I think what's happening is that down here inside of my paginate results Is that I'm only waiting for the s search result?
selector this is an individual result on the page and the very first Item on page 3 actually fulfills this condition. So it looks like maybe I need to wait for a different selector that ensures all of those individual items on the page are set and rendered properly. Let's go back here and do a little digging about what we could look for. I'll inspect the element.
And so that I have something different between each page, let's look for the term page one, since we're currently on page one. It looks like the span tag that displays the current page has an area label of current page, page, whatever the number is. This will actually work great because this renders after all of the search results. So we can be assured that each of the individual results is in the DOM.
and it's also specific per page so that I know the next page has fully loaded. All we need to do is dynamically enter this number right here, which I think is going to be pretty simple. So let's grab this selector, come back over to the code, and inside of the paginate results function, let's call waitForSelector, area label, current page, and then add in the page number here as a dynamic value. Once again, we'll need to change our surrounding quotes to tick marks.
Let's clear out the console and then run our scraper one more time. Perfect. This time I get five pages. And if I go to the very last result, that is a big history.
Let's take a look at that in the browser. Search for big history. Perfect. It is the last result on the last page.
In conclusion, sometimes you'll have to massage your scrapers a little bit, try something that doesn't work, and then refine it as you go along. When you're looking to work with paginated data, you can basically emulate the user's journey of loading the page, clicking the next button, and then waiting for the next page to load before scraping the new data. Doing this in a loop ensures you can get all the pages until the very end. So far, we've only gotten the first page of data for our search result on Amazon.
There are still plenty of other pages of relevant data for us to scrape. Just like we submitted the search form earlier, we can also click on the pagination buttons now in order to move between the different pages of our search results. The relevant selector that I want to use for this trigger is the a tag combined with the class S pagination next, as this is the most specific thing I could find without having to update the area label and change this page number every time I click on a new page. OK, so how do we click through the pages programmatically in the code?
Back over in my IDE, let me paste in a new function to handle this pagination for us. And then let's walk through what it's doing step by step. So, notice that I have called it paginateResults. This is another really good example of where breaking up our scraper into individual, focused functions makes things easier to understand.
The paginateResults function takes the page, that is the page object provided by Playwright, and then it also takes a callback function called processPage. Essentially, we'll be able to pass it our getBooks function and get all the results for every single page. Now, we start off with the current page being assumed as one. We also have a boolean called hasNextPage set to true.
And then we simply loop while hasNextPage is true. We'll console log the current page, just kind of an indicator that this is the page we're currently scraping. Just like we did before, we will wait for a selector to indicate that the page has fully loaded, and then we will call that processPageCallback function. As opposed to hardcoding and the getBooks function here, means that our paginate results function is more flexible and reusable, and can be used for more than just getting books.
After we've scraped the current page, then we go and find that next button using the selector that we just talked about over in the browser. If the next button doesn't exist, then we've reached the last page and has next page becomes false, meaning our loop will end. Otherwise, if there is still a next button, we'll click it and go to the next page, starting the loop all over again. So how do we put this to use?
Up inside of our main function, I'll simply call awaitPaginateResults, pass it the page from Playwright, and then provide a callback function that calls getBooks. That means we can get rid of the individual call here of getBooks, instead preferring getBooks to happen for every page and not just the first one. And with this, our scraper should work. However, just so we aren't sitting here and waiting for all the different pages of the books about Mars to be scraped, let's provide a more specific query that will give less results.
To do that, let's create a new variable called booksSearch, and then I'll paste in this query that I want to search for, live on Mars books. I searched this a little bit earlier and know that it only has five pages of results. Then, we'll replace the hard-coded books about Mars right here with the variable booksSearch. To do this, we'll need to use string interpolation though, so I should change the quotation marks to tick marks.
Finally, let's use the encodeUriComponent function in order to make sure that booksSearch is properly encoded for use inside of a query string variable. Whoops, it looks like this should be booksSearch. instead of book search. All right, moment of truth.
Let's see if this works. All right, it looks like we have one page, two pages, three pages of data. From my manual search off camera, I was actually expecting five pages. So what's going on here?
Well, let me first just go ahead and confirm in the browser that I should be expecting five pages. Right down here. Yeah, we should have five pages Well, I think what's happening is that down here inside of my paginate results Is that I'm only waiting for the s search result? selector this is an individual result on the page and the very first Item on page 3 actually fulfills this condition.
So it looks like maybe I need to wait for a different selector that ensures all of those individual items on the page are set and rendered properly. Let's go back here and do a little digging about what we could look for. I'll inspect the element. And so that I have something different between each page, let's look for the term page one, since we're currently on page one.
It looks like the span tag that displays the current page has an area label of current page, page, whatever the number is. This will actually work great because this renders after all of the search results. So we can be assured that each of the individual results is in the DOM. and it's also specific per page so that I know the next page has fully loaded.
All we need to do is dynamically enter this number right here, which I think is going to be pretty simple. So let's grab this selector, come back over to the code, and inside of the paginate results function, let's call waitForSelector, area label, current page, and then add in the page number here as a dynamic value. Once again, we'll need to change our surrounding quotes to tick marks. Let's clear out the console and then run our scraper one more time.
Perfect. This time I get five pages. And if I go to the very last result, that is a big history. Let's take a look at that in the browser.
Search for big history. Perfect. It is the last result on the last page. In conclusion, sometimes you'll have to massage your scrapers a little bit, try something that doesn't work, and then refine it as you go along.
When you're looking to work with paginated data, you can basically emulate the user's journey of loading the page, clicking the next button, and then waiting for the next page to load before scraping the new data. Doing this in a loop ensures you can get all the pages until the very end.