In this lesson, let's get hands on and scrape some Amazon data for ourselves. Let's say that I'm interested in scraping data for books about Mars. If I were to do this manually myself, I would go to the search bar at the top of the Amazon website and then just type books about Mars and hit the search button here. This would of course give me a result of all of the relevant books.
What if I want to do this in our scraper though? Well, we essentially wanna follow the exact same process just with code. screenshot line because it'll come in handy in just a few moments. com.
Now we're on the homepage and we want to fill out that search input at the top of the page. So what I'll do is back over on the homepage here, let's inspect this input and see what kind of unique identifier is available. Well, unfortunately, I don't have something as solid as a data test ID, but it does look like I have a regular ID attribute here set to toTabSearchTextBox. Let's try using that.
fill and then say what input we want to fill. That was this element with the ID of toTabSearchTextBox. So I'll start this with a hash and paste in the ID. Notice this is just a regular CSS query selector.
You'll use them all the time throughout your scrapers. And in a later lesson, we'll talk about some best practices when it comes to picking out which CSS query selectors you should use. But for now, this will work great. Then the data that we want to fill out with is of course books about Mars.
Now we want to click that submit button here. I'll right click on it, inspect. And this also has a nice ID to use. So let's copy that.
click, passing it hashtag and the ID. and that will click on the button. Now let's go back over to the browser and actually run our query again and take a look at the results page here to see what element I need to look for on the page to exist before continuing my scraping. This will ensure that I don't try to scrape elements that aren't there.
So here I can see that it looks like each of the different books, each of the different results in the list are a div with a data component type of S search result. waitForSelector. This says don't do anything else until this selector is found. And of course, the selector that we're looking for is the one with the attribute data component type and the value as search result.
Perfect. Now let's paste back in that screenshot line so that we can see the results in the screenshot to make sure it actually looks like what we expect it to. Of course, I could also use the bright data debugger, but I think this is pretty quick and easy, so we'll just stick with this. Run the scraper again.
Wait for just a few moments for it to complete. And great, it has finished. png here, there is a screenshot of the books about Mars. Now we want to get data from this though, so our job isn't quite done.
The next thing I wanna do is I wanna select each of the results in that list. We already know what the query selector for that looks like. It's this right here. $$, and then pass the selector.
querySelector does in modern browsers. It just grabs all of those elements, and now we have an array-like structure that contains all of the book divs. Let's loop over all of those book divs now. With some code, I'll just copy and paste into here.
This is just a for loop that sets an index to zero and increment that index so long as the index is less than the length of the books. Then we search for an H2 that's nested down inside each of the results, an A inside of that H2, and then finally a span. And that gives us the title element. The reason we had to do it like this is because over in the browser, if you'll inspect one of the titles here, you'll notice that there's not a lot kind of differentiating it.
Here is the actual, the surrounding div right here. Then the H2 is right down here. It's the only thing inside of each individual book result that is an H2. That gives us the title.
Then we nest down inside of that a tag down into the span tag. And here we have the title of the book. All of these classes are just stylistic classes. So in this case, our tag is probably the best bet for a selector.
Finally, if that title element exists, so if that span exists, then we will grab the inner text of it and then just console log out that title along with the number. of the index plus one. So we get a list labeled one book title, two book title, so on and so forth. Let's see what that looks like now.
When it's finished, we do indeed have a list of all of the Mars books on the first page of the results. In this lesson, let's get hands on and scrape some Amazon data for ourselves. Let's say that I'm interested in scraping data for books about Mars. If I were to do this manually myself, I would go to the search bar at the top of the Amazon website and then just type books about Mars and hit the search button here.
This would of course give me a result of all of the relevant books. What if I want to do this in our scraper though? Well, we essentially wanna follow the exact same process just with code. screenshot line because it'll come in handy in just a few moments.
com. Now we're on the homepage and we want to fill out that search input at the top of the page. So what I'll do is back over on the homepage here, let's inspect this input and see what kind of unique identifier is available. Well, unfortunately, I don't have something as solid as a data test ID, but it does look like I have a regular ID attribute here set to toTabSearchTextBox.
Let's try using that. fill and then say what input we want to fill. That was this element with the ID of toTabSearchTextBox. So I'll start this with a hash and paste in the ID.
Notice this is just a regular CSS query selector. You'll use them all the time throughout your scrapers. And in a later lesson, we'll talk about some best practices when it comes to picking out which CSS query selectors you should use. But for now, this will work great.
Then the data that we want to fill out with is of course books about Mars. Now we want to click that submit button here. I'll right click on it, inspect. And this also has a nice ID to use.
So let's copy that. click, passing it hashtag and the ID. and that will click on the button. Now let's go back over to the browser and actually run our query again and take a look at the results page here to see what element I need to look for on the page to exist before continuing my scraping.
This will ensure that I don't try to scrape elements that aren't there. So here I can see that it looks like each of the different books, each of the different results in the list are a div with a data component type of S search result. waitForSelector. This says don't do anything else until this selector is found.
And of course, the selector that we're looking for is the one with the attribute data component type and the value as search result. Perfect. Now let's paste back in that screenshot line so that we can see the results in the screenshot to make sure it actually looks like what we expect it to. Of course, I could also use the bright data debugger, but I think this is pretty quick and easy, so we'll just stick with this.
Run the scraper again. Wait for just a few moments for it to complete. And great, it has finished. png here, there is a screenshot of the books about Mars.
Now we want to get data from this though, so our job isn't quite done. The next thing I wanna do is I wanna select each of the results in that list. We already know what the query selector for that looks like. It's this right here.
$$, and then pass the selector. querySelector does in modern browsers. It just grabs all of those elements, and now we have an array-like structure that contains all of the book divs. Let's loop over all of those book divs now.
With some code, I'll just copy and paste into here. This is just a for loop that sets an index to zero and increment that index so long as the index is less than the length of the books. Then we search for an H2 that's nested down inside each of the results, an A inside of that H2, and then finally a span. And that gives us the title element.
The reason we had to do it like this is because over in the browser, if you'll inspect one of the titles here, you'll notice that there's not a lot kind of differentiating it. Here is the actual, the surrounding div right here. Then the H2 is right down here. It's the only thing inside of each individual book result that is an H2.
That gives us the title. Then we nest down inside of that a tag down into the span tag. And here we have the title of the book. All of these classes are just stylistic classes.
So in this case, our tag is probably the best bet for a selector. Finally, if that title element exists, so if that span exists, then we will grab the inner text of it and then just console log out that title along with the number. of the index plus one. So we get a list labeled one book title, two book title, so on and so forth.
Let's see what that looks like now. When it's finished, we do indeed have a list of all of the Mars books on the first page of the results.