Writing our own scraper from scratch really wasn't that difficult, but we only scraped two pieces of data about each book, its title and its price. In a real world scenario, we'd probably wanna scrape even more data than that. Plus we would have to maintain that scraper over time. So for a very large collection of popular websites, Bright Data has already done the heavy lifting for us with their product called the Web Scraper API.
The Web Scraper API can be found inside of the Bright Data dashboard. It is the second item within the main navigation. On the page, you can see all of the different APIs that they have available. There's some for Instagram, some for LinkedIn, some for TikTok.
Well, let's say we want to get the same kind of data from Amazon that we just got with our own personal scraper. Let's see if that's available. We'll search for Amazon. And sure enough, there are some APIs for Amazon.
The first one is Amazon reviews, collect them by URL. No, we weren't dealing with reviews. Amazon products, that's a little bit closer. but we're not interested in best sellers.
Okay, look at this. Amazon products discover by keyword. This is perfect. Remember the keyword we were searching for or the key phrase was books about Mars.
Let's see if we can do that with a web scraper API. On the overview page of this API, You can see the input that we have to provide that is the keyword to search by for us. That's books about Mars. And this is an example of what the API results will look like.
Looks like we'll get the title or the name of the product, the brand of the product, the description, the price, and a lot of other data as well. This is a lot more than just name and price. Okay, let's give it a try under the data collection APIs. Here we can quickly configure our payload for the request and then copy a curl command to run the request.
I've zoomed out a little bit, so this looks just a little bit better. And then the inputs could be, it looks like multiple keywords. Let's take it down to just a single keyword, the one we were using earlier. Then I will provide my API token in this input.
I got that by clicking get API token. You could do the same. And then let's take a look at what other options are available. It looks like we can limit the result per input.
So just for the sake of time, let's limit our result to three books. We can also include errors with the results, error reports. We could notify some URL whenever the data is ready. Or we could deliver the results to some external storage like an Amazon S3 bucket.
or Google Cloud Storage, or any of these other options. We can also send a webhook when the data is ready. So this could be some API endpoint on your website of your choice that will take in that array of products and then do whatever you need to do with it. Store it to a database, analyze it in some way, whatever logic you need to perform.
That is awesome. It looks like we even have control over the format that the webhook data is sent in. For most cases, you probably just want to use JSON. Okay, I don't want to store my results anywhere or send my results to a webhook.
Instead, I want to use this curl request right here to just get the results immediately. So let's copy it and then open up my terminal and paste in a command. Oh, here I get a validation error saying limit per input must be greater than or equal to one. It looks like I accidentally changed this to negative six somehow.
Let's back that up, change it to three, and then try the request one more time. Nice. The result now is a snapshot ID, not the ready data. It does take just a bit.
in order for that data to actually be ready for you. So what you can do is grab this snapshot ID back over in the interface, paste the snapshot ID under the delivery options here, which allows us to either download the snapshot or deliver the snapshot to storage. We'll just download the snapshot so that we can see the results inline in our terminal. Going back there, I'll paste in this new curl command, hit enter, and the result is that the snapshot isn't quite ready.
So once again, it does take a little time for this data to be ready for you. No problem. We'll just jump back on the video in about 10 seconds when it is ready and try again. All right, we're back and now I'm running the curl command for a second time.
It took about, eh, 20 or 30 seconds for this to be ready. But now here it is. And it's just one huge blob of JSON. This is definitely more data than we were scraping ourselves.
Let's copy this to the clipboard and then paste it over in VS code so we can read it a little bit better. All right. So it looks like each book is an object. Of course, there are three books because we limited it to such.
And each book has a title. A seller name, a brand, a description, an initial price. Reviews count a ton of other information here as well. Information that it would have taken time for us to inspect the element in the browser, figure out exactly how to target that element, and then clean the data and get the data by doing things like removing the period, like we did with the price, and so on and so forth.
So the Bright Data Web Scraper API is going to save you a lot of time and effort when it comes to scraping data from popular websites, leaving you to write your own scrapers for websites that are a little less well-known. Writing our own scraper from scratch really wasn't that difficult, but we only scraped two pieces of data about each book, its title and its price. In a real world scenario, we'd probably wanna scrape even more data than that. Plus we would have to maintain that scraper over time.
So for a very large collection of popular websites, Bright Data has already done the heavy lifting for us with their product called the Web Scraper API. The Web Scraper API can be found inside of the Bright Data dashboard. It is the second item within the main navigation. On the page, you can see all of the different APIs that they have available.
There's some for Instagram, some for LinkedIn, some for TikTok. Well, let's say we want to get the same kind of data from Amazon that we just got with our own personal scraper. Let's see if that's available. We'll search for Amazon.
And sure enough, there are some APIs for Amazon. The first one is Amazon reviews, collect them by URL. No, we weren't dealing with reviews. Amazon products, that's a little bit closer.
but we're not interested in best sellers. Okay, look at this. Amazon products discover by keyword. This is perfect.
Remember the keyword we were searching for or the key phrase was books about Mars. Let's see if we can do that with a web scraper API. On the overview page of this API, You can see the input that we have to provide that is the keyword to search by for us. That's books about Mars.
And this is an example of what the API results will look like. Looks like we'll get the title or the name of the product, the brand of the product, the description, the price, and a lot of other data as well. This is a lot more than just name and price. Okay, let's give it a try under the data collection APIs.
Here we can quickly configure our payload for the request and then copy a curl command to run the request. I've zoomed out a little bit, so this looks just a little bit better. And then the inputs could be, it looks like multiple keywords. Let's take it down to just a single keyword, the one we were using earlier.
Then I will provide my API token in this input. I got that by clicking get API token. You could do the same. And then let's take a look at what other options are available.
It looks like we can limit the result per input. So just for the sake of time, let's limit our result to three books. We can also include errors with the results, error reports. We could notify some URL whenever the data is ready.
Or we could deliver the results to some external storage like an Amazon S3 bucket. or Google Cloud Storage, or any of these other options. We can also send a webhook when the data is ready. So this could be some API endpoint on your website of your choice that will take in that array of products and then do whatever you need to do with it.
Store it to a database, analyze it in some way, whatever logic you need to perform. That is awesome. It looks like we even have control over the format that the webhook data is sent in. For most cases, you probably just want to use JSON.
Okay, I don't want to store my results anywhere or send my results to a webhook. Instead, I want to use this curl request right here to just get the results immediately. So let's copy it and then open up my terminal and paste in a command. Oh, here I get a validation error saying limit per input must be greater than or equal to one.
It looks like I accidentally changed this to negative six somehow. Let's back that up, change it to three, and then try the request one more time. Nice. The result now is a snapshot ID, not the ready data.
It does take just a bit. in order for that data to actually be ready for you. So what you can do is grab this snapshot ID back over in the interface, paste the snapshot ID under the delivery options here, which allows us to either download the snapshot or deliver the snapshot to storage. We'll just download the snapshot so that we can see the results inline in our terminal.
Going back there, I'll paste in this new curl command, hit enter, and the result is that the snapshot isn't quite ready. So once again, it does take a little time for this data to be ready for you. No problem. We'll just jump back on the video in about 10 seconds when it is ready and try again.
All right, we're back and now I'm running the curl command for a second time. It took about, eh, 20 or 30 seconds for this to be ready. But now here it is. And it's just one huge blob of JSON.
This is definitely more data than we were scraping ourselves. Let's copy this to the clipboard and then paste it over in VS code so we can read it a little bit better. All right. So it looks like each book is an object.
Of course, there are three books because we limited it to such. And each book has a title. A seller name, a brand, a description, an initial price. Reviews count a ton of other information here as well.
Information that it would have taken time for us to inspect the element in the browser, figure out exactly how to target that element, and then clean the data and get the data by doing things like removing the period, like we did with the price, and so on and so forth. So the Bright Data Web Scraper API is going to save you a lot of time and effort when it comes to scraping data from popular websites, leaving you to write your own scrapers for websites that are a little less well-known.