We've already seen how web scraping plays a huge part in the modern world of AI. It's the technology that allows models to be trained on such large swaths of data from around the web. But did you know that AI is also helpful for writing our scrapers? In fact, it's how I wrote a lot of the code for the scrapers in this course, along with some of the documentation from the Bright Data website.
So how did I go about it? ai account. I have found that Clawed is the best model out there for writing code, and so that's why I chose Clawed. But you could probably use ChatGBT as well, or some of the other models that are on the web.
The first thing I said is I said to use Playwright to search Amazon for books about Mars. What that gave me was a Python script for scraping Amazon. That's not what I wanted. We like to use JavaScript.
So then I just told Claude to use JavaScript instead. This gave us a script that looks a lot like what we did at the very beginning of coding our scraper. com. We click on the search box.
We type in books about Mars. Now, both of these steps weren't necessary, so I ended up just using this one. You didn't have to click and then fill it, but this gave me a great starting place. Then we could click the search button.
Once again, I modified this because this selector was a little too generic, but the rest of everything here looks a decent amount like what we actually ended up with. As I continued on developing the scraper further, I added more messages to the chat. Down here you can see how I asked it to handle pagination and here it gave us some code to find the next button click on the next button it used a timeout in order to wait for the next page to load which really isn't best practice we use the wait for selector instead and then all of this happens inside of a loop that loops over the total number of pages so once again we didn't use this exact same code but it gave me a good idea on how to start solving the issue at hand and can give you a nice start as well. To summarize, web scraping plays a large role in AI, and AI can play a good role in writing your scrapers as well.
We've already seen how web scraping plays a huge part in the modern world of AI. It's the technology that allows models to be trained on such large swaths of data from around the web. But did you know that AI is also helpful for writing our scrapers? In fact, it's how I wrote a lot of the code for the scrapers in this course, along with some of the documentation from the Bright Data website.
So how did I go about it? ai account. I have found that Clawed is the best model out there for writing code, and so that's why I chose Clawed. But you could probably use ChatGBT as well, or some of the other models that are on the web.
The first thing I said is I said to use Playwright to search Amazon for books about Mars. What that gave me was a Python script for scraping Amazon. That's not what I wanted. We like to use JavaScript.
So then I just told Claude to use JavaScript instead. This gave us a script that looks a lot like what we did at the very beginning of coding our scraper. com. We click on the search box.
We type in books about Mars. Now, both of these steps weren't necessary, so I ended up just using this one. You didn't have to click and then fill it, but this gave me a great starting place. Then we could click the search button.
Once again, I modified this because this selector was a little too generic, but the rest of everything here looks a decent amount like what we actually ended up with. As I continued on developing the scraper further, I added more messages to the chat. Down here you can see how I asked it to handle pagination and here it gave us some code to find the next button click on the next button it used a timeout in order to wait for the next page to load which really isn't best practice we use the wait for selector instead and then all of this happens inside of a loop that loops over the total number of pages so once again we didn't use this exact same code but it gave me a good idea on how to start solving the issue at hand and can give you a nice start as well. To summarize, web scraping plays a large role in AI, and AI can play a good role in writing your scrapers as well.