All right, so you've seen how we can use Puppeteer within the built-in playground in order to scrape a website. For real projects though, you'll probably want to write your scrapers in your own IDE, where you can get all the good auto-completion you're used to and so on and so forth. Okay, so I've already set up a local project just by creating a directory and then running npm init. json file, and a node module's directory.
Then I also ran npm i playwright in order to install the playwright library. Remember, this is the same kind of thing as puppeteer, and I'm choosing to use it just because of personal preference. If you want to follow along with me during this course, you should install Playwright as well, but feel free to experiment with Selenium and Puppeteer and see which one you like better. Now, since I have Playwright installed, I can start creating my first scraper.
js because, well, we're going to try to scrape Amazon. Back over in the browser, let's move to this Code Examples tab, and this provides us with some code we can copy and paste to get started. However, this is written with Puppeteer. Using this dropdown right here, I can change to Playwright.
Yes, here is that same exact MPMI Playwright command that I ran to install the dependency, and here is some code to do our web scraping. Notice what it starts out with. This is a URL that points to the Bright Data proxies, and we're going to connect over that proxy instead of directly to the website that we want to scrape. This is auto-filled out with my credentials and the data I need for my account.
By the time this video is released, my account will be deleted, so this won't work for you, but make sure you provide the proper data for your account here. Then, down inside of this main function, after we have connected, via the proxy, we will go to a new page. com. Okay, we'll change this out to Amazon.
Then we'll take a screenshot and grab the full content of the page, that is all the HTML, and then just log it to the console. com. com. In the terminal, let's use Node to run this script.
First it connects to the scraping browser, then we see that we're connected and we start navigating to the Amazon homepage. Now this might take more than just a few seconds. I had to wait about 30 to 45 seconds. I fast forwarded the video so that you didn't have to sit through all of that, but do note that if you're sitting and waiting, it's not broken, it's just taking a bit.
Now we're taking a screenshot of the page. Wait until that finishes and all of the HTML is logged to the console. It's quite a bit of it. Surprise, surprise.
png file, here is a screenshot of the Amazon homepage. And look at that, it's near Thanksgiving when I'm creating this course. And look, there's that nice fall theme going on. And sure enough, here are Black Friday deals all over the place.
So this is definitely live, relevant, real data from Amazon dot com. One thing I do want you to notice is that during that entire scraping process, there was not an actual browser visible anywhere that showed the Amazon Web page. It didn't show up. side by side anywhere, just like we saw it inside of the playground.
But that doesn't mean that Bright Data's scraping browser isn't running a real browser behind the scenes. It is. And that's why it's so good at what it does. What if we actually want to get eyes on what's happening in that browser in real time?
We can do that back inside of the Bright Data website here in the overview tab. I can open up this Chrome DevTools debugger. And it just lets us know that we couldn't find a live session. So let's start our scraper one more time in order to start a session.
Open it up one more time, then open this up in the browser. And now we can see under the hood what that bright data browser is actually doing. All right, so you've seen how we can use Puppeteer within the built-in playground in order to scrape a website. For real projects though, you'll probably want to write your scrapers in your own IDE, where you can get all the good auto-completion you're used to and so on and so forth.
Okay, so I've already set up a local project just by creating a directory and then running npm init. json file, and a node module's directory. Then I also ran npm i playwright in order to install the playwright library. Remember, this is the same kind of thing as puppeteer, and I'm choosing to use it just because of personal preference.
If you want to follow along with me during this course, you should install Playwright as well, but feel free to experiment with Selenium and Puppeteer and see which one you like better. Now, since I have Playwright installed, I can start creating my first scraper. js because, well, we're going to try to scrape Amazon. Back over in the browser, let's move to this Code Examples tab, and this provides us with some code we can copy and paste to get started.
However, this is written with Puppeteer. Using this dropdown right here, I can change to Playwright. Yes, here is that same exact MPMI Playwright command that I ran to install the dependency, and here is some code to do our web scraping. Notice what it starts out with.
This is a URL that points to the Bright Data proxies, and we're going to connect over that proxy instead of directly to the website that we want to scrape. This is auto-filled out with my credentials and the data I need for my account. By the time this video is released, my account will be deleted, so this won't work for you, but make sure you provide the proper data for your account here. Then, down inside of this main function, after we have connected, via the proxy, we will go to a new page.
com. Okay, we'll change this out to Amazon. Then we'll take a screenshot and grab the full content of the page, that is all the HTML, and then just log it to the console. com.
com. In the terminal, let's use Node to run this script. First it connects to the scraping browser, then we see that we're connected and we start navigating to the Amazon homepage. Now this might take more than just a few seconds.
I had to wait about 30 to 45 seconds. I fast forwarded the video so that you didn't have to sit through all of that, but do note that if you're sitting and waiting, it's not broken, it's just taking a bit. Now we're taking a screenshot of the page. Wait until that finishes and all of the HTML is logged to the console.
It's quite a bit of it. Surprise, surprise. png file, here is a screenshot of the Amazon homepage. And look at that, it's near Thanksgiving when I'm creating this course.
And look, there's that nice fall theme going on. And sure enough, here are Black Friday deals all over the place. So this is definitely live, relevant, real data from Amazon dot com. One thing I do want you to notice is that during that entire scraping process, there was not an actual browser visible anywhere that showed the Amazon Web page.
It didn't show up. side by side anywhere, just like we saw it inside of the playground. But that doesn't mean that Bright Data's scraping browser isn't running a real browser behind the scenes. It is.
And that's why it's so good at what it does. What if we actually want to get eyes on what's happening in that browser in real time? We can do that back inside of the Bright Data website here in the overview tab. I can open up this Chrome DevTools debugger.
And it just lets us know that we couldn't find a live session. So let's start our scraper one more time in order to start a session. Open it up one more time, then open this up in the browser. And now we can see under the hood what that bright data browser is actually doing.