com. You can do that on their homepage with the Start with Google button. Now, I have already created an account, so everything is already set up. You'll have just a small step or two to get the process started, but it's just basic account setup stuff.
Then you'll land on the dashboard for the Bright Data website. Now, notice that Bright Data offers several different products in order to help you work with data from around the web. The one that we'll be focusing on right now exists under the proxies and scraping tab. But do also note this web data tab, it is particularly interesting and we will look at it in detail later on.
Essentially this web data collection is data from websites, popular websites that they have already scraped and cleaned up for you. But for now, let's focus on building our own scraper. We can do that by pressing get started on the scraping browser solution on the proxies and scraping page. So I'll hit get started, provide a zone name.
I'll just stick with the default. but you can set this to whatever your project relates to, you know, whatever website you're scraping. Then we'll leave the CAPTCHA solver on as this comes at no extra cost and will just help us bypass pesky CAPTCHA solvers automatically. Then I'll hit add.
Am I sure? Yes, let's go ahead and create this scraping browser. Now I'm presented with some instructions to set up the scraping browser with some code written on my local machine. Let's just hit continue with scraping browser playground for now, as this is the easiest way to get started.
com website. When I run this request, you'll be able to see the browser actually opening the website right over here and going through the different steps to scrape the code. you'll also see the output of the code down here in the console. But first, before we run this, let's just walk through it and see what it's actually doing.
Well, notice that first of all, in this case, we're using Puppeteer in order to do the scraping. But remember, this could be any number of different solutions. This could be Selenium, this could be Playwright, they just happen to use Puppeteer here. com.
We are going to use the Bright Data Proxy in order to actually connect to this URL. Then we have a couple of utility functions, one that adds days to a current date, one that just formats a date to a certain string format. And this is where things get a little bit interesting. It looks like we're going to be searching the booking website for, I guess, hotels inside of New York City, and we're going to check in to that hotel one day from now and check out of that hotel two days from now.
com URL. Well, what does the run function look like? Here is the definition for that. pretty straightforward.
First, we use Puppeteer to connect, passing it that browser WS in order to use the Bright Data Proxy. com URL, wait until all the DOM content is loaded. So wait until the page loads. And that's great.
com. The first thing they do then is close a pop-up that might appear on the page. If you take a look at the code for this function, notice that it calls a function called waitForSelector on the page, which selects an element based upon its area label. Then it clicks on that button in order to close the pop-up.
But this is nested inside of a try catch, so Just in case the pop-up actually doesn't appear, it doesn't matter. We can continue on with our page interactions. What does that pop-up look like? com, I do indeed get a pop-up.
And if I inspect this close button right here, Notice it has an area label that matches what we saw in the code. So the way they knew what to write in that code is by simply visiting the website and inspecting the page element. The code does exactly what I just did manually by clicking the X. Back over here now, we call an interact function.
The definition of the interact function searches the page for a selector. with data test ID destination container, and then an input down inside of there. This is this search input right here. And this is where it will type out New York.
The reason they used a data test ID is because it's common to use these data attributes to hook up unit test to the DOM and check for certain elements to exist or to type in them or whatever. And so it's very unlikely that these data attributes will change, which makes this test more resilient over time. Great. Next, we actually type into that search input, the search text, which was New York.
We click on the search box dates container, which is this picker right here to check the dates. Then we set the check in date to our variable, the check out date to the variable that we set earlier. And then we click the submit button, waiting for the results page to appear. Awesome, so now we've gotten to where we want to be on the website.
So we're done with the interact phase. And then we can just look through the page, find the data we want, and parse it to a legible format. In this case, a JavaScript object. Okay, so what does this parse function actually look like?
Right down here, we're just doing more query selection using CSS selectors in order to get certain things from the page. Once again, using the data test IDs because they're more resilient. com website and type in New York, I'll choose tomorrow and the next day to come back just like it did inside of the script. Then if I right click, on one of the titles for a result.
Notice here's that data test ID title. So in order to write this script, someone literally manually went through exactly like we're doing and just grabbed the selectors that they needed from the source code. And so here, this is exactly what you can see. We're getting the title by checking the element with the data test ID of title.
And then we're just grabbing the inner text of that. Then we're doing the same thing for price and discounted price, as well as the review score, doing a little bit of formatting on that review score. And then ultimately we are saving each of those results to a JavaScript object. At the very end of the run function, we just take that data and log it to the console.
In a real life scenario, you'd probably want to send this off to an API or save it in a database. but we'll talk about that more later on. Okay, so that's how this works. Let's see it in action by hitting the Run button.
Hit Run, you can see the output here in the console. First, we have connected to the browser, and then we navigate to the site. After we navigate, we wait for the pop-up to appear. Great, there it is.
pop-up appeared, we're going to manually close it or rather automatically close it. I did not click on anything over there on the right-hand side. That happened from the code. We wait for the search form.
Once it's appeared, we start filling it in, which is what you can see happening in the browser right now. So this is a totally live browser. It's visiting via a real browser, which is why it can bypass some of those bot measures that aren't caught by other scraping technologies. Now, after the form is filled out, we wait for the results page to appear.
And after it does, we start parsing the data. And there it is. Here we have our parsed data results with real live data about hotels in New York. That's how easy it is to start collecting valuable data from around the web.
And in the next lesson, I'll show you how to get things set up locally so that you can work in the IDE you're already familiar with. com. You can do that on their homepage with the Start with Google button. Now, I have already created an account, so everything is already set up.
You'll have just a small step or two to get the process started, but it's just basic account setup stuff. Then you'll land on the dashboard for the Bright Data website. Now, notice that Bright Data offers several different products in order to help you work with data from around the web. The one that we'll be focusing on right now exists under the proxies and scraping tab.
But do also note this web data tab, it is particularly interesting and we will look at it in detail later on. Essentially this web data collection is data from websites, popular websites that they have already scraped and cleaned up for you. But for now, let's focus on building our own scraper. We can do that by pressing get started on the scraping browser solution on the proxies and scraping page.
So I'll hit get started, provide a zone name. I'll just stick with the default. but you can set this to whatever your project relates to, you know, whatever website you're scraping. Then we'll leave the CAPTCHA solver on as this comes at no extra cost and will just help us bypass pesky CAPTCHA solvers automatically.
Then I'll hit add. Am I sure? Yes, let's go ahead and create this scraping browser. Now I'm presented with some instructions to set up the scraping browser with some code written on my local machine.
Let's just hit continue with scraping browser playground for now, as this is the easiest way to get started. com website. When I run this request, you'll be able to see the browser actually opening the website right over here and going through the different steps to scrape the code. you'll also see the output of the code down here in the console.
But first, before we run this, let's just walk through it and see what it's actually doing. Well, notice that first of all, in this case, we're using Puppeteer in order to do the scraping. But remember, this could be any number of different solutions. This could be Selenium, this could be Playwright, they just happen to use Puppeteer here.
com. We are going to use the Bright Data Proxy in order to actually connect to this URL. Then we have a couple of utility functions, one that adds days to a current date, one that just formats a date to a certain string format. And this is where things get a little bit interesting.
It looks like we're going to be searching the booking website for, I guess, hotels inside of New York City, and we're going to check in to that hotel one day from now and check out of that hotel two days from now. com URL. Well, what does the run function look like? Here is the definition for that.
pretty straightforward. First, we use Puppeteer to connect, passing it that browser WS in order to use the Bright Data Proxy. com URL, wait until all the DOM content is loaded. So wait until the page loads.
And that's great. com. The first thing they do then is close a pop-up that might appear on the page. If you take a look at the code for this function, notice that it calls a function called waitForSelector on the page, which selects an element based upon its area label.
Then it clicks on that button in order to close the pop-up. But this is nested inside of a try catch, so Just in case the pop-up actually doesn't appear, it doesn't matter. We can continue on with our page interactions. What does that pop-up look like?
com, I do indeed get a pop-up. And if I inspect this close button right here, Notice it has an area label that matches what we saw in the code. So the way they knew what to write in that code is by simply visiting the website and inspecting the page element. The code does exactly what I just did manually by clicking the X.
Back over here now, we call an interact function. The definition of the interact function searches the page for a selector. with data test ID destination container, and then an input down inside of there. This is this search input right here.
And this is where it will type out New York. The reason they used a data test ID is because it's common to use these data attributes to hook up unit test to the DOM and check for certain elements to exist or to type in them or whatever. And so it's very unlikely that these data attributes will change, which makes this test more resilient over time. Great.
Next, we actually type into that search input, the search text, which was New York. We click on the search box dates container, which is this picker right here to check the dates. Then we set the check in date to our variable, the check out date to the variable that we set earlier. And then we click the submit button, waiting for the results page to appear.
Awesome, so now we've gotten to where we want to be on the website. So we're done with the interact phase. And then we can just look through the page, find the data we want, and parse it to a legible format. In this case, a JavaScript object.
Okay, so what does this parse function actually look like? Right down here, we're just doing more query selection using CSS selectors in order to get certain things from the page. Once again, using the data test IDs because they're more resilient. com website and type in New York, I'll choose tomorrow and the next day to come back just like it did inside of the script.
Then if I right click, on one of the titles for a result. Notice here's that data test ID title. So in order to write this script, someone literally manually went through exactly like we're doing and just grabbed the selectors that they needed from the source code. And so here, this is exactly what you can see.
We're getting the title by checking the element with the data test ID of title. And then we're just grabbing the inner text of that. Then we're doing the same thing for price and discounted price, as well as the review score, doing a little bit of formatting on that review score. And then ultimately we are saving each of those results to a JavaScript object.
At the very end of the run function, we just take that data and log it to the console. In a real life scenario, you'd probably want to send this off to an API or save it in a database. but we'll talk about that more later on. Okay, so that's how this works.
Let's see it in action by hitting the Run button. Hit Run, you can see the output here in the console. First, we have connected to the browser, and then we navigate to the site. After we navigate, we wait for the pop-up to appear.
Great, there it is. pop-up appeared, we're going to manually close it or rather automatically close it. I did not click on anything over there on the right-hand side. That happened from the code.
We wait for the search form. Once it's appeared, we start filling it in, which is what you can see happening in the browser right now. So this is a totally live browser. It's visiting via a real browser, which is why it can bypass some of those bot measures that aren't caught by other scraping technologies.
Now, after the form is filled out, we wait for the results page to appear. And after it does, we start parsing the data. And there it is. Here we have our parsed data results with real live data about hotels in New York.
That's how easy it is to start collecting valuable data from around the web. And in the next lesson, I'll show you how to get things set up locally so that you can work in the IDE you're already familiar with.