So far, we've done nothing but print our data to the console. This isn't extremely helpful. Instead, we should structure our data and then store it somehow. Well, we can store all of our books in an array and make each book an object.
Let me show you how we can do that. Here inside of the getBooks function, what I'm going to do is I'm going to move this title condition to the line right before our if statement. So we'll say the title is equal to and then add the title element conditionally. And so if the title element exists, then we will grab its inner text.
Otherwise, we'll just say there is no title. Underneath now, we can create our book object and give it the title property. Let's also add some more information about each book. Over in the browser, one other piece of information I see that would be handy to have is the price.
So let's inspect that and then see how we can select this. Okay, it looks like there is a price whole class on the whole number of the price, then a price fraction class on the cents. Do notice inside of a price whole though, we have another span called a price decimal. So we'll have to get rid of this bit here.
Then once we have the whole price, the dollar amount and the cents, we'll combine them together to be the full price. Let's see what that looks like in the code. I'll just copy and paste this in for the sake of time and then explain what I'm doing. Okay, so first I get the price whole element.
based on the class name, but notice I am nesting down inside of our current book. So there's not a lot of elements to choose from. This is great. I'm also using a price whole class, which is very descriptive of the actual data.
So this is a pretty good selector. Then I'm just checking, just like I did with the title, does that price whole element exist? If it does, grab its inner text and make sure to replace that period with a blank string. Then I do the exact same thing for the price fraction element.
And then finally assign the combination of these two together with a period in between them to the price property on the book. I'm also turning this into a number since a price would be good to work with as an actual number. And if there is no price available, we're just setting the value to null. But Where does this book object go now?
Well, let's create an array called results, where we can gather all the books together. Then inside the loop, let's push each book to the results. Finally, we should return all the results from this page from the function. Now we have the results for each page gathered up into an array, but we want all the results from all the pages together in a single array.
So up here inside of the paginate results call, the callback function, instead of just calling getBooks, let's await the result of getBooks, which is remember the result from each page. Make sure we mark this callback function as async, so the await keyword actually works. Capture the results of each page in a variable. We'll call it results per page.
and then we'll create a new array to gather the results for all the pages in. Lastly, let's just combine the results for all pages with the results for the current page as it's being iterated over. That looks something like this. Setting results for all pages to a new array where we spread what's currently in results for all pages along with the new results for the current page.
Now let's console log our structured data and we should be pretty much there. Run the scraper again. We still get our console logs for each page being scraped. And then at the very end, we get our data in a nice structured format.
Notice here we have an object with a title, a price. This one looks good. They are all in a very consistent format with each of the objects having the same type of information. Now that we have it all gathered up in an easily read and parsable format, we can store it wherever we'd like.
There are a ton of different options for this. There are self-hostable NoSQL databases like MongoDB, fully managed databases like Firebase, which is also a NoSQL database. There's plenty of relational databases out there like Postgres, MySQL, or SQLite. or even fully managed SQL databases like SuperBase, Terso, or PlanetScale.
Using any of these solutions to actually store the data is beyond the scope of this course. However, once you've got the data structured as we have it, it's pretty straightforward from here. So far, we've done nothing but print our data to the console. This isn't extremely helpful.
Instead, we should structure our data and then store it somehow. Well, we can store all of our books in an array and make each book an object. Let me show you how we can do that. Here inside of the getBooks function, what I'm going to do is I'm going to move this title condition to the line right before our if statement.
So we'll say the title is equal to and then add the title element conditionally. And so if the title element exists, then we will grab its inner text. Otherwise, we'll just say there is no title. Underneath now, we can create our book object and give it the title property.
Let's also add some more information about each book. Over in the browser, one other piece of information I see that would be handy to have is the price. So let's inspect that and then see how we can select this. Okay, it looks like there is a price whole class on the whole number of the price, then a price fraction class on the cents.
Do notice inside of a price whole though, we have another span called a price decimal. So we'll have to get rid of this bit here. Then once we have the whole price, the dollar amount and the cents, we'll combine them together to be the full price. Let's see what that looks like in the code.
I'll just copy and paste this in for the sake of time and then explain what I'm doing. Okay, so first I get the price whole element. based on the class name, but notice I am nesting down inside of our current book. So there's not a lot of elements to choose from.
This is great. I'm also using a price whole class, which is very descriptive of the actual data. So this is a pretty good selector. Then I'm just checking, just like I did with the title, does that price whole element exist?
If it does, grab its inner text and make sure to replace that period with a blank string. Then I do the exact same thing for the price fraction element. And then finally assign the combination of these two together with a period in between them to the price property on the book. I'm also turning this into a number since a price would be good to work with as an actual number.
And if there is no price available, we're just setting the value to null. But Where does this book object go now? Well, let's create an array called results, where we can gather all the books together. Then inside the loop, let's push each book to the results.
Finally, we should return all the results from this page from the function. Now we have the results for each page gathered up into an array, but we want all the results from all the pages together in a single array. So up here inside of the paginate results call, the callback function, instead of just calling getBooks, let's await the result of getBooks, which is remember the result from each page. Make sure we mark this callback function as async, so the await keyword actually works.
Capture the results of each page in a variable. We'll call it results per page. and then we'll create a new array to gather the results for all the pages in. Lastly, let's just combine the results for all pages with the results for the current page as it's being iterated over.
That looks something like this. Setting results for all pages to a new array where we spread what's currently in results for all pages along with the new results for the current page. Now let's console log our structured data and we should be pretty much there. Run the scraper again.
We still get our console logs for each page being scraped. And then at the very end, we get our data in a nice structured format. Notice here we have an object with a title, a price. This one looks good.
They are all in a very consistent format with each of the objects having the same type of information. Now that we have it all gathered up in an easily read and parsable format, we can store it wherever we'd like. There are a ton of different options for this. There are self-hostable NoSQL databases like MongoDB, fully managed databases like Firebase, which is also a NoSQL database.
There's plenty of relational databases out there like Postgres, MySQL, or SQLite. or even fully managed SQL databases like SuperBase, Terso, or PlanetScale. Using any of these solutions to actually store the data is beyond the scope of this course. However, once you've got the data structured as we have it, it's pretty straightforward from here.