One other useful feature of message prompts, besides just being useful within a chat UI, is that they are great for sending different types of input to multimodal LLMs. What are multimodal LLMs? Any LLM that takes input besides just text. Things like images, video, audio, PDFs, and so on.
ts, that gives you an example of sending Gemini Flash an image and asking it to describe the image. Notice that I'm using the messages prompt, sending in an array of messages, just like before. This time, I'm sending in a single message in that array with the role of user, and then the content this time is more than just a string. It's an array of objects, the first one which has a type of image, and a property image which points to a publicly available image URL.
This is some random image I found on Google. Here it is in the browser. It's just a cat who, yeah, looks like he's in a pretty good mood. Don't know why I tend to reach for cats when I'm looking for random images.
Anyways, the next object within this content array has the type of text and then instructions on what we want the LLM to do with that image. In this case, describe it and be concise. Okay, everything else is exactly as we've already been doing. Let's run this script to see if it can accurately describe the image.
The response, human hands gently caress the face of a tabby cat whose eyes are closed in blissful contentment, highlighting its striped fur and prominent white whiskers. I'd say that is spot on. What if you don't have a public image to work with, but an image that's stored on your server or on your local machine? That's totally possible as well.
Let me save this image to my Mac. Yeah, downloads is fine. I'll open up my file explorer as well as my Finder window. The Finder window is available just off camera, but I'm going to drag in this image to my project.
Then back inside of the script. Let's import FS from node. Then we'll use read file sync, pointing it to the image at the proper path. I think that's right.
Yeah, looks good. And then instead of passing in the image as a URL, let's pass in the Base64 encoded version. To test, we'll run the script again, which gives us yet another accurate output.