You should be fine, assuming the toaster oven isn’t malfunctioning. In general, granite is extremely heat resistant. The stone comes from igneous rock that cooled down, meaning it should be able to withstand high temperatures before being damaged by heat. That said, there is a risk of heat shock, where you heat one part too rapidly and cause a crack to form in the stone. The other danger is that the granite countertops can discolor over time and with repeated exposure to heat.
That’s why the best practice is to use a hot pad or trivet when putting hot things on the countertops. All that to say, though, is that a toaster oven is usually pretty good at venting its heat up instead of down. In fact, most have legs to prevent burning whatever you set them on.