In general: King Arthur Flour carries many hard-to-track-down ingredients for kosher baking, stuff like dough enhancer, 00 flour, and pain au chocolat sticks. All their kosher-certified products are listed here; not all symbols are visible in the packaging pics, so you may have to email about some.
Extracts + Flavorings
Specialty extracts: You can probably drum up some good-quality workhorse extracts at your local supermarket, your vanillas (OU) and almonds (Kof-K) and peppermints (KSA). These are the ones I keep on hand, though I use them sparingly. But maybe coconut, eggnog, "Buttery," Butter Rum, "Butternut," "Princess," cream cheese, or pumpkin extract floats your boat (all Star-K parve)?
Vanilla bean paste or powder: Nielsen-Massey makes a variety of both, and can be found at some markets or online (CRC). I save this for things like vanilla ice cream base or vanilla pastry cream.
Espresso powder: For years I kept a little jar of Medaglia d'Oro instant espresso powder in the freezer, a serviceable supermarket brand. However, I always found it to give baked goods a distinct mocha flavor. Then on a whim I bought Delallo instead (it was in the supermarket baking aisle), which is mild and wonderful and wholly recommendable. King Arthur makes one that is supposed to be more of a chocolate booster (OU; also on Amazon).
Lime juice: If you forgot to get some on your last grocery run (that would be me), 365 (the Whole Foods house organic brand) has a bottled lime juice (KSA). There is also lime juice powder for your pantry (OU).
Boiled Cider: Good in apple cider doughnuts and a nice alternative flavor agent for places you might otherwise add maple syrup. Mountain Cider Company can be found, for a price, on Amazon (KVH). Or make it yourself, it's an easy and satisfying DIY (I make less and skip the canning; just store in a jar in the fridge).
Gelatin + Marshmallow
Gelatin: I buy a fish-based Israeli brand at my kosher market. I've never seen kosher gelatin at a mainstream market in the US. This is a tricky one, and you'll need it for easy but fancy-pants dessert numbers like panna cotta (especially parve versions).
Marshmallows: If you are on poor terms with your candy thermometer and/or have a bunch of kids and/or need some in the pantry for hot chocolate emergencies, we like Elyon brand (OU) and Dandies (cRc). Oh Nuts carries a bunch of fun ones.
Marshmallow Fluff/Crème: Surprisingly, the classic American supermarket brand is kosher. If you want to make it yourself, I've had success with this simple recipe which, bonus, makes the equivalent of a store-bought container. (Add ½ tsp cream of tartar to stabilize.)
Pastes + Fillings
Cookie butter: I don't know of any national brands that are kosher. Biscoff does kosher runs of it, but they are not widely available. You can sometimes find them at kosher or Israeli markets (usually imported from Israel); same with the Biscoff biscuits themselves. This is not a DIY for the faint of heart. If you're not super committed, maybe just throw some petit beurre type biscuits and a shake of speculaas spice (premade here, OU) in the food processor? Tzarich iyyun.
Marzipan: Sweetened almond paste with a distinctive texture, you can sculpt it into candy, roll and bake it in yeasted cakes and breads, or add it to cake batter (think rainbow cookies). Love 'n Bake brand (OK-parve) can sometimes be found at the supermarket, or order from Nuts.com. Make your own from almond flour or whole almonds.
Almond paste: Similar to marzipan, but with less sugar and a slightly different texture; if you can't find Love 'n Bake brand (OK-parve) at the supermarket, you can order it from Nuts.com. Also easy to DIY.
Pistachio paste: Fiddyment Farms is a California-based brand with many products certified, including their pistachio paste, which you can sometimes find on Amazon. Love 'n Bake makes a parve pistachio paste (OK), available on Nuts.com and sometimes Amazon and King Arthur, you have to check. Or buy a bag of pistachios at Costco and make your own.
Dutch-processed cocoa: A lot of ink has been spilled on the merits and demerits of alkalized vs. natural cocoa powder. The latter is the common kind in the US, while alkalized tastes "Israeli" to me. Droste (OU) is the Dutchiest. I haven't tried the Guittard, which makes an intriguingly red Dutch process cocoa (OU-D, dairy equipment).
Chocolate melting discs: Guittard. I've never bothered because I find chopping chocolate oddly soothing, and more pertinent, those discs are pricey.
Specialty chips: There are parve ones out on the market, but don't bother, they're straight outta the Sigma catalog. If you have people in your life who don't do dairy, make them something with good-quality chocolate chunks instead. Most of the standard-issue certified dairy peanut butter and butterscotch chips are terrible as well, not that my kids notice. The best I've found are Guittard: they make white chocolate and butterscotch (OU-D). If you can find them, SunSpire makes peanut butter chips (OU-D; try Whole Foods or Vitacost).
Crystallized ginger: Theis is actually fairly easy to find with a hechsher. If you're in the Ralph's or Kroger neck of the woods, their house-brand organic label, Simple Truth, makes crystallized ginger (KSA). Amazon carries Swanson Organic (OK). Or, DIY, especially if you have a mandolin and need to justify its purchase/cabinet space.
Fondant: Satin Ice makes bulk tubs of it (OK-parve). Or DIY, in which case you'll need glycerin, on which see below (it's easy to find).
Glycerin: Wilton makes food grade glycerin, which you need a small amount of to make fondant at home (cRc-parve). You can get it at Michael's and other big-box craft stores in the cake dec aisle. Some glycerin is not made to be eaten; it's for skincare.
Glucose syrup: This one on Amazon is cRc-parve. You can generally substitute light corn syrup, which is easy to find certified, or light golden syrup in the UK. I keep meaning to see if agave syrup would work.
Diastatic Malt Powder: This is the bread and pretzel stuff, not the milkshake stuff (for which, see above). Modernist Pantry (their house brand, OU), King Arthur (OU), or Breadtopia on Amazon (UMK parve).