Once your Million Dollar Pound Cake is fully cooled (and ideally baked the day before), cut it into neat 1-inch cubes.
Dry the cake slightly for better soak. Once your pound cake is completely cool, cut it into neat 1-inch cubes. Spread the cubes out on a sheet pan and let them sit uncovered for a couple of hours, or pop them into a 275°F oven for about 10 minutes just to dry the edges. You don’t want croutons, just lightly “staled” cake that can drink up flavor without falling apart.
Make a sherry simple syrup. In a small saucepan, combine the sherry (1/2 cup), water (1/3 cup), sugar (1/3 cup), and orange peel (1). Bring just to a simmer, stirring until the sugar dissolves, then remove from the heat. Stir in the vanilla (1 teaspoon) or brandy (1 tablespoon) and let the syrup cool to room temperature. This gives you all the sherry perfume with a round, soft sweetness and no harsh boozy edge. Prepare the fruit & jam layer. In a medium bowl, gently toss the raspberries, sliced strawberries, and any additional berries with the sugar (3 tablespoons) and sherry (or orange juice) (1-2 tablespoons) until lightly coated. Let the fruit sit for 10–15 minutes, just until it looks glossy and releases some juices. Meanwhile, place the raspberry jam (3/4 cup) in a small saucepan or microwave-safe bowl and warm it just until loosened and pourable, then let it cool slightly so it’s not hot when you assemble the trifle. Loosen the custard (if needed). Once the custard is thoroughly chilled, whisk it well to smooth it out. If it seems very thick and stiff, whisk in 1–3 tablespoons cold heavy cream, a little at a time, until it’s luscious, creamy, and easily spoonable but still thick enough to hold soft layers.
Build the first cake layer. Place about half of the pound cake cubes in the bottom of a large glass trifle dish (about 3 quarts). Spoon about half of the cooled sherry syrup evenly over the cake cubes, aiming to moisten them rather than drench them. The cake should feel tender and flavored, but not soggy or swimming in liquid.
Add jam, custard, and fruit. Drizzle or dab a thin layer of the warmed raspberry jam over the sherry-soaked cake cubes—it doesn’t need to be perfectly even; little pockets of jam are wonderful. Spoon about half of the custard over the cake and jam, smoothing it into an even layer. Then, using a slotted spoon, scatter about half of the macerated berries over the custard, leaving most of the extra juices behind in the bowl. Repeat the layers. Add the remaining pound cake cubes on top of the berries and drizzle with the rest of the sherry syrup. Follow with another light drizzle of raspberry jam, then the remaining custard, and finally the rest of the berries. If your dish is very tall, you can build thinner layers and repeat more times as needed, keeping the same general order: cake + syrup → jam → custard → berries. Chill the trifle. Cover the trifle and refrigerate for at least 4 hours, or up to overnight, to let the cake fully soak, the flavors mingle, and the custard set into dreamy, spoonable layers.
Whip the cream topping. Just before serving (or up to a couple of hours ahead), add the heavy whipping cream(1 cup), powdered sugar (3-4 tablespoons), and vanilla (1 teaspoon) to a mixing bowl. Whip with a hand mixer or stand mixer until soft-to-medium peaks form—fluffy and luscious, but not stiff or grainy. Finish and serve. Spread or pipe the whipped cream over the top of the chilled trifle in big swoops. Garnish with extra berries for a showy finish. To serve, use a big spoon to dig straight down so every portion gets some buttery sherry-soaked cake, rich custard, jammy berries, and whipped cream.