Spread the pound cake cubes on a tray and gently brush or spoon the jam over them to coat all sides. Arrange half of the jam-coated cake cubes in an even layer in the bottom of a large glass trifle bowl.
Sprinkle half of the sherry evenly over the cake layer in the bowl. This will add moisture and flavor to the dessert.
Scatter half of the fresh raspberries and sliced strawberries over the soaked cake layer, arranging them to create an even distribution of fruit.
In a large mixing bowl, vigorously whisk the instant pudding mix with the cold milk for 2 full minutes until the mixture is smooth and has begun to thicken. Let it stand for 5 minutes to set further.
In a separate chilled bowl, use an electric mixer to beat the cold heavy cream, sugar, and vanilla on medium-high speed until stiff peaks form, which should take about 3-4 minutes. Gently fold half of this whipped cream into the set pudding until no white streaks remain.
Spoon all of the pudding and cream mixture over the fruit layer in the trifle bowl and spread it into a smooth, even layer with a spatula.
Repeat the layering process: add the remaining jam-coated cake cubes, sprinkle with the remaining sherry, and top with the rest of the fresh berries.
Spread the remaining plain whipped cream over the top layer of berries, creating decorative peaks. Cover the bowl and refrigerate for at least 4 hours, or up to overnight, to allow the flavors to meld. Just before serving, garnish with the toasted slivered almonds.