Pirate Cove
In this Pirate Adventure style concept we experimented with using photogrammetry for scene reconstruction.
The plate for this concept didn't just look cinematic, exotic and like great adventure... it also had a very interesting feature: in the full length source clip the orbiting drone move rotated by a full 90 degrees, making this plate suitable for a scene reconstruction using photogrammetry. Most VFX shots these days come with on-set lidar, but stock footage obviously does not. We had used photogrammetry for shots with witness cameras before, but not for single camera setups. This would be an interesting test to see if the geometry derived from a single clip would be good enough to be used in a VFX shot.
But back to the concept: after seeing the plate, we checked if kitbash.com had a pirate themed asset pack, and they sure did. As with most KitBash assets they didn't quite look photoreal, but we figured with a little up-resing and for a wide shot these would be OK. For the concept art we used both Stable Diffusion XL and Midjourney with image2image, and Midjourney was the clear winner: SDXL was not able to adapt to a given style while using image2image, and seemed stuck on a very simple looking illustration style. Midjourney already performed a bit better to start with - the trick was then to do one round of image2image first, which produced OK-ish results, followed by a remix allowing for strong variation. This produced great looking concepts with a given style, while sticking close enough to the original image.
Next we had a go at the scene reconstruction of the shot. We used RealityCapture, the fastest and most accurate photogrammetry solution on the market. It turns out that - as we had hoped - the 90 degrees orbit provided enough parallax and the 4K plate enough resolution to produce accurate geometry. It was actually so detailed that we had to down-res it in order to be practical. We then used Syntheyes to track the camera using the provided geometry as a baseline.
With all the technical stuff out of the way, we could now get to the creative part: creating a layout and then lighting it in Blender. The scene geometry was extremely useful here since it allowed us to nestle those shacks underneath the trees and to get correct shadowing and bounce light from them. We upres-ed the model of the ship to make it look less goofy, and did some additional cloth sims for the sails and flags in order to make the scene less static.
With the lighting done we then moved on to the comp, where we integrated the CG with the help of some roto. Since the 3D assets were a bit on the low res side, we decided to project additional hand-painted details onto them. We then added moving elements for flags, birds and smoke to bring the scene to life, and added a character in the center of frame to give the camera orbit motivation.
This shot was certainly fun to work on! We learned a lot, particularly while creating a workflow for RealityCapture to Syntheyes which we hope to refine in upcoming shows.