Filling in the sky on drone generated equirectangular panoramas

The following outlines one possible way of filling in the sky from drone generated equirectangular panoramas. While simple blue skies can use blending approaches, as used here, the approach described works for more complex clouds. The general principle is to extract a perspective view, fill in the clouds, and then re-project back into equirectangular. This is a common work flow for editing out the tripod legs, as presented here, where one transforms to/from cube maps. The difference with drone panoramas is the "hole" is typically much larger, even larger than the 90 degree field of view of the top face of a cube map. The "obvious" solution is to extract a much larger field of view perspective projection. The stitched drone image below will be used to illustrate the work flow.

The following is a 130 degree perspective projection of an up pointing virtual camera. There is nothing magical about 130 degrees, but the field of view needs to be large enough that the content aware fill has enough imagery to be successful. This perspective view is extracted using the authors command line tools, in this case "sphere2persp", but there are many other options.

   sphere2persp -w 4000 -h 4000 -t 130 -x -90 original.jpg

Probably obvious but the x axis is to the "right", so -x -90 pans the camera up 90 degrees. The -t option specifies the horizontal field of view, which is this case is the same as the vertical field of view since the requested image is square (determines by -h and -w).

In this example the hole is filled using the context aware fill in Photoshop, but similar capabilities can be found in many other image editing packages. In this case a circular lasso was created around the hole, it was feathered in order to avoid a sharp transition, the result of the context aware fill is shown below. In this case no further adjustments were necessary but in some cases one may need to tweak the result to hide any artefacts.

The above perspective image now needs to be mapped back into an equirectangular projection with exactly the same orientation and parameters as the forward transform. Again, in this case the authors tool "persp2sphere" is used but other options are undoubtedly available.

   persp2sphere -w 10000 -x -90 -t 130 filled.jpg

The requested width (-w) of 10000 matches the width of the original equirectangular. Obviously the authors "persp2sphere" and "sphere2persp" are symmetric, that is, if one conversion is performed and then the other then the result aligns perfectly with the original.

And finally, the above is blended with the original equirectangular image.