The original intention of using photospheres was to make photographs, especially site photographs, more navigable, and capture as much contextual information as possible. On the whole, this has proved really effective for certain kinds of projects where site access is more difficult, or spending extended periods of time at a site isn’t desirable.
However, dark basements were foiling this approach. I wasn’t able to light the scene simultaneously, so the result was a series of partially lit photospheres (with 75% or so of the photosphere in complete darkness). Each of these was no more useful than a non-contextualised photograph, I have several photospheres that I cannot accurately place.
Fortunately, I was able to draw on my previous work on with LightTraceMemory. Here, a multiple exposure technique was used (employing processing) to overlay several images. Whe the camera is kept exactly still on a monopod, and several photographs are taken with the lighting cast around the camera so that the entire scene is lit in at least one photograph of the series, then these can be overlaid to create a single, fully lit, photosphere. To overlay, the same pixel is compared in all the images, and the brightest one chosen for the final image.
However, my PPE (orange high visiblity shirt and trousers) were causing an issue. Frequently, the brightest of the pixels from the images would be my, or someone else’s, orange PPE from a photograph where they were accidentally partially lit. This lead to useful detail being obscured behind bright orange streaks. As I come across bright orange objects quite infrequently in the basements, a good way to differentiate between useful and non-useful data is how orange it is: useful information is rarely orange, non-useful information frequently is. Therefore, I needed a way to filter out pixels that were ‘too orange’ from each image. I achieved this crudely, using the dot product of the normalised RGB vector of the pixel color and a normalised RGB vector for pure orange, and setting a limit on this, beyond which it would be rejected – 0.9 to 0.95 seemed to work well.
Taking a typical scene (outside this time, as there are a few more orange objects to choose from), and replacing any ‘too orange’ pixels with pure black (note the barriers, my PPE, and the yellow on the legible London sign):
Typical outdoor scene with ‘too orange’ pixels removed (as part of testing)
However, this technique proved good enough, although it did tend to ‘rub out’ brickwork that was quite orange in colour. Therefore, I added one more step – accumulating the rejected ‘too orange’ pixels onto another overlaid image (taking the brightest for each pixel location), and adding in the pixels from the ‘too orange’ image as a final step for any pixel locations where the existing overlay is near-black (so no pixel data has made it through the ‘too orange’ filter for any of the photos, or that area has remained unlit in any photos, so doesn’t carry any useful information anyway). This helps to fill in some of the areas of brickwork and other fairly orange items, without allowing PPE to obscure detail on other photographs.
Basement example 1, 5 photos are taken with different parts of the scene lit by a lamp:
For each pixel, the orange pixels are filtered out, and the brightest remaining pixel is left, creating an overlay:
Overlay, without any ‘too orange’ pixels
Overlay of the ‘too orange’ pixels
Where the original overlay is black, and there is information in the ‘too orange’ overlay, the orange overlay is added back on, a little more detail is preserved. Note that the original overlay may not have been dark enough in some areas (e.g. the inner parts of the ladder) so the ‘too orange’ overlay has not been used. This can be corrected, where useful, by increasing the threshold at which the ‘too orange’ overlay is used (for this overlay it was set to almost completely black, so quite strict).
These overlaid images can be run through the previous photosphere scripts to create a 360 view using krpano. The photospheres shown in this post are all available at this link [User: UsefulSimple, Password: RrlMrKArf2eP] to see the different images and overlays.
Other technical issues, before I forget that I dealt with these:
- the Hugin scripts used to process the Samsung ‘2 circles’ format of photosphere image in a equirectangular projection used by krpano required the exif data for each image to list the camera model as a Gear 360, this meant that all of the photos produced using the processing script, initially, would not pass through the existing scripts. I used a check within the python script to ensure every image passed to the Hugin scripts had its exif data set correctly.
- the process of overlaying the photos usually took place off the memory card of the camera, so the suggested directories for the photo download locations did not work as expected when I ran the code without the memory card in place
- when taking the photos (e.g. on stairwells and in basements) the typical timing of the photos came in a series of bursts (e.g. 5 photos at approximately a 3 second spacing, followed by at least 15 seconds whilst I moved the monopod to the next location). I used sklearn’s implimentation of the k-means algorithm to group the photos automatically using their exif timestamp for each of them. This can easily be taken much further if I want to do so later, for example:
- taking the distance from the group mean for each point, decreasing the number of means from the number of photos until the largest and second largest group mean to point distances increase very suddenly (as the k-means will suddenly be forced to combine two bursts of photos into one scene). Use this to count the number of scenes rather than doing this manually.
- making a small GUI that allows the user to change the number of scenes manually (maybe with some help from the above program), and manually assign images to scenes if needed