Whilst we finished scanning the specimens we had already scanned with the iPhones and a couple of new ones with the scAnt last week, we managed to process all the photos taken and complete the 3D reconstructions this week. We tried completing the processing on a gaming laptop last week, but it wasn't working. Instead, we used the gaming pc in the office upstairs and managed to get 3D models from the scAnt data.
Top to bottom: Buff-tailed bumblebee, Julodis viridipes, Elephant Hawk-moth.
This week Bea has been working on adding 3D model viewing functionality to her app. You can view 3D models on a white background, pan and magnify the models. Since ARKit recognises very similar physical motions as pinching and rotation, there is code to rotate the model too, but it seems impossible to activate it. Not featured in the photos, we have started working on using the moth color scheme here too, so that the two apps may be integrated.
This week we read a large range of gaussian splatting research papers mainly on different variations of gaussian splatting and real-world use cases. These often compare results to NeRF models and photogrammetry. We also read papers focusing on 3D reconstruction of insects and found prices of current existing methods. Reading these also allowed us to become more familiar with the typical methodology of both types of papers.
We've looked into trying to train our own gaussian splatting model, and installed Brush. This can be used for 3D reconstruction using gaussian splatting and for training a gaussian splatting model on our data. We've used it to display 3D models from the .ply GS file format, and are going to install nerfstudio to convert from the files we have from the iPhone/scAnt scans to a file format acceptable by Brush for training.
After our meeting with Anil and Sadiq last week, Anil has put us in contact with Aurojit Panda, a researcher at NYU working on a distributed method of training gaussian splatting models. We are looking to send over the data we have collected on the iPhones and scAnt and get help with training another model. We also considered the idea of writing a paper comparing the two trained models and the general polycam one, if we manage to get them.
We had our bi-weekly meeting with Ed on Thursday morning to let him know about our progress before our break in the next 4 weeks. We also had our lab meeting at lunch, where Tiff presented the scans we've managed to get so far with the scAnt and iPhones, and we talked about what we've noticed about the various methods used so far and what we've worked on.
Insect fact of the week 1: The caterpillars of the Large Blue drop to the ground after three weeks. Here, they give off a scent that attracts red ants and tricks them into thinking they are a queen in distress. The red ants bring them back to the colony where they feed on ant larvae. By the time they enter the pupa stage, they will have consumed most of the colony's larvae.
Insect fact of the week 2: When dragonflies die, they lose their blue colour because the structure that causes it gets disrupted. Here you can see a stacked photo of a pinned Azure Damselfly taken by scAnt this week:
On Monday we sidequested to the MAA to demonstrate how we'd create a 3D model of a shadow puppet. It was lovely to meet Mei and Mark from the museum, although our model wasn't the best. Taking a scan of a larger object was quite different and proved to be a challenge, but it was still fun!