Weeknote 01/09/25

nerfstudio preprocessing

Through this week, Bea has been setting up the nerfstudio environment so we can use their preprocessing to train some 3DGS using Brush, which accepts data in the nerfstudio format. Currently, our workflow involves aligning the images in Metashape and using nerfstudio to preprocess the data.

3D printing and designing our dome

The first version of our dome was 3d printed once again, this time by Anna, who split it into 8 smaller parts and added snap connectors so we could try assembling a smaller version of the dome. After our meetings on Thursday (see below), we decided to redesign our dome to only have light sources around the camera for better lighting, a smooth sphere for a more uniform background (so light wouldn't bounce differently on the sharp edges) and to scale it down by a few centimeters (<50 cm in diameter) so it would fit in the printer when spliced into 8 sections. Some modifications were made to the phone holder to allow a wider range of phone sizes to be used. After we agreed on a redesign, both Anna and Arissa started working on a second version (Anna in Fusion, and Arissa in Blender), with the plan to print both versions at a smaller scale before deciding which one to go with. We are hoping to have this done by the start of next week.

Photo 1 Photo 2

iPhone auto camera app

Arissa finished writing a camera app for the iPhones, that has the option to choose the time interval at which photos should be taken, and auto focuses on the specimen before taking photos. This is so the person working on the scans would need to interact with the setup as little as possible, so they could just start the process of taking photos and the phone would keep going until they stopped it.

After finishing the camera app, it was added as another functionality to the 3D scan catalogue displaying app Bea made prior to our break.

Photo 3

Thursday meetings

We went to the Insect Ecology and Agroecology lab meeting this week, where Jerry Liu presented his findings from the past nine weeks. He gave an overview of drone images and LiDAR data from the RERTA river margin restoration project in Indonesia that the Insect Ecology group has been working on.

Later that day we also met with Anil Madhavapeddy, Alex Ho and Michael Dales who gave us some invaluable input into the 3D printing and designing of our dome, lighting and photography system.

Segment anything in 3D

A few weeks ago, Sadiq mentioned that he would be interested in whether Meta's new tool, Segment Anything, would be useful in identifying body parts of insects and segmenting them for further measurements and analysis. Since Segment Anything is designed for images, Bea looked into 3D alternatives and found SAM3D and SA3D. SAM3D works on point clouds with posed RGB images by first predicting 2D segmentation based on the posed images and then projects the 2D masks onto the 3D point cloud. From there, the 3D masks of partial scenes (from individual angles) are merged with adjacent frames to generate 3D masks of the whole scene. The model doesn't require fine-tuning or training so it appears to be simple to use once the workflow is established. SA3D works on a NeRF or 3DGS, so would be suitable for the models we aim to produce. It boasts 3D segmentation within minutes, although it doesn't mention any hardware requirements.

Cataloguing

Cataloguing efforts have continued this week, with Anna finishing off the Cream Wave specimens (Scopula floslactata) and moving onto the next column of Smoky Wave (Scopula ternata) specimens and finishing off that too. Bea did some cataloguing of Rusty Wave (Idaea inquinata) specimens and Small Fan-footed Wave (Idaea biselata) specimens.

Insect corner

Insect fact of the week: When Elephant Hawk-moth caterpillars feel threatened they can partially retract their head and puff up their bodies to mimic a snake!

Photo 4