Synthesizing pill images using ThreeJS
Bulk synthesis of prescription pill images from existing set of images to be used for training computer vision models.
For
Microsoft Research
Role
Designer / Developer
Year
2019
Categories
- Web app
- 3D
- Tool
Tech
- Node JS
- Three JS
Objective
Our team was developing a computer vision model to identify prescription pill from photograps. However, we did not have sufficient real world images to train the model for each of the thousands of pills we wanted to support. So we wanted a system that can autogenerate photorealistic images of prescription pills from a set of source images. Images needed to be realistic enough to be used for training a computer vision model that will identify medical pills from photographs.
Research
01. Exploring possibilities
I researched papers on image synthesis, generative graphics to understand the nuances of creating photo-realistic images. It helped me outline a potential solution with 3 major steps
- Extract texture from existing pill image
- Create a 3D model for the pill type & apply the texture on it
- Build a system which renders this 3D model, applies environment, lights and camera to generate photorealistic images
02. Identifying technical solution
I identified ImageMagik as the right tool to generate textures from thousands of existing pill images. ImageMagik has enough features to let me further generate more assets like normal maps, height maps from the images, that can be used to render realistic 3D models. I chose Blender to build the 3D models of the pills and create meshes for different types of pill shapes like round, oval, capsule etc. Finally, I decided to use ThreeJS to
- Apply texture to the 3d mesh to generate the 3D model of the pill
- Put light, camera and realistic environment to create a 3D scene
- Use the scene within a NodeJS container to automate randomization of the lights/camera to generate images
Development
I built an app with ThreeJS that would
- take a folder of 3d pill textures as input
- select the right pill mesh
- apply the texture on the mesh to generate the 3d model
- put light and camera and environment map to make the 3d scene realistic
- put normal map and ambient occlusion maps on the pill to make the images respond to light in a realistic way
I created a web app with NextJs which will
- orchastrate the movement of camera/lights within the ThreeJS app, to randomize photos
- Save the images on a remote server to be picked up by computer vision training scripts
Finished product
My solution enabled our team to generate thousands of photo realistic pill images for all the pills we wanted our CV model to support. The tool can generate about 200 images per minute, helping our CV engineers to train the model and improve the accuracy.