MobiDev, a US-based software engineering company with development centers in Ukraine, shares the experience of developing the virtual fitting room solution with ARKit and ACGPN model.
The virtual fitting room technology market provides offerings for accessories, watches, glasses, hats, clothes, and others. The logic of this solution can be considered in the following example. MobiDev had a project of Augmented Reality (AR) footwear fitting room development, which works in the following way:
- The input video is split into frames and processed with a deep learning model which estimates the position of a set of specific leg and feet keypoints.
- A 3D model of footwear is placed according to the detected key points to display the orientation to a user naturally.
- A 3D footwear model is rendered so that each frame displays realistic textures and lighting.
In the course of development MobiDev came across two main challenges: the tracking accuracy was too low to use it for footwear positioning and body parts were poorly identified by the ARKit algorithm. The conclusion was that virtual fitting room apps might require additional functionality along with the standard AR libraries.
Since there are no ready pre-trained models for the virtual dressing room, another example would be with the use of ACGPN model. MobiDev’s idea was to explore outputs of this model in practice for 2D cloth transferring by utilizing various approaches. The model was applied to people’s images in constrained (samples from the training dataset, VITON) and unconstrained (any environment) conditions. In addition, the limits of the model’s capabilities were tested by not only running it on custom persons’ images but also using custom clothing images that were quite different from the training data.
The main limiting factor that holds back the development of better models is the lack of diverse datasets with people captured in outdoor conditions.
More detailed overview of virtual fitting room examples and research results can be found at: https://mobidev.biz/blog/ar-ai-technologies-virtual-fitting-room-development
Contact:
Nana Hrytsenko
8883800276
