Lower the boundaries to create and train AI/ML models

Computer monitor displaying 3D modeling software with a red agricultural sprayer and green wireframe trees in the scene, against a green gradient background.

Overview

The Synthetics team was a small group of individuals looking to build a novel solution for training computer vision models. I came on as the sole designer to help craft a more intuitive experience by ideating on previous designs and refining the overall direction.

Role

Lead UX Designer

Timeline

4 months

The challenge

Training ML models take a large amount of data, time, and specialized skills leading to long timelines and barriers to entry for some organizations and people

What is Synthetics?

A small team building a tool that allows digital artists to create 3D models of real-world objects and synthetically put that object into various environments with different conditions such as lighting, backgrounds, and orientations to train ML models without requiring real-world training data.

This digital training data was scalable, perfectly labeled, and allowed for quicker training cycles.

Four LinkedIn profile cards for different professionals, each with a profile picture, name, title, and icons for messaging, team, video, call, and LinkedIn.

What we’ve learned

  1. Customers are entering from ML teams, where talent is in demand and supply is low

  2. Real world data does not scale

  3. Dataset iteration is a new process

Solution goals

Black and gray business card with a square logo on the left and lines of text on the right.

Template based solutions

Allowing users to select from pre-made templates affords the ability to do much of the heavy lifting before the user starts

A black-and-white icon of a stopwatch.

Shorter iteration cycles

Iterating on training sets for ML models is necessary, but there is an opportunity to reduce the time between cycles

Outline of a magnifying glass.

Improve the debugging process

The debugging step requires highly trained ML experts currently; we aim to provide tools so more people to do this

Target users

A woman working at a computer with code on a monitor in an office setting.

ML Model Experts

  • Domain Familiarity

  • Knows about the entire creation process

  • Knows how to debug

A man in black shirt using a computer with a large monitor displaying a 3D CAD design of a mechanical component.

Digital Artist

  • Low knowledge of ML model training process

  • 3D modeling/environment expert

  • Does not know how to debug

The previous MVP

The team had already created a proof-of-concept using existing tools to prove that this new approach was possible

Initial thoughts

Going through previous concepts with the team, I was able to identify where they wanted to go, the problems they faced, and uncover potential design opportunities

Screenshot of a user interface for choosing a recipe. It includes filters for description, rating, region, engine, contributor, ingredient, components, supplementary, and manufacturer. There is a list of recipes with images, titles, authors, submission dates, ratings, and download counts. Handwritten notes on the right mention setting the category to 'Mazy,' with options to add a GUI and search, and buttons to preview, download, or close the interface.
A digital interface with preset and adjustable lighting cages, a 3D globe model, and a lighting positioning diagram. The interface includes options for day, night, dusk, indoor, fluorescent, LED lighting. Handwritten notes describe features like weight adjustment and HDR imaging.
A screenshot of a digital interface with handwritten notes and sketches. The main window shows a diagram of a ring-shaped object with blocks along the perimeter. Text in red on the right lists features like main interface, model/scene preview, variability weight, console, and pipeline flow. Additional handwritten notes are on the margins, including a challenge note about opening in a new window.
A notebook page with various scientific graphs, charts, and handwritten notes. There is a 3D scatter plot, heatmap, bar graph, and line plots. A note written in black ink says 'Model Performance.' A highlighted comment states, 'This was an area I flagged that had a significant opportunity.'

Ideation & Refinement

Starting with pen and paper, I worked with key stakeholders to reimagine the framework of this new tool incorporating feedback and refining the concept

Sketching

Used low-fidelity sketches to work quickly and efficiently to include all the necessary parts

Hand-drawn wireframe of a software interface with sections labeled for current project, catalogs, extensions, debug, account, settings, a status bar, a ribbon, a command bar, and icons representing project and model details, assets, and project assets.

Refining layout

Created wireframes with more real-world data to validate design frameworks and determine information hierarchy

Screenshot of a 3D modeling software interface showing a scene with a partially assembled 3D Rubik's cube surrounded by numerous wireframe models of cars in a virtual environment.

Finalized framework

Refined content layout and product flow based on continuous stakeholder back-and-forth

Screenshot of 3D modeling software showing an array of green wireframe objects surrounding a colored Rubik's Cube with a Microsoft logo on it.
Computer monitor displaying 3D design software with a 3D model of a red tanky device floating above a wireframe landscape, set against a green gradient background.

Final design

I was able to successfully deliver on building a comprehensive design encompassing all the product's key user flows and provide a prototype to be used to pitch further investment from leadership

Computer monitor displaying a software interface with categories such as Visual, Audio, Data, and Open project. The interface includes templates for object detection, human understanding, geospatial analysis, sky monitoring, and object targeting.

The home page

The home page allows users to choose premade templates to kick off their training projects. They can choose from different categories and start a project with a sample model or import their own.

3D rendering of an agricultural machine or equipment, with a red cover, surrounded by green wireframe models of other similar objects against a light gray background.

A new view

Giving users a 3D perspective into how their digital object will be turned into training data

Screenshot of 3D rendering software displaying a model of a red water tank mounted on a metal frame with wheels.

Plug-ins for the best 3rd party tools

Allowing users to leverage the industry-leading tools they are accustomed to

A computer screen displaying a lighting catalog with six images of nighttime outdoor scenes, each with spherical light reflections in the foreground. The scenes include a lit palace, park, rural landscape, city street, and open land, all under dark skies.

Asset libraries

Catalogs of digital objects, lighting, sensor, camera, and background packages

Screenshot of a software interface showing two sections titled 'Added Fire Extinguisher' and 'Next iteration' with their details and testing results histograms, including bar charts with confidence scores, false positives, true positives, and false negatives.

Debugging made easy

An intuitive way to understand how each build performs and, most importantly, a way to understand what went wrong

The debugging feature

The debugging feature was a massive area of opportunity for design innovation. After learning how it's currently handled, we realized there were very few best practices in place, and it requires an individual with immense AI/ML knowledge and training.

Two screenshots of a computer interface showing image analysis of a brown outdoor cart on a brick ground. The first screenshot displays a bounding box around the cart with confidence at 12%. The second screenshot includes a heatmap overlay highlighting areas of interest on the cart, indicating where the analysis detected key features.

First approach

First iterations were focused on individual instances and aimed to communicate what the model was seeing

This was helpful but did a poor job of highlighting patterns from the whole training set

Multiple screenshots showing data analysis, neural network training, testing results, warnings, and error messages related to deep learning model performance and validation.

Going back to the drawing board…

Focusing on surfacing insights to the user about where the model was failing

Designing to give a health check for the entire training cycle

Two histograms showing testing results with confidence scores. The top histogram indicates few false positives and some true positives, suggesting the model performs well in identifying target objects. The bottom histogram shows a high number of true positives and minimal false negatives, further indicating good model performance.

Final debugging feature

This histogram approach allowed for a better high-level understanding of each training cycle and communicated where the model was having issues

What we accomplished

In a short amount of time, we were able to get buy-in from key stakeholders with the improved designs for Synthetics. Training computer vision models started with a highly manual process requiring much of the effort spent on establishing infrastructure. We reduced the time spent training models from months to just days, completely removing the manual component.

Impact

  • Democratizing computer vision model creation

  • 3D artists can now train ML models

  • Reduced the iteration cycle from months to days

Lessons Learned

  1. Designing novel tools is an iterative process

  2. Model Explainability is key

  3. As ML tools evolve, so do the users