I’m not really into the niche of “virtual YouTubers” – people who post YouTube videos and/or stream using a virtual avatar – but to each their own, and if this technology enables people to remain anonymous while doing what they love on YouTube or Twitch, I’m all for it. Since these virtual avatars also do things like face-tracking, there’s a whole cottage industry of software tools to make this all work, but Adrian “asie” Siekierka decided to take a look at where the training data used to make such face-tracking work actually comes from.
One day, some years ago, I decided to look at the data used to train OpenSeeFace. OpenSeeFace is the most popular open source face tracking solution for virtual YouTubers. It is supported by both open source and commercial model rendering tools; in particular, VTube Studio allows using it as an option for webcam tracking.
↫ Adrian “asie” Siekierka
The results of the investigation are not exactly great. Much of the data used by OpenSeeFace comes with serious restrictions on commercial use, and many of the underlying datasets contain images that you would need consent for from the people inside the image to actually use. On top of that, a lot of these data sets seem to have just scraped the internet for images of faces without asking anyone of the people in those images for consent, which raises a whole number of troubling issues.
I find this a very interesting topic of discussion, if only because you’d be hard-pressed to argue that the average cartoon-esque virtual avatars even remotely resemble real human faces, so it’s not like you’re going to suddenly run into your own face somewhere on YouTube or Twitch, but plastered into another person. On the other hand, the underlying datasets still contain a ton of people’s faces without those people’s consent, and even for those that did give consent, there’s often a commercial use restriction which earning revenue on YouTube or Twitch might violate.
It’s a fascinating microcosm of a whole slew of issues we’re dealing with right now, neatly contained in a relatively small niche.
I was totally unaware of this space, but now vaguely knowing about it from reading this, what I find interesting about it is a potential, eventual, future in expanded VR (Coming Soon(tm)!) usage where we actually have face tracking/response while keeping the VR avatars not necessarily looking like the actual people. Think a bit like Snow Crash avatars, and the potential for some real non-verbal communication/cues that could be carried through VR, even with fantasy/etc type avatars.
I think it’s something that’s a potential nice, interesting building block that I’m glad some people are pursuing, even if I personally don’t think I’ll be using them in the current incarnations.