Tapestry would train an open sovereign model without sharing data

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 28:29

   As if that wasn't enough to keep you busy. Well, it's kind of an idea I've been forming for the last three years or so. It's the fact that people increasingly use AI assistants for various things, right? I mean, you see a decrease in the use of traditional search engines, and you just ask a question to your favorite AI assistant. and you know if the plan that meta and others are are developing of you know having smart devices like smart glasses and stuff like that uh you know is realized basically you'd just be talking to your ai assistant you know by voice with you know to your smart glasses or maybe some other smart device and so all of your information diet will be mediated by ai assistants and And if you are someone, you know, somewhere in the world, let's say outside the US or China, and you have an AI assistant, and that AI assistant was built in California or, you know, Beijing or Shanghai or Shenzhen, it's not good for you.

2. 29:38

   Like you may speak a language that those systems really haven't been trained to handle particularly well. You may have a culture that is not particularly well understood by people in Silicon Valley and China, not well represented by the training data that is publicly available on the Internet. You may have a value system that is absolutely not represented by, you know, people building those models.

3. 30:03

   And certainly you will almost certainly have political opinions that are absolutely not represented by the handful of AI assistance you might be able to get from the, you know, West Coast tech companies or from Chinese companies. So what is the solution to this? Like how do you serve, you know, a farmer in India or even a philosopher in France or Germany? And what you need is a platform, which basically is an open, free foundation model, LLM style, that is fine-tunable by anyone to cater to the interest of people speaking a particular language, having a particular culture, having particular value systems, political biases, creeds, whatever it is.

4. 31:02

   And so what you need is a wide diversity of AI assistance. There's a lot of countries around the world that are neither the US nor China who absolutely want some level of sovereignty for AI, not just for the industry, but also for the citizen. They don't want the citizen to get brainwashed by a Chinese model or a Californian model, actually. And so they want sovereignty. How do you get that? So the way you get an open platform like this to get to the frontier is you just train it on more and higher quality data than the proprietary systems.

5. 31:43

   If you talk to people in India, in France, in Vietnam, in Morocco, in Switzerland, in Korea, Japan, Kazakhstan, everyone... wants basically sovereignty and you tell them like you guys have been training your model you know locally you don't have to share your data so that's the crucial aspect of tapestry you would have international contributor contributors to tapestry contributing to training a global model that would basically constitute a repository of all the world knowledge and culture, if you want.

6. 32:26

   But the contributors would contribute data and computing resources, but they would preserve the control on the data. They would not have to share the data with the other contributors. What they would contribute is parameter vectors. So it would be a kind of federated learning style thing where you have a bunch of data centers, they get the parameter vector from the global consensus of a model. Think of it as an average of all the parameter vectors of all the contributors, right? So all the contributors periodically tell everyone else through maybe a central server, here is my parameter vector. What is yours? Okay. And so you exchange parameter vectors like this. And a local worker basically, whenever it updates its parameter vector, it tries to also make it as close as possible to the global consensus vector.

7. 33:28

   So as the training of this thing kind of progresses, all those parameter vectors converge towards like a consensus model, essentially, which is kind of repository of all human knowledge. Now you have an open model that is as good as if it had been trained on all the data in the world. and now you can fine-tune it for your own purpose your own political cultural and linguistic biases whatever you want or centers of interest and i think there is a natural force for this to happen because you know most countries that are not the u.s nor china want sovereignty but also because ai is fast becoming a platform and there is a natural tendency for platforms to become open that's what happened with linux right and that's what happened with the software infrastructure of the internet or the wireless network it's all open source um it was proprietary

8. 34:30

   initially but that was all wiped out it's a really clever way to get around uh you know what seemed like this trend of you know decreasing open source and yeah and obviously i think there's been many fears that as like the closed source models get better they'll be held back and And they'll be used to train the next generation. And, you know, they'll kind of be this almost like escape scenario for closed source models where they get, you know, so much better than their open source counterparts.

9. 34:53

   So remember what, you know, who the big players of the Internet infrastructure were in 1996, Sun Microsystems, HP, Dell, and a few others. So Sun Microsystems was selling you Solaris with their, you know, proprietary hardware, HP with HP UX. They were claiming, you know, Unix is so much more reliable than Windows. You're not going to run a web server on Windows. Dell was doing this, you know, with Windows NT.

10. 35:22

    They're like, who is running Windows NT now as a web server? All of this was totally wiped out by Linux. Like, the entire internet runs on Linux. Even Azure, right? Even Microsoft runs Linux. So basically, OpenAI, Anthropic, et cetera, today are the Sun Microsystem and HPUX of yesterday. Yeah, I mean, I guess implicit in that is obviously, you know, I think your view of like the limitations of what like, you know, these models can only get so good.

11. 35:59

    and so