This post is going up nearly a month late — first it sat on paper for a few weeks, then right after I scanned it with AI the blog itself went down, which set me back another few weeks. At least it's fixed now, so I can finally stop turning it over in my head.
Original, Revised (Accurate Transcript + AI Polish)
In my view, anyone who supports “anti-distillation” has to answer “three hard questions” — namely, “a faulty analogy, a disregard for the law, and an unearned name”.
1. Faulty Analogy: A False Comparison
What do I mean by “faulty analogy”? People who back “anti-distillation” love to compare it to “copying homework” or “cheating on an exam” to argue that distillation is somehow illegal. But I think that comparison misses the mark: in those examples, the person who copies homework or cheats gets a final result that doesn't match what they actually know, learns nothing real in the process, and also hurts things like exam fairness. In model distillation, by contrast, the distiller genuinely gains knowledge and ability, and doesn't hurt the fairness of the rules. What it actually resembles is a student learning from a teacher: the teacher (a high-capability model) produces output (training data), and the student (a lower-capability model) learns from it and applies the knowledge. By the “anti-distillation” crowd's logic, everyone on Earth would have to wear the “cheater” label!
2. Disregard for the Law: Contracts and Clauses
Someone might object: “When a student learns from a teacher, the teacher agreed to it, and usually some fee was paid.” Then let's actually look at how distillation works: the student's parent (the LLM user) pays a fee (a subscription) to the teacher (the high-capability model maker), and lets the student (the small model) study the teacher's output. Doesn't that fit normal business logic? Or are we supposed to tear up the laws that already exist and prop up the “prohibited use” clauses some vendors unilaterally assert in their take-it-or-leave-it terms? That is what I call “disregard for the law.”
3. Unearned Name: Double Standards and Long-Arm Jurisdiction
Now let's talk about the “unearned name.” I have to ask certain companies: didn't their own models “distill” from open-source models at some point? Yet now they bite the hand that fed them — isn't that laughable? What's more, today's large AI models are, at their core, distillations of human knowledge to begin with. And some “anti-distillation” companies routinely scold distillers with one hand while using “burn-after-reading” tactics to dodge laws and regulations with the other — that's some well-practiced double standard. Then you have certain government agencies waving their domestic laws across borders in a bit of “long-arm jurisdiction.” Their true intentions, as the saying goes, are obvious to every passerby!
Expanded Version (I)
“
Revision notes: Kept the original structure and biting tone, smoothed out some of the long sentence transitions, strengthened the classical echo of “if the name is not right, the words won't ring true,” and added a closing line so the piece comes full circle with more force.
On the Three Hard Problems Facing “Anti-Distillation”
The debate over “knowledge distillation” in large AI models is getting louder by the day. In my view, anyone who supports “anti-distillation” and wants to stay coherent first has to clear “three hard hurdles” — namely, “a faulty analogy, a disregard for the law, and an unearned name”.
First is the “faulty analogy”: a bait-and-switch of concepts and a comparison that doesn't hold.
Supporters of “anti-distillation” love to reach for “copying homework” or “cheating on an exam” as everyday comparisons, trying to slap an immoral or even illegal label on model distillation as a technique. Yet this comparison is a textbook case of misrepresentation: in everyday life, someone who copies homework or cheats uses shortcuts to steal a fake result that doesn't match their real ability. They don't actually internalize any knowledge, and they undermine the fairness of the selection process. Model distillation, by contrast, has the distilled “student model” genuinely rebuilding its weights and internalizing its logical-reasoning ability, then getting tested for real in public benchmarks — so where is the harm to the fairness of the rules? This actually looks far more like how human education is supposed to work: the teacher (a high-capability model) provides lectures and worked solutions (training data), and the student (a lightweight model) grasps the method and applies it to new problems. If we follow the “anti-distillation” logic to its conclusion, wouldn't everyone on earth who ever learned from a teacher or drew nourishment from the works of predecessors have to wear the “cheater” label?
Second is the “disregard for law”: trampling contracts and declaring private dominion.
Someone might object: “When a student studies with a teacher, the teacher has to agree and the student pays tuition.” Then let's take a closer look at the real commercial loop behind model distillation: the developer (like a student's parent) pays a high-capability model maker real money in API fees or subscriptions, buying its compute and output services, and then lets its own model study the reasoning from that output — cash in one hand, information services in the other. How does that not fit the most basic logic of business and contracts? Stepping back, the big vendors themselves declare in their user agreements that they “make no claim to copyright over model outputs,” then turn around and use take-it-or-leave-it terms they wrote unilaterally to strip users of how they can use the output they already paid for. Setting aside the existing principles of copyright and contract law just so a few giants' monopolistic self-interest can be blessed by public legal doctrine — that's what I call “disregarding the law.”
Third is the “unearned name”: double standards and sheer bullying.
As the ancients said, “If the name is not right, the words won't ring true.” I'd like to ask certain business giants who wave the “anti-distillation” banner: back when you were starting out, did you hold back from fine-tuning and distilling open-source code and models? Now that your wings are full-grown and you've built high walls, you put on this “bite the hand that fed you” act — isn't it laughable? What's more, the essence of every large AI model today is, without exception, a “super-distillation” of thousands of years of human knowledge and public internet data. Certain companies shout “fair use” while scraping web-wide data without authorization, and even delete original training data or go “burn after reading” to dodge copyright accountability; but when others learn from their model outputs, they suddenly scream “intellectual property theft.” Having taken double standards this far, some even team up with politicians to try to use one country's domestic laws to impose “long-arm jurisdiction” over the global open-source ecosystem and other nations' technological development. Such conduct has long since left technical ethics behind; it's just using compliance as a cover for tech monopolization. As the saying goes, their true intentions are obvious to every passerby!
Expanded Version (II)
“
Revision notes: If this piece is meant for publication, debate, or academic/industry exchange, the other side will often push back from angles like “freedom of contract” and “the commercial value of synthetic data.” Within your “three hard problems” framework, this version fills in the copyright-law reasoning (the idea–expression dichotomy) and the history of technical evolution (the synthetic-data flywheel), making the argument airtight on a professional level.
On the Three Hard Problems of “Anti-Distillation” in Large Models: Faulty Analogy, Disregard for Law, and an Unearned Name
Recently, several leading closed-source large-model vendors have launched a public-relations siege and terms-of-service crackdown on “model knowledge distillation.” Yet if you examine it from three angles — technical principles, legal logic, and industry ethics — the “anti-distillation” position actually falls into a hard-to-justify “triple bind” — namely, “a faulty analogy, a disregard for the law, and an unearned name”.
1. Faulty Analogy: Confusing “Result Fraud” with “Skill Acquisition”
Supporters of “anti-distillation” love to reach for “copying homework” or “cheating on an exam” as a mass-market analogy to paint distillation as illegitimate. But that analogy completely fails at the level of underlying logic.
Cheating is, at bottom, “fake results without effort” — the cheater skips the process of building real ability and steals an evaluation that doesn't match their actual level. They learn no skill at all and directly damage the reliability of the selection system. Model distillation, on the other hand, is at bottom “internalizing ability through expert demonstration.” In distillation, a teacher model (a high-parameter model) outputs chains of thought (CoT) and reasoning patterns, and a student model (a small-parameter model) learns those logical distributions and, in doing so, optimizes its own neural-network weights. It not only genuinely gains the generalization ability to solve new problems on its own, but also creates real incremental value by cutting compute costs and pushing on-device AI into wider use. To smear a teaching process that “teaches a person to fish” as “cheating” is no different from branding every human learning behavior — reading the classics or studying under a mentor — as “academic plagiarism.”
2. Disregard for Law: Elevating Unilateral Form Contracts Above Statutory Rights
The second line of defense the other side often raises is commercial contract: “Even if learning from a great teacher is reasonable, it still requires the teacher's consent; since the vendor's terms of service (ToS) already forbid using outputs to train a competing model, distillation is a breach of contract.” That argument sounds reasonable on its face but actually falls into the “disregard for law” trap — it tries to use unilaterally drafted take-it-or-leave-it clauses to hollow out the fundamental principles of existing intellectual-property law.
First, from the standpoint of commercial consideration, developers who get training data through an API have already paid full commercial fees based on token count, completing a standard “paid procurement of services” loop. Second, from the standpoint of copyright law's core principles, the law protects only original “expression” and does not monopolize the underlying “ideas, logic, algorithms, or methods of reasoning.” What's more, mainstream judicial practice today generally holds that purely AI-generated content has no copyright, and major vendors' disclaimers typically hand the usage rights of outputs over to the paying user anyway. If vendors don't even hold exclusive statutory copyright over the output text itself, then what gives them the right to use a one-sided, competition-restricting form clause to forbid others from learning the “reasoning logic” within it? If we allow giants' terms of service to expand without limit and override statutory fair use and antitrust principles, that is plainly a distortion of normal commercial order and the spirit of the rule of law.
3. Unearned Name: Standing on Humanity's Commons While Biting the Hand That Fed You
The hardest thing for the “anti-distillation” argument to sell is the “unearned name” of its own advocates.
From the standpoint of technical lineage, no closed-source large model today was built from nothing. Every large model in the industry is, in essence, an unprecedented “super-distillation” of thousands of years of accumulated human civilization, open-source community code, and the public corpus of the global internet. Even now, when synthetic data has become an industry standard, leading vendors themselves — without exception — cross-use multi-source generated data, including from open-source models, for data cleaning and capability distillation when training new models.
When these same vendors scrape web-wide copyrighted works and personal data, they wave the banners of “advancing technology for all humanity” and “fair use,” even going so far as to delete original training sets (“burn after reading”) to dodge judicial review and copyright tracing. But when latecomers and the open-source community pay for API access to learn their reasoning patterns and break their compute monopoly, they suddenly turn into self-appointed “guardians of intellectual property.” This “bite the hand that fed you” double standard, combined with certain administrative powers trying to use domestic law to impose “long-arm jurisdiction” over global technology research and development, has long since left the realm of technical protection behind. It is nothing but incumbents trying to lock down the channels of technological diffusion and dig a moat to protect monopoly profits — as the saying goes, their true intentions are obvious to every passerby!
Loading...
Loading...
Loading...
Loading...
Loading...
The post opens by explaining that publication was delayed nearly a month because of paper drafts, scanning, and a blog outage. Its core is a rebuttal of the "anti-distillation" position, whose backers, it argues, can't escape a trilemma. First, the analogies don't hold: likening distillation to copying homework or cheating on an exam misrepresents it—distillation is a small model learning from a large one and acquiring capability. Second, the legal reasoning doesn't hold: learning from model outputs you've paid to call is sound commercial practice, and unilateral "take-it-or-leave-it" terms shouldn't override the law. Third, the naming doesn't hold: some vendors themselves depend on open source and on distillation from human knowledge, yet apply double standards, sidestep regulation, and push for "long-arm jurisdiction."
Three Challenges to “Anti-Distillation”
This post is going up nearly a month late — first it sat on paper for a few weeks, then right after I scanned it with AI the blog itself went down, which set me back another few weeks. At least it's fixed now, so I can finally stop turning it over in my head.
Original, Revised (Accurate Transcript + AI Polish)
In my view, anyone who supports “anti-distillation” has to answer “three hard questions” — namely, “a faulty analogy, a disregard for the law, and an unearned name”.
1. Faulty Analogy: A False Comparison
What do I mean by “faulty analogy”? People who back “anti-distillation” love to compare it to “copying homework” or “cheating on an exam” to argue that distillation is somehow illegal. But I think that comparison misses the mark: in those examples, the person who copies homework or cheats gets a final result that doesn't match what they actually know, learns nothing real in the process, and also hurts things like exam fairness. In model distillation, by contrast, the distiller genuinely gains knowledge and ability, and doesn't hurt the fairness of the rules. What it actually resembles is a student learning from a teacher: the teacher (a high-capability model) produces output (training data), and the student (a lower-capability model) learns from it and applies the knowledge. By the “anti-distillation” crowd's logic, everyone on Earth would have to wear the “cheater” label!
2. Disregard for the Law: Contracts and Clauses
Someone might object: “When a student learns from a teacher, the teacher agreed to it, and usually some fee was paid.” Then let's actually look at how distillation works: the student's parent (the LLM user) pays a fee (a subscription) to the teacher (the high-capability model maker), and lets the student (the small model) study the teacher's output. Doesn't that fit normal business logic? Or are we supposed to tear up the laws that already exist and prop up the “prohibited use” clauses some vendors unilaterally assert in their take-it-or-leave-it terms? That is what I call “disregard for the law.”
3. Unearned Name: Double Standards and Long-Arm Jurisdiction
Now let's talk about the “unearned name.” I have to ask certain companies: didn't their own models “distill” from open-source models at some point? Yet now they bite the hand that fed them — isn't that laughable? What's more, today's large AI models are, at their core, distillations of human knowledge to begin with. And some “anti-distillation” companies routinely scold distillers with one hand while using “burn-after-reading” tactics to dodge laws and regulations with the other — that's some well-practiced double standard. Then you have certain government agencies waving their domestic laws across borders in a bit of “long-arm jurisdiction.” Their true intentions, as the saying goes, are obvious to every passerby!
Expanded Version (I)
On the Three Hard Problems Facing “Anti-Distillation”
The debate over “knowledge distillation” in large AI models is getting louder by the day. In my view, anyone who supports “anti-distillation” and wants to stay coherent first has to clear “three hard hurdles” — namely, “a faulty analogy, a disregard for the law, and an unearned name”.
First is the “faulty analogy”: a bait-and-switch of concepts and a comparison that doesn't hold.
Supporters of “anti-distillation” love to reach for “copying homework” or “cheating on an exam” as everyday comparisons, trying to slap an immoral or even illegal label on model distillation as a technique. Yet this comparison is a textbook case of misrepresentation: in everyday life, someone who copies homework or cheats uses shortcuts to steal a fake result that doesn't match their real ability. They don't actually internalize any knowledge, and they undermine the fairness of the selection process. Model distillation, by contrast, has the distilled “student model” genuinely rebuilding its weights and internalizing its logical-reasoning ability, then getting tested for real in public benchmarks — so where is the harm to the fairness of the rules? This actually looks far more like how human education is supposed to work: the teacher (a high-capability model) provides lectures and worked solutions (training data), and the student (a lightweight model) grasps the method and applies it to new problems. If we follow the “anti-distillation” logic to its conclusion, wouldn't everyone on earth who ever learned from a teacher or drew nourishment from the works of predecessors have to wear the “cheater” label?
Second is the “disregard for law”: trampling contracts and declaring private dominion.
Someone might object: “When a student studies with a teacher, the teacher has to agree and the student pays tuition.” Then let's take a closer look at the real commercial loop behind model distillation: the developer (like a student's parent) pays a high-capability model maker real money in API fees or subscriptions, buying its compute and output services, and then lets its own model study the reasoning from that output — cash in one hand, information services in the other. How does that not fit the most basic logic of business and contracts? Stepping back, the big vendors themselves declare in their user agreements that they “make no claim to copyright over model outputs,” then turn around and use take-it-or-leave-it terms they wrote unilaterally to strip users of how they can use the output they already paid for. Setting aside the existing principles of copyright and contract law just so a few giants' monopolistic self-interest can be blessed by public legal doctrine — that's what I call “disregarding the law.”
Third is the “unearned name”: double standards and sheer bullying.
As the ancients said, “If the name is not right, the words won't ring true.” I'd like to ask certain business giants who wave the “anti-distillation” banner: back when you were starting out, did you hold back from fine-tuning and distilling open-source code and models? Now that your wings are full-grown and you've built high walls, you put on this “bite the hand that fed you” act — isn't it laughable? What's more, the essence of every large AI model today is, without exception, a “super-distillation” of thousands of years of human knowledge and public internet data. Certain companies shout “fair use” while scraping web-wide data without authorization, and even delete original training data or go “burn after reading” to dodge copyright accountability; but when others learn from their model outputs, they suddenly scream “intellectual property theft.” Having taken double standards this far, some even team up with politicians to try to use one country's domestic laws to impose “long-arm jurisdiction” over the global open-source ecosystem and other nations' technological development. Such conduct has long since left technical ethics behind; it's just using compliance as a cover for tech monopolization. As the saying goes, their true intentions are obvious to every passerby!
Expanded Version (II)
On the Three Hard Problems of “Anti-Distillation” in Large Models: Faulty Analogy, Disregard for Law, and an Unearned Name
Recently, several leading closed-source large-model vendors have launched a public-relations siege and terms-of-service crackdown on “model knowledge distillation.” Yet if you examine it from three angles — technical principles, legal logic, and industry ethics — the “anti-distillation” position actually falls into a hard-to-justify “triple bind” — namely, “a faulty analogy, a disregard for the law, and an unearned name”.
1. Faulty Analogy: Confusing “Result Fraud” with “Skill Acquisition”
Supporters of “anti-distillation” love to reach for “copying homework” or “cheating on an exam” as a mass-market analogy to paint distillation as illegitimate. But that analogy completely fails at the level of underlying logic.
Cheating is, at bottom, “fake results without effort” — the cheater skips the process of building real ability and steals an evaluation that doesn't match their actual level. They learn no skill at all and directly damage the reliability of the selection system. Model distillation, on the other hand, is at bottom “internalizing ability through expert demonstration.” In distillation, a teacher model (a high-parameter model) outputs chains of thought (CoT) and reasoning patterns, and a student model (a small-parameter model) learns those logical distributions and, in doing so, optimizes its own neural-network weights. It not only genuinely gains the generalization ability to solve new problems on its own, but also creates real incremental value by cutting compute costs and pushing on-device AI into wider use. To smear a teaching process that “teaches a person to fish” as “cheating” is no different from branding every human learning behavior — reading the classics or studying under a mentor — as “academic plagiarism.”
2. Disregard for Law: Elevating Unilateral Form Contracts Above Statutory Rights
The second line of defense the other side often raises is commercial contract: “Even if learning from a great teacher is reasonable, it still requires the teacher's consent; since the vendor's terms of service (ToS) already forbid using outputs to train a competing model, distillation is a breach of contract.” That argument sounds reasonable on its face but actually falls into the “disregard for law” trap — it tries to use unilaterally drafted take-it-or-leave-it clauses to hollow out the fundamental principles of existing intellectual-property law.
First, from the standpoint of commercial consideration, developers who get training data through an API have already paid full commercial fees based on token count, completing a standard “paid procurement of services” loop. Second, from the standpoint of copyright law's core principles, the law protects only original “expression” and does not monopolize the underlying “ideas, logic, algorithms, or methods of reasoning.” What's more, mainstream judicial practice today generally holds that purely AI-generated content has no copyright, and major vendors' disclaimers typically hand the usage rights of outputs over to the paying user anyway. If vendors don't even hold exclusive statutory copyright over the output text itself, then what gives them the right to use a one-sided, competition-restricting form clause to forbid others from learning the “reasoning logic” within it? If we allow giants' terms of service to expand without limit and override statutory fair use and antitrust principles, that is plainly a distortion of normal commercial order and the spirit of the rule of law.
3. Unearned Name: Standing on Humanity's Commons While Biting the Hand That Fed You
The hardest thing for the “anti-distillation” argument to sell is the “unearned name” of its own advocates.
From the standpoint of technical lineage, no closed-source large model today was built from nothing. Every large model in the industry is, in essence, an unprecedented “super-distillation” of thousands of years of accumulated human civilization, open-source community code, and the public corpus of the global internet. Even now, when synthetic data has become an industry standard, leading vendors themselves — without exception — cross-use multi-source generated data, including from open-source models, for data cleaning and capability distillation when training new models.
When these same vendors scrape web-wide copyrighted works and personal data, they wave the banners of “advancing technology for all humanity” and “fair use,” even going so far as to delete original training sets (“burn after reading”) to dodge judicial review and copyright tracing. But when latecomers and the open-source community pay for API access to learn their reasoning patterns and break their compute monopoly, they suddenly turn into self-appointed “guardians of intellectual property.” This “bite the hand that fed you” double standard, combined with certain administrative powers trying to use domestic law to impose “long-arm jurisdiction” over global technology research and development, has long since left the realm of technical protection behind. It is nothing but incumbents trying to lock down the channels of technological diffusion and dig a moat to protect monopoly profits — as the saying goes, their true intentions are obvious to every passerby!