Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Can you patent a Neural Net training method?

By retinaburn in MLP
Thu Oct 12, 2000 at 03:34:48 PM EST
Tags: Technology (all tags)
Technology

I came across the patent at the U.S. Patent Office through my news wire. It seems that Ralph Rose is attempting to patent a learning method for Neural networks by which one NN teaches another, apparently leading to deeper NNs being trained.


This sounds like another silly patent. I don't believe anyone holds a patent on the basic method of training NNs, nor the actually idea of a NN itself.

Is anyone else scared that this may in fact fly ?
Can we (or more specifically the U.S. government) do anything to stem the tide of patent-crazy post-ipo junkies submitting patents for photosynthesis, the design of mitochondria, or DNA ?

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o patent
o U.S. Patent Office
o Also by retinaburn


Display: Sort:
Can you patent a Neural Net training method? | 18 comments (18 topical, editorial, 0 hidden)
I'm not sure we can (2.00 / 3) (#1)
by reshippie on Thu Oct 12, 2000 at 09:07:05 AM EST

Basing this only on the bad things that have come out of the US Patent Office lately, NO, we can't do anything.

I remember reading somewhere, I forget exactly where so it may not be credible, that there is a sort of quota system going on there. The more patents that a clerk proccesses, the happier their boss is. It makes sense to me, then they can have nice happy statistics that say how many patents have been approved, and therefore, how many brilliant American inventors have been protected.

Unfortunately people are patenting whatever they can possibly think of, and they are getting approved. (ie One Click Shopping) Things are fucked up in this country, and the government is so damn big, it takes forever for real things to get fixed. Then again we can't just kill off half of the US government, like Harry Browne wants to. Anyone have any practical ideas?

Those who don't know me, probably shouldn't trust me. Those who do DEFINITELY shouldn't trust me. :-)

ouch (3.50 / 2) (#2)
by Anonymous 242 on Thu Oct 12, 2000 at 09:35:36 AM EST

Things are fucked up in this country, and the government is so damn big, it takes forever for real things to get fixed. Then again we can't just kill off half of the US government, like Harry Browne wants to. Anyone have any practical ideas?

I'm no economist, but it seems to me that Harry Browne's idea would be almost certain to place this nation of ours in one of the biggest recessions we've ever seen. Spending cuts almost always hurt the economy more than the corresponding tax breaks help.

IMHO, the only way to fix the problem is to actually put more of the government to work. Make governmental workers more efficient and productive and get things done that ought to be done. The problem isn't that there's no work that needs to be done, the problem is that between Congress and the assorted federal bureaus much of the wrong work is being required to be done.

I would also contend that, for the most part, bad patents are indicative of bad training, not of moron beurocrats. The patent office managers can't expect to get workers on the caliber of Einstein strolling through the door. You have to train people to know what to look for.

[ Parent ]

Re: ouch (none / 0) (#9)
by el_guapo on Thu Oct 12, 2000 at 11:45:09 AM EST

"I'm no economist, but it seems to me that Harry Browne's idea would be almost certain to place this nation of ours in one of the biggest recessions we've ever seen" Huh!?!? Letting people keep their money, thereby letting them spend it in the fashion that is most "efficient" to them, is WORSE than FORCING them to spend it on something everyone knows is a complete waste?????? I don't get it....
mas cerveza, por favor mirrors, manifestos, etc.
[ Parent ]
it seems a bit counter-intuitive, no? (none / 0) (#10)
by Anonymous 242 on Thu Oct 12, 2000 at 12:16:17 PM EST

Letting people keep their money, thereby letting them spend it in the fashion that is most "efficient" to them, is WORSE than FORCING them to spend it on something everyone knows is a complete waste??????

For the economy in general, the answer to your question is yes.

And while this may seem counter-intuitive to most individuals, anyone who has studied systems of any sort and understand how entropy works will understand why an economic system works that way.

Tax cuts let people keep their money. Most (not all) tax cuts will go to well off people with the result of being invested. Investments tie up money. I'll concede that some investment money is spent in the form of investments in stocks and other forms that put money back into the system, but the over all net effect of investing is to take money out of the system.

Spending, even (perhaps especially when) when the money is wasted helps to keep the money circulating within the system.

In other words, money (like heat) has a tendency to travel downhill (to trickle down so to speak) the problem is that supply side economics mistake the direction of the slope and think that money will magically work its way up the slope from the investors to the workers. But money, following the path of least resistance, has a tendency to stay in the pockets of those with deep pockets. The end result is something resembling heat death in a closed system. To prevent this, (1) one needs to open the system, (2) put more energy (money) in the system, or (3) redistribute the existing money. The US has a large problem executing (1) called the trade deficit. Option (2) typically leads to inflation which is a bad thing. Item (3) is usually the best bet and governmental spending programs are one way to do that.

The big question the economist must ask is why should economic systems be different than any other system? The universe is running down, and so is our economic system. We can either hasten entropy or delay it. Hastening entropy leads to recessions. Delaying entropy leads (temporarily) to a more robust economy.

have a day,

-l

[ Parent ]

I'll say... (none / 0) (#17)
by bgalehouse on Mon Oct 16, 2000 at 02:51:41 AM EST

Investing takes money out of the system? I'm not sure I quite understand. Investing in T-bills takes money out of the system. Hence raising interest rates cools the economy - by encouraging the purchase of T-bills and so taking money out of the system. Where does that money come from? Not from the general populance. It comes from investments which would otherwise be in stocks and bonds.

But I'm not an economist. Am I missing something here?

Also, as somebody who has studied physics, I don't see your entropy analogy at all. Money goes to people who have what the economy needs. That means programmers when computers are hot. That means investors when companies are trying to grow. That means blue collar workers when labor is tight. That means politicians when companies find laws inconvenient :-(.

There might be some strong social factors causing classes to appear. Looking at history, class structures have occured fairly consistantly since we started farming. The reasons for this may be as rich and detailed as the reasons for entropy - or even more so. But that doesn't a good analogy make.

[ Parent ]

Does this mean... (2.16 / 6) (#3)
by greyrat on Thu Oct 12, 2000 at 09:39:22 AM EST

that I'll have to pay patent fees as well as tuition for my kid's education??
~ ~ ~
Did I actually read the article? No. No I didn't.
"Watch out for me nobbystyles, Gromit!"

How Many Of Us Are A.I. Experts? (3.66 / 3) (#4)
by Carnage4Life on Thu Oct 12, 2000 at 10:09:49 AM EST

I know this is an MLP but the author should have spent some time creating a write up instead. Neural networks are an Artificial Intelligence concept that I doubt many of us are able to comment reliably on.

The first few posts I have seen seem to think this is some trivial patent like the Amazon One-Click patent thing or whatever but I doubt that it is that trivial or obvious. Hopefully someone with an A.I. background (perhaps the story poster) can give us more information and perhaps proof that it is as obvious as the MLP states.



Re: How Many Of Us Are A.I. Experts? (none / 0) (#8)
by retinaburn on Thu Oct 12, 2000 at 11:39:11 AM EST

I considered doing a write up for this. But 1) I am at work and although I am a co-op student I don't think they would appreciate this much and 2)I have only taken one beginners course in AI, only part of which was NNs. I have asked the opinion of my AI professor and I will post his response (if there is one). He "seemed" to have a better grasp then I had :)

I think that we are a young species that often fucks with things we don't know how to unfuck. -- Tycho


[ Parent ]
How Many Of Us Are A.I. Experts? (5.00 / 1) (#16)
by rich!walker on Fri Oct 13, 2000 at 03:45:31 PM EST

When reading patents (having been beating my head against this recently) the important thing is not the body text, which is merely a scene-setting preamble to convince the patent office you know what you are talking about, but the Claims, which are the area of technology that is yours until overturned.

In this patent, Claim 1 is (concisely) a neural network trained by error-difference, with variations:

  1. a change in network input values between two data points is unusual, and weird in the context of network training.
  2. at least one node producing a product of... means that the network is specifically not the standard type of neural net, but includes a PI neuron as well as SIGMA neurons.
  3. adjusting said one network parameter is also unusual: usually you train all parameters (which I think are weights) at once.

Claim 16 is the real "your patent office sux" claim. Translated from the patentese, it reads: "I claim running this algorithm on a computer as a novel idea". How that got through your patent office, I don't know. Next they'll be patenting XOR-for-graphics.

But from reading the patent over, I think the idea is this. Instead of applying back-prop to the network, we transform the network and the data by a procedure (the "differencing"). We train the transformed system, and then map this back to the original. This transformation is an improvement on the basic back-prop method. So, the patent is patenting a commutative diagram, and claiming that one route round it is more efficient than another. You can patent maths in your country? :> hth, cheers, Rich. -- need a new frontier? try obsoleting DNA.

[ Parent ]

Think a bit, won't you? (3.00 / 4) (#5)
by streetlawyer on Thu Oct 12, 2000 at 10:31:14 AM EST

Look, you posted I don't believe anyone holds a patent on the basic method of training NNs, nor the actually idea of a NN itself.

Is this guy trying to patent "the basic method of training neural networks"? No. Is he trying to patent "the actually [sic] idea of a NN itself"? No. So why raise these spectres. By this standard, when someone invents a flying car you'd be citing Chitty Chitty Bang Bang as prior art.

As far as I know, having one neural net train another is a pretty bad idea. It looks like an obvious route to overfitting of the target network, and furthermore it's bad statistical practice as it encourages you to believe that you're using less training data than you actually are. On the other hand, if this guy has solved the first problem (the second is not soluble), then he might arguably have been thought to have invented something original. He might even be able to patent a specific device based on it, as long as the patent wasn't too broad.

There is a valid critique of patents to be made, but pretending that difficult work was "obvious" from the comfort of your armchair forms no part of it.

--
Just because things have been nonergodic so far, doesn't mean that they'll be nonergodic forever

Re: Think a bit, won't you? (3.00 / 1) (#7)
by retinaburn on Thu Oct 12, 2000 at 11:33:38 AM EST

Is this guy trying to patent "the basic method of training neural networks"? No. Is he trying to patent "the actually [sic] idea of a NN itself"? No. So why raise these spectres.

The point I was trying to make is you cannot (or should not) be able to patent a specific method for training a neural network. I am sure that someone has done this (or something similar) in the past and the thought of "hey I can patent this" never crossed their minds, because it is ridiculous.

By this standard, when someone invents a flying car you'd be citing Chitty Chitty Bang Bang as prior art.

Somehome I doubt that. However if some did invent a flying car and after some time passed more than one implementation of the flying car existed (similar to the automobile ..many different engines, etc.). Now no one owns the patent for the automobile (to my limited knowledge) and no one company would own a patent for the flying car. Then somebody comes along and says "Everyone is taught by a driving instructor. I will get people to learn to drive by learning from a racecar driver."

Can he patent this idea ? No. (Nor can he receive any other appropriate protection of his idea from the government)

At least this is what I hope, I am not a patent expert (or even a novice). I hope this clears up any confusion.

There is a valid critique of patents to be made, but pretending that difficult work was "obvious" from the comfort of your armchair forms no part of it.

I admit in the one AI course I took that this thought never crossed my mind. The explanation contained within the patent makes very little sense to me. It sounds like a great idea and not at all obvious. If I said that it was obvious or even implied that I apologize. This could theoretically revolutionize[sp?] the NN field (possible creating a decent grammar checker..which I really need :) but it should not be patentable.

What would be the end result of allowing silly patents like those which we have all been reading about ad nauseum in ? Has anyone seen a realistic look into the not so distant future ?

I think that we are a young species that often fucks with things we don't know how to unfuck. -- Tycho


[ Parent ]
You have a funny notion of neural networks (3.33 / 3) (#13)
by streetlawyer on Thu Oct 12, 2000 at 03:03:02 PM EST

Look this isn't a "learning" technology, because neural nets don't "learn" in the way people learn to drive. A neural net is a nonlinear regression, end of story. What this guy has discovered, if he has discovered anything, is an algorithm for fitting a model to some data, by partitioning the model into two neural nets and iterating between them. His algorithm (if he is not a crank, which he may be), is potentially very useful. If it is in fact very useful, he deserves seven years protection to commercially exploit it. So long as the patent isn't excessively broad

See, I'd scream blue murder if, say, the General Method of Moments was patented. But I'm pretty relaxed if SPSS want to patent its extremely clever, highly optimised GMM algorithm, then I'm content to pay up for their statistical package to do so. In the knowledge that if I want to, I can look up the patent and improve my own mathematical education because it's in the public domain.

And this is a good solution, because if there were no patents, then either SPSS would never have the time and resources to devote to the long and arduous business of optimising a GMM algorithm, or they'd distribute it as a trade secret in binary-only form, and I'd never be able to figure out the details from the black box. After all, in seven years, I'll be able to get my hands on *their* algorithm, that *they invented, and use it for free without paying them a cent or even giving them credit!

The problems with software patents (and algorithmic patents in general) are that they are often too broad, not that they are bad in essence. The real test is; if it is not possible to invent a competing product without infringing on the patent, the patent is too broad.

--
Just because things have been nonergodic so far, doesn't mean that they'll be nonergodic forever
[ Parent ]
Re: You have a funny notion of neural networks (1.00 / 1) (#15)
by retinaburn on Thu Oct 12, 2000 at 03:56:33 PM EST

neural nets don't "learn" in the way people learn to drive. A neural net is a nonlinear regression, end of story.

Huh, I thought that my metaphor was clear, apparently you missed something. In my situation the man creates a "new" algorithm for teaching students to drive.

How does a student learning to drive differ from a NN learning to drive. Both are shown correct examples, as the student (or NN) works the observer can change the data to finely hone the learning process. Eventually creating a student that can drive (or mimic) driving. Sure they will still get in accidents but thats part of the process.

But I agree with the rest of your statement. You brought up good points that I had neglected to think of. A unique algorithm should be allowed to be patented, but allow enough room for competing products.


I think that we are a young species that often fucks with things we don't know how to unfuck. -- Tycho


[ Parent ]
Patenting algorithms (4.00 / 2) (#6)
by Kaa on Thu Oct 12, 2000 at 11:22:30 AM EST

Well, this is just another case of patenting an algorithm. If you believe algorithms shouldn't be patentable, then, yes, this patent shouldn't have been granted. If you believe that patenting algorithms is OK, than this is OK, too.

Kaa
Kaa's Law: In any sufficiently large group of people most are idiots.


Looking a bit deeper (none / 0) (#11)
by cthulhu on Thu Oct 12, 2000 at 12:26:27 PM EST

Upon closer inspection, you will find the patent is for a specific algorithm, which essentially trains one neural net using differential information with another neural net.

As for the methodology, I haven't seen any prior art, but most of the work I've done has been in the realm of Bayesian nets and decision trees.

I couldn't find any published papers by Ralph Rose on Cora or google. Not that a publication history is a requirement for developing something innovative, but does anyone know of any?

Patent upside (4.50 / 2) (#12)
by _cbj on Thu Oct 12, 2000 at 01:17:01 PM EST

Patents weren't created with evil in mind. It's unfair, and until proven otherwise quite wrong, to assume the patent holder has purely selfish intent.

John Koza's genetic programming is patented, but anyone could use it however they wished, I'm certain, because the creators are good people whose desire was to share useful ideas.

On the other hand, where the originating scientist is less dominant, and has less of a reputation to keep spiffy (say, in the slew of derived ideas like genetic expression programming), I can well imagine the money men behind the scenes could wish to cause mischief. All speculation, of course...

Worry about the problems, not the symptoms (none / 0) (#14)
by mooshu on Thu Oct 12, 2000 at 03:25:50 PM EST

And in this case, the problem appears to be the organization responsible for administrating patents.



A free market solution. (none / 0) (#18)
by Cwalen on Tue Oct 17, 2000 at 02:29:20 AM EST

<shrugs> Not that I am a great fan, but walk with me on this. "More demand for patent clerks increases employment. Money circulates through the economy. "Bad" patents get through. Those that are too stupid are challenged in court. Shifting money off to the lawyers. Round and Round it goes. Sounds as good for the economy as bill gates and a lot easier to put up with. As for those who think "nah that's not worth patenting" A lot of the time, for my ideas, it's true. With work and persistence I could develop something from those ideas, and profit from that discovery. Then I wouldn't mind having some protection. If I have something I believe is really worth patenting, I can go through the process. If you are too meek to protect your own work, then grab a shotgun; it's time to go inherit the earth.

Can you patent a Neural Net training method? | 18 comments (18 topical, 0 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!