Dear Ray;
I’ve written a book about the future of software. While writing it, I came to the conclusion that your dates are way off. I talk mostly about free software and Linux, but it has implications for things like how we can have driverless cars and other amazing things faster. I believe that we could have had all the benefits of the singularity years ago if we had done things like started Wikipedia in 1991 instead of 2001. There is no technology in 2001 that we didn’t have in 1991, it was simply a matter of starting an effort that allowed people to work together.
Proprietary software and a lack of cooperation among our software scientists has been terrible for the computer industry and the world, and its greater use has implications for every aspect of science. Free software is better for the free market than proprietary software, and there are many opportunities for programmers to make money using and writing free software. I often use the analogy that law libraries are filled with millions of freely available documents, and no one claims this has decreased the motivation to become a lawyer. In fact, lawyers would say that it would be impossible to do their job without all of these resources.
My book is a full description of the issues but I’ve also written some posts on this blog, and this is probably the one most relevant for you to read: https://lifeboat.com/blog/2010/06/h-conference-and-faster-singularity
Once you understand this, you can apply your fame towards getting more people to use free software and Python. The reason so many know Linus Torvalds’s name is because he released his code as GPL, which is a license whose viral nature encourages people to work together. Proprietary software makes as much sense as a proprietary Wikipedia.
I would be happy to discuss any of this further.
Regards,
-Keith
—————–
Response from Ray Kurzweil 11/3/2010:
I agree with you that open source software is a vital part of our world allowing everyone to contribute. Ultimately software will provide everything we need when we can turn software entities into physical products with desktop nanofactories (there is already a vibrant 3D printer industry and the scale of key features is shrinking by a factor of a hundred in 3D volume each decade). It will also provide the keys to health and greatly extended longevity as we reprogram the outdated software of life. I believe we will achieve the original goals of communism (“from each according to their ability, to each according to their need”) which forced collectivism failed so miserably to achieve. We will do this through a combination of the open source movement and the law of accelerating returns (which states that the price-performance and capacity of all information technologies grows exponentially over time). But proprietary software has an important role to play as well. Why do you think it persists? If open source forms of information met all of our needs why would people still purchase proprietary forms of information. There is open source music but people still download music from iTunes, and so on. Ultimately the economy will be dominated by forms of information that have value and these two sources of information – open source and proprietary – will coexist.
———
Response back from Keith:
Free versus proprietary isn’t a question about whether only certain things have value. A Linux DVD has 10 billion dollars worth of software. Proprietary software exists for a similar reason that ignorance and starvation exist, a lack of better systems. The best thing my former employer Microsoft has going for it is ignorance about the benefits of free software. Free software gets better only as more people use it. Proprietary software is an inferior development model and an anathema to science because it hinders people’s ability to work together. It has infected many corporations, and I’ve found that PhDs who work for public institutions often write proprietary software.
Here is a paragraph from my writings I will copy here:
I start the AI chapter of my book with the following question: Imagine 1,000 people, broken up into groups of five, working on two hundred separate encyclopedias, versus that same number of people working on one encyclopedia? Which one will be the best? This sounds like a silly analogy when described in the context of an encyclopedia, but it is exactly what is going on in artificial intelligence (AI) research today.
Today, the research community has not adopted free software and shared codebases sufficiently. For example, I believe there are more than enough PhDs today working on computer vision, but there are 200+ different codebases plus countless proprietary ones. Simply put, there is no computer vision codebase with critical mass.
We’ve known approximately what a neural network should look like for many decades. We need “places” for people to work together to hash out the details. A free software repository provides such a place. We need free software, and for people to work in “official” free software repositories.
“Open source forms of information” I have found is a separate topic from the software issue. Software always reads, modifies, and writes data, state which lives beyond the execution of the software, and there can be an interesting discussion about the licenses of the data. But movies and music aren’t science and so it doesn’t matter for most of them. Someone can only sell or give away a song after the software is written and on their computer in the first place. Some of this content can be free and some can be protected, and this is an interesting question, but mostly this is a separate topic. The important thing to share is scientific knowledge and software.
It is true that software always needs data to be useful: configuration parameters, test files, documentation, etc. A computer vision engine will have lots of data, even though most of it is used only for testing purposes and little used at runtime. (Perhaps it has learned the letters of the alphabet, state which it caches between executions.) Software begets data, and data begets software; people write code to analyze the Wikipedia corpus. But you can’t truly have a discussion of sharing information unless you’ve got a shared codebase in the first place.
I agree that proprietary software is and should be allowed in a free market. If someone wants to sell something useful that another person finds value in and wants to pay for, I have no problem with that. But free software is a better development model and we should be encouraging / demanding it. I’ll end with a quote from Linus Torvalds:
Science may take a few hundred years to figure out how the world works, but it does actually get there, exactly because people can build on each others’ knowledge, and it evolves over time. In contrast, witchcraft/alchemy may be about smart people, but the knowledge body never “accumulates” anywhere. It might be passed down to an apprentice, but the hiding of information basically means that it can never really become any better than what a single person/company can understand.
And that’s exactly the same issue with open source (free) vs proprietary products. The proprietary people can design something that is smart, but it eventually becomes too complicated for a single entity (even a large company) to really understand and drive, and the company politics and the goals of that company will always limit it.
The world is screwed because while we have things like Wikipedia and Linux, we don’t have places for computer vision and lots of other scientific knowledge to accumulate. To get driverless cars, we don’t need any more hardware, we don’t need any more programmers, we just need 100 scientists to work together in SciPy and GPL ASAP!
Regards,
-Keith
While I agree with you that free software is important for the future, I think we must not be too fast to discard every benefit of proprietary software. If we want quality product that break the limits of software, we need money. All we need now it to stop software patenting on basic feature such as user interface or menu screen and everything will be allright. Free software help push the proprietary software to innovate because they push them by creating a free alternative to something. Anyway, that’s my PoV, I may be wrong
BTW, telling Ray Kurzweil that his predictions are “way off” by referring to your own book on free software is maybe a bit of a stretch don’t you think? Ray Kurzweil worked on software and AI for a big part of his life, just saying.
——
From Keith: Free software doesn’t preclude programmers being paid to write it. My book spends a lot of time on that topic. Linus Torvalds gets a paycheck. I added several more sentences to help clarify. There is no technical benefit from a lack of freedom. Free software isn’t better than proprietary software everywhere, but that is because it doesn’t have enough people working on it yet. That is a social problem.
Keith,
I agree with Simon. Ray’s views regarding what technology will accomplish in terms of intellectual tasks is considered by many in the software engineering field to be ‘optimistic’. Jaron Lanier is only one example. To say something even more ‘optimistic’, such as that these intelligent software platforms could already exist if only we were more willing to adopt a Linux attitude towards things, is immensely short-sighted. Its one thing to be in favor of more Linux like architectures in terms of software development, its a whole other thing to argue that very difficult computational tasks requiring very fast processors and ingenious engineering, could already be here if only we adopt that method.
———
From Keith: Science is not about what “many people” think. In areas of science not explored, there is no such thing as a consensus. Furthermore, I don’t know who is in this set of futurists you’ve polled for their pessimism, but it is much smaller than the set of computer programmers. My book starts with a quote by John McCarthy, the creator of Lisp:
“Some people think much faster computers are required for Artificial Intelligence, as well as new ideas. My own opinion is that the computers of 30 years ago were fast enough if only we knew how to program them.” (2004)
I believe there are many software programmers who don’t think the problem is a lack of hardware. Every programmer has seen magical improvements in hardware in their career! Their explanation for why we don’t have the technology yet is just that it takes more work on the chalkboard to have technical breakthroughs. That theory is also wrong, but for a totally different reason.
You can say I’m short-sighted, so I’ll leave you with a question: Pretend you aren’t aware of Wikipedia’s existence. How long do you think it would take for a group of volunteers to create the world’s biggest encyclopedia? Did you say 2.5 years? I realize that Wikipedia is different from Linux and AI, but the analogy is useful and I believe fully valid. It isn’t just Wikipedia and Linux that are successful examples of free software. There are many examples — but not in the AI / futurism world yet. This is the next step.
I agree with what you are saying Keith. As a software developer I see a lot of wasted time on dead end commercial projects like Encarta, Photoshop, etc. On the other hand, Open Source software has quality issues, because the structure, meeting of deadlines and commitment from developers can be quite low. However, I believe when money is no longer the driving force in society EVERYTHING will change.
Suppose you had:
1) robot gardener & chef (= free food)
2) recycling robot (= free materials)
3) Efficient solar panels (= free energy)
4) open source hardware paired with a 3D printer and a manufacturing robot (= free physical products)
5) Open source software (= free software / AI)
6) Wireless (self replicating & self repairing) mesh network (= free internet & communication)
7) Flying Solar Cars (= free flights & transportation)
8) Robot scientists, doctors & nurses (= free health care)
(etc)
Some of these ideas are already available, or partially available. It doesn’t matter if these ideas will be fully available in 100 years or 200 years — It is plain to see we are headed in this direction.
My question is, when is the tipping point? When will money stop controlling the masses? When will people program cool software because they want to, not because they are getting paid to do it.
If I didn’t have to work for money, I would get into open source AI development to speed up humanity’s ability to solve all the world’s problems. Or I would get into genetics and stem cell research and develop immortality… but sadly I have to work for money!
———
From Keith: Hi Chris;
Not all FOSS has quality issues. There are many examples of free software that are higher quality than their proprietary counterparts. I spend some pages detailing how the LInux kernel is better than the Windows kernel.
I agree that when energy, computing and software are free, etc. we may move into a post-capitalist era where money will no longer matter. But this isn’t a binary situation.
Furthermore, free software is the most important next step, and it succeeds just fine in a free market. I spend a number of pages on this topic. It turns programming onto a service business like lawyers and accountants.
Maybe you can’t get paid to write free software, but millions of other people are contributing to it today, and a fair percentage are being paid. The problem isn’t that we don’t have enough programmers, but that they aren’t working together.
We need free software before we go into our post-capitalist society. Put another way, the fact that free software might succeed in a post-capitalist society doesn’t get us there so we shouldn’t consider it a solution.
It is a good question when the tipping point happens. But consider that the free software movement grows one person at a time. It is why I wrote this open letter. Some of the important and influential people can get others on board faster.
I wrote down a list of Linux world domination challenges here: http://keithcu.com/wordpress/?p=272
Hi Keith,
Thanks for the response.
There are a number of important differences between the Wikipedia project and the AI project. Wikipedia is a pooling of resources, where as with AI you are creating something for which there is very little precedent. AI projects have been unsuccessful for as long as they have because we don’t understand the underlying principles that govern intelligence. We don’t understand the brain. And consciousness, we really don’t understand. I see more analogies between the Linus project and the AI project in that regard.
That being said, a cruicial component to our meeting goals with regards to AI depends on our understanding the brain. And that is something that depends on hardware that we don’t in fact have, which is more sophisticated MRI technology. Faster larger processors that are capable of more so that we can scan the brain in greater detail in hopes of understanding the principles that guide intelligence. This very cruicial component of AI progress has nothing to do with software engineering.
Ray has never to my knowledge spoken out against open source software, and would be very cautious in considering that it is not factored into timelines he makes. He seems quite aware of the power of crowd sourced projects like Wikipedia, as they show up in his speeches.
So I guess I am curious as to why you think he does not factor in the power of open-sourced software into the timelines he makes?
All the best,
Miley
——-
From Keith: I have since gotten a reply from Ray and posted a reply back up above, and he and I talk about some of these issues, but I wanted to reply to your points as well.
I agree there are differences between Wikipedia and AI. Wikipedia is mostly data and very little code. I use the Wikipedia analogy because it is in the top 10 most popular websites, and well demonstrates the potential for mass collaboration. My first chapter is on Wikipedia, and my second is on Linux, which is a more thorough discussion. Note, there are people doing language AI research using Wikipedia, so the analogy is better than it first appears ;-)
I don’t agree we need more sophisticated MRI technology. We don’t need more detailed pictures of the biological hardware, we need a software re-implementation. There are many neural network implementations that are just hundreds of lines of code and have been kicking around for decades. When you digitize the algorithms of our analog system of computation, you can shrink things down a ton.
I don’t see Ray speaking out about free software very much. And in his letter above, he confirms that he sees it as equivalent to proprietary software. I know he doesn’t take the power of FOSS into his timelines, because if he did, he’d realize that we already have everything we need and we have for decades. He wouldn’t have timelines.
Regards,
-Keith
What Ray writes gets to the heart of the matter: “If open source forms of information met all of our needs why would people still purchase proprietary forms of information”
It comes down to having a good intuition regarding human nature with regards to the inherent value and limitations of open and closed systems. I quite enjoyed, although it is outdated now, Eric Raymond’s ‘The Cathedral and the Bazaar’ for its portrayal of the struggle between bottom-down and top-up design. Some projects are better with one, some the other, most with both. I agree with Ray in that they will continue to co-exist. I am optimistic in the fact that if open-source truly were better for everything then it would prevail. Or, that there is not some ‘fascist’ element to our societal power structures preventing the ‘best’ method to prevail in any extreme way. Of course there could be improvements, and those that hold the power (PhD professionals, closed-source companies) sometimes do prevent efficiency for the sake of maintaining their power, but ultimately, the process is quite democratic, and the best method will prevail. The best method, in my opinion, is coexistence. And ultimately, the two benefit from the other. I see a co-evolution taking place between open and closed.
In terms of making ANNs without having better knowledge of ‘how the brain does it’ we will have to agree to disagree on this. ANNs simulate some kind of intelligence well, but not others. I am confident in the fact that there are at least some underlying principles, ‘blueprints’ for ‘intelligence’ and ‘conscousness’, and that through reverse engineering the brain via MRI information we will discover them. The debate between ‘connectionist AI’ and ‘classic AI’ is still a debate for a reason. Like open and closed software systems, both methods have value.
I will pick up your book. Thanks for the discussion.
-Miley
———-
From Keith:
Hello Miley,
I have two disagreements with Ray’s sentence you quote above:
1. This isn’t about open source forms of information, only software. It is the “free software movement”.
2. Ray is saying that because proprietary software exists, that it SHOULD exist or that it is a good thing. I would reply again that starvation exists, but this isn’t a good thing. My book explains why free software is better for a free market than proprietary software, and necessary for us to solve the remaining big problems in computing, but people have to use free software. Ultimately, proprietary software and starvation will mostly disappear, but it involves humans making choices.
Not all choices are equally good. And taking away freedoms from software never makes the software better. Co-existence is not the best method, and it is slowing progress. Why haven’t we solved computer vision? Because there is no de facto codebase. Those people don’t realize that working together is important, yet. So while we can have this philosophical / moral discussion, consider the state of the software industry today.
I believe that anything can be simulated via ANNs. I just also understand how it would be very inefficient in some scenarios. It is like needing to add two numbers, and like inventing a programming language to do it in, first. I am quite sure there is nothing new we would learn from more MRIs of neural networks. All that is doing is taking more detailed picture of the biological machinery. It is like saying that taking pictures of your Intel processor allow you to make better software. We have the same problem with ANNs as we have for computer vision: where is the de facto codebase containing this code?
Like I said, I agree with you Keith. I ran some simulations / extrapolations on current trends a year ago and figured most open source software would overtake commercial software around 2020. This was based on a number of software projects including Linux and I’m happy to see Android (based on Linux) is selling at over 150,000 per day right now. If we can trust Google’s long term intentions then that’s a win for Linux.
I see techno-evolution happening on the commercial/proprietary front and on the opensource front. Proprietary software gets licensed out to various commercial entities, and on the evolution goes. Open Source evolution has less red tape and is more efficient however.
It is my opinion that the commercial side will eventually die, but not without a fight. Open Source grows by word of mouth, but Commercial Products utilize “Advertising” … I think the end result will be people will trust Commercial Advertising less and less as they see Open Source products perform better and better than commercial products.
One thing that Commercial Products have over Open Source is the 24 hour customer support — but even that is changing because it turns out people love helping each other and get a kick out of participating in online forums, chat rooms, etc. As chat bots get better, so will open source support improve. So really the Open Source movement will eventually solve it’s own problems. I believe in it :)
What makes me sick is there are over 50 “alpha” level projects in almost every software category that never get off the ground because of lack of commitment or interest. When no money is invested and no jobs are at stake, there is almost no commitment (besides the Linux fanatics & etc). Additionally if each of those 50 projects could unite their resources and passion, the project could actually take off and the whole world could benefit by actually getting the finished product!
What is the solution here? How can we unite 100 similar projects under one roof? It’s like trying to get fierce competitors to cooperate! How can we get more public contribution? How can we get more commitment? I have some ideas, but what are yours?
———-
From Keith:
Hello Chris Johnson;
I think Linux can take over before 10 years from now. But, if things don’t get better, it won’t take over in 10 years. Linux will only take over when it does a better job! This is a social question, not a technical one.
The success of Linux is a matter of getting this army moving more efficiently. Linux’s inherent advantage is the 10000 programs it has. Well, imagine when every one of those kicks ass? When will that be? I don’t know but, I do spend a chapter on the problems getting in the way, and the biggest problem is the bad programming languages.
There can be advertising with free software. You see lawyers advertise on TV, and many of their writings become freely available in law libraries. Free software never means that that the programmer didn’t get paid to write it. Linus has been getting paychecks to work on Linux for years.
You can have support with Linux. IBM, HP, Red Hat, and many other companies will provide you with support. Wave money in front of their face, and they will give you any SLA you could want.
I agree that there is a lot of duplicate code and dead projects in Linux. Writing more code in better programming languages will help. Using C instead of Lisp was the biggest mistake in the history of computing.
Evangelizing Python is the best way to get people to work better together. I spend two chapters leading to that conclusion. I also have another chapter on other problems with free software. The thing to realize is that we have more programmers than we need. We just need a few more to work together efficiently!
Nice, I like your perspective on how Open Source projects are merging with main stream commercial projects. I think there are many reasons commercial entities are learning to make friends with open source rather than compete against it. I feel the public is starting to understand and prefer Open Source. It means faster evolution, and better technology for everyone.
Give a man a fish, feed him for a day.
Teach a man to fish, feed him for a lifetime.
Make everything open source, feed the entire world forever.
If only I could get my lawyer, doctor, mortgage company, and government taxes to go ‘open source’, ‘free’, and ‘communal’…
Proprietary software exists because of three reasons:
1. Users, especially businesses, need assurance that the product they use (buy) will work easily, without a CIS PhD and without a 40 hour a week hobby of playing with computers.
2. People need to make a living! Everyone has rent to pay (even if it is too dang high). The economic system of labor needs revenue; otherwise, the societal resources to create the intellectual focus and capacity of those systems will become unsustainable.
3. Historical system use and cultural bias within the owner and user base of computing is predisposed to proprietary software. This may change over time as the culture evolves, but that will take time and generational understanding.
Open source or Free software ONLY exists because of subsidized or hobby interest:
1. The economic benefits of open-source software are great, but it only exists due to hobby interest or other displaced economic benefit from universities, corporate interests, or other tax subsidies from governmental agencies.
The move to open and free software at the kernel and systemic level is good for everyone both as a viable system of delivery of applications, content, and general use. Open software creates the perfect checksum on the proprietary software makers supporting a competitive marketplace where one is not easily or likely to exist. Open software is the best way to subsidize social justice and economic advancement for much of the developing world. However, it is not an either / or decision. Perhaps the best solution is a hybrid whereby the proprietary model subsidizes quick, efficient, and profit focused solutions while the open system supports better cost structures and the leveraged talent base of a larger talent pool.
Ray’s predictions are not optimistic and not pessimistic in time. Generally, they are right on the money based on historical precedence. Most importantly, he is a seminal figure who has taken the important step of placing his personal reputation and standing on the line to focus our attention on these important matters. It isn’t important whether he is right or not, it is only important that he gets the rest of us to understand the transformative nature of the next couple of years and getting the ‘group’ of humanity focused on our future shock.
——
From Keith:
This discussion is mostly about changing things for our software scientists, not all the other professions. This is the “free software movement”. There can be profitable services businesses around free software. Many programmers making a living working on free software already. You act like they don’t exist and can’t exist. Proprietary software makes as much sense as proprietary Wikipedia.
I brought up the dates because we should quit wondering when things will happen and realize it already should have. it is a way of moving people down the path of free software, and realizing that our problems are social / software. Ray’s predictions are pessimistic, and that is an important point.