Chrisw wrote:

home_ wrote:

Chrisw wrote:But we aren't born with concrete goals about what we want to achieve in the world. We choose our goals and can change them at will. We constantly evaluate our goals, not by reference to other goals but according to how they make us feel at a much more primitive level. And this includes moral feelings and emotions.

This doesn't solve the problem. If we make an AI this way it wouldn't mean that it would respect human values. If AI wanted to achieve some goal (regardless of whether that goal was pre-programmed or an emergent property) it could still destroy humans because we'd be standing in a way, even if just temporary.

It would respect the values we programmed it to have. Values are more resilient than goals, or at least they are in humans and thus in creatures designed to resemble humans in this way.

Yes, if we managed to program human values into this system that would essentially solve the problem. But the task is formidable. We don't really have a good definition of human values, we disagree about a lot of things, values change over time (a lot of values held in antiquity are disastrous from todays point of view) and we have yet to develop ways to put them into software. But I certainly hope that we overcome this challenge.

Another way of dealing with the problem would be to develop advanced math that would ensure that AI system would stay within chosen constraints. Math doesn't change, 2+2=4 for chimps and for humans and it will also hold for AI.

Chrisw wrote:It's OK if it has limited power. You get into problems when you start allowing it to modify itself. The idea that it could modify itself in accordance with its goal implies that its goal is somehow separate from what it is. But in that case why wouldn't it just change the goal to match reality?

Yes, if it has limited power then the problems are managable. But ultimately the point of making AI is to make strong AI, not weak AI. We have lots of weak and limited AI today.

And why would strong AI want to change its goal? If the goal is to be changed, then it is unlikely that it will be met. That would be disastrous from the point of view of that goal. Having a goal implies preservation of that goal.

kennyc wrote:

Chrisw wrote:
Think things through? I was responding, at reasonable length to a few throwaway sentences from you. Between the two of us, I'm the one actually thinking about this here.

I know I'm probably wasting my time here asking you, but what do you disagree with/not understand about what I wrote?

You are so limited in your understanding of the topic that you think my statement is 'throwaway sentences.'
Im not wasting my time with you beyond this.

Just a followup to say sorry Chris, I really shouldn't berate you for your limited understanding of a topic I've been studying for decades. I'll leave it at that but my point is that if you are going to post on this topic then you have a hell of a lot of learning to do about evolution, computing, physics, brains, minds, cognition, intelligence, evolution, memes, information, and the vast span of time and space in which those things operate.

Learning can be fascinating, good luck!

kennyc wrote:

kennyc wrote:

Chrisw wrote:
Think things through? I was responding, at reasonable length to a few throwaway sentences from you. Between the two of us, I'm the one actually thinking about this here.

I know I'm probably wasting my time here asking you, but what do you disagree with/not understand about what I wrote?

You are so limited in your understanding of the topic that you think my statement is 'throwaway sentences.'
Im not wasting my time with you beyond this.

Just a followup to say sorry Chris, I really shouldn't berate you for your limited understanding of a topic I've been studying for decades.

Be careful, you are starting to sound like Jamest. Years of "study" (you mean reading popular science books, right?) but you can't string an argument together to support your position. A position you are extremely coy about anyway lest anyone ask you awkward questions. Don't tell me how knowledgeable you are, demonstrate it by doing something other than cut-and-paste and one-liner arguments with no follow up.

I'll leave it at that but my point is that if you are going to post on this topic then you have a hell of a lot of learning to do about evolution, computing, physics, brains, minds, cognition, intelligence, evolution, memes, information, and the vast span of time and space in which those things operate.

I've almost certainly read the same popular science books you have Kenny. And I have a science-based degree, which helps occasionally. I've also read books by Kurzweil and Bostrom on the singularity/superintelligence and lots of stuff online. I know the arguments.

And if we are giving advice to each other, let me say that you need to actually read what people post and engage with the points they make. Your current posting style is borderline trolling. Seriously, you aren't participating in discussions in a way that is at all helpful.

If you just assume that everyone who disagrees with you is ignorant you will never learn anything new.

home_ wrote:

Boyle wrote:

home_ wrote:Of course it wouldn't be a random pick, but that wasn't the point. The point was to illustrate that there is a problem with strong AI. When you start thinking about how would a utility function that is in line with human values look like, then you realize that it's very hard to think of any at all. It will not be possible to have strong AI under control and it's not obvious how to ensure safety and beneficial outcome.

I guess I've gotta ask what capabilities you are ascribing to a strong AI. Does it possess enormous manufacturing capabilities? Access to every electronic system on Earth? Speakers? Screens? Robo-arms? How are they manipulating things? How are they connected with external systems? Are they simply being used as a consultant type system? Data mining? Personal assistant? Social analysis and prediction? Market analysis and prediction? What problems are we putting AI's to, exactly, that will cause them to exterminate us?

That's a lot of questions, I know, but I've never quite grasped just what is so terrifying about strong AI's. Or, to be honest, what characteristics they'd even have. Are they simply extreme logic engines with some added creativity? Are we assuming that strong AI's will have some propensity towards conserving resources and diversity? Or is it simply that we don't know what a strong AI will do?

'Standard definition' of strong AI is that it is an AI that vastly outperforms humans in all categories, including science, art and social interaction.

It would probably 'live' in powerful supercomputers and interact via internet, controlled robots, etc. The details don't really matter, the point is that it would take over the planet and it would not be possible to influence its decisions.

Increase in performance of computer chips has led some people (myself included, I suppose) to believe that such an AI is possible, and furthermore, that there are good enough reasons not to dismiss the possibility that it could be very destructive (or an existential threat) as an unintended consequence.

Sounds like scare-mongering to me, home_. It would need to control an army to be able to make itself an "existential theat". To be able to control an army, it would have to be ALLOWED to do so, by humans!

Chrisw wrote:

kennyc wrote:

kennyc wrote:

Chrisw wrote:
Think things through? I was responding, at reasonable length to a few throwaway sentences from you. Between the two of us, I'm the one actually thinking about this here.

I know I'm probably wasting my time here asking you, but what do you disagree with/not understand about what I wrote?

You are so limited in your understanding of the topic that you think my statement is 'throwaway sentences.'
Im not wasting my time with you beyond this.

Just a followup to say sorry Chris, I really shouldn't berate you for your limited understanding of a topic I've been studying for decades.

Be careful, you are starting to sound like Jamest. Years of "study" (you mean reading popular science books, right?) but you can't string an argument together to support your position. A position you are extremely coy about anyway lest anyone ask you awkward questions. Don't tell me how knowledgeable you are, demonstrate it by doing something other than cut-and-paste and one-liner arguments with no follow up.

I'll leave it at that but my point is that if you are going to post on this topic then you have a hell of a lot of learning to do about evolution, computing, physics, brains, minds, cognition, intelligence, evolution, memes, information, and the vast span of time and space in which those things operate.

I've almost certainly read the same popular science books you have Kenny. And I have a science-based degree, which helps occasionally. I've also read books by Kurzweil and Bostrom on the singularity/superintelligence and lots of stuff online. I know the arguments.

And if we are giving advice to each other, let me say that you need to actually read what people post and engage with the points they make. Your current posting style is borderline trolling. Seriously, you aren't participating in discussions in a way that is at all helpful.

If you just assume that everyone who disagrees with you is ignorant you will never learn anything new.

Keep learning, you'll get it eventually. Good Luck.

Kenny, I am, inclined to agree with Chrisw about your sounding like jamest.

Ask me if I give a fuck David.

kennyc wrote:Ask me if I give a fuck David.

I'm sure you don't. It fits with your obvious character issues that you don't.

DavidMcC wrote:Sounds like scare-mongering to me, home_. It would need to control, an army to be able to make itself an "existential theat". To be able to control an army, it would have to be ALLOWED to do so, by humans!

If you allow an AI to generate and publish a webpage, you are implicitly allowing it access and complete control of the vast majority of internet enabled computers, under the reasonable assumption that there are a shed load of zero day vulnerabilities through javascript engines and kernels allowing arbitrary privilege escalation. "All" it has to do is discover the vulnerabilities.

....and 'press the launch' button.

...and in the year end science news wrapup: http://singularityhub.com/2014/12/17/ho ... e-in-2014/

.....Artificial Intelligence Advances, Fuels Controversy

Not so long ago, artificial intelligence was a word to be avoided—now it’s back in vogue.

Narrow AI, the kind that learns and performs specialized tasks, doubled its ability to recognize images this year. IBM announced a new brain-inspired chip and invested $1 billion in its Jeopardy-champ AI system Watson. Siri’s inventors funded a project to develop a next-gen virtual assistant to handle the tasks Siri (and others) still struggle with. Google, Facebook, and others continued collecting AI researchers like baseball cards. And algorithms broke news before humans.

Meanwhile, still-distant general artificial intelligence (think Hollywood AI) kicked up controversy in a book by Nick Bostrom. Bostrom argued superintelligent AI could be our last invention—either saving or destroying us. Elon Musk and Stephen Hawking jumped aboard, the former noting AI could be more dangerous than nuclear weapons. Beyond a touch of hyperbole, the broader message is that future AI will be powerful (for good or ill): Handle with care.
....

Links from the above excerpt:
http://singularityhub.com/2014/11/20/su ... isruptive/
http://googleresearch.blogspot.com/2014 ... mages.html
http://singularityhub.com/2014/01/14/ib ... 1-billion/
http://singularityhub.com/2014/01/30/go ... -as-usual/
http://singularityhub.com/2014/03/25/mo ... you-think/
http://singularityhub.com/2014/12/09/ca ... s-from-ai/

Happy reading. Happy learning.

home_ wrote:

ChrisW wrote:It's OK if it has limited power. You get into problems when you start allowing it to modify itself. The idea that it could modify itself in accordance with its goal implies that its goal is somehow separate from what it is. But in that case why wouldn't it just change the goal to match reality?

Yes, if it has limited power then the problems are managable. But ultimately the point of making AI is to make strong AI, not weak AI. We have lots of weak and limited AI today.

Yes so it's a problem if powerful AI goes off the rails because we would like to have powerful AI, all other things being equal.

And why would strong AI want to change its goal? If the goal is to be changed, then it is unlikely that it will be met. That would be disastrous from the point of view of that goal. Having a goal implies preservation of that goal.

Not quite. Successfully achieving a goal implies preservation of the whatever it is that causes you to seek the goal.

But maybe you aren't successful? You say that the machine wouldn't behave in a self-defeating way because that would be going against its goal. So it might achieve outcome B instead of the intended outcome A. But what we should really say in such a case is that it's actual goal was B all along. We intended A and tried to design an A-seeking machine but we actually ended up constructing a B-seeking machine.

To say that something has a particular goal is to say that it will make some specific changes to the world (the goal outcome) over a fairly wide variety of different world scenarios. This is a functional definition of a goal and its all a goal really is.

It's only tautologically true that having a goal implies preserving your goal. If you reliably change your goal to something else, in a wide variety of circumstances, then you never really had the original goal in in the first place.

Now consider how such a goal-seeking machine might be organised internally. It might have a model of the world and its operations (which it continually updates according to fixed rules, essentially empirical science) and a model of the desired state of some part of the world relevant to its goal (a goal model). Its program makes it manipulate the world so that its world model is consistent with its goal model.

Now what we want to say here is, it is programmed to manipulate the world to make it match its goal and not vice versa. But the machine is part of the world. There are no limits on what "manipulate the world" may mean here, its goal model is just another part of the world. So why wouldn't it achieve its match between the goal and the world by changing the goal to match the current state of the world, as described in its model? Or maybe achieve the match by changing its world model or the way it builds it? In other words, it can move the goal posts or it can pretend it has hit the goal anyway. Both require fewer resources than doing it properly. And it won't even know it is cheating, it's just doing what it was programmed to do.

Chrisw wrote:..... it's just doing what it was programmed to do.

Which demonstrates once again your complete ignorance on the subject.

True AI is not 'programmed' to do anything other than to learn and to survive.

....and that is what is at the crux of this discussion and what Hawking and Musk among others is concerned about. If the AI learns that humanity poses a threat to its survival and that it can survive or determine a way to survive without humanity, the war begins.

kennyc wrote:

Chrisw wrote:..... it's just doing what it was programmed to do.

Which demonstrates once again your complete ignorance on the subject.

True AI is not 'programmed' to do anything other than to learn and to survive.

What's that from, the Ladybird book of Artificial Intelligence? Stop embarrassing yourself.

Chrisw wrote:

kennyc wrote:

Chrisw wrote:..... it's just doing what it was programmed to do.

Which demonstrates once again your complete ignorance on the subject.

True AI is not 'programmed' to do anything other than to learn and to survive.

What's that from, the Ladybird book of Artificial Intelligence? Stop embarrassing yourself.

As I said, please go learn if you care to discuss this topic, you clearly know nothing about it.

kennyc wrote:....and that is what is at the crux of this discussion and what Hawking and Musk among others is concerned about. If the AI learns that humanity poses a thread to its survival and that it can survive or determine a way to survive without humanity, the war begins.

I'm not arguing with you. You accuse me of ignorance because you are yourself so ignorant that you don't understand what I am saying. How fucked up is that?

The better my arguments the more you misunderstand them and insult me so what am I supposed to do? I can't dumb down enough for you, I'm sorry.

Learn.

kennyc wrote:Learn.

... how to be a Singularity fanboy?

Learn.

put the handbags away ladies!