Elf Sternberg

It seems to me that, while "emergent property" is to some extent a handwave, it is a necessary consequence of the limits of what we can explicitly model. Some of those limits are simply those of time and effort. How detailed a model of the weather can we make that will give a result soon enough to be a forecast? Other limits come from such things as Chaos, which can appear in what seems to be simplest of useful math.

So it's possible not only that we don't know how intelligence emerges from the complexity of our brains, but that, for reasons other than philosophical arguments about whether a system can ever understand itself, we cannot know.

But that's slightly different from the Chinese Room. That's saying that we can look inside the Black Box, and says something about what we find there. I think there's a real possibility that we can't do that. Though what we may have, and what we may be able to see, is a Chinese Room with a lot of parts we can understand, and a great big Black Box in the middle.

Which isn't really a Chinese Room any more.