|
131 :
naivetheorist said...
bee:
just a correction. Stephen Wolfram does NOT believe that the universe IS a computer (or a cellular automaton for that matter). if you read the beginning section of chapter 7 in his quitec literally weighty book "A New Kind of Science", you'll find that he takes the role of the theoretical physicist to be (in agreement with the view of von Neumann) constructing models that predict experimentally observed phenomena (so he is, like you a phenomenologist if i understand what you mean when you say you are a phenomenologist). he says in his lectures, blogs and book that "the universe (or any specific phenomena) behaves AS IF it is.... he does not claim that nature does in fact, follow the behavior of models which are simplified representations of aspects of reality (e.g. stephen might say that just because you can describe the flight path of a frisbee using differential equations does not mean that a person or a dog actually solves differential equations when playing frisbee catch).
best regards,
richard
9:25 AM, March 15, 2017
spink007 said...
The computer simulation hypothesis is what you get when you cross philosophy with journalism.
9:32 AM, March 15, 2017
SylviaFysica said...
Dear Sabine,
I think the simulation hypothesis belongs with other skeptical scenarios: yes, it is logically possible that the world could be very different from how it appears to us now or how we model it so far, but, no, there is no good reason to focus on this particular possibility. We should assign some probability to it, and then get on with science and critical thinking based on the higher probability cases. Eric Schwitzgebel has recently defended such a view under the name "1% skepticism". If we ever get evidence corroborating one of the - what are now considered to be - skeptical scenarios, then we should update our probabilities and develop theories for it, but prior to that it makes no sense: without evidence, there is nothing to work with.
One complication with this particular skeptical scenario is that Bostrom has offered an argument to the conclusion that we should assign high probability to the simulation hypothesis. Of course, there may be flaws in the argument, but even if you were to accept the argument, the fact that there is no direct evidence available seems a good reason for scientists not to spend too much time on it.
So, I definitely understand why all the attention to the simulation hypothesis annoys you, and the above points would be my way of responding to someone who brings it up.
Best wishes,
Sylvia
9:47 AM, March 15, 2017
Able Lawrence said...
An alternative way to refute the simulation hypothesis would be to estimate the computational requirements to run a simulation for the visible universe, even assuming qubits. What would be the memory requirement and can't we estimate the information content of the simulation vis a vis the simulated in terms of entropy. Perhaps that would show the meaninglessness of the simulation argument. It reminds me of Jorge Luis Borges short story on the country of cartographers who built a map as big as the country.
9:51 AM, March 15, 2017
AT said...
Hmm, somehow I am not entirely convinced by you arguments (or I did not read them carefully enough) that there is no simulation. If some really wants to pretend that there is quantum mechanics to (virtual) experimental physicists he/she/it will find a way. Just compute some time into the future and adjust past results as necessary to prevent suspicions (just think superdeterminism).
Of course, the question remains, why should one wish to degrade a virtual/real intelligence to a lab mouse? So, yes, overall I agree with you that the whole idea seems (like the multiverse) abandoning the theoretical physicist's job to describe nature.
Half off-topic, I wish to mention the Boltzmann Brain here, yet another mad idea about which I would like to read a blog post ;-)
10:07 AM, March 15, 2017
Uncle Al said...
Future of Humanity is a mumble factory trading apocalypse for self-importance. It is vacuum eager for gilded containers, the macroeconomics of political connivance. Future of Humanity demands credulity for shaking an aspergillum dispensing ubiquitous dread.
"today’s intelligencia is full of shit" and so willing to share, at gunpoint. Quis custodiet ipsos custodes? Entropy. Fear soft landings - the Third World.
10:18 AM, March 15, 2017
Sabine Hossenfelder said...
Hi Richard,
ok, thanks for pointing out. I should have been more careful phrasing that, will fix it.
10:32 AM, March 15, 2017
Sabine Hossenfelder said...
Able,
Well, Bostrom claims he's estimated it.
10:35 AM, March 15, 2017
Sabine Hossenfelder said...
AT,
Well, quite possibly you're not convinced there's no simulation because that wasn't what I was trying to argue. I was merely saying that the argument that it is likely we live in a simulation is wrong, or rather not even wrong - it's simply not an argument that lives up to the scientific standard. Maybe we live in a simulation - but I think it's unlikely for the reasons mentioned above: The difficulties in doing this seem to me vastly underestimated.
Yes, I've meant to write a rant on Boltzmann brains for some while, thanks for reminding me of this...
Best,
B.
10:40 AM, March 15, 2017
paperpandao1o said...
Very interesting, when it comes to deciphering junk from real science, you are my spirit animal, Sabine.
10:42 AM, March 15, 2017
Phillip Helbig said...
" It reminds me of Jorge Luis Borges short story on the country of cartographers who built a map as big as the country."
As far as I know, this scenario first appeared in Sylvie and Bruno Concluded by Lewis Carroll. Interestingly, this book also contains the idea of a train running in a tunnel which is a chord of the Earth, powered by gravity, with the journey always taking the same amount of time (about 42 minutes), regardless of the length of the chord.
10:43 AM, March 15, 2017
Phillip Helbig said...
"Yes, I've meant to write a rant on Boltzmann brains for some while, thanks for reminding me of this... "
Don't wait too long, otherwise one will materialize out of the void and write the post before you do!
10:44 AM, March 15, 2017
Georg said...
Hello Bee,
thank You for this Deep Thought(s) :=)
Georg
10:46 AM, March 15, 2017
aburt said...
Back when Bostrom first proposed this, I wrote a rebuttal, "Simulations and Reality in WYSIWYG Universes", exposing flaws in his math. (The paper was rejected by the journal he'd published his piece in, where he was, it appeared from the comments, the peer reviewer who rejected it. It was ultimately published in the SFWA Bulletin, in 2009, and is now available on Amazon/etc. as a little standalone thought piece ebook.)
Bottom line, by my analysis, the odds are much higher that the universe is exactly as we see it, not a simulation. (Though some other interesting conclusions follow from the math, about the nature of such universes, and the end game for intelligent inhabitants.) :) It was a fun analysis.
11:07 AM, March 15, 2017
Rob van Son (Not a physicist, just an amateur) said...
@Able
"An alternative way to refute the simulation hypothesis would be to estimate the computational requirements to run a simulation for the visible universe, even assuming qubits."
That has actually been done by Seth Lloyd
Computational capacity of the universe
https://arxiv.org/abs/quant-ph/0110141
I do not think he used qubits, but I am not sure. It is relatively easy to argue that, given the laws of nature, a simulation of the universe takes (much) more space than the universe itself. If you assume that the simulation runs according to other laws, anything goes.
11:11 AM, March 15, 2017
Matt Mahoney said...
Bostrom's simulation argument mistakenly uses probability theory to make statements about reality instead of statements about belief. If you say that the probability that we are living in a simulation is p, it means that you tested n universes and found pn of them to be simulations. In reality, no such test exists so it is not even a meaningful question.
11:33 AM, March 15, 2017
Kenneth Wharton said...
" there’s a trivial way in which the simulation hypothesis is correct: You could just interpret the presently accepted theories to mean that our universe computes the laws of nature. Then it’s tautologically true that we live in a computer simulation."
I wouldn't even go that far. Here's an argument against even this "trivial" point: https://arxiv.org/abs/1211.7081 . Whether or not you buy the argument, I hope you would at least concede that the statement "the universe computes itself like a computer" is not a tautology.
11:43 AM, March 15, 2017
Kenneth Wharton said...
This comment has been removed by the author.
11:49 AM, March 15, 2017
Matt Mahoney said...
Just because the universe is computable (Lloyd says 10^120 quantum operations) does not mean it is computable in a way that is useful to us. Wolpert proves that two computers cannot mutually simulate each other, which implies that a computer cannot simulate itself. A computer cannot have enough memory to know its own state because it needs at least one more bit to make any observation from its simulation and then has to include that bit in its state. Any computer that models the exact physics of our universe would have to exist outside our observable universe.
11:55 AM, March 15, 2017
Ben Jones said...
Suggesting the universe we're in can't be a simulation by appealing to computational complexity limits based on what we see in this universe is a bit of a circular argument. 'It seems very difficult to me' is like a computer game character having a hard time believing in computer game designers. Unknown unknowns.
I think the point of the hypothesis is that if it's possible to simulate experience (likely), and it's possible to nest simulations (also likely), it's a strange assumption to say you must be at the top of the tree. Nothing more controversial than that. Neither do I see any genuine showstopper in quantum mechanics. Maybe whatever box we are all running on really does simulate every single interaction! Or maybe it takes clever shortcuts, or fools us every time, or collapses wavefunctions on its own or whatever - we don't know anything about it. I personally find any of those ideas unlikely to the point of lunacy, but I don't have any ideological issue with it like you seem to, Sabine.
12:01 PM, March 15, 2017
Evan Thomas said...
"It’s not that I believe it’s impossible to simulate a conscious mind with human-built ‘artificial’ networks – I don’t see why this should not be possible"
I think Roger Penrose made pretty compelling arguments that it might very well be impossible. Everyone gets hung up on Gödel's Theorem here and seems to have missed the simpler arguments he made, which I thought were more powerful in some ways. To show that consciousness is not (fully) algorithmic in nature, we just need simple examples where we can show consciousness can do something an algorithm can't. Penrose lists some math problems that do not have algorithmic solutions (even in principle). Not only have we solved these problems, for some we've even proved they cannot be solved by an algorithm. The ones he talked about that come to mind are the halting problem, tiling problem and general Diophantine equations, although there were others, I believe.
Sounds really hard to simulate a part of the universe that is non-algorithmic doing something non-algorithmic like proving a problem is non-algorithmic, by using an ... algorithm.
Although, I seem to recall Penrose leaving open room for non-computational "algorithms", while admitting he has no idea what this would be like!
12:08 PM, March 15, 2017
Paul Mayo said...
This comment has been removed by the author.
12:15 PM, March 15, 2017
Evan Thomas said...
Matt Mahoney, this was an interesting statement:
"A computer cannot have enough memory to know its own state because it needs at least one more bit to make any observation from its simulation and then has to include that bit in its state."
Sounds like an possible argument against a computer becoming self-aware, at least fully?
Then again, I'm not sure most humans are even close to fully self-aware!
Anyhow, thanks for sharing, will have to look into this.
12:17 PM, March 15, 2017
Serafino Cerulli-Irelli said...
Something more about Borges. “Nosotros (la indivisa divinidad que opera en nosotros) hemos soñado el mundo. Lo hemos soñado resistente, misterioso, visible, ubicuo con el espacio y firme en el tiempo; pero hemos consentido en su arquitectura tenues y eternos intersticios de sinrazón para saber que es falso." J.L.Borges, 'Avatares de la Tortuga'
"We (the undivided divinity that operates in us) have dreamed the world. We have dreamed it gnarly, mysterious, observable, continuous in space and reliable in time; but we have allowed into its architecture anomalous gaps of irrationality both indeterminate and timeless so we might know that it is a contrivance."
12:42 PM, March 15, 2017
L said...
Hi Bee, what's your opinion of https://arxiv.org/abs/1210.1847 ?
1:18 PM, March 15, 2017
עמיר ליבנה בר-און said...
It seems to me that there's another solution for the "accuracy of simulation" problem nobody talks about, a solution with amusing consequences.
The beings-that-are-not-called-gods simulate each mind individually, and turn people off once they study too much physics and strain the simulation resources. The simulation of society includes physicists of course, but the simulated minds only enjoy popular articles and the fruits of technology. (The simulators might need to exclude psychologists and sociologists, and some of the more observant philosophers. But still, the simulated beings would far outnumber the physical ones.)
Obviously, there's a very different distribution of professions on simulated beings than in our society, since the simulated physicists are killed early. But since the society is simulated, there is no need for it to match the distribution of brains.
The degree of belief in this simulation hypothesis would thus depend on your occupation. If you are a physicist, the argument above would imply you must be a real live mind, living its life in the physical universe. But if you are only a reader of physics blogs, you may as well be simulated - simulating just your mind likely doesn't even require any quantum effect. This effect is stronger the less you interact with society, since you require less simulation resources this way (as society is made up, just for you). So in this scenario we can except the less socially-fluent non-physicists to believe the most that they are living in a simulation.
2:05 PM, March 15, 2017
Louis Tagliaferro said...
Sabine said…
“The idea that our universe is discretized clashes with observations because it runs into conflict with special relativity. …the programmer could prevent simulated brains from ever noticing contradictions, for example contradictions between discretization and special relativity.”
I wanted to ask that you consider discussing the issue above (discretization and special relativity) in more depth for a future blog post? I appreciate how you often can explain the technical in a way that gives non-physicists like me a better understanding of the science and issues involved.
2:06 PM, March 15, 2017
Uncle Al said...
The simulation is analog not digital. It does not calculate. Its size is not strongly bound to be larger than its construct's size. The universe is its own simulation! There's little employment to be had here.
What simulates the simulation? God (retrograde intellectual diversity). Give generously to your local tax-exempt sales outlets.
3:11 PM, March 15, 2017
Kaleberg said...
Penrose is a mystic. I always have trouble taking him seriously.
I've always liked Bill Gosper's approach in HAKMEM from 1972. He considers the various ways that a computer program can tell the number system of the machine it is running on. Back then, not all computers were twos-complement. There were ones-complement and a variety of decimal machines. He then raised the simulation issue:
"By this strategy, consider the universe, or, more precisely, algebra: Let X = the sum of many powers of 2 = ...111111 (base 2). Now add X to itself: X + X = ...111110. Thus, 2X = X - 1, so X = -1. Therefore algebra is run on a machine (the universe) that is two's-complement."
To capture the spirit of the times: "If arithmetic overflow is a fatal error, some fascist pig with a read-only mind is trying to enforce machine independence." Ah, the 1970s.
4:12 PM, March 15, 2017
Dev Null said...
I don't disagree with the thrust of your argument, but one of your statements strikes me as unlikely, specifically "More likely, we will see a future in which rich nations can afford raising one or two artificial consciousnesses". This really reminds me of the early days of computing where many thought there'd only be a need for a handful of computers or that "640k ought to be enough for anybody."
5:24 PM, March 15, 2017
Sabine Hossenfelder said...
Dev,
Well, please allow me a little fun, I sometimes like to imagine what the future might look like. It's not that I think it'll stay that way forever. But keep in mind that in the early days indeed there were only a handful of computers. And they were huge. And not everyone got to book time on them. So it seems likely to me we'll pass through a similar state with quantum computers.
2:28 AM, March 16, 2017
Unknown said...
I see no reason to assume the "simulators" would go to extraordinary lengths to hide the truth of the simulation from us. So what if we discover it? Perhaps that's the point? Maybe that's when we "win the game"?
Who knows? Maybe it's just experimental cosmology.
What I find compelling about the hypothesis is the Bayesian argument of 'If we could simulate a universe, then so could someone else, and therefore it's more likely than not we're already in one (and maybe they are too!).' Also, if there turns out to be no TOE, then a simulation run on piecewise, discrete functions seems a perfectly compelling hypothesis. There being of course, no evidence to suggest a "theory of everything" exists, and much evidence to suggest it doesn't.
PS It's trivially easy to hide "the truth" from scientists if you're God, just throw the tiniest bit of noise into something and subtle effects are swept right under the rug. Good luck with your 5-sigma if God's against you.
2:54 AM, March 16, 2017
Serafino Cerulli-Irelli said...
Sometimes we read that QM is like an 'operating system' (OS). I tend to agree with that. A question then arises. If QM is a sort of OS (superpositions, quantum darwinism, quantum non-separability, quantum contextuality, all those famous quantum principles, etc., are then routines or subroutines of that OS?) can we say that QM can ***not*** be, itself, a 'simulation'? It seems so, to me. (I remember that Einstein asked: Is the Old One [or the Great Simulator?] ***free*** to choose the physical laws?)
5:16 AM, March 16, 2017
akidbelle said...
Hi Sabine,
thanks for the post; I guess the laws of nature enabling a computer to simulate our universe would be very different.
Why would this law be more natural than the ones we believe we know? If not, no interest.
J.
7:02 AM, March 16, 2017
nicolas poupart said...
Serafino,
The OS metaphor is doubly wrong. First, QM is Turing complete and it's compares much more to a programming language than to an OS. Second, an OS suggests that there is something more fundamental and necessary to the QM existence and this proposition is false. The Turing machine demonstrates that computability emerges with a minimum of components, the QM does not need to be simulated, its simple existence is enough to generate the whole computability.
9:11 AM, March 16, 2017
mlmcl said...
The simulation notion conceals professional envy. As a life-long programmer, I relate, even if I don't sympathize. Compared to today's virtual worlds we find the universe doesn't hang, doesn't crash, doesn't run out of resources, and doesn't need maintenance. There are no race conditions, no corrupt variables, no unintended side-effects. But this is like comparing a photo of a tree to a tree, it's not an analogy that flows in the other direction.
The term "artificial intelligence" should be abandoned. When we create intelligence there will be nothing artificial about it. And there is no special reason to believe that we'll properly recognize it or fully understand it. Consider our failure to communicate with other intellectually advanced species, like whales and elephants. We do it on our terms, not theirs, despite decades of study.
9:26 AM, March 16, 2017
JimV said...
I enjoyed the post, and don't like the simulation hypothesis either, but (like another commenter above, but independently) would very much like to hear more about this: "Indeed, there are good reasons to believe it’s not possible. The idea that our universe is discretized clashes with observations because it runs into conflict with special relativity."
I had concluded (again as several commenters above) that any such simulation would have to be done in some higher-level universe with higher computational capacities, which takes it out of the realm of our science - however, I will say this: it is the first instance of a god-hypothesis (god as the entities in control of the simulation) which makes any sense to me. For example, a miracle (something not possible for the simulation's code to produce) could be done by the equivalent of hex-editing the data.
As for Penrose's argument, mentioned above, here's an algorithm he seems not to have considered:
1) Try something, even randomly, such as Edison's search for a practical light-bulb filament (not the first feasible thing, but the best thing he ever found in years of searching, was bamboo fiber).
2) If it doesn't work, try something else.
3) If it does work, write it down so future generations will know about it and go on to improve it.
That's how blind nature produced Penrose. It is also similar to the way Dr. Bee found some equation-solutions mentioned in her previous post.
10:06 AM, March 16, 2017
Richard Burke-Ward said...
Sabine, four things:
1. Physics is a mathematical that hopes to predict how the universe will behave on every level. In other words, it *encodes* the universe into algorithms. Our brains also encode what we perceive (differently). Language does the same (differently). We don't perceive raw reality, we perceive multiple, divergent encodings.
One of these coding systems is maths. But we don't know how the universe *is*, we only know how we perceive it. We codify inputs into things that mean something to us.
I am not a solipsist, mainly because I am not smart enough to understand how others codify reality. I can't experience how a physicist sees the curls and spirals that denote the decay of a charm quark; I can't experience the sensorium of a frog. But I can see these things happen. Therefore I am not alone.
I can only code (and therefore understand) things that are within the boundaries of my own physical, experiential limits. (These limits grow with time and experience.) The fact that I can see that there is a larger 'super-set' of coding systems - and that there are possibly *super*-super-sets beyond - is proof, to me, that there is no way to directly apprehend reality.
We can model reality's behaviour, somewhat, but we can't *experience* it in the raw.
So, we absolutely *do* live in a simulated universe. The question is, is it just a product of our moment-by-moment encoding, or is there some larger principle at work?
2. Sabine, you are quite right to say that it's mathematically impossible for a small fraction of the universe to be capable of perfectly simulating the entire universe it is part of - or, by extension, for that simulation to also contain another perfect simulation... But, like it or not, and however fuzzy the quantum edges might be, we do live in a granular universe. Our universe is not infinitely divisible.
So, what if the 'simulation' that we live in is actually more granular than the universe that created the simulation? What if any simulation *we* make is more granular than our own universe?
The paradox about universes-within-universes disappears. I haven't done any maths on this; but I assume you'd end up with a finite integral, even though there is an asymptotic infinity.
3. You make assumptions about *why* some entity would run a simulation: that it is in order to receive some expected or unexpected output - a 'result'. This is anthropomorphic, and may not be the case. I personally can't imagine any other reason; but then I am human, and my thinking is anthropomorphic by definition. It doesn't mean such reasons don't exist.
Maybe some meta-entity wants to know just how granular a simulation can become before it ceases to have meaning (on its layer of reality).
Maybe Boltzmann brains are real (in some finer-grained universe), and the 'brain' happened to emerge with a coding system that simulates the universe we experience. (If it didn't, we wouldn't be here.)
4. Why assume that the physics of our universe are the same physics as the universe running the simulation? Or even that the simulator-universe si part of our larger meta-verse? It could be something else entirely.
One other observation, from Iain M Banks's novel "Matter". The novel is an exploration of what a universal simulation might look like, and how we can be sure that what we experience is real. One character points out that the best argument *against* our universe being simulated is that we suffer. We experience (and inflict) pain, mortality, horror, grief.
What conceivable programmer could be so cruel?
Then again, we are talking by definition about things we cannot conceive.
Thanks Sabine, as always,
Richard BW
11:25 AM, March 16, 2017
Old Man said...
Bee
Your truth is your perception of reality interpreted by your human experience.
In a nutshell this means that every human brain is wired differently. This would require a simulation for not only every human on planet earth, but also every mammal and possibly every living thing.
david z
12:25 PM, March 16, 2017
Son Tran said...
Hi Sabine,
You are looking at it from very limited perspective. Had you ever thought of reading about Rene Descartes? Everything comes down to energy. Even a thought and idea is made of energy. And energy can be neither created nor destroyed; rather; it transforms from one form to another.
1:26 PM, March 16, 2017
APDunbrack said...
I think one should separate out two kinds of simulation arguments: anthropocentric and non-anthropocentric. The first sort, I think, is, while not entirely impossible, certainly implausible for a number of reasons you point out. Unfortunately, people tend to make their arguments for what we should do on this basis - "humans should be interesting" and so on. The second sort, I'll come back to.
There are also two ways to take simulation arguments: as science and as philosophy. The former is concerned with predictions from the hypothesis; the latter is concerned with arguments for or against the hypothesis as "true" (metaphysically). Many scientists, when discussing the simulation hypothesis, are doing so not as scientists, but as (typically naive) philosophers, dealing vaguely with ideas rather than actual concrete predictions (although there are a few who make predictions, whom you point out and whom I was previously unaware of).
That said: let's take a non-anthropogenic simulation hypothesis. I don't think this is entirely unreasonable; it just takes simulating a sufficiently complex universe. However, since we have no idea what people who simulate us are simulating (or what technology they're using to do it), that seems to tell us nothing new scientifically. Insofar as the nested-simulation-argument is valid, the simulation hypothesis makes no predictions.
My own issue with the argument in principle: you can then ask "what counts as a simulation." Sure, an actual computer simulation does - but what about a by-hand calculation of the same thing? What about mathematically solving the differential equations for the same phenomena, providing the same solution? What about just thinking about the solution? If a teacher teaches his students how to solve this differential equation, does that spawn a bunch of new universes? I think if you take this to its logical conclusion (combined with something like mathematical Platonism - things turn out differently if you think the math has to actually be discovered to exist), you find that universes can only be the laws themselves in some abstract sense, and you more-or-less end up being Max Tegmark. Not that there's anything wrong with that, but again, I think that ends up being unhelpful and unpredictive metaphysics rather than anything scientific.
4:26 PM, March 16, 2017
Count Iblis said...
We do live in our own brain's simulation of a virtual world based on real world information. What we experience is not the real world but the virtual world, albeit event in the two worlds are highly correlated (unless you suffer from schizophrenia). You can do simple experiments to verify this, e.g. optical illusions like this one
https://twistedsifter.files.wordpress.com/2012/05/optical-illusion-same-gray-color-thumb-over-middle.jpg?w=800&h=600
if you obscure the boundary between the two squares with a finger, then both squares become equally dark. Or take the McGurk Effect:
https://www.youtube.com/watch?v=G-lN8vWm3m0
8:08 PM, March 16, 2017
Kaleberg said...
It helps also if you remember that simulating a brain doesn't mean simulating something that always gets right answers. Most conscious animals we know use a biologically implemented Bayesian scheme that only indirectly requires any QM effects. It's a pretty good scheme, but it isn't always right and isn't particularly "quantum".
9:18 PM, March 16, 2017
Dana said...
From what I understand the simulation hypothesis is supported by mathematics but is not science. It's not falsifiable from my understanding, it doesn't make any predictions which can yield a true or false discovery, it does not reveal anything new about the universe. It does imply a multiverse in my opinion.
The simulation hypothesis suffers from the same problem that the problem of other minds suffers from. It cannot be tested, it's not science, but is a thought experiment. You can create a proof or come up with probabilities but if it doesn't produce something testable by scientists, physicists in particular, then what good is it?
So we live in a simulation? So what?
10:23 PM, March 16, 2017
David Schroeder said...
While evidently not viable as a scientific theory, the simulation hypothesis would make a nice sci-fi background plot for a Star Trek episode.
5:59 AM, March 17, 2017
Uncle Al said...
@mlmcl... "Consider our failure to communicate with other intellectually advanced species, like whales and elephants. We do it on our terms, not theirs."
If cetaceans broadcast (encoded) sonar images, we are fools for assuming our linear versus their volumetric language. Perhaps lines of physical theory should be 2D+ε holographic projections (photographic thick emulsion versus binary optics embossed film) re Contact (1997) and the primer. Holograms do not calculate. Lenses do not calculate, but they lose phase information.
10:23 AM, March 17, 2017
Serafino Cerulli-Irelli said...
Nicolas [9:11 AM, March 16, 2017]. I can agree. QM compares more to a programming language (or a syntax or, maybe, a many-valued language like the Aymara language) than to an operating system. I also think that QM does not need to be simulated. I'm in trouble when I think about the very meaning of computation, in the QM context. What is computed? Information? What is information? How is this information Turing computable? Via wavefunctions? Another interesting subject is, of course, the Turing-Church-Deutsch principle, and related consequences. S.
11:44 AM, March 17, 2017
David Brown said...
"The idea that our universe is discretized clashes with observations because it runs into conflct with special relativity." If nature is a multiverse that needs to be explained in terms of a Fredkin-Wolfram network underlying the Planck scale, then all arguments involving energy, spacetime, and/or quantum information might be wrong unless justified by some approximation that uses Fredkin-Wolfram information. Google "einstein's field equations: 3 criticisms" for my viewpoint.
1:13 PM, March 17, 2017
Jonathan Miller said...
I never could tell the difference between 'the universe is a computer simulation' and 'there exists a god, and perhaps that god is even similar to one in one of our religions'. As a scientist I don't think it is an interesting hypothesis and as a practicing religionist, after a little thought, I concluded that it also wasn't relevant to my relationship with God. That doesn't mean that the hypothesis doesn't imply anything interesting to theologians or philosophers, for example it might imply that god (or that which is doing the simulation, to use less loaded language) wasn't perfectly omniscient/omnipotent.
3:34 PM, March 17, 2017
Unknown said...
The whole notion of creating an "artificial consciousness" in a simulation, including encoding a "sense of agency" and self-awareness, seems outlandish.
Many very smart physical scientists & engineers, who often have a reductionist world-view, seem to significantly misunderstand & underestimate the problem. Ray Kurzweil's "singularity" mumbo-jumbo also falls into that trap.
Consider the model organism, the nematode c.elegans. Every worm has an essentially identical "brain" of about 300 neurons. I believe its neural connections -- its "connectome" -- have been completely mapped.
Yet, how the connectome leads to the worm's most basic behaviours is poorly understood, eg see:
http://www.cell.com/current-biology/abstract/S0960-9822%2805%2900940-1?_returnURL=http%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0960982205009401%3Fshowall%3Dtrue
IMO a more likely scenario for "artificial consciousness" is creating a "chinese room" type of simulation. It might very well pass any Turing test, but it's still just an unconscious "chinese room".
-- TomH
4:37 PM, March 17, 2017
Steve said...
> If you try to build the universe from classical bits, you won’t get quantum effects, so forget about this – it doesn’t work. This might be somebody’s universe, maybe, but not ours. You either have to overthrow quantum mechanics (good luck), or you have to use qubits.
Actually, if you have exponential classical resources available, there's no problem. This was proved in one of the first papers on quantum computation: http://epubs.siam.org/doi/abs/10.1137/S0097539796300921
>But how does the programmer notice a simulated mind is about to notice contradictions and how does he or she manage to quickly fix the problem?
This also seems like a non-problem. If it's a simulation, it could simply be restored from an earlier state as necessary, like save-scumming.
>Besides, if the reason you build an artificial intelligences is consultation, making them believe reality is not what it seems is about the last thing you’d want.
However, I'm totally in agreement that the motivations of the programmers and hardware-owners are so bizarre and inexplicable as to throw the whole enterprise into serious doubt. Of what possible value is our universe to anyone who doesn't live here?
9:07 PM, March 17, 2017
Sabine Hossenfelder said...
Steve,
Yes, first statement was very sloppy, sorry :/ Quantum mechanics should be possible as long as you have some kind of non-locality (in a suitable 'space-time'), it's quantum field theory that nobody knows how to do from classical bits. Hence my remark 'good luck': it might not be impossible, but nobody knows how to do it, and it will take some work to convince anyone it can be done. (The issue of locality is a thorny one. I am not at all sure it's even possible to replace the actual 'space-time' with a made-up space-time since locality is basically what defines the thing which we call 'space-time'.)
Second, I don't see how this helps - you'll still have to do the calculation which is what you were trying to avoid in the first place.
Either way, the message I was trying to get across is that if one wants to claim we live in a computer simulation one has to show it's possible to reproduce our observations this way. That's not easy, and vague words won't do.
3:27 AM, March 18, 2017
Henning Dekant said...
Whenever somebody espouses the simulation hypothesis to me I have to remind myself to be tolerant of religion. It is after all no more or less absurd than the idea that some guy thousands of years ago was nailed to a piece of wood to wash away our sins (whatever that is supposed to mean).
6:21 AM, March 18, 2017
Rob van Son (Not a physicist, just an amateur) said...
@Henning Dekant
" It is after all no more or less absurd than the idea that some guy thousands of years ago was nailed to a piece of wood to wash away our sins "
Human sacrifice was quite common throughout human history. Nothing singular about this story I think. Just that this particular way of torture was a common way to execute criminals.
9:26 AM, March 18, 2017
gurugeorge said...
Two random comments:-
1) For me the simulation idea falls down with "combinatorial explosion", which Daniel Dennett talked about in Consciousness Explained when discussing the Brain-in-a-Vat idea. Something like what you said about the choices an intelligent mind is going to make - a simulation would have to "anticipate" all the possible choices, and the possible results of those choices, etc., etc., ad infinitum.
2) Re. AI, I think that even with the great strides machine learning has been taking recently, it's not going to amount to actual intelligence: that will require different AIs talking to each other and making their own world together. IOW, intelligence is partly a function of sociality, most of the most intelligent creates are intensely social animals (with the odd exception of the octopus).
11:56 AM, March 18, 2017
Count Iblis said...
It's not clear to me whether the result in the paper cited by Steve:
http://epubs.siam.org/doi/abs/10.1137/S0097539796300921
applies here. Suppose we just do a brute force simulation of QFT, just put everything on a lattice and compute the time evolution of the wave-functional. Then the classical simulation will yield a MWI-like evolution, you'll end up with different sectors that have a different classical history that you can assign probabilities to using Born's rule. But we're interested in the inside view, and at that level Born's rule doesn't apply because all sectors end up being simulated, an observer has equal probability to find him/herself in any of the sectors where he/she is present regardless of the complex numbers referring to amplitudes that are supposed to yield the probabilities for the sectors.
2:14 AM, March 19, 2017
Sabine Hossenfelder said...
Count,
What's the lattice spacing?
2:55 AM, March 19, 2017
Count Iblis said...
You want our familiar length scales like the size of atoms to be much larger than the lattice spacing. One should therefore tune the bare couplings of the model defined on the lattice such that our familiar physics appears at scales much larger than the lattice spacing. The lattice artifacts lead to irrelevant operators, so they then become invisible at the scales we can probe. This can be done by using a model that has a critical fixed point and choosing the bare parameters such that the RG flow will let you hover near that fixed point for a large number of RG steps before you veer off toward the Standard Model.
5:13 AM, March 19, 2017
Sabine Hossenfelder said...
Count,
As you have noted yourself, you can do it up to a limited precision, highlighting the problem I pointed out: How does the programmer in advance what precision is about to be tested and if the precision limit is reached, how does he-or-she know what data to fill in?
8:20 AM, March 19, 2017
Count Iblis said...
Sabine,
Of course, I don't believe it's true, but I guess that a simulation on a lattice would not allow field configurations to arise that correspond an observer breaching the limits of the simulation.
Another problem is that the simulation actually doesn't seem to matter. Suppose that you simulate the early universe just after the Big Bang for a few seconds and then you stop. We then don't seem to appear in that simulation. However, the universe as it exists today is obtained by applying the time evolution operator to the early universe. Therefore, that simulation of the early universe does contain us in a scrambled way. This suggest to me that the simulation wasn't necessary at all, leading me to Tegmark's mathematical multiverse.
4:34 PM, March 19, 2017
George Rush said...
Sabine H.>> I think most people are much smarter than many self-declared intellectuals like to admit. Most readers will instead correctly conclude that today’s intelligentsia is full of shit. - I agree 100% with this part, but not the rest. Thanks for your interesting post!
12:13 AM, March 20, 2017
zarzuelazen 27 said...
Hi Sabine, I feel you haven't fully grasped the implications of the theory that 'reality is information' (or reality is computation).
It's *not* saying that 'reality is a simulation'. No, it's saying something much more radical than that - it's saying that there is literally no difference between 'the real world' and 'a simulation'. It's saying that the whole notion of a 'real world' is meaningless - it's like the aesther and can be dispensed it. The idea is that there's no hardware at all, no 'base level' exists - it's all 'software'.
This can work by allowing the quantifier 'real' to be continuous instead of binary. Instead of 'real' as being a binary yes/no, we could think of 'real' as a continuous, like the brightness of a light bulb. Then the idea is that there are only *degrees* of reality, things are more or less real, but no 'base level' of reality is needed.
Some questions for you: Is a simulated hurricane a 'real' hurricane. Answer: To some degree yes. Is a simulated you, really you? Answer, again, to some degree yes, up to and including 100%.
Is the 'simulation' of a thing actually drawing that thing into existence, by making it more 'real'?
And now here's the really huge kicker for you, the 'astonishing thought': Is the creation of the universe really completed yet, or is it still going on?
Think: what happens when one part of the universe simulates another part? (Remember: 'real' can be a matter of degree, and the simulation of a thing can literally summon that thing).
12:38 AM, March 20, 2017
Sabine Hossenfelder said...
zarzuelazen,
I think you haven't fully grasped what this blogpost is about.
3:11 AM, March 20, 2017
qsa said...
Actually this issue I think is misunderstood. We do not need to simulate the whole universe to prove it. We only need the correct form of the theory of nature to calculate, let's say, interactions of 100 atoms to the experimental precision(all aspects). It is clear that present day theories are lacking, but it is very clear that today's theories are also very close. So obviously we are almost there to prove the simulation hypothesis, at least that it is possible(maybe by an advanced civilization that has solved the problems of the human ones).
8:12 AM, March 20, 2017
John Anderson said...
To paraphrase G. Orwell, "They must be real intellectuals, no ordinary person would believe such nonsense."
Quis simuladiet ipsos simulades?
Did J. A. Wheeler know bit from Shinola?
So many other interesting problems: dark matter, matter-antimatter asymmetry, measurement problem, dark energy, neutrino oscillations, etc.
9:09 AM, March 20, 2017
akidbelle said...
zarzuelazen,
in any case HW or SW or both, what does that change? That you and I can refer reality to some macroscopic category inherited from our immediate environment. Why would there be even the beginning of a similarity between that dichotomy and reality?
If only information and no substrate, then information is active and interacts: I do not name this information.
J.
12:17 PM, March 20, 2017
Sabine Hossenfelder said...
qsa,
if you think we're close to simulating the presently accepted fundamental laws of nature, I recommend you read my post again
12:28 PM, March 20, 2017
qsa said...
Sabine,
I have read the article(many of the same arguments elsewhere) and many of the comments. It would help if your objections are spelled out more explicitly. Thanks.
1:54 PM, March 20, 2017
nicolas poupart said...
The problem is a curriculum problem, computer scientists have to go to quantum computing courses (at least as an option) and physicists have to take courses in the theory of computation (at least as an option). It is indispensable that the notions of computable numbers, formal incompleteness, automatons, complexity of the calculation be well grasped by physicists. This to avoid reinventing the wheel and to marvel at what is obvious to the other.
2:56 PM, March 20, 2017
David Pearce said...
A nice post: thank you. One small correction: I've asked Nick several times over the years what credence he personally assigns to our living in an ancestor-simulation. He's never reported a figure higher than 20%. That's rather different from believing that if posthuman superintelligence runs ancestor-simulations, then the principle of mediocrity dictates we're probably one of them – which is Nick's view.
3:29 PM, March 20, 2017
margarita mc said...
Mmm. I've only recently started to read your (most enjoyable) blog as part of my self education in science, and so have never heard of the simulation hypothesis before today.
You wrapped up your piece with:
"Most readers will instead correctly conclude that today’s intelligencia is full of shit. And I can’t even blame them for it."
And I was relieved that you had written that, Sabine, because as I was getting to the end of the blog post the thought that was pounding in my head was, "Do people actually get PAID for thinking up this **##* stuff?!"
I'm far too well mannered to have written a comment saying what you yourself said - but it was marvellous to have it said for me!
5:49 PM, March 20, 2017
Uncle Al said...
@qsa, "interactions of 100 atoms to the experimental precision"
Orthorhombic sulfur: space group Fddd (#70); a = 10.4646 Å, b = 12.8660 Å, c = 24.4860 Å; α,β,γ = 90°, has 128 atoms in its unit cell (DOI: 10.1107/S0108270187088152). You might need more than 100 atoms given that sphere close packing is 74% occupancy and sulfur manages only 17%.
7:52 PM, March 20, 2017
Tom Aaron said...
Im a geologist.
On all of these issues I use a geologic timescale. We are still motes on a dot. In a thousanf years? A million? A billion?
We dont know what consciousness is. We don't yet have full AI. We still haven't had any interaction with some alien intelligence. The bottom line is we are primitive creatures. Proposing some type of simulation is just another level that our brains can 'sort of' get around...the same with multiverses, parallel universes, etc. There is ZERO evidence for it yet its tossed out as a possibility. Its akin to
Creationism...no less legitimate yet no more absurd. Nothing in the physical properties of the Universe suggest a god or a simulated existence. It needs not be refuted as there is nothing to refute.
The next hundred years are going to offer some exponentisl advancements in technology and our understanding of existence. We may discover that 'reality' is more mind blowing than the possibilities we toss out today like 'simulation'. Today we our frustrated by our physics and impatient but the legitimate answesr will still be evidence based.
7:54 PM, March 20, 2017
Sabine Hossenfelder said...
qsa,
I explained in my post that your argument 'we do not need to simulate the whole universe' is an unproved and non-trivial assertion, so is your belief that we can reproduce the presently accepted fundamental laws of nature (the standard model and general relativity) on computers. We can't. We can only approximate them. I don't know why you expect me to repeat this and I'm not interested in repeating it yet again.
3:39 AM, March 21, 2017
Scott said...
The notion that we have solved problems that can't be solved by algorithms seems suspect. What counts as a "solution?" As long as it can be described and verified in finite time, it can be found (after a very long time) by a simple trial and error algorithm. And if it can't be described and verified in finite time, then how can we claim to have solved it?
Proving a problem unsolvable isn't the same thing as solving it! The so-called halting problem is a great example. The name is misused; is the HP finding a general program for determining halting behavior, or is the HP determining whether such a program can be found? The latter is solvable and solved; the answer is no, which is why the former is not and cannot be solved. Claims that the HP is an unsolvable solved problem trade on confusion between these two definitions.
That being said, Penrose is a pretty brilliant mathematician, and I think this is evidence that these things are hard rather than that some people are stupid or malicious for believing false or confused statements.
This is actually why I take Bostrom seriously! I don't necessarily think his argument is right, but then I'm not sure HE does. The point of pushing an argument like this is to see where and how it fails. I'd argue that Sabine's criticism here only applies if the argument can't fail — if it can't, then it's "not even wrong" and so frivolous. But I think it can fail, and in interesting ways that can teach us something. The fact that it can't be rejected based on physical evidence doesn't mean it can't be rejected, nor that such a rejection would invalidate the exercise. This is how philosophy (and math, I should add!) makes progress. It's just as hard and just as important as physics, though I don't necessarily expect physicists to agree!
My favorite analogy is with go (or chess if you prefer). There are some moves that you don't ever see in a pro game because they end badly. But knowing that they end badly requires playing them out! That's what I think Bostrom is doing; or at least that's why I think what he's doing is valuable, even if it turns out he's a confused true believer.
I'd be happier if this post engaged with the actual probabilistic argument he makes. It's not trivial, it can be justified by certain assumptions, and those assumptions are accepted in some other contexts. The question is, why should we reject them in this context? I feel that Sabine has given only half an answer here, and has not done anything to link that answer, in a careful way, to Bostrom's actual claims. As someone who takes the call to quantify humanistic and philosophical arguments very seriously, I'm slightly bothered by this unwillingness to meet half-way.
3:41 AM, March 21, 2017
Sabine Hossenfelder said...
David,
I've read your comment 5 times, but failed to make sense of it, sorry. He believes we live in a simulation with 20% probability or he believes we probably live in one of them (assuming that 'probably' means 'with almost 100%')? Your comment seems to say both.
3:47 AM, March 21, 2017
Sabine Hossenfelder said...
Scott,
I think you misunderstood the message I tried to get across. It's not that I think it's uninteresting, or that nothing can be learned from it, or that one shouldn't try to estimate the probabilities. I'm saying if you want to do that, you have to take into account the physical laws that we have confirmed to high precision. That's a non-trivial requirement, and without demonstrating that you can actually reproduce all our observations the simulation hypotheses is merely fiction. Best,
B.
3:51 AM, March 21, 2017
Cairo Silver said...
What's wrong with treating the universe as a computer?
I mean, if you argued against it by saying 'So where is that computer? Inside another universe? Is that a computer? Computers all the way down?', that's a valid argument against it.
But otherwise if a universe works by fixed rules and does not deviate from them, what's wrong with calling it a computer? Is the problem that 'computer' seems too intentful a word?
5:12 AM, March 21, 2017
David Pearce said...
Sabine, apologies for the ambiguity. I just checked the Simulation Argument FAQ (last updated 2011). Nick explicitly states that he assigns a probability to the simulation hypothesis of something in the 20% region. This is quite consistent with believing that _if_ posthuman superintelligence runs full-blown ancestor simulations, then we probably inhabit one of them. Maybe such simulations will prove technically infeasible; maybe humans will shortly go extinct; maybe any posthuman superintelligence will find ancestor-simulations too unethical or uninteresting to run. Maybe (and this would be one of my doubts, not Nick's) phenomenally bound subjects of experience can't arise at different levels of computational abstraction. Either way, Nick is much more cautious than some of his popularisers. Compare Elon Musk's recent claim that the probability we live in basement reality is “one in billions”.
6:41 AM, March 21, 2017
Sabine Hossenfelder said...
David,
Ok, I see. My biggest issue with Nick isn't the number but that I think it would be wiser to not mix up the simulation hypothesis with existential risks that actually have a sound footing in science.
11:16 AM, March 21, 2017
Sabine Hossenfelder said...
Cario,
Why don't you read what I explained in my post: It isn't compatible with what we know about the laws of nature. Unless you can prove that it can be made compatible, it's merely a mildly interesting tale.
11:17 AM, March 21, 2017
Tom Aaron said...
Cairo...is the Universe a computer?
Maybe yes. maybe no. But evidence is needed. We have none.
We don't know what existence is so we don't know if it follows some rational process. Our senses perceive fragments of reality that our minds are capable of detecting. Our math and physics are akin to watching a baseball game. Everything seems rational to the fans watching. Patterns are just accepted and some theory might be developed about how the number of strikes relate to the number of outs, etc. However, when we stand back and ask someone who has never seen a baseball game if the patterns represent reality he would go 'huh?...it has nothing to do wih gathering coconuts'.
The computer model, simulation, etc. 'assumes' a greater grasp on existence than what we have. Its akin to the ancients assuming that some god is responsible for the annual seasons. We are are still clueless to the extent of knowledge that we dont know. There may be a simulated existence, there may be a Sun god, the Universe may be a cimputer but 'so what?'. No evidence for any of these. Zero. We have finite knowledge and, like baseball, it may be irrelevent in understanding the bigger picture.
11:52 AM, March 21, 2017
George Rush said...
Sabine, obviously we might "live in a simulation". OTOH obviously you know what you're talking about. So, there's an apparent contradiction.
Minimally, a virtual reality simulation has to reproduce my experiences - or, from your point of view, yours. Most people say their consciousness can't be simulated, but not you. So, surely you agree that you personally could be a "brain in a vat"? Unlikely, but doesn't violate any physical laws, right? Moving beyond solipsism, if it can do one human, it can do 7 billion. A few orders of magnitude extra sim power is no big deal. Now, let's consider some of your objections.
From your previous post on the subject, "To avoid the inconsistencies, you’ll have to carry on all results for all future measurements that humans could possibly make, the problem being you don’t know which measurements they will make because you haven’t yet done the simulation." I guess you're talking about (to take simple example) a classic spacelike-separated Alice & Bob EPRB gedanken? The sim can easily provide random spin up/down results for A and B. But later, when results are compared, they must be consistent, with the right correlations. If that's what you mean, I claim it's no problem, happy to explain why.
Not sure why you think sim has to look at "all future measurements" Alice and Bob might make. But the objection is easily disposed of. As far as we know there may be only one future path. The entire history of the universe may be pre-known: no calculation necessary, sim just looks it up in a large database.
Then you say: "But there is a better way to test whether we live in a simulation: Build simulations ourselves, ... Eventually, the ... lowest level will find some strange artifacts. Something that is clearly not compatible with the laws of nature they have found so far and believed to be correct." This is no argument against sim hypothesis. Leaving aside some (rather important) issues, there's no reason this scenario couldn't happen. It may seem "crazy", but that's no argument.
In this current post, you say "... there’s a trivial way in which the simulation hypothesis is correct: You could just interpret the presently accepted theories to mean that our universe computes the laws of nature. Then it’s tautologically true that we live in a computer simulation. It’s also a meaningless statement." Wrong, it's far from meaningless. If we really are brains in vats, fed by thick conduits carrying encoded qualia, generated by a physical computer of some sort, tended by insectoid alien programmers - that's non-trivial! If the computer happens to generate those inputs by solving Standard Model equations, so what?
You assume sim must use the most advanced technology we ignorant naked apes can (sort of) understand at this stage of our development: digital quantum computers. But we have no idea what powerful computing resources might be available to us 100 years from now, much less a million. We can't constrain hypothetical alien programmers to our primitive techniques.
Although you're aware that only human experience needs to be simulated, you seem to think in order to do that, simulation of the entire universe, at all scales, must ultimately be involved. As you show, with primitive digital quantum computers, that probably can't be done. But your assumptions are flawed.
The solution to the "apparent contradiction" mentioned above may be: you're really not debating whether sim is possible, logical, or reasonable. Instead, you're investigating whether we can prove it or not: whether it's a scientific hypothesis, in the Popperian sense. But before going on I'll wait for response (if, indeed, you consider it worthwhile). BTW I don't "believe in" sim: although there's absolutely no evidence against it, there's not much for it either. Thanks for your time.
3:04 PM, March 21, 2017
Sabine Hossenfelder said...
George,
"Although you're aware that only human experience needs to be simulated, you seem to think in order to do that, simulation of the entire universe, at all scales, must ultimately be involved. "
What I am saying is, if you believe you can get away with not simulating parts of the universe when some of your simulated consciousnesses - whose actions you can't predict - aren't trying to probe them, I want to see how you do this. Unless you demonstrate that, it's just bla-bla but nothing any scientist can take seriously. The idea that you can somehow produce an explanation for all our observations that does not require the laws of nature to be how we have extracted them today is an extraordinary claim and nobody should accept it without a very solid backing. Best,
B.
3:10 AM, March 22, 2017
Topi Rinkinen said...
Hi Bee,
Let's assume the universe could be described with set of equations, and an initial condition. Then there (outside of our universe) can be a number of computers which can run the simulation and all of them would get the same result (assuming the functions are predeterministic).
If the number of computers running this simulation is more than 1, could someone inside the simulation have means to check out how many computers there are runnig the simulation? I guess not.
If a number of those computers are running the simulation in synchronization (so that you and me are writing excactly the same letters in same their-time), and then one by one those computers are shut down. Would we notice any difference? Even when the last computer is shut down? I believe not.
What if the equations are such, that the simulation can be done in kind of frequency domain (as opposed to time domain). What would be the point in time (in the simulating computer's wolrd time) that I press this T-letter? It's an ill-defined question.
If the frequency domain simulation was such that it could be distributed to several computers, which could run in parallel or totally in different times. How would it affect our capability of detecting the simulation? I'd say we couldn't sense it.
What if someone invented the equations, and started simulation in frequency domain, but aborted it after running a small part of it. Would we exist, and could we feel the simulation is incomplete? I say no, even if the small part is like one millionth of the whole simulation.
And the logical next question is, is there a need to start the simulation for us to feel our own existence? Or is there a need to "invent" the equations in the first place, for our everyday feelings?
...
Редактирано от Mod vege на 13.04.17 02:37.
|
+ Comments to "No, we probably don’t live in a..." [re: Mod vege]
(старо куче)
