How to spot “bad science”
Or how to do some critical thinking when reading (any) news or analyzing claims
Most of the readers here are experienced and seasoned engineers and researchers. I am therefore largely preaching to the choir, but for some (new) it may be useful.
I am sure we can all agree that over the years there have been many, many often spectacular claims on the Internet about all kinds of inventions. Energy inventions however have been especially abundant, with some claims even wilder than the previous. Massive over unity is one of the most popular ones – meaning that basically the machine needs to be started just once and will run evermore since it provides its own fuel (motive power). To use an understatement, this is obviously not very easy to achieve, which of course explains why never one of such inventions has been independently proven to actually work. Yet, with each passing week (and sometimes, every day) a new claimant is published somewhere and, subsequently, hailed as the newest savior of mankind. Or at least “RSN”: Real Soon Now. I suppose I am not alone in the observation that RSN so far aways seems to be a moving target, constantly a few months away.
Another observation is that claims lately seem to be become more and more numerous, but also more and more shrill: we now even have the (albeit self-proclaimed) Messiah in our midst, so all our (energy) troubles surely will be resolved RSN.
I agree that sometimes this is mildly amusing, and all nice and dandy, but the problem is of course that many people have been misled and even financially severely adversely affected due to investments that turned out to be, mildly put, not so wise. Of course it is the responsibility of the potential investor to perform due diligence for any investment, but since many sites “advertise” claims in a way as if it’s already a proven fact or at least “extremely likely”, many individuals were outright scammed or at least misled. This is compounded by the fact that those that ask reasonable, intelligent critical questions are often actively depicted as “negativists” (if not far worse) and therefore irrelevant. Fear not: Invest! Support! Donate! After all, the reader is assured the product is forthcoming RSN.
When talking to inventors and individuals offering investments in the energy field it is wise to look for “red flags” and -politely- ask critical questions. But often even that is not necessary: armed with some general knowledge and common sense it is often already obvious that something “fishy” is going on in one way or the other.
Examples of “red flags” in a person’s behavior are:
Don’t bother me with the facts, my mind is made up. And if you don’t believe me you are a sheep.
This type of reaction and/or behavior is a clear red flag. It signifies that the person preaching his beliefs to you is close-minded (while accusing you and others -especially scientists- of the very same thing, i.e. close mindedness) and absolutely convinced of his own intellectual superiority. But he/she has usually nothing to show for it, other than unfounded claims. Accusing non-believers (or simply those using their critical faculties) of being “sheeple” is of course the ultimate proof of complete and utter intellectual bankruptcy. Best to simply politely end the conversation and head for the hills.
It’s all a conspiracy, or it’s suppressed, or the black ops stole it/suppressed it/threatened me
Often a sign of a rather delusional or disturbed personality. Stories abound, but no tangible proof is usually offered. Perhaps inventions have been bought up in the past by large corporations and subsequently “shelved”, but that is probably rare and more often than not solely for commercial interests – not deliberate suppression. Stories of stolen, suppressed inventions and especially threats from “black ops” and/or other shadowy Government institutions should be regarded as not completely impossible but quite unlikely, especially when no shred of proof can be produced.
Let’s face it: what politician and his party would not want to be seen as the (literal!) savior of mankind? Suppression is not in the direct interest of most politicians.
What the public doesn’t know, I for one will not tell them
This is a tactic often used to mislead people. Some people will tell/sell reasonably plausible-sounding half-truths to mislead investors or interested persons. For example, they won’t tell that a certain invention was already patented years prior by other inventors. Or they will make claims that have been proven wrong and/or impossible long ago. Peddling half-truths is a huge red flag.
Attack the critics, whilst not providing data or proof
Instead of reinforcing their argument with (new or better) facts many an inventor resorts to ad hominem attacks on those that on often very reasonable grounds question the claims made. This is a very clear red flag. It basically means (actually, proves) that the accuser has no valid arguments whatsoever, but only opinions. It is most likely therefore that the invention is bogus as well.
Outright demonization of critics and/or accusations founded in religion
A huge red flag. Utter intellectual bankruptcy, of course, since never any proof whatsoever or rational quantifiable argument is given. Nothing but lots of hot air. Personally I have found it enlightening to investigate those that are criticized and demonized most relentlessly. One will often find that these are, contrary to what the accusers say, the actual good guys.
Perform science by proclamation, since real investigation and real research is too difficult
Many claims are often made whilst proclaiming that “science has aways denied the possibility of this and that” or “impossible according to science” – but proof of such accusations is hardly ever given. Statements like that indicate that the individual simply makes proclamations, without actually investigating or researching what the current state of science is, or what (real) science professionals actually think of the claim. “doing science” by proclamation can also mean that the claimant simply invented “new science” (new physics, for example) to bolster his claims, but in reality no such science or field of research exists, let alone documented proof. Asking for some references will visibly anger the claimant most of the time. When that happens one knows what time it is: head for the hills, once again.
Claims from authority
Sometimes inventors boast all kinds of expertise and (try to) extrapolate authority from it, such as impressive academic credentials or absolutely stellar prior careers. That may be true, but more often it is not. A genuine inventor with a genuine invention doesn’t need to boast of his career working for the past three Presidents whilst providing counsel to royalty, the Pope, and whatnot. The invention -and especially the all-important data!- will speak for itself. It is like good wine: an impressive label doesn’t guarantee a nice wine.
Claims from religion
More often than not: run for the hills immediately. Nothing wrong with religion -as long as it is practiced behind the front door- but religion never had anything to do with science or technology, or progress in those fields. To the contrary, even. Therefore: claims like “God gave me these plans for all mankind” (but please do donate first) are red flags of the first magnitude.
Some examples of “red flags” in media articles are:
Headlines of articles regarding certain claims are commonly designed to entice viewers into clicking on and reading the article. At best, they over-simplify the findings of research. At worst, they sensationalize and (completely) misrepresent them. General rule of thumb: if it sounds too good to be true, it most likely is.
News articles sometimes distort or misinterpret the findings of research for the sake of a good story, intentionally or otherwise. Do your homework and find the original research, and read it carefully – not just the summary. Never rely on the news article alone.
Conflict of interest
Many companies employ scientists to carry out and publish research – whilst this does not necessarily invalidate research, it should be analyzed with this in mind. Research can (and has been) misrepresented before.
Confusing Correlation & Causation
Be wary of confusion of correlation & causation. Correlation between two variables doesn’t automatically mean one causes the other. Prices of commodities such as foods have been increasing for many years, while at the same time the number Internet connections increased exponentially. But the abundance of Internet users doesn’t cause increased prices.
Speculations and weazelwords
Speculations from research are just that – speculation. Be especially aware of weazelwords such as ‘may’, ‘could’, ‘might’, and others, as it is unlikely the research provides hard evidence for any conclusions they precede.
Not “Conclusions only”
Don’t ever just read the mere summary or conclusions of a paper (assuming a paper has been produced). In this authors experience, even major multi-million research papers spanning a full decade of painstaking research have data in them that is absolutely at odds with the conclusions or recommendations written at the very end. These are often produced by some champion or principal investigator of the (research) project, and far too often not based on the factual data of the study itself! However these conclusions are used the lazy media (where have all the investigative reporters of yore gone?) in order to produce something completely false and misleading – but politically correct.
Claims of “inventor eccentricity” (as an excuse)
Sometimes inventors are (mildly put) “eccentric” and do very strange and irrational things. Although this doesn’t mean that inventors or claimants in general do not have the right to be their own (odd) self, it is however often a sign of potential (perhaps mental) trouble or sometimes delusion. Inventors can be their own worst enemy. The question is: has this adversely affected their findings and/or claims?
No control (or control group)
A control test should be used where all variables are controlled. Huge red flag where that is not the case. For example, when a invention is tested a identical control should be present minus the “special ingredient” that makes the invention work. If that control shows the same results, something is horribly wrong with the invention…
Videos do not constitute (scientific) proof
What’s a huge (energy) claim without a shaky, blurry YouTube video? Apart from the obvious fact that in this day and age it is extremely simple to fake a video, real science doesn’t operate with shaky videos. Some people actually believe that a person who posts many “interesting” videos must be a great scientist or at least inventor. Conversely, many believe that researchers that do not post many videos (or none at all) cannot be “real” researchers or have no credibility. That is false logic.
Watch out for lame excuses. Often it is best to “sit on the fence” when grandiose plans and claims are made. Within several weeks the first excuses will surface. The more lame the excuses, the more likely it is the invention is of the same caliber. Excuses like “the dog ate the plans” means: abandon ship.
“Third party” excuses
When the inventors credibility has already reached new lows, most likely due to a complete lack of results, third parties are often used with stories as to why no results have been obtained. Or, when results can be expected or products shipped RSN. Another big red flag.
Cherry picked results
This involves selecting data from experiments which supports the conclusion of the research, whilst ignoring those data points that do not. Obviously this completely invalidates the conclusions. Unfortunately it happens – far too often. And sometimes even in real scientific research. Be cautious and check the data yourself.
The science of measuring things is called metrology. For a number of very good reasons this is a science in itself. Many inventors and researchers are not capable of interpreting instruments correctly or making meaningful measurements.
No lab notes
It is a good practice to carefully document experiments and their outcomes. If that is not done, the scientific method is not followed and the results may be skewed or unreproducible.
Results should be replicable by independent research, and tested over a wide range of conditions (where possible) to ensure they are generalizable. Sometimes inventors make a claim, and subsequently dismantle the apparatus “for safekeeping”. This is utter madness and completely unscientific, and therefore completely destroys any credibility.
Just as “claims from authority” a large number of citations in a paper does not necessarily indicate that research is highly regarded or of very high quality.