Total Posts:13|Showing Posts:1-13
Jump to topic:

too many scientific studies

debate_power
Posts: 726
Add as Friend
Challenge to a Debate
Send a Message
3/15/2015 7:15:09 PM
Posted: 1 year ago
At 3/15/2015 7:04:36 PM, Wylted wrote:
According to a recent scientific study, there are too many scientific studies.

http://science.slashdot.org...

Thoughts?

"Too many studies" is subjective, and not scientific... I'll have a good look at that article.
You can call me Mark if you like.
Maikuru
Posts: 9,112
Add as Friend
Challenge to a Debate
Send a Message
3/15/2015 8:05:58 PM
Posted: 1 year ago
You are describing a major problem in academia today. As more and more up-and-coming researchers via for a limited number of positions, the priority will always be on finding a new result, a new revolutionary way of thinking. So long as there are degrees to be earned and grants to be won, there will be an over-abundance of studies.

I do believe this problem is self-correcting, though. There are more candidates for academic positions than there are positions available. This means those students who are unable to find work right away will start to look into post-doc positions or less demanding research. Both of these lend themselves to replications of previous studies or meta-analyses of existing work, which each serve to narrow down a field and separate academic fact from fiction.
"You assume I wouldn't want to burn this whole place to the ground."
- lamerde

https://i.imgflip.com...
Welfare-Worker
Posts: 1,200
Add as Friend
Challenge to a Debate
Send a Message
3/16/2015 10:23:48 AM
Posted: 1 year ago
The information in this article is typical of many I have in my files.

I don't particularly like long copy and paste, but it seem needed in this case:

A recent article in the Wall Street Journal underscores both how prevalent errors are in scientific studies and how long it takes to uncover these errors. Moreover, the number of retractions of scientific papers (retractions indicate an error has been detected) has been increasing rapidly this past decade. Here are the numbers: In the approximately 11,000 peer-reviewed scientific journals, the retraction rate has increased fifteen-fold from 2001. There were only 22 retraction notices in 2001, but 339 last year and 210 through July 2011.
The time that it takes for science to "self-correct," that is, for the retraction of a flawed study to appear, is also increasing in length. In 2011, retractions were published, on average, within about five months of the original study. The average now is over thirty-one months.
Three years may not seem like a long time to get it right, but if you have been taking medicine based on the original flawed study, that may be little consolation. And, of course, in some cases it can take much longer. The Journal article details the history of one flawed study that resulted in a significant change in the use of drugs to control hypertension. Seven years later, after serious side-effects to some patients on the new regimen, the original study was shown to be flawed and possibly conducted unethically, without proper controls.
Perhaps most disturbingly, the Journal study reveals that about 26% of flawed studies are not the result of mere human error, but rather are the product of scientific fraud. In other words, a scientist has failed to follow proper procedures, for example, by cooking the data. Why would a scientist engage in fraudulent conduct, especially if science is self-correcting?

~ ~
But irreproducibility is much more widespread. A few years ago scientists at Amgen, an American drug company, tried to replicate 53 studies that they considered landmarks in the basic science of cancer, often co-operating closely with the original researchers to ensure that their experimental technique matched the one used first time round. According to a piece they wrote last year in Nature, a leading scientific journal, they were able to reproduce the original results in just six. Months earlier Florian Prinz and his colleagues at Bayer HealthCare, a German pharmaceutical giant, reported in Nature Reviews Drug Discovery, a sister journal, that they had successfully reproduced the published results in just a quarter of 67 seminal studies.

The governments of the OECD, a club of mostly rich countries, spent $59 billion on biomedical research in 2012, nearly double the figure in 2000. One of the justifications for this is that basic-science results provided by governments form the basis for private drug-development work. If companies cannot rely on academic research, that reasoning breaks down. When an official at America"s National Institutes of Health (NIH) reckons, despairingly, that researchers would find it hard to reproduce at least three-quarters of all published biomedical findings, the public part of the process seems to have failed.

Academic scientists readily acknowledge that they often get things wrong. But they also hold fast to the idea that these errors get corrected over time as other scientists try to take the work further. Evidence that many more dodgy results are published than are subsequently corrected or withdrawn calls that much-vaunted capacity for self-correction into question. There are errors in a lot more of the scientific papers being published, written about and acted on than anyone would normally suppose, or like to think.

First, the statistics, which if perhaps off-putting are quite crucial. Scientists divide errors into two classes. A type I error is the mistake of thinking something is true when it is not (also known as a "false positive"). A type II error is thinking something is not true when in fact it is (a "false negative"). When testing a specific hypothesis, scientists run statistical checks to work out how likely it would be for data which seem to support the idea to have come about simply by chance. If the likelihood of such a false-positive conclusion is less than 5%, they deem the evidence that the hypothesis is true "statistically significant". They are thus accepting that one result in 20 will be falsely positive"but one in 20 seems a satisfactorily low rate.

Understanding insignificance
In 2005 John Ioannidis, an epidemiologist from Stanford University, caused a stir with a paper showing why, as a matter of statistical logic, the idea that only one such paper in 20 gives a false-positive result was hugely optimistic. Instead, he argued, "most published research findings are probably false." As he told the quadrennial International Congress on Peer Review and Biomedical Publication, held this September in Chicago, the problem has not gone away.
Dr Ioannidis draws his stark conclusion on the basis that the customary approach to statistical significance ignores three things: the "statistical power" of the study (a measure of its ability to avoid type II errors, false negatives in which a real signal is missed in the noise); the unlikeliness of the hypothesis being tested; and the pervasive bias favouring the publication of claims to have found something new.
A statistically powerful study is one able to pick things up even when their effects on the data are small. In general bigger studies"those which run the experiment more times, recruit more patients for the trial, or whatever"are more powerful. A power of 0.8 means that of ten true hypotheses tested, only two will be ruled out because their effects are not picked up in the data; this is widely accepted as powerful enough for most purposes. But this benchmark is not always met, not least because big studies are more expensive. A study in April by Dr Ioannidis and colleagues found that in neuroscience the typical statistical power is a dismal 0.21; writing in Perspectives on Psychological Science, Marjan Bakker of the University of Amsterdam and colleagues reckon that in that field the average power is 0.35.

Unlikeliness is a measure of how surprising the result might be. By and large, scientists want surprising results, and so they test hypotheses that are normally pretty unlikely and often very unlikely. Dr Ioannidis argues that in his field, epidemiology, you might expect one in ten hypotheses to be true. In exploratory disciplines like genomics, which rely on combing through vast troves of data about genes and proteins for interesting relationships, you might expect just one in a thousand to prove correct.

With this in mind, consider 1,000 hypotheses being tested of which just 100 are true (see chart). Studies with a power of 0.8 will find 80 of them, missing 20 because of false negatives. Of the 900 hypotheses that are wrong, 5%"that is, 45 of them"will look right because of type I errors. Add the false positives to the 80 true positives and you have 125 positive results, fully a third of which are specious. If you dropped the statistical power from 0.8 to 0.4, which would seem realistic for many fields, you would still have 45 false positives but only 40 true positives.

More than half your positive results would be wrong.

From: Science Is Self-Correcting ... Sort Of
August 16, 2011

http://sixdayscience.com...
UndeniableReality
Posts: 1,897
Add as Friend
Challenge to a Debate
Send a Message
3/16/2015 11:40:42 AM
Posted: 1 year ago
At 3/16/2015 10:23:48 AM, Welfare-Worker wrote:
The information in this article is typical of many I have in my files.

I don't particularly like long copy and paste, but it seem needed in this case:
[...]
From: Science Is Self-Correcting ... Sort Of
August 16, 2011

http://sixdayscience.com...

There is definitely a problem with lack of education in statistics in many fields of science, maybe especially with psychology/neuroscience. It's something I complain about a lot =P. I can't think of a very good solution, though. Should we make it mandatory that anyone who wants to become a scientist at least double-major in statistics during their undergraduate degree? That would help, and I personally don't think it's a bad idea... As it is now, a lot of scientists only have a very surface level understanding of statistics, and so don't really understand how to evaluate their hypotheses and interpret their results rigorously.

Anyway, there is an issue with the quoted text where it assumes (for the sake of simpler calculation, I suppose), that all studies use a significance threshold of 0.05. The average alpha used in any field is quite a bit lower than that. But the main flaw is in overestimating the importance of a single paper. A scientific result is not validated by its associated type 1 and type 2 error. It's repeatedly finding the same result in different analogous studies which reduces the probability of these errors to quite acceptable levels. If we drop the implicit assumption that the 1000 studies contain no repeated trials of the same hypothesis, then the calculation of how many false positives and false negatives we would expect changes by a wide margin.
Welfare-Worker
Posts: 1,200
Add as Friend
Challenge to a Debate
Send a Message
3/16/2015 12:14:52 PM
Posted: 1 year ago
~~ quoye
Statisticians have ways to deal with such problems. But most scientists are not statisticians. Victoria Stodden, a statistician at Columbia, speaks for many in her trade when she says that scientists" grasp of statistics has not kept pace with the development of complex mathematical techniques for crunching data. Some scientists use inappropriate techniques because those are the ones they feel comfortable with; others latch on to new ones without understanding their subtleties. Some just rely on the methods built into their software, even if they don"t understand them.

This fits with another line of evidence suggesting that a lot of scientific research is poorly thought through, or executed, or both. The peer-reviewers at a journal like Nature provide editors with opinions on a paper"s novelty and significance as well as its shortcomings. But some new journals"PLoS One, published by the not-for-profit Public Library of Science, was the pioneer"make a point of being less picky. These "minimal-threshold" journals, which are online-only, seek to publish as much science as possible, rather than to pick out the best. They thus ask their peer reviewers only if a paper is methodologically sound. Remarkably, almost half the submissions to PLoS One are rejected for failing to clear that seemingly low bar.

Other data-heavy disciplines face similar challenges. Models which can be "tuned" in many different ways give researchers more scope to perceive a pattern where none exists. According to some estimates, three-quarters of published scientific papers in the field of machine learning are bunk because of this "overfitting", says Sandy Pentland, a computer scientist at the Massachusetts Institute of Technology.

The idea that there are a lot of uncorrected flaws in published studies may seem hard to square with the fact that almost all of them will have been through peer-review. This sort of scrutiny by disinterested experts"acting out of a sense of professional obligation, rather than for pay"is often said to make the scientific literature particularly reliable. In practice it is poor at detecting many types of error.

John Bohannon, a biologist at Harvard, recently submitted a pseudonymous paper on the effects of a chemical derived from lichen on cancer cells to 304 journals describing themselves as using peer review. An unusual move; but it was an unusual paper, concocted wholesale and stuffed with clangers in study design, analysis and interpretation of results. Receiving this dog"s dinner from a fictitious researcher at a made up university, 157 of the journals accepted it for publication.

Dr Bohannon"s sting was directed at the lower tier of academic journals. But in a classic 1998 study Fiona Godlee, editor of the prestigious British Medical Journal, sent an article containing eight deliberate mistakes in study design, analysis and interpretation to more than 200 of the BMJ"s regular reviewers. Not one picked out all the mistakes. On average, they reported fewer than two; some did not spot any.

Another experiment at the BMJ showed that reviewers did no better when more clearly instructed on the problems they might encounter. They also seem to get worse with experience. Charles McCulloch and Michael Callaham, of the University of California, San Francisco, looked at how 1,500 referees were rated by editors at leading journals over a 14-year period and found that 92% showed a slow but steady drop in their scores.

As well as not spotting things they ought to spot, there is a lot that peer reviewers do not even try to check. They do not typically re-analyse the data presented from scratch, contenting themselves with a sense that the authors" analysis is properly conceived. And they cannot be expected to spot deliberate falsifications if they are carried out with a modicum of subtlety.

Fraud is very likely second to incompetence in generating erroneous results, though it is hard to tell for certain. Dr Fanelli has looked at 21 different surveys of academics (mostly in the biomedical sciences but also in civil engineering, chemistry and economics) carried out between 1987 and 2008. Only 2% of respondents admitted falsifying or fabricating data, but 28% of respondents claimed to know of colleagues who engaged in questionable research practices.

Peer review"s multiple failings would matter less if science"s self-correction mechanism"replication"was in working order. Sometimes replications make a difference and even hit the headlines"as in the case of Thomas Herndon, a graduate student at the University of Massachusetts. He tried to replicate results on growth and austerity by two economists, Carmen Reinhart and Kenneth Rogoff, and found that their paper contained various errors, including one in the use of a spreadsheet.

[Spoiler alert: "Customers" discussed:]
People who pay for science, though, do not seem seized by a desire for improvement in this area. Helga Nowotny, president of the European Research Council, says proposals for replication studies "in all likelihood would be turned down" because of the agency"s focus on pioneering work. James Ulvestad, who heads the division of astronomical sciences at America"s National Science Foundation, says the independent "merit panels" that make grant decisions "tend not to put research that seeks to reproduce previous results at or near the top of their priority lists". Douglas Kell of Research Councils UK, which oversees Britain"s publicly funded research argues that current procedures do at least tackle the problem of bias towards positive results: "If you do the experiment and find nothing, the grant will nonetheless be judged more highly if you publish."

And scientists themselves, Dr Alberts insisted, "need to develop a value system where simply moving on from one"s mistakes without publicly acknowledging them severely damages, rather than protects, a scientific reputation." This will not be easy. But if science is to stay on its tracks, and be worthy of the trust so widely invested in it, it may be necessary.
http://www.economist.com...-
Welfare-Worker
Posts: 1,200
Add as Friend
Challenge to a Debate
Send a Message
3/16/2015 12:20:28 PM
Posted: 1 year ago
~~ ~ quote
The ability to self-correct is considered a hallmark of science. However, self-correction does not always happen to scientific evidence by default. The trajectory of scientific credibility can fluctuate over time, both for defined scientific fields and for science at-large.
History suggests that major catastrophes in scientific credibility are unfortunately possible and the argument that "it is obvious that progress is made" is weak. Careful evaluation of the current status of credibility of various scientific fields is important in order to understand any credibility deficits and how one could obtain and establish more trustworthy results. Efficient and unbiased replication mechanisms are essential for maintaining high levels of scientific credibility. Depending on the types of results obtained in the discovery and replication phases, there are different paradigms of research: optimal, self-correcting, false nonreplication, and perpetuated fallacy. In the absence of replication efforts, one is left with unconfirmed (genuine) discoveries and unchallenged fallacies. In several fields of investigation, including many areas of psychological science, perpetuated and unchallenged fallacies may comprise the majority of the circulating evidence. I catalogue a number of impediments to self-correction that have been empirically studied in psychological science. Finally, I discuss some proposed solutions to promote sound replication practices enhancing the credibility of scientific results as well as some potential disadvantages of each of them. Any deviation from the principle that seeking the truth has priority over any other goals may be seriously damaging to the self-correcting functions of science.
http://pps.sagepub.com...

Then to say that all theories will eventually be self corrected is a matter of faith and is not a deduction. Why? It can be possible that every theory to date has been improved, and will continue to be improved, but it does not follow that all new theories will also fit this paradigm.
Also, it has not been demonstrated that all theories now "in play" have been self-correcting. It could very well be, and there is some evidence to suggest, that some theories are racing down blind alleys, self-destructing, as it were. This usually happens when theories are based on a false philosophies"and all physics must first needs a philosophy. For example, that the "laws" of physics work everywhere and everywhen identically is a philosophical and not physical idea. Multiverses and many-worlds seem to be examples of blind-alley theories.

Scientism is also incapable of ultimate self-correction. Scientism is the false belief that all theories are ultimately scientific; i.e. it is a futile attempt at supplanting philosophy, but which is a religion which only succeeds in masquerading its philosophy.
But let these examples pass, because they are beside the point. What is true is that to say that all theories are capable of self-correction is a matter of faith and is not a deduction. Given mankind"s pertinacious grip of error, nothing would seem more obvious than some theories can be perpetually wrong.
The Self-Correcting Fallacy is rarely stated blankly as the scientist who insists he is right because Science is self-correcting. But it"s not too far off, either. How often have we heard the phrase "The Science is settled"? If the science is settled, it is not in need of self-correction, and is therefore purged of error. Or perhaps some small amount of error is allowed"which, it is assumed, will itself be self-corrected"but because science is self-correcting, theories that reach public awareness must be "good enough" already. This is obviously false.
What remains true is that each theory must be judged on its own merits, and not on the merits of its expounders or that it was capable of self-correction.
Lastly, there is also a whiff of arrogance in the SCF. Scientists boast of science making improvements, and imply that other intellectual endeavors do not share this superior attribute. This is ridiculously false, a belief which can only be the result of an ignorance of human thought. For instance, history routinely improves its understanding, and even theology improves in time. Even a cursory reading in, say, the theology of Christology confirms this.
Of course, history, theology, and other humanities are awful prone to blind alley theories, too. But we have already seen science is not immune to these. We leave with the wisdom of Max Planck:
A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.
Which is at least proof that not all scientists are capable of self-correction.
http://wmbriggs.com...
johnlubba
Posts: 2,892
Add as Friend
Challenge to a Debate
Send a Message
3/16/2015 2:42:22 PM
Posted: 1 year ago
At 3/15/2015 7:04:36 PM, Wylted wrote:
According to a recent scientific study, there are too many scientific studies.

http://science.slashdot.org...

Thoughts?

Scientists have discovered, that people will believe anything if you say, "Scientists have discovered".
Garbanza
Posts: 1,997
Add as Friend
Challenge to a Debate
Send a Message
3/17/2015 2:07:06 AM
Posted: 1 year ago
At 3/15/2015 7:04:36 PM, Wylted wrote:
According to a recent scientific study, there are too many scientific studies.

http://science.slashdot.org...

Thoughts?

Nowadays, academics' jobs are linked to their publication/citation numbers. For instance, in my school all full time academics are expected to publish a minimum of 3 papers a year, and all promotions/internal funding etc. will depend on their publication/citation numbers. It's a system that is heavily gamed. The most common thing is for people to put other authors on their papers who had only trivial input, or sometimes no input at all. Also, people will break up a piece of research into as many papers as possible. They will also design research with papers and citations in mind. And of course all students are pressured to publish anything and everything they can. There's never the consideration that the quality might be too low, it's always will it get in and what's the journal's impact factor. That's all that matters.

Sometimes I read papers from the 60s before this all started and they're so charming because they may be short papers reporting on a whole series if experiments and years of thought. Definitely higher quality in that respect.
Wylted
Posts: 21,167
Add as Friend
Challenge to a Debate
Send a Message
3/17/2015 7:43:02 AM
Posted: 1 year ago
At 3/17/2015 2:07:06 AM, Garbanza wrote:
At 3/15/2015 7:04:36 PM, Wylted wrote:
According to a recent scientific study, there are too many scientific studies.

http://science.slashdot.org...

Thoughts?

Nowadays, academics' jobs are linked to their publication/citation numbers. For instance, in my school all full time academics are expected to publish a minimum of 3 papers a year, and all promotions/internal funding etc. will depend on their publication/citation numbers. It's a system that is heavily gamed. The most common thing is for people to put other authors on their papers who had only trivial input, or sometimes no input at all. Also, people will break up a piece of research into as many papers as possible. They will also design research with papers and citations in mind. And of course all students are pressured to publish anything and everything they can. There's never the consideration that the quality might be too low, it's always will it get in and what's the journal's impact factor. That's all that matters.

Sometimes I read papers from the 60s before this all started and they're so charming because they may be short papers reporting on a whole series if experiments and years of thought. Definitely higher quality in that respect.

This makes me sad.
Garbanza
Posts: 1,997
Add as Friend
Challenge to a Debate
Send a Message
3/17/2015 8:02:35 AM
Posted: 1 year ago
At 3/17/2015 7:43:02 AM, Wylted wrote:
At 3/17/2015 2:07:06 AM, Garbanza wrote:
At 3/15/2015 7:04:36 PM, Wylted wrote:
According to a recent scientific study, there are too many scientific studies.

http://science.slashdot.org...

Thoughts?

Nowadays, academics' jobs are linked to their publication/citation numbers. For instance, in my school all full time academics are expected to publish a minimum of 3 papers a year, and all promotions/internal funding etc. will depend on their publication/citation numbers. It's a system that is heavily gamed. The most common thing is for people to put other authors on their papers who had only trivial input, or sometimes no input at all. Also, people will break up a piece of research into as many papers as possible. They will also design research with papers and citations in mind. And of course all students are pressured to publish anything and everything they can. There's never the consideration that the quality might be too low, it's always will it get in and what's the journal's impact factor. That's all that matters.

Sometimes I read papers from the 60s before this all started and they're so charming because they may be short papers reporting on a whole series if experiments and years of thought. Definitely higher quality in that respect.

This makes me sad.

Me too. It's a bit like debating, though, and the votes being gamed. It's annoying, but it only invalidates the voting system and elo. There are still good and interesting debates. Anyway, maybe it was always like this. People always game whatever system there is.
slo1
Posts: 4,361
Add as Friend
Challenge to a Debate
Send a Message
3/20/2015 10:36:10 AM
Posted: 1 year ago
At 3/15/2015 7:04:36 PM, Wylted wrote:
According to a recent scientific study, there are too many scientific studies.

http://science.slashdot.org...

Thoughts?

I disagree with the authors conclusion. Just because there is an increase in studies and on average published studies are not cited for as long as they were in the past does not mean there is too many studies.

Is it possible this change in citation rate is because there are more scientists and technology to enable more studies. Coupled with the fast pace of knowledge and discovery, it explains why new knowledge may not be hemmed and hawed over for as long is it did in the past.

Also with technology and organizations that house studies, it is very easy for scientists to find relevant studies. Why should they focus on knowledge and studies that are irrelevant to their area?

By coming to the conclusion that he has, the author risks reducing the rate and pace of new discoveries in an era unprecedented in scope and breath as far as contributions to human knowledge. Why risk slowing that progress down?

The true determiner of whether there are too many studies is whether the quality has gone down because the system can not or does not perform the same level of validation prior to publishing.
Wylted
Posts: 21,167
Add as Friend
Challenge to a Debate
Send a Message
3/20/2015 10:42:09 AM
Posted: 1 year ago
At 3/20/2015 10:36:10 AM, slo1 wrote:
At 3/15/2015 7:04:36 PM, Wylted wrote:
According to a recent scientific study, there are too many scientific studies.

http://science.slashdot.org...

Thoughts?

I disagree with the authors conclusion. Just because there is an increase in studies and on average published studies are not cited for as long as they were in the past does not mean there is too many studies.

Is it possible this change in citation rate is because there are more scientists and technology to enable more studies. Coupled with the fast pace of knowledge and discovery, it explains why new knowledge may not be hemmed and hawed over for as long is it did in the past.

Also with technology and organizations that house studies, it is very easy for scientists to find relevant studies. Why should they focus on knowledge and studies that are irrelevant to their area?

By coming to the conclusion that he has, the author risks reducing the rate and pace of new discoveries in an era unprecedented in scope and breath as far as contributions to human knowledge. Why risk slowing that progress down?

The true determiner of whether there are too many studies is whether the quality has gone down because the system can not or does not perform the same level of validation prior to publishing.

It probably just means there needs to be more specialized scientific fields.