Fixing Science...

The prominence of science and technology in society is expanding rapidly by most measures, but this expansion could be healthier.

Outside Threats:

Fundamentalists are attacking science and winning.
While the importance of science is technology are increasing extremely fast, actual scientific literacy in this country is said to be declining.

Threats from Within:

"Discoveries of the Year" are turning out to be complete hoaxes, slowing progress and hurting the scientific spirit.
Many fields are flooded with an over-abundance of "noise" papers that nobody believes anyway, again slowing progress and hurting the scientific spirit.
In my own field I witness data manipulation and suppression regularly.

There must be a better way - in the coming century, the international funding and practical impact of science will scale by an order of magnitude - we must be ready with an efficient and reliable apparatus to absorb this growth stably.

1. A Human Problem and Solution

A Thought about How to Improve Peer Review

The Problem: The current peer review process seems increasingly politicized, sloppy, or both.

Politicized: reviewers under intense competitive pressure are biased against findings that are contrary to their position, either defensively or vengefully.]]>
Sloppy: otherwise busy reviewers are not rewarded for doing a good job, and other pressures win out for their time.]]>

The Cause: Many have argued that this stems almost entirely from the fact that it is anonymous.

No Carrot: Reviewers get almost no payoff for doing a good job on a review.
No Stick: Reviewers can get away with sabotaging papers or passing off papers to inexperienced students because they are not held accountable.

The Solution: Publish reviewers' names with the papers for accepted papers only.

This in a sense turns the reviewer into an additional author, and a very careful and skeptical one at that.
The quality of papers will increase dramatically as reviewers' standards and advice for the authors will improve, since their own credibility will be on the line.
Furthermore, reviewers will be more enticed to perform reviews and see their name "up in lights," as this will increase their visibility and perceived authority.
Reviewers are still free to reject bad papers without fearing for their safety.

Do you see a hidden cost? Contact me and tell me!

Note: Another great idea in the New York Times for correcting fraud was amusingly written.

2. A Human Problem and Computer Solution

A Thought about How to Improve Data Integrity

The Problem: A lot of the current literature is based on biased data, flaky reporting or faulty analysis.

The Cause: Due to the economics of science, investigators are rushing papers, grants and posters to press and bypassing documentation and oversight - there is inadequate verification within labs and certainly between labs.

The Solution: Computers will bring the solution, by enabling honest researchers to share their data online (as physicists already do) and making it (one day) increasingly suspicion-arousing for investigators not to do so.

But How: There is obviously a growing field devoted to this, which is great. Tim Berners-Lee had a good explanation of Web 2.0 technologies that would be relevant. I for one am hoping to put the entire results from my next project fully online using a data abstraction language that I have been working on:

Data is broken down into fundamental object types, each with a class definition and specific parameters.
Data is acquired and stored in structured database tables that expand as more information on the object types is desired.
Data is acquired directly to this structured data store from the experimental apparatus in real time. Ideally the history of the acquisition (or a checksum thereof) is simultaneously logged on a remote trusted site.
The data tables and their fundamental types are related by "transforms," ideally represented in a universal languages such as Matlab. These transforms can then be chained together by a meta-language and stored for reference in tables alongside the data itself.
The structured data store is inherently web accessible, and is posted on the lab's website (and perhaps that of the funding agency) at the time of publication.

The Result:

All higher-order data is therefore simply the result of transform functions operating on links to the lower-order data.
Clearly this will take at least a lifetime to realize, but as young postdocs it is exciting to start thinking about these issues!
Once implemented, all published conclusions will be retrogradely traceeable back to their individual data elements.
Investigators will be able to quickly perform meta-analyses on one another's work, to verify and extend it. Data / bench hours can continue to be reused.
The use of the system will likely evolve voluntarily as a few early adopters benefit from the elevated credibility of their work. Alternatively, best practices could be enforced by grant agencies, who would then receive propagated credibility.
The end result would be a decrease in the amount of flaky reporting, faulty analysis and spurious claims.