The next-gen search engine?
The Defense Advanced Research Projects Agency (DARPA), Pentagon's scientific-innovation department, is developing a search engine capable of mining the entirety of the internet - the Surface Web, the Dark Web and the Deep Web. The $50 million project is headed by DARPA's Dr. Chris White.
The Deep Web and the Surface Web
The Deep Web consists of web pages which cannot be indexed by traditional search engines like Google. The Surface Web, commonly called the World Wide Web, consists of what traditional search engines can index. The Surface Web accounts for only 5% of the internet.
What is the Dark Web?
While most of the information on the Deep Web is mundane, it houses a collection of websites which are hidden from search engines and their IP addresses are masked using the Tor encryption tool. These websites are publicly accessible through the Tor browser. This collection of websites is called the Dark Web - websites run and used by terrorists, gangsters, pedophiles, human traffickers etc.
So what can you find on the Dark Web?
The Dark Web hosts a variety of websites from online black markets (like the Silk Route) to child pornography websites. Using bitcoins, any user can anonymously purchase drugs, firearms, explosives, hitmen, forged papers, counterfeit currency, human organs, sex slaves and so on.
DARPA's Memex project
Since goods can be bought or sold anonymously through the Dark Web's unregulated free markets, lots of illegal activities are carried on through the Dark Web. Therefore, DARPA's Memex project isn't intended to just index hidden websites. The creators of the project also want to analyze the content using automated methods to uncover patterns and relationships which might help law enforcement and the military.
The story behind the name "Memex"
The word "Memex" - a portmanteau of "memory" and "index" - was coined in 1945 by Vannevar Bush, the then-director of the Office of Scientific Research and Development. He envisioned it as a mechanical device which could compress and store all books, records and communications.
What is Memex?
Memex is a set of super-search-engine parts and tools which, in theory, can be organized for any number of real world operations. It could be used for listing Dark Web websites and unsearchable Deep Web and Dark Web online forums, for monitoring social media trends, reading handwriting, connecting photos, etc. Memex could then be used to cross-index the information into data maps.
Dr. Chris White on one of the uses of Memex
"We're trying to move toward an automated mechanism of finding [Hidden Services sites] and making the public content on them accessible," said Dr. White on one of the goals of the Memex project.
The first real-world use of Memex
In 2014, to test its effectiveness, Dr. Chris White decided to use Memex to help US law enforcement to track sex traffickers. Using Memex tools like TellFinder and Dig, police were not only able to capture a trafficker named Rosado, but also able to dig up enough evidence to imprison him. Since then, it has been actively used in the hunt for human traffickers.
Human trafficking victims
According to data by the International Labour Organization (ILO), there are 20.9 million victims of human trafficking globally. According to ILO estimates, human trafficking is a $150 billion industry.
Other potential real-life uses for Memex
Some of the varied potential uses of Memex includes tracking the flow of illegal guns and labour, the movements of ISIS recruits or propaganda. It could be used to establish links between money laundering and shell companies. It could also be used to track public sentiments and moods in real-time, and the frequency of social media mentions for words and ideas.
Who is Dr. Chris White?
Dr. Chris White served as DARPA's lead in Afghanistan before moving up the ranks of the organisation. One of the top DARPA inventors today, White created DARPA's Open Catalog tool. He also created and currently runs DARPA's XDATA, part of Obama's Big Data Initiative.
