IBM Watson WINS human Jeopardy enemies

Wrapping up a three-day run on jeopardy game show, IBM Watson computer has beaten two former champions in a match against the town of man machine.

Execution has successfully demonstrated not only that a computer can beat humans in a quiz question of curiosity, but, above all, it shows how the computer can answer questions much like people, potentially opening up a new form of human/computer interaction.

In the final episode of game two-and three nights pre-recorded, Watson had saw competition, accumulating US $ 77,147 in winnings over the two Jeopardy champions played, Brad Rutter and Ken Jennings. Rutter scored $ 21,600 and Jennings scored $ 24,000. Watson took the prize sample of 1 million dollars, that IBM will donate to charity.

Managed by Sony Pictures Television, danger is a longtime U.S. television game where three contestants compete to answer questions of curiosity, divided into multiple categories and sorts of difficulties growing. Contestants are given an average of about 5 seconds to answer a question.

Researchers at IBM spent four years building Watson. The machine can process trillions of 80 (teraflops) operations per second. Runs approximately 2800 processor core and has 16 terabytes of memory.

Construction of such a system to play on the danger has proved to be a huge project, a much more challenging to build a supercomputer-play chess, that IBM did in the 1990s.

"It is a very different kind of problem. Chess was very challenging for the time because of mathematics. This was a very different type of program, "said Watson lead manager David Ferrucci, in an IBM viewing party held in New York to show Wednesday. "There's more problem or a space. You are dealing with ambiguity and contextual nature of language. "

On the software side, the machine uses Apache Hadoop distributed file system and the Apache UIMA (unstructured information Management Architecture), a framework for the analysis of unstructured data. Perhaps the most useful software, however, is a program of natural language processing called DeepQA that IBM supports phrase human can understand. This program is Watson what differentiates a typical search engine, which just might return a results list of a set of keywords.

The questions were recruited Watson from the text; It has used speech recognition technology. For these tours, Jeopardy avoided questions involving audio or video snippets. Watson, however, answer questions in a synthesized voice.

To build a body of knowledge for Watson, researchers have accumulated 200 million pages of content, both structured and unstructured, across 4 terabytes of disks. It looks for matches and then uses rules to approximately 6 million euro to determine the best answers. When a query is specified, the software analyzes initially, identifying any names, dates, geographical locations or other entities. In addition, it examines the sentence structure and grammar of the question for hints of what you're asking the question.

The first night of jeopardy game, held on Monday, both man and machine seemed to be on equal footing, with Watson tied with Rutter $ 5,000 and Jennings followed with $ 2000. From Tuesday, however, Watson has started to show muscles: Watson has led the evening with $ 35,734, Rutter followed with $ 10,400 and Jennings towed $ 4800.

On Wednesday, the machine scored well above the man competitors, thanks not only to his immense body of knowledge, but also for algorithms researchers have put in place to make the best bets. To twice daily, one question hidden special where the competitor is allowed to bet any amount of its companies, Watson bet a seemingly arbitrary 2127 $, a number that the public found it funny.

These computerized wages "are seemingly random to us mere mortals," says Ferrucci. "But what is happening in reality is that it is considering its confidence in the category. Also where is considering what is at stake, how far ahead or behind you forward, how much money still can potentially be won or lost. All that adds up to a rather complex calculation. Get the numbers that are optimized for this precision down. "

While Watson performed flawlessly in many cases, it was also capable of flubs also casual Jeopardy watchers could laugh. Show on Tuesday, when asked for the largest airport in the U.S. take a hero of World War II, he responded with Toronto, the name of a Canadian city. Show on Wednesday asking question lost the name of a known reference book, "the elements of style". This question, Watson was inscrutably and confidently replied "Dorothy Parker".

While IBM has no plan for revenge or a version 2 of Watson, Watson mean technology in various fields such as health, where, through a specific body of knowledge, could answer tough questions.

"I think Watson has the potential to transform the way that people interact with computers," said Jennifer Chu-Carroll, an IBM researcher working for the project, told Computerworld. "Watson is a significant step, allowing people to interact with a computer as they would a human being. Watson does not give you a list of documents to go through but offers the user a reply ".

Joab Jackson covers enterprise software and General technology breaking news for the IDG News Service. Follow Joab on Twitter at @ Joab_Jackson. E-mail address of Joab is Joab_Jackson@idg.com

IBM Watson brain similar to the human being

IBM Supercomputer, which Watson shellacked higher human samples of danger during the exhibition game news this week, is powered by a 90-cluster server and attacked the network storage (NAS) with 9 .6TB of data.

In the end, though, his brain has only 80% of the processing power of a human brain.

Tony Pearson, master inventor and senior consultant at IBM, he explained that Watson uses only about 1 TB to process real-time answers to the questions of danger after configuring its storage back-end as RAID and then killing over data to be loaded into memory the system server in the cluster.

Pearson cited estimate technology futurist and author Ray Kurzweil that the human brain can hold approximately 1.25TB of data and performs approximately 100 teraflops. In comparison, Watson is an 80-teraflop with 1 TB of memory.

"So is 80% human," said Pearson. "Yes, we could have dealt with many other information. We could have put more memory on each server, but once we have the answers to three seconds, we don't need to go beyond ".

Pearson explained that reach the threshold of three second response was only a matter of simple math.

The original algorithm identified threaded on a single core processor took two hours to scan and memory to produce a response. So IBM technologists divided only two hours to 2880 CPU, who produced the ability to answer questions in three seconds.

If IBM Watson were only a few other human Jeopardy contestant, viewers probably would have tuned in using such an overwhelming victory. However, interest in the man vs machine battle gave the show its highest ratings in nearly six years.

Competition between humans and computers have long captured the imagination of the public. Remember that the chess match between the 1996 computer Deep Blue IBM and world champion Garry Kasparov?

In this case, however, Watson has more in common with humans more Deep Blue. Like us, he uses only a small percentage of its huge storage capacity to answer questions.

Behind Watson simple scribble in front of the monitor that used it as a competitor of danger are 90 IBM Power 750 Express powered by 8-core processors--four on each machine for a total of 32 processors per machine. The servers are virtualized by using an implementation KVM (kernel-based Virtual Machine), creating a server cluster with a total processing capacity of 80 teraflops. A teraflop is a trillion operations per second.

On top of processing power, each server has 160 GB DRAM to provide the complete machine with nearly 15 TERABYTES of memory.

On the backend of the computer is a general parallel File System SONAS (GPFS from IBM). SONAS, or NAS, scalability is a Linux-based cluster system files that IBM released almost exactly a year ago.

The clustered storage model provides massive throughput due to a larger port count that comes from the compilation of many archiving servers together into a single pool of drives and processors all work on a similar task and everyone can share the same data through a single global namespace. In other words, all disk drives appear as a large pool of storage capabilities from which they can draw Watson.

Watson SONAS is populated with 48 450 GB serial ATA (SATA) hard drives for a total of 21.61TB capacity in a RAID 1 (mirrored); that leaves 10 TB of raw data that is used by Watson every time it starts. Three terabytes of which, however, is used for the operating system and applications.

But this is not the disk-based storage that makes Watson SONAS so darned fast; This is CPU and memory. Each time you start Watson, 10.8 TB data is automatically loaded in RAM 15 TB Watson and that, only about 1 TB is parsed for use in answering questions of danger, said Pearson.

In case you're wondering, 1 TB of capacity is still fairly significant; It can hold 220 million pages of text or 111 DVD.

"The remarkable thing is that you can get all the answers with such a small set of data," said John Webster, an analyst with research firm Evaluator Group. "After more iterations of load and test and test and loading and updating the database on IBM SONAS, came up with a version of the database that would generate the dataset that you have loaded into memory."

Enter the Australian and SAMBA developer programmer Andrew Tridgell.

Tridgell created the algorithm of computers that are running on top of Watson that culls out hardware for data set. Tridgell developed an open source Database clustering (CTDB) banal, that the SAMBA file protocol uses to access the memory between 90 server Watson.

More importantly, the CTDB ensures that none of the server are stepping on each other as they also update the information after a show of danger.

During the show, Watson is read-only-meaning that backend SONAS gets written anything. After the show, Watson is off and the computer scientists go to work to update information and debugging--trying to figure out why gave incorrect answers, such as the choice of Toronto as the answer to a question about American city.

"I'm sure they're scratching their head about that," said Pearson.

Lucas Mearian covers storage, disaster recovery and business continuity, the infrastructure of the financial services and healthcare IT for Computerworld. Follow Lucas on Twitter @ lucasmearian or Subscribe to the RSS feed of Lucas. His e-mail address is lmearian@computerworld.com.

Learn more about mainframes and supercomputers in Computerworld Mainframes and supercomputers topic Center.

For more enterprise computing news, visit Computerworld. Story copyright © 2010 Computerworld Inc. All rights reserved.