Artificial Intelligence: How to Measure 'I' in AI

Last week, South Korea Go Champion Lee Se-dol announced his retirement from professional sports after losing in a historic match with DeepMind's Artificial Intelligence Algorithm AlphaGo in 2016.

Lee Yonhap told the news agency, "With the introduction of AI into the Go Game, I realized that I was not top, even though I was ranked first by Ventric's efforts." "Even though I'm number one, there's an unbeatable company."

Importantly, C-Dole's comments have quickly made the rounds of major tech publications, some of them using groundbreaking headlines with AI-dominated themes.

Since the inception of AI Internet Technology, games have been one of the main criteria for assessing the efficiency of algorithms. Thanks to advances in deep learning and reinforcement learning, AI researchers are creating programs that can learn the most complex games and defeat the most experienced players worldwide. Unaware of these achievements, analysts are coming to think that AI is smarter than humans.

But at the same time, contemporary AI fails some fundamentally worse that every human being can do.

This also raises the question, does mastering the game prove anything? If not, how can you measure the intelligence level of AI systems?

Take the following example. In the image below, you have three problems and their solutions. The fourth function is also unresolved. Can you afford the solution?

Do you think this is too easy. By looking at these three examples, you will be able to solve different forms of the same problem with multiple walls, and many lines and different colored lines. But right now, there are no AI systems, and the most ambitious research being developed in laboratories can learn to solve such a problem with very few examples.

The above example is a letter of "Measure of Intelligence" from Francois Cholet, the creator of the Cares Deep Learning Library. Cholet published the paper a few weeks before Le-Sedol announced his retirement. In it, he provided several important guidelines for understanding and measuring intelligence.

Ironically, chocolate paper doesn’t need to be noticed. Unfortunately, the media is more interested in covering exciting AI news that gets more clicks. The 62-page paper contains invaluable information and is a must-read for anyone who wants to understand the state of AI beyond propaganda and sensationalism.

I will try my best to summarize the key recommendations of how Cholet measures AI systems and compares their performance with human intelligence.

What is intelligence?

One of the major challenges facing the AI ​​community is defining intelligence. Scientists have been debating for decades about providing a clear definition that allows AI systems to be evaluated and intelligent.

Cholette defines this definition by Deep Mind cofounder Shane Legg and AI scientist Marcus Hutter: "Intelligence measures an agent's ability to achieve goals in a wide range of environments."

The key here is “achieving goals” and “the wider environment”. Most current AI systems are good in the first part, achieving very specific goals, but doing so in broad environments is bad. For example, the AI ​​system that can detect and classify objects in images does not perform some other related functions, such as images of objects.

Cholette examines two main approaches in building intelligence systems: symbolic AI and machine learning.

Symbolic AI vs Machine Learning

Early generations of AI research focused on symbolic AI, which provides a clear representation of knowledge and behavior in computer programs. This approach requires human engineers to carefully write down the rules that define the behavior of the AI ​​agent.

"It is widely accepted in the AI ​​community that the 'intelligence problem' will be solved if human skills can be incorporated into official rules and human knowledge can be incorporated into a clear database," sees Cholette.

But rather than being smarter themselves, these symbolic AI systems reveal the intelligence of their creators in creating complex programs that can solve specific tasks.

The second approach, Machine Learning Systems, relies on the AI ​​model to deliver data from the problem area and develop its own behavior. The most successful machine learning architecture so far is artificial neural networks, which are complex mathematical functions that can create complex mappings between inputs and outputs.

For example, instead of manually coding the cancer detection rules on X-ray slides, a neural network with multiple slides that you quote with your results is called "training." AI examines the data and develops the mathematical model that it introduces. General features of the cancer model. It processes new slides and outputs what patients have cancer.

The advancement of neural networks and deep learning has enabled AI scientists to solve many tasks that were previously difficult or impossible with classic AI, such as natural language processing, computer vision, and speech recognition.

Neural network-based models, also known as Connection AI, are named after their biological counterparts. The mind is based on the idea of ​​a "blank slate" (tabala rasa), which transforms experience (data) into behavior. Therefore, the general trend in intensive learning has become to solve problems by providing more training data to create larger neural networks and improve their accuracy.

Cholet rejects both approaches because they both lack the ability to create a common mind, which is as flexible and fluid as the human mind.

“We see the world through the lens of the devices we know. Today, these two ideas of the nature of human intelligence - a collection of special-purpose programs or a general-purpose tabla rasa - are clearly wrong, ”he writes.

Truly intelligent systems must be able to develop high-level skills that can accomplish many tasks. For example, Masters Quake 3 is an AI program that can play other first-person shooter games better. Unfortunately, the best the current AI system can do is "local generalization", a limited maneuvering room in its own narrow domain.

Comprehensive and general AI requirements

In his paper, Cholette argues that "generalization" or "normalization power" for any AI system

Comprehensive and general AI requirements

In his paper, he argues that the "normalization" or "normalization power" of any AI system is "the ability to handle situations (or tasks) that are different from those encountered in the past."

Interestingly, this is a missing part of both symbolic and connectionist AI. The former requires engineers to clearly define its range of behavior, and the latter requires examples that support its problem-solving domain.

Cholette goes further and talks about "developer-aware generalization", the ability of an AI system to handle "the system or system's developer has never encountered it".

This is the flexibility you would expect from a robot-butler who can perform various tasks inside the home without explicit instruction or training data. One example is Steve Wozniak's famous coffee test, in which the robot enters a random house and makes coffee without knowing about the layout of the house or the equipment it contains.

Somewhere in the paper, Cholett makes it clear that AI systems that manipulate priests (rules) and experience (data) are not intelligent enough to manipulate their goals. For example, consider the best rule-based chess-game program Stockfish. Stockfish, an open-source project, is the result of thousands of rules being created and provided by tens of thousands of developers. One neural network-based example is Alfajero, a multipurpose AI that has played many board games millions of times.

Two systems have been adopted to perform a particular task using resources beyond the capacity of the human mind. The bright man cannot remember thousands of chess rules. Similarly, no human being can play millions of chess games in a lifetime.

"Solving any task with human-level functionality by offering unlimited priests or the benefit of unlimited data is not close to comprehensive AI or general AI, be it chess, football or any e-sport." Chocolate note.

That is why comparing Deep Blue, Alpha Zero, AlphaStar or any other game-playing AI to human intelligence is completely wrong.

Similarly, a middle school student does not have the same knowledge as other AI models, such as Aristo, who can pass the eighth grade science test. It owes its scientific capabilities, not to the understanding of the world of science, but to the vast enterprise of science.

(Note: Some AI researchers, such as computer scientist Rich Sutton, believe artificial intelligence research should be the right direction for data availability and resource quantification.)

Extraction is a logical corpus

In the paper, Cholet demonstrates the Abstraction Reasoning Corpus (ARC), which aims to evaluate the efficiency of AI systems and compare their performance to human intelligence. ARC is a set of problem-solving tasks tailored to AI and humans.

One of the main ideas behind ARC is to level the playing field between humans and AI. Humans are designed to take advantage of the vast background knowledge of their world to adopt AI. For example, AI systems do not include language-related problems that have historically been fought.

On the other hand, it is designed to prevent AI (and its developers) from betraying their way of success. The system does not provide access to large amounts of training data. As shown in the example at the beginning of this article, each concept is presented with some examples.

AI developers need to build a system that can handle various concepts such as object coordination, object persistence and object effects. AI systems need to learn tasks like scaling, drawing, connecting points, rotating and translating.