The Complex Dance of Language Models and Tabular Data
November 11, 2024, 9:48 pm
In the world of artificial intelligence, language models are the stars of the show. They can generate text, answer questions, and even hold conversations. But when it comes to understanding tabular data, the spotlight dims. Tables are like intricate puzzles, each cell a piece that fits into a larger picture. Language models, however, are trained on linear sequences of text, making the two-dimensional nature of tables a challenge.
This article delves into the evaluation of language models in their ability to interpret and respond to questions based on tabular data. The research team at IDP has embarked on a journey to uncover how well these models can navigate the complexities of tables, especially in technical documentation where such data is prevalent.
The team’s interest in this topic was sparked by a project involving a question-answering system for technical documents. Many of the inquiries centered around tables, which often contained lengthy column names, formulas, and multi-level headers. This complexity led to a ceiling in performance metrics, prompting the team to conduct a more thorough investigation.
The research focused on a single task and a specific type of question. The team selected tables from two primary sources: encyclopedias like Wikipedia and technical documentation such as GOST standards. They also synthesized additional datasets to assess the impact of value homogeneity and text complexity within cells. The tables were preprocessed, converting them from HTML to markdown, a format commonly used in language model training. Multi-level headers were flattened, merged cells were split, and text in split cells was duplicated. The tables varied in size, with widths ranging from 2 to 16 columns and heights from 1 to 64 rows.
The tables were categorized based on several criteria: dimensions, value homogeneity, and text volume within cells. This categorization was crucial for understanding how different models would perform under varying conditions.
The core task for the models was to comprehend the relationships between different rows and columns. To test this, the team formulated questions that required comparisons across at least two rows or columns. The most common question structure was: “What is the value in the target column if the value in the query column equals X?” This question format necessitated not only locating the specified columns but also identifying the target row based on the provided value, followed by extracting the answer from the corresponding cell.
The researchers generated questions algorithmically, ensuring that each question had a unique answer. This approach minimized the chances of models guessing the answer based on frequency. For instance, a question might be: “What is the surface if the opponent in the final is Leslie Kerhov?” The answer would be “Hard.”
The prompt format for the language models was carefully crafted to ensure clarity. The system prompt instructed the models to respond based solely on the data within the provided table. The evaluation metric used was exact match: a score of 1 for a correct answer and 0 otherwise.
The results were visualized through heat maps, illustrating the performance of various models across different table sizes and column proximities. The heat maps revealed patterns in model performance, highlighting areas where models excelled or struggled.
The research included several models, both open-source and commercial, ranging from medium to large sizes. The models were evaluated on their ability to answer questions accurately across different datasets. The findings indicated that while some models performed admirably, others struggled, particularly with complex or homogenous data.
Notably, the Qwen-2.5 model, despite having fewer parameters than some of its counterparts, outperformed expectations. Among larger models, Llama-3.1-405B and GigaChat-Max emerged as leaders in accuracy. The analysis revealed that models performed better on datasets with heterogeneous values, where semantic understanding could guide them to the correct answer.
The challenges faced by the models were evident, especially with synthetic datasets where values were uniform. In such cases, the models needed to “count” the number of columns to arrive at the correct answer, a task that proved difficult for smaller models. The ability to count and navigate the table structure seemed to correlate with model size and training quality.
The research concluded that even the most advanced language models struggle with complex tabular data. This limitation raises questions about the future of AI in processing such information. Will these models evolve to handle the intricacies of tables, or will this remain a hurdle for text-based models?
The team expressed interest in exploring multimodal models that combine visual and textual data, such as GigaChat Vision. They plan to investigate more complex tasks involving tables and integrate new benchmarks into their evaluation framework.
In summary, the journey of language models through the realm of tabular data is fraught with challenges. While they have made significant strides, the path ahead is still laden with obstacles. As researchers continue to push the boundaries of AI, the hope remains that future models will master the art of understanding tables, unlocking new possibilities in data interpretation and analysis.
This article delves into the evaluation of language models in their ability to interpret and respond to questions based on tabular data. The research team at IDP has embarked on a journey to uncover how well these models can navigate the complexities of tables, especially in technical documentation where such data is prevalent.
The team’s interest in this topic was sparked by a project involving a question-answering system for technical documents. Many of the inquiries centered around tables, which often contained lengthy column names, formulas, and multi-level headers. This complexity led to a ceiling in performance metrics, prompting the team to conduct a more thorough investigation.
The research focused on a single task and a specific type of question. The team selected tables from two primary sources: encyclopedias like Wikipedia and technical documentation such as GOST standards. They also synthesized additional datasets to assess the impact of value homogeneity and text complexity within cells. The tables were preprocessed, converting them from HTML to markdown, a format commonly used in language model training. Multi-level headers were flattened, merged cells were split, and text in split cells was duplicated. The tables varied in size, with widths ranging from 2 to 16 columns and heights from 1 to 64 rows.
The tables were categorized based on several criteria: dimensions, value homogeneity, and text volume within cells. This categorization was crucial for understanding how different models would perform under varying conditions.
The core task for the models was to comprehend the relationships between different rows and columns. To test this, the team formulated questions that required comparisons across at least two rows or columns. The most common question structure was: “What is the value in the target column if the value in the query column equals X?” This question format necessitated not only locating the specified columns but also identifying the target row based on the provided value, followed by extracting the answer from the corresponding cell.
The researchers generated questions algorithmically, ensuring that each question had a unique answer. This approach minimized the chances of models guessing the answer based on frequency. For instance, a question might be: “What is the surface if the opponent in the final is Leslie Kerhov?” The answer would be “Hard.”
The prompt format for the language models was carefully crafted to ensure clarity. The system prompt instructed the models to respond based solely on the data within the provided table. The evaluation metric used was exact match: a score of 1 for a correct answer and 0 otherwise.
The results were visualized through heat maps, illustrating the performance of various models across different table sizes and column proximities. The heat maps revealed patterns in model performance, highlighting areas where models excelled or struggled.
The research included several models, both open-source and commercial, ranging from medium to large sizes. The models were evaluated on their ability to answer questions accurately across different datasets. The findings indicated that while some models performed admirably, others struggled, particularly with complex or homogenous data.
Notably, the Qwen-2.5 model, despite having fewer parameters than some of its counterparts, outperformed expectations. Among larger models, Llama-3.1-405B and GigaChat-Max emerged as leaders in accuracy. The analysis revealed that models performed better on datasets with heterogeneous values, where semantic understanding could guide them to the correct answer.
The challenges faced by the models were evident, especially with synthetic datasets where values were uniform. In such cases, the models needed to “count” the number of columns to arrive at the correct answer, a task that proved difficult for smaller models. The ability to count and navigate the table structure seemed to correlate with model size and training quality.
The research concluded that even the most advanced language models struggle with complex tabular data. This limitation raises questions about the future of AI in processing such information. Will these models evolve to handle the intricacies of tables, or will this remain a hurdle for text-based models?
The team expressed interest in exploring multimodal models that combine visual and textual data, such as GigaChat Vision. They plan to investigate more complex tasks involving tables and integrate new benchmarks into their evaluation framework.
In summary, the journey of language models through the realm of tabular data is fraught with challenges. While they have made significant strides, the path ahead is still laden with obstacles. As researchers continue to push the boundaries of AI, the hope remains that future models will master the art of understanding tables, unlocking new possibilities in data interpretation and analysis.