1. Introduction

CityGML (City Geography Markup Language) is one important 3D city model standard supporting the creation of comprehensive digital twin of cities. However, despite its importance and utility, working with CityGML still remains a challenging task for users, especially those without a technical background. The format is complex, containing intricate hierarchical relationships between entities such as buildings, roads, and infrastructure. The size of these datasets, often spanning entire cities, further exacerbates the problem, making them difficult to query, analyse, and interpret without specialized knowledge, tools, and expertise.

Current methods for interacting with CityGML involve GIS platforms or customized software tools. These tools typically rely on spatial databases like PostGIS along with querying languages like SQL. They often demand significant time and expertise to master. Consequently, users who are not GIS professionals, such as urban planners, architects, or local government officials, may find it difficult to extract meaningful insights from large-scale CityGML datasets.

To address these challenges, we propose a novel solution named KCitychatBot: a knowledge graph-based chatbot system that leverages Large Language Models (LLMs) multi-agents to provide an intuitive, conversational natural language interface for querying and analysing large-scale CityGML data. KCitychatBot enables users to interact with complex CityGML urban models using ordinary natural languages as interface. The test demonstrated the usability of the proposed solution. This system not only simplifies querying but also encourage CityGML users without a technical background to ask questions, which can be particularly valuable in urban planning and decision-making processes with large-scale CityGML dataset.

2. State of the Art

The steep learning curve of traditional GIS platforms has driven research into developing new interaction methods, such as graphic workflows or chatbots, to lower the barrier for interacting with geospatial data. Among these methods, geospatial chatbots are often considered the most intuitive. However, early solutions (Tsai et al., 2019) had limited abilities to handle large, complex geospatial datasets due to their reliance on simple keyword-based queries. Recent developments of Large Language Models (LLMs) offer opportunities for advancing the capabilities of geospatial chatbots. The use of LLMs for geospatial tasks has already demonstrated promising results with 2D geospatial data (Wang et al., 2024). However, LLMs come with limitations that need to be addressed for large-scale geospatial applications. One key issue is the token input limit. LLMs like GPT-4.5 have a restricted number of tokens that can be processed in a single query. Another key issue is hallucination in LLMs, which refers to the model's tendency to generate plausible-sounding but incorrect or fabricated information.

Aiming at these limitations, some studies using RAG-based approaches in geospatial chatbots (Yu et al., 2025) have shown promising results but also highlight the complexity of synchronizing LLMs with large-scale geospatial datasets through RAG. Many researchers recognize the importance of knowledge graphs in chatbot systems (Luo et al., 2022). Knowledge graphs could not only enhance the chatbot's ability to interpret and process complex queries but also allow for more advanced reasoning capabilities. They may help mitigate the LLM chatbot hallucination issue by providing a structured and accurate reference for the model.

3. Methodology

The research described in this manuscript presents a chatbot system for a large-scale CityGML dataset. It follows the basic process of Knowledge-graph Retrieval-Augmented Generation (RAG) and employs a multi-agent framework. The chatbot integrates a city knowledge-graph database derived from the CityGML dataset and three collaborative LLM agents. For knowledge graph construction, this study proposes a hierarchical construction method tailored for CityGML using the Neo4j graph database. The functionalities of each intelligent agent are as follows:

Input2Cypher Crew: This agent handles conversations with users and automatically constructs Cypher queries (a query language for graph databases) based on the dialogue. It manages two sub-tasks: Extract_Task and Query_Task. The Extract_Task receives domain knowledge from the CityGML knowledge graph and analyses user input, processing it through an embedding model to extract user intent, relevant tags, and attribute information. Based on the results of Extract_Task, the Query_Task uses multiple preset template cases to automatically construct appropriate Cypher queries tailored to the city knowledge graph.

Execute Crew: This agent executes the Cypher queries generated by the Input2Cypher Crew and validates the query results from the knowledge graph. The system incorporates a three-iteration mechanism to mitigate the inherent randomness of LLMs. If the query result is empty or contains a syntax error, the process returns to the Input2Cypher Crew for further refinement.

Generate Crew: This agent processes the query results provided by the Execute Crew and converts them into natural language expressions aligned with human communication conventions.

When users interact with the system, the Input2Cypher Crew transforms the conversation input into a structured Cypher query, which is passed to the Execute Crew. The Execute Crew retrieves results from the knowledge graph. If valid results are returned, they are forwarded to the Generate Crew to produce natural language responses; otherwise, the results are sent back to the Input2Cypher Crew to iterate and refine the query generation process. Additionally, if users engage in multi-round conversations, the Generate Crew concatenates user inputs and system outputs into a contextual list to maintain coherence for subsequent interactions.

4. Evaluation

We utilize the PLATEAU dataset from Japan (Seto et al., 2023) to develop and validate our chatbot system through comprehensive case studies. Given that our system is based on a knowledge graph, the evaluation workflow comprises two consecutive steps: evaluating the accuracy of the knowledge graph and assessing the performance of the chatbot.

To assess the accuracy of the knowledge graph, we compare query results obtained from both the 3D CityDB (SQL) and Neo4j (Cypher). The comparison demonstrates identical outcomes from both querying approaches, thereby confirming the validity of the constructed city knowledge graph and indirectly validating the effectiveness of our graph construction method.

To evaluate chatbot performance, we built a test set with over 200 cases covering diverse question types, and utilized the RAGAS evaluation tool. Results indicate strong performance across two core metrics: answer relevance and answer faithfulness. Specifically, the system achieves an answer relevance score of 0.95 and an answer faithfulness score of 0.7 in tasks such as property querying, data statistics, and contextual querying, demonstrating high usability. For instance, a user can directly request all available information regarding the road '首都高速5号池袋線' (Metropolitan Expressway Route 5 Ikebukuro Line), and the system promptly indicates its location in 'Toshima City', classifies its usage as a 'Primary Emergency Transport Road', and provides additional contextual information. At the current development stage, the system performs effectively in single-feature property extraction, data statistics, contextual information extraction, and multilingual queries. However, there remains room for improvement in handling complex spatial analyses and ambiguous user inputs.

5. Conclusion

This manuscript presents a novel chatbot system using multi-LLM agents and a knowledge graph for large-scale CityGML datasets, aiming to lower the interaction barrier for ordinary users. The main innovations are a hierarchical knowledge graph construction method tailored for CityGML and a knowledge graph RAG-based multi-agent framework for the chatbot. Extensive evaluation with the PLATEAU dataset affirms the system’s usability. This work is one of the latest attempts in knowledge graph-supported city digital twin applications and establishes a new methodological contribution in bridging CityGML, knowledge graphs, and LLMs. The resulting chatbot system significantly alleviates the burden for ordinary users’ querying information from large-scale CityGML datasets, thus bearing broad practical implications. It also has the potential to be embedded into other spatial natural language-driven multi-agent systems. In the future, we plan to strengthen the system’s capability for complex spatial analysis and robustness in handling user input. It would also be valuable to explicitly define the value gained by using a city knowledge graph in the proposed chatbot system and its capability boundaries.

References

Es, S., James, J., Anke, L.E., and Schockaert S. 2024: Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 150-158.

Luo, B., Lau, R.Y.K., Li, C., 2022: A critical review of state‐of‐the‐art chatbot designs and applications. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(1). doi.org/10.1002/widm.1434.

Seto, T., Furuhashi, T., Uchiyama, Y., 2023: Role of 3d city model data as open digital commons: a case study of openness in japan's digital twin" project plateau". The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 48, 201-208. doi.org/10.5194/isprs-archives-XLVIII-4-W7-2023-201-2023.

Tsai, M.H., Chen, J.Y., Kang, S.C., 2019: Ask Diana: A keyword-based chatbot system for water-related disaster management. Water, 11(2), 234. doi.org/10.3390/w11020234.

Wang, S., Hu, T., Xiao, H., 2024: GPT, large language models (LLMs) and generative artificial intelligence (GAI) models in geospatial science: a systematic review. International Journal of Digital Earth, 17(1). doi.org/10.1080/17538947.2024.2353122.

Yu, D., Bao, R., Mai, G., 2025. Spatial-rag: Spatial retrieval augmented generation for real-world spatial reasoning questions. arXiv preprint arXiv:2502.18470. doi.org/10.48550/arXiv.2502.18470.