Extracting and classifying spatial language terms from planning documents

ABSTRACT:<br><br>Generative Artificial Intelligence (GenAI) and Large Language Models (LLMs)present new opportunities to advance text-based analyses, particularly for identify-ing and disambiguating geospatial information expressed as natural language text.This study explores the capabilities of an LLM to identify, classify, and create map-based representations of geospatial expressions found in urban planning documents.As a profession concerned with the spatial arrangement and interrelationships ofhuman activities, land uses, and environmental systems, much of the regulatoryinformation, professional expertise, and local knowledge that urban planning en-codes as text in reports and plans is geographic in nature. Interpreting and cross-referencing the geographic information in these documents is challenging and timeconsuming for both domain professionals (e.g., planners, lawyers) and communitymembers. This paper presents an automated pipeline that was developed to assessan LLM’s ability to extract geospatial expressions, determine their relevant spatialscale (micro, meso, macro), and to identify urban features referenced by a specificdevelopment proposal. Three types of planning documents were examined from amid-sized city (Kitchener, Ontario, Canada), namely a City-authored Official Plan(OP), private consultants’ development proposal reports, and transcripts from pub-lic meetings. A custom Retrieval-Augmented Generation (RAG) model was createdto process text hierarchically, extracting explicit and implicit geospatial expressionswith varying accuracy (e.g., F1-scores ranging from 0.440 to 0.800 across sources). Amulti-agent architecture, leveraging tools like Nominatim and DuckDB, was used toconvert these expressions into SQL queries and map representations. Results revealdistinct patterns in expression types and scales across document types, with OPdocuments favoring macro-scale expressions, plan justification reports emphasizingmicro-scale details, and public meeting transcripts reflecting informal micro-to-mesoreferences. While LLMs demonstrate strong potential, challenges such as factual in-consistencies, attribute errors, and context limitations highlight the need for humanoversight and improved metadata integration. This work advances the understand-ing of LLMs in geospatial analysis and highlights the potential to enhance urbanplanning processes through automated interpretation of geographic expressions intext<br><br>create_vector_store -&gt; explained in figure 4<br><br># Policy Encoder<br><br>This repository provides tools and pipelines for extracting, encoding, and querying geospatial policy data using LLMs and DuckDB with spatial extensions.<br><br>## Features<br><br>- **PDF Extraction**: Extracts geospatial expressions and scales from policy documents.<br>- **Geospatial Data Import**: Loads GeoJSON, GPKG, and other geospatial files into DuckDB as spatial tables.<br>- **Metadata Export**: Exports table schemas and geometry columns to JSON for downstream use.<br>- **GeoJSON Export**: Exports DuckDB tables with geometry columns as GeoJSON files.<br>- **LLM Integration**: Uses LangChain and OpenAI models for geospatial language understanding and SQL generation.<br>- **Agent-based Querying**: Modular agent workflow for geocoding, spatial language detection, and spatial SQL querying.<br><br>## Folder Structure<br><br>```<br>scripts/<br> agents/<br> constants/<br> create_figure_7.py<br> create_figure_8.py<br> create_vector_store.py<br> extract_expressions_and_scale.py<br> geocoder.py<br> prepare_duckdb_db.py<br> prompts.py<br> test.py<br> table_schema.json<br>sqls/<br>split_language/<br>scale/<br>maps/<br>input_files/<br>data/<br>csv/<br>```<br><br><br>| Folder | Description |<br>|----------------|----------------------------------------------------------------------------------------------|<br>| `scripts/` | Main Python scripts for data processing, extraction, database preparation, and LLM workflows. |<br>| `scripts/agents/` | Agent modules for geocoding, spatial language detection, and spatial SQL generation. |<br>| `scripts/constants/` | Configuration files for paths, LLM settings, and input files. |<br>| `sqls/` | Saved SQL queries and outputs generated by the system or agents. |<br>| `split_language/` | Intermediate files and outputs related to spatial language splitting and detection. |<br>| `scale/` | Intermediate files and outputs for scale extraction and analysis. |<br>| `maps/` | Map images, figures, and visualizations used in the project. |<br>| `input_files/` | Raw input documents such as policy texts and justification reports. |<br>| `data/` | Geospatial data files (GeoJSON, GPKG, etc.) and DuckDB database files. |<br>| `csv/` | Intermediate CSV files containing extracted policies, scales, spatial language,<br><br>## Setup<br><br>1. **Install dependencies** <br> Recommended: Python 3.10+ <br> ```sh<br> pip install -r requirements.txt<br> ```<br><br>2. **Set up environment variables** <br> - Add your OpenAI API key and other secrets as needed. You can set it up in the `scripts/constants/llm_config.py` and replace the value for `OPEN_AI_API_KEY`<br><br>3. **Prepare Data** <br> - The paper's geospatial files (GeoJSON, GPKG, etc.) are in the `data/` directory.<br> - The current model only reads from the Official plan file. you can replace it in the `scripts/constants/input_files.py` file and replace your new policy file for `PLAN_JUSTIFICATION` variable.<br> - Make sure to set `PARENT_DIR` correctly in the root folder<br><br>## Usage<br><br>### 1. Import Geospatial Data into DuckDB<br><br>Run:<br>```sh<br>python scripts/prepare_duckdb_db.py<br>```<br>- This will import all geospatial files from `data/` into a DuckDB database and export metadata to `db_metadata.json`.<br>- Later you can add correct table and column descriptions to the `db_metadata.json` and save it as `table_schema.json`. this file will be fed to the llm. The current `table_schema.json` is already filled with necessary data. <br><br>### 2. Extract Geospatial Expressions from Policy Documents<br><br>Run:<br>```sh<br>python scripts/extract_expressions_and_scale.py<br>```<br>- This will process the specified PDF and extract geospatial expressions and their scales.<br><br>### 3. Querying with Agents<br><br>- See [`scripts/test.py`](scripts/test.py) and [`scripts/create_figure_7.py`](scripts/create_figure_7.py) for examples of agent-based workflows that:<br> - Geocode locations<br> - Detect geospatial language<br> - Generate and execute spatial SQL queries on DuckDB<br><br># 4. Figures and tables<br><br>- [`scripts/create_figure_7.py`](scripts/create_figure_7.py) is used for an example of the figure 7 data. The logs of this file is converted to the figure 7<br><br>- [`scripts/create_figure_8_table3_4.py`](scripts/create_figure_8_table3_4.py) is used for an example of the figure 8 data. I generates one of the graph images and also exports an example data for the table 3 and 4<br><br>- [`answers-justification_spatial_languages.xlsx`](answers-justification_spatial_languages.xlsx) is used for evaluation of the table 2 and 5 for the plan justification data<br><br>- [`answers-op_spatial_languages.xlsx`](answers-op_spatial_languages.xlsx) is used for evaluation of the table 2 and 5 for the official plan data<br><br>- [`answers-transcript_spatial_languages.xlsx`](answers-transcript_spatial_languages.xlsx) is used for evaluation of the table 2 and 5 for the transcript data<br><br>- [`Figure 9.xlsx`](Figure 9.xlsx) This includes a simple copy of the data from previous `xls` files into new file to create figure 9<br><br>- for the `Table 6` sample sql queries from `scripts/create_figure_7.py` is selected and visualized in the Qgis. <br><br><br>## Customization<br><br>- **Prompts**: Modify or extend prompts in [`scripts/prompts.py`](scripts/prompts.py).<br>- **LLM Configs**: Set your LLM endpoints and parameters in [`scripts/constants/llm_configs.py`](scripts/constants/llm_configs.py).<br>- **Input Files**: Configure paths in [`scripts/constants/input_files.py`](scripts/constants/input_files.py).<br><br><br>## License<br><br>MIT License<br><br>---<br><br>**Contact:** <br>For questions or contributions, please contact the author<br>