GSoC 2026 Projects

Pipeline for Data Quality Verification and Anomaly Detection in Governmental Data with Automated Reports

Currently, GovHub performs daily ingestions of data from various public sources, organizing it into data layers (bronze, silver, and gold). Although there are some point validations, the systematic identification of anomalies, inconsistencies, and data quality issues still depends on manual inspections or ad hoc checks.

The objective of this project is to build an automated data quality verification module for the databases of the core government systems (ComprasNet, SIAFI, TransfereGov, SIAPE, and SIORG), capable of analyzing daily ingestions and generating consolidated periodic reports (weekly or monthly) highlighting statistical anomalies, semantic inconsistencies, and potential structural issues. The focus is not on automatically correcting the data, but on providing visibility, traceability, and issue prioritization for pipeline maintainers.

As the desired final state, the student should deliver a system integrated into the existing GovHub pipeline that applies automated (statistical and semantic) checks on processed tables and produces structured, versioned, and auditable reports, facilitating data governance and the reliability of published data.

RAG for Natural Language Queries over Gold Data

Transforming complex governmental data into clear answers through natural language.

Brazil is a global reference in budgetary transparency, with milestones such as the Fiscal Responsibility Law and the Freedom of Information Law (LAI). However, a paradox remains: despite the vast availability of open data, these datasets often become "unexplored digital archives." The technical barrier is high; to monitor public policies in GovHub, a citizen or researcher must master SQL and understand complex data models (bronze, silver, and gold layers), which restricts active transparency to technical specialists.

This project proposes to democratize access by transforming natural language into structured queries. The goal is not to create a generic chatbot that hallucinates answers, but a precise analytical agent.

This project aims to develop a system based on Retrieval-Augmented Generation (RAG) that enables natural language queries to be executed directly over qualified GovHub data. The system should leverage metadata, table descriptions, and vectorized schemas to feed the LLM, automatically generate SQL queries, manage them, and return structured responses.

The desired final state is a tool that abstracts the complexity of SQL queries while maintaining traceability of user questions, generated queries, and the data used—operating not as a generic chatbot, but as an analytical component integrated into the data pipeline and ensuring data governance.

Data Lakehouse (MinIO + Iceberg) for Versioning (Time Travel) and AI Consumption in GovHub

Currently, GovHub operates on a solid architecture for analytical processing and consumption, with processed data organized into Medallion architecture layers and stored in Postgres. However, as a database oriented toward the current state (a typical OLTP characteristic), Postgres presents some limitations:

• Native historical versioning (time travel) in a standardized and efficient manner;
• Traceability and reproducibility of datasets for AI consumption (ML, embeddings, RAG);
• Cost-efficient scalability for large volumes and frequent reprocessing;
• Retention governance and auditing at the snapshot and file level.

This project proposes evolving the GovHub architecture toward a versioned Data Lakehouse, built on widely adopted open-source technologies: MinIO (object storage) and Apache Iceberg (table format), with Spark as the processing engine and Trino as the federated query layer. The goal is to enable time travel, schema/partition evolution, and metadata governance, while keeping the team experience close to the current Airflow + dbt workflow, avoiding significant rework.

The scope also includes analyzing and documenting essential operational considerations for the lakehouse, such as retention and maintenance of Iceberg metadata, strategies to handle small files, organization of layers, catalog patterns, and infrastructure patterns for execution on Kubernetes.

Pipeline for Extraction, Structuring, and Ingestion of Unstructured Data into a Governmental Datalakehouse

A significant portion of relevant government information remains in unstructured documents, such as PDFs, presentations, and technical reports. Although these documents are publicly available, their content remains difficult to access for systematic analysis, integration with analytical pipelines, and reuse in data-driven applications.

This project proposes developing a modular, reprocessable pipeline to ingest unstructured documents into GovHub, combining visual layout analysis, region-based OCR, and semantic enrichment. The pipeline should automatically identify the structural elements of documents (titles, paragraphs, tables, and images), extract textual content in a contextualized manner, and transform it into semi-structured and structured data.

Unlike approaches focused on immediate indexing or search, the main focus of this project is the structuring and persistence of these data in a datalakehouse, organized into well-defined layers (raw, parsed, and enriched), enabling governance, versioning, auditing, and reprocessing. The desired final state is a structured data foundation derived from documents, ready to support analyses, reports, and future analytical or AI-based applications within the GovHub ecosystem.