Case study

How the system was built, measured and judged, in one page. Long tables are folded; every number keeps its source.

The problem

Fill any JSON schema from a Spanish text, and return null for whatever the text does not say.

Schema-guided extraction in Spanish

An instance of the task is a triple: a Spanish source text, a natural-language instruction, and a JSON Schema that defines what to extract. The schema is only seen at inference time, so a system cannot memorise entity types: it has to read field names, types and descriptions and act on them.

A valid output must satisfy two constraints at once:

1. Structural validity: the output is a JSON object that conforms to the schema.
2. Source faithfulness: every non-null value is supported by the text, and every field the text does not support is returned as null.

Below is an illustrative instance written for this page, in the format of the synthetic corpus (schema abridged); it is not a corpus record. Note that the average flow is not in the text, so the correct answer for that field is null.

{
  "input_text": "El río Arga es uno de los principales ríos de Navarra. Nace en el área de Quinto Real, en los Pirineos navarros, y atraviesa Pamplona y Puente la Reina antes de desembocar en el río Aragón. Tiene una longitud de 145 km.",
  "instruction": "Extrae el perfil estructurado del río descrito: su nombre, el río en el que desemboca, su longitud en kilómetros, la zona donde nace, las localidades que atraviesa y su caudal medio.",
  "target_schema": "RiverProfile: name, tributary_of, length_km (integer), source_range, passes_through (array), avg_flow_m3s (number or null)",
  "output": {
    "name": "Arga",
    "tributary_of": "Aragón",
    "length_km": 145,
    "source_range": "Quinto Real",
    "passes_through": ["Pamplona", "Puente la Reina"],
    "avg_flow_m3s": null
  }
}

Null-traps

Fields that the text does not answer are called null-traps. A model that invents a plausible value for them is hallucinating, and the evaluation penalises it. This makes the task a test of restraint as much as of extraction.

The GenSIE 2026 shared task

GenSIE (General-purpose Schema-guided Information Extraction) was a shared task at IberLEF 2026. Two rules shaped the work:

• No-Training policy. Fine-tuning the model weights (LoRA, QLoRA, full fine-tuning) was prohibited. Only inference-time techniques were allowed: prompting, retrieval, constrained decoding and few-shot examples.
• Several small open models. Systems were evaluated on several open-weight models below 14B parameters from different families, some of them held out until the results, so tuning a pipeline to a single model was discouraged.

Roughly half of the private test schemas were entirely new.

The metric

The primary metric is Flattened Schema Scoring: gold and system outputs are flattened into key–value sets, each shared key receives a similarity score between 0 and 1 (exact match for rigid types such as enums and numbers, an embedding and lexical blend for free text), and the final score is a Micro-F1:

F1 = 2 · TPS / (|G| + |S|), where TPS is the sum of the similarities of the shared keys, and |G| and |S| are the sizes of the gold and system key sets.

The GenSIE leaderboard aggregates results as the fraction of the baseline-to-perfect gap each system closes, averaged over the evaluation models. This site reports raw Micro-F1 per model.

Approach

One synthetic corpus feeds two branches: a zero-shot pipeline with frozen weights and a fine-tuning branch.

The work does not set fine-tuning against no fine-tuning. It solves one problem and explores several techniques, with and without training, on the same three small open models. A single synthetic corpus feeds both branches.

Block	What it does
Synthetic data generation	Produces a schema-guided Spanish corpus with a five-stage pipeline adapted from GuideX
Zero-shot branch	Keeps the weights frozen and combines prompting, retrieved few-shot examples, constrained decoding and post-processing
Fine-tuning branch	Updates the weights with LoRA and with full fine-tuning on the same corpus

Synthetic data generation

The corpus follows the idea of GuideX (guided synthetic data generation for zero-shot information extraction), adapted to Spanish and to the GenSIE record format.

Stage	What happens
0	Prepare a heterogeneous Spanish corpus (mostly Spanish Wikipedia passages), segment it into self-contained passages and tag the domain
1	A teacher LLM writes a structured summary of each passage and lists null-trap candidates, questions the passage does not answer
2	Synthesise a unique JSON Schema and instruction for each passage
3	Extract a JSON instance that conforms to the schema, retrying with validator feedback if it does not
4–5	Non-LLM quality control (schema violations, degenerate or under-specified records) and packaging into the GenSIE record format

The main departure from GuideX is that every passage gets its own schema, so a model prompted or trained with the corpus has to read the schema at inference time instead of memorising a fixed ontology.

Zero-shot branch

A generate-then-verify cascade around a single LLM call. No field name or example from the development set is hard-coded.

1. Prompt construction. A system prompt fixes the role of a structured extractor. The user message holds the schema, the text and a few few-shot demonstrations retrieved from the synthetic library by schema similarity, using multilingual sentence embeddings. An optional field guide restates each field's description, enumeration and nullability in plain language.
2. Constrained decoding. The JSON Schema is compiled into a grammar that masks the model's token logits, so structural validity holds by construction and the model's effort goes to semantic correctness.
3. Schema-aware post-processing. A deterministic pass with five conservative layers: source grounding, lexical normalisation, schema-driven canonicalisation, verbatim snapping and a hallucination filter that replaces unsupported values of nullable fields with null.

Fine-tuning branch

Supervised fine-tuning on chat-formatted examples from the same corpus, with the usual cross-entropy loss on the gold JSON and no reinforcement learning. Two regimes run side by side under the same data and objective:

• LoRA: freezes the base weights and trains low-rank adapters that can be merged back into a standard checkpoint.
• Full fine-tuning: updates every parameter, sharded across several GPUs for the larger models.

Three prompt modes are tried at training time, including in-context tuning, where a few demonstrations are placed inside the training prompt itself.

Experiments

Four experiments, organised in phases, each answering one question on the same three models.

The study is organised in phases, each one answering a single question and feeding its best configuration to the next.

Data

Set	Size	Use
Starter set	40 instances	Early screening of inference-time technique families
Development set	149 tasks	All later experiments and the comparisons reported on this site
Synthetic corpus	3,948 pairs	Few-shot retrieval pool and supervised fine-tuning data
Eval-loss holdout	197 examples	Internal validation split used as the proxy metric during the LoRA search (the cleaned release of the corpus has a 200-example holdout)

The synthetic corpus has 3,748 training pairs and a 200-example holdout. The official GenSIE test set is evaluated in the published paper; the results on this site are on the development set.

Models

Model	Checkpoint	Notes
Qwen3-14B	Qwen/Qwen3-14B	Largest of the three
Salamandra-7B	BSC-LT/salamandra-7b-instruct	Spanish-focused, trained at the Barcelona Supercomputing Center
Llama-3.2-3B	meta-llama/Llama-3.2-3B-Instruct	Smallest of the three

The same three models are used in every experiment, so cross-model comparisons are interpretable. The published paper also reports zero-shot results with Gemma-4-E4B.

The four experiments

1. Inference-time technique families. Baseline, chain-of-thought, schema-aware reasoning and few-shot retrieval are screened on the starter set; the best family is then refined on the development set with constrained decoding, a field guide and post-processing layers.
2. LoRA hyperparameter search. Three phases on the eval-loss holdout: rank and dropout, then scheduler, then prompt mode. Each phase carries its best setting forward.
3. Effect of fine-tuning on F1. For each model, four cells on the development set: untuned baseline, untuned with the best zero-shot pipeline, LoRA, and full fine-tuning.
4. Effect of fine-tuning on errors. Every error is classified into one of five categories, and the null-trap rate is measured separately as a probe for fabrication.

Evaluation protocol

• Metric: Flattened Schema Scoring Micro-F1, as defined by the task (see the problem).
• All runs are single-seed. Training runs use early stopping with a patience of 2 evaluations.
• Compute cost of an inference pipeline is estimated from token counts and parameter count.

Hardware and software

Training runs on the MareNostrum 5 supercomputer at the Barcelona Supercomputing Center, on NVIDIA H100 GPUs, scheduled with SLURM. Each LoRA run takes between 3 and 6 GPU hours. The software stack is listed under tech stack.

Results

The best zero-shot pipeline reached the highest F1; training mainly reduced hallucination.

Numbers and observations only; interpretation is on What I learned. All results are on the GenSIE development set (149 tasks), single-seed.

Best zero-shot pipeline: progression on Qwen3-14B

Data table

Step	F1	Δ
Baseline (zero-shot)	0.6577
E021 · 1-shot	0.6994	+0.0417
E025 · 3-shot by schema similarity	0.7155	+0.0161
E211 · + grounding post-processing	0.7426	+0.0271
E222 · + hallucination filter	0.7840	+0.0414
E225 · enriched selector + synthetic library	0.7910	+0.0070
E226 · + field guide + source cleaning	0.7999	+0.0089

Pipeline	F1
Baseline (zero-shot)	0.6577
E021 (1-shot)	0.6994
E025 (3-shot, schema-similarity)	0.7155
E211 (E025 + grounding post-processing)	0.7426
E222 (E025 + full post-processing, including the hallucination filter)	0.7840
E225 (enriched selector, synthetic library, behavioural addenda)	0.7910
E226 (E225 + field guide + source cleaning)	0.7999

Across the whole post-processing stack, the hallucination error count fell from 84 (E025, without post-processing) to 21 (E222). A variant of E226 with a 40% shorter prompt (E227) scored 0.7895.

Best result per regime and model

Model	Untuned baseline	Best zero-shot	Best LoRA	Best full fine-tuning
Qwen3-14B	0.6577	0.7999	0.6977	0.7054
Llama-3.2-3B	0.6703	0.7459	0.7449	0.7481
Salamandra-7B	0.4843	0.6276	0.6406	0.6193

Best value per model in bold. The tuned Qwen and Llama models were evaluated only with the baseline and 1-shot prompts: the richest pipelines (E225, E226) were not run on them, and the Qwen LoRA run with E226 was abandoned because of prompt length and latency. The table therefore does not show how much training adds on top of the best pipeline for those two models. The single highest F1 across all configurations is Qwen3-14B zero-shot (0.7999). On Llama the best zero-shot pipeline (0.7459) and full fine-tuning (0.7481) are essentially tied. On Salamandra the best result is LoRA combined with an enriched few-shot prompt.

Hallucination on nullable fields

Rates over the 505 nullable-field opportunities of the development set, with the plain baseline prompt (no few-shot examples, no post-processing).

Model	Untuned null-trap rate	Best tuned variant
Qwen3-14B	14.65%	12.08% (full fine-tuning; LoRA 12.48%)
Salamandra-7B	24.55%	7.33% (full fine-tuning; LoRA 14.26%)
Llama-3.2-3B	25.15%	6.34% (LoRA; full fine-tuning 12.67%)

The rate fell for every model under training. All rates were measured with the plain baseline prompt, not with the best pipelines.

LoRA search

The search converged on the same configuration for the three models: rank 64, alpha 128, dropout 0.10, learning rate 5e-5. Going from rank 64 to rank 128 lowered eval-loss by at most 0.003 while the train–eval gap kept growing. In-context tuning gave the lowest eval-loss on every model:

Model	Default prompt eval-loss	In-context tuning eval-loss
Qwen3-14B	0.5698	0.3569
Salamandra-7B	0.5078	0.3829
Llama-3.2-3B	0.671	0.4248

Under the hood

How training and inference are traced end to end: MLflow for training runs, Langfuse for inference.

During development every training run was logged to MLflow, and every model call and evaluated task to Langfuse; the live demo is not traced. Below, one real record of each.

MLflow runGenSIE_Llama3.2-3B_Phase1

llama_v2_r64_a128_d0_10

model: meta-llama/Llama-3.2-3B-Instruct

lora_r / alpha / dropout

64 / 128 / 0.1

learning_rate

5e-05

scheduler

cosine

epochs · early stopping

6 · True

batch × grad. accum.

1 × 8

max_length

4096

duration

46 min

train losseval loss

Langfuse traceOpenAI-generation

model call · 1565 ms

OpenAI-generation1565 ms

model

Qwen3-14B

tokens in / out

334 / 71

evaluated task · separate trace · gensie.eval.task

completion tokens

75

gold keys

10

prompt tokens

1243

system keys

8

total tokens

1318

TPS (true-positive score)

8

Exported read-only and sanitised: no prompts, outputs, paths or keys.

What I learned

Training appears to help less as the inference pipeline improves, and it matters more for restraint than for raw F1.

Interpretation of the results. It rests on three models, one task and single-seed runs, so each claim is hedged accordingly.

Fine-tuning appears to help less as the pipeline gets better

On Qwen3-14B the best zero-shot pipeline (0.7999) beats both LoRA and full fine-tuning. On Llama the two are essentially tied. Only on Salamandra, the weakest model without training, does LoRA add on top of the best prompting pipeline. The comparison has a limit: the tuned Qwen and Llama models were not evaluated with the richest pipelines, so it is untested whether training adds anything on top of them. A plausible reading is that fine-tuning teaches the model schema-handling behaviour that the post-processing layers and the retrieved examples already provide at inference time: when those layers are absent, training compensates; when they are present, it has less to add.

Training matters on a different axis: hallucination

Headline F1 hides a consistent effect. With the plain baseline prompt, the null-trap rate dropped on every model under training, strongly on Salamandra and Llama and only slightly on Qwen. This suggests that, compared with an untuned model under the same prompt, training makes the model more willing to abstain. It does not show that training beats the best prompting pipeline on this axis: in the source analysis, a well-designed pipeline stays competitive with LoRA on grounding.

The optimum was stable across model sizes

The same LoRA configuration (rank 64, alpha 128, dropout 0.10) came out best for the 3B, 7B and 14B models, and in-context tuning helped all three. Together with the flat gain from rank 64 to rank 128, this appears to indicate that the difficulty of the task lies in reasoning about the schema, and that the levers acting on that reasoning behave similarly across this size range.

What did not work

Run E229 added a second, targeted LLM pass for the 34 development tasks with nested objects, on top of E226 with Llama-3.2-3B. The target subset improved by 0.0037, but the global F1 fell from 0.7459 to 0.7432 and the estimated cost grew to about 1.23 times that of E226. In addition, 20 of the 115 tasks without nested objects changed between two runs of an unchanged first pass, which points to non-determinism in the serving stack rather than to the method. The branch was dropped.

Limits

• Three models, one task, single-seed runs: the interpretation is not a cross-family proof.
• The richest pipeline overflows Salamandra's 8k context window on 14 of the 149 development tasks, so its best reported pipeline is a lighter one.
• The synthetic corpus is dominated by general-domain, Wikipedia-derived text, so transfer to specialised domains is not tested here.

Reproducibility

Where the code lives and how every number on this site is traced to a file.

The work is split across two repositories, plus the repository of this site. The repositories are currently private, so they are named here without links; they will be linked once they are opened. The published paper is linked on the Recognition page.

Repository	Contents
gensie-solution	The competition solution: baselines, prompting variants, few-shot agents, grammar-constrained decoding, post-processing, experiment registry and result files
gensie-dataset	The synthetic-data pipeline, the LoRA and full fine-tuning code, SLURM job scripts for MareNostrum 5 and the evaluation scripts
gensie-portfolio	The source of this site

The solution repository started from the official GenSIE starter kit, and the task description, metric and data formats come from there.

How the numbers in this site are traced

Every figure in the markdown source of this site is preceded by an HTML comment <!-- source: ... --> naming the file it was read from. Figures cite files from the author's working copies of these repositories; large generated artefacts such as the synthetic dataset are not all committed, and some analysis notes live outside main. A test suite in the site repository fails the build if a decimal metric or percentage has no source comment, if a link points anywhere other than the main branch, or if page structure or numbers differ between the two languages. It also checks that cited source files exist when it runs on a machine that has the project repositories.

Tech stack

Only tools used in the two project repositories.

Training and fine-tuning

• PyTorch, Hugging Face Transformers, PEFT and TRL for LoRA and supervised fine-tuning (SFTTrainer).
• Fully Sharded Data Parallel for full fine-tuning of the larger models.
• SLURM jobs on the MareNostrum 5 supercomputer (NVIDIA H100).

Inference and structured output

• vLLM for serving base and fine-tuned models.
• Outlines and xgrammar for grammar-constrained JSON decoding.
• Pydantic and jsonschema for schema handling and output validation.

Retrieval and evaluation

• Sentence-Transformers (multilingual MiniLM) for schema-similarity few-shot retrieval, plus a TF-IDF variant.
• fastembed, rapidfuzz and scikit-learn for the Flattened Schema Scoring metric and the analysis scripts.
• Agent variants for chain-of-thought, schema decomposition, self-correction and multi-agent post-processing.

Experiment tracking

• MLflow and Langfuse for run tracking and tracing, with a self-hosted MLflow server run through Docker Compose.

Data generation

• A five-stage synthetic-data pipeline adapted from GuideX, with jsonschema validation and LLM calls through an OpenAI-compatible API.

Retrieval in this project is retrieval-augmented few-shot selection over a synthetic example library; it does not use a vector database.