Fine-Tuned LLM Microservice

This project takes Qwen2.5-3B-Instruct, a small open instruct model, and fine-tunes it with QLoRA (4-bit quantized LoRA) to perform one narrow, well-defined task: reading a raw job posting and extracting a structured JSON record of required skills, tech stack, seniority, and compensation range. The fine-tuned adapter is merged, quantized to GGUF, and served behind a FastAPI endpoint, containerized with Docker, deployed to an AWS EC2 (Graviton/ARM) instance, and wired into a GitHub Actions CI/CD pipeline that tests, builds, and publishes the image on every push. The project deliberately avoids agentic complexity or a bigger model — the point is the production tier: proving a single small model can be fine-tuned, evaluated rigorously against its own zero-shot baseline, and operated as a real service.

Job postings are unstructured text, but hiring analysis, market research, and sourcing tools all need structured data: required skills, tech stack, seniority, and compensation range. Doing this by hand doesn't scale, and general-purpose frontier models are too expensive and too heavy to run this kind of extraction at volume in production. There's also a portfolio gap this targets directly: job-posting analysis shows cloud in 61% of postings, AWS in 41%, and MLOps/model deployment in roughly 29% each, yet the 'deploy and operate a model' tier is almost entirely absent from typical early-career portfolios. A companion project (an agentic RAG system) proves the ability to architect LLM systems; this one proves the complementary skill: fine-tuning one model for a specific task and running it as a monitored production service.

Structured as teacher → baseline → student so the result isolates exactly one variable, the effect of fine-tuning: Claude Sonnet was used once, offline, as a teacher to bootstrap labels for a corpus of job postings, gated through a pydantic schema so malformed labels were rejected rather than silently kept; Qwen2.5-3B-Instruct zero-shot became the baseline, measured on a held-out test set; the fine-tuned Qwen2.5-3B + LoRA adapter is the student that ships, always scored as Student − Baseline via disable_adapter() so decoding conditions stay identical between the two. Data was bootstrapped, pydantic-validated, and split 70/15/15, then fine-tuned with QLoRA (4-bit nf4 base + LoRA r=16). The first run, on 70 examples with uncorrected labels, actually regressed performance below baseline at a 60% valid-JSON rate. Rather than reaching for decoding-time fixes, the raw generations were inspected directly, surfacing degenerate repetition loops under greedy decoding on the longest postings. Decoding fixes broke the loops but degraded content, and a stricter schema-validation eval exposed that the real root cause was thin, noisy training data, not decoding parameters. The dataset was doubled (100 → 188 examples, deduplicated), the labeling prompt tightened, and labels re-bootstrapped and re-verified — the retrain lifted valid-JSON output from 60% to 100% and turned the model into an across-the-board win over baseline. The adapter was then merged into the base model, quantized to a Q4_K_M GGUF (~1.8 GB), and served via llama-cpp-python on CPU, reusing the same generate → parse → validate pipeline from evaluation. Finally: containerized with Docker, published to Docker Hub, deployed to an AWS EC2 Graviton instance, and wired to a GitHub Actions pipeline that runs tests and builds/pushes the image on every push, with structured request logging and per-request latency middleware as the monitoring layer.

Field-level accuracy, base model vs. fine-tuned, scored identically (same decoding, same run, via disable_adapter()) on a held-out test set of 29 postings: seniority 0.72 → 0.76, compensation (±10%) 0.86 → 0.97, required skills (set-F1) 0.17 → 0.27, tech stack (set-F1) 0.08 → 0.68, valid-JSON rate 1.00 → 1.00. The standout result is tech stack (0.08 → 0.68): fine-tuning taught the model to cleanly separate named technologies (Python, PyTorch, AWS, Docker) from general competencies, a distinction the base model consistently failed to make. Getting there required a real diagnostic detour, arguably the strongest part of the project: the first fine-tuning attempt regressed performance (60% valid-JSON, worse than baseline on 3 of 4 fields) due to degenerate repetition loops in generation. Tracing that back to thin, noisy training data, rather than patching it over with decoding-time hacks, and fixing it at the source is what turned the project into a clean win across every measured field. Operationally, the model is fine-tuned, quantized, containerized, and deployed live on AWS behind a public, monitored /extract endpoint, with CI/CD testing, building, and publishing the image automatically on every push. A concrete infrastructure finding from deployment: on a free-tier 2 GB / 2-core EC2 instance, the Q4_K_M model is RAM-bound rather than compute-bound, running correctly but slowly because it doesn't stay resident in memory — a documented, well-understood cost/performance tradeoff with a clear scale-up path rather than a defect.

outcome: Fine-tuning lifted tech-stack extraction from 0.08 to 0.68 F1 over the zero-shot baseline, after diagnosing and fixing a training-data regression that had initially made the model worse.

• QLoRA
• FastAPI
• Docker
• AWS EC2
• GitHub Actions