Projects

Motivated after applying PPO on Atari and MuJoCo, I wanted to apply policy optimization to language-model post-training. I built an RLHF pipeline around Qwen2.5-0.5B-Instruct and HelpSteer3: supervised fine-tuning, pairwise reward modeling, and guarded token-level PPO with LoRA, a frozen reference policy, value learning, KL control, and qualitative response auditing.

The final pipeline used 36K+ filtered preference pairs, a reward model with 65.6% held-out pairwise accuracy, and 12K domain-balanced PPO rollouts with responses up to 768 tokens. The full explorer exposes 2,017 held-out prompts. The guarded PPO policy remained stable and less prone to stopping and repetition failures than an earlier high-reward checkpoint, but it did not beat Base under the learned reward proxy; the project therefore emphasizes inspectable training, reward-hacking safeguards, and honest qualitative analysis.

I studied the policy-optimization thread in reinforcement learning: Conservative Policy Iteration, Natural Policy Gradient, Trust Region Policy Optimization, and Proximal Policy Optimization. The project follows how constrained or clipped policy updates try to keep improvement stable while still letting the policy move.

I implemented TRPO, PPO, and NPG from scratch in PyTorch, benchmarked them on MuJoCo locomotion and Atari environments, and ran a Hopper step-size ablation to study where NPG becomes brittle. In the empirical runs, PPO reached stronger Hopper and Walker results with roughly one-tenth of the sampled environment interactions used by the TRPO setup. See the code, project report, and my broader RL reflection here.

InfraNet is my undergraduate research project on real-time elephant detection from infrared thermal imagery for human-wildlife conflict mitigation. The work uses a 50,694-frame thermal dataset with humans, elephants, and other animals across varied field conditions, then studies how preprocessing choices affect detector robustness.

The final approach combined four preprocessed thermal variants with a lightweight pretrained YOLO detector. Inversion and bilateral filtering were especially useful, and the ensemble improved elephant recall from 0.35 to 0.62 while preserving the fast inference profile needed for edge deployment. The paper was published at IEEE AVSS 2025.

StanLyric is an information-retrieval system for identifying songs from remembered lyric fragments. The public browser app searches a 36,545-song corpus with BM25-Okapi and shows ranked candidates, match terms, confidence, and score-gap diagnostics without publishing full lyric text.

I also built an embedding explorer over Cohere Embed v4 song representations: 1,024-D embeddings are projected with UMAP, clustered with Leiden community detection, and rendered as an interactive atlas for moving from keyword search into broader lyric-theme structure.

WordPlay uses bidirectional character n-gram language models for two small browser-side demos. WordGuesser plays a Hangman-style challenge by scoring letters from visible forward and reverse contexts, while WordGenerator samples plausible new English-like words from the same learned character distributions.

The generator now supports fixed letter positions, so a user can choose a word length, lock in known characters, and repeatedly ask the model to complete the remaining blanks. Both demos run from a static JSON model export with backoff, interpolation, and smoothing.

I studied personalized Warfarin dose selection as an offline contextual-bandit problem over 5,528 International Warfarin Pharmacogenetics Consortium records. The learner chooses among low, medium, and high weekly dose arms from patient context, then receives feedback from the known therapeutic dose class.

I evaluated LinUCB, regularized ridge and Lasso variants, and Linear Thompson Sampling against fixed-dose and pharmacogenetic references. The strongest learned configuration reached about 68.36% mean online accuracy, close to the pharmacogenetic reference near 69%, while sparse Lasso policies made the decision drivers easier to inspect. This is an experimental ML study, not a clinical prescribing tool.