Bentonville / Rogers, AR
Ashvi Soni
Senior Data Scientist at Walmart Global Tech, specializing in Generative AI, LLM evaluation, agentic systems, and applied machine learning.
Profile
Ashvi Soni is a Senior Data Scientist at Walmart Global Tech, where she leads data science for Ask My Doc, a production GenAI document-intelligence system. Her work centers on the evaluation layer of generative AI: LLM-as-a-judge pipelines, benchmarking frameworks, and simulation-based testing that determine whether a model change is a genuine improvement before it reaches production.
Prior to Walmart, she spent three years at eClinical Solutions building GenAI and RAG systems for clinical trial data, and previously worked at the George Washington University Biostatistics Center on federally funded public health research, including a CDC-funded COVID-19 study spanning ten-plus institutions. That combination of regulated clinical data and production-scale GenAI systems informs her focus on rigorous, reproducible evaluation.
Professional Experience
- Led data science for Ask My Doc, a production GenAI document-intelligence system, improving multi-file summarization accuracy from 70% to 95% and reaching 98% OCR extraction accuracy.
- Designed LLM benchmarking studies across GPT and Gemini model families using 130+ question evaluation suites spanning text, code, audio, video, and multimodal inputs to inform model selection and upgrades.
- Built LLM-as-a-judge pipelines to automatically assess response quality, grounding accuracy, and reasoning effectiveness across production AI systems at scale.
More detail
- Built an automated evaluation framework for a conversational AI agent, including a simulation-based user bot with adaptive behavioral criteria, cutting manual evaluation effort by roughly 80%.
- Ran root cause analysis on LLM behavioral failures, including a content-filtering incident affecting 8% of queries, and developed a rephraser-based proof of concept now moving into implementation.
- Developed proof-of-concept agentic routing systems using semantic search, RAG-based retrieval, and hybrid reflection-based routing architectures, with phase two underway.
- Partnered with engineering teams in an embedded data science organization to ship GenAI capabilities and drive adoption of AI-powered solutions.
- Built a GenAI model using fine-tuned LLMs for named entity recognition and medical term standardization, cutting manual drafting time by 70% on compliant reports.
- Developed an LLM and RAG-based medical coding system with a Chroma vector database to standardize SDTM dataset headers, reducing manual coding errors by 70% and speeding processing by 50%.
- Built ML models to score clinical trial site risk from historical compliance, data integrity, and enrollment performance, cutting monitoring costs by 30% and improving high-risk site prioritization by 25%.
More detail
- Executed ML models to identify mismatched medication-indication pairs against an open-source dataset, cutting manual review effort by 30%.
- Wrote Python (pandas) and SQL for anomaly detection, dataset reconciliation, trend analysis, and custom reporting, reducing manual data validation effort by 30%.
- Built Qlik dashboards and interactive visualizations to make complex clinical data usable for stakeholder decision-making.
- Contributed to the CDC- and North Carolina-funded COVID-19 Community Research Partnership across 10+ institutions, running exploratory, prescriptive, and predictive analysis in Python.
- Led data wrangling across Python, R, and SQL, improving data quality and integrity by 40% and cutting analysis time by 25%.
- Identified a critical HIPAA violation through data analysis, driving improved security protocols and staff training.
- Analyzed 10+ years of patient data for the NIDDK-funded EDIC study, contributing to a 15% improvement in study accuracy.
- Built and tuned six predictive models (logistic regression, random forest, SVM, k-NN, XGBoost, neural network) to forecast hypoglycemia in Type 1 diabetes participants, improving sensitivity and specificity by 50%.
Selected Work
Designed proof-of-concept routing architectures so a conversational AI agent picks the right retrieval or reasoning strategy per query, instead of relying on one static pipeline. Combines semantic search, RAG-based retrieval, and a hybrid reflection-based routing layer that lets the agent re-evaluate its own routing decision before responding.
Built an automated evaluation framework for a conversational AI agent, centered on a simulation-based user bot with adaptive behavioral criteria that stands in for real user variation. It scores response quality, grounding, and reasoning at scale, cutting manual evaluation effort by roughly 80% and giving the team a fast, repeatable way to tell whether a model or prompt change actually helped.
Peer-reviewed work
Williamson, J. C., Soni, A., et al. — "COVID-19 Outcomes in the Immunocompromised Population of the COVID-19 Community Research Partnership"
View on PubMed Central →Skills & Competencies
Programming Languages
GenAI & LLM Tools
Libraries
Tools & Platforms
Core Competencies
Education
Get in touch
Open to conversations on production LLM systems, evaluation methodology, and applied research collaboration.