0 ft Elevation
Boulder, Colorado · Data Science Leader

John Tribbia

Senior Staff Data Scientist · Google Workspace AI

Specializing in LLM evaluation and model quality measurement - linking offline model performance to online user engagement. PhD researcher turned Data Scientist.

Tackling mountains of data like I tackle mountains on the run - with unrelenting determination and a sense of adventure.

0 Years of Data Experience
0 Years as Team Lead
Scroll to ascend
01 —

About

Data science leader with expertise in LLM evaluation, model quality measurement, and user-based systems analysis. Currently leading response quality measurement for Google Workspace AI, linking offline model evaluations to online user engagement and retention.

Specialized in translating observational insights into actionable improvements for AI systems - backed by a PhD in Sociology and 15+ peer-reviewed publications (800+ citations) in the human dimensions of climate change.

Open to leadership opportunities
Current
Senior Staff Data Scientist
Google Workspace AI
Location
Boulder, Colorado
Education
PhD Sociology · BS Mathematics
Contact
LinkedIn ↗
john@bouldergearlab.com
02 —

Technical Expertise

ML & AI Evaluation
LLM Development & Evaluation
Model Quality Metrics
Response Quality Measurement
Sentiment Analysis
Text Summarization
Automated Classification
Offline → Online Metrics
Statistics & Analysis
Statistical Modeling
Causal Inference
Forecasting
A/B Testing
Multi-Touch Attribution
Predictive Analytics
Time Series Decomposition
Tools & Engineering
R
SQL
Python (pandas)
ETL / Data Modeling
Data Visualization
Leadership
Technical Strategy & Roadmap
Team Building (IC → Senior Staff)
Cross-functional Collaboration
Executive Stakeholder Management
Mentorship & Development
Quality Standards
03 —

Experience

Sep 2025 - Present
Google
Workspace AI
Boulder, CO
Senior Staff Data Scientist · Tech Lead IC
  • Leading model quality strategy for Workspace AI: defining evaluation standards for single-turn, multi-turn, and agentic surfaces and connecting offline model performance to real-world user behavior at scale
  • Quality-Retention Framework: Developed a within-version causal estimation method that isolates model quality's effect on retention by exploiting variation in how users' prompt mix interacts with category-level model strengths without requiring an A/B experiment
  • Investment Prioritization Model: Extended framework into counterfactual simulations that forecast per-category retention lift, giving product teams a ranked roadmap for where model fine-tuning buys the most engagement per point of quality improvement
  • Eval Audit: Spearheaded a cross-org audit of 50+ Workspace AI features, surfacing a 15% post-rollout quality decay pattern. Now building an automated regression-detection scorecard to catch degradations before they reach users
  • Evaluation Enrichment Pipeline: Built a methodology to overlay behavioral engagement data onto offline evaluation sets to surface prompt gaps, directly influencing evaluation standards across major Sheets, Docs, and Gmail releases
Dec 2020 - Sep 2025
Google
Ads Operations
Boulder, CO
Senior Manager, Data Science
Team · 3–8 Data Scientists
  • Progressive journey: Forecasting Manager → Data Science Manager → Senior Manager across Google Ads customer support, strategy, and operations
  • Built AI Metrics: Championed NLP-derived sentiment scoring and LLM-based explanations as core KPIs, directly informing A/B testing roadmaps for AI-driven features
  • Automated Classification: LLM-based issue classification achieving 85% accuracy vs. 60% manual - key inputs for offline AI model evaluation
  • Influenced C-Suite: Leveraged causal impact modeling to reverse deprecation of live support for SMB customers
  • Deployed AI Toolkit: End-to-end LLM insight generation toolkit driving a 25% increase in identified product improvements
  • Crisis Response: Led time series decomposition during COVID to maintain forecast accuracy for strategic staffing
2019 - 2020
Infor
Boulder, CO · Remote
Head of Marketing & Ops Analytics
Team · 8 · Manager of Managers
  • Led cross-functional teams developing multi-touch attribution pipeline for data-driven ROI analysis
2017 - 2019
Oracle
Broomfield, CO
Sr. Marketing Data Scientist
  • Optimized audience targeting through causal impact analysis - persona-based planner contributing to +30% incremental revenue
2016 - 2017
InsideSales.com
Salt Lake City, UT
Sr. Marketing Data Scientist
  • Built multi-touch attribution database and decision tool - 60% increase in lead-appointment conversion rate
2013 - 2016
Backcountry
Park City, UT
Business Analyst → Manager, Marketing Analytics
Team · 3 · Analysts, DS, BI
  • Developed AHP bid optimization solutions shifting strategies from transactional to lifetime customer value
  • Built demand forecasting models; managed budgets and KPIs across channels
2012 - 2013
RRC Associates
Boulder, CO
Senior Data Analyst
  • Designed multi-modal surveys; applied advanced statistical modeling for high-profile ski resort clients
04 —

Education

2012
PhD in Sociology
University of Colorado · Boulder
Dissertation: "What's so local about global climate change?"
Certificate: Demography
Focus: Social Statistics · Demography · Climate & Population
Research: Collaborated with Nobel Peace Prize-winning scientists at NCAR. 15 peer-reviewed articles · 800+ citations.
Related Roles
Research Assistant
Nat'l Center for Atmospheric Research
2006–2011
Statistics Instructor
University of Colorado at Boulder
2010–2012
2004
BS in Mathematics
University of Puget Sound
Focus: Statistics · Geometry · Environmental Studies (minor)

Foundation in quantitative reasoning and analytical problem-solving that bridges data science and systems thinking.
05 —

Beyond the Data

01
2024 Site Citizenship Award
Recognized for exceptional contributions to the Boulder community through Google-Boulder partnership development.
02
Business Leader in Residence
Google-Boulder partnership with CU-Boulder Leeds School of Business - mentoring MBA students.
03
Former Sponsored Athlete
La Sportiva · Pearl Izumi · Salomon. Multiple podium finishes on the global mountain and trail running stage.
04
Running Shoe Tester
Boulder Gear Lab · 200+ reviews · Road Trail Run · 100K+ miles in the Rockies.
05
Eagle Scout
Boy Scouts of America, 1999. Early foundation in leadership and service.
06
NOLS Graduate
National Outdoor Leadership School, 2005. Wilderness leadership and risk management under pressure.
06 —

Selected Research

Title Journal Year
More than information: what coastal managers need to plan for climate change Environmental Science & Policy 2008
Vulnerability to inundation and climate change impacts in California Marine Technology Society Journal 2006
Testing theories to explore the drivers of cities' atmospheric emissions AMBIO: Journal of the Human Environment 2009
Development and greenhouse gas emissions deviate from the 'modernization' theory Climate Research 2008
View all on Google Scholar ↗