Senior Data Scientist - ML Operations & Analytics 🧑💻
Filevine is forging the future of legal work with cloud-based workflow tools. We have a reputation for intuitive, streamlined technology that helps professionals manage their organization and serve their clients better. We're also known for our team of extraordinary and passionate professionals who love working together to help organizations thrive. Our success has catapulted Filevine to the forefront of our field - we are ranked as one of the most innovative and fastest-growing technology companies in the United States by both Deloitte and Inc. Our Mission is building the seamless intersection between legal and business by creating a world-class platform to help professionals scale.
We're a team of driven, enthusiastic problem solvers with strong backgrounds in machine learning, engineering, product management, legal and operations, on a mission to help attorneys resolve cases faster, for better outcomes. With two established ML teams pushing boundaries in transcription, NLP and GenAI applications, we're now building a critical analytics layer to ensure operational excellence, quality assurance, and continuous improvement across all our AI systems.
Key Responsibilities
Build ML operations analytics tracking pipeline health, costs, latency, and quality metrics across all services
Lead LLM evaluation initiatives using frameworks like Braintrust or Langfuse, designing custom metrics, quality monitoring and LLM observability in general
Collaborate with our data annotation team on prompt engineering and evaluation dataset creation
Aggregate and operationalize user feedback for model improvement
Create actionable dashboards and insights for ML teams and leadership
Help define SLAs, cost optimization and failure prevention strategies for our ML pipelines
Requirements
3+ years in data science, ML operations, or analytics engineering
Experience with LLM evaluation frameworks and quality metrics design
Strong Python, SQL, and data visualization skills
Knowledge of prompt optimization techniques
Excellent communication skills in English to translate technical metrics into business insights
Collaborative mindset with ability to work across machine learning teams
Ability to meet periodically in our Prague office as we value in-person collaboration
Estimated Time Commitment: 160 hours/month
We’d love to connect