Organisational and infrastructural challenges

Who does what around a model in production: the business data scientist as point of contact, and a qualification platform that matches production.

Authors

Affiliation

Nathan Randriamanana

Insee

Hugo Simon

Insee

Draft

This chapter is a draft, to be completed and validated by its authors.

Most of the difficulties of keeping a model in production are not technical. Practitioner surveys and case studies of machine learning deployments agree that the hardest problems lie at the boundaries between teams: agreeing on requirements, negotiating data quality, and integrating the model into an existing system (Amershi et al. 2019; Nahar et al. 2022; Paleyes et al. 2022). APE coding is no exception. This chapter describes two answers that proved decisive: a business data scientist who is the point of contact for the model, and a qualification platform that matches production.

Amershi, Saleema, Andrew Begel, Christian Bird, et al. 2019. “Software Engineering for Machine Learning: A Case Study.” Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 291–300.

Nahar, Nadia, Shurui Zhou, Grace Lewis, and Christian Kästner. 2022. “Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process.” Proceedings of the 44th International Conference on Software Engineering (ICSE). https://arxiv.org/abs/2110.10234.

Paleyes, Andrei, Raoul-Gabriel Urma, and Neil D. Lawrence. 2022. “Challenges in Deploying Machine Learning: A Survey of Case Studies.” ACM Computing Surveys 55 (6). https://doi.org/10.1145/3533378.

1 Three teams around one model

Three teams share responsibility for the model, each with its own constraints:

Table 1: Teams involved in APE coding.

Team	Owns	Main concern
Business team (Sirene register)	The classification, the coding rules, the manual coding teams	Quality of the APE code and workload of the coders
IT production teams	The register’s information system and the production infrastructure	Availability, security, change control
Data science team (Innovation Lab)	The training pipeline, the libraries, the development platform	Methods, tools, model quality

Without someone who belongs to the business side and understands the model, each question crosses team boundaries. Is a drop in the automation rate a drift or a seasonal effect? Is this reported error a one-off or a pattern? May a new model go to production? Those questions then wait, or get answered by whoever is closest, not by whoever should decide.

2 The business data scientist as point of contact

2.1 Why a dedicated role

The literature on machine learning in organisations describes the role names and responsibilities around a model in production (Kreuzberger et al. 2023). It also documents the benefits of placing data scientists within the business teams they serve, close to the domain and its decisions (Stelmaszak and Kline 2023). The AIML4OS framework, written for official statistics, makes it a recommendation: every model in production should have a named owner (Tayebjee et al. 2025, ch. 3).

Kreuzberger, Dominik, Niklas Kühl, and Sebastian Hirschl. 2023. “Machine Learning Operations (MLOps): Overview, Definition, and Architecture.” IEEE Access 11: 31866–79. https://doi.org/10.1109/ACCESS.2023.3262138.

Stelmaszak, Marta, and Kelsey Kline. 2023. “Managing Embedded Data Science Teams for Success: How Managers Can Navigate the Advantages and Challenges of Distributed Data Science.” Harvard Data Science Review 5 (2). https://hdsr.mitpress.mit.edu/pub/wfr9k3vq.

Tayebjee, Meilame, Nina Niederhametner, Bogdan Levagin, and Ariane Lestrade. 2025. From Text to Code (WP10.4) – MLOps Tutorial. AIML4OS project, Work Package 10. https://aiml4os.github.io/WP10_Cluster4_MLOps/.

For APE coding, this owner is a data scientist placed in the business team. They are distinct from the innovation team that built the first models. They speak both languages. They know the classification and the coders’ work, and they can read a risk–coverage curve, run a training job and review a pull request.

2.2 Responsibilities

Table 2: Responsibilities of the business data scientist.

Area	The business data scientist…
Requirements	Translates business needs into model requirements: target accuracy \(p^*\), acceptable manual workload, codes that must never be wrong
Annotation	Organises the annotation campaigns and the golden set: sampling, guidelines, the expert teams’ schedule
Business feedback	Collects reported errors, sorts one-off errors from patterns, and turns patterns into rules validated by experts
Retraining	Runs the retraining protocol and signs off the quality gates
Monitoring	Gives a first analysis of monitoring alerts and decides whether they need action
Release	Is the interlocutor of the IT production teams for every model release, from qualification to rollback
Communication	Reports on quality to management and to users of the APE code

The data science team stays responsible for the tools: libraries, pipelines, development platform. The business data scientist is responsible for the model as a service to the business. This separation lets the innovation team move on to other projects without leaving the model without an owner.

2.3 What it takes

The role works only if a few conditions hold:

time: retraining, annotation and monitoring are recurring work, not a side task;
access: the same development platform and the same repositories as the data science team;
skills: Python, Git, statistics, and knowledge of the classification;
a mandate: authority to decide on releases, within rules agreed with the business and IT teams.

3 A qualification platform that matches production

3.1 The gap between development and production

The model is developed and trained on SSP Cloud, an open data science platform with GPUs and internet access (see Architecture). It runs in production on INSEE’s internal infrastructure: CPU only, no internet access, strict resource limits and change control. Anything that differs between the two environments can break a release, or change the model’s behaviour without anyone noticing:

a library version that differs, and changes the preprocessing or the numerical results;
a resource that the model downloads at start-up, unavailable in a closed network;
a latency or memory footprint measured on a development machine, not on the production CPU;
a configuration difference in the way the API is exposed to the register.

3.2 What “iso-production” means

The qualification platform removes this gap. It reproduces production as closely as possible:

the same runtime: base image, Python and library versions, CPU-only builds;
the same resources: CPU and memory limits, number of replicas;
the same network constraints: no internet access, dependencies served from the internal repository manager;
the same deployment mechanism and the same configuration as production.

3.3 What is measured there

Each candidate model goes through the same series of checks before release (see Retraining protocol):

latency of unit calls (median, p95, p99) and throughput in batch mode, under a realistic load;
memory footprint and start-up time;
online/offline parity: the served model must reproduce the offline predictions on the test golden set exactly;
robustness: malformed or unexpected inputs must be rejected cleanly, not crash the service.

These are the non-functional metrics of the evaluation. They are only meaningful here.

flowchart LR
  DEV["Development and training<br/>(SSP Cloud: GPU, internet)"] --> REG[("MLflow<br/>registry")]
  REG --> QUAL["Qualification platform<br/>(iso-production: CPU, closed network)"]
  QUAL -->|"latency, memory,<br/>parity, robustness"| GO{"Go / no-go"}
  GO -->|"no"| DEV
  GO -->|"yes"| PROD["Production<br/>(internal infrastructure)"]

Figure 1: From development to production.

3.4 Who does what

Table 3: Sharing of responsibilities around qualification.

Step	Data science team	Business data scientist	IT production / architects
Model artifact and test suite	Provides	Reviews	—
Qualification platform	Uses	Uses	Provides and maintains
Go / no-go	Advises	Decides	Checks the operational criteria
Release and rollback	—	Requests	Executes

4 Open challenges

Sustaining annotation. The golden set needs a small but continuous annotation effort from business experts, in competition with their daily coding work.
Skills and continuity. A single business data scientist is a single point of failure. The role needs backup and documentation.
Keeping the platforms aligned. Every change to production (base image, resource limits, network rules) has to be carried over to the qualification platform, or qualification stops meaning anything.