An ML platform team
31
DAYS PR
[ abstract ]
An ML platform team needed an event-level feature set: one row per event, with last-touch attribution and rolling counts. By hand, that was about three days of an engineer's joins. The agent took eleven attempts and four hours, and it merged as one PR that ties back to the source tables exactly.
| ask | one row per event, with features |
| sources | events, catalog, attribution |
| checked against | the source tables |
| attempts | 11 |
| wall clock | 4 hours |
| before | ~3 engineer-days |
[ context ]
Feature work is mostly joins, and joins are where feature sets go wrong without telling anyone. A fanout inflates counts. An off-by-one window shifts a label. The model still trains, and you find out in production, if you find out at all.
New features have no old output to compare against, so the check came from the source data. There's one row per event, counts and sums tie back to the raw tables, and every event has exactly one last touch.
[ method ]
- 01
Write down the checks
The data scientists described the features and what would make them wrong. Those became the checks.
- 02
Attempt
The agent wrote the joins and ran them on a 4.20% scale model, with whole users and full history.
- 03
Miss, read the diff, go again
Each miss came back with the rows that broke the check. Eleven attempts in four hours.
- 04
Merge
One PR with the evidence attached. The team reviewed that instead of three days of SQL.
[ the run log ]
One feature set. Eleven attempts.
attempt 01
· missed · rows drift 1,847,002
fanout: events × catalog exploded the row count 41×
attempt 04
· missed · rows drift 203,441
window boundary: last touch credited to the next UTC day
attempt 11
✓ matched · rows drift 0
zero diff. KS 0.0000. 4 hours wall clock.
[ what missed, and why ]
The log has three turning points. The attempts in between were smaller versions of the same two problems:
attempt 01
feature_set
Events joined to the catalog without the right key, so every event matched every product variant.
showed up as · row count 41× the events table
fix · joined on sku and variant
attempt 04
feature_set
Last touch used a UTC day boundary, so late-evening touches were credited to the next day.
showed up as · 203,441 rows drifted
fix · windowed on event time, not calendar day
attempt 11
feature_set
Nothing. Counts tied to the source, one last touch per event.
showed up as · zero diff, KS 0.0000
fix · merged
[ result ]
| engineer-days before | 3 |
| review after | 1 PR |
| attempts | 11 |
| wall clock | 4 hours |
| drift at merge | 0 |
[ who did what ]
The data scientists chose the features and wrote down what correct meant. They didn't write the joins, and they spent the three days on the model instead.
“Our data scientists stopped writing joins and started choosing features.”
[ notes ]
- n1
A feature set can match its source and still be a bad feature. The check proves the data is right. It doesn't prove the model gets better, and that part stays with the data scientists.
[ read next ]
Working on something like this? Chat with Founders →