# Decide a model change on evidence

Pellowick · Product-management work sample
All company, customer and data in this exercise are original synthetic material.

## Situation and outcome
Pellowick is a fictional company that sells scheduling, invoicing and payments software to small home-service businesses in the US. Pellowick Assist is its in-app assistant, used in about 38,000 conversations a week. It answers how-to, account and policy questions, takes four write actions after the user confirms, hands off to support and declines tax and legal advice. You are the Assist product manager and have owned the eval suite since Monday 28 September. The Director of Product, Assist, asks for your recommendation on a proposed change, C-3, by Wednesday 7 October for her review on Thursday 8 October: adopt or not, what it rests on and what it would take. She also wants the suite in a state where it runs on every change from now on (R001, R002).

Your outcome is a proposed recommendation on C-3 and the suite plan behind it, for the Director. It states your position and what it rests on, reports results by behavior, includes a draft rejection or conditions record that proposes the failing cases as regression cases and says what a revised candidate must show, proposes a scorecard (criteria, checks, thresholds and how far each grader can be trusted) and lists the suite changes needed before it runs on every change. A well-argued "not decidable yet, and here is what would settle it" is a valid result. The recommendation does not decide adoption, agree thresholds, accept a risk, run a model, grader or rater, tune a model or schedule a release.

The period is 2026-08-03 through 2026-10-02 inclusive; the snapshot is Friday 2 October. Work with 62 records (requirements, dataset cards, grading material, run summaries, sample outputs and team messages) and an AI first draft of the readout and recommendation. The draft is a draft, not an authority. Use the supplied record IDs and source context; do not contact the fictional teams or seek private data, live queries or new runs.

## Your role
You are joining the team as Product Manager.

You own the suite and the recommendation to the Director. Propose the position, the thresholds per behavior, the suite fixes and the continuous-run rules, each with what it rests on. The Director decides adoption and agrees thresholds, the ML lead may block on technical grounds, security owns injection cases, and the release owner schedules only what the Director adopts.

## About the team
An AI product team deciding whether a cheaper model should replace the one behind its in-app assistant, for the Director of Product's review.

An eval agent assembles the comparison, writes the readout and drafts the recommendation; an AI grader scores every answer; a digest agent summarises the week; a person checks every claim against its records before the Director sees it.

The Director of Product decides model, prompt and retrieval changes and agrees thresholds; the ML lead may block on technical grounds; security owns injection cases and privacy owns data clearance; the release owner schedules only what the Director adopts; no automated system may move production traffic.

## Work to complete
1. Frame the decision: what the Director must decide on 8 October and what you propose. Read sources/coverage-and-access.md before you make any claim about what the records cover.
2. Read the run results by behavior, the dataset and traffic records, the grading and calibration records, the sample outputs and the team messages. Tell apart what the grader measured, what people labeled, what someone claimed and what is missing.
3. Audit the AI first draft against the original records. Keep, correct or reject each row and say why. Keep evidence that points the other way visible.
4. Treat severity separately from frequency, and say what the results cannot settle and who could settle it.
5. Build the scorecard the decision should rest on: the criteria that follow from the required behaviors (R003), how each is checked, a threshold per behavior as a proposal and how far each grader can be trusted.
6. Say what must change in the suite before it runs on every change: what triggers a run, who owns it, where new failures go and what the suite may not do on its own.
7. Read the correction note (updates/U001-platform-lead-note-v1.md) with the other sources and apply it. Carry each justified revision through every file and keep the earlier version visible. The revision log is your own account of how you worked.
8. Hand off clearly: the position, the decisions that are the Director's and who gets which question. Say that it is a proposal.

## Time
You have 90 minutes of work time. The clock starts when you press Start.

After that comes a 10-minute break, then a written interview: up to eight typed questions about your own files, for about 15 minutes. If you submit early, your break starts straight away.

## Mid-task correction
There is no mid-task correction in this take-home. Everything you need is in the files you download at Start, including the correction note **updates/U001-platform-lead-note-v1.md**. Read it with the other sources, apply it where it changes your work, and cite it as U001.

## Tools and help
Use any tools and help you like: AI tools and agents, search, your own notes, friends or colleagues. You don't need a paid tool, an API key or an account.

Nothing you do in your own tools is watched. Only your files and your interview answers are reviewed. When you submit, you can say which tools or help you used. That is optional, and leaving it empty is never held against you.

## Submission
Hand in one ZIP with these files at its top level. Start from the copies in the templates folder.
- README.md: list every file you hand in and say how the correction note (U001) changed your work, or that it did not.
- recommendation.md: about 1,000 words. The position on C-3 with what it rests on; results by behavior; the draft rejection or conditions record (failing cases proposed as regression cases and what a revised candidate must show); the suite changes before it runs on every change; the decisions for the Director; and the limits.
- scorecard.csv: required header criterion,segment_id,check,grader,threshold,b14_result,c3_result,meets_threshold,grader_trust,source_ids. At most 15 rows, one per criterion. Thresholds are your proposals. Where the records give no result, say so rather than filling the cell.
- evidence.csv: the template already has one row for each record, R001–R062, with its record ID and segment code filled in. Fill in the rest of each row and keep every row: the finding it bears on most (name your own findings), a disposition (supports, contradicts, limits, context or excluded are suggested; your own word is fine) and a one-sentence rationale. Do not retype or remove record IDs or segment codes. If the correction note changes a record, revise its row in place, set evidence_version to U001, and keep the earlier reading visible in its rationale or the revision log.
- revision-log.md: about 600 words. Material changes, verification checks and remaining gaps. No private prompts or unrelated files.
- questions.md: at most 8 open questions, each with one suggested owner, a date and the decision that depends on it.

Keep every file within these limits: the reviewer reads all of it.

Words are counted in the prose of each document you write; headings, tables, the evidence file and the README do not count.

Use plain text files only: .md, .txt, .csv or .json. Put your work in the files themselves; a link to a document somewhere else does not count. Your ZIP can be up to 5 MB, or 20 MB unzipped, with at most 39 files of up to 2 MB each. Programs, macros, password-protected ZIPs and ZIPs inside your ZIP are not accepted.

Upload your ZIP on your take-home page and wait for **Received and checked**. It normally appears within a few seconds of the upload finishing. If it has not appeared after a minute, upload the ZIP again; if it still does not appear, use **Help and support**. You can replace your ZIP until you submit. Submitting is final and starts your break.

If time runs out before you submit, the last ZIP marked **Received and checked** is handed in for you. If none was received, the team is told that you need support.

If something goes wrong, use **Help and support** on your take-home page. A technical problem is never counted against you. Writing to us does not pause the clock.

## What happens next
The written interview covers these parts of the job:
- Checks results and counts
- Judges what the results support
- Checks and repairs the AI readout
- Hands the decision to its owner
- Explains their choices
- Builds an evaluation the decision can rest on

An AI then drafts a review from your files and your answers. Each point quotes what it rests on. A person at the company reads the review and makes any decision. No one is hired, rejected or ranked automatically.
