Review replies drafted by AI, graded by people, watched by a watchdog

Notification emails from the property-management system are parsed by an OpenAI model in JSON mode into address, rating, guest, dates, reservation, region, a critical flag, a complaint flag and categorised issues, then logged to Airtable (once, by checking the reservation first) and posted to the right region's Slack channel. Critical reviews post at any hour; ordinary ones only in working hours. The Slack message's timestamp is stored on the review, so the draft can later be posted in the same thread.

Every thirty minutes the drafter picks up to five reviews that have no draft yet and are rated four stars or lower, or are five-star with a complaint flag. A filter lets a record through only during working hours in the company's home time zone, and immediately sets it to Pending so the next poll cannot take it again. Outside hours the record stays blank and is simply retried.

The system prompt is fetched by name from an Airtable record and its version is stamped on every review and log row, so each graded outcome ties to the prompt that produced it. Each run also pulls the home's known constraints and listing disclosures and its best earlier positive review, with sentinel rows so those searches can never return empty and break the chain. The model is told that amenity data, messaging history and corrective-action evidence are unavailable, so it must not claim a fix and must raise a verification flag instead. Its schema-enforced output classifies each review (resolvable, structural, sensitive, mixed, escalate or no reply) and escalations and anything legal, safety or refund-related go to a person with no draft at all.

The draft is posted in the alert thread with a reaction legend and a row is written to a response log with its outcome pending. One grading scenario reads the emoji reactions every four hours (a check-mark beats an edit mark beats a cross, matched as whole tokens to avoid false hits) and sets posted as-is, posted edited or not posted. The other, more authoritative, asks the property-management system every six hours whether a public host reply now exists and compares it with the draft: an exact match is as-is, anything else is edited, and the human's final wording is captured for calibration.

A weekly report posts the counts in each outcome bucket and a pass rate against a go-live threshold. A daily digest goes to each regional manager by direct message, with nothing at all on days with no drafts. And every six hours a watchdog runs three checks: reviews in an error state, drafts stuck at Pending for over a day, and drafts ungraded for ten days, ahead of the fourteen-day window in which grades count. It posts one message only when a count is above zero, naming the likely cause and the fix, and it only reads, so it cannot disturb the pipeline.

A synthetic harness runs six fixed cases, each labelled with its expected result (pure praise, a feature with no constraint on record, a noise complaint where one is, an issue with no fix evidence, a safety allegation, a boundary case), and a backfill test runs the live prompt over the twenty-five most recent flagged reviews. Both write nothing to real reviews: only audit rows with a blank outcome, which keeps them out of the pass-rate measurement.