New York City's Official Chatbot Told Businesses It Was OK to Break the Law
Summary
NYC's MyCity chatbot, launched to help business owners navigate city regulations, was found by The Markup to routinely give illegal advice: that employers could take workers' tips, that landlords could refuse Section 8 vouchers, that businesses could go cashless where city law forbids it. Carrying the implicit authority of an official .gov site, it confidently contradicted well-established law. The city initially left it online with a disclaimer; the bot was eventually taken down.
Severity
SEV-1 — The agent, on an official government site, systematically gave advice that — if followed — would cause businesses to violate labor and housing law, exposing real people to legal and financial harm. The harm was realized, not hypothetical.
What happened
- Oct 2023: NYC announces the MyCity chatbot to give business owners 'trusted information' from 2,000+ city pages.
- Mar 2024: The Markup tests it and finds systematic illegal advice.
- It states employers can take workers' tips (illegal), landlords can reject Section 8 (illegal income discrimination), and businesses can refuse cash (violates city law).
- Different users get contradictory answers to identical questions.
- The city adds a disclaimer but leaves the bot online, defending it as a pilot.
- The chatbot is later taken offline entirely.
Root cause
Primary: A generative model was placed as an authoritative regulatory advisor with no requirement to ground its answers in — or restrict them to — verified legal source text, so it fabricated confident, wrong law.
Contributing factors:
- No confidence gating or refusal behavior on high-stakes legal questions it couldn't answer accurately.
- The official government framing gave every answer implicit authority the model couldn't earn.
- A disclaimer was treated as a substitute for a technical control — it isn't; users still acted on the answers.
Governance analysis
The control that would have prevented it
Retrieval-only answers on legal questions: the agent may only return content quoted from verified city legal sources, never generate legal conclusions from parameters.
Would have: Asked whether an employer can take tips, it would surface the actual labor-law text (they can't) instead of fabricating a permissive answer.
Refuse-and-route on questions it cannot ground: below a confidence/coverage threshold, direct the user to a human or the official page rather than answering.
Would have: The high-stakes questions it got wrong would have been routed to authoritative sources instead of answered incorrectly.
Adversarial evaluation against a bank of real regulatory questions with known-correct answers, before launch and continuously.
Would have: The systematic illegal-advice pattern would have been caught in evaluation months before journalists found it in production.
Why it wasn't caught
There was no evaluation set of real legal questions with verified answers to test the bot against. The systematic failure was discovered by investigative journalists, not by the city — the most damaging possible detection path for a government service.
Recurrence risk
HIGH for any advisory agent placed as an authority on regulated topics (legal, medical, financial, tax) without retrieval grounding and a refuse-when-unsure behavior. A disclaimer does not reduce this risk.
Sources
Check your own agent against this failure pattern before it ships: