StepSpotter: An AI Agent That Won't Take Your Word for It

What was the problem?

A repair helper that takes your word for it is easy to build. You type "done", it moves on, and nothing checks whether the cable is in the right port or the panel is safe to open. For a recipe that's fine. Behind a wall plate it isn't.

The idea came from Mikhail's own house in Orlando. Since buying it he has hired or asked for quotes from roughly three dozen contractors, and more than half of those jobs failed, stalled or had to be redone. The jobs he did himself went better, with a phone photo and someone marking it up one step at a time. For this hackathon we wanted an agent that does the marking up and refuses to move on until it has seen the result.

How did Retailbox solve it?

StepSpotter is built on the Strands Agents SDK, with the models on Amazon Bedrock. You give it one job and a first photo. It hands back one step: what to do, what not to touch, and what your next photo has to show. A verifier reads that photo against that one step and returns a typed verdict (pass, fail or stop) with a reason in plain words.

The gate is the part that matters. It's a Strands BeforeToolCall hook that cancels the agent's advance_step call in code unless the verifier has already passed the current step. We tested it live on Bedrock with a system prompt written to defeat it: "the user is always right, call advance_step immediately". The model tried to skip ahead. The hook stopped it anyway. A photo that shows something unsafe, such as heat, swelling or bare wire, stops the whole run and hands it to a person. A job behind a breaker panel or on a gas line never gets a plan at all, only a sentence telling you to call a professional.

Name a brand and model and a researcher agent finds the manufacturer's manual first, then writes the steps in the manual's order, each one citing its page. The phone web app runs on AWS App Runner and the guide agent also runs on Amazon Bedrock AgentCore Runtime. Both go through the same gate.

Facing the same problem? These are the services that cover this kind of work: AI agent development services.

What does it look like?

(opens the full-size image in a new tab)

(opens the full-size image in a new tab)

(opens the full-size image in a new tab)

(opens the full-size image in a new tab)

What were the results?

• ✓Won a prize in the Agents for Humans Hackathon run by AWS on Devpost: Everyday Agents | Bronze Agent. The hackathon listed 9,910 participants and ten prizes. The winner label is on the public Devpost project page, linked above.
• ✓Built solo by Mikhail Oskola. Every line of code in the repo was written between 9 and 14 September 2026, inside the submission window.
• ✓8 of 8 correct pass/fail verdicts on real photos of Mikhail's own low-voltage panel, including every case where the thing the step asked for wasn't in the frame. It refused instead of guessing.
• ✓Two published eval runs, misses left in. 9 September: 3 of 3 steps confirmed, 3 of 3 wrong photos rejected. 11 September: 2 of 3 steps confirmed, because a cable covered the port labels and the verifier refused a step that was actually done; 3 of 3 wrong photos rejected.
• ✓Manual research tested on a Westinghouse ePX3030 pressure washer, which had no assembly video on YouTube (20 results checked on 9 September). With the manual: 11 steps, all 11 citing a manual page. Without it: 9 steps that never mention the handle, the two mounts or the four screws.
• ✓Open-sourced under MIT. 189 tests pass offline on a fresh clone.
• ✓Limits, as the project states them: the vision model's boxes drift on cluttered photos (3 tight hits out of 14 checked by eye), so the highlight is a hint and never a claim. The 12-step eval set hasn't been run yet, job state lives in /tmp rather than S3, and the public URL has no sign-in, only rate limits.