Jiayu Shao

Did the agent actually complete the task?

An agent can follow every expected step and still miss the goal. It can also succeed through an unexpected path. At Microsoft, I worked on separating these outcomes so the team could better understand what needed fixing.

For Sico, an open-source Digital Worker platform, I designed a four-tier error attribution framework and split evaluation into two independent criteria: procedural compliance and task completion.

The framework helped distinguish evaluation errors from agent failures and supported more focused issue triage.

Finding where a design agent goes wrong

An unsatisfactory design can come from a misunderstood request, the wrong tool, incorrect parameters, or a weak final output. I worked on making those differences visible and useful to the team.

I analyzed agent interactions and tool calls to define evaluation criteria across the workflow. The analysis surfaced issues including intent drift, generation-parameter errors, and model-routing failures.

I also built an interactive evaluation prototype with AI-assisted coding, translating the framework into a workflow the team could try and discuss.

The evaluation workflow helped the team identify failure patterns and prioritize improvements.

Helping travelers who haven’t decided what they want yet

Travel plans often start with an open-ended question, without a clear destination or date. My main focus was designing dialogue and recommendation strategies for these ambiguous requests in the unified travel agent.

I designed context-aware recommendations that reduced upfront questions and helped users explore options before committing to a plan. I also defined how travel dates and destination should carry across conversation turns and services, so users would not need to repeat information they had already shared. Analysis of family-travel requests informed this work.

I also contributed to AI ticket-bundling recommendations for mixed-age families, translating their needs and discount eligibility into recommendation flows and tool-call requirements.

Connecting everyday routines with in-car intelligence

People refer to familiar destinations in personal ways. I worked on destination recommendations and intent understanding to make destination selection more intuitive.

I also worked with spatial and temporal features to support contextual recommendations.

For automotive AI applications, I defined training data categories, sampling requirements, and annotation quality checks.