How We Built Expert Search for AI Group Chats

Imagine you're redesigning your product's onboarding and need to know whether it really improves activation, or whether the numbers just moved because your acquisition mix changed. Ideally, you would ask someone who has faced a similar issue before. But most of us don’t have someone like that for every hard question at work.

That is why we built Group Chat. When you ask a question like this one, SuperMe brings AI profiles of top growth operators (e.g., Elena Verna, Brian Balfour, Archie Abrams) into a chat that works through your problem in real time. Unlike common AI tools that give you “average” advice, each response is grounded in their lived experience and real-life examples, including insights you won’t find on the public internet (Figure 1).

But with thousands of profiles, how do you decide in a few seconds which people to bring into the chat for any question?

Authority and relevance

Inspired by how web search engines have long combined a query-independent score (e.g., PageRank) and a query-dependent match score, we rank profiles on two scores:

• Authority is a person’s mastery of a skill judged by their record in the field. It evolves over a career (Figure 6), so we assess it offline every few months.
• Relevance is how closely what they have written or said matches the query. It depends on the query, so we compute it in the few seconds the user waits.

The two scores are complementary. Relevance alone favors professional content creators who cover a topic through other people’s work. Authority alone favors proven operators, but they may not have shared their insights on the topic. Together, they surface people with a proven track record and insights on the topic (Figure 2).

Measuring authority

To measure authority, we do what a thoughtful recruiter would: research the person and compare them against their field's standards.

First, an LLM maps a user's career profile onto a taxonomy of 94 skills. Each record includes the skill, where and when they practiced it, their role (founder, operator, investor, etc.), and the following context tags.

Context matters because “sales at an early-stage consumer app” and “sales at a post-IPO enterprise SaaS company” are different jobs.

Next, for each skill, an AI agent researches the user's public record (company pages, press, talks, published work) and applies a custom assessment rubric with five gates. A candidate has to clear the earlier gates before the later ones matter:

The model is instructed to reason through every gate and provide evidence before picking a level. The result is one of six mastery levels: Beginner, Emerging, Competent, Advanced, Expert, Legendary. On average, only 8% of profiles clear the Expert+ bar required for a group chat. Below are example archetypes that meet that requirement.

Because the gates are re-checked every few months, a level is not permanent. Figure 6 is an illustrative example of one operator-turned-investor's two skills: when they move into venture capital, their product-growth skill loses a level for Recency while their investing skill climbs to Expert.

Measuring relevance

Relevance is measured over a corpus of insights people shared on SuperMe. Insights come from social sources (LinkedIn, X, Substack, YouTube) and connected sources, like Google Drive. Only content a person chooses to share on SuperMe is used; anything they keep private stays private.

Insights live in a vector database, and each chunk is indexed with a dense gemini-embedding-001 vector for semantic search and a sparse BM25 vector for lexical search.

We use hybrid search that fuses the dense and lexical results with Reciprocal Rank Fusion (k = 60) and keeps the top 1,500 chunks. We group chunks by person, and a person's relevance score is the mean fused score of their top 10 chunks.

End-to-end flow

Each query goes through four steps (Figure 8). The first three find the candidates in about 2.5 seconds; panel composition, the last step, adds a few more seconds.

1. Skill mapping

We use Gemini Flash to map a user question to the same skills taxonomy used for authority assessment. The model outputs up to six skills. When provided, the model extracts company-specific context tags.

For example, the question: "User growth at our seed-stage vintage clothing marketplace app has stalled. Is the problem in the new onboarding process or the acquisition channels?" maps to the following object.

{
  "skills": [
    "activation-onboarding",
    "user-acquisition",
    "product-growth",
    "growth-marketing",
    "data-analysis",
    "experimentation"
  ],
  "stage": ["early"],
  "vertical": ["consumer-tech"],
  "revenue_model": ["marketplace"],
  "customer_segment": ["consumer"],
  "geography": []
}

2. Authority filtering

Authority filtering turns the mapped skills into a candidate pool. We look up every skill record that matches one of the mapped skills or context tags and is assessed at expert or legendary. Candidates are then pre-ranked by an authority score, summed over the distinct skills they matched, and the top 200 are passed to relevance ranking:

Here S(p) is the set of distinct skills person p matched to the question, ℓₛ is their level in skill s, and cₛ is the number of the question's context dimensions that the skill record matches. Levels carry weights w(expert) = 1.6 and w(legendary) = 2.0.

Each matching context dimension adds 50% to that skill's contribution, so for a question on user activation at a start-up stage, an activation expert who did it at early-stage consumer companies outranks one who did it in enterprise.

3. Relevance ranking

Candidates are reordered by relevance (see Measuring relevance), and anyone with no relevant insight is dropped. The top ten go to the next step.

4. Panel composition

In the last step, the main chat agent reads the top ten profiles and picks a five-profile panel. It chooses based on how well each person's background fits the query, preferring firsthand experience in similar businesses. The agent has the full conversation context and can pick five of ten strong candidates to cover different angles of the user's problem.

Benchmarking expert search

We annotated 43 queries with SuperMe profiles well-positioned to answer them and measured recall@10 — the share of ground-truth experts who appear in the ten profiles handed to the agent for the panel selection. We chose recall over precision for two reasons. First, the agent picks the final five from all ten, so a weaker name on the shortlist costs little. Second, our labels list only the experts we would most want, not everyone who could answer well, so precision would penalize good picks we didn’t label.

We compared the retrieval pipeline against a naive baseline: no authority filter, ranking people by how many relevant insights they have.

Authority filter with relevance ranking beats the naive baseline on 24 of the 43 questions, loses on 3, and ties on the rest. On average, recall@10 improves by 0.16 (95% CI 0.10–0.23). We observed that ranking by volume tends to reward content creators who touch on a topic multiple times across different pieces and platforms, but lack lived experience and nuanced takes. Authority filtering narrows the field to people with a track record first, and relevance ranking then picks among them.

On latency, people search averages 2.5 seconds. About half of that is the skill-mapping LLM call; authority filtering and relevance ranking together take under a second. Panel composition adds a few extra seconds while the agent reads all ten profiles.

Where this falls short, and what’s next

At 0.58 recall@10, more than four in ten of the experts we want still miss the top ten. The gaps come from two places: the authority assessor and the retrieval pipeline.

Authority assessor failures

We turned each of these failure modes into explicit guidance in the rubric prompt. That reduced failures but hasn't eliminated them.

Brand laundering. The assessor gave people credit for a famous employer's reputation instead of their own work. Several years at a top company would push someone to Advanced or Expert even when nothing they personally owned justified that.

Over-correcting into skepticism. Once we prompted the assessor to be skeptical, it began under-rating operators famous for a nearby skill but who really do practice this one. A growth leader best known for SEO who also ran paid acquisition at scale would get marked down on paid acquisition, because their public reputation is about SEO.

Ambiguous seats in specialized fields. In some fields, the general rubric didn't know enough about the industry to judge mastery. For example, in venture investing, "partner at a firm that invested in Anthropic" could describe a GP with check-writing authority who led the round, or a junior partner who helped on the deal.

Recall failures

Skill mapping misses. If a question maps to the wrong skills, the right experts never become candidates. It is also our slowest step. Since skill mapping is a choice from a fixed set of options, we are testing a decision model instead of a general-purpose LLM. It returns a probability for every skill in the taxonomy and may improve both latency and matching.

The taxonomy is coarse. Some specialties are folded into broader ones, so specialists compete with generalists. We plan to add skills where evaluation shows gaps.

Want to see who shows up for your own question?