01

Research coverage record

Enough evidence to design the first study. Not enough evidence to judge Search+.

The October 13 scoping round located a substantial method and authority base. Its 4/5 coverage rating means the Lab can publish a falsifiable protocol. It does not support a claim that Search+ performs well or poorly in practice.

What the scoping round did and did not cover
StatusCoverageResearch consequence
Material11 Yale official research pages; 11 relevant papers or preprints; 6 BCHRT, CanLII or Justice Canada official sources; 3 existing Lab research draftsSufficient to define the access theory, study design and evidence-handling method.
Failed3 SSRN full-text requests were blocked; official abstracts onlyAbstracts were not used to support detailed claims.
SkippedReddit, 1Point3Acres and College Confidential; commercial legal databases unavailableAdmissions anecdotes do not establish research method; unavailable databases are recorded as a limitation.
Not yet doneNo logged-in Search+ benchmark; no authorized BCHRT corpus; no human-participant testingNo performance, completeness or user-outcome claim can yet be made.

02

Corrected research question

The test is whether access survives four separate thresholds.

Search+ accepts ordinary-language questions, creates and refines a structured CanLII search and can analyze the first 25 results. That corrects the earlier idea that ordinary people simply have no AI-assisted search. Individual access to a search interface and nonprofit access to a reusable research corpus remain different questions.

BCHRT directs people to CanLII while stating that the signed original in the Tribunal file is authoritative. Retrieval reliability, completeness and traceability therefore affect the operation of a public rights system; they are not merely product-review preferences.

Four thresholds of meaningful decision access

  1. Visible — a person can open a decision

  2. Searchable — a person can retrieve it from their facts

  3. Computationally usable — an authorized tool can index, compare and reproduce the search

  4. Practically actionable — the person can understand the authority and complete the next step

SourcesCanLII BlogCanLIIBC Human Rights TribunalCanLIIYale Law School Information Society Project

03

Methods drawn from adjacent Yale work

Four research traditions become one BC workflow.

This is not a copy of one Yale project. ISP supplies the access theory; DEC informs the audit discipline; Legal AI work informs hybrid reasoning; HRL informs evidence preservation. The Human Rights Access Lab supplies the BCHRT problem, the user journey and the implementation setting.

Method map: source tradition to Human Rights Access Lab practice
Research traditionWhat the source contributesHow the Lab applies it
Yale Legal AI LabLanguage interfaces combined with theorem provers, formal rules and guided questions; CLEAR tests factual and counterfactual queries with symbolic methods.Use guided questions for facts; deterministic rules for deadlines, forum and procedural conditions; language models for language, classification and retrieval.
Information Society Project · Access to KnowledgeAccess includes raw information, secondary analysis and knowledge-embedded tools and services—not merely online visibility.Test visible, searchable, computationally usable and practically actionable access separately.
Digital Ethics CenterA research programme concerned with auditable AI claims, governance and the effects of AI on public-facing administration.Set falsifiable criteria in advance; publish versions, failures and human-review rules; do not bend results to the advocacy position.
Humanitarian Research LabSource evaluation, data fusion and evidence preservation for human-rights and humanitarian investigation.Store canonical URL, retrieval time, hash, version and correction history so each finding returns to its source.

SourcesYale NewsarXivYale Law School Information Society ProjectYale Digital Ethics CenterYale Humanitarian Research Lab

04

Paper one · proposed

From an ordinary-language problem to authoritative BCHRT decisions.

Working title: “Searchable Is Not Actionable: Auditing Public Access to BC Human Rights Tribunal Decisions in the Age of AI.” The study asks whether ordinary users can retrieve the right decisions and do something verifiable with them.

  • Can an ordinary-English account retrieve genuinely relevant BCHRT decisions?
  • Does Search+ materially improve retrieval over traditional CanLII search?
  • Do results change across legal English, ordinary English, Cantonese-to-English phrasing, and short or misspelled descriptions?
  • After retrieval, can the user identify material similarity, judicial-review status, the supporting passage and a next step?
  • Do quota, completeness, corrections, licensing and bulk-access constraints prevent reproducible nonprofit research?
Proposed pilot

  1. Create 40 de-identified or synthetic BCHRT scenarios

  2. Write four query variants for each scenario

  3. Compare traditional CanLII search and Search+

  4. Add a BM25 baseline only after corpus authorization

  5. Two annotators judge relevance independently

  6. Resolve disagreements and publish the locked protocol

SourcesBC Human Rights TribunalCanLIICanLII

05

Evaluation before conclusions

A polished answer cannot compensate for missed authority.

CLERC reports that zero-shot retrieval models reached only 48.3% recall@1000 in its US case-law benchmark, while GPT-4o produced the highest ROUGE F-scores and the most hallucination in the generation test. LegalBench-RAG therefore focuses on retrieving minimal, highly relevant passages. These are not BCHRT performance estimates; they justify measuring retrieval and citation directly.

Metrics locked before viewing the study results
MeasureWhat it answersWhy it matters
Recall@5 and Recall@10How many known relevant decisions appear near the top?Missed authorities remain invisible to the user.
MRR and nDCG@10How early and how consistently are relevant results ranked?The first useful result affects time and abandonment.
Time and query revisionsHow much work is required to reach the first relevant authority?A usable system must reduce practical research burden.
Relevant-passage precisionDoes the retrieved excerpt contain the proposition the user needs?A related document can still supply the wrong passage.
Authority and citation traceabilityCan the result be traced to the correct source, version and pinpoint?The research pack must be independently reviewable.
Task completionCan the user complete a specified next step?Perceived helpfulness is not the same as actionable access.

SourcesACL Anthology · Findings of NAACL 2025arXiv

06

Research sequence

The corpus audit comes before testing vulnerable participants.

Three-paper programme
StudyScopeRelease condition
1 · Searchable Is Not ActionableTraditional search versus Search+ across controlled BCHRT scenarios and language variants.Can begin with bounded public-interface testing; BM25 waits for an authorized corpus.
2 · Visible, Searchable, Computable, ActionableCompare BCHRT, CHRT, CRT, WCAT and OIPC infrastructure: completeness, IDs, metadata, search, language, corrections, machine readability, authorized access, cost and provenance.Adapt Justice Canada’s user-centred Access to Justice Index and publish the scoring protocol.
3 · From Legal Problem to Research PackCompare an ordinary website with the Pathfinder for forum, deadline, authority, time, comprehension and confidence calibration among racialized newcomers and self-represented people.Requires formal consent, data minimization and academic or independent ethics review.

SourcesDepartment of Justice Canada

07

From evidence report to paper

Publish the protocol, the failures and the reproduction route.

  • Required reviewer: a lawyer or scholar familiar with BCHRT and administrative law.
  • Required reliability role: a second independent relevance annotator.
  • Required challenge function: an academic methods adviser willing to test the protocol rather than endorse the conclusion.
  • A 20–40 query pilot creates a concrete basis for later discussion with Yale researchers: a method, a result and a specific question.
Publication route

  1. Issue an NGO evidence report

  2. Prepare an English preprint

  3. Publish protocol, query set, code, results and limitations

  4. Create a DOI through Zenodo

  5. Submit to an access-to-justice, legal-informatics or NLP venue

  6. If text cannot be redistributed, publish IDs, annotations, hashes and reproduction steps

08

Evidence record

Research sources and limits

This page publishes the study design, not results. The October 13 scoping round is preserved with failed, skipped and unfinished work. Search+ has not yet been benchmarked, no authorized BCHRT corpus has been obtained and no participant study has begun.

  1. Yale Law School’s AI lab

    Yale News · November 12, 2025

    Lab mission, formal-reasoning approach, legal-aid distribution and Scott Shapiro’s “AI or nothing” statement.

  2. Access to Knowledge

    Yale Law School Information Society Project · Accessed October 13, 2026

    Defines access broadly to include raw information, secondary analysis and knowledge-embedded tools and services.

  3. Publications

    Yale Digital Ethics Center · Accessed October 13, 2026

    Current research programme covering auditable AI claims, governance, public-sector systems and street-level bureaucracy.

  4. Remote Sensing and OSINT Analysis for Human Rights and Humanitarian Applications

    Yale Humanitarian Research Lab · Accessed October 13, 2026

    Human-rights investigation training in source evaluation, data fusion and evidence preservation.

  5. ‘Put the Car on the Stand’: SMT-based Oracles for Investigating Decisions

    arXiv · May 9, 2023

    CLEAR uses symbolic execution and SMT solving to investigate factual and counterfactual questions about algorithmic decisions.

  6. CLERC: A Dataset for U.S. Legal Case Retrieval and Retrieval-Augmented Analysis Generation

    ACL Anthology · Findings of NAACL 2025 · April 2025

    The authors report 48.3% recall@1000 for zero-shot retrieval models; GPT-4o had the highest ROUGE F-scores and the most hallucination in their generation benchmark.

  7. LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain

    arXiv · August 19, 2024

    A retrieval benchmark centred on minimal, highly relevant legal passages rather than document IDs or large imprecise chunks.

  8. Development of an Access to Justice Index for Federal Administrative Bodies

    Department of Justice Canada · 2017 · updated index released 2025

    Measures access through technology, procedure, representation, information, costs, functionality and outcome transparency from a user perspective.

  9. Introducing CanLII Search+

    CanLII Blog · February 25, 2026

    Available to anyone with a free myCanLII account, subject to a daily limit.

  10. CanLII Search+ User Guide

    CanLII · 2026

    Search+ creates and refines a structured search; Analyze reads the full text of the first 25 results and assigns relevance scores.

  11. Terms of Use

    CanLII · Updated June 3, 2026

    Systematic downloading is prohibited; automated or large-scale retrieval is directed to original sources or authorized channels. Search+ availability, continuity and preservation are not warranted, and the service may be changed, suspended or discontinued.

  12. Decisions

    BC Human Rights Tribunal · Updated April 1, 2026

    CanLII and yearly pages; the signed original in the Tribunal file is authoritative.