Kế hoạch di trú dữ liệu học sinh — quy tắc lưu trữ + di trú tại chỗ

Bối cảnh · Context

The Foundation's student data lives in a coordinator's personal Google Drive archive ("Thông tin học sinh"), outside the Foundation Shared Drive, across 3 school years — uploaded by year, then by school, with many consolidated files (a roster / KQ sheet / SE batch that covers a group of students, not one).

Over 2026-06-09/10, prior sessions already loaded 142 children into the live care store via one-off scratch scripts. That work is real but ad-hoc: the raw source still sits outside the Foundation Drive, provenance back to source documents is partial, there are known unresolved gaps (phantom students, grade-label mismatches), and there is no documented repeatable process.

This plan does two things, in order:

1. Establish the storage rules as a canonical reference.
2. Validate-and-formalize the existing migration in place so the live data conforms — without wiping live data (the care store is the sole copy of beneficiary data, and a known write-race makes re-disturbing it risky).

What already exists (reused, not rebuilt)

• Data model: partner_school, child, benefit, profile_entry, school_report, watchlist_item, ghi_danh. profile_entry already carries provenance: source, by_person, raw_link, source_ref. No register tables yet.
• Per-student "sổ tay" handbook: compiles a student's full record (Hồ sơ / Ghi danh / Học bạ all-years / Sổ phúc lợi / Nhật ký) into a native Google Doc in the per-student folder 2_Hoc-sinh/Ho-so/<mã> — <Tên>/, idempotent.
• Reports (child / school / cohort) → markdown + native Google Docs in …/Bao-cao/.
• Two namespaces already decoupled: human-facing Shared Drive band 2_Hoc-sinh/ vs internal machine store. This is the "two spaces" rule — already the architecture; this plan fills the gaps.
• Unattended intake: weekly drops → promote / extract, flag-on-failure.

Gaps vs. the 5 rules

Phần 1 — Thiết lập quy tắc (tài liệu tham chiếu)

New rule file .claude/rules/care-data.md — the canonical, agent-facing reference for student-data storage. Codifies:

1. Two spaces. Every input is persisted in both: the human-readable Foundation Shared Drive (as Google Doc/Sheet/Word/Excel/PDF, read-only to users) and the machine store (as .md/.txt/.sqlite).
2. Consolidated (group) docs. Stored intact in 2_Hoc-sinh/Tai-lieu-tong-hop/ (human) + a .md/.txt extraction (machine), cataloged in a new source_doc table. Each per-student fact pulled from it is written to that student's sổ tay as a journal entry whose source_ref = the source-doc id/section. The journal records both the entry and its source.
3. Student-specific docs. Stored in the student's folder 2_Hoc-sinh/Ho-so/<mã> — <Tên>/Tai-lieu-goc/ (human) + a machine md/txt mirror; key details → sổ tay. Reports do a full record review and present every time-series chronologically early→latest.
4. School + cohort registers. A standing per-school and per-cohort record (a Google Doc users read + a lightweight register_entry log) that accretes high-level events (prizes, hardships, milestones, headcount/funding roll-ups) as documents arrive. Reports read the register and roll up per-student data.
5. Migration discipline. Idempotent, dry-run-first, backups + write-race quiesce before any machine-store write; the live 142 are validated/formalized in place, never wiped.

Cross-references: a pointer in CLAUDE.md §Pointers and a short §16 "Storage & migration rules" in the beneficiary-care subplan (which stays the design-of-record).

Phần 2 — Sơ đồ thư mục chuẩn (hai không gian)

Human — Foundation Shared Drive band 2_Hoc-sinh/

2_Hoc-sinh/
  Ho-so/<mã> — <Tên>/            # per-student folder (EXISTS)
    So-tay · <mã> · <Tên>        # the handbook Doc (EXISTS)
    Tai-lieu-goc/                # NEW: this student's source docs (rule 3)
  Tai-lieu-tong-hop/             # NEW: consolidated/group source docs (rule 2)
    <school-year>/<school>/…
  So-tay-truong/<school>         # NEW: per-school register Doc (rule 4)
  So-tay-lua/<programme>-<year>  # NEW: per-cohort register Doc (rule 4)
  Bao-cao/<YYYY>/…               # generated reports (EXISTS)
  Khao-sat/<ung-vien>/…          # candidate surveys PRE-admission only

Machine — internal care store (prefix unchanged)

so-dang-ky.sqlite                # registry (the queryable shape)
Ho-so/journey-NNNN.md            # canonical profile (EXISTS)
So-phuc-loi/journey-NNNN/…       # benefits ledger (EXISTS)
Tai-lieu-vao/<week>/<journey>/…  # weekly intake inbox (EXISTS)
Tai-lieu-tong-hop/               # NEW: .md/.txt extraction of each consolidated doc
Tai-lieu-goc/journey-NNNN/       # NEW: .md/.txt mirror of student-specific docs

Phần 2b — Loại tài liệu & nguồn (lập kế hoạch ngay)

The ad-hoc loads added one document type at a time. To stop retrofitting, every source doc is typed (source_doc.doc_type) and classified by scope (per-student → rule 3 / consolidated → rule 2 / multi-scope → register, rule 4), with a routing target and a petal anchor known up front. Only the four highest-volume, recognisable types have structured extractors today (hoc-ba, so-diem, chuyen-can, hop-phu-huynh); we add a new one only where volume + shape justify it.

Sources (profile_entry.source, unchanged): school | coordinator | student | family | partner.

Per-student (rule 3 → student folder + sổ tay)

Consolidated / group (rule 2 → Tai-lieu-tong-hop/ + per-student extraction)

Multi-scope (rule 4 → register_entry): a school/cohort aggregate prize, an event recap, a hardship affecting a group, and the headcount/funding roll-ups — written to the school or cohort register.

Build implication: extend the school-doc extractor with three new schemas (ket-qua-thi, se, giay-khen) + their intake routing tokens. Sensitive types (giay-to, hoan-canh, health) honour subplan §3: the source doc is stored, but identifying medical / precise-circumstance detail is not copied into the profile body — only the care-relevant fact, scoped.

Phần 2c — Kênh tiếp nhận & định tuyến (tài liệu đến từ đâu, về đâu)

Four ways a document arrives; all funnel into one classify → route → extract → file pipeline; anything unmatched goes to a triage queue, never silently filed (the flag-on-failure discipline already built).

1. Hub upload (preferred). Staff upload a file and pick scope (student / school / cohort / candidate) + doc_type at upload time. Explicit metadata = deterministic routing. Written to both spaces + queued for extraction.
2. Email. A care-documents mailbox/label (mirrors the existing bank-notification lane): attachments pulled → triage queue, doc_type/scope inferred (filename token + roster name-match) or set by a human.
3. General drop folder. A catch-all care inbox swept by a watch job (mirrors ingest_watch): same triage.
4. Form-attached uploads. When a Form allows file/photo upload, the onFormSubmit doorbell already knows the form's context (which candidate/student/event), moves the files into the matching folder + machine store tagged with that context, then queues extraction — no guessing.

Routing/association: explicit metadata (hub pick or form context) → deterministic; else a filename token (<journey-id|student_code>__<doc-type>); a consolidated doc fans out per student via the roster name-matcher. Unmatched / low-confidence / multi-child-needing-split → the triage queue (a hub screen), never silently filed.

Phần 3 — Bổ sung lược đồ (cộng thêm, idempotent)

Extend the registry migration tool — following the existing IF NOT EXISTS / additive-column discipline.

source_doc — catalog of every source file

doc_id (PK, src-YYYY-NNNN) · title · kind (consolidated|student)
doc_type (Part-2b taxonomy: hoc-ba|se|giay-khen|…) · scope_type · scope_key
school_year · drive_url (human copy) · bucket_path (machine extraction)
original_name · transcription (1 = AI/OCR) · added_at · added_by

The journal's source_ref and the school-report's source field reference doc_id.

register_entry — one lightweight log for both register kinds

reg_id (PK) · scope_type (school|cohort) · scope_key · entry_date
kind (prize|hardship|milestone|headcount|funding|note) · detail
journey_id (nullable) · source_doc_id (nullable) · by_person · created_at

Phần 4 — Thay đổi mã nguồn · Code changes

1. Schema — add source_doc + register_entry (tables, index, helper fns); extend the selftest.
2. Chronological early→latest (rule 3) — flip the per-student journal + transcript display to oldest-first, while leaving the "latest value" selectors used by the school/cohort reports unchanged. Touches the handbook compiler, the child-report renderer, and a shared ordering helper. The child report's default window also becomes full programme history (not a trailing year) so it is the "full record review" rule 3 asks for.
3. Report tools & flows (rules 3–4) — beyond ordering: the child report surfaces each entry's source_ref → source_doc (traceable to its source); the school + cohort reports read the standing register_entry log (prizes/hardships/milestones) and roll up per-student data; and the two register compilers get the same delivery path as reports — a one-click hub trigger + scheduled run, tracked in report_index, upserted as native Docs.
4. Registers (rule 4) — deterministic per-school and per-cohort register Docs (headcount by status/year, lifetime funding, roster, and the register_entry event timeline), upserted to So-tay-truong/ and So-tay-lua/ via the existing Drive seam.
5. Migration tool — tools/care_migrate.py (NEW; supersedes the scratch one-offs): one documented idempotent CLI with --dry-run / --selftest and subcommands copy-source (copy the personal-Drive archive into the Foundation Drive + catalog it), backfill-provenance (link the loaded 142 to their source docs), reconcile (apply held-back fixes from a coordinator decisions manifest), build-registers (seed register_entry + emit register Docs).
6. Tests — extend the matching tool tests for new tables, chronological order, register compile; add a migration-tool test (fake Drive + temp store).

Phần 4b — Danh mục báo cáo (đặc tả đầu ra)

From the administrator's output notes. The current tools cover part of this; the new cuts are the school-year report, the all-programme Tổng hợp report, the grade-level (khối 6–12) breakdown, and window as a first-class parameter. Note lứa (cohort = programme-entry year) ≠ năm học (academic year) — both are reportable. Window: tu-dau (lifetime) · nam (trailing 12 mo) · nam-hoc (one academic year) · latest.

Build deltas: --window first-class on child/cohort (adds nam-hoc, latest); khối grouping in cohort/năm-học/overall (derive current khối from latest học bạ, already done in the banner); a thin Tổng hợp report across all programmes; the student talking-points variant = report_child latest + the already-built coordinator guidance block. "Bài viết cho media" stays the Area-2 output, listed for completeness.

Phần 5 — Thực thi di trú (giám sát, tại chỗ)

Per the known write-race discipline: quiesce before any machine-store write, back up first, verify after.

1. Back up the registry + a Drive snapshot of 2_Hoc-sinh/.
2. Schema migrate — run against a copy; quiesce; upload; verify the two new tables hold.
3. copy-source --dry-run → eyeball consolidated-vs-student classification → execute; populate source_doc.
4. backfill-provenance over the 142.
5. reconcile against the coordinator decisions manifest (phantom-student call is the coordinator's).
6. build-registers → register_entry + School/Cohort Docs.
7. Regenerate all sổ tay Docs + reports with chronological order.
8. Validate — profile validator, consistency check, every changed tool's selftest, the test suite.

Quyết định / ngoài phạm vi · Decisions / out of scope

• Phantom-student removal is the coordinator's call (feeds the reconcile manifest); this plan wires the mechanism, not the verdict.
• Copying from the coordinator's personal Drive into the Foundation Drive is itself the governance fix (rule 1); confirm read access at run time.
• Any Cloud Run hub redeploy (e.g. to surface registers in the hub UI) is a separate, per-deploy go-ahead; the register Docs are readable on the Drive without a redeploy, so a hub UI is optional/later.
• No machine-store prefix rename, no Phase-A change, no egress.