Technical evaluation

1.Key findings

All axes reach the reference level (a score of 3) or higher. Automated testing and CI/CD fall short of a score of 4 for a shared reason: no setting makes a failing CI run block the merge (and there are records of merges made before CI finished or while it was failing). CI/CD also has yet to pin external actions to commit hashes. The Technology choices axis stayed at a score of 3 because, by the subject date, there had been periods when merging of automated dependency updates stopped and vulnerability alerts had been fixed in batches, so updates and fixes cannot be said to run continuously as a mechanism. The conditions for a score of 5 that remain on the four axes that stayed at a score of 4 concern mainly records and disclosure: publishing the record of accessibility checks and extending automated enforcement of the accessibility rules, a verification record mapped to the OWASP ASVS Level 2 requirements and a threat model document, retrospectives that include comparing estimates with actuals, and disclosing the limits of the use of generative AI.

1.1Strengths

• The design conventions themselves run as automated tests, automatically detecting layer violations and misuse of the database entry points (Architecture and maintainability, Automated testing).
• Authorization centers on row-level security in the database, and a dedicated test suite checks, for each user role, which tables and operations are permitted, against a real database (Data design, Security).
• The Cloud Run runtime settings and the scheduler and monitoring settings are applied to live resources from declarations in code, and checks after each deployment and every day automatically detect drift from the declarations (CI/CD, Cloud platform and operations).
• Statements in the documents are automatically reconciled with values derived from the implementation, and requirement, feature, and screen IDs trace the documents to the implementation (Documentation).
• Business domain requirements such as external sharing and receiving, virus scanning, retention and deletion, and audit are implemented as part of the product, including monitoring, anomaly detection, and settings for operators (Features).

1.2Room for improvement

• A setting under which a CI failure mechanically blocks the merge (required checks), and a mechanism that makes production deployments wait for CI to finish (Automated testing, CI/CD).
• Operations that keep dependency updates and vulnerability fixes running without interruption (automated vulnerability checks in CI, deadlines for merging updates), a record comparing alternatives for the core runtime, data store, and authentication platform together with a format for decision records (in the form of ADRs), and pinning third-party GitHub Actions to commit SHAs (Technology choices, CI/CD, Documentation).
• Publishing a record of checking the WCAG 2.2 Level AA items, extending automated enforcement of the accessibility rules, and introducing automated accessibility checks that scan rendered screens, such as axe (UI, UX, and design).
• A verification record mapped to the OWASP ASVS Level 2 requirements, and a threat model document that brings assets, trust boundaries, and threats together in one place (Security).
• Retrospectives that include comparing estimates with actuals, with a record of the resulting improvements to the procedures, and disclosure of the limits of the use of generative AI (patterns of error, findings rejected on verification, and the burden of verification loops), together with defining the scope in which people read code diffs (Development process, Use of generative AI).

Back to contents

2.Evaluation method

Evaluator

At the developer's request, generative AI (Claude Code) analyzed the source code, configuration, and development records.

Basis

Five-level criteria for each axis were defined first with reference to published guidelines, and the facts (code, configuration, and figures) were scored against them. Each rating links to the diagrams, documents, and figures on this site (in Japanese) as evidence.

As of

The subject is the source code and development records as of 2026-09-17, evaluated on 2026-09-29.

Note

This is not an assessment by a third-party organization. It covers two perspectives, technical and market, that the project overview slides do not.

In defining the criteria, ISO/IEC 25010^[1], the international standard for product quality models, is used to organize the aspects, and among guidelines that define levels, the levels of OWASP ASVS^[2] for security and of WCAG 2.2^[3] for accessibility are borrowed as reference points on the scale. Borrowing a guideline's levels as reference points does not claim conformance or any certification. The full list of guidelines consulted for each axis is in the table in the appendix.

The reference point of the scale is the level a typical business system development team usually meets (a score of 3). Scores of 3 through 5 are cumulative: a score of n requires meeting every condition from level 3 through level n. The fact that one person developed the system is presented in figures in the next section, and no adjustment is applied to the scores for it. Every axis lists room for improvement: for an axis below a score of 5 it states the conditions not yet met for moving up to the next level, and for an axis that scores 5 it gives points for maintaining and extending the practices.

The evaluation has limits. A score of 3 is a level defined by the practices that published guidelines treat as basic, not a measured average of real teams. The evidence comes from reading the source code, configuration, and records as of the subject date, and it does not include runtime tests (operation with assistive technologies, checks of the runtime configuration, and actual delivery of notifications) or third-party inspections such as vulnerability assessments. The criteria set no numeric thresholds such as test coverage rates. Published guidance also advises against a uniform target^[4].

For the three axes Features, Documentation, and Use of generative AI, no published standard defines levels, so their levels are this report's own definitions, built up from practices in published material. The Use of generative AI axis evaluates the use of the same generative AI that serves as the evaluator; the provider's documentation is used as a list of practices, not as evidence of effectiveness. Scoring used one shared evidence log. Separate instances of the generative AI (separate conversations), not shown the first round of scores, judged each condition independently; another instance compared the two sets of judgments, and discrepancies and conditions judged as not met were rechecked directly against the source code as of the subject date. Where the readings of a condition diverged, the developer reviewed the evaluator's recommendation and confirmed the result.

Back to contents

3.Scale and team

Matsuba was built by one developer who handled every phase: requirements analysis, design, implementation, testing, deployment, and documentation. The development period is 4 months, and the recorded work time is 727 hours. Development uses generative AI (Claude Code) in every phase, and the methods and quality assurance of that use are evaluated under the Use of generative AI axis.

The source code is kept in a single repository with 5 applications and 12 shared packages. The frontend and the API are written in TypeScript, file scanning and synchronization to external storage are written in Go, and the database structure is managed with 79 migrations. The implementation is divided into 10 stages; each unit of work within a stage keeps its plans and work logs, and most units of work also keep specifications.

The breakdown of work hours by phase is in the Work hours table in the appendix, and the lines of code by language and by workspace are in the Lines of code tables.

Back to contents

4.Evaluation by axis

Each card below presents, in this order, the axis name and its focus, the score and its level name, the criteria for that score, a summary, the evidence, and the room for improvement.

Back to contents

5.Notable design decisions

5.1Contracts generated from schemas at three boundaries

At three boundaries (between the frontend and the API, for events between services, and at the database), contracts are not written twice by hand. The frontend and the API share the same validation schemas, and the API definition is generated from the input and output validation schemas declared for each route and committed to the repository; CI regenerates it every time and fails if there is any difference. For events between services, TypeScript and Go types are generated from Protocol Buffers definitions; the committed Go output passes the same diff check, and the TypeScript output is not committed but regenerated on every build. Database enumerations are derived from the schema definition, and static analysis blocks hard-coded values.

What the frontend and the API share is the shape of the validation schemas and fixed constants that break unless both sides agree on the same value. Values that vary with administrator settings or the environment are not shared but passed in responses, because when the frontend and the API are deployed separately, there are moments when only one side is new. The frontend does not import the API's types directly; type sharing is handled entirely by the validation schemas in the shared package. This decision is part of the evidence for Technology choices and Architecture and maintainability. The overall picture is in the application architecture diagram, and the technologies used are in the API and monorepo sections.

5.2Authorization centered on row-level security in the database

The unit of authorization is not a tenant but a shared space (a Box), and the same user can belong to several Boxes with different roles. A single role-based permission table is shared by the frontend, the API, and the authentication platform, and as the last layer, row-level security in the database holds the same role distinctions in SQL and enforces the scope of reads and writes with 103 policies. Rows without permission are returned as "not found" rather than denied, so that whether they exist is not revealed. Entry to the database is limited to three paths (users, recipients of share links, and the system) plus a dedicated entry for the authentication platform (with archiving of the audit log as the only documented exception), and static analysis (import restrictions) enforces which code can use which entry.

The policies follow conventions for naming and for stating the target roles explicitly, and a dedicated test suite applies the definitions to a real PostgreSQL database in the same order as production (migrations, then permission definitions, then policies) and checks, for each user role, which tables and operations are permitted. The authentication and REST API features provided by the managed database service are not used, and only the system's own services connect to the database. Tables that the authentication platform reads and writes without a user context are excluded from row-level security, and the reasons are documented. The diagrams are the security and authorization architecture diagram, the ER diagram (overview), and the storage and database architecture diagram, and the related axes are Data design and Security.

5.3Isolating side effects through an outbox

A convention keeps business transactions from directly triggering external side effects such as email, messaging, task queues, storage operations, and synchronization to external storage. Side effects that can take place afterward are written to the outbox in the same transaction as the update of the business data, a periodic dispatch job delivers them to Pub/Sub, and each service receives them. Updates and notifications do not diverge, and rows that fail are isolated at the retry limit and can be resubmitted from the administration screen. Receivers deduplicate deliveries by event ID (with records of processed events or unique constraints), and email sending is split into two transactions, before and after the send, to prevent sending twice.

Every topic has a dead-letter topic and its subscription paired with it, so messages that cannot be delivered do not disappear silently. The decision on who receives a notification is also gathered in one place. There are 57 notification types, and each process that resolves recipients must declare its "grounds for notifying each recipient" as a type. Exceptions to the conventions, such as sending authentication emails directly, are stated in the documents with reasons. The diagrams are the event-driven notification delivery diagram and the detailed system overview diagram, and the related axes are Architecture and maintainability and Cloud platform and operations.

5.4Configuration management by reconciling declarations with live resources

The cloud configuration does not use an infrastructure-as-code tool such as Terraform; the setup procedures are treated as the source of truth, and the parts that change often are declared in code and reconciled with live resources. The project does not put only some resources under such a tool, because that would create dual ownership by the tool and by hand, making it unclear which one is the source of truth. The deployment workflows declare the Cloud Run runtime settings and the injection of secrets, and every deployment reads them back from the live service to confirm that they match the declarations and that the secrets come from Secret Manager. For the scheduler and monitoring settings, declarations generated from a registry in code are committed, so the pull request diff serves as the change plan, and after deployment the declarations are applied and checked to confirm that no difference remains.

Deployments issue no key files at all and instead impersonate service accounts through Workload Identity Federation, and the paths that call services (event pushes, task delivery, and internal endpoints) authenticate with OIDC. Resources outside the declarations are reproduced from setup procedures with check items, and a daily check job reports drift between declarations and live resources as in-app alerts. The diagrams are the CI/CD pipeline diagram and the Google Cloud service architecture diagram, the technologies are in the observability section, and the related axes are CI/CD and Cloud platform and operations.

5.5The scanning and quarantine lifecycle for uploads

Uploaded files are handled in three tiers of buckets: unscanned, clean, and quarantine. The file content is transferred directly from the browser to storage with a signed URL, a scanning service that receives the upload-completion event from storage runs a virus scan, and the file moves to the clean or the quarantine bucket depending on the result. Moves always follow the order "copy, commit in the database, delete the original", so the original is not lost even if a step fails midway. The current version visible to users is only the latest version that passed the scan, and the previous version stays usable while a replacement is in progress.

Access to the quarantine bucket is split between a principal that "can delete but cannot read" and a principal that "can read only when impersonated", so that quarantined files cannot be retrieved through the normal download path. The thumbnail generation service, which parses users' files, has no permissions on storage, the database, or secrets. The scanning path runs a monthly self-test with the EICAR test file, and failures to update the virus definition files are also reported as anomalies. The diagrams are the file lifecycle diagram and the Google Cloud service architecture diagram, the document is "virus scanning of uploaded files", and the related axes are Security and Features.

5.6Letting machines enforce the conventions

The design conventions are not left to documents and reviews alone; they also run as automated tests. There are 78 convention tests in the dedicated directories (other checks of the same kind exist as well), and the rules for building them are to find their targets by scanning the code rather than from a hand-kept list and to assert that detection does not pass vacuously; in many of the checks, the detector itself also has positive and negative samples. Most convention tests run in a job whose start depends on repository variables and the scope of the change, so the checks that must hold even on pull requests where that job does not run are named in the job that runs every time, and the diff checks of generated artifacts are placed there as well.

The same idea extends to the documents: CI runs, every time, a check that reconciles statements in the documents (API endpoints, screen routes, topics, and the DNS records for sending email) with values derived from the implementation at run time, and a check of the writing rule that documents describe only the current state, not its history. Rules that relied on human attention broke down repeatedly, which led to pairing each convention with a mechanical check. The diagram is the design rules diagram, the technologies are in the static analysis section, and the related axes are Architecture and maintainability, Documentation, and Use of generative AI.

Back to contents

6.Appendix

6.1Axes and the guidelines consulted

These are the published guidelines consulted when defining the five-level criteria for each axis. The numbers correspond to the list of sources.

6.2Full text of the criteria

6.3Work hours

6.4Lines of code

6.5How the figures are counted

These are the names, values, and counting methods of the figures used in the text, the figure cards, and the evidence.

Screens 68

Number of page.tsx files in the web app's App Router tree, excluding the developer preview screens

API endpoints 375

Number of paths in the OpenAPI definition (402 operations by HTTP method)

Database tables 61

Number of model declarations in the Prisma schema

Background jobs 30

Number of registered scheduled and on-demand jobs (27 scheduled, 3 on-demand; the two standalone Go services are not included)

Test files 1,582

Number of automated test files (1,491 TypeScript unit and integration tests, 58 E2E specs, and 33 Go tests)

Lines of code 835,979

Lines in program, test, and configuration files (TypeScript, Go, SQL, and others), excluding documents, generated files, and blank lines, and including comments (481,442 lines of product code and 354,537 lines of test code)

Work hours 727 hours

Sum of the work hours recorded per phase during the development period (27 items including requirements analysis, design, development, testing, and documentation)

Development period 4 months

Months from the start of requirements analysis to the completion of implementation, including testing and deployment (mid-May to mid-September 2026, rounded down)

Applications 5

Number of top-level directories under apps/ (web, api, and thumbnailer in Node.js; sync and scanner in Go)

Shared packages 12

Number of top-level directories under packages/ (11 pnpm packages and 1 Go module)

Migrations 79

Number of database schema migrations (migration directories, from the initial one through the subject date; applied migrations are never rewritten)

Implementation stages 10

Number of stages into which the implementation was divided (environment setup, authentication, the file-sharing foundation, extended file features, reliability, retention, administration screens and audit, public pages, in-app help, and release preparation). Each stage holds smaller units of work that keep plans and work logs, and most also keep specifications and review records

RLS policies 103

Number of policies declared for row-level security in the database (CREATE POLICY statements in the definition file, excluding comments). Row-level security is enabled on 53 tables, 48 of which have policies; the 8 tables that the authentication platform reads and writes without a user context are excluded

Notification types 57

Number of notification types sent to users and administrators (the number of template identifiers): 46 whose text administrators can edit plus 11 fixed in code for authentication, including 5 delivered only inside the app. Variants of the text by theme or recipient are not counted

Convention tests 78

Number of automated test files that mechanically check design conventions (those in the dedicated convention-test directories: 72 for the web app and 6 for the API; similar tests outside those directories, in shared packages or in each app's unit-test locations, are not counted)

Help articles 95

Number of user-facing help articles built into the app (MDX; sample articles used only for checks are not counted). CI checks that the articles cover every entry of the feature catalog and the screen catalog

Design and implementation documents 343

Number of Markdown documents under the repository's docs directory (six sections: baseline concept 18, design candidates 2, feature catalog 37, page catalog 57, design rules 16, and implementation plans and records 213; help articles and development process documents are not counted)

Agent skills 13

Number of skills for generative AI agents managed in the repository (6 for domain knowledge, 6 for phase procedures, and 1 for document design; the 5 role-specific agent definitions are counted separately)

6.6Sources

1. [1]

   ISO/IEC 25010:2023 Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model(opens in a new tab) ISO/IEC

   https://www.iso.org/standard/78176.html accessed 2026-09-26

2. [2]

   OWASP Application Security Verification Standard (ASVS)(opens in a new tab) OWASP Foundation

   https://owasp.org/projects/asvs accessed 2026-09-26

3. [3]

   Web Content Accessibility Guidelines (WCAG) 2.2(opens in a new tab) W3C

   https://www.w3.org/TR/WCAG22/ accessed 2026-09-26

4. [4]

   Code Coverage Best Practices(opens in a new tab) Google Testing Blog

   https://testing.googleblog.com/2020/08/code-coverage-best-practices.html accessed 2026-09-26

5. [5]

   Node.js Releases(opens in a new tab) Node.js

   https://nodejs.org/en/about/previous-releases accessed 2026-09-26

6. [6]

   Next.js Support Policy(opens in a new tab) Vercel

   https://nextjs.org/support-policy accessed 2026-09-26

7. [7]

   Release History (Release Policy)(opens in a new tab) The Go Project

   https://go.dev/doc/devel/release accessed 2026-09-26

8. [8]

   Versioning Policy(opens in a new tab) PostgreSQL Global Development Group

   https://www.postgresql.org/support/versioning/ accessed 2026-09-26

9. [9]

   The Twelve-Factor App(opens in a new tab) Adam Wiggins

   https://12factor.net/ accessed 2026-09-26

10. [10]

    Check Documentation (Scorecard checks)(opens in a new tab) OpenSSF

    https://github.com/ossf/scorecard/blob/main/docs/checks.md accessed 2026-09-26

11. [11]

    Capabilities: Code maintainability(opens in a new tab) DORA

    https://dora.dev/capabilities/code-maintainability/ accessed 2026-09-26

12. [12]

    Capabilities: Loosely coupled teams(opens in a new tab) DORA

    https://dora.dev/capabilities/loosely-coupled-teams/ accessed 2026-09-26

13. [13]

    5.5. Constraints (PostgreSQL 18 Documentation)(opens in a new tab) PostgreSQL Global Development Group

    https://www.postgresql.org/docs/current/ddl-constraints.html accessed 2026-09-26

14. [14]

    Capabilities: Database change management(opens in a new tab) DORA

    https://dora.dev/capabilities/database-change-management/ accessed 2026-09-26

15. [15]

    ウェブアクセシビリティ導入ガイドブック(opens in a new tab) デジタル庁 (in Japanese)

    https://www.digital.go.jp/resources/introduction-to-web-accessibility-guidebook accessed 2026-09-26

16. [16]

    10 Usability Heuristics for User Interface Design(opens in a new tab) Nielsen Norman Group

    https://www.nngroup.com/articles/ten-usability-heuristics/ accessed 2026-09-26

17. [17]

    What is the ASVS? (ASVS 5.0.0)(opens in a new tab) OWASP Foundation

    https://github.com/OWASP/ASVS/blob/master/5.0/en/0x03-What-is-the-ASVS.md accessed 2026-09-26

18. [18]

    OWASP Top 10:2025(opens in a new tab) OWASP Foundation

    https://top10.owasp.org/2025/ accessed 2026-09-26

19. [19]

    Well-Architected Framework: Security, privacy, and compliance pillar(opens in a new tab) Google Cloud

    https://docs.cloud.google.com/architecture/framework/security accessed 2026-09-26

20. [20]

    Software Engineering at Google, Chapter 11: Testing Overview(opens in a new tab) Google

    https://abseil.io/resources/swe-book/html/ch11.html accessed 2026-09-26

21. [21]

    Capabilities: Test automation(opens in a new tab) DORA

    https://dora.dev/capabilities/test-automation/ accessed 2026-09-26

22. [22]

    Capabilities: Continuous integration(opens in a new tab) DORA

    https://dora.dev/capabilities/continuous-integration/ accessed 2026-09-26

23. [23]

    Capabilities: Deployment automation(opens in a new tab) DORA

    https://dora.dev/capabilities/deployment-automation/ accessed 2026-09-26

24. [24]

    Secure use reference (GitHub Actions)(opens in a new tab) GitHub

    https://docs.github.com/en/actions/reference/security/secure-use accessed 2026-09-26

25. [25]

    Google Cloud Well-Architected Framework(opens in a new tab) Google Cloud

    https://docs.cloud.google.com/architecture/framework accessed 2026-09-26

26. [26]

    Well-Architected Framework: Operational excellence pillar(opens in a new tab) Google Cloud

    https://docs.cloud.google.com/architecture/framework/operational-excellence accessed 2026-09-26

27. [27]

    Site Reliability Engineering, Chapter 4 - Service Level Objectives(opens in a new tab) Google

    https://sre.google/sre-book/service-level-objectives/ accessed 2026-09-26

28. [28]

    Site Reliability Engineering, Chapter 6 - Monitoring Distributed Systems(opens in a new tab) Google

    https://sre.google/sre-book/monitoring-distributed-systems/ accessed 2026-09-26

29. [29]

    Capabilities: Monitoring and observability(opens in a new tab) DORA

    https://dora.dev/capabilities/monitoring-and-observability/ accessed 2026-09-26

30. [30]

    Diátaxis(opens in a new tab) Daniele Procida

    https://diataxis.fr/ accessed 2026-09-26

31. [31]

    C4 model(opens in a new tab) Simon Brown

    https://c4model.com/ accessed 2026-09-26

32. [32]

    arc42 Template Overview(opens in a new tab) arc42

    https://arc42.org/overview accessed 2026-09-26

33. [33]

    Architecture decision records overview(opens in a new tab) Google Cloud

    https://docs.cloud.google.com/architecture/architecture-decision-records accessed 2026-09-26

34. [34]

    Capabilities: Documentation quality(opens in a new tab) DORA

    https://dora.dev/capabilities/documentation-quality/ accessed 2026-09-26

35. [35]

    Capabilities: Working in small batches(opens in a new tab) DORA

    https://dora.dev/capabilities/working-in-small-batches/ accessed 2026-09-26

36. [36]

    Capabilities: Trunk-based development(opens in a new tab) DORA

    https://dora.dev/capabilities/trunk-based-development/ accessed 2026-09-26

37. [37]

    Capabilities: Version control(opens in a new tab) DORA

    https://dora.dev/capabilities/version-control/ accessed 2026-09-26

38. [38]

    Capabilities: Streamlining change approval(opens in a new tab) DORA

    https://dora.dev/capabilities/streamlining-change-approval/ accessed 2026-09-26

39. [39]

    DORA AI Capabilities Model report(opens in a new tab) DORA

    https://dora.dev/ai/capabilities-model/report/ accessed 2026-09-26

40. [40]

    Capabilities: Clear and communicated AI stance(opens in a new tab) DORA

    https://dora.dev/capabilities/clear-and-communicated-ai-stance/ accessed 2026-09-26

41. [41]

    Best practices for Claude Code(opens in a new tab) Anthropic

    https://code.claude.com/docs/en/best-practices accessed 2026-09-26

42. [42]

    Security (Claude Code Docs)(opens in a new tab) Anthropic

    https://code.claude.com/docs/en/security accessed 2026-09-26