PMDA makes its draft GenAI review points available in English
Japan’s Pharmaceuticals and Medical Devices Agency (PMDA) published a provisional English translation of its Review Points for Medical Devices Utilizing Generative AI (Draft) on 14 October 2026. The document is dated 14 September, and PMDA announced the start of feedback solicitation on 18 September. The October update makes the existing draft more accessible to international developers. It does not finalise the review points or introduce newly effective requirements. The Japanese original prevails if the texts differ.
The draft sets out possible approaches to evaluating generative AI medical devices, with particular attention to variable outputs, clinical context and changes after deployment. It states that PMDA had not yet reviewed a GenAI-enabled medical device for approval when the document was issued. Its approaches are provisional and illustrative; justified alternatives may be appropriate for individual products.
Define a clinical claim that the evidence can support
The document covers GenAI systems that qualify as medical devices under Japan’s Pharmaceuticals and Medical Devices Act. Its technical focus is on systems, typically built around large language or vision-language models, that generate varied responses through natural-language text or speech interactions. It does not settle all questions about which healthcare applications qualify as medical devices.
Developers need to establish who will use the product, its clinical purpose and how its outputs will influence decisions. For a product with broad diagnostic ambitions, a feasible evaluation protocol may be difficult to design without limiting the diseases or symptoms covered. The draft suggests considering a narrower initial intended use followed by gradual expansion. The appropriate evidence depends on the product’s clinical positioning and the scope of its claims.
Clinical usefulness may be established by demonstrating improved clinical outcomes or by showing that performance results justify clinical significance, depending on the available evidence and product context. A prospective clinical trial is anticipated where prospective evaluation of usefulness is needed. These are product-specific considerations rather than a single evidence route for every GenAI device.
Move from benchmark scores to clinical task performance
PMDA separates performance into three levels: general language capability and instruction-following, basic medical knowledge, and task-specific performance. General benchmarks and medical examination questions can provide a baseline. Task-specific assessment requires datasets that reflect intended clinical settings, including patient characteristics, comorbidities and atypical symptoms.
A high aggregate score cannot establish clinical usefulness or safety on its own. Developers should justify the representativeness of test data, define acceptable answers and assess consequential failures such as missed contraindications or incorrect emergency recommendations. Public benchmarks may also be present in training data, creating a risk of inflated performance estimates.
The draft gives particular attention to property-based testing. Instead of requiring a single exact answer, developers identify properties that acceptable behaviour must preserve and search for inputs that violate them. For example, a change in wording should not cause the system to recommend a contraindicated treatment. Additional clinically important information should lead to an appropriate change in the response.
Relevant test conditions include repeated inputs, paraphrasing, reordered information, accumulated conversation history, irrelevant material and requests outside the claimed function. Rule-based scoring, expert assessment and evaluation by another language model may be combined according to the task and testing scale. Evaluation criteria and reproducibility conditions need to be defined. These methods are examples, not an exclusive testing prescription.
Account for changes outside the manufacturer’s direct control
A foundation-model update delivered through a cloud API can change the behaviour of an otherwise unchanged product. Retrieval-augmented generation also introduces dependencies: changes to reference material, deleted data or service failures may affect outputs.
The draft distinguishes development models according to the applicant’s control over the system. Relevant issues include the ability to fix model or inference-environment versions, access training information and logs, manage updates and maintain security. Accountability and management arrangements, including contracts and service-level agreements, should reflect those dependencies.
As a practical preparation step, regulatory and engineering teams can document which conditions can be fixed, which changes can be detected and what responses remain available if performance deteriorates. Establishing these capabilities during supplier selection can prevent an evaluation plan from depending on information or controls that the supplier does not provide. This is an implementation inference from the draft’s discussion of controllability, rather than a prescribed submission checklist.
Plan post-market monitoring during development
PMDA emphasises a total life-cycle approach. Evaluation under specified premarket conditions needs to be supported by ongoing management that maintains the evaluated state, actively identifies incidents and enables an appropriate response. The document discusses collection of post-market performance, adverse-event and hallucination information, together with periodic reporting of whether findings raise clinical concerns.
For systems with agentic functions, monitoring may need to extend to execution logs. An apparently acceptable clinical outcome can conceal hazardous intermediate actions. The information collected, reporting arrangements and responses to problematic events should be discussed with the authority in light of the product’s risks and premarket evidence.
Post-market measures complement the evidence package. The draft does not present monitoring as an unrestricted substitute for adequate premarket evaluation. Its central challenge is to establish what can be demonstrated before marketing and how those properties can be maintained as the product and its operating environment change.
Address how users interpret and act on outputs
Evaluation also needs to consider how intended users understand generated information and how that understanding affects their decisions. The draft discusses presenting verifiable sources, distinguishing evidence-supported content from extrapolated inference, rejecting unsuitable requests through guardrails, and providing user education. Patient-facing products require particular attention to users’ ability to recognise errors and to safety measures built into the design.
The draft states that residual incompleteness in an output does not, by itself, preclude approval. The relevant question is whether remaining risks can be managed in the intended operating environment. For developers preparing Japanese market entry, useful discussion materials include a bounded clinical claim, an evaluation plan that represents its actual use and a post-market management approach consistent with the controls available from model and infrastructure suppliers.
A separate scheduling update on PMDA’s English SaMD page, also dated 14 October, lists availability for fee-based consultations by the Office of Software as a Medical Device, excluding Simple Consultation, as “After January 2027.” Where preliminary advice is requested, the page says the consultation meeting takes place two weeks after that advice is issued. Teams should check the current schedule when planning development discussions; this notice concerns consultation availability.
Sources
- Primary official sourceReview Points for Medical Devices Utilizing Generative AI (Draft): Provisional Translation as of October 2026 (opens in a new tab)Pharmaceuticals and Medical Devices Agency
- Supporting source新着情報（審査関連業務） (opens in a new tab)Pharmaceuticals and Medical Devices Agency
- Supporting sourceSoftware as a Medical Device (SaMD) (opens in a new tab)Pharmaceuticals and Medical Devices Agency