Specification · draft

Draft. This is the draft of the METAIS specification at commit f7f57020f4fe5067f6107bae90bb3ae07e1fa803, not a released revision. It may change before it is released.

10. Transforms

Normative. Areas: TRN.

An output may be transformed after it renders, and only by the transforms this specification defines. This chapter specifies zip container outputs, with the settings the Office Open XML (ECMA-376), OpenDocument (ODF) and EPUB (EPUB) formats require of them, conversion of line endings to CRLF, and validation of an output against a schema the package carries.

10.1 Zip containers

A container output is one zip file whose entries are declarations of the forms an output takes, written inline beneath it. A part that must also be a file of its own is declared again at the top level: an entry is written only into its container.

MTS-TRN-001 A processor MUST write a container output as one zip file, at the container's path in its enclosing tree, whose entries are the files of the container's entries declarations and nothing else: each entry's file rendered exactly as a top-level output with that declaration is, at its path relative to the root of the container, and none of them written on its own. A container MUST hold file entries only, never an entry for a directory. Within an entry's template, output.path is the entry's path within its container and output.root leads from the entry back to the container's root.

MTS-TRN-002 A container MUST be its own tree for §11.4's portable-path rules: a processor MUST hold each entry's path to them within its container, MUST look for collisions and file–directory conflicts among the container's entries only, and MUST report each with the code §11.4 gives it, located at the entry's declaration. Entry ids MUST be unique within their container, and an entry MUST NOT declare executable, since a container records no file mode; a processor refuses both as it refuses any other manifest that breaks its rules (chapter 3).

MTS-TRN-003 A container MAY hold a container as an entry, to the depth of the container-depth limit (chapter 14), counting the outermost container as the first. A processor MUST refuse a container nested deeper with limit.package naming container-depth, located at the deeper container's container member.

MTS-TRN-004 A processor MUST write a container's entries in this order, whatever the order of their declarations: first the entries its profile places first (§10.2), in the profile's order; then the files of every entry declaration with first: true, in declaration order, each declaration's own files in byte order of their entry paths; then every other file, in byte order of its entry path, comparing the paths' UTF-8 bytes. Each file MUST appear once, at the earliest of these places.

MTS-TRN-005 A processor MUST store, with compression method 0, every file of an entry declaration with stored: true and the mimetype entry a profile writes; every other entry it MUST deflate (method 8) when that is strictly shorter than its content and store otherwise, so that an empty entry is always stored. A conforming processor's choice between the two for an entry without stored: true is its encoder's, and is not compared.

MTS-TRN-006 A processor MUST write each container's zip with every field a package's zip fixes (chapter 2): the fixed version, flags, time and date and external attributes; no extra field and no comment on any entry; sizes and CRC in each local header, with no data descriptor; no zip64 record; and an end record that is the file's last 22 bytes. The entries follow one another with nothing between them, in the order of MTS-TRN-004.

MTS-TRN-007 A processor MUST refuse a container that plans more than 65,534 entries, a profile's mimetype among them, with output.container.too_many_entries, since a zip without zip64 holds no more.

MTS-TRN-008 A container without a profile MUST be written exactly as declared: no entry is added to it, none is required, and an entry may have any portable path, mimetype included.

MTS-TRN-012 A container with for_each MUST be written once for each evaluation whose when holds, as one zip at the path that evaluation renders (chapter 9), with its entries planned and rendered in that evaluation: every entry's when, path and template MUST see the evaluation's item and fan_out. An entry of such a container MUST NOT have a for_each of its own (MTS-MAN-016).

10.2 Container profiles

A profile states a format's packaging rules once, so an author who names it gets them right without knowing them. It applies the settings the format requires, and verifies what can be seen without reading a part's content; it never validates a part's XML. Each of a profile's refusals and its warning is located at the entry's declaration, and a missing entry at the container's profile.

MTS-TRN-009 A container with profile: ooxml MUST have an entry whose path is [Content_Types].xml, which a processor MUST write first; a processor MUST refuse a container without one with output.container.missing_entry, and an entry whose path holds a character outside ASCII with output.container.non_ascii.

MTS-TRN-010 For a container with profile: odf, a processor MUST write an entry mimetype itself, first, whose content is exactly the container's mimetype with no line ending, stored, with no extra field and no data descriptor. It MUST refuse an entry of the author's whose path is mimetype with output.container.reserved_entry, and a container with no entry META-INF/manifest.xml with output.container.missing_entry.

MTS-TRN-011 For a container with profile: epub, a processor MUST write an entry mimetype itself, first, whose content is exactly application/epub+zip with no line ending, stored, with no extra field and no data descriptor. It MUST refuse an entry of the author's whose path is mimetype with output.container.reserved_entry, and a container with no entry META-INF/container.xml with output.container.missing_entry; and it MUST report output.container.space, a warning, for each entry whose path holds U+0020.

10.3 CRLF conversion

The transforms run on an output's bytes in memory, after the output renders and before anything is written: conversion of line endings first, then validation, so that a schema judges exactly the bytes that will be written.

A template is best kept with LF line endings, since a checkout that converts them would change the package; an output a platform needs with CR LF gets them from the declaration instead.

MTS-TRN-013 When a template output, a mirror, or a container's template entry declares newline: crlf, a processor MUST convert the bytes of its file after it renders and before it is validated (§10.4), and before a container holding it is assembled: every LF (U+000A) not immediately preceded by CR (U+000D) becomes CR LF, and a CR LF and a lone CR are left as they are, so the conversion is idempotent and a template written with CR LF converts to the same bytes. The conversion applies to the whole file, the bytes a value or a tag placed in it included. Under newline: as-written, the default, a processor MUST write the bytes the template produced.

10.4 Output validation

An output may name a schema its file must satisfy, so that a package whose output feeds another program can promise that the data it writes is valid, and a supplier whose data would make it invalid hears so before anything is written.

MTS-TRN-014 When a template output, or a container's template entry, names a schema with validate, a processor MUST parse the final bytes of each of its files, after §10.3, by its kind (chapter 9): a json file by the rules of a JSON input file and a yaml file by those of a YAML input file (chapter 6), after a single leading byte order mark is removed; and MUST validate the value against the schema by the dialect, keywords, formats and reference rules of chapter 5, with inputs/ read as the top-level directory that holds the validate file. The schema files compiled with it are the validate file and each JSON or YAML file beneath that directory that its references reach by path, transitively; beneath inputs/, every schema file there. A reference into .compiled/ escapes the package's content. A schema that does not compile MUST refuse the render with the findings chapter 5 gives it, and an output whose kind is neither json nor yaml with output.declaration.validate_kind, located at its validate.

MTS-TRN-015 An output whose text does not parse by its kind's rules, or whose value fails its schema, MUST refuse the render, and the processor MUST write nothing. It MUST report every finding of every validated file: a rule of the form the text breaks as output.parse, whose rule names the code an input file breaking the same rule gets; a limit the text breaches as limit.exceeded.output; and each failure of the schema by the output. family of §12.3. Each is about the output, named by its id (an entry without one by the nearest enclosing declaration's), and located with origin output, its file the output's path (for a container's entry, the container's path and the entry's, joined by /), its pointer the JSON Pointer into the output's value, and its line and column where the parser gives them.