Conformance tests: Working Draft 1.x

aliases

/conf/aliases/chain: An alias defined through another alias, without a cycle, is valid.
/conf/aliases/compose: An alias defined through another alias expands to the full path.
/conf/aliases/cycle: Two aliases that refer to each other are invalid.
/conf/aliases/name: An alias name starting with a digit is invalid.
/conf/aliases/name-valid: An alias name of letters, digits, underscores, and hyphens starting with a letter is valid.
/conf/aliases/scoped-form-missing-formats: A scoped alias target without formats is invalid.
/conf/aliases/scoped-form-valid: A scoped alias whose every target has formats and given is valid.
/conf/aliases/scoped-last-target: For an OpenAPI 3.1 document, which has both oas3 and oas3_1, only the last matching target is used.
/conf/aliases/scoped-no-target: A scoped alias with no target for the document's formats matches nothing; only the format warning is reported.
/conf/aliases/scoped-oas2: On an OpenAPI 2 document, the scoped alias uses the oas2 target.
/conf/aliases/scoped-oas3: On an OpenAPI 3 document, the scoped alias uses the oas3 target.
/conf/aliases/simple-form-string: A simple alias given as a bare string is invalid.
/conf/aliases/simple-form-valid: A simple alias that is an array of path expressions is valid.

custom-functions

/conf/custom-functions/declaration-not-array: A functions value that is a string is invalid.
/conf/custom-functions/declaration-valid: functions as an array of strings with a functionsDir string is valid.
/conf/custom-functions/invoke: The function receives the value and the options, and each problem it returns becomes a result with its message.
/conf/custom-functions/javascript-promise: A function that returns a promise is awaited, and the problems it resolves to are reported.
/conf/custom-functions/load-custom-dir: With functionsDir, the function is loaded from that directory.
/conf/custom-functions/load-default-dir: A function named in functions is loaded from functions/<name>.js next to the ruleset.
/conf/custom-functions/missing: A ruleset naming a function whose file does not exist is rejected.
/conf/custom-functions/present: A ruleset naming a function whose file exists is loaded and the function runs.
/conf/custom-functions/problem-path: A problem with a path is reported there; one whose path is not in the document is reported at the root.
/conf/custom-functions/returns-nothing: A function that returns nothing reports no problems, while one that returns a problem does.

document

/conf/document/empty-file: An empty file is the document null, and a rule on $ sees it.
/conf/document/root-array: A document whose root is an array is linted, and $ selects the array.
/conf/document/utf8-keys: A document is read as UTF-8, so non-ASCII keys arrive intact in path.

extends

/conf/extends/base-invalid: A ruleset that extends a ruleset with an invalid given is rejected.
/conf/extends/entry-forms-valid: A string entry and tuples with each of the three modes are valid.
/conf/extends/entry-invalid-mode: An extends tuple with the mode strict is invalid.
/conf/extends/entry-tuple-length: An extends tuple with a third element is valid; elements after the mode are ignored.
/conf/extends/load: A rule defined only in the extended ruleset is available to the extending one by name.
/conf/extends/mode-all: With mode all, both rules run.
/conf/extends/mode-all-extra-element: With [./base.yaml, all, extra], the mode all applies and the extra element is ignored.
/conf/extends/mode-default: Without a mode, only the rule that is recommended runs.
/conf/extends/mode-off: With mode off, no inherited rule runs until it is enabled by name.
/conf/extends/mode-recommended: With mode recommended, only the rule that is recommended runs.
/conf/extends/precedence: The extending ruleset's definition of r wins over both extended rulesets.
/conf/extends/precedence-order: Of two extended rulesets defining the same rule, the later one wins.
/conf/extends/redefine-keeps-properties: A redefinition that sets only given and then keeps the inherited rule's severity and message.
/conf/extends/resolvable: A ruleset extending a file that exists is loaded and the inherited rule runs.
/conf/extends/shape-array: A ruleset whose extends is an array mixing a string and a tuple is valid.
/conf/extends/shape-object: A ruleset whose extends is an object is invalid.
/conf/extends/shape-string: A ruleset whose extends is a single string is valid.
/conf/extends/unresolvable: A ruleset extending a file that does not exist is rejected.

extension-properties

/conf/extension-properties/accepted: A ruleset with extension properties of several types at the root and on a rule is accepted.
/conf/extension-properties/ignored: Extension properties at the root and on a rule change nothing about the results.

formats

/conf/formats/detect-aas2-6: A document with asyncapi: 2.6.0 is aas2_6 and not aas2_5.
/conf/formats/detect-aas2-needs-patch: asyncapi: "2.6" without a patch number is not aas2, while 2.6.0 is.
/conf/formats/detect-arazzo: A document with arazzo: 1.0.1 is arazzo1_0.
/conf/formats/detect-json-schema-2020-12: A $schema of draft 2020-12 is jsonSchemaDraft2020_12 and not draft 7.
/conf/formats/detect-json-schema-loose: A document with type: object and no $schema is jsonSchemaLoose but not jsonSchema.
/conf/formats/detect-number-version: An unquoted openapi: 3.0 is the number 3, whose string form is 3, so the document is oas3 but not oas3_0.
/conf/formats/detect-oas2: A document with swagger: "2.0" is oas2 and not oas3.
/conf/formats/detect-oas3-1: An OpenAPI 3.1 document is detected as oas3_1 and not as oas3_0.
/conf/formats/detection-declared-only: oas3_1 is not detected when no rule names it, so for an OpenAPI 3.1 document the alias uses its oas3 target, though an oas3_1 target comes last.
/conf/formats/rule-overrides-ruleset: A rule's own formats replaces the ruleset's.
/conf/formats/rule-scope: Of two rules scoped to different formats, only the one whose format the document has runs.
/conf/formats/ruleset-default: A rule without formats takes the ruleset's, so it does not run on a document of another format, while a rule scoped to that format does.
/conf/formats/shape-string: A ruleset whose formats is a string is invalid.
/conf/formats/shape-valid: formats as an array of strings on the ruleset, a rule, an override, and an alias target is valid.
/conf/formats/unknown-identifier: A rule whose formats names an identifier outside the table, oas31, is invalid.
/conf/formats/unrecognized-document: A document that has none of the ruleset's formats gets one warning and no rule results.
/conf/formats/unrecognized-unscoped: A document with none of the ruleset's formats gets the warning, the ruleset's own rule does not run, and an unscoped rule from an extended ruleset does.
/conf/formats/unscoped: Without formats anywhere, a rule runs on a document of no known format.

functions

/conf/functions/alphabetical: alphabetical fails on an unsorted array and passes on a sorted one.
/conf/functions/alphabetical-case: Strings are compared with localeCompare, so a comes before B and B after a.
/conf/functions/alphabetical-keyed-by: With keyedBy, an array of objects is compared by that property.
/conf/functions/alphabetical-numbers: Numbers are compared by value, so [2, 10] is in order and [10, 9] is not.
/conf/functions/casing: casing with type: kebab fails on a camel-case string.
/conf/functions/casing-non-ascii: Only ASCII letters count, so éclair is not camel case; the empty string passes.
/conf/functions/casing-types: Each case accepts a value written in it and rejects one written in another case.
/conf/functions/defined: defined fails on an absent property and passes on a present one.
/conf/functions/enumeration: enumeration fails on a value outside values.
/conf/functions/enumeration-types: "1" is not one of [1], while 1.0 is.
/conf/functions/falsy: falsy fails on an empty array, an empty object, and a string, and passes on false, "", 0, and null.
/conf/functions/input-types: pattern passes on a number and on an absent value, and fails only on the string that does not match; xor passes on a string.
/conf/functions/length: length with max: 4 fails on a string of six characters and an array of five elements, and passes on a number below the bound.
/conf/functions/length-code-units: A string's length is its number of UTF-16 code units: é is one, an emoji two.
/conf/functions/length-min: length with min: 2 fails on an object with one property and a string of one character, and passes on an object with two.
/conf/functions/or: or fails unless at least one of properties is present.
/conf/functions/or-null: A key whose value is null counts as present, so {a: null} satisfies or and {} does not.
/conf/functions/pattern: match fails on a string that does not match, notMatch fails on one that does.
/conf/functions/schema: schema fails on a value that does not validate against the given JSON Schema.
/conf/functions/schema-absent: schema fails on an absent value.
/conf/functions/schema-all-errors: With allErrors: true, a value that fails two keywords gets two problems.
/conf/functions/schema-default-draft: Without $schema or dialect, the schema is read as draft 7: prefixItems is ignored, required is not.
/conf/functions/schema-dialect: With dialect: draft2020-12, prefixItems applies.
/conf/functions/schema-first-error: Without allErrors, a value that fails two keywords gets one problem.
/conf/functions/schema-format: The format keyword is asserted, so not-an-email fails format: email.
/conf/functions/schema-ref: A $ref inside the option resolves against the option's own $defs.
/conf/functions/truthy: truthy fails on false, "", 0, and null, and passes on an empty array, an empty object, and a string.
/conf/functions/undefined: undefined fails on a present property and passes on an absent one.
/conf/functions/unreferenced-reusable-object: unreferencedReusableObject reports a member of the container that nothing references.
/conf/functions/xor: xor fails unless exactly one of properties is present.
/conf/functions/xor-null: A key whose value is null counts as present, so {a: null, b: 1} has two of the properties.

given

/conf/given/alias-declared: A rule referencing an alias the ruleset declares is valid.
/conf/given/alias-expansion: #Info.description is evaluated as $..info.description.
/conf/given/alias-redefined-both: A rule that redefines an inherited rule with an alias both rulesets declare is valid.
/conf/given/alias-redefined-own: A rule that redefines an inherited rule with an alias only the extending ruleset declares is invalid.
/conf/given/alias-undefined: A rule referencing an alias the ruleset does not declare is invalid.
/conf/given/bare-property-name: A rule whose given is a bare property name is invalid.
/conf/given/evaluate-all-matches: Two operations without summary produce one result each.
/conf/given/evaluate-multiple-expressions: Two path expressions in given each produce their results.
/conf/given/jsonpath-filter: A filter expression in parentheses, ?(@.in === "header"), selects the header parameter with an empty name; the index in its path is a string.
/conf/given/match: The same rule on a document where given matches produces results.
/conf/given/no-match: A rule whose given matches nothing produces no results, even though its function would fail on any node.
/conf/given/once-per-node: A node matched by three path expressions of one rule is checked once.
/conf/given/path-expression-forms: A JSONPath expression starting with $ and an alias reference starting with # are both valid.
/conf/given/present: A rule with given and then is valid.
/conf/given/required: A rule with then but no given is invalid.
/conf/given/resolved-default: By default given sees the resolved document, where the $ref has been replaced by a schema with type.
/conf/given/resolved-false: With resolved: false, given sees the $ref object, which has no type.
/conf/given/shape-empty-array: A rule with given: [] is invalid.
/conf/given/shape-object: A rule whose given is an object is invalid.
/conf/given/shape-valid: given as a single path expression and as an array of them are both valid.

linter

/conf/linter/accept-valid: A valid ruleset is loaded and linting produces its results.
/conf/linter/reject-invalid: Given an invalid ruleset, the linter rejects it and produces no results.
/conf/linter/walkthrough: The running example of the Linting chapter: a ruleset that extends a base, enables a rule by shorthand, declares an alias, and overrides a severity for legacy documents, linting one legacy document.

message

/conf/message/default-description: Without message, the result carries the rule's description.
/conf/message/override: A rule's message replaces the reported message.
/conf/message/placeholder-description: {{description}} resolves to the rule's description.
/conf/message/placeholder-path: {{path}} is the JSON Pointer to the offending value, with a leading #.
/conf/message/placeholder-property: {{property}} resolves to the key of the offending value.

overrides

/conf/overrides/aliases-existing-rule: An alias an override declares does not change a rule the ruleset defines: the rule keeps the ruleset's alias and only its severity changes.
/conf/overrides/aliases-new-rule: A rule an override defines resolves aliases against the override's, which take precedence over the ruleset's alias of the same name.
/conf/overrides/aliases-outside-globs: Outside the override's globs its aliases have no effect, and the rule keeps its default severity.
/conf/overrides/extends: An override's extends adds rules for the documents it matches.
/conf/overrides/extends-with-rules: An applying override with extends and rules adds to the rulesets the ruleset extends: a, b, and r all run.
/conf/overrides/extends-without-rules: An applying override with extends and no rules replaces the rulesets the ruleset extends: b and the ruleset's own r run, a does not.
/conf/overrides/file-scoped-empty: A file-scoped override with neither extends nor rules is invalid.
/conf/overrides/file-scoped-valid: A file-scoped override with rules, extends, formats, parserOptions, and aliases is valid.
/conf/overrides/mixed-globs: An override whose files mix a pointer glob and a plain glob is file-scoped, so it can define rules with shorthands and is valid.
/conf/overrides/order: Of two overrides that both match, the later one wins.
/conf/overrides/pointer-scope: A pointer-scoped override silences the rule under its pointer only.
/conf/overrides/pointer-scope-at: A pointer that names the result's own location silences it.
/conf/overrides/pointer-scoped-extends: A pointer-scoped override with extends is invalid.
/conf/overrides/pointer-scoped-rule-object: A pointer-scoped override that defines a rule object is invalid.
/conf/overrides/pointer-scoped-valid: A pointer-scoped override whose rules are severities only is valid.
/conf/overrides/redefined-rule-alias-applied: An override that redefines a rule with an alias only the override declares makes a linter reject the ruleset for a document it applies to.
/conf/overrides/redefined-rule-alias-declared: An override that redefines a rule with an alias the ruleset declares is loaded, and the redefined rule runs.
/conf/overrides/scope-match: A document inside the override's globs gets the override's severity.
/conf/overrides/scope-no-match: A document outside the override's globs keeps the base severity.
/conf/overrides/scope-outside-ruleset-directory: The same glob does not match a document in a legacy/ directory that is not next to the ruleset, whatever the working directory.
/conf/overrides/scope-ruleset-directory: A glob is matched against the document's path relative to the ruleset file's directory, so legacy/** matches a document in legacy/ next to the ruleset.
/conf/overrides/shape-empty: An empty overrides array is invalid.
/conf/overrides/shape-no-files: An override without files is invalid.
/conf/overrides/shape-valid: An override with a non-empty files array is valid.
/conf/overrides/unknown-property-applied: An override with a property the Ruleset Object does not have makes a linter reject the ruleset for a document it applies to.
/conf/overrides/unknown-property-not-applied: The same override, applying to no document being linted, does not stop the ruleset from loading.

parser-options

/conf/parser-options/duplicate-keys-default: A duplicate key is reported as an error by default.
/conf/parser-options/duplicate-keys-off: With duplicateKeys: off, a duplicate key is not reported.
/conf/parser-options/duplicate-keys-warn: With duplicateKeys: warn, a duplicate key is reported as a warning.
/conf/parser-options/incompatible-values: A YAML mapping key that is a number, not a string, has no JSON representation and is reported as an error by default.
/conf/parser-options/shape-extension-property: An extension property inside parserOptions is invalid: the permission covers the root and rules only, and parserOptions admits no other properties.
/conf/parser-options/shape-unknown-option: An unknown parser option is invalid.
/conf/parser-options/shape-unknown-severity: duplicateKeys: critical is invalid.
/conf/parser-options/shape-valid: Both parser options with severity values are valid.

parsing

/conf/parsing/several-documents: A YAML file with two documents gets a parser error, and the rules run on the first document.
/conf/parsing/syntax-error: A YAML document with an unclosed flow sequence is reported as a parser error.

resolution

/conf/resolution/cycle: A schema that references itself resolves without error, and a rule sees its type.
/conf/resolution/external: A $ref to another file, relative to the document, is replaced by the referenced schema.
/conf/resolution/local: A $ref to a pointer in the same document is replaced by the referenced schema, whose type the rule then sees.
/conf/resolution/shared-reference: A schema referenced from two operations fails a rule once, at the place where the schema is written.
/conf/resolution/unresolvable: A $ref to a file that does not exist is reported as invalid-ref at severity error.
/conf/resolution/unresolvable-kept: After a $ref fails to resolve, a rule still sees the reference object.

result

/conf/result/code: The code of a result is the rule name.
/conf/result/fields: Every result has the five required fields.
/conf/result/order: Results come in order of position, and by code at the same position, whatever the order of the rules.
/conf/result/range: A result about the root value starts at line 0, character 0.
/conf/result/range-utf16: A character before the value that needs two UTF-16 code units counts as two, and the end is just after the value.
/conf/result/severity: A rule with severity: hint reports severity: 3.
/conf/result/source: A result for a document read from a file carries that file's path in source.

rule

/conf/rule/all-properties: A rule using every property with its documented type is valid.
/conf/rule/recommended-not-boolean: A rule whose recommended is a string is invalid.
/conf/rule/then-present: A rule with then and given is valid.
/conf/rule/then-required: A rule with given but no then is invalid.
/conf/rule/type-unknown: A rule whose type is neither style nor validation is invalid.
/conf/rule/unknown-property: A rule with the unknown property severty is invalid.

rules

/conf/rules/recommended-default: A rule the ruleset defines runs whether recommended is true or absent.
/conf/rules/recommended-false: A rule the ruleset defines with recommended: false does not run, while its other rules do.

ruleset

/conf/ruleset/all-properties: A ruleset using every root property with its documented type is valid.
/conf/ruleset/contributes-empty: A ruleset with only description is invalid.
/conf/ruleset/contributes-extends-only: A ruleset with only extends is valid.
/conf/ruleset/description-not-string: A ruleset whose description is a number is invalid.
/conf/ruleset/document-json: A JSON text whose root is an object is valid.
/conf/ruleset/document-object: A YAML document whose root is an object is valid.
/conf/ruleset/document-root-array: A ruleset whose root value is an array is invalid.
/conf/ruleset/document-single-yaml-document: A YAML stream with one document, with an explicit document marker, is valid.
/conf/ruleset/document-syntax-error: A ruleset that is neither JSON nor YAML, here an unclosed flow sequence, is invalid.
/conf/ruleset/document-two-yaml-documents: A YAML stream with two documents is invalid.
/conf/ruleset/documentation-url-intranet: A ruleset whose documentationUrl names a single-label intranet host, http://wiki/docs, is invalid.
/conf/ruleset/documentation-url-invalid: A ruleset whose documentationUrl is not a URI is invalid.
/conf/ruleset/documentation-url-mailto: A ruleset whose documentationUrl is a mailto: URI is invalid: only http, https, and ftp URLs are accepted.
/conf/ruleset/extension-root-property: A ruleset with the extension property x-owner at the root is valid.
/conf/ruleset/rules-not-object: A ruleset whose rules is an array is invalid.
/conf/ruleset/unknown-root-property: A ruleset with rules and the unknown root property rule is invalid.

severity

/conf/severity/default: A rule without severity reports with the severity of warn.
/conf/severity/numeric-equivalence: A rule with severity: 0 reports with the severity of error.
/conf/severity/off: A rule with severity: off produces no results although it would fail.
/conf/severity/off-control: The same rule at warn produces the results that off suppresses.
/conf/severity/result: A rule inherited at warn and raised to error reports with the severity of error.
/conf/severity/values-invalid: A rule with severity: critical is invalid.
/conf/severity/values-name: A rule with severity: hint is valid.
/conf/severity/values-number: A rule with severity: 3 is valid.

shorthands

/conf/shorthands/false: false disables an enabled rule.
/conf/shorthands/false-control: Without the false shorthand, the inherited rule runs.
/conf/shorthands/inherited: A shorthand for a rule inherited through extends is valid.
/conf/shorthands/not-inherited: A shorthand for a rule the ruleset does not inherit is invalid.
/conf/shorthands/numeric: A numeric severity in the shorthand position is invalid.
/conf/shorthands/override-own-rule: A shorthand in an applying override for a rule the ruleset defines itself sets its severity.
/conf/shorthands/override-unknown: A shorthand in an override for a rule the ruleset neither defines nor inherits makes a linter reject the ruleset for a document the override applies to.
/conf/shorthands/override-unknown-not-applied: A shorthand for an unknown rule in an override that applies to no document being linted does not stop the ruleset from loading.
/conf/shorthands/severity: A severity name enables the rule and sets its severity.
/conf/shorthands/true-enables: true enables a rule that is off by default, at the severity it was defined with.
/conf/shorthands/true-keeps-severity: A rule disabled with false in one ruleset and re-enabled with true in the next keeps the severity it was defined with.
/conf/shorthands/true-off: true on a rule whose severity is off does not enable it.
/conf/shorthands/unknown-name: An unknown name in the shorthand position is invalid.
/conf/shorthands/values-valid: Every severity name and both booleans are valid in the shorthand position.

then

/conf/then/apply-each: Two invocations that both fail on the same node produce two results.
/conf/then/apply-each-problem: One invocation whose function returns two problems produces two results.
/conf/then/field: With field, the function sees the named property of the matched node.
/conf/then/field-jsonpath: A field that is a JSONPath expression applies the function to every value it selects.
/conf/then/field-key: With field: "@key" on a matched object, the function sees each of its property keys; the key that does not match the pattern is reported.
/conf/then/field-missing: With field naming a property the node lacks, the function sees no value, so defined fails.
/conf/then/field-path: A field that is a path of property names reaches a nested value.
/conf/then/function-core: A rule naming a core function is valid.
/conf/then/function-missing: An invocation without function is invalid.
/conf/then/function-options: The max option reaches the length function.
/conf/then/function-unknown: A rule naming a function that is neither core nor declared is invalid.
/conf/then/invocation-form-valid: An invocation with function, field, and functionOptions of the right types is valid.
/conf/then/options-not-object: An invocation whose functionOptions is a string is invalid.
/conf/then/options-on-no-option-function: truthy with options is invalid, because it takes none.
/conf/then/options-required-missing: length with empty options is invalid, because it needs min or max.
/conf/then/options-unknown-value: casing with type: lower, which is not a case the function knows, is invalid.
/conf/then/options-valid: Core functions with options that match their tables are valid.
/conf/then/shape-empty-array: A rule with then: [] is valid.
/conf/then/shape-string: A rule whose then is a string is invalid.
/conf/then/shape-valid: then as a single invocation and as an array of them are both valid.