Add defaults to check_row_counts_per_partition to simplify schema files

[krivard]

Overview

The custom dbt test check_row_counts_per_partition has two arguments:

• table_name
• partition_expr

The table name can be computed from the default model argument for any instance of the test run against a parquet file (currently all instances).

The partition expression is blank in ~70 instances.

If we add defaults to these arguments, we can convert the check_row_counts_per_partition entry to a single line in ~70 schema files, and reduce duplication in the remaining couple hundred.

Example:

before:

        data_tests:
          - expect_columns_not_all_null
          - check_row_counts_per_partition:
              arguments:
                table_name: core_eia__codes_balancing_authorities
                partition_expr:
        columns:
          - name: code
          - name: label

after:

        data_tests:
          - expect_columns_not_all_null
          - check_row_counts_per_partition
        columns:
          - name: code
          - name: label

Success Criteria

How will we know that we're done?

• check_row_counts_per_partition no longer requires the table_name argument unless used to test something other than a parquet file
• check_row_counts_per_partition no longer requires the partition_expr if the expression would be blank
• No schema file lists the table_name argument unless testing something other than a parquet file
• No schema file lists the partition_expr argument unless the expression is non-empty

Next steps

• draft changes to check_row_counts_per_partition
• write a script to apply the changes to all the schema files
• one of:
  • acquire the will to git add -p only the changes to the rowcounts spec and not the other changes resulting from re-serializing each yaml file (4-600 little yes/no/split decisions)
  • write a different script that doesn't require re-serializing all the yaml files

[krivard]

Script that works but re-serializes all the yaml:
fixit.py.zip

Undesirable changes during re-serialization include:

Changing block descriptions to inline:

-              description: >
-                The can_switch_when_operating column is only reported in a single year,
-                but assumed to be static (and thus is included in the entity table).
-                However, if the one year in which it was reported isn't processed, then
-                it ends up being entirely null, so it is excluded from the null check.
+              description: 'The can_switch_when_operating column is only reported in a single year, but assumed to be static (and thus is included in the entity table). However, if the one year in which it was reported isn''t processed, then it ends up being entirely null, so it is excluded from the null check.'

Changes to significant figures:

-                    min_fraction: 0.00
-                    max_fraction: 0.10
+                    min_fraction: 0.0
+                    max_fraction: 0.1

Changing lists from inline to block:

-                    value_set: ["simulated"]
+                    value_set:
+                      - simulated

Removing explicit quoting:

-                row_condition: "demand_imputed_pudl_mwh_imputation_code IS NULL"
+                row_condition: demand_imputed_pudl_mwh_imputation_code IS NULL

[jdangerx]

We don't have concrete plans to actively work on this, but if this bugs you and you have a couple hours to throw at it, go for it @krivard ! Seems like some of those serialization problems could be fixed by some configuration or running the yaml through another linter?