Automatically set up airflow variable defaults with descriptions.

catalog/variables.tsv

@@ -2,4 +2,3 @@ key	default	description
 SILENCED_SLACK_NOTIFICATIONS	{}	Configuration for a silencing Slack notifications from a DAG. Mapping of DAG ID to a list of dictionaries containing the following keys: "issue" (a link to a GitHub issue which describes why the notification is silenced and tracks resolving the problem), "predicate" (Slack notifications whose text or username contain the predicate will be silenced, matching is case-insensitive), and "task_id_pattern" (a regex pattern that matches the task_id of the task that triggered the notification, optional). Declaration: https://github.com/WordPress/openverse/blob/d500b7764c411f7d228ae12c57dce519c8709610/catalog/dags/common/slack.py#L72-L86 Example: {     "finnish_museums_workflow": [         {             "issue": "https://github.com/WordPress/openverse/issues/1605",             "predicate": "AirflowTaskTimeout",             "task_id_pattern": "clean_data"         }     ] }
 SKIPPED_INGESTION_ERRORS	{}	Configuration for silencing an ingestion error and preventing a Slack message from being sent. Mapping of DAG ID to a list of dictionaries containing the following keys: "issue" (a link to a GitHub issue which describes why the error is silenced and tracks resolving the problem), and "predicate" (errors whose classname or message contain the predicate will be skipped, matching is case-insensitive). Declaration: https://github.com/WordPress/openverse/blob/6636dcfbb57abca19ef32027975f78548e10411f/catalog/dags/providers/provider_api_scripts/provider_data_ingester.py#L53-L64 Example: {     "science_museum_workflow": [         {             "issue": "https://github.com/WordPress/openverse/issues/4013",             "predicate": "Service unavailable for url"         }     ] }
 CONFIGURATION_OVERRIDES	{}	DAG configuration overrides for the provider ingestion workflows. Currently only supports overriding the execution timeout for certain tasks, but allows dynamic overrides at DAG run time. Mapping of DAG ID to a list of dictionaries containing the following keys: "task_id" (a regex pattern that matches the task_id of the task to be modified), and "timeout" (str in "%d:%H:%M:%S" format giving the amount of time the task may take, example: 6d:10h:30m). Declaration: https://github.com/WordPress/openverse/blob/2cffcb9f8da6961e84a00854a3cd472fd0f9dad8/catalog/dags/providers/provider_workflows.py#L42-L58 Example: {     "brooklyn_museum_workflow": [         {             "task_id_pattern": "pull_image_data",             "timeout": "10h"         }     ] }
-TESTING_CONFIGURATION_OVERRIDES	{}	DAG configuration overrides for the provider ingestion workflows. Currently only supports overriding the execution timeout for certain tasks, but allows dynamic overrides at DAG run time. Mapping of DAG ID to a list of dictionaries containing the following keys: "task_id" (a regex pattern that matches the task_id of the task to be modified), and "timeout" (str in "%d:%H:%M:%S" format giving the amount of time the task may take, example: 6d:10h:30m). Declaration: https://github.com/WordPress/openverse/blob/2cffcb9f8da6961e84a00854a3cd472fd0f9dad8/catalog/dags/providers/provider_workflows.py#L42-L58 Example: {     "brooklyn_museum_workflow": [         {             "task_id_pattern": "pull_image_data",             "timeout": "10h"         }     ] }