Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up airflow variable defaults with descriptions automatically #4297

Prev Previous commit
Update comments
  • Loading branch information
obulat committed May 16, 2024
commit f9b0a5dac3ac23542930b29eb14576d188435974
10 changes: 5 additions & 5 deletions catalog/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ if [[ $* == "webserver" ]]; then
output=$(airflow variables list -o plain | tail -n +2)
found_existing_vars=true

# if there are no existing variable, print this notification and continue
# If there are no existing variables, print this notification and continue
if [[ -z $output || $output == "No data found" ]]; then
echo "No existing variables found, proceeding to set all variables"
found_existing_vars=false
Expand All @@ -87,15 +87,15 @@ if [[ $* == "webserver" ]]; then
done <<<"$output"

if $found_existing_vars; then
echo -e "Found the following existing variables(the values of these will not be overwritten):\n"
echo -e "Found the following existing variables (the values of these will not be overwritten):\n"
for variable in "${existing_variables[@]}"; do
echo "$variable"
done
fi

# now iterate through each row of variables.tsv and and only
# Now, iterate through each row of variables.tsv and and only
# run airflow variables set --description <description> <key> <value>
# if the key doesn't already exist in the database i.e not found in
# if the key doesn't already exist in the database, i.e it is not found in
# $existing_variables
while IFS=$'\t' read -r column1 column2 column3; do
# skip the first meta row or a row with empty data
Expand All @@ -121,7 +121,7 @@ if [[ $* == "webserver" ]]; then
echo -e "The following variables are now set:\n"
echo "$new_varibles_list"

# if the last line in variables.tsv did not correctly terminate
# If the last line in variables.tsv did not correctly terminate
# with a new line character then this variable would not be empty
# and this means the last line would not be read correctly.
if [ -n "$column1" ]; then
Expand Down