BigQuery Metadata Sync Automation
Introduction
BigQuery Metadata Sync is an automation that synchronizes DataHub Tags, Table and Column descriptions, and Column Glossary Terms with BigQuery. This automation is exclusively available in DataHub Cloud (Acryl).
Use-Cases
- Maintain consistent metadata across DataHub and BigQuery
- Improve data discovery by propagating rich descriptions back to BigQuery
- Enhance data governance by applying Policy Tags based on DataHub Glossary Terms
- Streamline data classification by syncing DataHub Tags to BigQuery Labels
- Facilitate compliance efforts by automatically tagging sensitive data columns
- Support data lineage tracking by keeping metadata aligned across platforms
Sync Capabilities
| DataHub Source | BigQuery Target | Sync Direction | Notes |
|---|---|---|---|
| Table Tags | Table Labels | Bi-directional | Changes in either system reflect in both |
| Table Descriptions | Table Descriptions | Bi-directional | Changes in either system reflect in both |
| Column Descriptions | Column Descriptions | Bi-directional | Changes in either system reflect in both. Thes sync doesn't delete table description from BigQuery |
| Column Glossary Terms | Column Policy Tags | DataHub → BigQuery | Created under DataHub taxonomy |
Setup Instructions
1. Verify Permissions
Ensure your service account has the following permissions:
| Task | Required Permissions | Available Role |
|---|---|---|
| Policy Tag Management | • datacatalog.taxonomies.create• datacatalog.taxonomies.update• datacatalog.taxonomies.list• datacatalog.taxonomies.get• bigquery.tables.createTagBinding | Policy Tag Admin |
| Policy Tag Assignment | • bigquery.tables.updateTag | - |
| Description Management | • bigquery.tables.update | - |
| Label Management | • bigquery.tables.update | - |
Note: bigquery.tables permissions must be granted in every project where metadata sync is needed.
2. Enable the Automation
Navigate to Automations: Click on 'Govern' > 'Automations' in the navigation bar.
Create An Automation: Click on 'Create' and select 'BigQuery Tag Propagation'.
Configure Automation:
- Select a Propagation Action
Propagation Type DataHub Entity BigQuery Entity Note Table Tags as Labels Table Tag BigQuery Label - Column Glossary Terms as Policy Tags Glossary Term on Table Column Policy Tag - Assigned Policy tags are created under DataHub taxonomy.
- Only the latest assigned glossary term set as policy tag. BigQuery only supports one assigned policy tag.
- Policy Tags are not synced to DataHub as glossary term from BigQuery.
Table Descriptions Table Description Table Description - Column Descriptions Column Description Column Description - noteYou can limit propagation based on specific Tags and Glossary Terms. If none are selected, ALL Tags or Glossary Terms will be automatically propagated to BigQuery tables and columns. (The recommended approach is to not specify a filter to avoid inconsistent states.)
note- BigQuery supports only one Policy Tag per table field. Consequently, the most recently assigned Glossary Term will be set as the Policy Tag for that field.
- Policy Tags cannot be applied to fields in External tables. Therefore, if a Glossary Term is assigned to a field in an External table, it will not be applied.
- Fill in the required fields to connect to BigQuery, along with the name, description, and category
- Finally, click 'Save and Run' to start the automation
3. Propagating for Existing Assets (Optional)
To ensure that all existing table Tags and Column Glossary Terms are propagated to BigQuery, you can back-fill historical data for existing assets. Note that the initial back-filling process may take some time, depending on the number of BigQuery assets you have.
To do so, follow these steps:
- Navigate to the Automation you created in Step 3 above
- Click the 3-dot "More" menu
- Click "Initialize"
This one-time step will kick off the back-filling process for existing descriptions. If you only want to begin propagating descriptions going forward, you can skip this step.
Viewing Propagated Tags
You can view propagated Tags inside the BigQuery UI to confirm the automation is working as expected.
Troubleshooting BigQuery Propagation
Q: What metadata elements support bi-directional syncing between DataHub and BigQuery?
A: The following metadata elements support bi-directional syncing:
- Tags (via BigQuery Labels): Changes made in either DataHub Table Tags or BigQuery Table Labels will be reflected in the other system.
- Descriptions: Both table and column descriptions are synced bi-directionally.
Q: Are Policy Tags bi-directionally synced?
A: No, BigQuery Policy Tags are only propagated from DataHub to BigQuery, not vice versa. This means that Policy Tags should be mastered in DataHub using the Business Glossary.
It is recommended to avoid enabling extract_policy_tags_from_catalog during
ingestion, as this will ingest policy tags as BigQuery labels. Our sync process
propagates Glossary Term assignments to BigQuery as Policy Tags.
In a future release, we plan to remove this restriction to support full bi-directional syncing.
Q: What metadata is synced from BigQuery to DataHub during ingestion?
A: During ingestion from BigQuery:
- Tags and descriptions from BigQuery will be ingested into DataHub.
- Existing Policy Tags in BigQuery will not overwrite or create Business Glossary Terms in DataHub. It only syncs assigned column Glossary Terms from DataHub to BigQuery.
Q: Where should I manage my Business Glossary?
A: The expectation is that you author and manage the glossary in DataHub. Policy tags in BigQuery should be treated as a reflection of the DataHub glossary, not as the primary source of truth.
Q: Are there any limitations with Policy Tags in BigQuery?
A: Yes, BigQuery only supports one Policy Tag per column. If multiple glossary terms are assigned to a column in DataHub, only the most recently assigned term will be set as the policy tag in BigQuery. To reduce the scope of conflicts, you can set up filters in the BigQuery Metadata Sync to only synchronize terms from a specific area of the Business Glossary.
Q: How frequently are changes synced between DataHub and BigQuery?
A: From DataHub to BigQuery, the sync happens instantly (within a few seconds) when the change occurs in DataHub.
From BigQuery to DataHub, changes are synced when ingestion occurs, and the frequency depends on your custom ingestion schedule. (Visible on the Integrations page)
Q: What happens if there's a conflict between DataHub and BigQuery metadata?
A: In case of conflicts (e.g., a tag is modified in both systems between syncs), the DataHub version will typically take precedence. However, it's best to make changes in one system consistently to avoid potential conflicts.
Q: What permissions are required for bi-directional syncing?
A: Ensure that the service account used for the automation has the necessary permissions in both DataHub and BigQuery to read and write metadata. See the required BigQuery permissions at the top of the page.
Q: Can table description removed?
No, the sync can only modify table description but it won't remove or clear a description from a table.