Synchronize Azure AI Foundry

Synchronizing Azure AI Foundry is the process of integrating metadata from Azure AI Foundry and making the data available in Collibra Platform.

You can synchronize manually, or you can automate it by adding a synchronization schedule.

Before you begin

Required permissions

  • You have the following Azure AI Foundry permissions:
    • Microsoft.MachineLearningServices/workspaces/serverlessEndpoints/read
    • Microsoft.MachineLearningServices/workspaces/onlineEndpoints/read
    • Microsoft.MachineLearningServices/workspaces/onlineEndpoints/deployments/read
    • Microsoft.MachineLearningServices/workspaces/read
    • Microsoft.MachineLearningServices/workspaces/connections/read
    • Microsoft.MachineLearningServices/workspaces/*/read
    • Microsoft.Insights/Metrics/Read
    • Microsoft.Insights/Logs/Read
    • Microsoft.CognitiveServices/accounts/deployments/read
    • Microsoft.CognitiveServices/accounts/models/read
    • Microsoft.CognitiveServices/accounts/OpenAI/assistants/read
    • Microsoft.CognitiveServices/accounts/OpenAI/assistants/files/read
    • Microsoft.CognitiveServices/accounts/OpenAI/assistants/vector_stores/read
    • Microsoft.CognitiveServices/accounts/OpenAI/assistants/vector_stores/files/read
    • Microsoft.CognitiveServices/accounts/OpenAI/files/read
    • Microsoft.CognitiveServices/accounts/OpenAI/fine-tunes/read
    • Microsoft.CognitiveServices/accounts/OpenAI/1p-jobs/read
    • Microsoft.CognitiveServices/accounts/OpenAI/*/read
  • You have a resource role with the Configure external system resource permission, for example, Owner.
  • You have a global role with the Catalog global permission, for example, Catalog Author.
  • You have a global role with the View Edge connections and capabilities global permission, for example, Edge integration engineer. For example, Edge integration engineer.

Steps

  1. On the main toolbar, click Products icon → Catalog.
    The Catalog homepage opens.
  2. On the main toolbar, click Plus icon.
    The Create dialog box appears.
  3. In the Register with Edge section of the Create dialog box, click Integration Configuration.
    The Integration Configuration page opens.
  4. In the Connection name column, locate the Azure connection that you used when you added the Azure AI Foundry capability and click the link in the Capabilities column.
    The Synchronization page opens.
  5. In the Synchronization Configuration section, click Add Configuration.
  6. If you want to refresh the dynamic fields pulled from Azure AI Foundry, click Updated: <timestamp> next to Synchronization Configuration. The timestamp indicates the last time when the data was loaded from Azure AI Foundry.
    Note If the Updated: <timestamp> is green, you don't need to refresh as the data is up to date. If it is red, refresh the data. If you don't see the option, the metadata hasn't been refreshed yet. As metadata is refreshed daily, try checking again later or tomorrow.
  7. In Domain, select the Domain asset in which you want to add the Azure AI Foundry assets.

    Important Ensure that you select a domain of the type Technology Asset.

  8. In Workspace Name (in preview), select at least one or more workspaces in Azure AI Foundry that contains the data you want to ingest in Collibra. The list is prepopulated from your Azure AI Foundry platform and includes all Resource Groups, as well as their hubs and projects. You must add at least one Workspace Name. For more information, go to Azure's resources documentation.
    Important The Workspace Name list shows workspaces available in your Azure AI Foundry platform. It shows the Resource Group first, and then any hubs or projects within that Resource Group. The format of the list items does not follow the exact format found in Azure AI Foundry:
    • "Resource Group name"/: Ingests everything within the Resource Group.
    • "Resource Group name"/"Hub or Project name": Ingests everything in either the Hub, such as all projects, models, and agents or the Project, such as models and agents.
    If this is a first-time integration, Collibra creates a new Project asset. If you are synchronizing the data source, Collibra updates the existing Project asset in Collibra.
    Note 

    Selecting both a Resource Group and the same Resource Group with a hub won't duplicate information.

    If a Resource Group, hub, or project is missing, click Updated: <timestamp> and verify the permissions of the credentials used to configure the Azure AI Foundry capability.

  9. Select the Associate AI Use Cases With AI Models checkbox (in preview) if you want to automatically create a relation between an AI Use Case and the synchronized model when they are both linked to the same AI Project.
  10. Select the Do you want to create file assets? checkbox (in preview) if you want to ingest the dataset folder and files used to fine-tune your foundational AI models and add data to agents. These are ingested as a Storage Container asset and File assets. For more information, go to AI model traceability: automatic linking of AI Governance assets.

  11. Optionally, in Custom AI Metrics Mappings, define which custom Azure AI Foundry model metrics you want to integrate. You do this by adding the mapping between the custom metric and the Collibra attribute. The attribute list contains all attribute types that are assigned to the Azure AI Foundry Model asset type.

    After you synchronize the capability, the specified custom Azure AI Foundry Model metrics are mapped to the corresponding attributes.

    Important 

    If you use this feature, make sure to add any custom attributes/characteristics, as needed, to the asset type assignment.

    1. Click Add Custom AI Metrics Mappings.
    2. In Metric, select the custom metric from the list of available Azure AI Foundry metrics.
    3. In Attribute, select the attribute in which you want to see the value.
      Make sure to select an attribute that is included in the Azure AI Foundry Model asset type assignment.
  12. Click Save.
  13. Click Synchronize.
    A notification indicates the synchronization has started.
  1. On the main toolbar, click Products icon → Catalog.
    The Catalog homepage opens.
  2. On the main toolbar, click Plus icon.
    The Create dialog box appears.
  3. In the Register with Edge section of the Create dialog box, click Integration Configuration.
    The Integration Configuration page opens.
  4. In the Connection name column, locate the Azure connection that you used when you added the Azure AI Foundry capability and click the link in the Capabilities column.
    The Synchronization page opens.
  5. In the Synchronization Configuration section, click Add Configuration.
  6. If you want to refresh the dynamic fields pulled from Azure AI Foundry, click Updated: <timestamp> next to Synchronization Configuration. The timestamp indicates the last time when the data was loaded from Azure AI Foundry.
    Note If the Updated: <timestamp> is green, you don't need to refresh as the data should be up to date. If it is red, you should refresh the data. If you don't see the option, the metadata hasn't been refreshed yet. As metadata is refreshed daily, try checking again later or tomorrow.
  7. In Domain, select the Domain asset in which you want to add the Azure AI Foundry assets.

    Important Ensure that you select a domain of the type Technology Asset.

  8. In Workspace Name (in preview), select at least one or more workspaces in Azure AI Foundry that contains the data you want to ingest in Collibra. The list is prepopulated from your Azure AI Foundry platform and includes all Resource Groups, as well as their hubs and projects. You must add at least one Workspace Name. For more information, go to Azure's resources documentation.
    Important The Workspace Name list shows workspaces available in your Azure AI Foundry platform. It shows the Resource Group first, and then any hubs or projects within that Resource Group. The format of the list items does not follow the exact format found in Azure AI Foundry:
    • "Resource Group name"/: Ingests everything within the Resource Group.
    • "Resource Group name"/"Hub or Project name": Ingests everything in either the Hub, such as all projects, models, and agents or the Project, such as models and agents.
    If this is a first-time integration, Collibra creates a new Project asset. If you are synchronizing the data source, Collibra updates the existing Project asset in Collibra.
    Note 

    Selecting both a Resource Group and the same Resource Group with a hub won't duplicate information.

    If a Resource Group, hub, or project is missing, click Updated: <timestamp> and verify the permissions of the credentials used to configure the Azure AI Foundry capability.

  9. Select the Associate AI Use Cases With AI Models checkbox (in preview) if you want to automatically create a relation between an AI Use Case and the synchronized model when they are both linked to the same AI Project.
  10. Select the Do you want to create file assets? checkbox (in preview) if you want to ingest the dataset folder and files used to fine-tune your deployed AI models. These are ingested as a Storage Container asset and File assets. For more information, go to AI model traceability: automatic linking of AI Governance assets.

  11. Optionally, in Custom AI Metrics Mappings, define which custom Azure AI Foundry model metrics you want to integrate. You do this by adding the mapping between the custom metric and the Collibra attribute. The attribute list contains all attribute types that are assigned to the Azure AI Foundry Model asset type.

    After you synchronize the capability, the specified custom Azure AI Foundry Model metrics are mapped to the corresponding attributes.

    Important 

    If you use this feature, make sure to add any custom attributes/characteristics, as needed, to the asset type assignment.

    1. Click Add Custom AI Metrics Mappings.
    2. In Metric, select the custom metric from the list of available Azure AI Foundry metrics.
    3. In Attribute, select the attribute in which you want to see the value.
      Make sure to select an attribute that is included in the Azure AI Foundry Model asset type assignment.
  12. Click Save.
  13. In the Synchronization Schedule section, click Add schedule.
  14. Enter the required information and click Save:
    FieldDescription
    RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
    Cron

    The Quartz Cron expression that determines when the synchronization takes place.

    This field is only visible if you select Cron expression in the Repeat field.

    Every

    The day on which you want to synchronize, for example, Sunday.

    This field is only visible if you select Weekly in the Repeat field.

    Every first

    The day of the month on which you want to synchronize, for example, Tuesday.

    This field is only visible if you select Monthly in the Repeat field.

    At

    The time at which you want to synchronize automatically, for example, 14:00.

    • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
    • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
    Time zoneThe time zone for the schedule.

What's next

The synchronization job synchronizes the Azure AI Foundry data.
After the synchronization: