Integrating DataHub into Jira: A Practical Guide Using DataHub Actions

, we’ll go over how to integrate DataHub events into Jira workflows using the DataHub Actions framework. Before diving in, we’ll give a little background into what DataHub is and how to use its Actions framework for effective data management. Finally, we’ll walk through a specific example of writing a custom action for creating a Jira ticket upon creating a data product in DataHub.

Hopefully, this article can serve as a general template for how to integrate DataHub events of interest into your specific Jira processes.

Contents

What is DataHub?

DataHub is a data catalog that supports data discovery, governance, and metadata management. It provides features that allow organizations to implement their own data mesh — a decentralized method of data management that empowers individual business domains to take initiative over their own data quality/requirements.

A metadata platform such as DataHub is incredibly valuable for various reasons:

• Cross-domain data discovery and analysis — data consumers (ex: data analysts/scientists) may use DataHub to explore relevant datasets that they can use for their analyses. To help data consumers make sense of data across various domains, each domain needs to enrich their data assets with sufficient business context.
•  — DataHub allows organizations to establish a single source of truth for their data, implement strong security by enforcing access policies on data assets, and ensure regulatory compliance by supporting data classification.

• DataHub provides the Actions Framework to help integrate metadata changes that occur in DataHub into a larger event-based architecture.
• The framework allows you to specify configuration to trigger certain actions depending on events that occur in DataHub.
• Common use cases may include notifying relevant parties upon changes in a dataset, integrating metadata changes into organizational workflows, etc.

One of the common use cases of the DataHub Actions framework is to integrate metadata changes into organization-specific notifications. DataHub provides native support for this into certain 3rd party platforms, such as Slack and Teams.

In this article, we’ll look at how to use the Actions framework to integrate DataHub metadata changes into Jira workflows. Specifically, we’ll implement a custom action that creates a Jira ticket upon creating a new data product. However, the configuration and code here may be easily altered to trigger Jira workflows for other DataHub events.

Developing a Custom Jira Action

Each DataHub Action is run in a separate pipeline, which is a continuously running process that repeats the following steps: poll event data from relevant sources, apply transformation/filtering to those events, and execute the desired action.

• We can define the configuration for our action pipeline via YAML.
• We can define the logic to execute for our custom action by extending DataHub’s base Action class.

Let’s walk through both of these steps.

YAML configuration

Our YAML configuration requires specifying 4 key aspects:

1. Action Pipeline Name (should be unique and static)
2. Event Source Configurations
3. Transform + Filter Configurations
4. Action Configuration

There are two possible event sources for DataHub Actions:

Kafka is the framework’s default event source. Unless you’re on an instance of DataHub Cloud above v0.3.7, you’ll be processing event data from Kakfa.

The Kafka event source emits two types of events:

Since we are listening for the creation of data products, and the data structure of the Entity Change Event is simpler to work with, we will filter for Entity Change Events.

Here is what our YAML file will look like (jira_action.yaml).

# jira_action.yaml

# 1. Action Pipeline Name
# This may be whatever you like as long as it's unique.
name: "jira_action"

# 2. Event Source - Where to source event data from.
# Kafka is the default event source (https://docs.datahub.com/docs/actions/sources/kafka-event-source).
source:
  type: "kafka"
  config:
    connection:
      bootstrap: ${KAFKA_BOOTSTRAP_SERVER:-localhost:9092}
      schema_registry_url: ${SCHEMA_REGISTRY_URL:-http://localhost:8081} 

# 3. Filter - Filter events that reach the Action
# We'll listen for Data Product CREATE events.
# For more information about other events: https://docs.datahub.com/docs/actions/events/entity-change-event#entity-create-event
filter:
  event_type: "EntityChangeEvent_v1"
  event:
    entityType: "dataProduct"
    category: "LIFECYCLE"
    operation: "CREATE"

# 4. Action - What action to take on events.
# Here, we'll reference our custom Action file (jira_action.py).
action:
  type: "jira_action:JiraAction"
  config:
    # Action-specific configs (map)

Defining our Custom Action Class

Next, we need to implement our logic for creating a Jira ticket in a custom Action class, which we’ll define in a separate jira_action.py file.

Our action class will extend DataHub’s base action class, which consists of overriding the following methods:

• create() — invoked when instantiating the action. If you specified any action-specific config in your YAML file, this method will pass that config as a dictionary to all instances of this action.
• act() — invoked when an event is received. This method will contain the core logic of our action i.e. creating the Jira ticket.
• close() — invoked when our action pipeline is shutdown.

Since we didn’t specify any action-specific config, and we don’t have to worry about any special cleanup once our action is terminated, our work will mainly consist of overriding act().

We will use Python Jira, a python wrapper around the Jira REST API, to programmatically interact with our Jira instance. For more information/examples for how to programmatically interact with Jira, check out the docs.

Here is what our code will look like.

# jira_action.py

from datahub_actions.action.action import Action
from datahub_actions.event.event_envelope import EventEnvelope
from datahub_actions.event.event import Event
from datahub_actions.pipeline.pipeline_context import PipelineContext
from jira import JIRA

class JiraAction(Action):
    @classmethod
    def create(cls, config_dict: dict, ctx: PipelineContext) -> "Action":
        """
        Shares any action-specific config across all instances of the action.
        """
        return cls(ctx, config_dict)
    def __init__(self, ctx: PipelineContext, config_dict: dict):
        self.ctx = ctx
        self.config = config_dict
        
    def act(self, event: EventEnvelope) -> None:
        """
        Create a Jira ticket when a data product is created in DataHub with its DataHub link.
        We'll use the Python Jira API to programmatically interact with Jira (https://jira.readthedocs.io/index.html).
        """
        event_object = event.event
        entity_urn = event_object.entityUrn
        # Extract DataHub link for data product
        DATAHUB_DOMAIN = "http://localhost:9002/" # replace with link to your DataHub instance
        data_product_link = f"{DATAHUB_DOMAIN}{entity_urn}"
        
        # Authenticate into your Jira instance (https://jira.readthedocs.io/examples.html#authentication).
        jira = JIRA(
            token_auth="API token",  # Self-Hosted Jira (e.g. Server): the PAT token
            # basic_auth=("email", "API token"),  # Jira Cloud: a username/token tuple
            # basic_auth=("admin", "admin"),  # a username/password tuple [Not recommended]
            # auth=("admin", "admin"),  # a username/password tuple for cookie auth [Not recommended]
        )
        # Create Jira issue (https://jira.readthedocs.io/examples.html#issues).
        # For more info about the attributes you can specify in a Jira Issue,
        # check out the Issue class (https://github.com/pycontribs/jira/blob/main/jira/resources.py)
        issue_dict = {
            'project': {}, # JIRA project to create issue under in dict form (ex: {'id': 123})
            'summary': 'New Data Product',
            'description': f'Data Product Link: {data_product_link}',
            'issuetype': {}, # define issue type in dict form (ex: {'name': 'Bug'})
            'reporter': '',
            'assignee': ''
        }
        jira.create_issue(fields=issue_dict)
    def close(self) -> None:
        """
        Cleanup any processes happening inside the Action.
        """
        pass

Although the configuration & code here is specific for data product create events, these can be altered to integrate other DataHub events into Jira workflows, such as adding/removing tags, terms, domains, owners, etc.

• You can find the list of different Entity Change Events here.
• Listening for these events would involve changing the Filter configuration in our YAML to the field values of the specific Entity Change Event.

For instance, to create a Jira ticket for an Add Tag Event on a Dataset, we would update our Filter configuration in our YAML as follows:

filter:
  event_type: "EntityChangeEvent_v1"
  event:
    entityType: "dataset"
    category: "TAG"
    operation: "ADD"

Running our Action

Now that we’ve created our action configuration and implementation, we can run this action by placing these two files (jira_action.yaml and jira_action.py) in the same python runtime environment as our DataHub instance.

Then, we can run our action via CLI using the following command:

datahub actions -c jira_action.yaml

For more information on developing/running a custom action, check out the docs.

Wrap-up

Thanks for reading! To briefly recap about what we talked about:

• DataHub is a data catalog that facilitates efficient data discovery, management, and governance.
• DataHub provides its own Actions Framework to integrate metadata changes into organizational workflows in real time.
• Using the framework, we can write our own action to integrate DataHub events into Jira workflows by simply defining the action pipeline in YAML and the implementation logic in a custom python class.

If you have any other ideas/experiences with using DataHub Actions to implement real-time data governance, I’d love to hear it in the comments!

Sources

DataHub Actions:

Python Jira: