Skip to main content

How to use Catalog Ingestion API to push data into Harness IDP

Introduction​

In this tutorial we will be creating Jira tickets using Workflows and add the information (i.e. ticket numbers) to the corresponding Software Component in the Catalog and display the same in the UI using Additional Info Card. The aim of this tutorial is to help you understand the usage of REST APIs to push information in the Catalog and then use them in different parts of IDP according to your use-cases.

Pre-Requisite​

  1. We assume you have JIRA set-up as your ticket management system, and you have access to the same to create tickets in your corresponding project as well as administrator access to create projects.

Create the Workflow​

Create Jira ticket​

Using RUN Step​

  1. Go to Admin in your IDP
  2. Now select the project where you you want to create the pipeline for the Workflows.
  3. Begin by selecting the Create a Pipeline button followed by adding a name for the pipeline and set up your pipeline as inline.
  4. Now select the Developer Portal Stage and give it a name.
  5. Add a RUN step, name it as create jira ticket and select the Shell as Bash
  6. Now add the following under the Command.
EMAIL_ID="<+pipeline.variables.email-id>"
JIRA_TOKEN="<+pipeline.variables.jiratoken>"
PROJECT_KEY="<+pipeline.variables.projectkey>"
COMPONENT_NAME="<+pipeline.variables.componentname>"
ISSUE_TYPE="<+pipeline.variables.issuetype>"
ISSUE_SUMMARY="<+pipeline.variables.issuesummary>"
ISSUE_CONTENT="<+pipeline.variables.issuecontent>"
LABELS="<+pipeline.variables.labels>"

# Perform the POST request with curl and capture the response
response=$(curl --silent --request POST \
  --url 'https://harness.atlassian.net/rest/api/3/issue' \
  --user "$EMAIL_ID:$JIRA_TOKEN" \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data '{
    "fields": {
        "project": {
            "key": "$PROJECT_KEY"
        },
        "components": {
            "name": "$COMPONENT_NAME"
        },
        "issuetype": {
            "name": "$ISSUE_TYPE"
        },
        "summary": "$ISSUE_SUMMARY",
        "description": {
            "version": 1,
            "type": "doc",
            "content": [
                {
                    "type": "paragraph",
                    "content": [
                        {
                            "type": "text",
                            "text": "$ISSUE_CONTENT"
                        }
                    ]
                }
            ]
        },
        "labels": [
            "$LABELS"
        ]
    }
}')

# Extract the key from the JSON response using jq
issue_key=$(echo "$response" | jq -r '.key')

# Export the issue key as an environment variable
export ISSUE_KEY="$issue_key"

# Print the issue key (optional)
echo "The created issue key is: $ISSUE_KEY"

The above given request body can create a Jira ticket based on project and component and add a label to the same.

We have used few pipeline variables in the body, which will be used to take input from the IDP Workflows and for the users to choose project, add the summary, description for the tickets.

  1. Now under Optional Configuration add the Output Variables as ISSUE_KEY.

  2. Apply the Changes.

  3. Go to Variables on the right side of the page and add the following variables and have the input type as Runtime Input.

    • emailid
    • jiratoken
    • projectkey
    • componentname
    • issuetype
    • issuesummary
    • issuecontent
    • labels
    • usergroup

  4. Apply the changes.

Use Catalog Metadata Ingestion API​

  1. Start by adding another RUN step.
  2. Name it as Ingestion API and select the Shell as Bash
  3. Now add the following under the Command.
curl --location 'https://app.harness.io/gateway/v1/catalog/custom-properties' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <+secrets.getValue('account.TOKEN_ID')>' \
--data '{
    "properties": [
        {
            "field": "metadata.openTicket",
            "filter": {
                "kind": "Component",
                "type": "service",
                "owners": [
                    "<+pipeline.variables.usergroup>"
                ]
            },
            "value_overrides": [
                {
                    "entity_ref": "YOUR_COMPONENT_LINK",
                    "override_value": "<+stage.spec.execution.steps.create_jira_ticket.output.outputVariables.ISSUE_KEY>"
                }
            ],
            "value": "0"
        }
    ]
}'

  1. Under header x-api-key: <+secrets.getValue('account.TOKEN_ID')>, add the token ID for your API key. Get your token ID from your Profile

In the above body the openTicket which got created in JIRA will be added, to kind component and type service owned by the usergroup selected in the Workflows. Under entity_ref add the component link to which you want to add the ticket ID, the unique entity reference could be found using inspect entity for the component in Catalog.

  1. Now Apply the changes.

Create Workflow​

Now we have to create a workflow, which takes the input from the user and triggers the pipeline. Here's the workflow YAML

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: jira-ticket
  title: Create a JIRA Ticket
  description: A template to create a Jira Ticket
  tags:
    - jira
    - management
spec:
  owner: owner@company.com
  type: service
  parameters:
    - title: Service Details
      required:
        - projectkey
        - issuetype
        - jiracomponentname
        - issuesummary
        - issuecontent
        - labels
        - usergroup
      properties:
        emailid:
          title: Email ID used for Atlassain account
          type: string
          description: Use the email ID associated with JIRA
        jiratoken:
          title: Jira Access Tokens
          type: string
          ui:widget: password
          description: Add the API access tokens created under the above mentioned email id 
        projectkey:
          title: Jira Project Key
          type: string
          default: IDP
          description: Your ticket will be created under this project
        issuetype:
          type: string
          title: Add your Jira component type
          default: Story
        jiracomponentname:
          type: string
          title: Add the component for this issue
          desciption: Component for ticket tracing eg., Misc, Backstage Core, Platform etc. 
        issuesummary:
          title: Give a title for your ticket
          type: string
        issuecontent:
          title: Describe the details needed in the ticket
          type: string
        labels:
          type: string
          title: Select a label for the issue
          default: roadmap     
        usergroup:
          title: Choose an Owner for the Service
          type: string
          ui:field: OwnerPicker
          ui:options:
            allowedKinds:
              - Group
        # This field is hidden but needed to authenticate the request to trigger the pipeline
        token:
          title: Harness Token
          type: string
          ui:widget: password
          ui:field: HarnessAuthToken
  steps:
    - id: trigger
      name: Bootstrapping your new service
      action: trigger:harness-custom-pipeline
      input:
        url: "PIPELINE URL"
        inputset:
          emailid: ${{ parameters.emailid }}
          jiratoken: ${{ parameters.jiratoken }}
          projectkey: ${{ parameters.projectkey }}
          issuetype: ${{ parameters.issuetype }}
          componentname: ${{ parameters.jiracomponentname }}
          issuesummary: ${{ parameters.issuesummary }}
          issuecontent: ${{ parameters.issuecontent }}
          labels: ${{ parameters.labels }}
          usergroup: ${{ parameters.usergroup }}
        apikey: ${{ parameters.token }}

  output:
    links:
      - title: Pipeline Details
        url: ${{ steps.trigger.output.PipelineUrl }}

In the above YAML just replace the url with the pipeline URL we created above, also make sure the key values under inputset exactly matches with the pipeline variable names.

Also for Jira token input the input should be the personal access token from JIRA.

  1. Now go to your git provider and add this workflow yaml and save it, make sure it's public incase it's in private repo make sure you have the git integration setup.

  2. Once the file is created in your git repo, copy the full URL to the file. For example, https://github.com/harness-community/idp-samples/blob/main/tutorial-jira-ticket-catalog-ingestion.yaml.

  1. In the left navigation, select Create, and then select Register Software Component.

  1. Enter the URL to your new workflow.yaml.

  1. Click Import.

  1. Now go to the workflow and select the workflow you just added.

  1. Add the input values in the field that would be used by the workflow

Create Additional Info Card​

In case you want to display the same information you have ingested on your Overview page as an additional card, follow the steps below.

  1. Go to the Layout Page and under Admin and add the following for Services and Save it.
- component: EntityAdditionalInfoCard
    specs:
    props:
        title: Open JIRA Tickets
        items:
        - label: JIRA TICKET KEY
            value: <+metadata.openTicket>
            type: string
            style:
            bold: true
    gridProps:
        md: 6

navigation-layout

  1. Now go to the Software Component in the Catalog and you'll find an additional info card populated with information we ingested using the API above. You can read more about additional info card