Cloud Services

Cost & Asset Management

Data transformation template
Published On Oct 18, 2024 - 1:26 PM

Data transformation template

Create your own data template for S3 bucket and GPD files by mapping cost, asset, and utilization attributes to match and generate GPD files.
The Data Transformation Template allows you to create a template that is applied to raw data files to create a Generic Provider Data (GPD) file. GPD format is required to import data into Cost and Asset Management. The template maps attributes or fields in the raw data to GPD file attributes.
To learn more about navigating to the different services from each tenant, refer to Landing page navigation or Kyndryl Bridge Landing page navigation.
A new window displays where you can see a table with a list of templates that you have already created, this table specifies:
  • Provider
    : The specific cloud provider.
  • Type
    : The type of template (cost, asset, or metric).
  • Date
    : The date on which the template was created.
  • Status
    : Template status, valid or error.
This page contains the different templates that can be generated for GPD files:
  • Using transformation template generator for S3 Bucket
  • Transform raw data file into GPD format

Using transformation template generator for S3 Bucket

Upload raw data through S3 bucket to match the CAM portal which then creates Generic Provider Data (GPD) files with the information the user wishes to view.
Users authorized to set up providers and upload Generic Provider Data (GPD) files, can do it via S3 bucket. The files must be in zip format and must contain
JSON formatted Asset, Cost, or Utilization data.
Note: A provider is not needed to do run the transformation, but it is needed for the transformed data to be ingested properly.
The provider is created through an API, for more information on this API, go to: Object Storage APIs.
To map cost, asset, and utilization attributes in a transformation template to validate that the attributes match Generic Provider data (GPD) format, go to: Data transformation template.

Upload raw data to generate Generic Provider Data (GPD) files using S3 bucket

Complete the steps to upload GPD files using S3 bucket:
  1. Go to
    CAM
    , on settings you see the provider already created.
  2. Upload raw data to generate GPD file using S3 bucket. You have to make different API call throughout the process of creating, uploading, and transforming your data. To complete the following APIs call see steps details in Uploading raw data to generate GPF files API

Creating S3 Bucket template

Follow the steps to create a new cost, metric, or asset template, you can create only one type of template per specific provider.
  1. On the Data Transformation Templates page, click
    Create Template
     located at the beginning of the page.
  2. A new window displays, add the required information:
    • Provider
      : You can select the provider from the list or add a new one.
    • Template type
      : Select the type of template from cost, asset, or metric.
    • Upload raw data file
      : Upload a sample of the raw data file. This file must be in JSON format. To learn more details about JSON file guidelines, please visit: JSON File format.
  3. Once the raw data file is uploaded, click Create. A new window displays where you can map raw data fields to attributes in a GPD file. The available GPD attributes depends on the selected template type. On this window, a table with different specific values shows:
    • Type of update
      : This indicates if the data is required or optional.
    • Cost, metric, or asset field
      : The name of your type of template selected.
    • Raw data field
      : In this column, you must enter the raw data that matches the GPD file attribute.
    • Errors
      : This column indicates if there is an error, and the type of error so you can update the values on the raw data field.
    • Status
      : Indicates if the information is correct “updated” or requires updates “failed”.
  4. Once you have added all the information, click
    Save
    .
  5. An alert message pops up confirming that your template was created and successfully validated.
  6. You can continue to upload raw data to a designated S3 bucket for GPD files. The raw data will be combined with the Data Transformation Template to create a Generic Provider Data (GPD) file which will be uploaded into Cost and Asset Management.

Editing S3 Bucket templates

Follow the steps to edit a template you have already created:
  1. On the data transformation template page, a table with the list of existing templates displays, click the template you want to edit.
  2. A new window opens with the map raw data fields, on this page you can quick update data or view transformation template.
    Quick update data tab provides a table with different information
    :
    • Type of update
      : This indicates if the data is required or optional.
    • Cost, metric, or asset field
      : The name of your type of template selected.
    • Raw data field
      : In this column, you must enter the raw data that matches an attribute of a GPD file. Available GPD attributes depend on the template type.
    • Value
      : A static user-defined value that appears in all rows of the GPD file.
    • Status
      : Indicates if the information is correct “updated” or requires updates “failed”.
    You have to update the Raw data field to add data from the drop-down menu which should match the metric field. You can also manually add a value.
    View transformation template tab: On this tab you can compare the raw data against the transformed data.

Deleting S3 Bucket templates

Follow the steps to delete a template you already created:
  1. On the data transformation template page, a table with the list of existing templates displays, click the template you want to eliminate. 
  2. A new window opens, click
    Delete
     located at the beginning of the page.
  3. Confirm that you want to delete the template and click
    Delete
    .

Transform raw data file into GPD format

Transform your raw data file into a GPD format file to generate cost, asset, and metric data:
  1. To transform the raw data into GPD format
    , you have to consume
    POST
     /api/ecam/extensions/flow
    • QueryParams
      : No query parameters
    • Request payload
      : { “vars”: {“providerCode”:” ”}}
    • Response
      : 200 Returns the status and message along with the transformation job id and 500 Internal server error
  2. To list all the files (uploaded and transformed) on the S3 repository
    , consume GET /api/ecam/extensions/data/list
  3. To delete any file on the S3 repository
    , consume
    POST
     /api/ecam/extensions/data/delete
    • Query Parameters
      : List of files to be deleted in the below format (The key and size can be obtained from the List API mentioned above [ {“key” : “ ”, “Size”:” ”}
    • Response
      : 200 Returns the status along with count of files deleted and 500 Internal server error
  4. To download/view any file on the S3 repository
    , consume
    GET
     /api/ecam/extensions/data/download
    • QueryParams
       key = file_path_on_s3
    • Response
      : 200 Returns the file contents and 500 Internal server error.
From the
Custom Provider
 tab, select the provider and click Start Enrichment. CAM recognizes data in a 60-day window. Older data remains in the system, but new data is ingested for only the last 60 days. CAM requires a minimum of 60 days of data for proper utilization readings.

Upload extension file

  1. Check the connection to the tenant
     consume GET /api/ecam/extensions/test.
  2. Create a signed URL, to the AWS S3, you have to upload the extension file and
    consume
    POST
    : /api/ecam/extensions/S3ExtensionRepo.
    • Parameters required:
      Filename
      :
      name_of_the_extensionFile_to_be_uploaded
       and
      ProviderCode
      :
      providerName
    • Response
      : 200 Returns the preSignedUrl to upload the extension file to AWS S3
      500 Internal server error
  3. To upload the compressed extension file you have to consume
    PUT
    : preSignedURL.
    • QueryParams
      : Auto-populated
    • Request payload
      : Select type as Binary and upload the required extension file.
    • Response
      : 200 empty response and 500 Internal server error.
  4. Validate the uploaded extension file and consume
    POST
     /api/ecam/extensions/S3ExtensionRepo/validate.
    • QueryParams
      : Filename = name_of_the_extensionFile_to_be_uploaded and ProviderCode = providerName
    • Response
      : 200 Returns the validation result and 500 Internal server error

Upload raw data file

Once the extension file is validated and successfully uploaded, you have to upload the raw data file indicating the specific assets fields that you wish to view.
  1. To create a signed URL to the AWS S3 and upload the compressed raw data file
    , consume
    POST
     /api/ecam/extensions/S3InputHandler
    • Query parameters:
      Filename: name_of_the_rawData_to_be_uploaded and ProviderCode: providerName
    • Response:
      200: Returns the preSignedUrl to upload the raw data file to AWS S3 and 500: Internal server error.
  2. To upload the raw data file
     and consume
    PUT
     preSignedURL
    • QueryParams
      : Auto-populated
    • Request payload
       – Select type as Binary and upload the required raw data zip file.
    • Response
      : 200 Empty response and 500 Internal server error.
  3. To expand the contents of the compressed raw data file
    , consume
    POST
     /api/ecam/extensions/notify/S3InputHandler
    • QueryParams
      : Filename: name_of_the_rawData_to_be_uploaded and ProviderCode: providerName
    • Response
      : 200: {“results”:”ok”} and 500: Internal server error
Do you have two minutes for a quick survey?
Take Survey