Mappings

Mappings are the conversion of a dictionary to a flattened table structure. The mapping class allows you to perform operations to view and alter the settings for a single event type.

Mappings.__init__(api, event_name, preview_full=True, preview_changes=True, apply_changes=False, pprint_indent=2, pprint_width=250, pprint_depth=5)

View and change Alooma mappings. Mappings are the conversion of a dictionary to a flattened table structure. The class is initiated with the following variables:

Parameters:
  • api – The Alooma API client authentication
  • event_name – The name of the event to view or change settings for
  • preview_full – Prints the mapping or mapping changes if True. The default is True.
  • preview_changes – Prints the changes in the mapping by category if True
  • apply_changes – Executes the mapping changes if True
  • pprint_indent – The indent value to pprint dictionaries
  • pprint_width – The width value to pprint dictionaries
  • pprint_depth – The depth value to pprint dictionaries

Get Mapping For Event

managealooma.mappings.Mappings.get_mapping_for_event()

Gets the mapping for the event name that the class was initialized with

Returns:a dictionary with the mapping

Get Mapping For Event

Mappings.view_mapping(view_field_mappings=False)

Gets the mapping for an event and allows printing with or with the field details

Parameters:view_field_mappings – Hides the field details when set to false. Useful for quick view of consolidation and mapping details without all the fields
Returns:Returns the mapping dictionary the user viewed

Sample Mapping printed with view_field_mappings=False. The mapping[‘fields’] key with the list of fields IS NOT printed.

{'autoMappingError': None,
 'consolidation': {'consolidatedSchema': 'MY_SCHEMA',
                  'consolidatedTableName': 'MY_TABLE',
                  'consolidationKeys': ['ID'],
                  'viewSchema': None},
 'inputObjects': {'12345-asdfg': ['98765-zxcvb']},
 'mapping': {'isDiscarded': False,
             'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}',
             'outputId': 'a1s2d3-f4g5h6',
             'readOnly': False,
             'schema': 'MY_SCHEMA',
             'tableName': 'MY_TABLE_LOG'},
  'mappingMode': 'AUTO_MAP',
  'name': 'MY_SCHEMA.MY_TABLE',
  'origInputLabel': 'production_database',
  'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table',
              'schema?id=d=12345-asdfg&sschema_object=deleted_rows'],
  'state': 'MAPPED',
  'usingDefaultMappingMode': False}

Sample Mapping printed with view_field_mappings=True. The mapping[‘fields’] key with the list of fields IS printed.

{'autoMappingError': None,
 'consolidation': {'consolidatedSchema': 'MY_SCHEMA',
                  'consolidatedTableName': 'MY_TABLE',
                  'consolidationKeys': ['ID'],
                  'viewSchema': None},
  'fields': [ {'fieldName': 'id',
              'fields': [],
              'mapping': {'columnName': 'ID',
              'columnType': {'nonNull': True,
                              'precision': 38,
                              'scale': 0,
                              'type': 'NUMERIC'},
                              'isDiscarded': False,
                              'machineGenerated': False,
                              'subFields': None}},
              {'fieldName': 'name',
              'fields': [],
              'mapping': {'columnName': 'NAME',
              'columnType': {'length': 16777216,
                              'nonNull': False,
                              'truncate': False,
                              'type': 'VARCHAR'},
              'isDiscarded': False,
              'machineGenerated': False,
              'subFields': None}}
              ],
 'inputObjects': {'12345-asdfg': ['98765-zxcvb']},
 'mapping': {'isDiscarded': False,
             'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}',
             'outputId': 'a1s2d3-f4g5h6',
             'readOnly': False,
             'schema': 'MY_SCHEMA',
             'tableName': 'MY_TABLE_LOG'},
  'mappingMode': 'AUTO_MAP',
  'name': 'MY_SCHEMA.MY_TABLE',
  'origInputLabel': 'production_database',
  'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table',
              'schema?id=d=12345-asdfg&sschema_object=deleted_rows'],
  'state': 'MAPPED',
  'usingDefaultMappingMode': False}

Change Mapping Mode

managealooma.mappings.Mappings.change_mapping_mode(self, new_mapping_mode='STRICT')

Change the mapping mode. Alooma has 3 modes of AUTO_MAP, STRICT, and FLEXIBLE. We only use AUTO_MAP or STRICT

Parameters:new_mapping_mode – The new mapping mode to set: AUTO_MAP, STRICT, and FLEXIBLE
Returns:The altered mapping

The mapping settings keys and sample values

{'autoMappingError': None,
'mappingMode': None,
'usingDefaultMappingMode': True}

The mapping mode with the specified new mappingMode to alter

{'autoMappingError': None,
'mappingMode': 'AUTO_MAP',
'usingDefaultMappingMode': True}

Change Mapping Consolidation Settings

managealooma.mappings.Mappings.change_mapping_consolidation_settings(self, consolidation_schema, consolidation_table_name, consolidation_keys)

Updates the consolidation information for an event

Parameters:
  • consolidation_schema – The schema of consolidated table
  • consolidation_table_name – The name of the consolidated table
  • consolidation_keys – The consolidation keys (primary key) for the table. Takes a single field or a list
Returns:

The altered mapping

The old consolidation settings

{'consolidatedSchema': 'MY_SCHEMA_TEMP',
'consolidatedTableName': 'NY_TABLE_ITEMS',
'consolidationKeys': ['IDENTIFIER'],
'viewSchema': None}

The new consolidation settings to apply

{'consolidatedSchema': 'MY_SCHEMA',
'consolidatedTableName': 'MY_TABLE',
'consolidationKeys': ['ID'],
'viewSchema': None}

Change Mapping Consolidation Key

Mappings.change_mapping_consolidation_key(new_consolidation_key)

Change the conolidation key only

Parameters:new_consolidation_key – The new consolidation key. This is the primary key for the table.
Returns:The altered mapping

The consolidation key is set to the field name ID when tables are auto-mapped

{'consolidatedSchema': 'MY_SCHEMA',
'consolidatedTableName': 'MY_TABLE',
'consolidationKeys': ['ID'],
'viewSchema': None}

This example will change the key to a field named IDENTIFIER

{'consolidatedSchema': 'MY_SCHEMA',
'consolidatedTableName': 'MY_TABLE',
'consolidationKeys': ['IDENTIFIER'],
'viewSchema': None}

Change Mapping to Use Log

Mappings.change_mapping_to_use_log()

Changes the mapping from using the consolidated table to the log table. Used when adjusting manual mappings

Parameters:event_name – The event name of the mapping to alter
Returns:The altered mapping

Events without a _log do not have consolidated table information

'mapping': {'isDiscarded': False,
 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}',
 'outputId': 'a1s2d3-f4g5h6',
 'readOnly': False,
 'schema': 'MY_SCHEMA',
 'tableName': 'MY_TABLE'}

This example will change the key to a field named IDENTIFIER

'mapping': {'isDiscarded': False,
  'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}',
  'outputId': 'a1s2d3-f4g5h6',
  'readOnly': False,
  'schema': 'MY_SCHEMA',
  'tableName': 'MY_TABLE_LOG'}

Change Mapping for Manual Consolidation Creation

Mappings.change_mapping_for_manual_consolidation_creation(consolidation_schema, consolidation_table_name, consolidation_keys, case='UPPER')

Updates the mapping after creating the log table manually

Parameters:
  • consolidation_schema – The schema of consolidated table
  • consolidation_table_name – The name of the consolidated table
  • consolidation_keys – The consolidation keys (primary key) for the table. Takes a single field or a list
  • case – UPPER if your events are MY_SCHEMA.MY_EVENT and LOWER if the events are my_schema_my_event
Returns:

The altered mapping

First adjust the tables in the data warehouse.

ALTER TABLE MY_TABLE RENAME TO MY_TABLE_LOG;
CREATE TABLE MY_TABLE LIKE MY_TABLE_LOG;

Then run change_mapping_for_manual_consolidation_creation to change the mapping to insert data into the new log table. This is the old mapping:

 'consolidation': {'consolidatedSchema': None,
                 'consolidatedTableName': None,
                 'consolidationKeys': ,
                 'viewSchema': None},
'mapping': {'isDiscarded': False,
            'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}',
            'outputId': 'a1s2d3-f4g5h6',
            'readOnly': False,
            'schema': 'MY_SCHEMA',
            'tableName': 'MY_TABLE'},

These are the changes that will be applied with the new mapping

 'consolidation': {'consolidatedSchema': 'MY_SCHEMA',
                 'consolidatedTableName': 'MY_TABLE',
                 'consolidationKeys': ['ID_FIELD'],
                 'viewSchema': None},
'mapping': {'isDiscarded': False,
            'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}',
            'outputId': 'a1s2d3-f4g5h6',
            'readOnly': False,
            'schema': 'MY_SCHEMA',
            'tableName': 'MY_TABLE_LOG'},

Lastly add consolidation queries to combine the new data with the existing data using create_consolidation()

Delete Field From Mapping

Mappings.delete_field_from_mapping(field_name)

Gets a mapping, deletes a field from the mapping, and resets the mapping table statistics.

Parameters:
  • event_name – the name of event for which to make the change
  • field_name – the field name that should be deleted
Returns:

The altered mapping

If print_changes is specified the details for the field to remove is printed

{'fieldName': 'name',
             'fields': [],
             'mapping': {'columnName': 'NAME',
             'columnType': {'length': 16777216,
                             'nonNull': False,
                             'truncate': False,
                             'type': 'VARCHAR'},
             'isDiscarded': False,
             'machineGenerated': False,
             'subFields': None}}

Then the entire new mapping is printed. The ‘name’ field is not longer in the mapping

{'autoMappingError': None,
 'consolidation': {'consolidatedSchema': 'MY_SCHEMA',
                  'consolidatedTableName': 'MY_TABLE',
                  'consolidationKeys': ['ID'],
                  'viewSchema': None},
  'fields': [ {'fieldName': 'id',
              'fields': [],
              'mapping': {'columnName': 'ID',
              'columnType': {'nonNull': True,
                              'precision': 38,
                              'scale': 0,
                              'type': 'NUMERIC'},
                              'isDiscarded': False,
                              'machineGenerated': False,
                              'subFields': None}}
              ]
 'inputObjects': {'12345-asdfg': ['98765-zxcvb']},
 'mapping': {'isDiscarded': False,
             'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}',
             'outputId': 'a1s2d3-f4g5h6',
             'readOnly': False,
             'schema': 'MY_SCHEMA',
             'tableName': 'MY_TABLE_LOG'},
  'mappingMode': 'AUTO_MAP',
  'name': 'MY_SCHEMA.MY_TABLE',
  'origInputLabel': 'production_database',
  'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table',
              'schema?id=d=12345-asdfg&sschema_object=deleted_rows'],
  'state': 'MAPPED',
  'usingDefaultMappingMode': False}

Change Field Mapping Settings

Mappings.change_field_mapping_settings(field_name, new_data_type, truncate=False, non_null=False)

Updates a single filed in a mapping

Parameters:
  • field_name – The field name to alter
  • new_data_type – The new datatype for the mapping such as VARCHAR(1024) or INT
  • truncate – Set to True if the event should be truncated when it’s longer than the specific mapping length. Redshift’s max VARCHAR is 65535 and Snowflake’s max VARCHAR is 16777216.
  • non_null – Set to true if the field is needs to be not null
Returns:

The altered mapping

The current mapping field settings

{'fieldName': 'name',
             'fields': [],
             'mapping': {'columnName': 'NAME',
             'columnType': {'INT':
                             'nonNull': False,
                             'truncate': False,
                             'type': 'VARCHAR'},
             'isDiscarded': False,
             'machineGenerated': False,
             'subFields': None}}

The new mapping field settings to apply

{'fieldName': 'name',
             'fields': [],
             'mapping': {'columnName': 'NAME',
             'columnType': {'length': 1024,
                             'nonNull': False,
                             'truncate': True,
                             'type': 'VARCHAR'},
             'isDiscarded': False,
             'machineGenerated': False,
             'subFields': None}}

Change Field Varchar Length

Mappings.change_field_varchar_length(field_name, new_length)

Updates only the length of a varchar for a field.

Parameters:
  • field_name – The field name to alter
  • new_length – The new length for the varchar
Returns:

None

{'fieldName': 'name',
             'fields': [],
             'mapping': {'columnName': 'NAME',
             'columnType': {'length': 1024,
                             'nonNull': False,
                             'truncate': False,
                             'type': 'VARCHAR'},
             'isDiscarded': False,
             'machineGenerated': False,
             'subFields': None}}
{'fieldName': 'name',
             'fields': [],
             'mapping': {'columnName': 'NAME',
             'columnType': {'length': 16777216,
                             'nonNull': False,
                             'truncate': True,
                             'type': 'VARCHAR'},
             'isDiscarded': False,
             'machineGenerated': False,
             'subFields': None}}

Change Field Null Constraint

Mappings.change_field_null_constraint(field_name, nonnull=False)

Removes the NULL constraint from a column

Parameters:
  • field_name – The column for which to remove the constraint
  • nonnull – The new null setting for the field
Returns:

The altered mapping

The field mapping details with the current nonnull setting

{'fieldName': 'id',
'fields': [],
'mapping': {'columnName': 'ID',
           'columnType': {'nonNull': True,
                           'precision': 38,
                           'scale': 0,
                           'type':'NUMERIC'},
           'isDiscarded': False,
           'machineGenerated': False,
           'subFields': None}}

The field mapping details with the new nonnull setting

{'fieldName': 'id',
'fields': [],
'mapping': {'columnName': 'ID',
           'columnType': {'nonNull': False,
                           'precision': 38,
                           'scale': 0,
                           'type':'NUMERIC'},
           'isDiscarded': False,
           'machineGenerated': False,
           'subFields': None}}

Check Consolidation Uses Log

Mappings.check_if_consolidation_uses_log()

Checks if the event uses a log table

Parameters:event_name – The name of the event for which to check
Returns:None

Prints the event name, consolidation key list, the table data is inserted to, and True/False if the insert contains _LOG in the name event_name [‘consolidation_key_list’] mapping_table_name True/False

Copy Mapping

Mappings.copy_mapping(new_event)

Copies mapping from the event the class is instantiated with to a new event. Only the name changes

Parameters:
  • new_event – The name of the event to copy to
  • print_mapping – Prints the changes if specified
  • apply_changes – Applies the changes if specified
Returns:

The altered mapping

Set Mapping from Existing Mapping

Mappings.set_mapping_from_existing_mapping(new_event_name, new_schema, new_table, new_input_label)

Takes the mapping from the event the class is instantiated with event and uses it to set a new mapping

Parameters:
  • new_event_name – The event for which to set the new mapping
  • new_schema – The schema for the new mapping
  • new_table – The table for the new mapping
  • new_input_label – The input label of the new mapping
Returns:

The altered mapping

Add Field To Mapping

Mappings.add_field_to_mapping(field_name, column_name, data_type, precision=38, scale=0, length=16777216, truncate=False, non_null=False)

Adds a single field without sub-fields to the mapping. Works for Snowflake data types only.

Parameters:
  • field_name – The field name to add
  • column_name – The name of the column in the target DB
  • data_type – The data type for the field out of NUMBER, VARIANT, BOOLEAN, VARCHAR, FLOAT, TIMESTAMP
  • precision – The precision for numeric data
  • scale – The scale for numeric data
  • length – The length for a varchar
  • truncate – Set to True if the event should be truncated when it’s longer than the specific mapping length. Redshift’s max VARCHAR is 65535
  • non_null – Set to true if the field is needs to be not null
Returns:

The altered mapping

Prints the entire existing mapping before the field is added

{'autoMappingError': None,
 'consolidation': {'consolidatedSchema': 'MY_SCHEMA',
                  'consolidatedTableName': 'MY_TABLE',
                  'consolidationKeys': ['ID'],
                  'viewSchema': None},
  'fields': [ {'fieldName': 'id',
              'fields': [],
              'mapping': {'columnName': 'ID',
              'columnType': {'nonNull': True,
                              'precision': 38,
                              'scale': 0,
                              'type': 'NUMERIC'},
                              'isDiscarded': False,
                              'machineGenerated': False,
                              'subFields': None}}
              ]
 'inputObjects': {'12345-asdfg': ['98765-zxcvb']},
 'mapping': {'isDiscarded': False,
             'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}',
             'outputId': 'a1s2d3-f4g5h6',
             'readOnly': False,
             'schema': 'MY_SCHEMA',
             'tableName': 'MY_TABLE_LOG'},
  'mappingMode': 'AUTO_MAP',
  'name': 'MY_SCHEMA.MY_TABLE',
  'origInputLabel': 'production_database',
  'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table',
              'schema?id=d=12345-asdfg&schema_object=deleted_rows'],
  'state': 'MAPPED',
  'usingDefaultMappingMode': False}

Then prints the entire new mapping to set after the field is added. This example adds the NAME field as a VARCHAR

 {'autoMappingError': None,
'consolidation': {'consolidatedSchema': 'MY_SCHEMA',
                 'consolidatedTableName': 'MY_TABLE',
                 'consolidationKeys': ['ID'],
                 'viewSchema': None},
 'fields': [ {'fieldName': 'id',
             'fields': [],
             'mapping': {'columnName': 'ID',
             'columnType': {'nonNull': True,
                             'precision': 38,
                             'scale': 0,
                             'type': 'NUMERIC'},
                             'isDiscarded': False,
                             'machineGenerated': False,
                             'subFields': None}},
             {'fieldName': 'name',
             'fields': [],
             'mapping': {'columnName': 'NAME',
             'columnType': {'length': 16777216,
                             'nonNull': False,
                             'truncate': False,
                             'type': 'VARCHAR'},
             'isDiscarded': False,
             'machineGenerated': False,
             'subFields': None}}
             ]
'inputObjects': {'12345-asdfg': ['98765-zxcvb']},
'mapping': {'isDiscarded': False,
            'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}',
            'outputId': 'a1s2d3-f4g5h6',
            'readOnly': False,
            'schema': 'MY_SCHEMA',
            'tableName': 'MY_TABLE_LOG'},
 'mappingMode': 'AUTO_MAP',
 'name': 'MY_SCHEMA.MY_TABLE',
 'origInputLabel': 'production_database',
 'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table',
             'schema?id=d=12345-asdfg&sschema_object=deleted_rows'],
 'state': 'MAPPED',
 'usingDefaultMappingMode': False}

Remove Unmapped Fields and Clear Table Stats

Mappings.remove_unmapped_fields_and_clear_table_stats()

Remove unmapped fields from the a mapping and clear the UI stats. The mapper can become very slow over time and clearing the table stats periodically will help speed it back up.

Returns:none