Mappings¶
Mappings are the conversion of a dictionary to a flattened table structure. The mapping class allows you to perform operations to view and alter the settings for a single event type.
-
Mappings.
__init__
(api, event_name, preview_full=True, preview_changes=True, apply_changes=False, pprint_indent=2, pprint_width=250, pprint_depth=5)¶ View and change Alooma mappings. Mappings are the conversion of a dictionary to a flattened table structure. The class is initiated with the following variables:
Parameters: - api – The Alooma API client authentication
- event_name – The name of the event to view or change settings for
- preview_full – Prints the mapping or mapping changes if True. The default is True.
- preview_changes – Prints the changes in the mapping by category if True
- apply_changes – Executes the mapping changes if True
- pprint_indent – The indent value to pprint dictionaries
- pprint_width – The width value to pprint dictionaries
- pprint_depth – The depth value to pprint dictionaries
Get Mapping For Event¶
-
managealooma.mappings.Mappings.
get_mapping_for_event
()¶ Gets the mapping for the event name that the class was initialized with
Returns: a dictionary with the mapping
Get Mapping For Event¶
-
Mappings.
view_mapping
(view_field_mappings=False)¶ Gets the mapping for an event and allows printing with or with the field details
Parameters: view_field_mappings – Hides the field details when set to false. Useful for quick view of consolidation and mapping details without all the fields Returns: Returns the mapping dictionary the user viewed Sample Mapping printed with view_field_mappings=False. The mapping[‘fields’] key with the list of fields IS NOT printed.
{'autoMappingError': None, 'consolidation': {'consolidatedSchema': 'MY_SCHEMA', 'consolidatedTableName': 'MY_TABLE', 'consolidationKeys': ['ID'], 'viewSchema': None}, 'inputObjects': {'12345-asdfg': ['98765-zxcvb']}, 'mapping': {'isDiscarded': False, 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}', 'outputId': 'a1s2d3-f4g5h6', 'readOnly': False, 'schema': 'MY_SCHEMA', 'tableName': 'MY_TABLE_LOG'}, 'mappingMode': 'AUTO_MAP', 'name': 'MY_SCHEMA.MY_TABLE', 'origInputLabel': 'production_database', 'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table', 'schema?id=d=12345-asdfg&sschema_object=deleted_rows'], 'state': 'MAPPED', 'usingDefaultMappingMode': False}
Sample Mapping printed with view_field_mappings=True. The mapping[‘fields’] key with the list of fields IS printed.
{'autoMappingError': None, 'consolidation': {'consolidatedSchema': 'MY_SCHEMA', 'consolidatedTableName': 'MY_TABLE', 'consolidationKeys': ['ID'], 'viewSchema': None}, 'fields': [ {'fieldName': 'id', 'fields': [], 'mapping': {'columnName': 'ID', 'columnType': {'nonNull': True, 'precision': 38, 'scale': 0, 'type': 'NUMERIC'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}}, {'fieldName': 'name', 'fields': [], 'mapping': {'columnName': 'NAME', 'columnType': {'length': 16777216, 'nonNull': False, 'truncate': False, 'type': 'VARCHAR'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}} ], 'inputObjects': {'12345-asdfg': ['98765-zxcvb']}, 'mapping': {'isDiscarded': False, 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}', 'outputId': 'a1s2d3-f4g5h6', 'readOnly': False, 'schema': 'MY_SCHEMA', 'tableName': 'MY_TABLE_LOG'}, 'mappingMode': 'AUTO_MAP', 'name': 'MY_SCHEMA.MY_TABLE', 'origInputLabel': 'production_database', 'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table', 'schema?id=d=12345-asdfg&sschema_object=deleted_rows'], 'state': 'MAPPED', 'usingDefaultMappingMode': False}
Change Mapping Mode¶
-
managealooma.mappings.Mappings.
change_mapping_mode
(self, new_mapping_mode='STRICT')¶ Change the mapping mode. Alooma has 3 modes of AUTO_MAP, STRICT, and FLEXIBLE. We only use AUTO_MAP or STRICT
Parameters: new_mapping_mode – The new mapping mode to set: AUTO_MAP, STRICT, and FLEXIBLE Returns: The altered mapping The mapping settings keys and sample values
{'autoMappingError': None, 'mappingMode': None, 'usingDefaultMappingMode': True}
The mapping mode with the specified new mappingMode to alter
{'autoMappingError': None, 'mappingMode': 'AUTO_MAP', 'usingDefaultMappingMode': True}
Change Mapping Consolidation Settings¶
-
managealooma.mappings.Mappings.
change_mapping_consolidation_settings
(self, consolidation_schema, consolidation_table_name, consolidation_keys)¶ Updates the consolidation information for an event
Parameters: - consolidation_schema – The schema of consolidated table
- consolidation_table_name – The name of the consolidated table
- consolidation_keys – The consolidation keys (primary key) for the table. Takes a single field or a list
Returns: The altered mapping
The old consolidation settings
{'consolidatedSchema': 'MY_SCHEMA_TEMP', 'consolidatedTableName': 'NY_TABLE_ITEMS', 'consolidationKeys': ['IDENTIFIER'], 'viewSchema': None}
The new consolidation settings to apply
{'consolidatedSchema': 'MY_SCHEMA', 'consolidatedTableName': 'MY_TABLE', 'consolidationKeys': ['ID'], 'viewSchema': None}
Change Mapping Consolidation Key¶
-
Mappings.
change_mapping_consolidation_key
(new_consolidation_key)¶ Change the conolidation key only
Parameters: new_consolidation_key – The new consolidation key. This is the primary key for the table. Returns: The altered mapping The consolidation key is set to the field name ID when tables are auto-mapped
{'consolidatedSchema': 'MY_SCHEMA', 'consolidatedTableName': 'MY_TABLE', 'consolidationKeys': ['ID'], 'viewSchema': None}
This example will change the key to a field named IDENTIFIER
{'consolidatedSchema': 'MY_SCHEMA', 'consolidatedTableName': 'MY_TABLE', 'consolidationKeys': ['IDENTIFIER'], 'viewSchema': None}
Change Mapping to Use Log¶
-
Mappings.
change_mapping_to_use_log
()¶ Changes the mapping from using the consolidated table to the log table. Used when adjusting manual mappings
Parameters: event_name – The event name of the mapping to alter Returns: The altered mapping Events without a _log do not have consolidated table information
'mapping': {'isDiscarded': False, 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}', 'outputId': 'a1s2d3-f4g5h6', 'readOnly': False, 'schema': 'MY_SCHEMA', 'tableName': 'MY_TABLE'}
This example will change the key to a field named IDENTIFIER
'mapping': {'isDiscarded': False, 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}', 'outputId': 'a1s2d3-f4g5h6', 'readOnly': False, 'schema': 'MY_SCHEMA', 'tableName': 'MY_TABLE_LOG'}
Change Mapping for Manual Consolidation Creation¶
-
Mappings.
change_mapping_for_manual_consolidation_creation
(consolidation_schema, consolidation_table_name, consolidation_keys, case='UPPER')¶ Updates the mapping after creating the log table manually
Parameters: - consolidation_schema – The schema of consolidated table
- consolidation_table_name – The name of the consolidated table
- consolidation_keys – The consolidation keys (primary key) for the table. Takes a single field or a list
- case – UPPER if your events are MY_SCHEMA.MY_EVENT and LOWER if the events are my_schema_my_event
Returns: The altered mapping
First adjust the tables in the data warehouse.
ALTER TABLE MY_TABLE RENAME TO MY_TABLE_LOG; CREATE TABLE MY_TABLE LIKE MY_TABLE_LOG;
Then run change_mapping_for_manual_consolidation_creation to change the mapping to insert data into the new log table. This is the old mapping:
'consolidation': {'consolidatedSchema': None, 'consolidatedTableName': None, 'consolidationKeys': , 'viewSchema': None}, 'mapping': {'isDiscarded': False, 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}', 'outputId': 'a1s2d3-f4g5h6', 'readOnly': False, 'schema': 'MY_SCHEMA', 'tableName': 'MY_TABLE'},
These are the changes that will be applied with the new mapping
'consolidation': {'consolidatedSchema': 'MY_SCHEMA', 'consolidatedTableName': 'MY_TABLE', 'consolidationKeys': ['ID_FIELD'], 'viewSchema': None}, 'mapping': {'isDiscarded': False, 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}', 'outputId': 'a1s2d3-f4g5h6', 'readOnly': False, 'schema': 'MY_SCHEMA', 'tableName': 'MY_TABLE_LOG'},
Lastly add consolidation queries to combine the new data with the existing data using
create_consolidation()
Delete Field From Mapping¶
-
Mappings.
delete_field_from_mapping
(field_name)¶ Gets a mapping, deletes a field from the mapping, and resets the mapping table statistics.
Parameters: - event_name – the name of event for which to make the change
- field_name – the field name that should be deleted
Returns: The altered mapping
If print_changes is specified the details for the field to remove is printed
{'fieldName': 'name', 'fields': [], 'mapping': {'columnName': 'NAME', 'columnType': {'length': 16777216, 'nonNull': False, 'truncate': False, 'type': 'VARCHAR'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}}
Then the entire new mapping is printed. The ‘name’ field is not longer in the mapping
{'autoMappingError': None, 'consolidation': {'consolidatedSchema': 'MY_SCHEMA', 'consolidatedTableName': 'MY_TABLE', 'consolidationKeys': ['ID'], 'viewSchema': None}, 'fields': [ {'fieldName': 'id', 'fields': [], 'mapping': {'columnName': 'ID', 'columnType': {'nonNull': True, 'precision': 38, 'scale': 0, 'type': 'NUMERIC'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}} ] 'inputObjects': {'12345-asdfg': ['98765-zxcvb']}, 'mapping': {'isDiscarded': False, 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}', 'outputId': 'a1s2d3-f4g5h6', 'readOnly': False, 'schema': 'MY_SCHEMA', 'tableName': 'MY_TABLE_LOG'}, 'mappingMode': 'AUTO_MAP', 'name': 'MY_SCHEMA.MY_TABLE', 'origInputLabel': 'production_database', 'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table', 'schema?id=d=12345-asdfg&sschema_object=deleted_rows'], 'state': 'MAPPED', 'usingDefaultMappingMode': False}
Change Field Mapping Settings¶
-
Mappings.
change_field_mapping_settings
(field_name, new_data_type, truncate=False, non_null=False)¶ Updates a single filed in a mapping
Parameters: - field_name – The field name to alter
- new_data_type – The new datatype for the mapping such as VARCHAR(1024) or INT
- truncate – Set to True if the event should be truncated when it’s longer than the specific mapping length. Redshift’s max VARCHAR is 65535 and Snowflake’s max VARCHAR is 16777216.
- non_null – Set to true if the field is needs to be not null
Returns: The altered mapping
The current mapping field settings
{'fieldName': 'name', 'fields': [], 'mapping': {'columnName': 'NAME', 'columnType': {'INT': 'nonNull': False, 'truncate': False, 'type': 'VARCHAR'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}}
The new mapping field settings to apply
{'fieldName': 'name', 'fields': [], 'mapping': {'columnName': 'NAME', 'columnType': {'length': 1024, 'nonNull': False, 'truncate': True, 'type': 'VARCHAR'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}}
Change Field Varchar Length¶
-
Mappings.
change_field_varchar_length
(field_name, new_length)¶ Updates only the length of a varchar for a field.
Parameters: - field_name – The field name to alter
- new_length – The new length for the varchar
Returns: None
{'fieldName': 'name', 'fields': [], 'mapping': {'columnName': 'NAME', 'columnType': {'length': 1024, 'nonNull': False, 'truncate': False, 'type': 'VARCHAR'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}}
{'fieldName': 'name', 'fields': [], 'mapping': {'columnName': 'NAME', 'columnType': {'length': 16777216, 'nonNull': False, 'truncate': True, 'type': 'VARCHAR'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}}
Change Field Null Constraint¶
-
Mappings.
change_field_null_constraint
(field_name, nonnull=False)¶ Removes the NULL constraint from a column
Parameters: - field_name – The column for which to remove the constraint
- nonnull – The new null setting for the field
Returns: The altered mapping
The field mapping details with the current nonnull setting
{'fieldName': 'id', 'fields': [], 'mapping': {'columnName': 'ID', 'columnType': {'nonNull': True, 'precision': 38, 'scale': 0, 'type':'NUMERIC'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}}
The field mapping details with the new nonnull setting
{'fieldName': 'id', 'fields': [], 'mapping': {'columnName': 'ID', 'columnType': {'nonNull': False, 'precision': 38, 'scale': 0, 'type':'NUMERIC'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}}
Check Consolidation Uses Log¶
-
Mappings.
check_if_consolidation_uses_log
()¶ Checks if the event uses a log table
Parameters: event_name – The name of the event for which to check Returns: None Prints the event name, consolidation key list, the table data is inserted to, and True/False if the insert contains _LOG in the name event_name [‘consolidation_key_list’] mapping_table_name True/False
Copy Mapping¶
-
Mappings.
copy_mapping
(new_event)¶ Copies mapping from the event the class is instantiated with to a new event. Only the name changes
Parameters: - new_event – The name of the event to copy to
- print_mapping – Prints the changes if specified
- apply_changes – Applies the changes if specified
Returns: The altered mapping
Set Mapping from Existing Mapping¶
-
Mappings.
set_mapping_from_existing_mapping
(new_event_name, new_schema, new_table, new_input_label)¶ Takes the mapping from the event the class is instantiated with event and uses it to set a new mapping
Parameters: - new_event_name – The event for which to set the new mapping
- new_schema – The schema for the new mapping
- new_table – The table for the new mapping
- new_input_label – The input label of the new mapping
Returns: The altered mapping
Add Field To Mapping¶
-
Mappings.
add_field_to_mapping
(field_name, column_name, data_type, precision=38, scale=0, length=16777216, truncate=False, non_null=False)¶ Adds a single field without sub-fields to the mapping. Works for Snowflake data types only.
Parameters: - field_name – The field name to add
- column_name – The name of the column in the target DB
- data_type – The data type for the field out of NUMBER, VARIANT, BOOLEAN, VARCHAR, FLOAT, TIMESTAMP
- precision – The precision for numeric data
- scale – The scale for numeric data
- length – The length for a varchar
- truncate – Set to True if the event should be truncated when it’s longer than the specific mapping length. Redshift’s max VARCHAR is 65535
- non_null – Set to true if the field is needs to be not null
Returns: The altered mapping
Prints the entire existing mapping before the field is added
{'autoMappingError': None, 'consolidation': {'consolidatedSchema': 'MY_SCHEMA', 'consolidatedTableName': 'MY_TABLE', 'consolidationKeys': ['ID'], 'viewSchema': None}, 'fields': [ {'fieldName': 'id', 'fields': [], 'mapping': {'columnName': 'ID', 'columnType': {'nonNull': True, 'precision': 38, 'scale': 0, 'type': 'NUMERIC'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}} ] 'inputObjects': {'12345-asdfg': ['98765-zxcvb']}, 'mapping': {'isDiscarded': False, 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}', 'outputId': 'a1s2d3-f4g5h6', 'readOnly': False, 'schema': 'MY_SCHEMA', 'tableName': 'MY_TABLE_LOG'}, 'mappingMode': 'AUTO_MAP', 'name': 'MY_SCHEMA.MY_TABLE', 'origInputLabel': 'production_database', 'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table', 'schema?id=d=12345-asdfg&schema_object=deleted_rows'], 'state': 'MAPPED', 'usingDefaultMappingMode': False}
Then prints the entire new mapping to set after the field is added. This example adds the NAME field as a VARCHAR
{'autoMappingError': None, 'consolidation': {'consolidatedSchema': 'MY_SCHEMA', 'consolidatedTableName': 'MY_TABLE', 'consolidationKeys': ['ID'], 'viewSchema': None}, 'fields': [ {'fieldName': 'id', 'fields': [], 'mapping': {'columnName': 'ID', 'columnType': {'nonNull': True, 'precision': 38, 'scale': 0, 'type': 'NUMERIC'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}}, {'fieldName': 'name', 'fields': [], 'mapping': {'columnName': 'NAME', 'columnType': {'length': 16777216, 'nonNull': False, 'truncate': False, 'type': 'VARCHAR'}, 'isDiscarded': False, 'machineGenerated': False, 'subFields': None}} ] 'inputObjects': {'12345-asdfg': ['98765-zxcvb']}, 'mapping': {'isDiscarded': False, 'outputHint': '{"table":"my_table","schema":"MY_SCHEMA"}', 'outputId': 'a1s2d3-f4g5h6', 'readOnly': False, 'schema': 'MY_SCHEMA', 'tableName': 'MY_TABLE_LOG'}, 'mappingMode': 'AUTO_MAP', 'name': 'MY_SCHEMA.MY_TABLE', 'origInputLabel': 'production_database', 'schemaUrls': ['schema?id=12345-asdfg&schema_object=my_table', 'schema?id=d=12345-asdfg&sschema_object=deleted_rows'], 'state': 'MAPPED', 'usingDefaultMappingMode': False}