Common

API Reference

class forepaas.core.common.AESCipher(key, iv)

Bases: object

AESCipher is an object used to encrypt/decrypt a string using AES

Parameters:

  • bs – int - Block size used for encryption/decryption

  • iv – string -

  • key – string - Key used for encryption/decryption

Example:

>>> cipher = AESCipher(key)
>>> encrypted = cipher.encrypt('hello world')
>>> decrypted = cipher.decrypt(encrypted)
decrypt(enc)
encrypt(message)
pad(s)
class forepaas.core.common.Request(session=None)

Bases: object

forepaas.core.common.addslashes(s)
forepaas.core.common.chunks(to_chunk, chunk_size)

Get chunksize of a list

Parameters:

  • to_chunk (mixed) – List or anything that you can do a range on

  • chunk_size (int) – Chunk size

forepaas.core.common.convert2unicode(mydict)
forepaas.core.common.convert_connection_str(connection_str)

Convert any connection string to readable string for the factory connector class

Parameters:

  • connection_str (str) – connection string to be parsed

  • conf (dict) – User or Worker configuration

Raises:

  • Exception – Table … not found in dwh

  • Exception – Can’t find database … in dwh

Returns:

ready-to-use connection string for connect

Return type:

str

Examples of formats :

  • mysql.data_prim/prim_data (DBMS.DB_NAME/TABLE_NAME)

  • dwh/data_prim/prim_data (dwh/DB_NAME/TABLE_NAME)

  • dwh/data (dwh/OBJECT_NAME)

forepaas.core.common.find(lst, key, value)

Find in a list of dict an element where dict[key] equals to value

Parameters:

  • lst (list(dict)) – List where to find the element

  • key (str) – Key of the dict to use to do the compare

  • value (mixed) – Value to compare with

Returns:

Index of the element found

Return type:

int

forepaas.core.common.generate_object_id() → str
forepaas.core.common.get_account_key(configuration, params)

Get the account key from parameters

Parameters:

  • configuration (dict) – connector configuration

  • params (dict) – connector parameters

Returns:

account key

Return type:

str

forepaas.core.common.get_database(name: str)

Retrieve database from the Name of it

Parameters:

name (str) – Database name

Returns:

DWH database configuration

forepaas.core.common.get_destination(params)

Get the correct destination in parameters

Parameters:

params (dict) – connector parameters

Returns:

destination

Return type:

str

forepaas.core.common.get_filename(params, destination)

Get filename in parameters, if filename is not set, then use destination by default

Parameters:

  • params (dict) – connector parameters

  • destination (str) – destination file

Returns:

destination

Return type:

str

forepaas.core.common.get_load_from(params)

Get the load from from parameters

Parameters:

params (dict) – connector parameters

Returns:

Load from

Return type:

dict

forepaas.core.common.get_source(params)

Get the correct source in parameters :param params: connector parameters :type params: dict :return: source :rtype: str

forepaas.core.common.get_table(db_name: str, table_name: str)

Retrieve table from DWH configuration

Parameters:

  • db_name (str) – Database name

  • table_name (str) – Table name

Returns:

DWH table’s configuration

forepaas.core.common.get_table_dwh(params, destination)

Get the table conf from the dwh :param params: connector conf :type params: dict :param destination: destination :type destination: str :return: Table config :rtype: dict

forepaas.core.common.ignore_fields(rows, params)

Remove some key in a row depending of params

Parameters:

  • row (DataFrame) – Row to filter

  • params (dict) – Params containing ignore or fields in them

Returns:

Filtered rows

Return type:

DataFrame

forepaas.core.common.joins_to_query(joins)

Translate joins as a list to a string that you can use to make a query to the dbms

Parameters:

joins (list(dict)) – Joins to query for aggregate action

Returns:

Usable query for aggregate connectors

Return type:

str

forepaas.core.common.list_files(path)

List files from a directory

Parameters:

path (str) – Path to list the files from

Raises:

Exception – [description]

Returns:

List of files

Return type:

list(str)

forepaas.core.common.merge_stats(stat, stats)

Merge the stats from an insert into mariadb for instance If the key exists in stats we will add the value to stats[key] Else we will create the field in stats with the corresponding value

Parameters:

  • stat (dict) – Stat received after an insert

  • stats (dict) – Regroupment of stat received after an insert

Raises:

Exception – [description]

Returns:

Merged stat and stats

Return type:

dict

forepaas.core.common.remove_accents(string)

Remove accents from characters in a string

Parameters:

string (str) – string to replace

Returns:

string with accents replaced

Return type:

str

forepaas.core.common.request_retry(method, url, retries=3, timeout=15, time_sleep=1, on_http_error=None, on_request_exception=None, **kwargs)

Send a request with a retry if the last request failed

Parameters:

  • method (str) – Method for the request “GET”, “POST”, …

  • url (str) – Url to send the request to

  • retries (int, optional) – NUmber of retries in case of failure, defaults to 3

  • timeout (int, optional) – Request timeout, defaults to 15

  • time_sleep (int, optional) – Request sleeping time between retry, defaults to 1

  • on_http_error (lambda) – Lambda to launch when there is an http error. Status_code >= 400

  • on_request_exception (lambda) – Lambda to launch when requests launches an exception

Returns:

Result of the request

Return type:

requests.Response

forepaas.core.common.safe_str(obj, demojize=False)
forepaas.core.common.simpleescape(s)
forepaas.core.common.slugify(string, to_lower: bool = True)
forepaas.core.common.time_unit_to_seconds(time_unit)

Convert a time unit to a usable time in seconds

Parameters:

time_unit (str) – Time unit

Returns:

Time in second

Return type:

int

Example of time unit:

  • “1y 2w 1s”

  • “1h”

  • “2w 30s”

forepaas.core.common.transform_schema(schema)

Transform a schema version 2 to a compatible format with v1

Parameters:

schema (dict) – Schema v2 of aggregate

Returns:

The schema and the default_schema that is used in v1

Return type:

tuple(dict, dict)

forepaas.core.common.unescape(s)
forepaas.core.common.use_load_from(params)

Check if load from should be used

Parameters:

params (dict) – connector parameters

Returns:

True or False

Return type:

Boolean