Welcome to apu’s documentation!

apu Time

apu.datetime: anton python utils datetime module

class apu.time.DateTime

Datetime contains the datetime utility functions

static add_time(datetime_: datetime.datetime, day: int = 0, hour: int = 0, minute: int = 0, seconds: int = 0) → datetime.datetime
add time to a datetime object with respect to the

timezone

This function will keep the correct timezone

Parameters

  • datetime (datetime.datetime) – datetime object

  • day (int) – days

  • hour (int) – hours

  • seconds (int) – seconds

Returns

time in the correct timezone

Return type

(datetime.datetime)

static datetime2unix(datetime_: datetime.datetime) → float
convert a datetime object into

a unixtimestamp

Parameters

datetime (dt.datetime) – datetime

Returns

unix timestamp

Return type

(float)

Examples: .. example_code:

>>> from apu.time.date_time import DateTime
>>> DateTime.unixtime2date(datetime.now())
1234567890.123
static generate(start: datetime.datetime, end: datetime.datetime, random_callback=<class 'random.Random'>) → datetime.datetime

generate a random date.

Parameters

  • start (datetime.datetime) – start time

  • end (datetime.datetime) – end time

  • random_callback (random.Random) – random number generator

Returns

datetime

Return type

(datetime.datetime)

Raises

ValueError – the start end oder is wrong

Examples: .. example_code:

>>> import random
>>> import datetime as dt
>>> from apu.time.date_time import DateTime

>>> rand = random.Random()
>>> rand.seed(0)

>>> print(DateTime.generate(start=dt.datetime(2020,1,2),
>>>       end=dt.datetime(2020,2,3),
>>>       random_callback=rand))
2020-01-29 00:30:57.535096
static time_string(datetime: datetime.datetime = datetime.datetime(2021, 2, 12, 18, 11, 14, 789860), form: str = '%y%m%d_%H%M%S') → str

create string date.

Parameters

  • datetime (datetime.datetime) – date

  • form (str) – format string

Returns

date as string

Return type

(str)

Examples: .. example_code:

>>> import datetime as dt
>>> from apu.time.date_time import DateTime
>>> print(DateTime.time_string(
...       datetime=dt.datetime(2020,1,2,6,4,53),
...       form="%y%m%d_%H%M%S"))
200102_060453
static unixtime2date(unix: float) → datetime.datetime

convert a uniy timestamp to datetime object

Parameters

unix (float) – unix timestamp

Returns

datetime

Return type

(datetime.datetime)

Examples: .. example_code:

>>> from apu.time.date_time import DateTime
>>> DateTime.unixtime2date(1234567890.123)
2009-02-13 23:31:30.123000
apu.time.time_it(duration='s')

generate a random date.

Parameters

  • start (datetime.datetime) – start time

  • end (datetime.datetime) – end time

  • random_callback (random.Random) – random number generator

Returns

datetime

Return type

(datetime.datetime)

Raises

ValueError – the start end oder is wrong

Examples: .. example_code:

>>> import math
>>> from apu.time.timer import time_it

>>> @time_it()
>>> def test():
>>>    math.sqrt(1231)

>>> test()
Time taken by the function is [0.0003693103790283203] sec

apu Geography

apu.datetime: anton python utils geography module

apu.geo.carree2pix(coord: List[float], area: List[Tuple[float]], image_size: List[int], origin: str = 'upper') → tuple

Convert Carree lat/long to image poxel position

Parameters

  • coord (List[float]) – (lat, lon) coordinate in image

  • area (List[Tuple[float]]) – ((u,v)_min, (u,v)_max) the area in the image to search in

  • image_size (List[int]) – image size in width and height

  • origin (str) – image convention. where is the image origin. ‘upper’ means the origin [0,0] is in the upper left corner ‘lower’ means that the image origin is in the lower left corner

Returns

(u, v) pixel coordinate

Return type

tuple

apu.geo.km2pix(height: float, extention: float, radius: float = 6378.137, distorsion_scaling: float = 1.0) → float

convert from kilometer in pixel

Parameters

  • height (float) – image height

  • extention (float) – latitude extention of image in pixel

  • radius (float) – planet radius in kilometer default is the earth radius (6_378.137km)

  • distorsion_scaling (float) – Scaling factor for distortion between 0. and 1.

Returns

conversionsfactor pix/m

Return type

float

apu.geo.m2pix(height: float, extention: float, radius: float, distorsion_scaling: float = 1.0) → float

convert from meter in pixel

Parameters

  • height (float) – image height

  • extention (float) – latitude extention of image in pixel

  • radius (float) – planet radius in metern

  • distorsion_scaling (float) – Scaling factor for distortion between 0. and 1.

Returns

conversionsfactor pix/m

Return type

float

apu.geo.pix2carree(pixel: List[float], area: List[Tuple[float]], image_size: List[int], origin: str = 'upper') → tuple
Convert image pixel position to Carree lat/long

ASSAMTION: central median is 0 => (long [-180,180[)

Parameters

  • pixel (List[float]) – (u,v) coordinate in image

  • area (List[Tuple[float]]) – ((u,v)_min, (u,v)_max) the area in the image to search in

  • image_size (List[int]) – image size in width and height

  • origin (str) – image convention. where is the image origin. ‘upper’ means the origin [0,0] is in the upper left corner ‘lower’ means that the image origin is in the lower left corner

Returns

(lat, lon) coordinates in a Plate Carree image

Return type

tuple

apu Setup

apu.setup: anton python utils setup module

class apu.setup.BuildProtoBuf(dist)

find and execute protobuf related work

run()

run the proto_builder

class apu.setup.CleanProtoBuf(dist)

clean protobuf info

run()

Delete generated files in the code tree.

class apu.setup.Module

in this class all module containing functions are placed.

static check(module_name: str)

check if a module exists without importing it

Parameters

module_name (str) – module name

Returns

Module specification

Return type

(ModuleSpec)

Raises

  • ModuleNotFoundError – Module not found

  • AttributeError – given attribute error

static import_module_from_spec(module_name: str)

If the module is found utilizing check_module

Parameters

module_name (str) – name of the module to import

Returns

imported module

Return type

(module)

Raises

  • ModuleNotImportedError – cannot import module

  • ModuleNotFoundError – cannot find the module. install the module first

  • AttributeError – given attribute for the module check is not valid

apu.setup.find_protoc()

Locates protoc executable

apu.setup.replace_line_in_file(fpath, old_line_start, new_line)

find and delete string in a file

apu.setup.setversion(repopath, versionfile)

set the version number in a given file

apu Multiprocessing

apu.mp: anton python utils multiprocessing module

apu.mp.parallel_for(loop_callback: Callable[[Any], Any], parameters: List[Tuple[Any, …]], nb_threads: int = 8) → List[Any]

Execute a for loop body in parallel .. note:: Race-Conditions

Code executation in parallel can cause into an “race-condition” error.

Parameters

  • loop_callback (Callable) – function callback running in the loop body

  • parameters (List[Tuple]) – element to execute in parallel

Returns

list of values

Return type

(List[Any])

Examples: .. example-code:

>>> x = lambda x: x ** 2
>>> parallel_for(x, [y for y in range(10)])
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
apu.mp.thread_funcrun(func)

run function in thread

Examples: .. example_code:

>>> from apu.mp.thread_funcrun import thread_funcrun

>>> @thread_funcrun
... def test(*args, **kwargs):
...     for i in range(5):
...             print(f'elem: {i}')

elem: 0
elem: 1
elem: 2
elem: 3
elem: 4
Thread started for function "test"
apu.mp.thread_n_funcrun(number_of_threads=1)

run function in multiple threads

Examples: .. example_code:

>>> from apu.mp.thread_funcrun import thread_funcrun

>>> @thread_n_funcrun(number_of_threads=3)
... def test(*args, **kwargs):
...     pass
Thread started for function "test"
Thread started for function "test"
Thread started for function "test"

apu Design Patterns

apu.dp: anton python utils design pattern module

class apu.dp.AlphabeticalOrderCollection(collection: List[Any] = ())

Collection handling a collection with iterator

Examples: .. example-code:

>>> liste = AlphabeticalOrderCollection()
>>> liste.add_item(0)
>>> liste.add_item(1)
>>> liste.add_item(2)
>>> print("
“.join(liste))

0 1 2 >>> print(“

“.join(liste.get_reverse_iterator()), end=””)

2 1 0

add_item(item: Any) → None

add an element to the collection

item

an item for the collection

Type

Any

get_reverse_iterator() → apu.dp.iterator.AlphabeticalOrderIterator

return an iterator object

Returns

the iterator in reversed order

Return type

(AlphabeticalOrderIterator)

class apu.dp.AlphabeticalOrderIterator(collection: List[Any] = (), reverse_order: bool = False)

Alphabetical ordered collection. this class stores the traversal positon at all time

class apu.dp.Blackboard

a shared memory structure to handover information in different objects TODO: Memory info

clear()

clean the memory for all reference keys

clear_callbacks(key: str)

clear the callback

Parameters

key (str) – reference key

Raises

NonExistingKey – key does not exists

close()

delete the meta information and close the memory

drop(key: str) → bool

drop memory with pointer

Parameters

key (str) – pointer to memory

Returns

Ture if the drop was successful

Return type

(bool)

Raises

NonExistingKey – key to memory is unknown

get(key: str) → Any

get the memory

Parameters

key (str) – reference key

Returns

memory value

Return type

(Any)

Raises

NonExistingKey – key does not exists

keys(in_list: bool = False) → List[str]

get a list of all reference keys

Parameters

in_list (bool) – as a python list

Returns

an iterator or python list object

Return type

List[str]

load(dir_path: str = './.blackboard', safe: bool = True)

load the data from blackboard file

Parameters

  • dir_path (str) – filepath

  • safe (bool) – load safe

Raises
register_callback(key: str, callback: Callable[[Any], Any]) → int

register a callback on the memory values

Parameters

  • key (str) – reference key

  • callback (Callable[[Any], Any]) – callback function

Returns

python id Object

Return type

int

Raises

NonExistingKey – the key does not exists

remove_callback(key: str, callback: Callable[[Any], Any]) → int

remove the callback from the memory

Parameters

  • key (str) – reference key

  • callback (Callable[[Any], Any]) – delete this callback

Returns

python object id

Return type

int

Raises

NonExistingKey – key does not exists

save(dir_path: str = './.blackboard')

save the blackboard

Parameters

dir_path (str) – file path

set(key: str, value: Any, read_only: bool = False) → bool

set the memory related to the reference key

Parameters

  • key (str) – reference

  • value (Any) – memory value

  • read_only (bool) – store the data read only?

Returns

True if the memory could be placed

Return type

(bool)

Raises
update(key: str, value: Any) → bool

change the memory

Parameters

  • key (str) – reference key

  • value (Any) – Memory value

Returns

true if the data are places successfully

Return type

(bool)

Raises
class apu.dp.MetaInfo(read_only: bool = True)

additional information and data manipulation utilizing callbacks

property callback

get the callbacks

Returns

list of all callbacks

Return type

(List[Callable[[Any], Any]])

clean()

delete all callbacks

del_callback(callback: Callable[[Any], Any])

get one callback if the callback is in the list

Parameters

callback (Callable[[Any], Any]) – callable structure

class apu.dp.Null(*args, **kargs)

Null Object implementation

This class ignores all parameters parsed to the object on creation. The created instances do nothing and mark an object nor nothing

class apu.dp.Singleton

singleton metaclass

Examples: .. example_code:

>>> from apu.dp.singleton import Singleton

>>> class Duck(metaclass=Singleton):
>>>     pass

>>> Duck() is Duck()
True
apu.dp.singleton(cls)

singleton declerator

Examples: .. example_code:

>>> from apu.dp.singleton import singleton

>>> @singleton
>>> class Duck():
>>>     pass

>>> Duck() is Duck()
True

apu Exceptions

apu.exception: anton python utils excaptions module

exception apu.exception.ExistingKey

the key allready exists

exception apu.exception.KeyNotString

key type is not String

exception apu.exception.ModuleNotImportedError

Extends ModuleNotFoundError. Because a Module can also available but not imported

exception apu.exception.NonExistingDirectory

The requested directory do not exists

exception apu.exception.NonExistingKey

the key does not exists

exception apu.exception.NotEditable

you try to edit a not editable object

exception apu.exception.ReadWrongFile

read the wrong file error

exception apu.exception.UnsafeLoading

unsafe loading. are you sure

exception apu.exception.UnsupportedDataType

requested datatype is not supported

apu Input/Output

apu.io anton python utils input output module

class apu.io.FileFormat(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

base class. so each object implements the same functions

property access_time

date of last access

compair(filepath: str, method='sha1')

compair two files utilizing the fingerprint

property creation_time

date of creation

fingerprint(method: str = 'sha1')

build file fingerprint

meta() → Any

aditional meta informations

property modification_time

date of modification

abstract read()

read information from file into data buffer

classmethod suffix()

file extentions

abstract write(sink: str, create: bool = True) → None

write buffer into file

class apu.io.Path(*args, **kwargs)

path changes the type on different os

expand()

expand all together and resolve all

expand_vars()

expand vars

apu.io.download(source: str, sink: Optional[str] = None) → str

download from page

apu.io.load(data: str) → Any

serialize the data

Parameters

data (str) – string to deserialize

Returns

deserialized data

Return type

(Any)

Raises

UnsupportedDataType – the data are not deserializeable

apu.io.reconstruct(value: Any) → str

serialize the data

Parameters

value (Any) – object to serialize

Returns

serialized data

Return type

(str)

Raises

UnsupportedDataType – the data are not serializeable

apu.io.urlread(url: str, encoding: str = 'utf8') → str

read a page

file format input output realization with meta info

class apu.io.__fileformat.CSV(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

read write csv files

read()

read a csv file

classmethod suffix()

file extentions

write()

write a csv file

class apu.io.__fileformat.DILL(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

working with a dill file

read()

read dill

classmethod suffix()

file extentions

write()

“write dill

class apu.io.__fileformat.H5(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

h5 fileformat handler

read()

read information from file into data buffer

recursively_load_dict_contents_from_group(h5_file, path)

Load contents of an HDF5 group. If further groups are encountered, treat them like dicts and continue to load them recursively.

recursively_save_dict_contents_to_group(h5file: h5py._hl.files.File, path: str, dic: dict)

save recursivly

classmethod suffix()

file extentions

write()

write buffer into file

class apu.io.__fileformat.JSON(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

json file

read()

read json file

classmethod suffix()

json suffix

write()

write json files

class apu.io.__fileformat.JSONL(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

json line files

read()

read json line files

classmethod suffix()

json line suffix

write()

read json line

class apu.io.__fileformat.MAT(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

handle mat files

read()

read mat file

classmethod suffix()

matlab suffix

write()

write mat file

class apu.io.__fileformat.NPY(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

handle npy files

read()

read npy files

classmethod suffix()

numpy suffix

write()

write npy files

class apu.io.__fileformat.NPZ(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

handle npz files

classmethod suffix()

compressed numpy suffix

write()

write npz files

class apu.io.__fileformat.PICKLE(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

handle pickel

read()

read pickled file

classmethod suffix()

file extentions

write()

write pickel file

class apu.io.__fileformat.YAML(path: str, kwargs: Optional[Dict] = None, data: Optional[Any] = None)

reading and writing yaml files

read()

open a yaml file and read the data into the buffer

classmethod suffix()

yaml suffix

write()

write the data into a data sink

apu Datastructures

apu.datastructures anton python utils datastructures

class apu.datastructures.CircularBuffer(size=1)

ringbuffer based on deque

property average

calculate the avarage of all the componentes in the ringbuffer

Returns

the avarage

Return type

float

class apu.datastructures.Dictionary

shared Memory dictionary

property all

get the whole dict

Returns

the shared memory dictionary

Return type

(dict)

delete(key: str) → None

delete a key value pair

Parameters

key (str) – key pointing to memory position

exists(key: str) → bool
check if the key is in the list of

pointers in the memory

Parameters

key (str) – pointer to value

Returns

true if the key is a valid pointer

Return type

(bool)

flush()

clean the memory

get(key: str) → Any

get an dictionary element by key

Parameters

key (str) – key pointing to the memory related to it

Returns

the value or a Null object

Return type

(Any)

keys() → List[str]

get all keys of the data storage

Returns

return a list of strings

Return type

(List[str])

set(key: str, value: Any) → bool

set a key value pair into the dictionary

Parameters

  • key (str) – key value has to be string

  • value (Any) – any value is allowed

Returns

successful added

Return type

(bool)

class apu.datastructures.DictionaryWrapper(**kwargs)

Dictionary class wrapper class. This is used for using Dictionary as a memory.

close()

close the memory and delete all items

delete(key: str) → bool

delete the value related to the pointer

Parameters

key (str) – point to memory

Returns

successfully deleted

Return type

(bool)

get(key: str) → Any

get the memory utilizing a reference

Parameters

key (str) – key to memory

Returns

memory or Null-Object

Return type

(Any)

has(key: str) → bool

does points contain the key

Parameters

key (str) – pointer wo memory

Returns

True if the pointers contains the key

Return type

(bool)

set(key: str, value: Any) → bool

store data to memory

Parameters

  • key (str) – pointer to memory

  • value (Any) – memory to store

Returns

True if the memory are set successfull

Return type

(bool)

setup()

setup the Wrapper with a dictionary object

class apu.datastructures.EnhancedList(*args: Iterable[T])

extends list of a gerneric type

reject_indices(indices: List[int])

remove the elements utilizing a list

Parameters

indices – List[int]

Returns

EnhancedList

Return type

list without rejected elements

class apu.datastructures.MemoryWrapper(**kwargs)

wrap the memory and setup steps. future change to additional wrapper

abstract close()

set the Pointer to Null

abstract delete(key: str) → bool

delete the memory object

Parameters

key (str) – pointer to memory object

Returns

true if successfull deleted

Return type

(bool)

abstract get(key: str) → Any

get the memory value by string

Parameters

key (str) – pointer to memory object

Returns

memory object

Return type

(Any)

abstract has(key: str) → bool

has pointer to memory?

Parameters

key (str) – request key

Returns

True if the pointer is in memory

Return type

(bool)

load(file_path: str) → bool

load the data from file

Parameters

file_path (str) – the path to the fail containing the memory

Returns

True if the storage was successful

Return type

(bool)

save(file_path: str) → bool

store the memory to file

Parameters

file_path (str) – the path to the fail containing the memory

Returns

True if the storage was successful

Return type

(bool)

abstract set(key: str, value: Any) → bool

set a key to the memory value :param key: string typed key :param value: Any memory object

Returns

bool

Return type

true if successfull stored

abstract setup()

memory setup

static transform_pickle_to_value(data: str) → Any

deserialize object

Parameters

data (str) – serialized value

Returns

deserialized Data

Return type

(Any)

static transform_value_to_pickle(value: Any) → str

serialize an object

Parameters

value (Any) – the memory

Returns

serialized object

Return type

(str)

apu machine Lerning

apu.ml: anton python utils machine learning

apu.ml.init: anton python utils machine learning initializers

apu.ml.init.glorot_normal(tensor: torch.Tensor, gain: float = 1.0) → torch.Tensor

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a uniform distribution. The resulting tensor will have values sampled from \(\mathcal{U}(-a, a)\) where

\[a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}}\]

Also known as Glorot initialization.

Parameters

  • tensor – an n-dimensional torch.Tensor

  • gain – an optional scaling factor

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu'))
apu.ml.init.glorot_unified(tensor: torch.Tensor, gain: float = 1.0) → torch.Tensor

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a normal distribution. The resulting tensor will have values sampled from \(\mathcal{N}(0, \text{std}^2)\) where

\[\text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}}\]

Also known as Glorot initialization.

Parameters

  • tensor – an n-dimensional torch.Tensor

  • gain – an optional scaling factor

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.xavier_normal_(w)
apu.ml.init.he_normal(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')

Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a normal distribution. The resulting tensor will have values sampled from \(\mathcal{N}(0, \text{std}^2)\) where

\[\text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}}\]

Also known as He initialization.

Parameters

  • tensor – an n-dimensional torch.Tensor

  • a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')

  • mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.

  • nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu')
apu.ml.init.he_uniform(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')

Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a uniform distribution. The resulting tensor will have values sampled from \(\mathcal{U}(-\text{bound}, \text{bound})\) where

\[\text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}}\]

Also known as He initialization.

Parameters

  • tensor – an n-dimensional torch.Tensor

  • a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')

  • mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.

  • nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).

Examples

>>> w = torch.empty(3, 5)
>>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')

Indices and tables