gromozeka.concurrency package

Submodules

gromozeka.concurrency.commands module

Commands to synchronize processes or threads

class gromozeka.concurrency.commands.Command(command, args=None)

Bases: object

Helper class to make commands

args = None

as_dict()

as_tuple()

command = None

gromozeka.concurrency.commands.backend_chain_get_result(graph_uuid, chain_uuid)

gromozeka.concurrency.commands.backend_graph_get(graph_uuid)

gromozeka.concurrency.commands.backend_graph_init(graph_dict)

gromozeka.concurrency.commands.backend_graph_result_set(graph_uuid, task_uuid, result)

gromozeka.concurrency.commands.backend_graph_update(graph_uuid, verticies, graph_state=None, error_task_uuid=None, short_error=None)

gromozeka.concurrency.commands.backend_group_add_result(graph_uuid, group_uuid, task_uuid, result)

gromozeka.concurrency.commands.backend_group_get_result(graph_uuid, group_uuid)

gromozeka.concurrency.commands.backend_is_group_completed(graph_uuid, group_uuid, expected)

gromozeka.concurrency.commands.backend_result_del(task_uuid)

Command to delete result

Parameters:task_uuid – Task uuid Returns: Return type:Command

gromozeka.concurrency.commands.backend_result_get(task_uuid, graph_uuid)

Command to get result from backend_adapter

Parameters:

• task_uuid – Task uuid
• graph_uuid – Graph uuid

Returns:

Return type:

Command

gromozeka.concurrency.commands.backend_result_set(task_uuid, result, graph_uuid)

Command to set task result

Parameters:

• task_uuid – Task uuid
• result – Result
• graph_uuid – Graph uuid

Returns:

Return type:

Command

gromozeka.concurrency.commands.backend_results_del(*task_uuids)

Command to delete results

Parameters:task_uuids – Task uuids Returns: Return type:Command

gromozeka.concurrency.commands.broker_on_pool_size_changed()

Command to change prefetch_count

Returns: Return type:Command

gromozeka.concurrency.commands.broker_task_done(task_uuid, broker_point, delivery_tag)

Command to acknowledge task message by it’s broker_adapter point, delivery_tag or scheduler_tag

Parameters:

• task_uuid (str) – task identification
• broker_point (primitives.BrokerPointType) – Broker entry
• delivery_tag (int) – Task delivery tag

Returns:

Return type:

Command

gromozeka.concurrency.commands.broker_task_register(task_id, broker_point, options, deserializator)

Command to add new consumer with broker_point to task

Parameters:

• task_id (str) – Unique task identification
• broker_point (primitives.BrokerPointType) – Broker entry
• options – Specific broker options. See NatsOptions broker for example

Returns:

Return type:

Command

gromozeka.concurrency.commands.broker_task_reject(task_uuid, broker_point, delivery_tag)

Command to reject task message by it’s broker_adapter point, delivery_tag or scheduler_tag

Parameters:

• task_uuid (str) – task identification
• broker_point (primitives.BrokerPointType) – Broker entry
• delivery_tag (int) – Task delivery tag

Returns:

Return type:

Command

gromozeka.concurrency.commands.broker_task_send(task_uuid, request, broker_point, reply_to=None)

Command to publish primitives.Request to customer

Parameters:

• task_uuid (str) – task identification
• request – Serialized request
• broker_point (gromozeka.BrokerPoint) –
• reply_to (gromozeka.BrokerPoint) –

Returns:

Return type:

Command

gromozeka.concurrency.commands.broker_task_send_delayed(task_uuid, request, broker_point, delay)

Command to publish primitives.Request to customer

Parameters:

• task_uuid (str) – task identification
• request (primitives.Request) – Request to publish
• broker_point –
• delay –

Returns:

Return type:

Command

gromozeka.concurrency.commands.pool_grow(n)

Command to grow pool

Parameters:n – Number of workers to grow Returns: Return type:Command

gromozeka.concurrency.commands.pool_remove_worker(worker_ident)

Command to remove worker from pool

Parameters:worker_ident (int) – Worker identification Returns: Return type:Command

gromozeka.concurrency.commands.pool_shrink(n)

Command to shrink pool

Parameters:n – Number of workers to shrink Returns: Return type:Command

gromozeka.concurrency.commands.pool_size()

Command to get pool size

Returns: Return type:Command

gromozeka.concurrency.commands.pool_stop()

Command to stop pool

Returns: Return type:Command

gromozeka.concurrency.pool module

class gromozeka.concurrency.pool.Pool(max_workers=1, worker_class=<class 'gromozeka.concurrency.worker.ThreadWorker'>, logger=None)

Bases: threading.Thread

Pool of workers

logger

logging.Logger – Class logger

worker_queue

multiprocessing.Queue – Queue to receive input requests

cmd

multiprocessing.Queue – Command queue

cmd_out

multiprocessing.Queue – Command result queue

max_workers

int – Number of workers to start

worker_class

gromozeka.concurrency.Worker – worker class

workers

list of gromozeka.concurrency.Worker – list of worker instances

_stop_event

multiprocessing.Event – Stop event

Parameters:

• max_workers (int, optional) – Number of workers to start
• worker_class (gromozeka.concurrency.Worker, optional) – Worker class

cmd

cmd_out

grow(n)

Grow pool by n number of workers

Parameters:n (int) – Number to grow

init_max_workers

is_on_stop

listen_cmd()

Listen to commands

logger

remove_worker(worker_ident)

Remove worker from pool by it`s ident

Parameters:worker_ident – Worker identification

run()

Start workers in threads or processes

shrink(n)

Shrink pool by n number of workers

Parameters:n (int) – Number to grow

size

Not command do not use this in workers

Returns:

start()

Start pool

stop()

Stop pool and child workers

stop_()

worker_cls

worker_queue

workers

gromozeka.concurrency.scheduler module

Module for schedule tasks

class gromozeka.concurrency.scheduler.Scheduler

Bases: threading.Thread

Scheduler class

logger

logging.Logger – Class logger

_stop_event

threading.Event – Stop event

_shed

sched.scheduler – Scheduler instance

_event_queue

queue.Queue – Queue for scheduler events

add(wait_time, func, args=None, kwargs=None, priority=1)

Add event to scheduler queue

Parameters:

• wait_time (float) – Delay to run (seconds)
• func – Callable function to run
• args – Func arguments
• kwargs – Func keyword arguments
• priority – Event priority

listen_cmd()

Listen to commands

logger

run()

Run scheduler in thread

stop()

Stop scheduler

gromozeka.concurrency.scheduler.delay(last_run=None, every=None, interval=None, at=None)

Make new delay from arguments

Examples

delay(every='days', interval=2) # This event will run every 2 days

delay(every='weeks', interval=1, at='15:40') # This event will run every week at 15:40

Parameters:

• last_run (str, optional) – Last run time
• every (str, optional) – May be seconds, minutes, hours, days, weeks.
• interval (int, optional) – Uses with every.
• at (str, optional) – Time formatted as hour:minute or hour:minute:second.

Returns:

new event delay. Returns False when not need more event to run - last run.

Return type:

int or False

gromozeka.concurrency.scheduler.delay_from_eta(eta)

Make new delay from eta

Examples

delay_from_eta(countdown='2018-02-20 13:38:33.0') # This event will run 20.02.18 at 13:38:33 once

Parameters:eta (str) – If exceptions.Retry occurs other retry will run at this time. Other arguments ignored. Returns:Delay Return type:int or None

gromozeka.concurrency.scheduler.new_eta(seconds)

Make new eta from summary of current time and seconds

Parameters:seconds (float) – Seconds Returns:Datetime in format “%Y-%m-%d %H:%M:%S.%f” Return type:str

gromozeka.concurrency.worker module

class gromozeka.concurrency.worker.ProcessWorker(pool)

Bases: multiprocessing.context.Process, gromozeka.concurrency.worker.Worker

Worker class for gromozeka.concurrency.Pool. Uses as mixin.

logger

logging.Logger – class logger

pool

gromozeka.concurrency.Pool – parent pool instance

_stop_event

multiprocessing.Event or threading.Event

Parameters:pool (gromozeka.concurrency.Pool) – Parent pool instance

run()

Method to be run in sub-process; can be overridden in sub-class

stop(timeout=None)

Stop worker. multiprocessing or threading part

class gromozeka.concurrency.worker.ThreadWorker(pool)

Bases: threading.Thread, gromozeka.concurrency.worker.Worker

Worker class for gromozeka.concurrency.Pool. Uses as mixin.

logger

logging.Logger – class logger

pool

gromozeka.concurrency.Pool – parent pool instance

_stop_event

multiprocessing.Event or threading.Event

Parameters:pool (gromozeka.concurrency.Pool) – Parent pool instance

run()

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

stop(timeout=None)

Stop worker. multiprocessing or threading part

class gromozeka.concurrency.worker.Worker(pool)

Bases: object

Worker class for gromozeka.concurrency.Pool. Uses as mixin.

logger

logging.Logger – class logger

pool

gromozeka.concurrency.Pool – parent pool instance

_stop_event

multiprocessing.Event or threading.Event

Parameters:pool (gromozeka.concurrency.Pool) – Parent pool instance

ident = None

is_alive()

Check worker is alive. multiprocessing or threading part

Returns:True if worker is alive or False Return type:bool

name = None

run()

Run worker in thread or process

Raises:

• exceptions.MaxRetriesExceedException
• BaseException

stop()

Stop worker. multiprocessing or threading part

Module contents

class gromozeka.concurrency.Pool(max_workers=1, worker_class=<class 'gromozeka.concurrency.worker.ThreadWorker'>, logger=None)

Bases: threading.Thread

Pool of workers

logger

logging.Logger – Class logger

worker_queue

multiprocessing.Queue – Queue to receive input requests

cmd

multiprocessing.Queue – Command queue

cmd_out

multiprocessing.Queue – Command result queue

max_workers

int – Number of workers to start

worker_class

gromozeka.concurrency.Worker – worker class

workers

list of gromozeka.concurrency.Worker – list of worker instances

_stop_event

multiprocessing.Event – Stop event

Parameters:

• max_workers (int, optional) – Number of workers to start
• worker_class (gromozeka.concurrency.Worker, optional) – Worker class

cmd

cmd_out

grow(n)

Grow pool by n number of workers

Parameters:n (int) – Number to grow

init_max_workers

is_on_stop

listen_cmd()

Listen to commands

logger

remove_worker(worker_ident)

Remove worker from pool by it`s ident

Parameters:worker_ident – Worker identification

run()

Start workers in threads or processes

shrink(n)

Shrink pool by n number of workers

Parameters:n (int) – Number to grow

size

Not command do not use this in workers

Returns:

start()

Start pool

stop()

Stop pool and child workers

stop_()

worker_cls

worker_queue

workers

class gromozeka.concurrency.ThreadWorker(pool)

Bases: threading.Thread, gromozeka.concurrency.worker.Worker

Worker class for gromozeka.concurrency.Pool. Uses as mixin.

logger

logging.Logger – class logger

pool

gromozeka.concurrency.Pool – parent pool instance

_stop_event

multiprocessing.Event or threading.Event

Parameters:pool (gromozeka.concurrency.Pool) – Parent pool instance

run()

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

stop(timeout=None)

Stop worker. multiprocessing or threading part

class gromozeka.concurrency.ProcessWorker(pool)

Bases: multiprocessing.context.Process, gromozeka.concurrency.worker.Worker

Worker class for gromozeka.concurrency.Pool. Uses as mixin.

logger

logging.Logger – class logger

pool

gromozeka.concurrency.Pool – parent pool instance

_stop_event

multiprocessing.Event or threading.Event

Parameters:pool (gromozeka.concurrency.Pool) – Parent pool instance

run()

Method to be run in sub-process; can be overridden in sub-class

stop(timeout=None)

Stop worker. multiprocessing or threading part

class gromozeka.concurrency.Scheduler

Bases: threading.Thread

Scheduler class

logger

logging.Logger – Class logger

_stop_event

threading.Event – Stop event

_shed

sched.scheduler – Scheduler instance

_event_queue

queue.Queue – Queue for scheduler events

add(wait_time, func, args=None, kwargs=None, priority=1)

Add event to scheduler queue

Parameters:

• wait_time (float) – Delay to run (seconds)
• func – Callable function to run
• args – Func arguments
• kwargs – Func keyword arguments
• priority – Event priority

listen_cmd()

Listen to commands

logger

run()

Run scheduler in thread

stop()

Stop scheduler

class gromozeka.concurrency.Worker(pool)

Bases: object

Worker class for gromozeka.concurrency.Pool. Uses as mixin.

logger

logging.Logger – class logger

pool

gromozeka.concurrency.Pool – parent pool instance

_stop_event

multiprocessing.Event or threading.Event

Parameters:pool (gromozeka.concurrency.Pool) – Parent pool instance

ident = None

is_alive()

Check worker is alive. multiprocessing or threading part

Returns:True if worker is alive or False Return type:bool

name = None

run()

Run worker in thread or process

Raises:

• exceptions.MaxRetriesExceedException
• BaseException

stop()

Stop worker. multiprocessing or threading part