gromozeka.concurrency package¶
Submodules¶
gromozeka.concurrency.commands module¶
Commands to synchronize processes or threads
-
class
gromozeka.concurrency.commands.
Command
(command, args=None)¶ Bases:
object
Helper class to make commands
-
args
= None¶
-
as_dict
()¶
-
as_tuple
()¶
-
command
= None¶
-
-
gromozeka.concurrency.commands.
backend_chain_get_result
(graph_uuid, chain_uuid)¶
-
gromozeka.concurrency.commands.
backend_graph_get
(graph_uuid)¶
-
gromozeka.concurrency.commands.
backend_graph_init
(graph_dict)¶
-
gromozeka.concurrency.commands.
backend_graph_result_set
(graph_uuid, task_uuid, result)¶
-
gromozeka.concurrency.commands.
backend_graph_update
(graph_uuid, verticies, graph_state=None, error_task_uuid=None, short_error=None)¶
-
gromozeka.concurrency.commands.
backend_group_add_result
(graph_uuid, group_uuid, task_uuid, result)¶
-
gromozeka.concurrency.commands.
backend_group_get_result
(graph_uuid, group_uuid)¶
-
gromozeka.concurrency.commands.
backend_is_group_completed
(graph_uuid, group_uuid, expected)¶
-
gromozeka.concurrency.commands.
backend_result_del
(task_uuid)¶ Command to delete result
Parameters: task_uuid – Task uuid Returns: Return type: Command
-
gromozeka.concurrency.commands.
backend_result_get
(task_uuid, graph_uuid)¶ Command to get result from backend_adapter
Parameters: - task_uuid – Task uuid
- graph_uuid – Graph uuid
Returns: Return type:
-
gromozeka.concurrency.commands.
backend_result_set
(task_uuid, result, graph_uuid)¶ Command to set task result
Parameters: - task_uuid – Task uuid
- result – Result
- graph_uuid – Graph uuid
Returns: Return type:
-
gromozeka.concurrency.commands.
backend_results_del
(*task_uuids)¶ Command to delete results
Parameters: task_uuids – Task uuids Returns: Return type: Command
-
gromozeka.concurrency.commands.
broker_on_pool_size_changed
()¶ Command to change prefetch_count
Returns: Return type: Command
-
gromozeka.concurrency.commands.
broker_task_done
(task_uuid, broker_point, delivery_tag)¶ Command to acknowledge task message by it’s broker_adapter point, delivery_tag or scheduler_tag
Parameters: Returns: Return type:
-
gromozeka.concurrency.commands.
broker_task_register
(task_id, broker_point, options, deserializator)¶ Command to add new consumer with broker_point to task
Parameters: - task_id (str) – Unique task identification
- broker_point (primitives.BrokerPointType) – Broker entry
- options – Specific broker options. See NatsOptions broker for example
Returns: Return type:
-
gromozeka.concurrency.commands.
broker_task_reject
(task_uuid, broker_point, delivery_tag)¶ Command to reject task message by it’s broker_adapter point, delivery_tag or scheduler_tag
Parameters: Returns: Return type:
-
gromozeka.concurrency.commands.
broker_task_send
(task_uuid, request, broker_point, reply_to=None)¶ Command to publish primitives.Request to customer
Parameters: - task_uuid (str) – task identification
- request – Serialized request
- broker_point (gromozeka.BrokerPoint) –
- reply_to (gromozeka.BrokerPoint) –
Returns: Return type:
-
gromozeka.concurrency.commands.
broker_task_send_delayed
(task_uuid, request, broker_point, delay)¶ Command to publish primitives.Request to customer
Parameters: - task_uuid (str) – task identification
- request (primitives.Request) – Request to publish
- broker_point –
- delay –
Returns: Return type:
-
gromozeka.concurrency.commands.
pool_grow
(n)¶ Command to grow pool
Parameters: n – Number of workers to grow Returns: Return type: Command
-
gromozeka.concurrency.commands.
pool_remove_worker
(worker_ident)¶ Command to remove worker from pool
Parameters: worker_ident (int) – Worker identification Returns: Return type: Command
gromozeka.concurrency.pool module¶
-
class
gromozeka.concurrency.pool.
Pool
(max_workers=1, worker_class=<class 'gromozeka.concurrency.worker.ThreadWorker'>, logger=None)¶ Bases:
threading.Thread
Pool of workers
-
logger
¶ logging.Logger
– Class logger
-
worker_queue
¶ multiprocessing.Queue
– Queue to receive input requests
-
cmd
¶ multiprocessing.Queue
– Command queue
-
cmd_out
¶ multiprocessing.Queue
– Command result queue
-
worker_class
¶ gromozeka.concurrency.Worker
– worker class
-
workers
¶ list
ofgromozeka.concurrency.Worker
– list of worker instances
-
_stop_event
¶ multiprocessing.Event
– Stop event
Parameters: - max_workers (
int
, optional) – Number of workers to start - worker_class (
gromozeka.concurrency.Worker
, optional) – Worker class
-
cmd
-
cmd_out
-
init_max_workers
¶
-
is_on_stop
¶
-
listen_cmd
()¶ Listen to commands
-
logger
-
remove_worker
(worker_ident)¶ Remove worker from pool by it`s ident
Parameters: worker_ident – Worker identification
-
size
¶ Not command do not use this in workers
Returns:
-
start
()¶ Start pool
-
stop_
()¶
-
worker_cls
¶
-
worker_queue
-
workers
-
gromozeka.concurrency.scheduler module¶
Module for schedule tasks
-
class
gromozeka.concurrency.scheduler.
Scheduler
¶ Bases:
threading.Thread
Scheduler class
-
logger
¶ logging.Logger
– Class logger
-
_stop_event
¶ threading.Event
– Stop event
-
_shed
¶ sched.scheduler
– Scheduler instance
-
_event_queue
¶ queue.Queue
– Queue for scheduler events
-
add
(wait_time, func, args=None, kwargs=None, priority=1)¶ Add event to scheduler queue
Parameters: - wait_time (float) – Delay to run (seconds)
- func – Callable function to run
- args – Func arguments
- kwargs – Func keyword arguments
- priority – Event priority
-
listen_cmd
()¶ Listen to commands
-
logger
-
run
()¶ Run scheduler in thread
-
stop
()¶ Stop scheduler
-
-
gromozeka.concurrency.scheduler.
delay
(last_run=None, every=None, interval=None, at=None)¶ Make new delay from arguments
Examples
delay(every='days', interval=2) # This event will run every 2 days delay(every='weeks', interval=1, at='15:40') # This event will run every week at 15:40
Parameters: Returns: new event delay. Returns False when not need more event to run - last run.
Return type: int
or False
-
gromozeka.concurrency.scheduler.
delay_from_eta
(eta)¶ Make new delay from eta
Examples
delay_from_eta(countdown='2018-02-20 13:38:33.0') # This event will run 20.02.18 at 13:38:33 once
Parameters: eta ( str
) – If exceptions.Retry occurs other retry will run at this time. Other arguments ignored.Returns: Delay Return type: int
orNone
gromozeka.concurrency.worker module¶
-
class
gromozeka.concurrency.worker.
ProcessWorker
(pool)¶ Bases:
multiprocessing.context.Process
,gromozeka.concurrency.worker.Worker
Worker class for gromozeka.concurrency.Pool. Uses as mixin.
-
logger
¶ logging.Logger – class logger
-
pool
¶ gromozeka.concurrency.Pool – parent pool instance
-
_stop_event
¶ multiprocessing.Event or threading.Event
Parameters: pool (gromozeka.concurrency.Pool) – Parent pool instance -
run
()¶ Method to be run in sub-process; can be overridden in sub-class
-
stop
(timeout=None)¶ Stop worker. multiprocessing or threading part
-
-
class
gromozeka.concurrency.worker.
ThreadWorker
(pool)¶ Bases:
threading.Thread
,gromozeka.concurrency.worker.Worker
Worker class for gromozeka.concurrency.Pool. Uses as mixin.
-
logger
¶ logging.Logger – class logger
-
pool
¶ gromozeka.concurrency.Pool – parent pool instance
-
_stop_event
¶ multiprocessing.Event or threading.Event
Parameters: pool (gromozeka.concurrency.Pool) – Parent pool instance -
run
()¶ Method representing the thread’s activity.
You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.
-
stop
(timeout=None)¶ Stop worker. multiprocessing or threading part
-
-
class
gromozeka.concurrency.worker.
Worker
(pool)¶ Bases:
object
Worker class for gromozeka.concurrency.Pool. Uses as mixin.
-
logger
¶ logging.Logger – class logger
-
pool
¶ gromozeka.concurrency.Pool – parent pool instance
-
_stop_event
¶ multiprocessing.Event or threading.Event
Parameters: pool (gromozeka.concurrency.Pool) – Parent pool instance -
ident
= None¶
-
is_alive
()¶ Check worker is alive. multiprocessing or threading part
Returns: True if worker is alive or False Return type: bool
-
name
= None¶
-
run
()¶ Run worker in thread or process
Raises: exceptions.MaxRetriesExceedException
BaseException
-
stop
()¶ Stop worker. multiprocessing or threading part
-
Module contents¶
-
class
gromozeka.concurrency.
Pool
(max_workers=1, worker_class=<class 'gromozeka.concurrency.worker.ThreadWorker'>, logger=None)¶ Bases:
threading.Thread
Pool of workers
-
logger
¶ logging.Logger
– Class logger
-
worker_queue
¶ multiprocessing.Queue
– Queue to receive input requests
-
cmd
¶ multiprocessing.Queue
– Command queue
-
cmd_out
¶ multiprocessing.Queue
– Command result queue
-
worker_class
¶ gromozeka.concurrency.Worker
– worker class
-
workers
¶ list
ofgromozeka.concurrency.Worker
– list of worker instances
-
_stop_event
¶ multiprocessing.Event
– Stop event
Parameters: - max_workers (
int
, optional) – Number of workers to start - worker_class (
gromozeka.concurrency.Worker
, optional) – Worker class
-
cmd
-
cmd_out
-
init_max_workers
¶
-
is_on_stop
¶
-
listen_cmd
()¶ Listen to commands
-
logger
-
remove_worker
(worker_ident)¶ Remove worker from pool by it`s ident
Parameters: worker_ident – Worker identification
-
size
¶ Not command do not use this in workers
Returns:
-
start
()¶ Start pool
-
stop_
()¶
-
worker_cls
¶
-
worker_queue
-
workers
-
-
class
gromozeka.concurrency.
ThreadWorker
(pool)¶ Bases:
threading.Thread
,gromozeka.concurrency.worker.Worker
Worker class for gromozeka.concurrency.Pool. Uses as mixin.
-
logger
¶ logging.Logger – class logger
-
pool
¶ gromozeka.concurrency.Pool – parent pool instance
-
_stop_event
¶ multiprocessing.Event or threading.Event
Parameters: pool (gromozeka.concurrency.Pool) – Parent pool instance -
run
()¶ Method representing the thread’s activity.
You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.
-
stop
(timeout=None)¶ Stop worker. multiprocessing or threading part
-
-
class
gromozeka.concurrency.
ProcessWorker
(pool)¶ Bases:
multiprocessing.context.Process
,gromozeka.concurrency.worker.Worker
Worker class for gromozeka.concurrency.Pool. Uses as mixin.
-
logger
¶ logging.Logger – class logger
-
pool
¶ gromozeka.concurrency.Pool – parent pool instance
-
_stop_event
¶ multiprocessing.Event or threading.Event
Parameters: pool (gromozeka.concurrency.Pool) – Parent pool instance -
run
()¶ Method to be run in sub-process; can be overridden in sub-class
-
stop
(timeout=None)¶ Stop worker. multiprocessing or threading part
-
-
class
gromozeka.concurrency.
Scheduler
¶ Bases:
threading.Thread
Scheduler class
-
logger
¶ logging.Logger
– Class logger
-
_stop_event
¶ threading.Event
– Stop event
-
_shed
¶ sched.scheduler
– Scheduler instance
-
_event_queue
¶ queue.Queue
– Queue for scheduler events
-
add
(wait_time, func, args=None, kwargs=None, priority=1)¶ Add event to scheduler queue
Parameters: - wait_time (float) – Delay to run (seconds)
- func – Callable function to run
- args – Func arguments
- kwargs – Func keyword arguments
- priority – Event priority
-
listen_cmd
()¶ Listen to commands
-
logger
-
run
()¶ Run scheduler in thread
-
stop
()¶ Stop scheduler
-
-
class
gromozeka.concurrency.
Worker
(pool)¶ Bases:
object
Worker class for gromozeka.concurrency.Pool. Uses as mixin.
-
logger
¶ logging.Logger – class logger
-
pool
¶ gromozeka.concurrency.Pool – parent pool instance
-
_stop_event
¶ multiprocessing.Event or threading.Event
Parameters: pool (gromozeka.concurrency.Pool) – Parent pool instance -
ident
= None¶
-
is_alive
()¶ Check worker is alive. multiprocessing or threading part
Returns: True if worker is alive or False Return type: bool
-
name
= None¶
-
run
()¶ Run worker in thread or process
Raises: exceptions.MaxRetriesExceedException
BaseException
-
stop
()¶ Stop worker. multiprocessing or threading part
-