gromozeka.brokers package

Submodules

gromozeka.brokers.base module

class gromozeka.brokers.base.BrokerAdapter(broker)

Bases: gromozeka.concurrency.pool.Pool

listen_cmd()

Listen to commands

on_pool_size_changed()

This method will be run, when worker pool size change

start()

Start pool

task_done(task_uuid, broker_point, delivery_tag)
Parameters:
  • task_uuid (str) – task identification
  • broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
  • delivery_tag – Broker delivery tag (unique message identification)
task_register(broker_point, task_id, options=None, deserializator=None)

Register task in broker

Parameters:
  • broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
  • task_id (str) – Task identification
  • options – Specific broker options. See NatsOptions broker for example
task_reject(task_uuid, broker_point, delivery_tag)
Parameters:
  • task_uuid (str) – task identification
  • broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
  • delivery_tag – Broker delivery tag (unique message identification)
task_send(task_uuid, request, broker_point, reply_to=None)
Parameters:
task_send_delayed(task_uuid, request, broker_point, delay)
Parameters:
  • task_uuid (str) – task identification
  • request (gromozeka.primitives.protocol.Request) – Request object
  • broker_point –
  • delay –
class gromozeka.brokers.base.BrokerInterface(app)

Bases: object

configure()

Configure broker

on_pool_size_changed()

This method will be run, when worker pool size change

stop()

Stop broker

task_done(task_uuid, broker_point, delivery_tag)
Parameters:
  • task_uuid (str) – task identification
  • broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
  • delivery_tag – Broker delivery tag (unique message identification)
task_register(broker_point, task_id, options, deserializator)
Parameters:
task_reject(task_uuid, broker_point, delivery_tag)
Parameters:
  • task_uuid (str) – task identification
  • broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
  • delivery_tag – Broker delivery tag (unique message identification)
task_send(task_uuid, request, broker_point, reply_to=None)
Parameters:
task_send_delayed(task_uuid, request, broker_point, delay)
Parameters:
  • task_uuid (str) – task identification
  • request – serialized request
  • broker_point (gromozeka.BrokerPoint) –
  • delay –

Returns:

wait_for_start()

Not required method for check broker is started

Returns:

static worker_run(self)

This method will be in thread - start to work here

gromozeka.brokers.base.get_broker_factory(app)
Parameters:app (gromozeka.Gromozeka) –
Returns:
Return type:gromozeka.brokers.base.BrokerAdapter

gromozeka.brokers.rabbit module

class gromozeka.brokers.rabbit.Consumer(exchange, exchange_type, queue, routing_key, deserializator=None)

Bases: object

acknowledge_message(task_uuid, delivery_tag)

Acknowledge the message delivery from RabbitMQ by sending a Basic.Ack RPC method for the delivery tag.

Parameters:
  • task_uuid –
  • delivery_tag (int) – The delivery tag from the Basic.Deliver frame
add_on_cancel_callback()

Add a callback that will be invoked if RabbitMQ cancels the consumer for some reason. If RabbitMQ does cancel the consumer, on_consumer_cancelled will be invoked by pika.

add_on_channel_close_callback()

This method tells pika to call the on_channel_closed method if RabbitMQ unexpectedly closes the channel.

add_task(task_id)
change_prefetch_count(new_size)
close_channel()

Call to close the channel with RabbitMQ cleanly by issuing the Channel.Close RPC command.

on_bindok(unused_frame)

Invoked by pika when the Queue.Bind method has completed. At this point we will start consuming messages by calling start_consuming which will invoke the needed RPC commands to start the process.

Parameters:unused_frame (pika.frame.Method) – The Queue.BindOk response frame
on_cancelok(unused_frame)

This method is invoked by pika when RabbitMQ acknowledges the cancellation of a consumer. At this point we will close the channel. This will invoke the on_channel_closed method once the channel has been closed, which will in-turn close the connection.

Parameters:unused_frame (pika.frame.Method) – The Basic.CancelOk frame
on_channel_closed(channel, reply_code, reply_text)

Invoked by pika when RabbitMQ unexpectedly closes the channel. Channels are usually closed if you attempt to do something that violates the protocol, such as re-declare an exchange or queue with different parameters. In this case, we’ll close the connection to shutdown the object.

Parameters:
  • channel (pika.channel.Channel) – The closed channel
  • reply_code (int) – The numeric reason the channel was closed
  • reply_text (str) – The text reason the channel was closed
on_channel_open(channel)

This method is invoked by pika when the channel has been opened. The channel object is passed in so we can make use of it.

Since the channel is now open, we’ll declare the exchange to use.

Parameters:channel (pika.channel.Channel) – The closed channel
on_consumer_cancelled(method_frame)

Invoked by pika when RabbitMQ sends a Basic.Cancel for a consumer receiving messages.

Parameters:method_frame (pika.frame.Method) – The Basic.Cancel frame
on_exchange_declareok(unused_frame)

Invoked by pika when RabbitMQ has finished the Exchange.Declare RPC command.

Parameters:unused_frame (pika.Frame.Method) – Exchange.DeclareOk response frame
on_message(unused_channel, basic_deliver, properties, body)

Invoked by pika when a message is delivered from RabbitMQ. The channel is passed for your convenience. The basic_deliver object that is passed in carries the exchange, routing key, delivery tag and a redelivered flag for the message. The properties passed in is an instance of BasicProperties with the message properties and the body is the message that was sent.

Parameters:
  • unused_channel (pika.channel.Channel) – The channel object
  • basic_deliver (pika.Spec.Basic.Deliver) – Basic deliver method
  • properties (pika.Spec.BasicProperties) – Properties
  • body (str|bytes) – The message body
on_queue_declareok(method_frame)

Method invoked by pika when the Queue.Declare RPC call made in setup_queue has completed. In this method we will bind the queue and exchange together with the routing key by issuing the Queue.Bind RPC command. When this command is complete, the on_bindok method will be invoked by pika.

Parameters:method_frame (pika.frame.Method) – The Queue.DeclareOk frame
open_channel()

Open a new channel with RabbitMQ by issuing the Channel.Open RPC command. When RabbitMQ responds that the channel is open, the on_channel_open callback will be invoked by pika.

publish(task_uuid, request, exchange=None, routing_key=None)
Parameters:
  • task_uuid –
  • request –
  • exchange (str) –
  • routing_key (str) –

Returns:

reject_message(task_uuid, delivery_tag, requeue=False)

Reject message by it`s delivery tag

Parameters:
  • task_uuid (str) –
  • delivery_tag (int) –
  • requeue (bool) – If True message will reject with requeue
setup_exchange(exchange_name)

Setup the exchange on RabbitMQ by invoking the Exchange.Declare RPC command. When it is complete, the on_exchange_declareok method will be invoked by pika.

Parameters:exchange_name (str) – The name of the exchange to declare
setup_queue(queue_name)

Setup the queue on RabbitMQ by invoking the Queue.Declare RPC command. When it is complete, the on_queue_declareok method will be invoked by pika.

Parameters:queue_name (str|unicode) – The name of the queue to declare.
start_consuming()

This method sets up the consumer by first calling add_on_cancel_callback so that the object is notified if RabbitMQ cancels the consumer. It then issues the Basic.Consume RPC command which returns the consumer tag that is used to uniquely identify the consumer with RabbitMQ. We keep the value to use it when we want to cancel consuming. The on_message method is passed in as a callback pika will invoke when a message is fully received.

stop_consuming()

Tell RabbitMQ that you would like to stop consuming by sending the Basic.Cancel RPC command.

class gromozeka.brokers.rabbit.DelayedMessagePublisher(task_uuid, broker_connection, request, original_exchange, original_routing_key, delay)

Bases: object

add_on_cancel_callback()

Add a callback that will be invoked if RabbitMQ cancels the consumer for some reason. If RabbitMQ does cancel the consumer, on_consumer_cancelled will be invoked by pika.

add_on_channel_close_callback()

This method tells pika to call the on_channel_closed method if RabbitMQ unexpectedly closes the channel.

close_channel()

Call to close the channel with RabbitMQ cleanly by issuing the Channel.Close RPC command.

exchange = 'delayed'
exchange_type = 'direct'
on_bindok(unused_frame)

Invoked by pika when the Queue.Bind method has completed. At this point we will start consuming messages by calling start_consuming which will invoke the needed RPC commands to start the process.

Parameters:unused_frame (pika.frame.Method) – The Queue.BindOk response frame
on_cancelok(unused_frame)

This method is invoked by pika when RabbitMQ acknowledges the cancellation of a consumer. At this point we will close the channel. This will invoke the on_channel_closed method once the channel has been closed, which will in-turn close the connection.

Parameters:unused_frame (pika.frame.Method) – The Basic.CancelOk frame
on_channel_closed(channel, reply_code, reply_text)

Invoked by pika when RabbitMQ unexpectedly closes the channel. Channels are usually closed if you attempt to do something that violates the protocol, such as re-declare an exchange or queue with different parameters. In this case, we’ll close the connection to shutdown the object.

Parameters:
  • channel (pika.channel.Channel) – The closed channel
  • reply_code (int) – The numeric reason the channel was closed
  • reply_text (str) – The text reason the channel was closed
on_channel_open(channel)

This method is invoked by pika when the channel has been opened. The channel object is passed in so we can make use of it.

Since the channel is now open, we’ll declare the exchange to use.

Parameters:channel (pika.channel.Channel) – The closed channel
on_consumer_cancelled(method_frame)

Invoked by pika when RabbitMQ sends a Basic.Cancel for a consumer receiving messages.

Parameters:method_frame (pika.frame.Method) – The Basic.Cancel frame
on_exchange_declareok(unused_frame)

Invoked by pika when RabbitMQ has finished the Exchange.Declare RPC command.

Parameters:unused_frame (pika.Frame.Method) – Exchange.DeclareOk response frame
on_queue_declareok(method_frame)

Method invoked by pika when the Queue.Declare RPC call made in setup_queue has completed. In this method we will bind the queue and exchange together with the routing key by issuing the Queue.Bind RPC command. When this command is complete, the on_bindok method will be invoked by pika.

Parameters:method_frame (pika.frame.Method) – The Queue.DeclareOk frame
open_channel()

Open a new channel with RabbitMQ by issuing the Channel.Open RPC command. When RabbitMQ responds that the channel is open, the on_channel_open callback will be invoked by pika.

publish()
setup_exchange(exchange_name)

Setup the exchange on RabbitMQ by invoking the Exchange.Declare RPC command. When it is complete, the on_exchange_declareok method will be invoked by pika.

Parameters:exchange_name (str) – The name of the exchange to declare
setup_queue(queue_name)

Setup the queue on RabbitMQ by invoking the Queue.Declare RPC command. When it is complete, the on_queue_declareok method will be invoked by pika.

Parameters:queue_name (str|unicode) – The name of the queue to declare.
class gromozeka.brokers.rabbit.RabbitMQPikaAdaptee(app)

Bases: gromozeka.brokers.base.BrokerInterface

Pika adapted broker

reconnect_max_retries

int – Maximum attempts to reconnect

reconnect_retry_countdown

int|float – Maximum attempts to reconnect

connection

pika.SelectConnection – Pika connection

add_on_connection_close_callback()

This method adds an on close callback that will be invoked by pika when RabbitMQ closes the connection to the publisher unexpectedly.

close_connection()

This method closes the connection to RabbitMQ.

configure()

Configure broker

connect()

This method connects to RabbitMQ, returning the connection handle. When the connection is established, the on_connection_open method will be invoked by pika.

Returns:
Return type:pika.SelectConnection
get_consumer(broker_point)
on_connection_closed(connection, reply_code, reply_text)

This method is invoked by pika when the connection to RabbitMQ is closed unexpectedly. Since it is unexpected, we will reconnect to RabbitMQ if it disconnects.

Parameters:
  • connection (pika.connection.Connection) – The closed connection obj
  • reply_code (int) – Reply code if given
  • reply_text (str) – The server provided reply_text if given
on_connection_open(unused_connection)

This method is called by pika once the connection to RabbitMQ has been established. It passes the handle to the connection object in case we need it, but in this case, we’ll just mark it unused.

Parameters:unused_connection (pika.SelectConnection) – Unused connection
on_pool_size_changed()

This method will be run, when worker pool size change

reconnect()

Will be invoked by the IOLoop timer if the connection is closed. See the on_connection_closed method.

stop()

Stop broker

stop_serve()

Cleanly shutdown the connection to RabbitMQ by stopping the consumers with RabbitMQ. When RabbitMQ confirms the cancellation, on_cancelok will be invoked by pika, which will then closing the channel and connection. The IOLoop is started again because this method is invoked when CTRL-C is pressed raising a KeyboardInterrupt exception. This exception stops the IOLoop which needs to be running for pika to communicate with RabbitMQ. All of the commands issued prior to starting the IOLoop will be buffered but not processed.

task_done(task_uuid, broker_point, delivery_tag)

Args: task_uuid(str): task identification broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry delivery_tag: Broker delivery tag (unique message identification)

task_register(broker_point, task_id, options, deserializator)

Args: broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry task_id(str): Task identification options: Specific broker options. deserializator(gromozeka.primitives.base.TaskDeserializator):

task_reject(task_uuid, broker_point, delivery_tag)

Args: task_uuid(str): task identification broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry delivery_tag: Broker delivery tag (unique message identification)

task_send(task_uuid, request, broker_point, reply_to=None)

Args: task_uuid(str): task identification request: serialized request broker_point(gromozeka.BrokerPoint): reply_to(gromozeka.BrokerPoint):

task_send_delayed(task_uuid, request, broker_point, delay)
Parameters:
  • task_uuid (str) – task identification
  • request – serialized request
  • broker_point (gromozeka.BrokerPoint) –
  • delay –

Returns:

wait_for_start()

Not required method for check broker is started

Returns:

static worker_run(self)
Parameters:self (gromozeka.brokers.base.BrokerAdapter) –

Returns:

gromozeka.brokers.rabbit.format_consumer_id_from_broker_point(broker_point)

Module contents

class gromozeka.brokers.BrokerAdapter(broker)

Bases: gromozeka.concurrency.pool.Pool

listen_cmd()

Listen to commands

on_pool_size_changed()

This method will be run, when worker pool size change

start()

Start pool

task_done(task_uuid, broker_point, delivery_tag)
Parameters:
  • task_uuid (str) – task identification
  • broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
  • delivery_tag – Broker delivery tag (unique message identification)
task_register(broker_point, task_id, options=None, deserializator=None)

Register task in broker

Parameters:
  • broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
  • task_id (str) – Task identification
  • options – Specific broker options. See NatsOptions broker for example
task_reject(task_uuid, broker_point, delivery_tag)
Parameters:
  • task_uuid (str) – task identification
  • broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
  • delivery_tag – Broker delivery tag (unique message identification)
task_send(task_uuid, request, broker_point, reply_to=None)
Parameters:
task_send_delayed(task_uuid, request, broker_point, delay)
Parameters:
  • task_uuid (str) – task identification
  • request (gromozeka.primitives.protocol.Request) – Request object
  • broker_point –
  • delay –
class gromozeka.brokers.RabbitMQPikaAdaptee(app)

Bases: gromozeka.brokers.base.BrokerInterface

Pika adapted broker

reconnect_max_retries

int – Maximum attempts to reconnect

reconnect_retry_countdown

int|float – Maximum attempts to reconnect

connection

pika.SelectConnection – Pika connection

add_on_connection_close_callback()

This method adds an on close callback that will be invoked by pika when RabbitMQ closes the connection to the publisher unexpectedly.

close_connection()

This method closes the connection to RabbitMQ.

configure()

Configure broker

connect()

This method connects to RabbitMQ, returning the connection handle. When the connection is established, the on_connection_open method will be invoked by pika.

Returns:
Return type:pika.SelectConnection
get_consumer(broker_point)
on_connection_closed(connection, reply_code, reply_text)

This method is invoked by pika when the connection to RabbitMQ is closed unexpectedly. Since it is unexpected, we will reconnect to RabbitMQ if it disconnects.

Parameters:
  • connection (pika.connection.Connection) – The closed connection obj
  • reply_code (int) – Reply code if given
  • reply_text (str) – The server provided reply_text if given
on_connection_open(unused_connection)

This method is called by pika once the connection to RabbitMQ has been established. It passes the handle to the connection object in case we need it, but in this case, we’ll just mark it unused.

Parameters:unused_connection (pika.SelectConnection) – Unused connection
on_pool_size_changed()

This method will be run, when worker pool size change

reconnect()

Will be invoked by the IOLoop timer if the connection is closed. See the on_connection_closed method.

stop()

Stop broker

stop_serve()

Cleanly shutdown the connection to RabbitMQ by stopping the consumers with RabbitMQ. When RabbitMQ confirms the cancellation, on_cancelok will be invoked by pika, which will then closing the channel and connection. The IOLoop is started again because this method is invoked when CTRL-C is pressed raising a KeyboardInterrupt exception. This exception stops the IOLoop which needs to be running for pika to communicate with RabbitMQ. All of the commands issued prior to starting the IOLoop will be buffered but not processed.

task_done(task_uuid, broker_point, delivery_tag)

Args: task_uuid(str): task identification broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry delivery_tag: Broker delivery tag (unique message identification)

task_register(broker_point, task_id, options, deserializator)

Args: broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry task_id(str): Task identification options: Specific broker options. deserializator(gromozeka.primitives.base.TaskDeserializator):

task_reject(task_uuid, broker_point, delivery_tag)

Args: task_uuid(str): task identification broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry delivery_tag: Broker delivery tag (unique message identification)

task_send(task_uuid, request, broker_point, reply_to=None)

Args: task_uuid(str): task identification request: serialized request broker_point(gromozeka.BrokerPoint): reply_to(gromozeka.BrokerPoint):

task_send_delayed(task_uuid, request, broker_point, delay)
Parameters:
  • task_uuid (str) – task identification
  • request – serialized request
  • broker_point (gromozeka.BrokerPoint) –
  • delay –

Returns:

wait_for_start()

Not required method for check broker is started

Returns:

static worker_run(self)
Parameters:self (gromozeka.brokers.base.BrokerAdapter) –

Returns:

gromozeka.brokers.get_broker_factory(app)
Parameters:app (gromozeka.Gromozeka) –
Returns:
Return type:gromozeka.brokers.base.BrokerAdapter
class gromozeka.brokers.LatexSema(sema_cls, value)

Bases: object

acquire()
change(delta)
release()