gromozeka.brokers package¶
Submodules¶
gromozeka.brokers.base module¶
-
class
gromozeka.brokers.base.
BrokerAdapter
(broker)¶ Bases:
gromozeka.concurrency.pool.Pool
-
listen_cmd
()¶ Listen to commands
-
on_pool_size_changed
()¶ This method will be run, when worker pool size change
-
start
()¶ Start pool
-
task_done
(task_uuid, broker_point, delivery_tag)¶ Parameters: - task_uuid (str) – task identification
- broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
- delivery_tag – Broker delivery tag (unique message identification)
-
task_register
(broker_point, task_id, options=None, deserializator=None)¶ Register task in broker
Parameters: - broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
- task_id (str) – Task identification
- options – Specific broker options. See NatsOptions broker for example
-
task_reject
(task_uuid, broker_point, delivery_tag)¶ Parameters: - task_uuid (str) – task identification
- broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
- delivery_tag – Broker delivery tag (unique message identification)
-
task_send
(task_uuid, request, broker_point, reply_to=None)¶ Parameters: - task_uuid (str) – task identification
- request – serialized request
- broker_point (gromozeka.BrokerPoint) –
-
-
class
gromozeka.brokers.base.
BrokerInterface
(app)¶ Bases:
object
-
configure
()¶ Configure broker
-
on_pool_size_changed
()¶ This method will be run, when worker pool size change
-
stop
()¶ Stop broker
-
task_done
(task_uuid, broker_point, delivery_tag)¶ Parameters: - task_uuid (str) – task identification
- broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
- delivery_tag – Broker delivery tag (unique message identification)
-
task_register
(broker_point, task_id, options, deserializator)¶ Parameters: - broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
- task_id (str) – Task identification
- options – Specific broker options.
- deserializator (gromozeka.primitives.base.TaskDeserializator) –
-
task_reject
(task_uuid, broker_point, delivery_tag)¶ Parameters: - task_uuid (str) – task identification
- broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
- delivery_tag – Broker delivery tag (unique message identification)
-
task_send
(task_uuid, request, broker_point, reply_to=None)¶ Parameters: - task_uuid (str) – task identification
- request – serialized request
- broker_point (gromozeka.BrokerPoint) –
- reply_to (gromozeka.BrokerPoint) –
-
task_send_delayed
(task_uuid, request, broker_point, delay)¶ Parameters: - task_uuid (str) – task identification
- request – serialized request
- broker_point (gromozeka.BrokerPoint) –
- delay –
Returns:
-
wait_for_start
()¶ Not required method for check broker is started
Returns:
-
static
worker_run
(self)¶ This method will be in thread - start to work here
-
-
gromozeka.brokers.base.
get_broker_factory
(app)¶ Parameters: app (gromozeka.Gromozeka) – Returns: Return type: gromozeka.brokers.base.BrokerAdapter
gromozeka.brokers.rabbit module¶
-
class
gromozeka.brokers.rabbit.
Consumer
(exchange, exchange_type, queue, routing_key, deserializator=None)¶ Bases:
object
-
acknowledge_message
(task_uuid, delivery_tag)¶ Acknowledge the message delivery from RabbitMQ by sending a Basic.Ack RPC method for the delivery tag.
Parameters: - task_uuid –
- delivery_tag (int) – The delivery tag from the Basic.Deliver frame
-
add_on_cancel_callback
()¶ Add a callback that will be invoked if RabbitMQ cancels the consumer for some reason. If RabbitMQ does cancel the consumer, on_consumer_cancelled will be invoked by pika.
-
add_on_channel_close_callback
()¶ This method tells pika to call the on_channel_closed method if RabbitMQ unexpectedly closes the channel.
-
add_task
(task_id)¶
-
change_prefetch_count
(new_size)¶
-
close_channel
()¶ Call to close the channel with RabbitMQ cleanly by issuing the Channel.Close RPC command.
-
on_bindok
(unused_frame)¶ Invoked by pika when the Queue.Bind method has completed. At this point we will start consuming messages by calling start_consuming which will invoke the needed RPC commands to start the process.
Parameters: unused_frame (pika.frame.Method) – The Queue.BindOk response frame
-
on_cancelok
(unused_frame)¶ This method is invoked by pika when RabbitMQ acknowledges the cancellation of a consumer. At this point we will close the channel. This will invoke the on_channel_closed method once the channel has been closed, which will in-turn close the connection.
Parameters: unused_frame (pika.frame.Method) – The Basic.CancelOk frame
-
on_channel_closed
(channel, reply_code, reply_text)¶ Invoked by pika when RabbitMQ unexpectedly closes the channel. Channels are usually closed if you attempt to do something that violates the protocol, such as re-declare an exchange or queue with different parameters. In this case, we’ll close the connection to shutdown the object.
Parameters:
-
on_channel_open
(channel)¶ This method is invoked by pika when the channel has been opened. The channel object is passed in so we can make use of it.
Since the channel is now open, we’ll declare the exchange to use.
Parameters: channel (pika.channel.Channel) – The closed channel
-
on_consumer_cancelled
(method_frame)¶ Invoked by pika when RabbitMQ sends a Basic.Cancel for a consumer receiving messages.
Parameters: method_frame (pika.frame.Method) – The Basic.Cancel frame
-
on_exchange_declareok
(unused_frame)¶ Invoked by pika when RabbitMQ has finished the Exchange.Declare RPC command.
Parameters: unused_frame (pika.Frame.Method) – Exchange.DeclareOk response frame
-
on_message
(unused_channel, basic_deliver, properties, body)¶ Invoked by pika when a message is delivered from RabbitMQ. The channel is passed for your convenience. The basic_deliver object that is passed in carries the exchange, routing key, delivery tag and a redelivered flag for the message. The properties passed in is an instance of BasicProperties with the message properties and the body is the message that was sent.
Parameters: - unused_channel (pika.channel.Channel) – The channel object
- basic_deliver (pika.Spec.Basic.Deliver) – Basic deliver method
- properties (pika.Spec.BasicProperties) – Properties
- body (str|bytes) – The message body
-
on_queue_declareok
(method_frame)¶ Method invoked by pika when the Queue.Declare RPC call made in setup_queue has completed. In this method we will bind the queue and exchange together with the routing key by issuing the Queue.Bind RPC command. When this command is complete, the on_bindok method will be invoked by pika.
Parameters: method_frame (pika.frame.Method) – The Queue.DeclareOk frame
-
open_channel
()¶ Open a new channel with RabbitMQ by issuing the Channel.Open RPC command. When RabbitMQ responds that the channel is open, the on_channel_open callback will be invoked by pika.
-
publish
(task_uuid, request, exchange=None, routing_key=None)¶ Parameters: Returns:
-
reject_message
(task_uuid, delivery_tag, requeue=False)¶ Reject message by it`s delivery tag
Parameters:
-
setup_exchange
(exchange_name)¶ Setup the exchange on RabbitMQ by invoking the Exchange.Declare RPC command. When it is complete, the on_exchange_declareok method will be invoked by pika.
Parameters: exchange_name (str) – The name of the exchange to declare
-
setup_queue
(queue_name)¶ Setup the queue on RabbitMQ by invoking the Queue.Declare RPC command. When it is complete, the on_queue_declareok method will be invoked by pika.
Parameters: queue_name (str|unicode) – The name of the queue to declare.
-
start_consuming
()¶ This method sets up the consumer by first calling add_on_cancel_callback so that the object is notified if RabbitMQ cancels the consumer. It then issues the Basic.Consume RPC command which returns the consumer tag that is used to uniquely identify the consumer with RabbitMQ. We keep the value to use it when we want to cancel consuming. The on_message method is passed in as a callback pika will invoke when a message is fully received.
-
stop_consuming
()¶ Tell RabbitMQ that you would like to stop consuming by sending the Basic.Cancel RPC command.
-
-
class
gromozeka.brokers.rabbit.
DelayedMessagePublisher
(task_uuid, broker_connection, request, original_exchange, original_routing_key, delay)¶ Bases:
object
-
add_on_cancel_callback
()¶ Add a callback that will be invoked if RabbitMQ cancels the consumer for some reason. If RabbitMQ does cancel the consumer, on_consumer_cancelled will be invoked by pika.
-
add_on_channel_close_callback
()¶ This method tells pika to call the on_channel_closed method if RabbitMQ unexpectedly closes the channel.
-
close_channel
()¶ Call to close the channel with RabbitMQ cleanly by issuing the Channel.Close RPC command.
-
exchange
= 'delayed'¶
-
exchange_type
= 'direct'¶
-
on_bindok
(unused_frame)¶ Invoked by pika when the Queue.Bind method has completed. At this point we will start consuming messages by calling start_consuming which will invoke the needed RPC commands to start the process.
Parameters: unused_frame (pika.frame.Method) – The Queue.BindOk response frame
-
on_cancelok
(unused_frame)¶ This method is invoked by pika when RabbitMQ acknowledges the cancellation of a consumer. At this point we will close the channel. This will invoke the on_channel_closed method once the channel has been closed, which will in-turn close the connection.
Parameters: unused_frame (pika.frame.Method) – The Basic.CancelOk frame
-
on_channel_closed
(channel, reply_code, reply_text)¶ Invoked by pika when RabbitMQ unexpectedly closes the channel. Channels are usually closed if you attempt to do something that violates the protocol, such as re-declare an exchange or queue with different parameters. In this case, we’ll close the connection to shutdown the object.
Parameters:
-
on_channel_open
(channel)¶ This method is invoked by pika when the channel has been opened. The channel object is passed in so we can make use of it.
Since the channel is now open, we’ll declare the exchange to use.
Parameters: channel (pika.channel.Channel) – The closed channel
-
on_consumer_cancelled
(method_frame)¶ Invoked by pika when RabbitMQ sends a Basic.Cancel for a consumer receiving messages.
Parameters: method_frame (pika.frame.Method) – The Basic.Cancel frame
-
on_exchange_declareok
(unused_frame)¶ Invoked by pika when RabbitMQ has finished the Exchange.Declare RPC command.
Parameters: unused_frame (pika.Frame.Method) – Exchange.DeclareOk response frame
-
on_queue_declareok
(method_frame)¶ Method invoked by pika when the Queue.Declare RPC call made in setup_queue has completed. In this method we will bind the queue and exchange together with the routing key by issuing the Queue.Bind RPC command. When this command is complete, the on_bindok method will be invoked by pika.
Parameters: method_frame (pika.frame.Method) – The Queue.DeclareOk frame
-
open_channel
()¶ Open a new channel with RabbitMQ by issuing the Channel.Open RPC command. When RabbitMQ responds that the channel is open, the on_channel_open callback will be invoked by pika.
-
publish
()¶
-
setup_exchange
(exchange_name)¶ Setup the exchange on RabbitMQ by invoking the Exchange.Declare RPC command. When it is complete, the on_exchange_declareok method will be invoked by pika.
Parameters: exchange_name (str) – The name of the exchange to declare
-
setup_queue
(queue_name)¶ Setup the queue on RabbitMQ by invoking the Queue.Declare RPC command. When it is complete, the on_queue_declareok method will be invoked by pika.
Parameters: queue_name (str|unicode) – The name of the queue to declare.
-
-
class
gromozeka.brokers.rabbit.
RabbitMQPikaAdaptee
(app)¶ Bases:
gromozeka.brokers.base.BrokerInterface
Pika adapted broker
-
reconnect_max_retries
¶ int – Maximum attempts to reconnect
-
reconnect_retry_countdown
¶ int|float – Maximum attempts to reconnect
-
connection
¶ pika.SelectConnection – Pika connection
-
add_on_connection_close_callback
()¶ This method adds an on close callback that will be invoked by pika when RabbitMQ closes the connection to the publisher unexpectedly.
-
close_connection
()¶ This method closes the connection to RabbitMQ.
-
configure
()¶ Configure broker
-
connect
()¶ This method connects to RabbitMQ, returning the connection handle. When the connection is established, the on_connection_open method will be invoked by pika.
Returns: Return type: pika.SelectConnection
-
get_consumer
(broker_point)¶
-
on_connection_closed
(connection, reply_code, reply_text)¶ This method is invoked by pika when the connection to RabbitMQ is closed unexpectedly. Since it is unexpected, we will reconnect to RabbitMQ if it disconnects.
Parameters:
-
on_connection_open
(unused_connection)¶ This method is called by pika once the connection to RabbitMQ has been established. It passes the handle to the connection object in case we need it, but in this case, we’ll just mark it unused.
Parameters: unused_connection (pika.SelectConnection) – Unused connection
-
on_pool_size_changed
()¶ This method will be run, when worker pool size change
-
reconnect
()¶ Will be invoked by the IOLoop timer if the connection is closed. See the on_connection_closed method.
-
stop
()¶ Stop broker
-
stop_serve
()¶ Cleanly shutdown the connection to RabbitMQ by stopping the consumers with RabbitMQ. When RabbitMQ confirms the cancellation, on_cancelok will be invoked by pika, which will then closing the channel and connection. The IOLoop is started again because this method is invoked when CTRL-C is pressed raising a KeyboardInterrupt exception. This exception stops the IOLoop which needs to be running for pika to communicate with RabbitMQ. All of the commands issued prior to starting the IOLoop will be buffered but not processed.
-
task_done
(task_uuid, broker_point, delivery_tag)¶ Args: task_uuid(str): task identification broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry delivery_tag: Broker delivery tag (unique message identification)
-
task_register
(broker_point, task_id, options, deserializator)¶ Args: broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry task_id(str): Task identification options: Specific broker options. deserializator(gromozeka.primitives.base.TaskDeserializator):
-
task_reject
(task_uuid, broker_point, delivery_tag)¶ Args: task_uuid(str): task identification broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry delivery_tag: Broker delivery tag (unique message identification)
-
task_send
(task_uuid, request, broker_point, reply_to=None)¶ Args: task_uuid(str): task identification request: serialized request broker_point(gromozeka.BrokerPoint): reply_to(gromozeka.BrokerPoint):
-
task_send_delayed
(task_uuid, request, broker_point, delay)¶ Parameters: - task_uuid (str) – task identification
- request – serialized request
- broker_point (gromozeka.BrokerPoint) –
- delay –
Returns:
-
wait_for_start
()¶ Not required method for check broker is started
Returns:
-
static
worker_run
(self)¶ Parameters: self (gromozeka.brokers.base.BrokerAdapter) – Returns:
-
-
gromozeka.brokers.rabbit.
format_consumer_id_from_broker_point
(broker_point)¶
Module contents¶
-
class
gromozeka.brokers.
BrokerAdapter
(broker)¶ Bases:
gromozeka.concurrency.pool.Pool
-
listen_cmd
()¶ Listen to commands
-
on_pool_size_changed
()¶ This method will be run, when worker pool size change
-
start
()¶ Start pool
-
task_done
(task_uuid, broker_point, delivery_tag)¶ Parameters: - task_uuid (str) – task identification
- broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
- delivery_tag – Broker delivery tag (unique message identification)
-
task_register
(broker_point, task_id, options=None, deserializator=None)¶ Register task in broker
Parameters: - broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
- task_id (str) – Task identification
- options – Specific broker options. See NatsOptions broker for example
-
task_reject
(task_uuid, broker_point, delivery_tag)¶ Parameters: - task_uuid (str) – task identification
- broker_point (gromozeka.primitives.protocol.ProtoBrokerPoint) – Broker entry
- delivery_tag – Broker delivery tag (unique message identification)
-
task_send
(task_uuid, request, broker_point, reply_to=None)¶ Parameters: - task_uuid (str) – task identification
- request – serialized request
- broker_point (gromozeka.BrokerPoint) –
-
-
class
gromozeka.brokers.
RabbitMQPikaAdaptee
(app)¶ Bases:
gromozeka.brokers.base.BrokerInterface
Pika adapted broker
-
reconnect_max_retries
¶ int – Maximum attempts to reconnect
-
reconnect_retry_countdown
¶ int|float – Maximum attempts to reconnect
-
connection
¶ pika.SelectConnection – Pika connection
-
add_on_connection_close_callback
()¶ This method adds an on close callback that will be invoked by pika when RabbitMQ closes the connection to the publisher unexpectedly.
-
close_connection
()¶ This method closes the connection to RabbitMQ.
-
configure
()¶ Configure broker
-
connect
()¶ This method connects to RabbitMQ, returning the connection handle. When the connection is established, the on_connection_open method will be invoked by pika.
Returns: Return type: pika.SelectConnection
-
get_consumer
(broker_point)¶
-
on_connection_closed
(connection, reply_code, reply_text)¶ This method is invoked by pika when the connection to RabbitMQ is closed unexpectedly. Since it is unexpected, we will reconnect to RabbitMQ if it disconnects.
Parameters:
-
on_connection_open
(unused_connection)¶ This method is called by pika once the connection to RabbitMQ has been established. It passes the handle to the connection object in case we need it, but in this case, we’ll just mark it unused.
Parameters: unused_connection (pika.SelectConnection) – Unused connection
-
on_pool_size_changed
()¶ This method will be run, when worker pool size change
-
reconnect
()¶ Will be invoked by the IOLoop timer if the connection is closed. See the on_connection_closed method.
-
stop
()¶ Stop broker
-
stop_serve
()¶ Cleanly shutdown the connection to RabbitMQ by stopping the consumers with RabbitMQ. When RabbitMQ confirms the cancellation, on_cancelok will be invoked by pika, which will then closing the channel and connection. The IOLoop is started again because this method is invoked when CTRL-C is pressed raising a KeyboardInterrupt exception. This exception stops the IOLoop which needs to be running for pika to communicate with RabbitMQ. All of the commands issued prior to starting the IOLoop will be buffered but not processed.
-
task_done
(task_uuid, broker_point, delivery_tag)¶ Args: task_uuid(str): task identification broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry delivery_tag: Broker delivery tag (unique message identification)
-
task_register
(broker_point, task_id, options, deserializator)¶ Args: broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry task_id(str): Task identification options: Specific broker options. deserializator(gromozeka.primitives.base.TaskDeserializator):
-
task_reject
(task_uuid, broker_point, delivery_tag)¶ Args: task_uuid(str): task identification broker_point(gromozeka.primitives.protocol.ProtoBrokerPoint): Broker entry delivery_tag: Broker delivery tag (unique message identification)
-
task_send
(task_uuid, request, broker_point, reply_to=None)¶ Args: task_uuid(str): task identification request: serialized request broker_point(gromozeka.BrokerPoint): reply_to(gromozeka.BrokerPoint):
-
task_send_delayed
(task_uuid, request, broker_point, delay)¶ Parameters: - task_uuid (str) – task identification
- request – serialized request
- broker_point (gromozeka.BrokerPoint) –
- delay –
Returns:
-
wait_for_start
()¶ Not required method for check broker is started
Returns:
-
static
worker_run
(self)¶ Parameters: self (gromozeka.brokers.base.BrokerAdapter) – Returns:
-
-
gromozeka.brokers.
get_broker_factory
(app)¶ Parameters: app (gromozeka.Gromozeka) – Returns: Return type: gromozeka.brokers.base.BrokerAdapter