cerebralcortex.core.messaging_manager package

Submodules

cerebralcortex.core.messaging_manager.kafka_handler module

class KafkaHandler[source]

Bases: object

create_direct_kafka_stream(kafka_topic: str, ssc) → pyspark.streaming.kafka.KafkaDStream[source]

Create a direct stream to kafka topic. Supports only one topic at a time

Parameters:kafka_topic – kafka topic to create stream against
Raises:Exception – if direct stream cannot be created.

Todo

Enable logging of errors

produce_message(topic: str, msg: str)[source]

Publish a message on kafka message queue

Parameters:
  • topic (str) – name of the kafka topic
  • msg (dict) – message that needs to published on kafka
Returns:

True if successful. In case of failure, it returns an Exception message.
Return type:

bool
Raises:
  • ValueError – topic and message parameters cannot be empty or None.
  • Exception – Error publishing message. Topic: topic_name - error-message
subscribe_to_topic(topic: str) → dict[source]

Subscribe to kafka topic as a consumer

Parameters:topic (str) – name of the kafka topic
Yields:dict – kafka message
Raises:ValueError – Topic parameter is missing.

cerebralcortex.core.messaging_manager.messaging_queue module

class MessagingQueue(CC: object, auto_offset_reset: str = 'largest')[source]

Bases: cerebralcortex.core.messaging_manager.kafka_handler.KafkaHandler

Module contents