cerebralcortex.core.data_manager.object package

Submodules

cerebralcortex.core.data_manager.object.data module

class ObjectData(CC)

Bases: cerebralcortex.core.data_manager.object.storage_filesystem.FileSystemStorage

cerebralcortex.core.data_manager.object.storage_filesystem module

class FileSystemStorage

Bases: object

create_bucket(bucket_name: str) → bool

creates a bucket aka folder in object storage system.

Parameters:bucket_name (str) – name of the bucket Returns:True if bucket was successfully created. On failure, returns an error with dict {“error”:”error-message”} Return type:bool Raises:ValueError – Bucket name cannot be empty/None.

Examples

>>> CC = CerebralCortex("/directory/path/of/configs/")
>>> CC.create_bucket("live_data_folder")
>>> True

get_bucket_objects(bucket_name: str) → dict

returns a list of all objects stored in the specified Minio bucket

Parameters:bucket_name (str) – name of the bucket aka folder Returns:{bucket-objects: [{“object_name”:”“, “metadata”: {}}…], in case of an error {“error”: str} Return type:dict

get_buckets() → dict

returns all available buckets in an object storage

Returns:{bucket-name: str, [{“key”:”value”}]}, in case of an error {“error”: str} Return type:dict

get_object(bucket_name: str, object_name: str) → dict

Returns stored object (HttpResponse) :param bucket_name: :param object_name: :return: object (HttpResponse), in case of an error {“error”: str}

Parameters:

• bucket_name (str) – name of a bucket aka folder
• object_name (str) – name of an object that needs to be downloaded

Returns:

object that needs to be downloaded. If file does not exists then it returns an error {“error”: “File does not exist.”}

Return type:

file-object

Raises:

• ValueError – Missing bucket_name and object_name params.
• Exception – {“error”: “error-message”}

get_object_stats(bucket_name: str, object_name: str) → dict

Returns properties (e.g., object type, last modified etc.) of an object stored in a specified bucket

Parameters:

• bucket_name (str) – name of a bucket aka folder
• object_name (str) – name of an object

Returns:

information of an object (e.g., creation_date, object_size etc.). In case of an error {“error”: str}

Return type:

dict

Raises:

• ValueError – Missing bucket_name and object_name params.
• Exception – {“error”: “error-message”}

is_bucket(bucket_name: str) → bool

checks whether a bucket exist :param bucket_name: name of the bucket aka folder :type bucket_name: str

Returns:True if bucket exist or False otherwise. In case an error {“error”: str} Return type:bool Raises:ValueError – bucket_name cannot be None or empty.

is_object(bucket_name: str, object_name: str) → dict

checks whether an object exist in a bucket :param bucket_name: name of the bucket aka folder :type bucket_name: str :param object_name: name of the object :type object_name: str

Returns:True if object exist or False otherwise. In case an error {“error”: str} Return type:bool Raises:Excecption – if bucket_name and object_name are empty or None

upload_object(bucket_name: str, object_name: str, object_filepath: str) → bool

Upload an object in a bucket aka folder of object storage system.

Parameters:

• bucket_name (str) – name of the bucket
• object_name (str) – name of the object to be uploaded
• object_filepath (str) – it shall contain full path of a file with file name (e.g., /home/nasir/obj.zip)

Returns:

True if object successfully uploaded. On failure, returns an error with dict {“error”:”error-message”}

Return type:

bool

Raises:

ValueError – Bucket name cannot be empty/None.

cerebralcortex.core.data_manager.object.storage_minio module

class MinioHandler

Bases: object

Todo

For now, Minio is disabled as CC config doesn’t provide an option to use mutliple object-storage

create_bucket(bucket_name: str) → bool

creates a bucket aka folder in object storage system.

Parameters:bucket_name (str) – name of the bucket Returns:True if bucket was successfully created. On failure, returns an error with dict {“error”:”error-message”} Return type:bool Raises:ValueError – Bucket name cannot be empty/None.

Examples

>>> CC = CerebralCortex("/directory/path/of/configs/")
>>> CC.create_bucket("live_data_folder")
>>> True

get_bucket_objects(bucket_name: str) → dict

returns a list of all objects stored in the specified Minio bucket

Parameters:bucket_name (str) – name of the bucket aka folder Returns:{bucket-objects: [{“object_name”:”“, “metadata”: {}}…], in case of an error {“error”: str} Return type:dict

get_buckets() → List

returns all available buckets in an object storage

Returns:{bucket-name: str, [{“key”:”value”}]}, in case of an error {“error”: str} Return type:dict

get_object(bucket_name: str, object_name: str) → dict

Returns stored object (HttpResponse) :param bucket_name: :param object_name: :return: object (HttpResponse), in case of an error {“error”: str}

Parameters:

• bucket_name (str) – name of a bucket aka folder
• object_name (str) – name of an object that needs to be downloaded

Returns:

object that needs to be downloaded. If file does not exists then it returns an error {“error”: “File does not exist.”}

Return type:

file-object

Raises:

• ValueError – Missing bucket_name and object_name params.
• Exception – {“error”: “error-message”}

get_object_stats(bucket_name: str, object_name: str) → dict

Returns properties (e.g., object type, last modified etc.) of an object stored in a specified bucket

Parameters:

• bucket_name (str) – name of a bucket aka folder
• object_name (str) – name of an object

Returns:

information of an object (e.g., creation_date, object_size etc.). In case of an error {“error”: str}

Return type:

dict

Raises:

• ValueError – Missing bucket_name and object_name params.
• Exception – {“error”: “error-message”}

is_bucket(bucket_name: str) → bool

checks whether a bucket exist :param bucket_name: name of the bucket aka folder :type bucket_name: str

Returns:True if bucket exist or False otherwise. In case an error {“error”: str} Return type:bool Raises:ValueError – bucket_name cannot be None or empty.

is_object(bucket_name: str, object_name: str) → dict

checks whether an object exist in a bucket :param bucket_name: name of the bucket aka folder :type bucket_name: str :param object_name: name of the object :type object_name: str

Returns:True if object exist or False otherwise. In case an error {“error”: str} Return type:bool Raises:Excecption – if bucket_name and object_name are empty or None

upload_object(bucket_name: str, object_name: str, object_filepath: object) → bool

Upload an object in a bucket aka folder of object storage system.

Parameters:

• bucket_name (str) – name of the bucket
• object_name (str) – name of the object to be uploaded
• object_filepath (str) – it shall contain full path of a file with file name (e.g., /home/nasir/obj.zip)

Returns:

True if object successfully uploaded. On failure, returns an error with dict {“error”:”error-message”}

Return type:

bool

Raises:

• ValueError – Bucket name cannot be empty/None.
• Exception – if upload fails

upload_object_to_s3(bucket_name: str, object_name: str, file_data: object, obj_length: int) → bool

Upload an object in a bucket aka folder of object storage system.

Parameters:

• bucket_name (str) – name of the bucket
• object_name (str) – name of the object to be uploaded
• file_data (object) – object of a file
• obj_length (int) – size of an object

Returns:

True if object successfully uploaded. On failure, throws an exception

Return type:

bool

Raises:

Exception – if upload fails

Module contents