src.system.projection.processor.multiple_process.MultiProcessProjectionProcessor

class src.system.projection.processor.multiple_process.MultiProcessProjectionProcessor(projection_parameters: ProjectionParameters)

ProjectionProcessor that uses Python’s multiprocessing module to calculate Projections.

Inheritance Diagram

Variables & Properties

Methods

__init__(projection_parameters)	Constructor method.
progress_bar(out_queue, projection_count, cpus)	Progress bar display that monitors and updates based on items in a queue.
run_projection(projection)	Runs a single projection and writes output.
run_projections([cpus])	Runs projections in parallel, using Python's multiprocessing module, by:
setup_output()	Calls setup_output() for each projection in projections.
worker(in_queue, out_queue)	Worker that consumes work from a queue, and executes work.

Details

__init__(projection_parameters: ProjectionParameters)

Constructor method. Creates projections and root data sources.

Parameters:

projection_parameters – Parameters to initialize model objects.

data_sources: DataSourcesRoot

Data sources to be read at runtime.

static progress_bar(out_queue: Queue, projection_count: int, cpus: int) → None

Progress bar display that monitors and updates based on items in a queue.

Parameters:

• out_queue – Progress bar monitoring target.

• projection_count – Number of expected projections.

• cpus – Number of parallel processes.

Returns:

Nothing.

projection: Type

Projection class definition.

projection_parameters: ProjectionParameters

Parameters to initialize model objects.

projections: List[Projection]

List of projections to run.

static run_projection(projection: Projection) → None

Runs a single projection and writes output.

Parameters:

projection – Projection to run.

Returns:

Nothing.

run_projections(cpus: int = None) → None

Runs projections in parallel, using Python’s multiprocessing module, by:

1. Creating Queue objects.

2. Feeding projections into the Queue.

3. Feeding poison pills into the Queue, one for each worker process.

4. Spinning up workers using a Pool.

5. Processing all items in the Queue using the Pool.

Note

If cpus is None, allow the system to determine the number of CPU’s to use. Typically, this would be the physical core count - 1.

Parameters:

cpus – Number of processes to run in parallel.

Returns:

Nothing.

setup_output() → None

Calls setup_output() for each projection in projections.

Returns:

Nothing.

classmethod worker(in_queue: Queue, out_queue: Queue) → None

Worker that consumes work from a queue, and executes work. If the worker consumes a PoisonPill, the worker quits and “dies”.

Parameters:

• in_queue – Work input queue.

• out_queue – Work output queue.

Returns:

Nothing.