Limit (Task Submission)

The previous section has shown how ecFlow provides simple load management.

Even when using limits, it is still possible to allow several hundred jobs to be submitted at once, which can cause problems:

Excessive disk/io in job generation
Server busy in job generation, and slow to respond to the GUI.
Overload queuing systems like PBS/SLURM

To avoid these issues, limits can also be used to restrict the number of submissions, and in this case, when the Job becomes active the limit token is released.

The following shows the use of inlimit -s option, which helps to control the number of tasks in the submitted state. There can be more than 2 active jobs, since only the number in the submitted state is controlled. The removal of the -s option would mean that only two active jobs would be allows to execute in parallel. The use of the -s allows the configuration of the suite, depending on the load the disk/io and queuing system can sustain.

The following is a simple illustration that modifies the previous example.

Modify the suite definition file, as follows:

# Definition of the suite test.
suite test
 edit ECF_INCLUDE "$HOME/course"
 edit ECF_HOME    "$HOME/course"
 limit l1 2
 family f5
     inlimit -s l1 # by default consume 1 token from the limit l1
     edit SLEEP 20
     task t1
     task t2
     task t3
     task t4
     task t5
     task t6
     task t7
     task t8
     task t9
 endfamily
endsuite

Listing 56 $HOME/course/test.py

import os
from ecflow import (
    Defs,
    Suite,
    Family,
    Task,
    Edit,
    Trigger,
    Complete,
    Event,
    Meter,
    Time,
    Day,
    Date,
    Label,
    RepeatString,
    RepeatInteger,
    RepeatDate,
    InLimit,
    Limit,
)


def create_family_f5():
    return Family(
        "f5",
        # limit_name(l1),limit_path(""),no_of_tokens_to_consume(1),limit node(False), limit submission(True)
        InLimit("l1", "", 1, False, True),
        Edit(SLEEP=20),
        [Task("t{}".format(i)) for i in range(1, 10)],
    )


print("Creating suite definition")
home = os.path.join(os.getenv("HOME"), "course")
defs = Defs(
    Suite(
        "test",
        Edit(ECF_INCLUDE=home, ECF_HOME=home),
        Limit("l1", 2),
        create_family_f5(),
    )
)
print(defs)

print("Checking job creation: .ecf -> .job0")
print(defs.check_job_creation())

print("Checking trigger expressions and inlimits")
assert len(defs.check()) == 0, defs.check()

print("Saving definition to file 'test.def'")
defs.save_as_defs("test.def")

What to do

Apply the changes to suite definition.
In ecflow_ui, observe the effects on the number of submitted tasks.
Change the value of the limit and inlimit, and observe the effects.