Limit
Limits provide simple load management by limiting the number of tasks submitted by a specific ecflow_server
At ECMWF, suite designers tend to use triggers in two different ways: to represent data dependency, or as courtesy triggers (i.e. a means to manage resources). Triggers where originally designed to represent data dependency, but can artificially prevent too many jobs from executing at once and thus be used to manage queues.
The use of triggers to manage queues, although possible, is undesired as it results in suites difficult to maintain.
The concept of limit was introduced as better and first class alternative to manage limited resources. Limits are declared with the limit keyword.
InLimit
Limits are used in conjunction with inlimit, which defines a need to consider the referred limit.
A limit must be defined using limit NAME N – the limit definition is typically placed at the suite scope.
Then, a limit can be imposed on a group of tasks by attaching inlimit NAME attribute to the restricted nodes.
Attaching the attribute to a task adds the task to the group. Attaching it to a family adds all tasks from that family.
The effect of a limit is that no more than N tasks of a group will run at once.
A node can be limited by several limits.
Suite Definition
Create family f5 with nine tasks, modifying the suite definition file, as follows:
# Definition of the suite test.
suite test
edit ECF_INCLUDE "$HOME/course"
edit ECF_HOME "$HOME/course"
limit l1 2
family f5
inlimit l1
edit SLEEP 20
task t1
task t2
task t3
task t4
task t5
task t6
task t7
task t8
task t9
endfamily
endsuite
import os
from ecflow import (
Defs,
Suite,
Family,
Task,
Edit,
Trigger,
Complete,
Event,
Meter,
Time,
Day,
Date,
Label,
RepeatString,
RepeatInteger,
RepeatDate,
InLimit,
Limit,
)
def create_family_f5():
return Family(
"f5",
InLimit("l1"),
Edit(SLEEP=20),
[Task("t{}".format(i)) for i in range(1, 10)],
)
print("Creating suite definition")
home = os.path.join(os.getenv("HOME"), "course")
defs = Defs(
Suite(
"test",
Edit(ECF_INCLUDE=home, ECF_HOME=home),
Limit("l1", 2),
create_family_f5(),
)
)
print(defs)
print("Checking job creation: .ecf -> .job0")
print(defs.check_job_creation())
print("Checking trigger expressions and inlimits")
assert len(defs.check()) == 0, defs.check()
print("Saving definition to file 'test.def'")
defs.save_as_defs("test.def")
Task script
Create new task scripts in $HOME/course/test/f5/ directory, each one containing:
%include <head.h>
echo "I will now sleep for %SLEEP% seconds"
sleep %SLEEP%
%include <tail.h>
What to do
Apply the changes to suite definition.
Apply the changes to task script.
In ecflow_ui