ecflow.Defs /////////// .. py:class:: Defs :module: ecflow Bases: :py:class:`~Boost.Python.instance` The Defs class holds the :term:`suite definition` structure. It contains all the :py:class:`ecflow.Suite` and hence acts like the root for suite node tree hierarchy. The definition can be kept as python code, alternatively it can be saved as a flat ASCII definition file. If a definition is read in from disk, it will by default, check the :term:`trigger` expressions. If however the definition is created in python, then checking should be done explicitly: Defs(string) string - The Defs class take one argument which represents the file name Defs(Suite | Edit ) :py:class:`ecflow.Suite`- One or more suites :py:class:`ecflow.Edit` - specifies user defined server variables Example: .. code-block:: python # Build definition using Constructor approach, This allows indentation, to show the structure # This is a made up example to demonstrate suite construction: defs = Defs( Edit(SLEEP=10,FRED='bill'), # user defined server variables Suite('s1' Clock(1, 1, 2010, False), Autocancel(1, 10, True), Task('t1' Edit({'a':'12', 'b':'bb'}, c='v',d='b'), Edit(g='d'), Edit(h=1), Event(1), Event(11,'event'), Meter('meter',0,10,10), Label('label','c'), Trigger('1==1'), Complete('1==1'), Limit('limit',10),Limit('limit2',10), InLimit('limitName','/limit',2), Defstatus(DState.complete), Today(0,30),Today('00:59'),Today('00:00 11:30 00:01'), Time(0,30),Time('00:59'),Time('00:00 11:30 00:01'), Day('sunday'),Day(Days.monday), Date(1,1,0),Date(28,2,1960), Autocancel(3) ), [ Family('f{}'.format(i)) for i in range(1,6)])) defs.save_as_defs('filename.def') # save defs into file defs = Defs() # create an empty defs suite = defs.add_suite('s1') family = suite.add_family('f1') for i in [ '_1', '_2', '_3' ]: family.add_task( 't' + i ) defs.save_as_defs('filename.def') # save defs into file Create a Defs from an existing file on disk: .. code-block:: python defs = Defs('filename.def') # Will open and parse the file and create the Definition print(defs) .. py:method:: Defs.add :module: ecflow object add(tuple args, dict kwds) : add(..) provides a way to append Nodes and attributes This is best illustrated with an example: .. code-block:: python defs = Defs().add( Suite('s1').add( Clock(1, 1, 2010, False), Autocancel(1, 10, True), Task('t1').add( Edit({'a':'12', 'b':'bb'}, c='v',d='b'), Edit(g='d'), Edit(h=1), Event(1), Event(11,'event'), Meter('meter',0,10,10), Label('label','c'), Trigger('1==1'), Complete('1==1'), Limit('limit',10),Limit('limit2',10), InLimit('limitName','/limit',2), Defstatus(DState.complete), Today(0,30),Today('00:59'),Today('00:00 11:30 00:01'), Time(0,30),Time('00:59'),Time('00:00 11:30 00:01'), Day('sunday'),Day(Days.monday), Date(1,1,0),Date(28,2,1960), Autocancel(3) ), [ Family('f{}'.format(i)) for i in range(1,6)])) We can also use '+=' with a list here are a few examples: .. code-block:: python defs = Defs(); defs += [ Suite('s2'),Edit({ 'x1':'y', 'aa1':'bb'}, a='v',b='b') ] .. code-block:: python defs += [ Suite('s{}'.format(i)) for i in range(1,6) ] .. code-block:: python defs = Defs() defs += [ Suite('suite').add( Task('x'), Family('f').add( [ Task('t{}'.format(i)) for i in range(1,6)] ), Task('y'), [ Family('f{}'.format(i)) for i in range(1,6) ], Edit(a='b'), [ Task('t{}'.format(i)) for i in range(1,6) ], )] It is also possible to use '+' .. code-block:: python defs = Defs() + Suite('s1') defs.s1 += Autocancel(1, 10, True) defs.s1 += Task('t1') + Edit({ 'e':1, 'f':'bb'}) +\ Event(1) + Event(11,'event') + Meter('meter',0,10,10) + Label('label','c') + Trigger('1==1') +\ Complete('1==1') + Limit('limit',10) + Limit('limit2',10) + InLimit('limitName','/limit',2) +\ Defstatus(DState.complete) + Today(0,30) + Today('00:59') + Today('00:00 11:30 00:01') +\ Time(0,30) + Time('00:59') + Time('00:00 11:30 00:01') + Day('sunday') + Day(Days.monday) +\ Date(1,1,0) + Date(28,2,1960) + Autocancel(3) .. warning:: We can only use '+' when the left most object is a node, i.e Task('t1') in this case .. py:method:: Defs.add_extern( (Defs)arg1, (str)arg2) -> None : :module: ecflow :term:`extern` refer to nodes that have not yet been defined typically due to cross suite :term:`dependencies` :term:`trigger` and :term:`complete expression`\ s may refer to paths, and variables in other suites, that have not been loaded yet. The references to node paths and variable must exist, or exist as externs Externs can be added manually or automatically. Manual Method: .. code-block:: python void add_extern(string nodePath ) Usage: .. code-block:: python defs = Defs('file.def') .... defs.add_extern('/temp/bill:event_name') defs.add_extern('/temp/bill:meter_name') defs.add_extern('/temp/bill:repeat_name') defs.add_extern('/temp/bill:edit_name') defs.add_extern('/temp/bill') Automatic Method: This will scan all trigger and complete expressions, looking for paths and variables that have not been defined. The added benefit of this approach is that duplicates will not be added. It is the user's responsibility to check that extern's are eventually defined otherwise trigger expression will not evaluate correctly .. code-block:: python void auto_add_externs(bool remove_existing_externs_first ) Usage: .. code-block:: python defs = Defs('file.def') ... defs.auto_add_externs(True) # remove existing extern first. .. py:method:: Defs.add_suite( (Defs)arg1, (Suite)arg2) -> Suite : :module: ecflow Add a :term:`suite` :term:`node`. See :py:class:`ecflow.Suite` If a new suite is added which matches the name of an existing suite, then an exception is thrown. Exception: - Throws RuntimeError is the suite name is not valid - Throws RuntimeError if duplicate suite is added Usage: .. code-block:: python defs = Defs() # create a empty defs suite = Suite('suite') # create a stand alone Suite defs.add_suite(suite) # add suite to defs s2 = defs.add_suite('s2') # create a suite and add to defs # Alternatively we can create Suite in place defs = Defs( Suite('s1', Family('f1', Task('t1'))), Suite('s2', Family('f1', Task('t1')))) add_suite( (Defs)arg1, (str)arg2) -> Suite : Create a empty Defs .. py:method:: Defs.add_variable( (Defs)arg1, (str)arg2, (str)arg3) -> Defs : :module: ecflow Adds a name value :term:`variable`. Also see :py:class:`ecflow.Edit` This defines a variable for use in :term:`variable substitution` in a :term:`ecf script` file. There can be any number of variables. The variables are names inside a pair of '%' characters in an :term:`ecf script`. The name are case sensitive. Special character in the value, must be placed inside single quotes if misinterpretation is to be avoided. The value of the variable replaces the variable name in the :term:`ecf script` at `job creation` time. The variable names for any given node must be unique. If duplicates are added then the the last value added is kept. Exception: - Writes warning to standard output, if a duplicate variable name is added Usage: .. code-block:: python task.add_variable( Variable('ECF_HOME','/tmp/')) task.add_variable( 'TMPDIR','/tmp/') task.add_variable( 'COUNT',2) a_dict = { 'name':'value', 'name2':'value2', 'name3':'value3' } task.add_variable(a_dict) add_variable( (Defs)arg1, (str)arg2, (int)arg3) -> Defs add_variable( (Defs)arg1, (Variable)arg2) -> Defs add_variable( (Defs)arg1, (dict)arg2) -> Defs .. py:method:: Defs.auto_add_externs( (Defs)arg1, (bool)arg2) -> None : :module: ecflow :term:`extern` refer to nodes that have not yet been defined typically due to cross suite :term:`dependencies` :term:`trigger` and :term:`complete expression`\ s may refer to paths, and variables in other suites, that have not been loaded yet. The references to node paths and variable must exist, or exist as externs Externs can be added manually or automatically. Manual Method: .. code-block:: python void add_extern(string nodePath ) Usage: .. code-block:: python defs = Defs('file.def') .... defs.add_extern('/temp/bill:event_name') defs.add_extern('/temp/bill:meter_name') defs.add_extern('/temp/bill:repeat_name') defs.add_extern('/temp/bill:edit_name') defs.add_extern('/temp/bill') Automatic Method: This will scan all trigger and complete expressions, looking for paths and variables that have not been defined. The added benefit of this approach is that duplicates will not be added. It is the user's responsibility to check that extern's are eventually defined otherwise trigger expression will not evaluate correctly .. code-block:: python void auto_add_externs(bool remove_existing_externs_first ) Usage: .. code-block:: python defs = Defs('file.def') ... defs.auto_add_externs(True) # remove existing extern first. .. py:method:: Defs.check( (Defs)arg1) -> str : :module: ecflow Check :term:`trigger` and :term:`complete expression`\ s and :term:`limit`\ s * Client Side: The client side can specify externs. Hence all node path references in :term:`trigger` expressions, and :term:`inlimit` references to :term:`limit`\ s, that are unresolved and which do *not* appear in :term:`extern`\ s are reported as errors * Server Side: The server does not store externs. Hence all unresolved references are reported as errors Returns a non empty string for any errors or warning Usage: .. code-block:: python # Client side defs = Defs('my.def') # Load my.def from disk .... print(defs.check()) # do the check # Server Side try: ci = Client() # use default host(ECF_HOST) & port(ECF_PORT) print(ci.check('/suite')) except RuntimeError, e: print(str(e)) .. py:method:: Defs.check_job_creation( (Defs)arg1 [, (bool)throw_on_error=False [, (bool)verbose=False]]) -> str : :module: ecflow Check `job creation` . Will check the following: - :term:`ecf script` files and includes files can be located - recursive includes - manual and comments :term:`pre-processing` - :term:`variable substitution` Some :term:`task`\ s are dummy tasks have no associated :term:`ecf script` file. To disable error message for these tasks please add a variable called ECF_DUMMY_TASK to them. Checking is done in conjunction with the class :py:class:`ecflow.JobCreationCtrl`. If no node path is set on class JobCreationCtrl then all tasks are checked. In the case where we want to check all tasks, use the convenience function that take no arguments. Usage: .. code-block:: python defs = Defs('my.def') # specify the defs we want to check, load into memory ... print(defs.check_job_creation()) # Check job generation for all tasks ... # throw on error and Output the tasks as they are being checked defs.check_job_creation(throw_on_error=TrueTrue,verbose=True) job_ctrl = JobCreationCtrl() job_ctrl.set_verbose(True) # Output the tasks as they are being checked defs.check_job_creation(job_ctrl) # Check job generation for all tasks, same as above print(job_ctrl.get_error_msg()) ... job_ctrl = JobCreationCtrl() job_ctrl.set_node_path('/suite/to_check') # will hierarchically check job creation under this node defs.check_job_creation(job_ctrl) # job files generated to ECF_JOB print(job_ctrl.get_error_msg()) ... job_ctrl = JobCreationCtrl() # no set_node_path() hence check job creation for all tasks job_ctrl.set_dir_for_job_creation(tmp) # generate jobs file under this directory defs.check_job_creation(job_ctrl) print(job_ctrl.get_error_msg()) ... job_ctrl = JobCreationCtrl() # no set_node_path() hence check job creation for all tasks job_ctrl.generate_temp_dir() # automatically generate directory for job file defs.check_job_creation(job_ctrl) print(job_ctrl.get_error_msg()) check_job_creation( (Defs)arg1, (JobCreationCtrl)arg2) -> None .. py:method:: Defs.delete_variable( (Defs)arg1, (str)arg2) -> None : :module: ecflow An empty string will delete all user variables .. py:property:: Defs.externs :module: ecflow Returns a list of :term:`extern`\ s .. py:method:: Defs.find_abs_node( (Defs)arg1, (str)arg2) -> Node : :module: ecflow Given a path, find the the :term:`node` .. py:method:: Defs.find_node( (Defs)arg1, (str)arg2, (str)arg3) -> Node : :module: ecflow Given a type(suite,family,task) and a path to a node, return the node. .. py:method:: Defs.find_node_path( (Defs)arg1, (str)arg2, (str)arg3) -> str : :module: ecflow Given a type(suite,family,task) and a name, return path of the first match, otherwise return an empty string .. py:method:: Defs.find_suite( (Defs)arg1, (str)arg2) -> Suite : :module: ecflow Given a name, find the corresponding :term:`suite` .. py:method:: Defs.generate_scripts( (Defs)arg1) -> None : :module: ecflow Automatically generate template :term:`ecf script`\ s for this definition Will automatically add :term:`child command`\ s for :term:`event`\ s, :term:`meter`\ s and :term:`label`\ s. This allows the definition to be refined with out worrying about the scripts. However it should be noted that, this will create a lot of *duplicated* script contents i.e in the absence of :term:`event`\ s, :term:`meter`\ s and :term:`label`\ s, most of generated :term:`ecf script` files will be the same. Hence should only be used an aid to debugging the definition. It uses the contents of the definition to parameterise what gets generated, and the location of the files. Will throw Exceptions for errors. Requires: - ECF_HOME: specified and accessible for all Tasks, otherwise RuntimeError is raised - ECF_INCLUDE: specifies location for head.h and tail.h includes, will use angle brackets, i.e %include , if the head.h and tail.h already exist they are used otherwise they are generated Optional: - ECF_FILES: If specified, then scripts are generated under this directory otherwise ECF_HOME is used. The missing directories are automatically created. - ECF_CLIENT_EXE_PATH: if specified child command will use this, otherwise will use ecflow_client and assume this accessible on the path. - ECF_DUMMY_TASK: Will not generated scripts for this task. - SLEEP: Uses this variable to delay time between calls to child commands, if not specified uses delay of one second Usage: .. code-block:: python defs = ecflow.Defs() suite = defs.add_suite('s1') suite.add_variable('ECF_HOME','/user/var/home') suite.add_variable('ECF_INCLUDE','/user/var/home/includes') for i in range(1,7) : fam = suite.add_family('f' + str(i)) for t in ( 'a', 'b', 'c', 'd', 'e' ) : fam.add_task(t); defs.generate_scripts() # generate '.ecf' and head.h/tail.h if required .. py:method:: Defs.get_all_nodes( (Defs)arg1) -> NodeVec : :module: ecflow Returns all the :term:`node`\ s in the definition .. py:method:: Defs.get_all_tasks( (Defs)arg1) -> TaskVec : :module: ecflow Returns all the :term:`task` nodes .. py:method:: Defs.get_server_state( (Defs)arg1) -> SState : :module: ecflow Returns the :term:`ecflow_server` state: See :term:`server states` Usage: .. code-block:: python try: ci = Client() # use default host(ECF_HOST) & port(ECF_PORT) ci.shutdown_server() ci.sync_local() assert ci.get_defs().get_server_state() == SState.SHUTDOWN, 'Expected server to be shutdown' except RuntimeError, e: print(str(e)) .. py:method:: Defs.get_state( (Defs)arg1) -> State :module: ecflow .. py:method:: Defs.has_time_dependencies( (Defs)arg1) -> bool : :module: ecflow returns True if the :term:`suite definition` has any time :term:`dependencies` .. py:method:: Defs.restore_from_checkpt( (Defs)arg1, (str)arg2) -> None : :module: ecflow Restore the :term:`suite definition` from a :term:`check point` file stored on disk .. py:method:: Defs.save_as_checkpt( (Defs)arg1, (str)arg2) -> None : :module: ecflow Save the in memory :term:`suite definition` as a :term:`check point` file. This includes all node state. .. py:method:: Defs.save_as_defs( (Defs)arg1, (str)arg2 [, (Style)arg3]) -> None : :module: ecflow Save the in memory :term:`suite definition` into a file. The file name must be passed as an argument .. py:property:: Defs.server_variables :module: ecflow Returns a list of server :term:`variable`\ s .. py:method:: Defs.simulate( (Defs)arg1) -> str : :module: ecflow Simulates a suite definition, allowing you predict/verify the behaviour of your suite in few seconds The simulator will analyse the definition, and simulate the ecflow server. Allowing time dependencies that span several months, to be simulated in a few seconds. Ecflow allows the use of verify attributes. This example show how we can verify the number of times a task should run, given a start(optional) and end time(optional): .. code-block:: shell suite cron3 # use real clock otherwise clock starts when the simulations starts. clock real 1.1.2006 # define a start date for deterministic behaviour endclock 13.1.2006 # When to finish. end clock is *only* used for the simulator family cronFamily task t cron -d 10,11,12 10:00 11:00 01:00 # run on 10,11,12 of the month at 10am and 11am verify complete:6 # task should complete 6 times between 1.1.2006 -> 13.1.2006 endfamily endsuite Please note, for deterministic behaviour, the start and end clock should be specified. However if no 'endclock' is specified the simulation will assume the following defaults. - No time dependencies: 24 hours - time || today : 24 hours - day : 1 week - date : 1 month - cron : 1 year - repeat : 1 year If there no time dependencies with an minute resolution, then the simulator will by default use 1 hour resolution. This needs to be taken into account when specifying the verify attribute If the simulation does not complete it creates defs.flat and defs.depth files. This provides clues as to the state of the definition at the end of the simulation Usage: .. code-block:: python defs = Defs('my.def') # specify the defs we want to simulate .... theResults = defs.simulate() print(theResults) .. py:method:: Defs.sort_attributes( (Defs)arg1, (AttrType)arg2) -> None :module: ecflow sort_attributes( (Defs)arg1, (AttrType)arg2, (bool)arg3) -> None sort_attributes( (Defs)arg1, (AttrType)arg2, (bool)arg3, (list)arg4) -> None sort_attributes( (Defs)arg1, (str)attribute_type [, (bool)recursive=True [, (list)no_sort=[]]]) -> None sort_attributes( (Defs)arg1, (AttrType)arg2, (bool)attribute_type [, (object)recursive=True]) -> None .. py:property:: Defs.suites :module: ecflow Returns a list of :term:`suite`\ s .. py:property:: Defs.user_variables :module: ecflow Returns a list of user defined :term:`variable`\ s