Title: preppy spec
Version: $Revision: 7752 $
Author: Aaron Watters <aaron@reportlab.com>
Status: Final
Type: Design Specification
Created: 5-Oct-2000


Introduction

    This document describes the preppy.py module operations.
    It is intended for an audience of proficient python programmers
    and is not intended as a user guide for novices.

	Preppy is intended as a simple and fairly general method for
	preprocessing text files containing special markup into python
	programs.  The markup may contain loops and conditional flow
	control and generic python code.  A module generated by preppy
	sends text output to standard output.  Nondirectives in the
	source file are sent unmodified and directives in the source
	file result in computation which may result in substitutions into
	the text output or evaluation of flow control that
	decides whether and how many times segments of text are injected
	into the output stream.

	An example use for Preppy would be to generate a series of similar
	HTML files from the results of database queries -- where the general
	form of the document remains fixed but certain data elements
	change.

	For example consider the Preppy source file

	<html><head><title>{{name}}</title></head><body>
	hello my name is {{name}} and I am
	{{if sex=="f":}} a gal
	{{elif sex=="m":}} a guy
	{{else:}} neuter {{endif}}
	</body></html>

	then with 

	dictionary = {"name": "Fred Flintstone", "sex": "m"}

	the preppy output for this preprocessor source wedded with the
	dictionary input is

        <html><head><title>Fred Flintstone</title></head><body>
        hello my name is Fred Flintstone and I am
         a guy
                </body></html>

	Alternatively with

	dictionary = {"name": "Wilma Flintstone", "sex": "f"}

	We obtain output

	<html><head><title>Wilma Flintstone</title></head><body>
        hello my name is Wilma Flintstone and I am
         a gal
                </body></html>

Intended Use

	Preppy is intended for use in a professional environment with testing.
	In particular there are few provisions to prevent infinite loops or
	NameErrors, or Divide by Zero errors, or other problems familiar to
	professional programmers.  In particular the preppy source

	{{script}}from foo import bar; bar(){{endscript}}

	may result in an infinite loop or a NameError and we make no provision 
	to attempt to avoid this problem.  Similarly for

	{{while 1}}
		INFINITE LOOP!!! {{X}}
	{{endwhile}}

	This is only one example of many
	where using the preppy module may result in errors or bugs.

Operations

	Load a preppy module (possibly from source text) using

	m = getModule(name, directory=".", source_extension=".prep", 
		verbose=0, savefile=1, sourcetext=None)

	This function will look for the python module named "name" and if
	if exists check to see that it matches the preppy source module
	name+source_extension in the specified directory.  If the
	module does not exist or does not match a new python module
	will be built from the source file, stored as a python module
	name+".py" in the directory (which must be writable if the modules
	don't match.)  

	If you do not want to save the regenerated module
	text then use savefile=0 and no file will be written during the
	load (but a matching existing python module will still be preferred).

	If the source module is not present, but the python module is present
	the python module will be assumed correct and used.  When the source
	text and the generated python module match the python code will not
	be regenerated (as an optimization).

	The preppy module itself will keep a cache of those modules previously
	built by preppy and will not reexecute the test/generation code when
	a module of the same name is requested twice -- instead it will provide
	the previously built module.

	It is also possible to use preppy to make a module without any
	use of the filesystem.  In this case provide source text as a string
	with a module name (of no significance) to getModule

		m = getModule("dummymodule", sourcetext=my_string)

	In this case (when sourcetext is provided) no file will be read and
	no file will be generated.  The module m will be constructed in memory
	for use by the current process only.

	The entry point for a Preppy module is the function

		m.run(dictionary, __write__=None, outputfile=None)

	Where dictionary provides external information to the run function.
	An example usage might be

	m = getModule("salutation", directory="/usr/lib/preppy")
	m.run({"name": "Wilma Flintstone", "sex": "f"})

	Run has a second argument which allows the default "write" operation
	to be redefined.  For example to append the output text segments to
	a list instead of writing to a file use:

		L = []
		app = L.append
		m.run(dictionary, __write__=app)

	Run has a third optional argument which allows the program to specify
	a file to use as the standard output for the function.  In this case
	print statements will work (since sys.stdout will be set to the file
	while run() executes and set back to whatever it was previously
	afterwards.  Thus you can call

		f = open("flintstone.html", "w")
		m.run(dictionary, outputfile=f)

	WARNING: the preppy module will signal an error if both __write__ and
	outputfile are defined -- use one or the other or neither, not both.

Recursive Imports of preppy modules:

	Recursive imports of preppy modules should use getModule which
	guarantees that once the module is loaded once it will not be loaded
	a second time. At present there is no special directive to do this.
	Use a script directive.

NonDirectives

	Text not recognized as directives (or partial erroneous directives)
	is sent to standard output unmodified by the run() function.

Directives

	Directives are set off by 

			STARTDELIMITER = "{{"
			ENDDELIMITER = "}}"

	these module constants can be altered, but alteration may
	result in parsing difficulties if the replacements are not chosen
	carefully.

	NOTE ABOUT ESCAPING: The string {${ will be translated to {{ anywhere
	in the text after delimiters have been identified.  Similarly }$} will
	be translated to }}.  Furthermore $$ will translate to $.  To specify
	two $'s in sequence use $$$$.  Single $'s will be left alone outside of
	{${ or }$}.

	Below are the discussions of the directives in turn.

token:	{{token}}

	This construct is recognized the token
	does not match any of the other directives.  The result of this directive
	is the value of token evaluated as a python expression in the context
	of the m.run() function.  An example might be

		{{dictionary["name"]+2}}

	The result of the evaluation is introduced into the output stream
	(using standard python string conversion, if needed).

eval:	{{eval}}python_expression{{endeval}}

	This is a longer way to spell {{python_expression}} :). It is useful
	for larger expressions like

	{{eval}}
	a_complex("and", "very", "verbose", function="call")
	{{endeval}}

	The expression is evaluated and the result is inserted in the output
	stream as in token.

	WARNING: for token, eval, and script any newlines in the code text
	will be automatically indented to the proper indentation level for
	the run() module at that insertion point.  This is only a concern
	for triple quoted strings.  If this may be an issue don't use triple
	quoted strings in preppy source. Instead of

	x = """
	a string
	"""

	use

	x = ("\n"
	"\ta string\n"
	)

	or similar.

	NOTE: inside the script and eval directives the STARTTAG {{ and ENDTAG
	}} will be ignored unless they occur within {{endeval}} or {{endscript}}
	respectively.

script:	{{script}}python_code_line{{endscript}}

	or
		{{script}}
		many
		lines
		indented
		properly
		{{endscript}}

	This executes a sequence of python code within the context of the
	run() function.  An example would be

		{{script}}import math; x = math.sin(math.pi/4.0){{endscript}}

	or

		{{script}}
		import math
		x = math.sin(math.pi/4.0){{endscript}}

	RESTRICTION: in the multiline case the indentation should follow
	Python conventions (of course) and any initial indentation should be
	character for character the same between the lines of code
	(except for completely white lines).  In particular if the first line
	begins with 8 spaces the next line should not begin with a TAB.

	SCRIPT RECOMMENDATION 1: If possible the script tags should not directly
	produce standard output, but instead define string variables that
	are introduced in token directives.

	SCRIPT RECOMMENDATION 2: If a script must produce standard output
	itself it should use the function

		__write__(string)

	which is always defined in the context of the run() function and
	allows the output to be redirected flexibly without modifying sys.stdout.
	In particular the script should not use the "print" statement or other
	direct writes to sys.stdout.

	SCRIPT RECOMMENDATION 3: If you ignore the first 2 recommendations then
	never use the second argument of the run() function to redirect the
	output -- reset sys.stdout instead using the outputfile=file third argument.
	In this case the generated function cannot send output to any structure
	which doesn't support the Python file protocol.

if)	{{if condition}} block {{endif}}
	{{if condition}} block1 {{else}} block2 {{endif}}
	{{if condition}} block1 {{elif condition2}} block2 {{endif}}
	{{if condition}} block1 {{elif condition2}} block2 
			{{else}} block3 {{endif}}
		...etcetera...

	This conditionally executes segments of text and directives.
	each condition should be a python expression in the context of the
	run() function.  The blocks may contain other directives and text.

for)	{{for for_target}} block {{endfor}}

	This implements a for loop in preppy source.  The for_target should follow
	normal python conventions for python for loops.  The resulting python 
	code is roughly

	for for_target:
		interpretation_of(block)

while)	{{while condition}} block {{endwhile}}

	This implements a while loop in preppy source.  The condition should be
	a python expression.  The resulting python code is roughly

	while condition:
		interpretation_of(block)

STATIC COMPILATION

	To statically compile a preppy module once and for all you can
	use preppy as command line program.  For example suppose the
	example file given above is in "./flintstone.prep" in the same
	directory as preppy.  Then compile it using
	
		C:\reportlab\repository\rlextra\preppy> preppy.py flintstone
		no module flintstone found (or error)
		CHECKSUMS DON'T MATCH
		regenerating python source from .\flintstone.prep

	You can then test it using a static import

		C:\reportlab\repository\rlextra\preppy>python
		Python 1.5.2 (#0, Apr 13 1999, 10:51:12) [MSC 32 bit (Intel)] on win32
		Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
		>>> from flintstone import run
		>>> D =  {'sex': 'm', 'name': 'george'}
		>>> run(D)

		        <html><head><title>george</title></head><body>
		        hello my name is george and I am
		         a guy

		        </body></html>
		>>>

	Note that preppy modules which do not dynamically import other preppy
	modules may be used "stand-alone" without the preppy module itself
	present.


SYNTAX CHECKING:

	Python code expressions and code blocks are checked for syntactic
	correctness using the Python compiler during generation in an attempt
	to detect errors in a helpful way during code generation.  Partial
	code blocks or expressions are not allowed.

	For example the following will result in errors reported at generation time.

		{{5*}} {{10}}  (5* is not a complete expression)

	and

		{{script}}
		if x=40: # syntax error in expression x=40
			y = 9
		{{endscript}}

	and

		{{script}}
		if x==40: # incomplete if statement
		{{endscript}}
			the value of x is 40

	and

		{{if x==40: y=9}} (not a valid if directive)

	However NameErrors or other run time errors which are not detected
	by the Python compiler will not be detected at generation time.  For
	example the compiler will not understand that this is an invalid
	function call (it is syntactically acceptible to the grammar)

		{{ "this"("doesn't work") }}

	This error will be detected during execution of the run() function
	if and when the code segment containing it is executed.

		...TypeError: call of non-function (type string)

ADDITIONAL NOTES:

	You may define classes and functions in scripts but remember that they
	are defined in the local context of the run() function and do not have access
	to local variables in the run() function.  Any variable you wish to
	use in a class or function (except for __write__ and dictionary and
	outputfile, the run function arguments) can be declared global in the script

		global variable

	(But note that this is a slightly dangerous thing to do in the unlikely
	event that the generated module is used in a multithreaded python
	application).  Also if you wish to make the classes or functions available
	to other modules you must declare them global and the run() function must
	be executed at least once before the exported functions or classes are used
	(NOT RECOMMENDED!).

	WARNING: extra whitespace at the end of a line is discarded in the source.
	(rationale: had problems with extra carriage returns at ends of some lines
	and this was an easy fix, but AFAIK it may need to be undone at some
	later time if trailing whitespace is needed sometime...).


# end of preppy spec

