Name

Outthentic

Synopsis

Multipurpose scenarios framework.

Build status

![Build Status](https://travis-ci.org/melezhik/outthentic.svg)

Install

    $ cpanm Outthentic

Short introduction

This is a quick tutorial on outthentic usage.

Run your scenarios

Scenario is just a script that you run and that yields something into stdout.

Perl scenario example:

    $ cat story.pl
    
    print "I am OK\n";
    print "I am outthentic\n";

Bash scenario example:

    $ cat story.bash
    
    echo I am OK
    echo I am outthentic

Python scenario example:

    $ cat story.py
    
    print "I am OK"
    print "I am outthentic"

Ruby scenario example:

    $ cat story.rb
    
    puts "I am OK"
    puts "I am outthentic"

Outthentic scenarios could be written on one of four languages:

Perl
Bash
Python
Ruby

Choose you favorite language ;) !

Outthentic relies on file names to determine scenario language.

This is the table to describe language / file name conventions:

    +-----------+--------------+
    | Language  | File         |
    +-----------+--------------+
    | Perl      | story.pl     |
    | Bash      | story.bash   |
    | Python    | story.py     |
    | Ruby      | story.rb     |
    +-----------+--------------+

Check file

Check file contains rules to verify a stdout produced by scenario script.

Here we require that our scenario should produce `I am OK' and `I am outthentic' lines in stdout:

    $ cat story.check
    
    I am OK
    I am outthentic

NOTE: You can leave check file empty, but it's required anyway

    $ touch story.check

Empty check file means you just want to ensure that your story succeed ( exit code 0 ) and don't want run any checks for story stdout.

Story

Outthentic story is a scenarios + check file. When outthentic run story it:

executes scenario script and saves stdout into file.
verifies stdout against a check file

Suite

Outthentic suites are a bunch of related stories. You may also call suites as (outthentic) projects.

One may have more then one story at the project.

Just create a new directories with a story data inside:

    $ mkdir perl-story
    $ echo 'print "hello from perl";' > perl-story/story.pl
    $ echo 'hello from perl' > perl-story/story.check
    
    $ mkdir bash-story
    $ echo 'echo hello from bash' > bash-story/story.bash
    $ echo 'hello from bash' > bash-story/story.check
    
    $ mkdir python-story
    $ echo 'print "hello from python"' > python-story/story.rb
    $ echo 'print from python' > python-story/story.check
    
    $ mkdir ruby-story
    $ echo 'puts "hello from ruby"' > ruby-story/story.rb
    $ echo 'hello from ruby' > ruby-story/story.check

Now, let's use strun command to run suite stories:

    $ strun --story perl-story
    
    /perl-story/ started
    
    hello from perl
    OK  scenario succeeded
    OK  output match 'hello from perl'
    ---
    STATUS  SUCCEED
    
    $ strun --story bash-story # so on ...

Summary:

Stories are just a directories with story scenarios and check files.
Strun - a [S]tory [R]unner - a console script to execute story scenarios and validate output by given check lists.
Outthentic suites are bunches of related stories.
By default strun looks for story.(pl|rb|bash) file at the project root directory. This is so called a default story.
One can point strun a distinct story by --story parameter:

$ strun --root /path/to/project/root/ --story /path/to/story/directory/inside/project/root

Story should point a directory relative to project root directory.

Project root directory - a directory holding all related stories.

A project root directory could be set explicitly using --root parameter:

    $ strun --root /path/to/my/root/

If --root is not set strun assumes project root directory to be equal current working directory:

    $ strun # all the stories should be here

Calculator project example

Here is more detailed tutorial where we will build a test suite for calculator program.

Let's repeat it again - there are three basic outthentic entities:

suite
scenarios
check files

Project

Outthentic project is a bunch of related stories. Every project is represented by a directory.

Let's create a project to test a simple calculator application:

    $ mkdir calc-app
    $ cd calc-app

Scenarios

Scenarios are just a scripts to be executed so that their output to be verified by rules defined at check files.

In other words, every story is like a small program to be executed and then gets tested ( by it's output )

Let's create two stories for our calc project. One story for `addition' operation and another for `multiplication':

    # story directories
    
    $ mkdir addition # a+b
    $ mkdir multiplication # a*b
    
    
    # scenarios
    
    $ cat  addition/story.pl
    use MyCalc;
    my $calc = MyCalc->new();
    print $calc->add(2,2), "\n";
    print $calc->add(3,3), "\n";
    
    $ cat  multiplication/story.pl
    use MyCalc;
    my $calc = MyCalc->new();
    print $calc->mult(2,3), "\n";
    print $calc->mult(3,4), "\n";

Check files

Check file contains validation rules to test script output. Every scenario is always accompanied by story check file.

Check files should be placed at the same directory as scenario and be named as story.check.

Lets add some rules for `multiplication' and `addition' stories:

    $ cat addition/story.check
    4
    6
     
    $ cat multiplication/story.check
    6
    12

And finally lets run our suite:

    $ strun

Story runner

Story runner - is a script to run outthentic stories. It is called strun.

Runner consequentially goes several phases:

A compilation phase.

Stories are converted into perl files *.pl ( compilation phase ) and saved into temporary directory.

An execution phase.

Perl executes a compiled perl file. As it was told before if not set explicitly strun looks for something like story.(pl|rb|bash) at the top of project root directory and then compiles it in regular perl script and then give it Perl to run to execute such a script.

Check files syntax

Outthentic consumes Outthentic::DSL, so check files contain rules defined in terms of Outthentic DSL - a language to validate unstructured text data.

Below some examples of check file syntax, you may learn more at Outthentic::DSL documentation.

plain strings checks

Often all you need is to ensure that stdout has some strings in:

    # scenario stdout
    
    HELLO
    HELLO WORLD
    123456
    
    
    # check file
    
    HELLO
    123
    
    # verification output
    
    OK - output matches 'HELLO'
    OK - output matches 'HELLO WORLD'
    OK - output matches '123'

regular expressions

You may use regular expressions as well:

    # check file
    
    regexp: L+
    regexp: \d
    
    
    # verification output
    
    OK - output matches /L+/
    OK - output matches /\d/

See check-expressions in Outthentic::DSL documentation pages.

inline code, generators and asserts

You may inline code from other language to add some extra logic into your check file:

Inline code

    # check file
    
    code: <<CODE
    !bash
    echo 'this is debug message will be shown at console'
    CODE
    
    code: <<CODE
    !python
    print 'this is debug message will be shown at console'
    CODE
    
    code: <<CODE
    !ruby
    puts 'this is debug message will be shown at console'
    CODE
    
    code: <<CODE
    # by default Perl language is used
    print("this is debug message will be shown at console\n");
    CODE

generators

You may generate new check entries on runtime:

    # check file
    # with 2 check entries
       
    Say
    HELLO
       
    generator: <<CODE
    !bash
    
    echo say 
    echo hello 
    echo again
    
    CODE
    
    # a new check list would be:
       
    Say
    HELLO
    say
    hello
    again

Here examples on using other languages in generator expressions:

Perl:

    generator: <<CODE
    !perl
    [ 
      qw { say hello again } 
    ]
    
    CODE

Python:

    generator: <<CODE
    !python
    print 'say'
    print 'hello'
    print 'again'
    
    CODE

Ruby:

    generator: <<CODE
    !ruby
    puts 'say'
    puts 'hello'
    puts 'again'
    
    CODE

asserts

Asserts are statements returning true of false with some extra text description.

Asserts are very powerful feature when combined with captures and generators:

    # scenario output
    
    ten       for 10
    twenty   for 20
    thirty    for 30
    
    # check file
    
    regexp: \w+\s+for\s(\d+)
    
    generator: <<CODE
    !ruby
      sum=0
      (captures()).each do |c|
        sum+=c.first
      end
    puts "assert: #{sum==60} sum should be 60!"
    CODE

Follow code expressions, generators and asserts in Outthentic::DSL documentation pages to learn more about code expressions, generators and asserts.

text blocks

Need to validate that some lines goes successively?

    # stdout
    
    this string followed by
    that string followed by
    another one string
    with that string
    at the very end.
    
    
    # check list
    # this text block
    # consists of 5 strings
    # goes consequentially
    # line by line:
    
    begin:
        # plain strings
        this string followed by
        that string followed by
        another one
        # regexp patterns:
    regexp: with (this|that)
        # and the last one in a block
        at the very end
    end:

See comments-blank-lines-and-text-blocks in Outthentic::DSL documentation pages.

Hooks

Story hooks are extension points to change story runner behavior.

It's just a scripts gets executed before scenario script.

You should name your hooks as hook.* and place them into story directory

    $ cat perl/hook.pl
    
    print "this is a story hook!";

Hooks could be written on one of three languages:

Perl
Bash
Python
Ruby

Here is naming convention for hook files:

    +-----------+--------------+
    | Language  | File         |
    +-----------+--------------+
    | Perl      | hook.pl      |
    | Bash      | hook.bash    |
    | Python    | hook.py      |
    | Ruby      | hook.rb      |
    +-----------+--------------+

Reasons why you might need a hooks:

execute some initialization code before running a scenario script
redefine scenario stdout
call downstream stories

Hooks API

Story hooks API provides several functions to hack into story runner execution process:

Redefine stdout

Redefining stdout feature means you define a scenario output on the hook side ( thus scenario script is never executed ).

This might be helpful when for some reasons you do not want to run or you don't have by hand a proper scenario script.

This is simple an example:

    $ cat hook.pl
    set_stdout("THIS IS I FAKE RESPONSE \n HELLO WORLD");
    
    $ cat story.check
    THIS IS FAKE RESPONSE
    HELLO WORLD

You may call set_stdout() more then once:

    set_stdout("HELLO WORLD");
    set_stdout("HELLO WORLD2");

An effective scenario stdout will be:

    HELLO WORLD
    HELLO WORLD2

Here is set_stdout() function signatures list for various languages:

    +-----------+-----------------------+
    | Language  | signature             |
    +-----------+-----------------------+
    | Perl      | set_stdout(SCALAR)    |
    | Bash      | set_stdout(STRING)    |
    | Python(*) | set_stdout(STRING)    |
    | Ruby      | set_stdout(STRING)    |
    +-----------+-----------------------+

IMPORTANT: You should only use a set_stdout inside story hook, not scenario file.

(*) you need to from outthentic import * in Python to import set_stdout function.

Upstream and downstream stories

It is possible to run one story from another with the help of downstream stories.

Downstream stories are reusable stories ( aka modules ).

Story runner never executes downstream stories directly.

Downstream story always gets called from the upstream one. This is example:

    $ cat modules/knock-the-door/story.rb
    
    # this is a downstream story
    # to make story downstream
    # simply create story files 
    # in modules/ directory
    
    puts 'knock-knock!'
     
    $ cat modules/knock-the-door/story.check
    knock-knock!
    
     
    $ cat open-the-door/hook.rb
    
    # this is a upstream story
    # to run downstream story
    # call run_story function
    # inside upstream story hook
    
    # with a single parameter - story path,
    # notice that you have to remove
    # `modules/' chunk from story path parameter
    
    run_story( 'knock-the-door' );
    
    $ cat open-the-door/story.rb
    puts 'opening ...' 
    
    $ cat open-the-door/story.check
    opening
    
    $ strun --story open-the-door/
     
    /modules/knock-the-door/ started
    
    knock-knock!
    OK  scenario succeeded
    OK  output match 'knock-knock!'
    
    /open-the-door/ started
    
    opening ...
    OK  scenario succeeded
    OK  output match 'opening'
    ---
    STATUS  SUCCEED

Summary:

to make story a downstream simply create a story in a modules/ directory.
to run downstream story call run_story(story_path) function inside upstream story hook.
you can call as many downstream stories as you wish.
you can call the same downstream story more than once.
downstream stories in turn may call other downstream stories.

Here is an example of multiple downstream story calls:

    $ mkdir module/up
    $ mkdir module/down
    $ echo 'UP!' > module/up/story.check
    $ echo 'and DOWN!' > module/down/story.check
    $ echo 'print qq{UP!}' > modules/up/story.pl 
    $ echo 'print qq{DOWN!}' > modules/down/story.pl 
    
    $ cat two-jumps/hook.pl
    
    run_story( 'up' );
    run_story( 'down' );
    run_story( 'up' );
    run_story( 'down' );

story variables

You may pass a variables to downstream story using second argument of run_story() function. For example:

    $ mkdir modules/greeting
    
    $ cat hook.pl
    
    run_story( 
      'greeting', {  name => 'Alexey' , message => 'hello' }  
    );

Or using Ruby:

    $ cat hook.rb
    
    run_story  'greeting', {  'name' => 'Alexey' , 'message' => 'hello' }

Or using Python:

    from outthentic import *
    run_story('greeting', {  'name' : 'Alexey' , 'message' : 'hello' })

Or Bash:

    $ cat hook.bash
    
    run_story  greeting name Alexey message hello

Here is the run_story signature list for various languages:

    +-----------+----------------------------------------------+
    | Language  | signature                                    |
    +-----------+----------------------------------------------+
    | Perl      | run_story(SCALAR,HASHREF)                    |
    | Bash      | run_story(STORY_NAME NAME VAL NAME2 VAL2 ... | 
    | Python    | run_story(STRING,DICT)                       | 
    | Ruby      | run_story(STRING,HASH)                       | 
    +-----------+----------------------------------------------+

Story variables are accessible via story_var() function.

Examples:

In Perl:

    $ cat modules/greeting/story.pl
    
    print story_var('name'), 'say ', story_var('message');

In Python:

    $ cat modules/greeting/story.py
    
    from outthentic import *
    print story_var('name') + 'say ' + story_var('message')

In Ruby:

    $ cat modules/greeting/story.rb
    
    puts "#{story_var('name')} say #{story_var('message')}"

In Bash (1-st way):

    $ cat modules/greeting/story.bash
    
    echo $name say $message

In Bash (2-nd way):

    $ cat modules/greeting/story.bash
    
    echo $(story_var name) say $(story_var message)

You may access story variables inside story hooks and check files as well.

And finally:

downstream stories may invoke other downstream stories.
you can't only use story variables inside downstream stories.

Here is the how you access story variable in all three languages

    +------------------+---------------------------------------------+
    | Language         | signature                                   |
    +------------------+---------------------------------------------+
    | Perl             | story_var(SCALAR)                           |
    | Python(*)        | story_var(STRING)                           | 
    | Ruby             | story_var(STRING)                           | 
    | Bash (1-st way)  | $foo $bar ...                               |
    | Bash (2-nd way)  | $(story_var foo.bar)                        |
    +------------------+---------------------------------------------+

(*) you need to from outthentic import * in Python to import story_var() function.

Story properties

Some story properties have a proper accessors functions. Here is the list:

project_root_dir() - Root directory of outthentic project.
test_root_dir() - Test root directory. Root directory of generated perl test files , see also story runner.
config() - Returns suite configuration hash object. See also suite configuration.

Helper functions and variables

Outthentic provides some helpers and variables:

    +------------------+-----------------------------------------------------+
    | Language         | Type     | Name | Comment                           |
    +------------------+-----------------------------------------------------+
    | Perl             | function | os() | get a name of OS distribution     |
    | Bash             | variable | os   | get a name of OS distribution     |
    | Python(*)        | function | os() | get a name of OS distribution     |
    | Ruby             | function | os() | get a name of OS distribution     |
    +------------------+-----------------------------------------------------+

(*) you need to from outthentic import * in Python to import os() function.

Meta stories

Meta stories are special type of outthentic stories.

The essential property of meta story is it has no scenario file at all:

    # foo/bar story
    mkdir foo/bar
    
    # it's a meta story
    touch foo/bar/meta.txt

Placing a special `meta.txt' file into story directory makes that story a meta.

You may live `meta.txt' empty file or add some useful description to be printed when story is executed:

    nano foo/bar/meta.txt
    
        This is my cool story.
        Take a look at this!

How one could use meta stories?

Meta stories are just containers for other downstream stories. Usually one defines some downstream stories call inside meta story's hook file:

    nano foo/bar/hook.pm
    
        run_story( '/story1' );
        run_story( '/story2' );

Meta stories are very similar to upstream stories with redefined stdout, with the only exclusion that as meta story has no scenario file there is no need for redefining a stdout.

You may also call meta stories as downstream stories:

    nano modules/foo/bar/meta.txt

Ignore unsuccessful story code

Every story is a script gets executed and thus returning an exit code. If exit code is bad (!=0) this is treated as story verification failure.

Use ignore_story_err(int) function to ignore unsuccessful story code:

    # Python
    
    $ cat hook.py
    from outthentic import *
    ignore_story_err(1)
    
    
    # Ruby
    
    $ cat hook.rb
    ignore_story_err 1
    
    # Perl
    
    $ cat hook.pl
    ignore_story_err(1)
    
    # Bash
    
    $ cat hook.bash
    ignore_story_err 1

Story libraries

Story libraries are files to keep your libraries code to automatically required into story hooks and check files context:

Here are some examples:

Perl:

    $ cat common.pm
    sub abc_generator {
      print $_, "\n" for a..z;
    } 
    
    $ cat story.check
    
    generator: <<CODE;
    !perl
      abc_generator()
    CODE

Ruby:

    $ cat common.rb
    def super_utility arg1, arg2
      # I am cool! But I do nothing!
    end
    
    $ cat hook.pl
    
    super_utility 'foo', 'bar'

Here is the list for library file names for various languages:

    +-----------+-----------------+
    | Language  | file            |
    +-----------+-----------------+
    | Perl      | common.pm       |
    | Bash      | common.bash     |
    | Ruby      | common.rb       |
    +-----------+-----------------+

NOTE! Story libraries are not supported for Python

Language libraries

Perl

PERL5LIB

$project_root_directory/lib path gets added to $PERL5LIB variable.

This make it easy to place custom Perl modules under project root directory:

    $ cat my-app/lib/Foo/Bar/Baz.pm
    package Foo::Bar::Baz;
    1;
    
    $ cat common.pm
    use Foo::Bar::Baz;

Story runner client

    $ strun <options>

Options

--root

Root directory of outthentic project. If root parameter is not set current working directory is assumed as project root directory.

--debug

Enable/disable debug mode:

    * Increasing debug value results in more low level information appeared at output
    
    * Default value is 0, which means no debugging 
    
    * Possible values: 0,1,2,3

--silent

Run in silent mode. By default strun prints all scenarios output, to disable this choose --silent option.

--purge-cache

Purge strun cache upon exit. By default --purge-cache is disabled ( cache remains to allow debugging and troubleshooting ).

--match_l

Truncate matching strings. In a TAP output truncate matching strings to {match_l} bytes; default value is 200.

--story

Run only single story. This should be path to a directory containing story inside project. A path should be relative against project root directory. Examples:

    # A project  with 3 stories
    foo/story.pl
    foo/bar/story.rb
    bar/story.pl
    
    # Run various stories
    --story foo # runs foo/ stories
    --story foo/story # runs foo/story.pl
    --story foo/bar/ # runs foo/bar/ stories

--ini

Configuration ini file path.

See suite configuration section for details.

--yaml

YAML configuration file path.

See suite configuration section for details.

--json

JSON configuration file path.

See suite configuration section for details.

--nocolor

If set - disable color output. By default strun prints with colors.

--dump-config

If set - dumps a suite configuration and exit not doing any other actions. See also suite configuration section.

Suite configuration

Outthentic projects are configurable. Configuration data is passed via configuration files.

There are three type of configuration files are supported:

Config::General format
YAML format
JSON format

Config::General style configuration files are passed by --ini parameter

    $ strun --ini /etc/suites/foo.ini
    
    $ cat /etc/suites/foo.ini
    
    <main>
    
      foo 1
      bar 2
    
    </main>

There is no special magic behind ini files, except this should be Config::General compliant configuration file.

Or you can choose YAML format for suite configuration by using --yaml parameter:

    $ strun --yaml /etc/suites/foo.yaml
    
    $ cat /etc/suites/foo.yaml
    
    main :
      foo : 1
      bar : 2

Unless user sets path to configuration file explicitly by --ini or --yaml or --json story runner looks for the files named suite.ini and then ( if suite.ini is not found ) for suite.yaml, suite.json at the current working directory.

If configuration file is passed and read a related configuration data is accessible via config() function, for example in story hook file:

    $ cat hook.pl
    
    my $foo = config()->{main}->{foo};
    my $bar = config()->{main}->{bar};

Examples for other languages:

Bash:

    $ cat hook.bash
    
    foo=$(config main.foo )
    bar=$(config main.bar )

Python:

    $ cat hook.py
    
    
    from outthentic import *
    
    foo = config()['main']['foo']
    bar = config()['main']['bar']

Ruby:

    $ cat hook.rb
    
    foo = config['main']['foo']
    bar = config['main']['bar']

Runtime configuration

Runtime configuration parameters is way to override suite configuration data. Consider this example:

    $ cat suite.ini
    <foo>
      bar 10
    </foo>
      
    $ strun --param foo.bar=20

This way we will override foo.bar to value `20'.

Environment variables

match_l - In a suite runner output truncate matching strings to {match_l} bytes. See also --match_l in options.
SPARROW_ROOT - if set, used as prefix for test root directory.
SPARROW_NO_COLOR - disable color output (see --nocolor option of strun)

Test root directory resolution table:

    +---------------------+----------------------+
    | Test root directory | SPARROW_ROOT Is Set? |
    +---------------------+----------------------+
    | ~/.outthentic/tmp/  | No                   |
    | $SPARROW_ROOT/tmp/  | Yes                  |
    +---------------------+----------------------+

Examples

An example outthentic project lives at examples/ directory, to run it say this:

    $ strun --root examples/

AUTHOR

Aleksei Melezhik

Home Page

https://github.com/melezhik/outthentic

Thanks

To God as the One Who inspires me to do my job!

To install Outthentic, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Outthentic

CPAN shell

perl -MCPAN -e shell
install Outthentic

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

Name

Synopsis

Build status

Install

Short introduction

Run your scenarios

Check file

Story

Suite

Calculator project example

Project

Scenarios

Check files

Story runner

A compilation phase.

An execution phase.

Check files syntax

plain strings checks

regular expressions

inline code, generators and asserts

Inline code

generators

asserts

text blocks

Hooks

Hooks API

Redefine stdout

Upstream and downstream stories

story variables

Story properties

Helper functions and variables

Meta stories

Ignore unsuccessful story code

Story libraries

Language libraries

Perl

Story runner client

Options

Suite configuration

Runtime configuration

Environment variables

Examples

AUTHOR

Home Page

See also

Thanks

Module Install Instructions