Introduction

This is the User Guide for dexy. It is intended to be a thorough guide to dexy features, but not an introductory tutorial or complete reference documentation. For links to other dexy documentation, please visit http://dexy.it/docs/

The User Guide is not complete yet, but it covers many topics and we have tried to focus on the most frustrating and confusing aspects of dexy as a priority (and hopefully we have also made them a little less frustrating in the process). In many places there are links to unit tests related to the content in this guide, you can click the shield icon to show/hide these sections. There are also embedded tickets which represent future planned development.

Telling Dexy What To Do

Dexy is an automation tool which will follow your instructions for what automated operations you would like done on your project. You give Dexy instructions about how to run your project using configuration files written using YAML syntax.

Configuration files should be named dexy.yaml.

You probably only need one dexy.yaml file in your project root, but you can place dexy.yaml files in subdirectories where they will apply only to that subdirectory and its children. The recurse option to the dexy command must be set in order for dexy.yaml files in subdirectories to be loaded.

TODO: Test that dexy.yaml files work as described.

Please feel free to ask questions or add comments about this issue on github.

The basic elements you can specify are:

  • file names/patterns and filters to apply to them

  • dependencies among documents (inputs are indented relative to parents)

  • settings for documents or filters (settings are indented under document)

Here’s a simple YAML config file demonstrating each of these:

def basic_yaml__test():
    with open(PROJECT_ROOT + "/examples/basic.yaml", 'r') as f:
        yaml = f.read()

    with wrap() as wrapper:
        with open("foo.md", 'w') as f:
            f.write("1 + 1 {{ 1+1 }}")

        with open("example.py", 'w') as f:
            f.write("print 'hello'")

        with open("dexy.yaml", 'w') as f:
            f.write(yaml)

        wrapper.run_from_new()

        assert sorted(wrapper.nodes) == [
                'doc:example.py|pyg',
                'doc:foo.md|jinja|markdown',
                'pattern:*.py|pyg']

        assert "<pre>1</pre>" in str(wrapper.nodes["doc:example.py|pyg"].output_data())
        assert wrapper.nodes["doc:foo.md|jinja|markdown"].setting('output')
foo.md|jinja|markdown:
    - output: True # this is a document setting for foo.md|jinja|markdown
    - .py|pyg: # indentation means this document is an input to foo.md|jinja|markdown
        - pyg: { linenos: True } # this is a pyg filter setting for the .py|pyg document

The format for a document specification is a file name or file pattern followed by zero or more filters. Here are a few examples:

- foo.txt # no filters
- .py|pyg # 1 filter
- docs/*.md|jinja|markdown # 2 filters

You can leave off the initial asterisk if your file pattern starts with a dot, i.e. if you are matching all files with a given file extension. This is done for convenience and because if you want to start a string with an asterisk in YAML you need to wrap it in quotes.

File Names and Patterns

foo.txt instructs dexy to create a document named foo.txt where the source is a file named foo.txt in the project root.

def config_txt_single_file__test():
    with wrap() as wrapper:
        write_text_files()
        with open("dexy.yaml", 'w') as f:
            f.write("foo.txt")
        wrapper.run_from_new()

        assert sorted(wrapper.nodes) == ['doc:foo.txt']

bar/foo.txt instructs dexy to create a document named bar/foo.txt where the source is a file named foo.txt in the bar directory under the project root.

def config_txt_single_file_in_subdir__test():
    with wrap() as wrapper:
        write_text_files()
        with open("dexy.yaml", 'w') as f:
            f.write("bar/foo.txt")
        wrapper.run_from_new()

        assert sorted(wrapper.nodes) == ['doc:bar/foo.txt']

.txt instructs dexy to add all available files with ".txt" extension. You could also write this as "*.txt" (note the quotes) but dexy assumes entries starting with a dot are wildcard expressions and adds the asterisk for you.

def config_txt_ext__test():
    with wrap() as wrapper:
        write_text_files()
        with open("dexy.yaml", 'w') as f:
            f.write(".txt")
        wrapper.run_from_new()

        assert sorted(wrapper.nodes) == [
                'doc:bar.txt',
                'doc:bar/baz/foo.txt',
                'doc:bar/foo.txt',
                'doc:foo.txt',
                'pattern:*.txt'
                ]

"*foo.txt" instructs dexy to add all files named foo.txt in any directory. Note that we need to wrap the expression in quotes because we are starting with an asterisk and can’t use the usual shortcut because we are matching a file name, not just an extension.

def config_txt_wildcard__test():
    with wrap() as wrapper:
        write_text_files()
        with open("dexy.yaml", 'w') as f:
            f.write("\"*foo.txt\"")
        wrapper.run_from_new()

        assert sorted(wrapper.nodes) == [
                'doc:bar/baz/foo.txt',
                'doc:bar/foo.txt',
                'doc:foo.txt',
                'pattern:*foo.txt']

Dexy won’t complain if a wildcard expression doesn’t match any files. However, if you specify a filename, there either must be a corresponding file on the file system, or you need to specify the file’s contents using the contents setting.

TODO: Demonstrate error when no file contents available.

Please feel free to ask questions or add comments about this issue on github.

Virtual Files

You can make dexy process a file which doesn’t really exist on the file system by using the contents setting:

def virtual_file_contents__test():
    with wrap() as wrapper:
        with open("dexy.yaml", 'w') as f:
            f.write(trim("""\
            hello.txt|jinja:
                - contents: "1 + 1 = {{ 1+1 }}"
            """))

        wrapper.run_from_new()
        doc = wrapper.nodes['doc:hello.txt|jinja']
        assert str(doc.output_data()) == "1 + 1 = 2"
hello.txt|jinja:
    - contents: "1 + 1 = {{ 1+1 }}"

TODO: Document what happens when both file exists and contents are set.

Please feel free to ask questions or add comments about this issue on github.

contents is one of the available Document Settings.

Custom Name Settings

You can change the name with which a file will be output by specifying the output-name document setting.

If the name starts with a slash / or contains any slashes it is assumed to be a full local (within the project) path to the desired destination of the file. If the name doesn’t contain a slash it is assumed you want to rename a file but keep it in its original directory.

Dexy will automatically apply string interpolation or string formatting to the name if it sees a % or { character. The environment for formatting will be populated with: - all document settings - name corresponding to the original document name - dirname corresponding to the original document directory - any local environment variables poopulated from a dexy-env.json file

Available variables are written to the dexy log at DEBUG level.

This example was run with these environment variables defined:

{
    "client" : "Acme INC",
    "year" : 2013
}

Here is the dexy.yaml:

baz/*.txt:
    - output-name: "%(name)s-file.abc"

bar/baz.txt:
    - output-name: foo-{year}-baz.txt

bar/bell.txt:
    - output-name: "/bell2.txt"

foo.txt:
    - output-name: foo-%(year)s.txt

And here are the input and output files:

$ find .
.
./bar
./bar/bell.txt
./bar/baz.txt
./baz
./baz/1.txt
./baz/2.txt
./baz/3.txt
./foo.txt
./.dexy
./.dexy/.dexy-generated
./dexy-env.json
./dexy.yaml
./dexy.conf
$ cd output/
$ find .
.
./.dexy-generated
./foo-2013.txt
./bar
./bar/foo-2013-baz.txt
./baz
./baz/baz
./baz/baz/1.txt-file.abc
./baz/baz/3.txt-file.abc
./baz/baz/2.txt-file.abc
./README.md
./bell2.txt

Document Settings

A key: value pair indented under a document specification is a document setting. For example, output: True.

To see available document settings, you can use the dexy nodes command.

$ dexy nodes -alias doc
doc

A single Dexy document.

Settings:
    apply-ws-to-content
        If you want to put website-related content (like the link()
        function) in your content, set this to True so your content
        gets put through the jinja filter for the website reporter.
        default value: False

    apply-ws-to-content-var-end-string
        Provide a custom jinja var-end-string to avoid clashes.
        default value: None

    apply-ws-to-content-var-start-string
        Provide a custom jinja var-start-string to avoid clashes.
        default value: None

    contents
        Custom contents for a virtual document.
        default value: None

    data-type
        Alias of custom data class to use to store document content.
        default value: None

    output
        Whether document should be included in output/ and output-site/
        default value: None

    output-name
        Override default canonical name.
        default value: None

    shortcut
        A nickname for document so you don't have to use full key.
        default value: None

    title
        Custom title.
        default value: None

    ws-template
        Key of the template to apply for rendering in website.
        Setting of 'None' will use default template, 'False' will
        force no template to be used.
        default value: None

Files Without Filters

When no filters are specified for a document, the original contents of the file are made available to other documents, and the document will appear in Dexy’s final output unless the output setting has been set to False.

Filters

To specify filters, follow the filename/pattern with a pipe symbol | and a filter alias. You can chain as many filters as you want, in order, by adding more pipes and aliases. The first filter operates on the original contents of the source file, subsequent filters operate on the output from the previous filter.

You can run a source file through different filter combinations, and each will be a separate document in dexy.

- example.py|pyg # html file with syntax highlighted python
- example.py|py # txt file with STDOUT from executing script

Filter Settings

To customize filter settings, you need to first give the filter alias, and then a dictionary of the desired settings for that filter alias.

- example.py|idio|pycon|pyg
    - pycon: { add-new-files: True}
    - pyg: { linenos: True}

To see available settings for a filter, you can use the dexy filters command with the -alias option. See Filter Documentation.

Aliases

Document keys consist of the file name plus the filters. Document keys must be unique in Dexy. This poses a problem when you want to run a file through the same filters with different combinations of settings.

- .py|pyg
- .py|pyg:
    - pyg: { noclasses: True }
- .py|pyg:
    - pyg: { linenos: True }

To differentiate, you can place an alias filter at the end of your document key. This just needs to start with a hyphen, and then can optionally have some descriptive text.

- .py|pyg
- .py|pyg|-:
    - pyg: { noclasses: True }
- .py|pyg|-with-linenos:
    - pyg: { linenos: True }

Desired Feature: Try to warn when multiple documents have same key

A situation like this:

    - .py|pyg
    - .py|pyg:
        - pyg: { noclasses: True }
    - .py|pyg:
        - pyg: { linenos: True }

Results in one dexy document, not the 3 that are expected. See if it’s possible to detect this and provide a warning to user.

  • Updated at: 2013-11-11T06:50:55Z

  • Assigned to: None

  • Milestone: None

Lists vs. Dicts

The syntax of the dexy.yaml file mixes list entries with dictionary (key: value) entries. When you have deep nesting of settings, such as in this example:

- foo.md|jinja|markdown:
    - example.py|py:
        - py:
            - add-new-files: True
            - additional-doc-filters:
                - .py: pyg
                - .rb: pyg

It helps to use a more dictionary-like syntax with curly braces and commas. This makes it more clear to read and also prevents compiler errors.

- foo.md|jinja|markdown:
    - example.py|py:
        - py: {
            add-new-files: True,
            additional-doc-filters: {.py: pyg, .rb: pyg}
          }

You can also try to reduce the amount of nesting by using named bundles.

Bundles

You can gather collections of documents together in named bundles and then refer to these bundles in other locations. This is helpful to make a more readable config file, reduce deep nesting and to re-use bundles of dependencies in different places.

docs:
    - user-guide.md|jinja|markdown:
        - title: "Dexy User Guide"
        - sources
    - developer-guide.md|jinja|markdown:
        - title: "Dexy Developer Guide"
        - sources

sources:
    - .py|pyg
    - .rb|pyg

Bundle names can also be used as target names, and using the -target command line option you can tell dexy to just run a single target rather than your entire project.

TODO: Document using bundle names with -target option.

Please feel free to ask questions or add comments about this issue on github.

You can use the dexy nodes command to view more information about the bundle node type:

$ dexy nodes -alias bundle
bundle

Acts as a wrapper for other nodes.

Scripts

Dexy guarantees that inputs are run before the documents which depend on them, but it doesn’t make any guarantees about the order in which sibling documents run. If you want to force dexy to run documents in a certain order, you do so by placing them in a bundle whose name is preceded by the script: prefix.

script:screenshots:
    - setup_script.sh|shint
    - screenshots.js|casperjs
    - teardown_script.sh|shint

The script: prefix instructs Dexy to construct a special kind of node which ensures its children are run in sequential order.

You can use the dexy nodes command to view more information about the script node type:

$ dexy nodes -alias script
script

Represents a bundle of nodes which need to run in order.

If any of the bundle siblings change, the whole bundle should be re-run.

Test that:

  • script bundles run code in order

  • change to any sibling triggers full rebuild

TODO: Test that script bundles work as described.

Please feel free to ask questions or add comments about this issue on github.

Running Dexy

Command Line Help

Dexy’s command-line interface uses python-modargs to process commands and arguments. All arguments can take any number of dashes, so -r and --r and ---r all do the same thing.

The dexy help command gives you access to information about dexy commands:

$ dexy help

For help on the main `dexy` command, run `dexy help -on dexy`.

The dexy tool includes several different commands:
  `dexy help --all` lists all available commands
  `dexy help --on XXX` provides help on a specific command

Commands for running dexy:
  `dexy` runs dexy
  `dexy setup` makes directories dexy needs
  `dexy cleanup` removes directories dexy has created
  `dexy reset` empties and resets dexy's working directories

Commands which print lists of dexy features:
  `dexy filters` filters like |jinja |py |javac
  `dexy reports` reporters like `output` and `run`
  `dexy nodes` node types and their document settings
  `dexy datas` data types and available methods
  `dexy env` elements available in document templates

Commands which print information about your project:
  (you need to be in the project dir and have run dexy already)
  `dexy grep` search for documents and keys in documents
  `dexy info` list metadata about a particular document
  `dexy targets` list target names you can run
  `dexy links` list all ways to refer to documents and sections

Other commands:
  `dexy serve` start a local static web server to view generated docs
  `dexy help` you're reading it
  `dexy version` print the version of dexy software which is installed

The --all flag will print out all the available dexy commands:

$ dexy help --all

Available commands for dexy are:
   cite
   cleanup
   conf
   datas
   dexy
   env
   fcmd
   fcmds
   filter
   filters
   gen
   grep
   help
   info
   it
   links
   nodes
   parsers
   plugins
   reporters
   reports
   reset
   serve
   setup
   targets
   template
   templates
   version

For help on a particular command type, e.g., 'dexy help -on version'

You can get help on a particular command using the -on flag:

$ dexy help --on setup
=====================
Help for 'dexy setup'
=====================
   Create the directories dexy needs to run.

   Arguments:
      artifactsdir - Where dexy should store working files.
            [optional, defaults to '.dexy'] e.g. 'dexy setup --artifactsdir .dexy'

Setting Up and Running Dexy

It can be inconvenient if you accidentally run the dexy command somewhere you didn’t mean to, like in your home directory, so dexy won’t run unless it finds a .dexy directory in the current working directory. If you try to run dexy by accident, you’ll see a message like this:

$ dexy
Oops, there's a problem running your command.
Here is some more information:
'You need to run 'dexy setup' in this directory first.'

Running dexy setup creates the .dexy directory:

$ dexy setup

And now you can run dexy:

$ dexy
dexy run finished in 0.260
dexy reports finished in 0.283

The .dexy directory is used to store working files, cached files, the dexy.log and some dexy reports:

$ ls .dexy
batch.args.pickle  dexy.log	 id_parsetab.py  reports
batches		   id_lextab.py  last		 work

Searching Dexy Batches

You can search the generated documents in the previous run via the dexy grep command, and you can get more detailed information about a document via the dexy info command.

TODO: Document "dexy grep" command.

Please feel free to ask questions or add comments about this issue on github.

There are some examples of using dexy info in this document, use control+F to search and find them.

TODO: Document "dexy info" command.

Please feel free to ask questions or add comments about this issue on github.

Serving Generated Files

The dexy serve command runs a simple web server to serve the static assets generated by dexy. This command first looks for the output-site directory generated by the ws reporter, and if it doesn’t find this it looks for the output directory generated by the output reporter. It launches a static server and prints out the port on which the files will be served.

Cache Management

Dexy stores cached files in the .dexy directory to help speed up subsequent runs. You shouldn’t have to manage this manually, but if you want to force dexy to re-run everything you can empty the cache by running dexy with the -r option or running the dexy reset command.

Dexy might also create a .trash directory although it should remove this automatically.

Filter Documentation

The filters command lets you list all available dexy filters:

$ dexy filters
Installed filters:
  abc : Runs `abcm2ps` on .abc music files.
  abcm : Runs `abcm2ps` on .abc music files, generating all output formats.
  ansi2html : Generates HTML from ANSI color codes using ansi2html.
  apis : Base class for filters which post content to a remote API.
  applytemplate : Apply template to file. Template should specify %(content)s.
  archive : Creates a .tgz archive of all input documents.
  asciidoc : Runs asciidoc command.
  asciidoctor : Runs `asciidoctor`.
  asciisyn : Surrounds code with highlighting instructions for Asciidoctor
  bash : Runs bash scripts using 'bash' and returns stdout.
  bashint : Runs bash. use to run bash scripts.
  bleach : Runs the Bleach HTML sanitizer. <https://github.com/jsocol/bleach>
  bn : Forces previous filter to output .bmp extension.
  botoup : Uses boto library to upload content to S3, returns the URL.
  bw : Converts color pdf to black and white.
  bwconv : Converts color pdf to black and white.
  c : Compile code using gcc and run.
  calibre : Runs `ebook-convert` command (part of calibre)
  casperjs : Runs scripts using casper js. Saves cookies.
  cb : Changes file extension to .sh
  cfussy : Compile code using gcc and run, raising an error if compiled code returns nonzero exit.
  ch : Changes file extension to .html
  chext : Dummy filter for allowing changing a file extension.
  cinput : Compile code using gcc and run with input.
  cj : Changes file extension to .json
  clang : Compile code using clang and run.
  clanginput : compile code using clang and run with input.
  clj : Runs clojure.
  cljws : Parse clojure code into sections based on whitespace and try to guess a
  cowsay : Runs input through 'cowsay'.
  cowthink : Runs input through 'cowthink'.
  cpickle : Forces previous filter to output .cpickle extension.
  cpp : Compile c++ code using cpp and run.
  cppinput : Compile c++ code using cpp and run with input.
  ct : Changes file extension to .txt
  customize : Add <script> tags or <link> tags to an HTML file's header.
  dexy : Filter which implements some default behaviors.
  dict : Returns an ordered dict with a single element.
  ditaa : Runs ditaa to generate images from ascii art.
  dot : Renders .dot files to either PNG or PDF images.
  dvilatex : Run Latex outputting a .dvi file.
  easyhtml : Wraps your text in HTML header/footer which includes Baseline CSS resets.
  easylatex : Wraps your text in LaTeX article header/footer.
  ebook : Runs `ebook-convert` command (part of calibre)
  elixir : Runs Elixir (.ex) files.
  embedfonts : Runs ghostscript ps2pdf with prepress settings.
  eps2pdf : Uses epstopdf to convert .eps files to .pdf
  epstopdf : Uses epstopdf to convert .eps files to .pdf
  escript : Runs Erlang scripts using the escript command.
  espeak : Runs espeak text to speech.
  f95 : Compiles and executes fortran code.
  figlet : Runs input through 'figlet'.
  filterargs : Prints out the args it receives.
  fn : Deprecated. No longer needed.
  fopdf : Uses asciidoctor-fopub to generate PDF.
  forcebmp : Forces previous filter to output .bmp extension.
  forcegif : Forces previous filter to output .gif extension.
  forcehtml : Forces previous filter to output .html extension.
  forcejpg : Forces previous filter to output .jpg extension.
  forcejson : Forces previous filter to output .json extension.
  forcelatex : Forces previous filter to output .tex extension.
  forcepdf : Forces previous filter to output .pdf extension.
  forcepng : Forces previous filter to output .png extension.
  forcer : Forces previous filter to output .R extension.
  forcesvg : Forces previous filter to output .svg extension.
  forcetext : Forces previous filter to output .txt extension.
  forcexml : Forces previous filter to output .xml extension.
  fortran : Compiles and executes fortran code.
  ft : Apply another file to bottom of file.
  gcc : Compile code using gcc and run.
  ghmd : Converts github-flavored markdown to HTML using redcarpet.
  gn : Forces previous filter to output .gif extension.
  go : Runs 'go run' command on an input .go file. http://golang.org/
  gotest : Runs 'go test' command on an input .go file. http://golang.org/
  graphviz : Renders .dot files to either PNG or PDF images.
  h : Forces previous filter to output .html extension.
  hd : Apply another file to top of file.
  head : Returns just the first 10 lines of input.
  htlatex : Generates HTML from Latex source using htlatex
  html2pdf : Deprecated, use casper.js instead.
  htmlsections : Splits files into sections based on comments like ### "foo"
  htmltidy : Uses tidy to clean and validate HTML.
  ipynb : Get data out of an IPython notebook.
  ipynbcasper : Launch IPython notebook and run a casperjs script against the server.
  ipynbx : Generates a static file based on an IPython notebook.
  ipython : Runs python code in the IPython console.
  irb : Runs ruby code in irb.
  irbout : Runs ruby scripts in irb.
  j : Forces previous filter to output .json extension.
  java : Compiles java code and runs main method.
  javac : Compiles java code and returns the .class object
  jinja : Runs the Jinja templating engine.
  jirb : Run jruby code in jirb.
  jlcon : Runs julia (.jl) files in the repl.
  jn : Forces previous filter to output .jpg extension.
  join : Takes sectioned code and joins it into a single section. Some filters which
  jruby : Run jruby code and return stdout.
  js : Runs code through rhino js interpreter.
  jsint : Runs rhino JavaScript interpeter.
  julia : Runs julia (.jl) files.
  jython : jython
  jythoni : jython in REPL
  keyvalueexample : Example of storing key value data.
  kshint : Runs ksh. Use to run bash scripts.
  kv : Creates a new key-value store.
  l : Forces previous filter to output .tex extension.
  latexdvi : Run Latex outputting a .dvi file.
  latextile : Converts textile to LaTeX using Redcloth.
  lines : Returns each line in its own section.
  livescript : Runs LiveScript (.ls) files.
  lua : Runs code through lua interpreter.
  lynxdump : Converts HTML to plain text by using lynx -dump.
  lyx : Runs lyx to generate LaTeX output.
  lyxjinja : Converts dexy:foo.txt|bar into << d['foo.txt|bar'] >>
  make : Runs make tasks.
  man : Read command names from a file and fetch man pages for each.
  markdown : Runs a Markdown processor to convert markdown to HTML.
  matlabint : Runs matlab in REPL.
  newdoc : A filter which adds an extra document to the tree.
  node : Runs scripts using node js
  nodejs : Runs scripts using node js
  org : Convert .org files to other formats.
  others : Example of accessing other documents.
  outputabc : Only outputs extension .abc
  p : Forces previous filter to output .pdf extension.
  pandoc : Convert documents to various available output formats using pandoc.
  pdf2cairo : Runs `pdftocairo` from the poppler library.
  pdf2img : Runs ghostscript to convert PDF files to images.
  pdf2jpg : Converts a PDF file to a jpg image using ghostscript.
  pdf2text : Uses pdftotext from the poppler library to convert PDFs to text.
  pdfcrop : Runs the PDFcrop script http://pdfcrop.sourceforge.net/
  pdfinfo : Uses the pdfinfo script to retrieve metadata about a PDF.
  pdftotext : Uses pdftotext from the poppler library to convert PDFs to text.
  pegdown : Converts extended markdown to HTML using pegdown.
  phantomjs : Runs scripts using phantom js.
  php : Runs php file.
  phpint : Runs PHP in interpeter mode.
  phrender : Renders HTML to PNG/PDF using phantom.js.
  pickle : Forces previous filter to output .pickle extension.
  pn : Forces previous filter to output .png extension.
  ppjson : Pretty prints JSON input.
  process : Calls `set_data` method to store output.
  processmanual : Writes output directly to output file.
  processtext : Uses process_text method
  processwithdict : Stores sectional data using `process` method.
  ps2pdf : Converts a postscript file to PDF format.
  pstopdf : Converts a postscript file to PDF format.
  py : Runs Python code and returns stdout.
  pycon : Runs python code in python's REPL.
  pydoc : Returns introspected python data in key-value storage format.
  pyg : Apply Pygments <http://pygments.org/> syntax highlighting.
  pyg4rst : Surrounds code with highlighting instructions for ReST
  pyin : Runs python code and passes input
  pyout : Runs Python code and returns stdout.
  pytest : Runs the tests in the specified Python modules.
  r : Runs R in REPL.
  ragel : Generates ruby source code from a ragel file.
  rageldot : Generates state chart in .dot format of ragel state machine.
  ragelruby : Generates ruby source code from a ragel file.
  ragelrubydot : Generates state chart in .dot format of ragel state machine for ruby.
  rb : Runs ruby scripts and return stdout.
  rbrepl : Runs ruby code in irb.
  rd2pdf : Generates a pdf from R documentation file.
  Rd2pdf : Generates a pdf from R documentation file.
  rdconv : Convert R documentation to other formats.
  redcloth : Converts textile to HTML using Redcloth.
  redclothl : Converts textile to LaTeX using Redcloth.
  regetron : Filter which loads .regex file into regetron and runs any input text against it.
  resub : Runs re.sub on each line of input.
  rhino : Runs code through rhino js interpreter.
  rhinoint : Runs rhino JavaScript interpeter.
  rint : Runs R in REPL.
  rintbatch : Runs R files in batch mode, returning an R console transcript.
  rintmock : Experimental filter to run R in sections without using pexpect.
  rlrb : Generates ruby source code from a ragel file.
  rlrbd : Generates state chart in .dot format of ragel state machine for ruby.
  rout : Runs R files in batch mode, returning just the output.
  routbatch : Runs R files in batch mode, returning just the output.
  rst : A 'native' ReST filter which uses the docutils library.
  rst2beamer : Runs rst2beamer command (docutils).
  rst2html : Convert rst to HTML
  rst2latex : Runs rst2latex command (docutils).
  rst2man : Runs rst2man command (docutils).
  rst2odt : Runs rst2odt command (docutils).
  rst2xml : Runs rst2xml command (docutils).
  rstbody : Returns just the body part of an ReST document.
  rstdocparts : Returns key-value storage of document parts.
  rstmeta : Extracts bibliographical metadata and makes this available to dexy.
  rust : Runs rust code.
  rustc : Runs rust code.
  rusti : Runs rust code in the rust repl (rusti). EXPERIMENTAL.
  scala : Compiles and runs .scala files.
  scalac : Compiles .scala code to .class files.
  scalai : Runs scala code in the REPL.
  sed : Runs a sed script.
  sh : Runs bash scripts using 'sh' and returns stdout.
  shint : Runs bash. use to run bash scripts.
  slides : Converts paragraphs to HTML and wrap each slide in a header and footer.
  sloc : Runs code through sloccount.
  sloccount : Runs code through sloccount.
  soups : Split a HTML file into nested sections based on header tags.
  split : Generate index page linking to multiple pages from single source.
  ss : Add a blank space to the start of each line.
  stata : Runs stata files.
  statai : Runs stata files.
  strings : Clean non-printing characters from text using the 'strings' tool.
  svg : Forces previous filter to output .svg extension.
  svg2pdf : Converts an SVG file to PDF by running it through casper js.
  t : Forces previous filter to output .txt extension.
  tags : Wrap text in specified HTML tags.
  taverna : Runs workflows in Taverna via command line tool.
  template : Base class for templating system filters such as JinjaFilter. Templating
  textile : Converts textile to HTML using Redcloth.
  tgzdir : Create a .tgz archive containing the unprocessed files in a directory.
  tidy : Uses tidy to clean and validate HTML.
  tidycheck : Runs `tidy` to check for valid HTML.
  tidyerrors : Uses tidy to print HTML errors.
  tikz : Renders Tikz code to PDF.
  used : Runs `sed` on the input file.
  wc : Runs input through wc command line tool.
  wiki2beamer : Converts wiki content to beamer.
  wkhtmltopdf : Deprecated, use casper.js instead.
  wordpress : Posts to a WordPress blog.
  wrap : Wraps text after 79 characters (tries to preserve existing line breaks and
  x : Forces previous filter to output .xml extension.
  xelatex : Runs .tex files using xelatex.
  xetex : Runs .tex files using xelatex.
  xmlsec : Stores all elements in the input XML document which have any of the
  yamlargs : Specify attributes in YAML at top of file.
  zip : Creates a .zip archive of all input documents.

For more information about a particular filter,
use the -alias flag and specify the filter alias.

To print the full docstring and available settings for a particular filter, use the -alias option:

$ dexy filters -alias jinja

Runs the Jinja templating engine.

aliases: jinja
tags:

Converts from file formats:
   .*

Converts to file formats:
   .*

Settings:
    add-new-files
        Boolean or list of extensions/patterns to match.
        default value: False

    added-in-version
        Dexy version when this filter was first available.
        default value:

    additional-doc-filters
        Filters to apply to additional documents created as side effects.
        default value: {}

    additional-doc-settings
        Settings to apply to additional documents created as side effects.
        default value: {}

    assertion-passed-indicator
        Extra text to return with a passed assertion.
        default value:

    block-end-string
        Tag to indicate the start of a block.
        default value: %}

    block-start-string
        Tag to indicate the start of a block.
        default value: {%

    changetags
        Automatically change from { to < based tags for .tex and .wiki files.
        default value: True

    comment-end-string
        Tag to indicate the start of a comment.
        default value: #}

    comment-start-string
        Tag to indicate the start of a comment.
        default value: {#

    data-type
        Alias of custom data class to use to store filter output.
        default value: generic

    examples
        Templates which should be used as examples for this filter.
        default value: []

    exclude-add-new-files
        List of patterns to skip even if they match add-new-files.
        default value: []

    exclude-new-files-from-dir
        List of directories to skip when adding new files.
        default value: []

    ext
        File extension to output.
        default value: None

    extension-map
        Dictionary mapping input extensions to default output extensions.
        default value: None

    filters
        List of template plugins to make into jinja filters.
        default value: ['assertions', 'highlight', 'head', 'tail', 'rstcode', 'stripjavadochtml', 'replacejinjafilters', 'bs4']

    input-extensions
        List of extensions which this filter can accept as input.
        default value: ['.*']

    jinja-path
        List of additional directories to pass to jinja loader.
        default value: []

    keep-originals
        Whether, if additional-doc-filters are specified, the
        original unmodified docs should also be added.
        default value: False

    mkdir
        A directory which should be created in working dir.
        default value: None

    mkdirs
        A list of directories which should be created in working dir.
        default value: []

    output
        Whether to output results of this filter by default by
        reporters such as 'output' or 'website'.
        default value: True

    output-extensions
        List of extensions which this filter can produce as output.
        default value: ['.*']

    override-workspace-exclude-filters
        If True, document will be populated to other workspaces
        ignoring workspace-exclude-filters.
        default value: False

    plugins
        List of plugins for run_plugins to use.
        default value: []

    preserve-prior-data-class
        Whether output data class should be set to match the input data class.
        default value: False

    require-output
        Should dexy raise an exception if no output is produced by this filter?
        default value: True

    skip-plugins
        List of plugins which run_plugins should not use.
        default value: []

    variable-end-string
        Tag to indicate the start of a variable.
        default value: }}

    variable-start-string
        Tag to indicate the start of a variable.
        default value: {{

    variables
        Variables to be made available to document.
        default value: {}

    vars
        Variables to be made available to document.
        default value: {}

    workspace-exclude-filters
        Filters whose output should be excluded from workspace.
        default value: ['pyg']

    workspace-includes
        If set to a list of filenames or extensions, only these will
        be populated to working dir.
        default value: ['.jinja']


For online docs see http://dexy.it/ref/filters/jinja

If you have suggestions or feedback about this filter,
please contact info@dexy.it

Node Documentation

You use nodes (often without knowing it) when you write dexy.yaml files. Dexy guesses the node type you want, for example a PatternNode when you use a wildcard or implicit wildcard, a Doc when you specify an individual file. You can force a node to be of particular type by prefixing its name with the node type alias and a colon, as when you create a script node via script:screenshots.

The nodes command lets you list available node types:

$ dexy nodes
bundle
doc
pattern
script
For info on a particular node type run `dexy nodes -alias doc`

To print the full docstring and available settings for a particular node, use the -alias option:

$ dexy nodes -alias doc
doc

A single Dexy document.

Settings:
    apply-ws-to-content
        If you want to put website-related content (like the link()
        function) in your content, set this to True so your content
        gets put through the jinja filter for the website reporter.
        default value: False

    apply-ws-to-content-var-end-string
        Provide a custom jinja var-end-string to avoid clashes.
        default value: None

    apply-ws-to-content-var-start-string
        Provide a custom jinja var-start-string to avoid clashes.
        default value: None

    contents
        Custom contents for a virtual document.
        default value: None

    data-type
        Alias of custom data class to use to store document content.
        default value: None

    output
        Whether document should be included in output/ and output-site/
        default value: None

    output-name
        Override default canonical name.
        default value: None

    shortcut
        A nickname for document so you don't have to use full key.
        default value: None

    title
        Custom title.
        default value: None

    ws-template
        Key of the template to apply for rendering in website.
        Setting of 'None' will use default template, 'False' will
        force no template to be used.
        default value: None

Template Env Documentation

The dexy env command gives you information about the template environment elements present. See the Templating Filters section.

$ dexy env
BeautifulSoup: The BeautifulSoup module.
DEXY_VERSION: The active dexy version. Currently 1.0.2d.
ET: The xml.etree.ElementTree module.
a: Another way to reference 'd'. Deprecated.
abs: The python builtin function abs
all: The python builtin function all
ansi2html: The convert method from ansi2html module.
any: The python builtin function any
args: The document args.
*assert_contains: Assert that input equals expected value.
*assert_does_not_contain: Assert that input equals expected value.
*assert_equals: Assert that input equals expected value.
*assert_matches: Assert that input matches the specified regular expressino.
*assert_selector_text: Asserts that the contents of CSS selector matches the expected text.
*assert_startswith: Assert that the input starts with the specified value.
attrgetter: The attrgetter method from Python's operator module.
basestring: The python builtin function basestring
bin: The python builtin function bin
bool: The python builtin function bool
bytearray: The python builtin function bytearray
cal: Shortcut for `calendar`.
caldates: List of calendar dates in current month.
calendar: A Calendar instance from Python calendar module.
callable: The python builtin function callable
camelize: The camelize method from Python inflection module.
chr: The python builtin function chr
cmp: The python builtin function cmp
complex: The python builtin function complex
d: The 'd' object.
dasherize: The dasherize method from Python inflection module.
datetime: The Python datetime module.
debug: A debugging method - prints content to command line stdout.
dict: The python builtin function dict
dir: The python builtin function dir
divmod: The python builtin function divmod
enumerate: The python builtin function enumerate
f: The filter instance for this document.
filter: The python builtin function filter
float: The python builtin function float
format: The python builtin function format
hasattr: The python builtin function hasattr
*head: Returns the first n lines of input string.
hex: The python builtin function hex
*highlight: Pygments syntax highlighter.
humanize: The humanize method from Python inflection module.
id: The python builtin function id
*indent: Jinja's indent function.
int: The python builtin function int
isinstance: The python builtin function isinstance
issubclass: The python builtin function issubclass
itemgetter: The itemgetter method from Python's operator module.
iter: The python builtin function iter
*javadoc2rst: Replace escape character with newlines and remove paragraph tags.
json: The Python json module.
len: The python builtin function len
list: The python builtin function list
load_yaml: Safely load YAML from a file.
locals: The python builtin function locals
long: The python builtin function long
map: The python builtin function map
markdown: Function which converts markdown to HTML.
max: The python builtin function max
md: Function which converts markdown to HTML.
min: The python builtin function min
month: Current month.
oct: The python builtin function oct
ord: The python builtin function ord
ordinal: The ordinal method from Python inflection module.
ordinalize: The ordinalize method from Python inflection module.
parameterize: The parameterize method from Python inflection module.
parse_yaml: Safely load YAML from text.
pformat: Pretty prints Python objects.
pluralize: The pluralize method from Python inflection module.
pow: The python builtin function pow
ppjson: Pretty prints valid JSON.
pprint: Pretty prints Python objects.
*prettify_html: Pretty-print HTML using BeautifulSoup
*pygmentize: Pygments syntax highlighter.
pygments: Dictionary of pygments stylesheets.
raise: Alias for `throw`.
range: The python builtin function range
re: The Python re module.
reduce: The python builtin function reduce
repr: The python builtin function repr
reversed: The python builtin function reversed
round: The python builtin function round
s: The data instance for this document.
set: The python builtin function set
singularize: The singularize method from Python inflection module.
slice: The python builtin function slice
sorted: The python builtin function sorted
str: The python builtin function str
*strip_javadoc_html: Replace escape character with newlines and remove paragraph tags.
subdirectories: List of subdirectories of this document.
sum: The python builtin function sum
*tail: Returns the last n lines of inptu string.
throw: A debugging utility which raises exception with argument.
time: The Python time module.
titleize: The titleize method from Python inflection module.
today: Result of datetime.today().
transliterate: The transliterate method from Python inflection module.
tuple: The python builtin function tuple
type: The python builtin function type
underscore: The underscore method from Python inflection module.
unicode: The python builtin function unicode
uuid: The Python uuid module. http://docs.python.org/2/library/uuid.html
w: The wrapper for the dexy run.
xrange: The python builtin function xrange
year: Current year.
zip: The python builtin function zip

* indicates the method can be used as a jinja template filter

Data Types Documentation

When Dexy processes a file and applies filters, each stage of processing is stored in a Data instance. There are different types of Data based on what sort of information you are storing.

The dexy datas command prints out a list of all data types:

$ dexy datas

bs4
etree
generic
keyvalue
sectioned

For more information about a particular data type,
use the -alias flag and specify the data type alias.

By default, documents start out using the Generic data type and subsequent filters may change this depending on how the filters alter the data. You can see which data type is being used for a particular document by running the dexy info command:

$ dexy info -expr hello
search expr: hello

  Info for Document 'hello.txt|jinja'

  document output data type: generic

  settings:
    canonical-name: hello.txt
    canonical-output: True
    output-name: None
    shortcut: None
    storage-type: generic
    title: None

  attributes:
    ext: .txt
    key: hello.txt|jinja
    name: hello.txt

  methods:
    basename(): hello.txt
    baserootname(): hello
    filesize(): 7
    long_name(): hello.txt-jinja.txt
    parent_dir():
    title(): Hello
    web_safe_document_key(): hello.txt-jinja.txt

    For website reporter tags, run this command with -ws option

For more information about methods available on this data type run
`dexy datas -alias generic`

You can then get more information about methods defined on the data type by running the dexy datas command, as suggested in the output of dexy info:

$ dexy datas -alias generic

generic

Data type representing generic binary or text-based data in a single blob.

Methods:
    basename

        Returns the local file name without path.


    baserootname

        Returns local file name without extension or path.


    filesize

        Returns size of file stored on disk.

        Takes arguments. Run with -source option to see source code.

    from_json

        Attempts to load data using a JSON parser, returning whatever objects
        are defined in the JSON.


    from_yaml

        Attempts to load data using a YAML parser, returning whatever objects
        are defined in the YAML.


    is_canonical_output

        Used by reports to determine if document should be written to output/
        directory.


    is_index_page

        Is this a website index page, i.e. named `index.html`.


    items

        List of sections in document.


    iteritems

        Iterable list of sections in document.


    keys

        List of keys (section names) in document.


    long_name

        A unique, but less canonical, name for the document.


    output_name

        Canonical name to output to, relative to output root. Returns None if
        artifact not in output_root.


    output_parent_dir

        Canonical output directory, taking into account custom outputroot and document name.


    output_to_file

        Write canonical output to a file. Parent directory must exist already.

        Takes arguments. Run with -source option to see source code.

    parent_dir

        The name of the directory containing the document.


    parent_output_dir

        The name of the directory containing the document based on final output
        name, which may be specified in a different directory.


    relative_path_to

        Returns a relative path from this document to the passed other
        document.

        Takes arguments. Run with -source option to see source code.

    rootname

        Returns the file name, including path, without extension.


    safe_setting

        Retrieves the setting value, but returns a default value rather than
        raising an error if the setting does not exist.

        Takes arguments. Run with -source option to see source code.

    set_data

        Shortcut to set and save data.

        Takes arguments. Run with -source option to see source code.

    setting

        Retrieves the setting value whose name is indicated by name_hyphen.

        Values starting with $ are assumed to reference environment variables,
        and the value stored in environment variables is retrieved. It's an
        error if thes corresponding environment variable it not set.

        Takes arguments. Run with -source option to see source code.

    setting_values

        Returns dict of all setting values (removes the helpstrings).

        Takes arguments. Run with -source option to see source code.

    settings_and_attributes

        Return a combined dictionary of setting values and attribute values.


    splitlines

        Returns a list of lines split at newlines or custom split.

        Takes arguments. Run with -source option to see source code.

    strip

        Returns contents stripped of leading and trailing whitespace.


    title

        Canonical title of document.

        Tries to guess from document name if `title` setting not provided.


    update_settings

        Update settings for this instance based on the provided dictionary of
        setting keys: setting values. Values should be a tuple of (helpstring,
        value,) unless the setting has already been defined in a parent class,
        in which case just pass the desired value.

        Takes arguments. Run with -source option to see source code.

    url_quoted_name

        Applies urllib's quote method to name.


    web_safe_document_key

        Returns document key with slashes replaced by double hypheens.


    websafe_key

        Returns a web-friendly version of the key.

Custom data types are a way of exposing custom methods on data. For example the bs4 data type lets you run BeautifulSoup queries on HTML content of a document.

Additional Commands

TODO: Document all remaining dexy commands.

Please feel free to ask questions or add comments about this issue on github.

Filters

This section deals with important concepts and features which are shared by all filters or groups of similar filters.

Workspaces

Many filters create a temporary workspace within the .dexy directory when they run. This workspace will mimic the directory structure of the main project and will be populated with the desired input files in their correct states (i.e. run through any applicable filters).

This provides a limited amount of isolation, in that processes are not changing files in the main project repository (unless there is a malicious or poorly-designed script), and any files generated as side effects do not clutter up the main project space.

Desired Feature: Provide a chroot option for process filters.

Provide a way to limit potential side effects by chroot-locking processes spawned by dexy filters by default.

  • Updated at: 2013-11-10T05:20:47Z

  • Assigned to: None

  • Milestone: None

In this example, a bash script is being run through the shint filter, and running the pwd command allows us to see the working directory where the code is being executed:

$ pwd
/home/ana/dev/dexy-user-guide/.dexy/work/c8/c863ad8b5cf0ea27fae7b0242812f315-002-idio-shint/examples

Check the filter documentation for each filter to see which of these workspace-related options are supported.

Using Working Directories

The use-wd boolean setting controls whether or not to create and populate a working directory and to set the process’s cwd to the working directory. The setting defaults to True.

def use_wd_option_defaults_to_true__test():
    shint = Filter.create_instance('shint')
    assert shint.setting('use-wd') == True

When use-wd is True (the default case), then a working directory is created within the .dexy/work directory.

def if_use_wd_true_code_runs_in_work_dir__test():
    with wrap() as wrapper:
        doc = Doc("test.sh|shint",
                wrapper,
                [],
                contents = "pwd",
                shint = { 'use-wd' : True }
                )

        wrapper.run_docs(doc)
        assert ".dexy/work" in str(doc.output_data())

When use-wd is set to False, the code runs directly in the project root.

def if_use_wd_false_code_runs_in_project_home__test():
    with wrap() as wrapper:
        doc = Doc("test.sh|shint",
                wrapper,
                [],
                contents = "pwd",
                shint = { 'use-wd' : False }
                )

        wrapper.run_docs(doc)
        assert not ".dexy/work" in str(doc.output_data())

Including and Excluding Inputs

Working directories can be populated with the documents specified as dependencies or inputs. This can end up being a lot of files, and sometimes we want to control more precisely which files are copied. Several settings help to manage which files are copied.

    def include_input_in_workspace(self, inpt):
        """
        Whether to include the contents of the input file inpt in the workspace
        for this filter.
        """
        workspace_includes = self.setting('workspace-includes')

        if workspace_includes is not None:
            if inpt.ext in workspace_includes:
                self.log_debug("Including %s because file extension matches." % inpt)
                return True
            elif inpt.output_data().basename() in workspace_includes:
                self.log_debug("Including %s because base name matches." % inpt)
                return True
            else:
                self.log_debug("Excluding %s because does not match workspace-includes" % inpt)
                return False

        elif not inpt.filters:
            self.log_debug("Including because %s has no filters." % inpt)
            return True

        elif inpt.filters[-1].setting('override-workspace-exclude-filters'):
            self.log_debug("Including %s because override-workspace-exclude-filters is set." % inpt)
            return True

        else:
            workspace_exclude_filters = self.setting('workspace-exclude-filters')

            if workspace_exclude_filters is None:
                self.log_debug("Including because exclude_filters is None.")
                return True
            elif any(a in workspace_exclude_filters for a in inpt.filter_aliases):
                self.log_debug("Excluding %s because of workspace-exclude-filters" % inpt)
                return False
            else:
                self.log_debug("Including %s because not excluded" % inpt)
                return True
  • workspace-exclude-filters A list of filter aliases. Input files which had these filters applied will be excluded.

  • override-workspace-exclude-filters A boolean specified on an input file. This input file will be included in working directories regardless of the parent’s workspace-exclude-filters setting.

  • workspace-include A list of filenames or wildcard patterns. These and only these will be written to the workspace. When this is set, workspace-exclude-filters and override-workspace-exclude-filters are ignored.

The workspace-exclude-filters setting takes a list of filter aliases and it doesn’t populate the working directory with any documents which include any of these filter aliases. So if jinja is in workspace-exclude-filters then a document named hello.txt|jinja will not be written to the working directory.

def workspace_exclude_filters_excluding_jinja__test():
    with wrap() as wrapper:
        yaml = """
        - test.sh|sh:
            - sh: { workspace-exclude-filters: ['jinja'] }
            - hello.txt|jinja
            - script.py|py
            - script.py|pyg

        """

        populate(yaml)
        wrapper.run_from_new()
        ls_wd = read_result(wrapper)
        assert not "hello.txt" in ls_wd
        assert ls_wd == [
                "script.py.html", # script.py|pyg
                "script.txt", # script.py|py
                "test.sh"]

To include all input files, set workspace-exclude-filters to an empty list.

def workspace_exclude_filters_no_excludes__test():
    with wrap() as wrapper:
        yaml = """
        - test.sh|sh:
            - sh: { workspace-exclude-filters: [] }
            - hello.txt|jinja
            - script.py|py
            - script.py|pyg

        """

        populate(yaml)
        wrapper.run_from_new()
        ls_wd = read_result(wrapper)

        assert ls_wd == [
                "hello.txt", # hello.txt|jinja
                "script.py.html", # script.py|pyg
                "script.txt", # script.py|py
                "test.sh"]

The workspace-exclude-filters setting defaults to ['pyg'] since usually syntax highlighted content is included in documents via templating, not via the file system. When pyg outputs image files or stylesheets, these have override-workspace-exclude-filters set to True by the filter.

def workspace_exclude_filters_pyg_defaults__test():
    with wrap() as wrapper:
        yaml = """
        - test.sh|sh:
            # Generate a pygments stylesheet.
            - pygments.css|pyg:
                - contents: ""
                - pyg: { 'ext' : '.css' }
            - script.py|pyg
            - script.py|pyg|pn

        """
        populate(yaml)
        wrapper.run_from_new()
        ls_wd = read_result(wrapper)
        assert "pygments.css" in ls_wd
        assert not "script.py.html" in ls_wd
        assert "script.py.png" in ls_wd

Making Extra Directories

Sometimes a tool expects a certain directory structure to exist when it runs, but this may not correspond to the directory structure of your project.

The mkdir and mkdirs settings let you specify extra directories which will be created in the working directory before the filter is run.

The mkdir setting creates a single directory based on a string.

def mkdir_creates_extra_directory_in_work_dir__test():
    with wrap() as wrapper:
        doc = Doc("test.sh|shint",
                wrapper,
                [],
                contents = "ls -l",
                shint = { 'mkdir' : "foo" }
                )

        wrapper.run_docs(doc)
        foo_line = str(doc.output_data()).splitlines()[2]
        assert foo_line.endswith("foo")
        assert foo_line.startswith("drw")

The mkdirs setting creates multiple directories based on a list.

def mkdirs_creates_extra_directories_in_work_dir__test():
    with wrap() as wrapper:
        doc = Doc("test.sh|shint",
                wrapper,
                [],
                contents = "ls -l",
                shint = { 'mkdirs' : ["foo", "bar"]}
                )

        wrapper.run_docs(doc)
        bar_line = str(doc.output_data()).splitlines()[2]
        foo_line = str(doc.output_data()).splitlines()[3]
        assert foo_line.endswith("foo")
        assert foo_line.startswith("drw")
        assert bar_line.endswith("bar")
        assert bar_line.startswith("drw")

Adding New Files

One of the reasons we tend to run scripts in their own working directories is because they generate extra files. LaTeX is notorious for generating .log, .aux, .bbl and a host of other files you usually aren’t interested in unless you need to debug somtehing. So by default Dexy just ignores any extra files which are created in working directories. If you need to do debugging, you can look in the working directory.

Sometimes, though, these extra files are useful and may even be the whole point of running a script. We may be generating a PNG file containing a graph, or a JSON or CSV file containing data.

The add-new-files setting controls how dexy treats these additional files.

    def add_new_files(self):
        """
        Walk working directory and add a new dexy document for every newly
        created file found.
        """
        wd = self.workspace()
        self.log_debug("adding new files found in %s for %s" % (wd, self.key))

        add_new_files = self.setting('add-new-files')
        if isinstance(add_new_files, basestring):
            add_new_files = [add_new_files]

        exclude = self.setting('exclude-add-new-files')
        skip_dirs = self.setting('exclude-new-files-from-dir')

        if isinstance(exclude, basestring):
            raise dexy.exceptions.UserFeedback("exclude-add-new-files should be a list, not a string")

        new_files_added = 0
        for dirpath, subdirs, filenames in os.walk(wd):
            # Prune subdirs which match exclude.
            subdirs[:] = [d for d in subdirs if d not in skip_dirs]

            # Iterate over files in directory.
            for filename in filenames:
                filepath = os.path.normpath(os.path.join(dirpath, filename))
                relpath = os.path.relpath(filepath, wd)
                self.log_debug("Processing %s" % filepath)

                if relpath in self._files_workspace_populated_with:
                    # already have this file
                    continue

                if isinstance(add_new_files, list):
                    is_valid_file_extension = False
                    for pattern in add_new_files:
                        if "*" in pattern:
                            if fnmatch.fnmatch(relpath, pattern):
                                is_valid_file_extension = True
                                continue
                        else:
                            if filename.endswith(pattern):
                                is_valid_file_extension = True
                                continue

                    if not is_valid_file_extension:
                        msg = "Not adding filename %s, does not match patterns: %s"
                        args = (filepath, ", ".join(add_new_files))
                        self.log_debug(msg % args)
                        continue

                elif isinstance(add_new_files, bool):
                    if not add_new_files:
                        msg = "add_new_files method should not be called if setting is False"
                        raise dexy.exceptions.InternalDexyProblem(msg)
                    is_valid_file_extension = True

                else:
                    msg = "add-new-files setting should be list or boolean. Type is %s value is %s"
                    args = (add_new_files.__class__, add_new_files,)
                    raise dexy.exceptions.InternalDexyProblem(msg % args)

                # Check if should be excluded.
                skip_because_excluded = False
                for skip_pattern in exclude:
                    if skip_pattern in filepath:
                        msg = "skipping adding new file %s because it matches exclude %s"
                        args = (filepath, skip_pattern,)
                        self.log_debug(msg % args)
                        skip_because_excluded = True
                        continue

                if skip_because_excluded:
                    continue

                if not is_valid_file_extension:
                    raise Exception("Should not get here unless is_valid_file_extension")

                with open(filepath, 'rb') as f:
                    contents = f.read()
                self.add_doc(relpath, contents)
                new_files_added += 1

        if new_files_added > 10:
            self.log_warn("%s additional files added" % (new_files_added))

By default, add-new-files is False so Dexy ignores any new files which appear in the working directory.

def process_filters_have_add_new_files_false_by_default__test():
    f = Filter.create_instance('process')
    assert not f.setting('add-new-files')

Some filters like casperjs which are almost always invoked for side effects will have add-new-files be True by default, so check the individual filter documentation.

def casperjs_has_add_new_files_true_by_default__test():
    f = Filter.create_instance('casperjs')
    assert f.setting('add-new-files')

When add-new-files is False, no new files are added to dexy.

def if_add_new_files_false_new_files_not_added__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write("touch foo.txt")

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh

            """))

        wrapper.run_from_new()
        assert not "doc:foo.txt" in wrapper.nodes.keys()

When add-new-files is True, new files are added to dexy.

def if_add_new_files_true_new_files_are_added__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write("touch foo.txt")

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: { add-new-files: True }

            """))

        wrapper.run_from_new()
        assert "doc:foo.txt" in wrapper.nodes.keys()

The add-new-files setting can also be a list of expressions to match.

Entries in the list can be file extensions which should be added.

def add_new_files_list__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write("touch foo.txt\ntouch bar.log\nls -l\n")

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: { add-new-files: [.txt] }

            """))

        wrapper.run_from_new()

        ls_wd = str(wrapper.nodes['doc:test.sh|sh'].output_data())
        assert "bar.log" in ls_wd
        assert "foo.txt" in ls_wd

        assert "doc:foo.txt" in wrapper.nodes.keys()
        assert not "doc:bar.log" in wrapper.nodes.keys()

They can also be glob-style file patterns to match.

def add_new_files_pattern__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write("touch foo.txt\ntouch bar.log\ntouch other.log\nls -l\n")

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: { add-new-files: ["b*.log"] }

            """))

        wrapper.run_from_new()

        ls_wd = str(wrapper.nodes['doc:test.sh|sh'].output_data())
        assert "foo.txt" in ls_wd
        assert "bar.log" in ls_wd
        assert "other.log" in ls_wd

        assert not "doc:foo.txt" in wrapper.nodes.keys()
        assert "doc:bar.log" in wrapper.nodes.keys()
        assert not "doc:other.log" in wrapper.nodes.keys()

There is also an exclude-add-new-files setting which lets you list exceptions so you can skip directories, file names or patterns which otherwise would be included.

def exclude_add_new_files__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write(trim("""\
                 touch foo.txt
                 touch bar.log
                 touch foo/hello.txt
                 ls -l"""))

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: {
                    add-new-files: True,
                    exclude-add-new-files: ["foo"]
                    }

            """))

        wrapper.run_from_new()

        ls_wd = str(wrapper.nodes['doc:test.sh|sh'].output_data())
        assert "foo.txt" in ls_wd
        assert "bar.log" in ls_wd

        assert not "doc:foo.txt" in wrapper.nodes.keys()
        assert "doc:bar.log" in wrapper.nodes.keys()

Additional Documents

Sometimes running a filter will cause extra documents to be added to the Dexy run. The split filter, for example, takes a HTML file and splits it into multiple files, each of which becomes an extra independent document. Extra documents may also be added as a result of the add-new-files setting (see the Adding New Files section).

When new documents are added, you may wish to customize some of their settings or specify additional filters which should be applied to the new documents. You can do this via additional-doc-filters and additional-doc-settings.

The additional-doc-filters setting can be a string listing a single filter or single filter chain (a sequence of filters separated with pipes just as you would write in a dexy file) in which case every new document has these additional filters applied.

def additional_doc_filters__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write(trim("""\
                 echo "1 + 1 = {{ 1+1 }}" > hello.txt
                 ls -l"""))

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: {
                    add-new-files: True,
                    additional-doc-filters: "jinja|ch"
                    }

            """))

        wrapper.run_from_new()

        assert not "doc:hello.txt" in wrapper.nodes
        hello_txt = wrapper.nodes["doc:hello.txt|jinja|ch"]
        assert str(hello_txt.output_data()) == "1 + 1 = 2"
        assert hello_txt.output_data().ext == ".html"

If additional-doc-filters is a list, then separate new documents are created for each filter combination in the list.

def additional_doc_filters_list__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write(trim("""\
                 echo "1 + 1 = {{ 1+1 }}" > hello.txt
                 ls -l"""))

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: {
                    add-new-files: True,
                    additional-doc-filters: ["jinja|markdown", "jinja"]
                    }

            """))

        wrapper.run_from_new()

        assert not "doc:hello.txt" in wrapper.nodes

        hello_txt = wrapper.nodes["doc:hello.txt|jinja"]
        assert str(hello_txt.output_data()) == "1 + 1 = 2"

        hello_html = wrapper.nodes["doc:hello.txt|jinja|markdown"]
        assert str(hello_html.output_data()) == "<p>1 + 1 = 2</p>"

additional-doc-filters can also be a dictionary which maps file extensions to the filters which should be applied to those file extensions. If a file is found whose extension is not in the dictionary, then that file is added without any extra filters being applied.

def additional_doc_filters_dict__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write(trim("""\
                 echo "1 + 1 = {{ 1+1 }}" > hello.txt
                 echo "print 'hello'\n" > example.py
                 touch "foo.rb"
                 ls -l"""))

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: {
                    add-new-files: True,
                    additional-doc-filters: {
                        '.txt' : 'jinja',
                        '.py' : ['py', 'pycon|pyg', 'pyg']
                    }
                    }

            """))

        wrapper.run_from_new()

        assert not "doc:hello.txt" in wrapper.nodes
        assert "doc:foo.rb" in wrapper.nodes

        assert len(wrapper.nodes) == 6

        hello_txt = wrapper.nodes["doc:hello.txt|jinja"]
        assert str(hello_txt.output_data()) == "1 + 1 = 2"

        example_py_out = wrapper.nodes["doc:example.py|py"]
        assert str(example_py_out.output_data()) == "hello\n"

        example_pycon = wrapper.nodes["doc:example.py|pycon|pyg"]
        assert "&gt;&gt;&gt;" in str(example_pycon.output_data())
        assert "&#39;hello&#39;" in str(example_pycon.output_data())

        example_pyg = wrapper.nodes["doc:example.py|pyg"]
        assert "&#39;hello&#39;" in str(example_pyg.output_data())
        assert not "&gt;&gt;&gt;" in str(example_pyg.output_data())

The keep-originals boolean setting can be combined with additional-doc-filters and it instructs Dexy to also add the original files without any extra filters applied.

def additional_doc_filters_keep_originals__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write(trim("""\
                 echo "1 + 1 = {{ 1+1 }}" > hello.txt
                 ls -l"""))

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: {
                    add-new-files: True,
                    keep-originals: True,
                    additional-doc-filters: "jinja"
                    }

            """))

        wrapper.run_from_new()

        assert "doc:hello.txt" in wrapper.nodes

        hello_txt = wrapper.nodes["doc:hello.txt|jinja"]
        assert str(hello_txt.output_data()) == "1 + 1 = 2"

The additional-doc-settings will apply extra settings to new documents. If this is a dictionary, then the entries in the dictionary are assumed to be setting names and values, and these will be applied to all new documents.

def additional_doc_settings__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write(trim("""\
                 touch hello.txt
                 ls -l"""))

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: {
                    add-new-files: True,
                    additional-doc-settings: {
                        'output' : True,
                        'ws-template' : "custom_template.html"
                        }
                    }

            """))

        wrapper.run_from_new()

        doc = wrapper.nodes["doc:hello.txt"]
        assert doc.setting('output') == True
        assert doc.setting('ws-template') == "custom_template.html"

additional-doc-settings can also be a list of lists where each element is a file extension and a dictionary of settings which will be applied to all files matching the extension. The ".*" extension can be used to provide default settings.

def additional_doc_settings_list__test():
    with wrap() as wrapper:
        with open("test.sh", 'w') as f:
            f.write(trim("""\
                 touch hello.html
                 touch hello.py
                 touch hello.rb
                 ls -l"""))

        with open("dexy.yaml", 'w') as f:
            f.write(trim("""
            - test.sh|sh:
                - sh: {
                    add-new-files: True,
                    additional-doc-filters: {
                        ".py" : "pyg",
                        ".rb" : "pyg"
                    },
                    additional-doc-settings: [
                        [".html", { "data-class" : "bs4" }],
                        [".*", { "ws-template" : "code-template.html" }],
                        ]
                    }

            """))

        wrapper.run_from_new()

        assert wrapper.nodes["doc:hello.html"].output_data().alias == 'bs4'
        assert wrapper.nodes["doc:hello.rb|pyg"].setting('ws-template') == "code-template.html"
        assert wrapper.nodes["doc:hello.py|pyg"].setting('ws-template') == "code-template.html"

Templating Filters

One of the most common things you will probably want do in dexy is to insert snippets of code into other documents using tags like {{ d['foo.py|pyg'] }} using the jinja filter. The jinja filter is an example of a templating filter, and this chapter describes how these filters work and what elements are available for you to use in your documents.

A templating tool lets you insert content into a document template. Templating tools typically evaluate template tags like {{ foo }} against an environment. An environment can be thought of as a hashmap like { "foo" : 123 }. The values in the hashmap can be simple values like 123, or they can be any type of object which is supported by the templating tool. Jinja2, for example, supports almost any kind of Python object including functions.

The TemplateFilter base class in Dexy prepares a giant hashmap containing various elements you might want to be able to refer to in your documents. It does so by running several Template Plugins, each of which returns a hashmap.

For example, the DexyVersion template plugin returns a hashmap with one entry, to let you refer to DEXY_VERSION (currently 1.0.2d) in your documents.

class DexyVersion(TemplatePlugin):
    """
    Exposes the current dexy version
    """
    aliases = ['dexyversion']
    def run(self):
        return { "DEXY_VERSION" : ("The active dexy version. Currently %s." % DEXY_VERSION, DEXY_VERSION) }

All these individual hashmaps are combined together to generate the full environment.

Subclasses of TemplateFilter take this full environment and pass it to the templating system so it can be used to evalute template tags.

Choosing Template Plugins

By default, Dexy’s TemplateFilter includes all registered template plugins when it generates the template environment. (See the Cashew docs for details about how plugin registration works.)

def by_default_dexy_runs_all_template_plugins__test():
        f = Filter.create_instance('template')
        n_plugins = len(list(f.template_plugins()))
        expected_plugins = len(list(instance for instance in TemplatePlugin))
        assert n_plugins == expected_plugins

If, instead, you want to specify which plugins to run, then you can use the plugins setting to specify a list of template plugin aliases to use.

def use_plugins_attribute_to_specify_whitelist__test():
        f = Filter.create_instance('template')
        f.update_settings({'plugins' : ['dexyversion']})
        n_plugins = len(list(f.template_plugins()))
        assert n_plugins == 1

If you just want to exclude a few plugins, then you can use the skip-plugins filter setting to list template aliases you don’t want to be used.

def use_skip_plugins_attribute_to_specify_blacklist__test():
        f = Filter.create_instance('template')
        f.update_settings({'plugins' : ['dexyversion']})
        n_plugins = len(list(f.template_plugins()))
        assert n_plugins == 1

The dexy env command prints all the environment elements which are available from running all the template plugins.

The Jinja Filter

The jinja filter is the recommended templating filter to use. It is the most widely tested and used. It uses the jinja2 templating system.

TODO verfiy this URL

Jinja has a lot of nice features, and you should familiarize yourself with the jinja template documentation.

One nice feature is jinja template filters. These are functions which you call by placing them after a pipe symbol like {{ foo | indent }}.

Yeah, we have a lot of "filters" and pipe symbols happening here, but that’s because both dexy and jinja are following similar conventions. Pipes are a classic unix concept and indicate that you take the output from one process and "pipe" it to the next process in a chain. It’s the same idea as when you write a pipe in a dexy.yaml file. I’ll refer to jinja’s implementation of filters as "jinja template filters" and dexy’s "jinja filter" as "the jinja dexy filter".

You can use all of the standard jinja template filters in dexy documents which you process through the jinja dexy filter. In addition to the built-in jinja template filters, dexy also implements some custom jinja template filters.

You can see the additional jinja template filters provided by dexy when you run dexy env. They are indicated by an asterisk.

Assertions in Documents

The jinja dexy filter allows you to make assertions about content you are including via jinja. For example, you may wish to assert that an error message does not appear in output, or you may wish to assert that a given HTML element is present, or that it contains certain text.

Here’s an example of a text file which includes output from a Python script:

Here is some python code:

    {{ d['broken.py|pycon'] | indent(4) }}

There’s no indication that anything is wrong when we run dexy, but the results show that there’s a problem:

Here is some python code:

    Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
    [GCC 4.8.1] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> printx "foo"
      File "", line 1
        printx "foo"
                   ^
    SyntaxError: invalid syntax

While Dexy does monitor for nonzero exit codes and notifies you (depending on the filter setting), not all errors will result in a nonzero exit code (for a variety of reasons). Making assertions is an alternative approach to ensuring that you are getting the content you expect from your inputs.

Now here’s the document with an assertion filter:

Here is some python code:

    {{ d['broken.py|pycon'] | assert_does_not_contain("SyntaxError") | indent(4) }}

When we run dexy, this is the result:

$ dexy
ERROR while running docs.md|jinja: input text contained 'SyntaxError'
dexy run finished in 0.488 WITH ERRORS
dexy reports finished in 0.284

After fixing the file, we can run this again:

Here is some python code:

    Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
    [GCC 4.8.1] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print 'foo'
    foo

The assertion filters return the original passed content if the assertion passes.

All available assertions can be seen by running the dexy env command:

$ dexy env | grep assert
*assert_contains: Assert that input equals expected value.
*assert_does_not_contain: Assert that input equals expected value.
*assert_equals: Assert that input equals expected value.
*assert_matches: Assert that input matches the specified regular expressino.
*assert_selector_text: Asserts that the contents of CSS selector matches the expected text.
*assert_startswith: Assert that the input starts with the specified value.

Alternative Templating Filters

There are other subclasses of TemplateFilter available, although many of these are proofs of concept. If you have a reason not to use jinja then please get in touch to discuss alternatives.

Stubbing Out Dynamic Content

#cookbook #dummyfilter

Occasionally you may want to work on the prose of a document without worrying about the automation. For example, a technical writer may wish to concentrate on writing explanations which a developer will later help pair with examples. Or you may be working on a machine which doesn’t have everything configured for generating screenshots, which aren’t important to your work anyway. You want to be able to run subsequent filters like a markdown to HTML filter without having jinja crud get in the way.

This can be accomplished by adding an alternative configuration target which calls the dummyjinja filter instead of the jinja filter. The dummyjinja filter evaluates jinja tags but instead of using a real dexy environment, it just inserts an insert stub which allows subsequent filters to run without choking on curly braces.

Reporters

Reporters are what present the output from your dexy run.

To see the available reporters, run:

$ dexy reporters
alias           default   info
d3tree          true      Generates a tree graph using d3.js
graph           true      Emits a plain text graph of the network structure.
graphviz        true      Emits a graphviz graph of the network structure.
long            false     Creates complete dexy output with files given long, unique filenames.
output          true      Creates canonical dexy output with files given short filenames.
run             true      Returns info about a dexy run.
ws              false     Applies a template to create a website from your dexy output.

The default column indicates whether the reporter is enabled by default. You can control which reporters run via the reports argument to the main dexy command. If this is blank then each report’s default setting is used. If reports is not blank, it is interpreted as a string containing space-separated report aliases to be run in order.

To see full documentation and settings for an individual reporter, use the alias argument:

$ dexy reporters -alias output
Output Reporter

settings:

  default
    Whether to run this report by default. Should be False for reports
    with side effects.
    (default: True)

  dir
    Directory where report output will be written.
    (default: 'output')

  filename
    Name of report file to generate (used when the report consists of
    just a single file).
    (default: None)

  in-cache-dir
    Whether to create report dir in the cache directory (instead of
    project root).
    (default: False)

  no-delete
    List of file names not to delete when resetting report dir (only
    effective if report dir is cleaned element-wise).
    (default: ['.git', '.nojekyll'])

  plugins
    List of template plugin aliases which should be included in jinja
    environment.
    (default: ['debug', 'inflection', 'builtins', 'operator', 'datetime', 'pprint', 'pygments', 'markdown'])

  readme-contents
    Contents to be written to README file.
    (default: 'This directory was generated by the %(alias)s dexy reporter and may be deleted without notice.\n')

  readme-filename
    Name of README file, or None to omit.
    (default: 'README.md')

  run-for-wrapper-states
    Wrapper states for which report is valid.
    (default: ['ran'])

  safety-filename
    Name of a file which will be created in generated dir to indicate
    it was created by dexy and is safe to auto-remove.
    (default: '.dexy-generated')

You can customize settings for a reporter using a dexyplugin.yaml file. An entry should start with reporter:alias: where alias is the reporter alias, and then have setting keys and new values listed in an indented block (YAML format, like dexy.yaml):

reporter:output:
    help: Store output in separate gh-pages branch.
    readme-filename: None
    dir: ../dexy-user-guide-generated

Output Reporter

The output reporter populates the output/ directory which contains short, canonical but potentially non-unique names for files. You can control whether a document appears in the output/ directory via the output setting. Setting this to True will ensure the document appears, setting this to False will ensure the document does not appear. The default behavior is determined by filters. If any filter’s default output setting is True then a document passed through that filter will default to a True setting. It’s easiest to start with defaults and then fine-tune to remove or add files from output/.

To change the name of the output/ directory, or to change any other report settings, requires writing a custom plugin, which can be in YAML or Python format.

The dir setting controls where the output is written. You can provide a different name for dir or even a different path, for example you can set the output to be in a gh-pages github branch to generate github pages content:

reporter:output:
    help: Store output in separate gh-pages branch.
    readme-filename: None
    dir: ../dexy-user-guide-generated

If you set the dir to be outside of the dexy project root then you will need to run dexy with the --writeanywhere setting, which you can pass on the command line or in a dexy.conf file:

writeanywhere: true
debug: True
loglevel: DEBUG

Website Reporter

The Website reporter is similar to the Output reporter, and it uses the same criteria to determine whether it should include a file in output-site/ or not. The difference is that the Website reporter adds some features intended to help create a website-like output, such as applying HTML templates and help with creating navigation and links to other pages.

HTML Templates

By default the website reporter looks for templates named _template.html. For a given file foo/bar/baz.html the website reporter will look first in foo/bar/, then in foo/, then in the project root for a _template.html file, so you can override templates in subdirectories without needing to do any configuration.

You can change the default template name from _template.html to something else via the default-template setting.

You can also use the ws-filter setting on an individual file to specify an alternative template file. The ws-filter setting can also take boolean values to override the default behavior of whether a template should be applied or not. By default, templates are applied unless HTML header tags are already found in the HTML.

When writing a template, a {{ content }} tag should indicate where the main content should go.

Here’s a simple HTML template:

<html>
    <head>
    </head>
    <body>
{{ content }}
    </body>
</html>

And here’s an index.html file containing some very simple content:

<p>This is page content.</p>

Here’s the index.html file in the output-site/ directory:

<html>
    <head>
    </head>
    <body>
<p>This is page content.</p>

    </body>
</html>

The project was configured with --reports setting including the ws reporter:

reports: run ws

Sectioned Documents

If HTML content has been split into sections then you can access section content via the content object, using either dictionary style access or jinja’s dot syntax (if the section name forms a valid python attribute name).

Here is an example of a HTML file with sections delimited using idio filter syntax:

<!-- @export "Introductory Section" -->
<p>This is the introductory section.</p>

<!-- @export "thanks" -->
<p>I'd like to thank Starfleet Academy.</p>

<!-- @export "footer" -->
<p>Some fine print to bury at the bottom.</p>

When this is put through the htmlsections filter the content will be split into sections:

index.html|htmlsections

The htmlsections filter is actually one of the aliases for the idio filter. It behaves a little differently than |idio: it splits content into sections following the idio rules, it doesn’t apply syntax highlighting, and then it sets output to True. It’s intended for use in just the scenario being described here.

The template can access these sections using either content.foo or content['foo']. The former is shorter, the latter works even when section names have spaces in them:

<html>
    <head>
    </head>
    <body>
    <div class="introduction">
        {{ content['Introductory Section'] }}
    </div>
    <div class="acknowledgements">
        {{ content.thanks }}
    </div>
    <footer>
        {{ content['footer'] }}
    </footer>
    </body>
</html>

Here’s the result:

None

You can also just call {{ content }} on a sectioned document and dexy will insert all sections combined.

Navigation

The website template provides several elements to assist with creating website navigation.

page_title

The page_title element contains the title of the current page:

<title>{{ page_title }}</title>

Here’s the result for index.html:

<title>Welcome</title>
Current Page Location

The source element contains the canonical output filename of the current page. The current_dir and parent_dir elements refer to the directory containing the current page and its parent (if any), respectively:

<p>This document name is: {{ source }}</p>
<p>This document is in: '{{ current_dir }}'</p>
<p>This document's parent dir is: '{{ parent_dir }}'</p>

Here’s the result for index.html:

<p>This document name is: index.html</p>
<p>This document is in: ''</p>
<p>This document's parent dir is: ''</p>

Here’s the result for foo/index.html:

<p>This document name is: foo/index.html</p>
<p>This document is in: 'foo'</p>
<p>This document's parent dir is: ''</p>

Here’s the result for foo/bar/index.html:

<p>This document name is: foo/bar/index.html</p>
<p>This document is in: 'foo/bar'</p>
<p>This document's parent dir is: 'foo'</p>
Current Page’s Data Object

The s element contains the current page’s data object. (The same information is contained in the d element, you can use either one.)

<p>This document's key is: {{ s.key }}</p>
<p>This document's file size is: {{ s.filesize() }}</p>
<p>'s' is an object of type: {{ s.__class__.__name__ }}</p>

Here’s the result for index.html:

<p>This document's key is: index.html</p>
<p>This document's file size is: 29</p>
<p>'s' is an object of type: Generic</p>

Here’s the result for foo/index.html:

<p>This document's key is: foo/index.html</p>
<p>This document's file size is: 0</p>
<p>'s' is an object of type: Generic</p>

Here’s the result for foo/bar/index.html:

<p>This document's key is: foo/bar/index.html</p>
<p>This document's file size is: 0</p>
<p>'s' is an object of type: Generic</p>
Navigation & Node Objects

The Website reporter creates a Navigation object which represents the tree of directories in the project. Each directory is represented by a Node object.

Two node objects are made directly available within a website template, the root object representing the root of the project, and the nav object representing the node corresponding to the file being processed. The whole navigation tree is availalbe via the navtree objects.

Here are some basic methods and attributes of nodes, using the nav object representing the node for the document being processed:

<p>'nav' is an object of type: {{ nav.__class__.__name__ }}</p>
<p>ancestors are {{ nav.ancestors }}</p>
<p>breadcrumbs are {{ nav.breadcrumbs() }}</p>
<p>children are {{ nav.children }}</p>
<p>children with index pages? {{ nav.has_children_with_index_pages() }}</p>
<p>docs are {{ nav.docs }}</p>
<p>index page is {{ "%r" % nav.index_page }}</p>
<p>level is {{ nav.level }}</p>
<p>location is '{{ nav.location }}'</p>
<p>parent is {{ nav.parent }}</p>

The children attribute represents subdirectories. The docs attribute represents all documents found in the node’s directory. The index_page attribute corresponds to the index.html page, if there is one, in the node’s directory. The level attribute represents the number of directories above the node’s directory in the project.

Here’s the result for index.html:

<p>'nav' is an object of type: Node</p>
<p>ancestors are [Node(/)]</p>
<p>breadcrumbs are <a href="/">Welcome</a></p>
<p>children are [Node(/foo)]</p>
<p>children with index pages? True</p>
<p>docs are [Generic('index.html')]</p>
<p>index page is Generic('index.html')</p>
<p>level is 0</p>
<p>location is '/'</p>
<p>parent is None</p>

Here’s the result for foo/index.html:

<p>'nav' is an object of type: Node</p>
<p>ancestors are [Node(/), Node(/foo)]</p>
<p>breadcrumbs are <a href="/">Welcome</a> &gt; <a href="/foo">Welcome</a></p>
<p>children are [Node(/foo/bar)]</p>
<p>children with index pages? True</p>
<p>docs are [Generic('foo/index.html')]</p>
<p>index page is Generic('foo/index.html')</p>
<p>level is 1</p>
<p>location is '/foo'</p>
<p>parent is Node(/)</p>

Here’s the result for foo/bar/index.html:

<p>'nav' is an object of type: Node</p>
<p>ancestors are [Node(/), Node(/foo), Node(/foo/bar)]</p>
<p>breadcrumbs are <a href="/">Welcome</a> &gt; <a href="/foo">Welcome</a> &gt; <a href="/foo/bar">Welcome</a></p>
<p>children are []</p>
<p>children with index pages? False</p>
<p>docs are [Generic('foo/bar/index.html')]</p>
<p>index page is Generic('foo/bar/index.html')</p>
<p>level is 2</p>
<p>location is '/foo/bar'</p>
<p>parent is Node(/foo)</p>

Here’s these attributes for the root node:

<p>'root' is an object of type: {{ root.__class__.__name__}}</p>
<p>ancestors are {{ root.ancestors }}</p>
<p>breadcrumbs are {{ root.breadcrumbs() }}</p>
<p>children are {{ root.children }}</p>
<p>children with index pages? {{ root.has_children_with_index_pages() }}</p>
<p>docs are {{ root.docs }}</p>
<p>index page is {{ "%r" % root.index_page }}</p>
<p>level is {{ root.level }}</p>
<p>location is '{{ root.location }}'</p>
<p>parent is {{ root.parent }}</p>

Here’s the result for index.html:

<p>'root' is an object of type: Node</p>
<p>ancestors are [Node(/)]</p>
<p>breadcrumbs are <a href="/">Welcome</a></p>
<p>children are [Node(/foo)]</p>
<p>children with index pages? True</p>
<p>docs are [Generic('index.html')]</p>
<p>index page is Generic('index.html')</p>
<p>level is 0</p>
<p>location is '/'</p>
<p>parent is None</p>

It’s the same for foo/bar/index.html:

<p>'root' is an object of type: Node</p>
<p>ancestors are [Node(/)]</p>
<p>breadcrumbs are <a href="/">Welcome</a></p>
<p>children are [Node(/foo)]</p>
<p>children with index pages? True</p>
<p>docs are [Generic('index.html')]</p>
<p>index page is Generic('index.html')</p>
<p>level is 0</p>
<p>location is '/'</p>
<p>parent is None</p>

A common usage is to iterate over children to produce a list of subdirectories. You can create relative navigation on each page by using the nav object, or navgation for the entire site using the root object and iterating recursively over children.

There are some macros bundled with dexy which illustrate ways to use these elements to construct site navigation. You can use them as-is or copy them and modify them for your own needs.

The Navigation object is primarily used to generate the tree, but it can also be accessed via the navtree element. Its nodes attribute is a dictionary of all nodes accessed by path. The root attribute is the root node, which you can already access via root. There is a debug method which prints out all nodes and their attributes which you can include in a document.

<p>'navtree' is an object of type: {{ navtree.__class__.__name__ }}</p>
        Nodes: {{ pprint(navtree.nodes) }}
        Root: {{ pprint(navtree.root) }}

        Debug Method:
        {{ navtree.debug() }}

Here’s the result for index.html:

<p>'navtree' is an object of type: Navigation</p>
        Nodes: {'/': Node(/), '/foo': Node(/foo), '/foo/bar': Node(/foo/bar)}
        Root: Node(/)

        Debug Method:

node: /
  index-page:
    index.html

  docs:
    index.html

  children:
    /foo


node: /foo
  index-page:
    foo/index.html

  docs:
    foo/index.html

  children:
    /foo/bar


node: /foo/bar
  index-page:
    foo/bar/index.html

  docs:
    foo/bar/index.html

The link() and section() methods allow you to create a HTML <a> link to a dexy page based on the page’s title or key and a section’s name.

The link() method links to a specified page, and optionally to a section on that page. The section() method lets you link to a specified section without needing to know which page it’s on.

Here’s an example of using these methods:

        {{ link("Home") }}
        {{ link("All About Foo") }}
        {{ link("All About Foo", section_name = "Welcome") }}
        {{ section("Welcome") }}
        {{ section("Foo") }}

And here are the results:

<a href="index.html">Home</a>
<a href="foo/index.html">All About Foo</a>
<a href="foo/index.html#welcome">Welcome</a>
<a href="foo/index.html#welcome">Welcome</a>
<a href="foo/index.html#foo">Foo</a>

The page entitled Home has its title specified in dexy.yaml:

- index.html:
    - title: Home
    - ws-template: _home.html

The page entitled All About Foo, on which we link to the Foo and Welcome sections uses two filters wihch are useful to know about.

Yamlargs

The yamlargs filter lets you add some YAML metadata to the top of a page. The YAML is stripped off and the metadata is added to the page’s settings.

title: All About Foo
---
<h1>Welcome</h1>
<p>This page is all about foo.</p>

<h2>Foo</h2>
<p>This is the foo section.</p>
Soups

The soups filter uses BeautifulSoup 4’s HTML parser to look for any <h1> or <hn> tags and creates a section for each one it finds, and also creates an id attribute for each header element so they can be linked to.

<html>
    <head>
    <title>All About Foo</title>
    </head>
    <body>
        <h1 id="welcome">Welcome</h1>
        <p>This page is all about foo.</p>
        <h2 id="foo">Foo</h2>
        <p>This is the foo section.</p>
    </body>
</html>

Listing Pages and Sections

You can use the dexy links command to print out a list of all valid keys which are possible to use as the target of the link() and section() commands:

$ dexy links
Nodes:
  'All About Foo'
    Sectioned('foo/index.html|yamlargs|soups')

  'Home'
    Generic('index.html')

  'foo/index.html'
    Sectioned('foo/index.html|yamlargs|soups')

  'foo/index.html|yamlargs|soups'
    Sectioned('foo/index.html|yamlargs|soups')

  'index.html'
    Generic('index.html')


Sections:
  'Actual Document Contents'
    Sectioned('foo/index.html|yamlargs|soups')

  'Foo'
    Sectioned('foo/index.html|yamlargs|soups')

  'Welcome'
    Sectioned('foo/index.html|yamlargs|soups')

Dexy will complain if you use a non-unique title or section name.

Other Plugins

In addiiton to the specific template elements covered above, the Website reporter makes use of many of dexy’s Template Plugins. You can configure this via the plugins setting.

Template Element Reference

The dexy info command provides information about indvidual documents in a dexy run, and this command takes a ws argument which adds customized documentation about the website template elements available, and their values, for this individual document.

Here’s an example. It includes website related tags, elements defined via template plugins, the contents of the Navigation tree, and methods available on nodes in the tree:

$ dexy info -expr hello.txt -ws
search expr: hello.txt

  Info for Document 'hello.txt|jinja'

  document output data type: generic

  settings:
    canonical-name: hello.txt
    canonical-output: True
    output-name: None
    shortcut: None
    storage-type: generic
    title: None

  attributes:
    ext: .txt
    key: hello.txt|jinja
    name: hello.txt

  methods:
    basename(): hello.txt
    baserootname(): hello
    filesize(): 7
    long_name(): hello.txt-jinja.txt
    parent_dir():
    title(): Hello
    web_safe_document_key(): hello.txt-jinja.txt

  website reporter methods:

    Website Template Environment Variables:

      Navigation and Content Related:
        current_dir: The directory containing the document being processed.
        d: 'generic' type data object representing the current doc.
        link: Function to create link to other page.
        nav: The node for the current document's directory.
        navtree: The complete navigation tree for the website.
        page_title: Title of the current page.
        parent_dir: The directory one level up from the document being processed.
        root: The root node of the navigation tree.
        s: 'generic' type data object representing the current doc.
        section: Function to create link to section on any page.
        source: Output name of current doc.
        title: Title of the current page.
        wrapper: The current wrapper object.

      navobj Node attributes (using root node):

        ancestors: [Node(/)]
        children: []
        docs: [Generic('hello.txt|jinja')]
        index_page: None
        level: 0
        location: '/'
        parent: None

      navobj Node methods (using root node):

        breadcrumbs()
          Navigation breadcrumbs showing each parent directory.


        has_children_with_index_pages()
          Boolean value.
          False


      Variables From Plugins:
        abs: The python builtin function abs
        all: The python builtin function all
        any: The python builtin function any
        attrgetter: The attrgetter method from Python's operator module.
        basestring: The python builtin function basestring
        bin: The python builtin function bin
        bool: The python builtin function bool
        bytearray: The python builtin function bytearray
        cal: Shortcut for `calendar`.
        caldates: List of calendar dates in current month.
        calendar: A Calendar instance from Python calendar module.
        callable: The python builtin function callable
        camelize: The camelize method from Python inflection module.
        chr: The python builtin function chr
        cmp: The python builtin function cmp
        complex: The python builtin function complex
        dasherize: The dasherize method from Python inflection module.
        datetime: The Python datetime module.
        debug: A debugging method - prints content to command line stdout.
        dict: The python builtin function dict
        dir: The python builtin function dir
        divmod: The python builtin function divmod
        enumerate: The python builtin function enumerate
        filter: The python builtin function filter
        float: The python builtin function float
        format: The python builtin function format
        hasattr: The python builtin function hasattr
        hex: The python builtin function hex
        humanize: The humanize method from Python inflection module.
        id: The python builtin function id
        int: The python builtin function int
        isinstance: The python builtin function isinstance
        issubclass: The python builtin function issubclass
        itemgetter: The itemgetter method from Python's operator module.
        iter: The python builtin function iter
        len: The python builtin function len
        list: The python builtin function list
        locals: The python builtin function locals
        long: The python builtin function long
        map: The python builtin function map
        markdown: Function which converts markdown to HTML.
        max: The python builtin function max
        md: Function which converts markdown to HTML.
        min: The python builtin function min
        month: Current month.
        oct: The python builtin function oct
        ord: The python builtin function ord
        ordinal: The ordinal method from Python inflection module.
        ordinalize: The ordinalize method from Python inflection module.
        parameterize: The parameterize method from Python inflection module.
        pformat: Pretty prints Python objects.
        pluralize: The pluralize method from Python inflection module.
        pow: The python builtin function pow
        pprint: Pretty prints Python objects.
        pygments: Dictionary of pygments stylesheets.
        raise: Alias for `throw`.
        range: The python builtin function range
        reduce: The python builtin function reduce
        repr: The python builtin function repr
        reversed: The python builtin function reversed
        round: The python builtin function round
        set: The python builtin function set
        singularize: The singularize method from Python inflection module.
        slice: The python builtin function slice
        sorted: The python builtin function sorted
        str: The python builtin function str
        sum: The python builtin function sum
        throw: A debugging utility which raises exception with argument.
        titleize: The titleize method from Python inflection module.
        today: Result of datetime.today().
        transliterate: The transliterate method from Python inflection module.
        tuple: The python builtin function tuple
        type: The python builtin function type
        underscore: The underscore method from Python inflection module.
        unicode: The python builtin function unicode
        xrange: The python builtin function xrange
        year: Current year.
        zip: The python builtin function zip

    navobj Nodes:

      node: /
        no index page.

        docs:
          hello.txt|jinja


  active template plugins are:
    debug, inflection, builtins, operator, datetime, pprint, pygments, markdown

For more information about methods available on this data type run
`dexy datas -alias generic`

Run Reporter

The Run reporter generates an HTML file with information about the dexy run. It is enabled by default, you can find its output in .dexy/reports/run/index.html.

Colophon

This documentation was generated by Dexy and the Asciidoctor implementation of Asciidoc.

The source code lives on Github. If you have feedback or suggestions about this document please fell free to email info@dexy.it or open a github issue.

Here is the dexy.yaml file for this document:

user-guide.adoc|jinja|asciidoctor|customize:
    # document settings
    - output-name: index.html

    # filter settings
    - customize: { scripts: ['jquery.js', 'view-result.js'] }
    - jinja: { assertion-passed-indicator: "<div style=\"background-color: green; width: 20px; height: 20px;\">OK</div>" }

    # inputs
    - assets
    - dexy-source
    - tests
    - examples
    - github-issues

assets:
    - .js

github-issues:
    - github.py|py:
        - py: {
            add-new-files: True,
            additional-doc-settings: { output: False }
        }

dexy-source:
    - modules.txt|pydoc:
        - contents: "dexy"

tests:
    - tests/*.py|pydoc:
        - tests/utils.py:
            - output: False

examples:
    - examples/*.py|idio|pycon|asciisyn
    - examples/*.sh|idio|shint|asciisyn:
        - except: "run"
        - shint: {
            add-new-files: True,
            exclude-new-files-from-dir: [".dexy"],
            keep-originals: True,
            additional-doc-filters: idio|t,
            additional-doc-settings: [['.html', { 'data-type' : 'bs4' }]]
            }
    - examples/run.sh|idio|shint|asciisyn:
        - examples/run-example.yaml
    - .conf|asciisyn:
        - asciisyn: { lexer: yaml }
    - .yaml|asciisyn
    - .yaml|idio|asciisyn:
        - idio: { ext : '.txt' }
    - .yaml|idio:
        - idio: { ext : '.txt' }