Introduction
This is the User Guide for dexy. It is intended to be a thorough guide to dexy features, but not an introductory tutorial or complete reference documentation. For links to other dexy documentation, please visit http://dexy.it/docs/
The User Guide is not complete yet, but it covers many topics and we have tried to focus on the most frustrating and confusing aspects of dexy as a priority (and hopefully we have also made them a little less frustrating in the process). In many places there are links to unit tests related to the content in this guide, you can click the shield icon to show/hide these sections. There are also embedded tickets which represent future planned development.
Telling Dexy What To Do
Dexy is an automation tool which will follow your instructions for what automated operations you would like done on your project. You give Dexy instructions about how to run your project using configuration files written using YAML syntax.
Configuration files should be named dexy.yaml.
You probably only need one dexy.yaml file in your project root, but you can place dexy.yaml files in subdirectories where they will apply only to that subdirectory and its children. The recurse option to the dexy command must be set in order for dexy.yaml files in subdirectories to be loaded.
Please feel free to ask questions or add comments about this issue on github.
The basic elements you can specify are:
-
file names/patterns and filters to apply to them
-
dependencies among documents (inputs are indented relative to parents)
-
settings for documents or filters (settings are indented under document)
def basic_yaml__test():
with open(PROJECT_ROOT + "/examples/basic.yaml", 'r') as f:
yaml = f.read()
with wrap() as wrapper:
with open("foo.md", 'w') as f:
f.write("1 + 1 {{ 1+1 }}")
with open("example.py", 'w') as f:
f.write("print 'hello'")
with open("dexy.yaml", 'w') as f:
f.write(yaml)
wrapper.run_from_new()
assert sorted(wrapper.nodes) == [
'doc:example.py|pyg',
'doc:foo.md|jinja|markdown',
'pattern:*.py|pyg']
assert "<pre>1</pre>" in str(wrapper.nodes["doc:example.py|pyg"].output_data())
assert wrapper.nodes["doc:foo.md|jinja|markdown"].setting('output')
foo.md|jinja|markdown:
- output: True # this is a document setting for foo.md|jinja|markdown
- .py|pyg: # indentation means this document is an input to foo.md|jinja|markdown
- pyg: { linenos: True } # this is a pyg filter setting for the .py|pyg document
The format for a document specification is a file name or file pattern followed by zero or more filters. Here are a few examples:
- foo.txt # no filters
- .py|pyg # 1 filter
- docs/*.md|jinja|markdown # 2 filters
You can leave off the initial asterisk if your file pattern starts with a dot, i.e. if you are matching all files with a given file extension. This is done for convenience and because if you want to start a string with an asterisk in YAML you need to wrap it in quotes.
File Names and Patterns
foo.txt instructs dexy to create a document named foo.txt where the source
is a file named foo.txt in the project root.
def config_txt_single_file__test():
with wrap() as wrapper:
write_text_files()
with open("dexy.yaml", 'w') as f:
f.write("foo.txt")
wrapper.run_from_new()
assert sorted(wrapper.nodes) == ['doc:foo.txt']
bar/foo.txt instructs dexy to create a document named bar/foo.txt where the
source is a file named foo.txt in the bar directory under the project root.
def config_txt_single_file_in_subdir__test():
with wrap() as wrapper:
write_text_files()
with open("dexy.yaml", 'w') as f:
f.write("bar/foo.txt")
wrapper.run_from_new()
assert sorted(wrapper.nodes) == ['doc:bar/foo.txt']
.txt instructs dexy to add all available files with ".txt" extension. You
could also write this as "*.txt" (note the quotes) but dexy assumes entries
starting with a dot are wildcard expressions and adds the asterisk for you.
def config_txt_ext__test():
with wrap() as wrapper:
write_text_files()
with open("dexy.yaml", 'w') as f:
f.write(".txt")
wrapper.run_from_new()
assert sorted(wrapper.nodes) == [
'doc:bar.txt',
'doc:bar/baz/foo.txt',
'doc:bar/foo.txt',
'doc:foo.txt',
'pattern:*.txt'
]
"*foo.txt" instructs dexy to add all files named foo.txt in any directory.
Note that we need to wrap the expression in quotes because we are starting with
an asterisk and can’t use the usual shortcut because we are matching a file
name, not just an extension.
def config_txt_wildcard__test():
with wrap() as wrapper:
write_text_files()
with open("dexy.yaml", 'w') as f:
f.write("\"*foo.txt\"")
wrapper.run_from_new()
assert sorted(wrapper.nodes) == [
'doc:bar/baz/foo.txt',
'doc:bar/foo.txt',
'doc:foo.txt',
'pattern:*foo.txt']
Dexy won’t complain if a wildcard expression doesn’t match any files. However, if you specify a filename, there either must be a corresponding file on the file system, or you need to specify the file’s contents using the contents setting.
Please feel free to ask questions or add comments about this issue on github.
Virtual Files
You can make dexy process a file which doesn’t really exist on the file system
by using the contents setting:
def virtual_file_contents__test():
with wrap() as wrapper:
with open("dexy.yaml", 'w') as f:
f.write(trim("""\
hello.txt|jinja:
- contents: "1 + 1 = {{ 1+1 }}"
"""))
wrapper.run_from_new()
doc = wrapper.nodes['doc:hello.txt|jinja']
assert str(doc.output_data()) == "1 + 1 = 2"
hello.txt|jinja:
- contents: "1 + 1 = {{ 1+1 }}"
Please feel free to ask questions or add comments about this issue on github.
contents is one of the available Document Settings.
Custom Name Settings
You can change the name with which a file will be output by specifying the
output-name document setting.
If the name starts with a slash / or contains any slashes it is assumed to be
a full local (within the project) path to the desired destination of the file.
If the name doesn’t contain a slash it is assumed you want to rename a file but
keep it in its original directory.
Dexy will automatically apply
string
interpolation or
string formatting
to the name if it sees a % or { character. The environment for formatting
will be populated with:
- all document settings
- name corresponding to the original document name
- dirname corresponding to the original document directory
- any local environment variables poopulated from a dexy-env.json file
Available variables are written to the dexy log at DEBUG level.
This example was run with these environment variables defined:
{
"client" : "Acme INC",
"year" : 2013
}
Here is the dexy.yaml:
baz/*.txt:
- output-name: "%(name)s-file.abc"
bar/baz.txt:
- output-name: foo-{year}-baz.txt
bar/bell.txt:
- output-name: "/bell2.txt"
foo.txt:
- output-name: foo-%(year)s.txt
And here are the input and output files:
$ find .
.
./bar
./bar/bell.txt
./bar/baz.txt
./baz
./baz/1.txt
./baz/2.txt
./baz/3.txt
./foo.txt
./.dexy
./.dexy/.dexy-generated
./dexy-env.json
./dexy.yaml
./dexy.conf
$ cd output/
$ find .
.
./.dexy-generated
./foo-2013.txt
./bar
./bar/foo-2013-baz.txt
./baz
./baz/baz
./baz/baz/1.txt-file.abc
./baz/baz/3.txt-file.abc
./baz/baz/2.txt-file.abc
./README.md
./bell2.txt
Document Settings
A key: value pair indented under a document specification is a document
setting. For example, output: True.
To see available document settings, you can use the dexy nodes command.
$ dexy nodes -alias doc
doc
A single Dexy document.
Settings:
apply-ws-to-content
If you want to put website-related content (like the link()
function) in your content, set this to True so your content
gets put through the jinja filter for the website reporter.
default value: False
apply-ws-to-content-var-end-string
Provide a custom jinja var-end-string to avoid clashes.
default value: None
apply-ws-to-content-var-start-string
Provide a custom jinja var-start-string to avoid clashes.
default value: None
contents
Custom contents for a virtual document.
default value: None
data-type
Alias of custom data class to use to store document content.
default value: None
output
Whether document should be included in output/ and output-site/
default value: None
output-name
Override default canonical name.
default value: None
shortcut
A nickname for document so you don't have to use full key.
default value: None
title
Custom title.
default value: None
ws-template
Key of the template to apply for rendering in website.
Setting of 'None' will use default template, 'False' will
force no template to be used.
default value: None
Files Without Filters
When no filters are specified for a document, the original contents of the file
are made available to other documents, and the document will appear in Dexy’s
final output unless the output setting has been set to False.
Filters
To specify filters, follow the filename/pattern with a pipe symbol | and a
filter alias. You can chain as many filters as you want, in order, by adding
more pipes and aliases. The first filter operates on the original contents of
the source file, subsequent filters operate on the output from the previous
filter.
You can run a source file through different filter combinations, and each will be a separate document in dexy.
- example.py|pyg # html file with syntax highlighted python
- example.py|py # txt file with STDOUT from executing script
Filter Settings
To customize filter settings, you need to first give the filter alias, and then a dictionary of the desired settings for that filter alias.
- example.py|idio|pycon|pyg
- pycon: { add-new-files: True}
- pyg: { linenos: True}
To see available settings for a filter, you can use the dexy filters command
with the -alias option. See Filter Documentation.
Aliases
Document keys consist of the file name plus the filters. Document keys must be unique in Dexy. This poses a problem when you want to run a file through the same filters with different combinations of settings.
- .py|pyg
- .py|pyg:
- pyg: { noclasses: True }
- .py|pyg:
- pyg: { linenos: True }
To differentiate, you can place an alias filter at the end of your document key. This just needs to start with a hyphen, and then can optionally have some descriptive text.
- .py|pyg
- .py|pyg|-:
- pyg: { noclasses: True }
- .py|pyg|-with-linenos:
- pyg: { linenos: True }
A situation like this:
- .py|pyg
- .py|pyg:
- pyg: { noclasses: True }
- .py|pyg:
- pyg: { linenos: True }
Results in one dexy document, not the 3 that are expected. See if it’s possible to detect this and provide a warning to user.
-
Updated at: 2013-11-11T06:50:55Z
-
Assigned to: None
-
Milestone: None
Lists vs. Dicts
The syntax of the dexy.yaml file mixes list entries with dictionary (key:
value) entries. When you have deep nesting of settings, such as in this example:
- foo.md|jinja|markdown:
- example.py|py:
- py:
- add-new-files: True
- additional-doc-filters:
- .py: pyg
- .rb: pyg
It helps to use a more dictionary-like syntax with curly braces and commas. This makes it more clear to read and also prevents compiler errors.
- foo.md|jinja|markdown:
- example.py|py:
- py: {
add-new-files: True,
additional-doc-filters: {.py: pyg, .rb: pyg}
}
You can also try to reduce the amount of nesting by using named bundles.
Bundles
You can gather collections of documents together in named bundles and then refer to these bundles in other locations. This is helpful to make a more readable config file, reduce deep nesting and to re-use bundles of dependencies in different places.
docs:
- user-guide.md|jinja|markdown:
- title: "Dexy User Guide"
- sources
- developer-guide.md|jinja|markdown:
- title: "Dexy Developer Guide"
- sources
sources:
- .py|pyg
- .rb|pyg
Bundle names can also be used as target names, and using the -target command line option you can tell dexy to just run a single target rather than your entire project.
Please feel free to ask questions or add comments about this issue on github.
You can use the dexy nodes command to view more information about the
bundle node type:
$ dexy nodes -alias bundle
bundle
Acts as a wrapper for other nodes.
Scripts
Dexy guarantees that inputs are run before the documents which depend on them,
but it doesn’t make any guarantees about the order in which sibling documents
run. If you want to force dexy to run documents in a certain order, you do so
by placing them in a bundle whose name is preceded by the script: prefix.
script:screenshots:
- setup_script.sh|shint
- screenshots.js|casperjs
- teardown_script.sh|shint
The script: prefix instructs Dexy to construct a special kind of node which
ensures its children are run in sequential order.
You can use the dexy nodes command to view more information about the
script node type:
$ dexy nodes -alias script
script
Represents a bundle of nodes which need to run in order.
If any of the bundle siblings change, the whole bundle should be re-run.
Test that:
-
script bundles run code in order
-
change to any sibling triggers full rebuild
Please feel free to ask questions or add comments about this issue on github.
Running Dexy
Command Line Help
Dexy’s command-line interface uses python-modargs
to process commands and arguments. All arguments can take any number of dashes,
so -r and --r and ---r all do the same thing.
The dexy help command gives you access to information about dexy commands:
$ dexy help
For help on the main `dexy` command, run `dexy help -on dexy`.
The dexy tool includes several different commands:
`dexy help --all` lists all available commands
`dexy help --on XXX` provides help on a specific command
Commands for running dexy:
`dexy` runs dexy
`dexy setup` makes directories dexy needs
`dexy cleanup` removes directories dexy has created
`dexy reset` empties and resets dexy's working directories
Commands which print lists of dexy features:
`dexy filters` filters like |jinja |py |javac
`dexy reports` reporters like `output` and `run`
`dexy nodes` node types and their document settings
`dexy datas` data types and available methods
`dexy env` elements available in document templates
Commands which print information about your project:
(you need to be in the project dir and have run dexy already)
`dexy grep` search for documents and keys in documents
`dexy info` list metadata about a particular document
`dexy targets` list target names you can run
`dexy links` list all ways to refer to documents and sections
Other commands:
`dexy serve` start a local static web server to view generated docs
`dexy help` you're reading it
`dexy version` print the version of dexy software which is installed
The --all flag will print out all the available dexy commands:
$ dexy help --all
Available commands for dexy are:
cite
cleanup
conf
datas
dexy
env
fcmd
fcmds
filter
filters
gen
grep
help
info
it
links
nodes
parsers
plugins
reporters
reports
reset
serve
setup
targets
template
templates
version
For help on a particular command type, e.g., 'dexy help -on version'
You can get help on a particular command using the -on flag:
$ dexy help --on setup
=====================
Help for 'dexy setup'
=====================
Create the directories dexy needs to run.
Arguments:
artifactsdir - Where dexy should store working files.
[optional, defaults to '.dexy'] e.g. 'dexy setup --artifactsdir .dexy'
Setting Up and Running Dexy
It can be inconvenient if you accidentally run the dexy command somewhere you
didn’t mean to, like in your home directory, so dexy won’t run unless it
finds a .dexy directory in the current working directory. If you try to run
dexy by accident, you’ll see a message like this:
$ dexy
Oops, there's a problem running your command.
Here is some more information:
'You need to run 'dexy setup' in this directory first.'
Running dexy setup creates the .dexy directory:
$ dexy setup
And now you can run dexy:
$ dexy
dexy run finished in 0.260
dexy reports finished in 0.283
The .dexy directory is used to store working files, cached files, the
dexy.log and some dexy reports:
$ ls .dexy
batch.args.pickle dexy.log id_parsetab.py reports
batches id_lextab.py last work
Searching Dexy Batches
You can search the generated documents in the previous run via the dexy grep
command, and you can get more detailed information about a document via the
dexy info command.
Please feel free to ask questions or add comments about this issue on github.
There are some examples of using dexy info in this document, use control+F to search and find them.
Please feel free to ask questions or add comments about this issue on github.
Serving Generated Files
The dexy serve command runs a simple web server to serve the static assets
generated by dexy. This command first looks for the output-site directory
generated by the ws reporter, and if it doesn’t find this it looks for the
output directory generated by the output reporter. It launches a static
server and prints out the port on which the files will be served.
Cache Management
Dexy stores cached files in the .dexy directory to help speed up subsequent
runs. You shouldn’t have to manage this manually, but if you want to force dexy
to re-run everything you can empty the cache by running dexy with the -r
option or running the dexy reset command.
Dexy might also create a .trash directory although it should remove this
automatically.
Filter Documentation
The filters command lets you list all available dexy filters:
$ dexy filters
Installed filters:
abc : Runs `abcm2ps` on .abc music files.
abcm : Runs `abcm2ps` on .abc music files, generating all output formats.
ansi2html : Generates HTML from ANSI color codes using ansi2html.
apis : Base class for filters which post content to a remote API.
applytemplate : Apply template to file. Template should specify %(content)s.
archive : Creates a .tgz archive of all input documents.
asciidoc : Runs asciidoc command.
asciidoctor : Runs `asciidoctor`.
asciisyn : Surrounds code with highlighting instructions for Asciidoctor
bash : Runs bash scripts using 'bash' and returns stdout.
bashint : Runs bash. use to run bash scripts.
bleach : Runs the Bleach HTML sanitizer. <https://github.com/jsocol/bleach>
bn : Forces previous filter to output .bmp extension.
botoup : Uses boto library to upload content to S3, returns the URL.
bw : Converts color pdf to black and white.
bwconv : Converts color pdf to black and white.
c : Compile code using gcc and run.
calibre : Runs `ebook-convert` command (part of calibre)
casperjs : Runs scripts using casper js. Saves cookies.
cb : Changes file extension to .sh
cfussy : Compile code using gcc and run, raising an error if compiled code returns nonzero exit.
ch : Changes file extension to .html
chext : Dummy filter for allowing changing a file extension.
cinput : Compile code using gcc and run with input.
cj : Changes file extension to .json
clang : Compile code using clang and run.
clanginput : compile code using clang and run with input.
clj : Runs clojure.
cljws : Parse clojure code into sections based on whitespace and try to guess a
cowsay : Runs input through 'cowsay'.
cowthink : Runs input through 'cowthink'.
cpickle : Forces previous filter to output .cpickle extension.
cpp : Compile c++ code using cpp and run.
cppinput : Compile c++ code using cpp and run with input.
ct : Changes file extension to .txt
customize : Add <script> tags or <link> tags to an HTML file's header.
dexy : Filter which implements some default behaviors.
dict : Returns an ordered dict with a single element.
ditaa : Runs ditaa to generate images from ascii art.
dot : Renders .dot files to either PNG or PDF images.
dvilatex : Run Latex outputting a .dvi file.
easyhtml : Wraps your text in HTML header/footer which includes Baseline CSS resets.
easylatex : Wraps your text in LaTeX article header/footer.
ebook : Runs `ebook-convert` command (part of calibre)
elixir : Runs Elixir (.ex) files.
embedfonts : Runs ghostscript ps2pdf with prepress settings.
eps2pdf : Uses epstopdf to convert .eps files to .pdf
epstopdf : Uses epstopdf to convert .eps files to .pdf
escript : Runs Erlang scripts using the escript command.
espeak : Runs espeak text to speech.
f95 : Compiles and executes fortran code.
figlet : Runs input through 'figlet'.
filterargs : Prints out the args it receives.
fn : Deprecated. No longer needed.
fopdf : Uses asciidoctor-fopub to generate PDF.
forcebmp : Forces previous filter to output .bmp extension.
forcegif : Forces previous filter to output .gif extension.
forcehtml : Forces previous filter to output .html extension.
forcejpg : Forces previous filter to output .jpg extension.
forcejson : Forces previous filter to output .json extension.
forcelatex : Forces previous filter to output .tex extension.
forcepdf : Forces previous filter to output .pdf extension.
forcepng : Forces previous filter to output .png extension.
forcer : Forces previous filter to output .R extension.
forcesvg : Forces previous filter to output .svg extension.
forcetext : Forces previous filter to output .txt extension.
forcexml : Forces previous filter to output .xml extension.
fortran : Compiles and executes fortran code.
ft : Apply another file to bottom of file.
gcc : Compile code using gcc and run.
ghmd : Converts github-flavored markdown to HTML using redcarpet.
gn : Forces previous filter to output .gif extension.
go : Runs 'go run' command on an input .go file. http://golang.org/
gotest : Runs 'go test' command on an input .go file. http://golang.org/
graphviz : Renders .dot files to either PNG or PDF images.
h : Forces previous filter to output .html extension.
hd : Apply another file to top of file.
head : Returns just the first 10 lines of input.
htlatex : Generates HTML from Latex source using htlatex
html2pdf : Deprecated, use casper.js instead.
htmlsections : Splits files into sections based on comments like ### "foo"
htmltidy : Uses tidy to clean and validate HTML.
ipynb : Get data out of an IPython notebook.
ipynbcasper : Launch IPython notebook and run a casperjs script against the server.
ipynbx : Generates a static file based on an IPython notebook.
ipython : Runs python code in the IPython console.
irb : Runs ruby code in irb.
irbout : Runs ruby scripts in irb.
j : Forces previous filter to output .json extension.
java : Compiles java code and runs main method.
javac : Compiles java code and returns the .class object
jinja : Runs the Jinja templating engine.
jirb : Run jruby code in jirb.
jlcon : Runs julia (.jl) files in the repl.
jn : Forces previous filter to output .jpg extension.
join : Takes sectioned code and joins it into a single section. Some filters which
jruby : Run jruby code and return stdout.
js : Runs code through rhino js interpreter.
jsint : Runs rhino JavaScript interpeter.
julia : Runs julia (.jl) files.
jython : jython
jythoni : jython in REPL
keyvalueexample : Example of storing key value data.
kshint : Runs ksh. Use to run bash scripts.
kv : Creates a new key-value store.
l : Forces previous filter to output .tex extension.
latexdvi : Run Latex outputting a .dvi file.
latextile : Converts textile to LaTeX using Redcloth.
lines : Returns each line in its own section.
livescript : Runs LiveScript (.ls) files.
lua : Runs code through lua interpreter.
lynxdump : Converts HTML to plain text by using lynx -dump.
lyx : Runs lyx to generate LaTeX output.
lyxjinja : Converts dexy:foo.txt|bar into << d['foo.txt|bar'] >>
make : Runs make tasks.
man : Read command names from a file and fetch man pages for each.
markdown : Runs a Markdown processor to convert markdown to HTML.
matlabint : Runs matlab in REPL.
newdoc : A filter which adds an extra document to the tree.
node : Runs scripts using node js
nodejs : Runs scripts using node js
org : Convert .org files to other formats.
others : Example of accessing other documents.
outputabc : Only outputs extension .abc
p : Forces previous filter to output .pdf extension.
pandoc : Convert documents to various available output formats using pandoc.
pdf2cairo : Runs `pdftocairo` from the poppler library.
pdf2img : Runs ghostscript to convert PDF files to images.
pdf2jpg : Converts a PDF file to a jpg image using ghostscript.
pdf2text : Uses pdftotext from the poppler library to convert PDFs to text.
pdfcrop : Runs the PDFcrop script http://pdfcrop.sourceforge.net/
pdfinfo : Uses the pdfinfo script to retrieve metadata about a PDF.
pdftotext : Uses pdftotext from the poppler library to convert PDFs to text.
pegdown : Converts extended markdown to HTML using pegdown.
phantomjs : Runs scripts using phantom js.
php : Runs php file.
phpint : Runs PHP in interpeter mode.
phrender : Renders HTML to PNG/PDF using phantom.js.
pickle : Forces previous filter to output .pickle extension.
pn : Forces previous filter to output .png extension.
ppjson : Pretty prints JSON input.
process : Calls `set_data` method to store output.
processmanual : Writes output directly to output file.
processtext : Uses process_text method
processwithdict : Stores sectional data using `process` method.
ps2pdf : Converts a postscript file to PDF format.
pstopdf : Converts a postscript file to PDF format.
py : Runs Python code and returns stdout.
pycon : Runs python code in python's REPL.
pydoc : Returns introspected python data in key-value storage format.
pyg : Apply Pygments <http://pygments.org/> syntax highlighting.
pyg4rst : Surrounds code with highlighting instructions for ReST
pyin : Runs python code and passes input
pyout : Runs Python code and returns stdout.
pytest : Runs the tests in the specified Python modules.
r : Runs R in REPL.
ragel : Generates ruby source code from a ragel file.
rageldot : Generates state chart in .dot format of ragel state machine.
ragelruby : Generates ruby source code from a ragel file.
ragelrubydot : Generates state chart in .dot format of ragel state machine for ruby.
rb : Runs ruby scripts and return stdout.
rbrepl : Runs ruby code in irb.
rd2pdf : Generates a pdf from R documentation file.
Rd2pdf : Generates a pdf from R documentation file.
rdconv : Convert R documentation to other formats.
redcloth : Converts textile to HTML using Redcloth.
redclothl : Converts textile to LaTeX using Redcloth.
regetron : Filter which loads .regex file into regetron and runs any input text against it.
resub : Runs re.sub on each line of input.
rhino : Runs code through rhino js interpreter.
rhinoint : Runs rhino JavaScript interpeter.
rint : Runs R in REPL.
rintbatch : Runs R files in batch mode, returning an R console transcript.
rintmock : Experimental filter to run R in sections without using pexpect.
rlrb : Generates ruby source code from a ragel file.
rlrbd : Generates state chart in .dot format of ragel state machine for ruby.
rout : Runs R files in batch mode, returning just the output.
routbatch : Runs R files in batch mode, returning just the output.
rst : A 'native' ReST filter which uses the docutils library.
rst2beamer : Runs rst2beamer command (docutils).
rst2html : Convert rst to HTML
rst2latex : Runs rst2latex command (docutils).
rst2man : Runs rst2man command (docutils).
rst2odt : Runs rst2odt command (docutils).
rst2xml : Runs rst2xml command (docutils).
rstbody : Returns just the body part of an ReST document.
rstdocparts : Returns key-value storage of document parts.
rstmeta : Extracts bibliographical metadata and makes this available to dexy.
rust : Runs rust code.
rustc : Runs rust code.
rusti : Runs rust code in the rust repl (rusti). EXPERIMENTAL.
scala : Compiles and runs .scala files.
scalac : Compiles .scala code to .class files.
scalai : Runs scala code in the REPL.
sed : Runs a sed script.
sh : Runs bash scripts using 'sh' and returns stdout.
shint : Runs bash. use to run bash scripts.
slides : Converts paragraphs to HTML and wrap each slide in a header and footer.
sloc : Runs code through sloccount.
sloccount : Runs code through sloccount.
soups : Split a HTML file into nested sections based on header tags.
split : Generate index page linking to multiple pages from single source.
ss : Add a blank space to the start of each line.
stata : Runs stata files.
statai : Runs stata files.
strings : Clean non-printing characters from text using the 'strings' tool.
svg : Forces previous filter to output .svg extension.
svg2pdf : Converts an SVG file to PDF by running it through casper js.
t : Forces previous filter to output .txt extension.
tags : Wrap text in specified HTML tags.
taverna : Runs workflows in Taverna via command line tool.
template : Base class for templating system filters such as JinjaFilter. Templating
textile : Converts textile to HTML using Redcloth.
tgzdir : Create a .tgz archive containing the unprocessed files in a directory.
tidy : Uses tidy to clean and validate HTML.
tidycheck : Runs `tidy` to check for valid HTML.
tidyerrors : Uses tidy to print HTML errors.
tikz : Renders Tikz code to PDF.
used : Runs `sed` on the input file.
wc : Runs input through wc command line tool.
wiki2beamer : Converts wiki content to beamer.
wkhtmltopdf : Deprecated, use casper.js instead.
wordpress : Posts to a WordPress blog.
wrap : Wraps text after 79 characters (tries to preserve existing line breaks and
x : Forces previous filter to output .xml extension.
xelatex : Runs .tex files using xelatex.
xetex : Runs .tex files using xelatex.
xmlsec : Stores all elements in the input XML document which have any of the
yamlargs : Specify attributes in YAML at top of file.
zip : Creates a .zip archive of all input documents.
For more information about a particular filter,
use the -alias flag and specify the filter alias.
To print the full docstring and available settings for a particular filter, use
the -alias option:
$ dexy filters -alias jinja
Runs the Jinja templating engine.
aliases: jinja
tags:
Converts from file formats:
.*
Converts to file formats:
.*
Settings:
add-new-files
Boolean or list of extensions/patterns to match.
default value: False
added-in-version
Dexy version when this filter was first available.
default value:
additional-doc-filters
Filters to apply to additional documents created as side effects.
default value: {}
additional-doc-settings
Settings to apply to additional documents created as side effects.
default value: {}
assertion-passed-indicator
Extra text to return with a passed assertion.
default value:
block-end-string
Tag to indicate the start of a block.
default value: %}
block-start-string
Tag to indicate the start of a block.
default value: {%
changetags
Automatically change from { to < based tags for .tex and .wiki files.
default value: True
comment-end-string
Tag to indicate the start of a comment.
default value: #}
comment-start-string
Tag to indicate the start of a comment.
default value: {#
data-type
Alias of custom data class to use to store filter output.
default value: generic
examples
Templates which should be used as examples for this filter.
default value: []
exclude-add-new-files
List of patterns to skip even if they match add-new-files.
default value: []
exclude-new-files-from-dir
List of directories to skip when adding new files.
default value: []
ext
File extension to output.
default value: None
extension-map
Dictionary mapping input extensions to default output extensions.
default value: None
filters
List of template plugins to make into jinja filters.
default value: ['assertions', 'highlight', 'head', 'tail', 'rstcode', 'stripjavadochtml', 'replacejinjafilters', 'bs4']
input-extensions
List of extensions which this filter can accept as input.
default value: ['.*']
jinja-path
List of additional directories to pass to jinja loader.
default value: []
keep-originals
Whether, if additional-doc-filters are specified, the
original unmodified docs should also be added.
default value: False
mkdir
A directory which should be created in working dir.
default value: None
mkdirs
A list of directories which should be created in working dir.
default value: []
output
Whether to output results of this filter by default by
reporters such as 'output' or 'website'.
default value: True
output-extensions
List of extensions which this filter can produce as output.
default value: ['.*']
override-workspace-exclude-filters
If True, document will be populated to other workspaces
ignoring workspace-exclude-filters.
default value: False
plugins
List of plugins for run_plugins to use.
default value: []
preserve-prior-data-class
Whether output data class should be set to match the input data class.
default value: False
require-output
Should dexy raise an exception if no output is produced by this filter?
default value: True
skip-plugins
List of plugins which run_plugins should not use.
default value: []
variable-end-string
Tag to indicate the start of a variable.
default value: }}
variable-start-string
Tag to indicate the start of a variable.
default value: {{
variables
Variables to be made available to document.
default value: {}
vars
Variables to be made available to document.
default value: {}
workspace-exclude-filters
Filters whose output should be excluded from workspace.
default value: ['pyg']
workspace-includes
If set to a list of filenames or extensions, only these will
be populated to working dir.
default value: ['.jinja']
For online docs see http://dexy.it/ref/filters/jinja
If you have suggestions or feedback about this filter,
please contact info@dexy.it
Node Documentation
You use nodes (often without knowing it) when you write dexy.yaml files. Dexy
guesses the node type you want, for example a PatternNode when you use a
wildcard or implicit wildcard, a Doc when you specify an individual file. You
can force a node to be of particular type by prefixing its name with the node
type alias and a colon, as when you create a script node via
script:screenshots.
The nodes command lets you list available node types:
$ dexy nodes
bundle
doc
pattern
script
For info on a particular node type run `dexy nodes -alias doc`
To print the full docstring and available settings for a particular node, use
the -alias option:
$ dexy nodes -alias doc
doc
A single Dexy document.
Settings:
apply-ws-to-content
If you want to put website-related content (like the link()
function) in your content, set this to True so your content
gets put through the jinja filter for the website reporter.
default value: False
apply-ws-to-content-var-end-string
Provide a custom jinja var-end-string to avoid clashes.
default value: None
apply-ws-to-content-var-start-string
Provide a custom jinja var-start-string to avoid clashes.
default value: None
contents
Custom contents for a virtual document.
default value: None
data-type
Alias of custom data class to use to store document content.
default value: None
output
Whether document should be included in output/ and output-site/
default value: None
output-name
Override default canonical name.
default value: None
shortcut
A nickname for document so you don't have to use full key.
default value: None
title
Custom title.
default value: None
ws-template
Key of the template to apply for rendering in website.
Setting of 'None' will use default template, 'False' will
force no template to be used.
default value: None
Template Env Documentation
The dexy env command gives you information about the template environment
elements present. See the Templating Filters section.
$ dexy env
BeautifulSoup: The BeautifulSoup module.
DEXY_VERSION: The active dexy version. Currently 1.0.2d.
ET: The xml.etree.ElementTree module.
a: Another way to reference 'd'. Deprecated.
abs: The python builtin function abs
all: The python builtin function all
ansi2html: The convert method from ansi2html module.
any: The python builtin function any
args: The document args.
*assert_contains: Assert that input equals expected value.
*assert_does_not_contain: Assert that input equals expected value.
*assert_equals: Assert that input equals expected value.
*assert_matches: Assert that input matches the specified regular expressino.
*assert_selector_text: Asserts that the contents of CSS selector matches the expected text.
*assert_startswith: Assert that the input starts with the specified value.
attrgetter: The attrgetter method from Python's operator module.
basestring: The python builtin function basestring
bin: The python builtin function bin
bool: The python builtin function bool
bytearray: The python builtin function bytearray
cal: Shortcut for `calendar`.
caldates: List of calendar dates in current month.
calendar: A Calendar instance from Python calendar module.
callable: The python builtin function callable
camelize: The camelize method from Python inflection module.
chr: The python builtin function chr
cmp: The python builtin function cmp
complex: The python builtin function complex
d: The 'd' object.
dasherize: The dasherize method from Python inflection module.
datetime: The Python datetime module.
debug: A debugging method - prints content to command line stdout.
dict: The python builtin function dict
dir: The python builtin function dir
divmod: The python builtin function divmod
enumerate: The python builtin function enumerate
f: The filter instance for this document.
filter: The python builtin function filter
float: The python builtin function float
format: The python builtin function format
hasattr: The python builtin function hasattr
*head: Returns the first n lines of input string.
hex: The python builtin function hex
*highlight: Pygments syntax highlighter.
humanize: The humanize method from Python inflection module.
id: The python builtin function id
*indent: Jinja's indent function.
int: The python builtin function int
isinstance: The python builtin function isinstance
issubclass: The python builtin function issubclass
itemgetter: The itemgetter method from Python's operator module.
iter: The python builtin function iter
*javadoc2rst: Replace escape character with newlines and remove paragraph tags.
json: The Python json module.
len: The python builtin function len
list: The python builtin function list
load_yaml: Safely load YAML from a file.
locals: The python builtin function locals
long: The python builtin function long
map: The python builtin function map
markdown: Function which converts markdown to HTML.
max: The python builtin function max
md: Function which converts markdown to HTML.
min: The python builtin function min
month: Current month.
oct: The python builtin function oct
ord: The python builtin function ord
ordinal: The ordinal method from Python inflection module.
ordinalize: The ordinalize method from Python inflection module.
parameterize: The parameterize method from Python inflection module.
parse_yaml: Safely load YAML from text.
pformat: Pretty prints Python objects.
pluralize: The pluralize method from Python inflection module.
pow: The python builtin function pow
ppjson: Pretty prints valid JSON.
pprint: Pretty prints Python objects.
*prettify_html: Pretty-print HTML using BeautifulSoup
*pygmentize: Pygments syntax highlighter.
pygments: Dictionary of pygments stylesheets.
raise: Alias for `throw`.
range: The python builtin function range
re: The Python re module.
reduce: The python builtin function reduce
repr: The python builtin function repr
reversed: The python builtin function reversed
round: The python builtin function round
s: The data instance for this document.
set: The python builtin function set
singularize: The singularize method from Python inflection module.
slice: The python builtin function slice
sorted: The python builtin function sorted
str: The python builtin function str
*strip_javadoc_html: Replace escape character with newlines and remove paragraph tags.
subdirectories: List of subdirectories of this document.
sum: The python builtin function sum
*tail: Returns the last n lines of inptu string.
throw: A debugging utility which raises exception with argument.
time: The Python time module.
titleize: The titleize method from Python inflection module.
today: Result of datetime.today().
transliterate: The transliterate method from Python inflection module.
tuple: The python builtin function tuple
type: The python builtin function type
underscore: The underscore method from Python inflection module.
unicode: The python builtin function unicode
uuid: The Python uuid module. http://docs.python.org/2/library/uuid.html
w: The wrapper for the dexy run.
xrange: The python builtin function xrange
year: Current year.
zip: The python builtin function zip
* indicates the method can be used as a jinja template filter
Data Types Documentation
When Dexy processes a file and applies filters, each stage of processing is stored in a Data instance. There are different types of Data based on what sort of information you are storing.
The dexy datas command prints out a list of all data types:
$ dexy datas
bs4
etree
generic
keyvalue
sectioned
For more information about a particular data type,
use the -alias flag and specify the data type alias.
By default, documents start out using the Generic data type and subsequent
filters may change this depending on how the filters alter the data. You can
see which data type is being used for a particular document by running the
dexy info command:
$ dexy info -expr hello
search expr: hello
Info for Document 'hello.txt|jinja'
document output data type: generic
settings:
canonical-name: hello.txt
canonical-output: True
output-name: None
shortcut: None
storage-type: generic
title: None
attributes:
ext: .txt
key: hello.txt|jinja
name: hello.txt
methods:
basename(): hello.txt
baserootname(): hello
filesize(): 7
long_name(): hello.txt-jinja.txt
parent_dir():
title(): Hello
web_safe_document_key(): hello.txt-jinja.txt
For website reporter tags, run this command with -ws option
For more information about methods available on this data type run
`dexy datas -alias generic`
You can then get more information about methods defined on the data type by
running the dexy datas command, as suggested in the output of dexy info:
$ dexy datas -alias generic
generic
Data type representing generic binary or text-based data in a single blob.
Methods:
basename
Returns the local file name without path.
baserootname
Returns local file name without extension or path.
filesize
Returns size of file stored on disk.
Takes arguments. Run with -source option to see source code.
from_json
Attempts to load data using a JSON parser, returning whatever objects
are defined in the JSON.
from_yaml
Attempts to load data using a YAML parser, returning whatever objects
are defined in the YAML.
is_canonical_output
Used by reports to determine if document should be written to output/
directory.
is_index_page
Is this a website index page, i.e. named `index.html`.
items
List of sections in document.
iteritems
Iterable list of sections in document.
keys
List of keys (section names) in document.
long_name
A unique, but less canonical, name for the document.
output_name
Canonical name to output to, relative to output root. Returns None if
artifact not in output_root.
output_parent_dir
Canonical output directory, taking into account custom outputroot and document name.
output_to_file
Write canonical output to a file. Parent directory must exist already.
Takes arguments. Run with -source option to see source code.
parent_dir
The name of the directory containing the document.
parent_output_dir
The name of the directory containing the document based on final output
name, which may be specified in a different directory.
relative_path_to
Returns a relative path from this document to the passed other
document.
Takes arguments. Run with -source option to see source code.
rootname
Returns the file name, including path, without extension.
safe_setting
Retrieves the setting value, but returns a default value rather than
raising an error if the setting does not exist.
Takes arguments. Run with -source option to see source code.
set_data
Shortcut to set and save data.
Takes arguments. Run with -source option to see source code.
setting
Retrieves the setting value whose name is indicated by name_hyphen.
Values starting with $ are assumed to reference environment variables,
and the value stored in environment variables is retrieved. It's an
error if thes corresponding environment variable it not set.
Takes arguments. Run with -source option to see source code.
setting_values
Returns dict of all setting values (removes the helpstrings).
Takes arguments. Run with -source option to see source code.
settings_and_attributes
Return a combined dictionary of setting values and attribute values.
splitlines
Returns a list of lines split at newlines or custom split.
Takes arguments. Run with -source option to see source code.
strip
Returns contents stripped of leading and trailing whitespace.
title
Canonical title of document.
Tries to guess from document name if `title` setting not provided.
update_settings
Update settings for this instance based on the provided dictionary of
setting keys: setting values. Values should be a tuple of (helpstring,
value,) unless the setting has already been defined in a parent class,
in which case just pass the desired value.
Takes arguments. Run with -source option to see source code.
url_quoted_name
Applies urllib's quote method to name.
web_safe_document_key
Returns document key with slashes replaced by double hypheens.
websafe_key
Returns a web-friendly version of the key.
Custom data types are a way of exposing custom methods on data. For example the
bs4 data type lets you run BeautifulSoup queries on HTML content of a document.
Additional Commands
Please feel free to ask questions or add comments about this issue on github.
Filters
This section deals with important concepts and features which are shared by all filters or groups of similar filters.
Workspaces
Many filters create a temporary workspace within the .dexy directory when
they run. This workspace will mimic the directory structure of the main project
and will be populated with the desired input files in their correct states
(i.e. run through any applicable filters).
This provides a limited amount of isolation, in that processes are not changing files in the main project repository (unless there is a malicious or poorly-designed script), and any files generated as side effects do not clutter up the main project space.
In this example, a bash script is being run through the shint filter, and
running the pwd command allows us to see the working directory where the code
is being executed:
$ pwd
/home/ana/dev/dexy-user-guide/.dexy/work/c8/c863ad8b5cf0ea27fae7b0242812f315-002-idio-shint/examples
Check the filter documentation for each filter to see which of these workspace-related options are supported.
Using Working Directories
The use-wd boolean setting controls whether or not to create and populate a
working directory and to set the process’s cwd to the working directory. The
setting defaults to True.
def use_wd_option_defaults_to_true__test():
shint = Filter.create_instance('shint')
assert shint.setting('use-wd') == True
When use-wd is True (the default case), then a working directory is created
within the .dexy/work directory.
def if_use_wd_true_code_runs_in_work_dir__test():
with wrap() as wrapper:
doc = Doc("test.sh|shint",
wrapper,
[],
contents = "pwd",
shint = { 'use-wd' : True }
)
wrapper.run_docs(doc)
assert ".dexy/work" in str(doc.output_data())
def if_use_wd_false_code_runs_in_project_home__test():
with wrap() as wrapper:
doc = Doc("test.sh|shint",
wrapper,
[],
contents = "pwd",
shint = { 'use-wd' : False }
)
wrapper.run_docs(doc)
assert not ".dexy/work" in str(doc.output_data())
Including and Excluding Inputs
Working directories can be populated with the documents specified as dependencies or inputs. This can end up being a lot of files, and sometimes we want to control more precisely which files are copied. Several settings help to manage which files are copied.
def include_input_in_workspace(self, inpt):
"""
Whether to include the contents of the input file inpt in the workspace
for this filter.
"""
workspace_includes = self.setting('workspace-includes')
if workspace_includes is not None:
if inpt.ext in workspace_includes:
self.log_debug("Including %s because file extension matches." % inpt)
return True
elif inpt.output_data().basename() in workspace_includes:
self.log_debug("Including %s because base name matches." % inpt)
return True
else:
self.log_debug("Excluding %s because does not match workspace-includes" % inpt)
return False
elif not inpt.filters:
self.log_debug("Including because %s has no filters." % inpt)
return True
elif inpt.filters[-1].setting('override-workspace-exclude-filters'):
self.log_debug("Including %s because override-workspace-exclude-filters is set." % inpt)
return True
else:
workspace_exclude_filters = self.setting('workspace-exclude-filters')
if workspace_exclude_filters is None:
self.log_debug("Including because exclude_filters is None.")
return True
elif any(a in workspace_exclude_filters for a in inpt.filter_aliases):
self.log_debug("Excluding %s because of workspace-exclude-filters" % inpt)
return False
else:
self.log_debug("Including %s because not excluded" % inpt)
return True
-
workspace-exclude-filtersA list of filter aliases. Input files which had these filters applied will be excluded. -
override-workspace-exclude-filtersA boolean specified on an input file. This input file will be included in working directories regardless of the parent’s workspace-exclude-filters setting. -
workspace-includeA list of filenames or wildcard patterns. These and only these will be written to the workspace. When this is set, workspace-exclude-filters and override-workspace-exclude-filters are ignored.
The workspace-exclude-filters setting takes a list of filter aliases and it
doesn’t populate the working directory with any documents which include any of
these filter aliases. So if jinja is in workspace-exclude-filters then a
document named hello.txt|jinja will not be written to the working directory.
def workspace_exclude_filters_excluding_jinja__test():
with wrap() as wrapper:
yaml = """
- test.sh|sh:
- sh: { workspace-exclude-filters: ['jinja'] }
- hello.txt|jinja
- script.py|py
- script.py|pyg
"""
populate(yaml)
wrapper.run_from_new()
ls_wd = read_result(wrapper)
assert not "hello.txt" in ls_wd
assert ls_wd == [
"script.py.html", # script.py|pyg
"script.txt", # script.py|py
"test.sh"]
def workspace_exclude_filters_no_excludes__test():
with wrap() as wrapper:
yaml = """
- test.sh|sh:
- sh: { workspace-exclude-filters: [] }
- hello.txt|jinja
- script.py|py
- script.py|pyg
"""
populate(yaml)
wrapper.run_from_new()
ls_wd = read_result(wrapper)
assert ls_wd == [
"hello.txt", # hello.txt|jinja
"script.py.html", # script.py|pyg
"script.txt", # script.py|py
"test.sh"]
The workspace-exclude-filters setting defaults to ['pyg'] since usually
syntax highlighted content is included in documents via templating, not via the
file system. When pyg outputs image files or stylesheets, these have
override-workspace-exclude-filters set to True by the filter.
def workspace_exclude_filters_pyg_defaults__test():
with wrap() as wrapper:
yaml = """
- test.sh|sh:
# Generate a pygments stylesheet.
- pygments.css|pyg:
- contents: ""
- pyg: { 'ext' : '.css' }
- script.py|pyg
- script.py|pyg|pn
"""
populate(yaml)
wrapper.run_from_new()
ls_wd = read_result(wrapper)
assert "pygments.css" in ls_wd
assert not "script.py.html" in ls_wd
assert "script.py.png" in ls_wd
Making Extra Directories
Sometimes a tool expects a certain directory structure to exist when it runs, but this may not correspond to the directory structure of your project.
The mkdir and mkdirs settings let you specify extra directories which will be created in the working directory before the filter is run.
def mkdir_creates_extra_directory_in_work_dir__test():
with wrap() as wrapper:
doc = Doc("test.sh|shint",
wrapper,
[],
contents = "ls -l",
shint = { 'mkdir' : "foo" }
)
wrapper.run_docs(doc)
foo_line = str(doc.output_data()).splitlines()[2]
assert foo_line.endswith("foo")
assert foo_line.startswith("drw")
def mkdirs_creates_extra_directories_in_work_dir__test():
with wrap() as wrapper:
doc = Doc("test.sh|shint",
wrapper,
[],
contents = "ls -l",
shint = { 'mkdirs' : ["foo", "bar"]}
)
wrapper.run_docs(doc)
bar_line = str(doc.output_data()).splitlines()[2]
foo_line = str(doc.output_data()).splitlines()[3]
assert foo_line.endswith("foo")
assert foo_line.startswith("drw")
assert bar_line.endswith("bar")
assert bar_line.startswith("drw")
Adding New Files
One of the reasons we tend to run scripts in their own working directories is
because they generate extra files. LaTeX is notorious for generating .log,
.aux, .bbl and a host of other files you usually aren’t interested in
unless you need to debug somtehing. So by default Dexy just ignores any extra
files which are created in working directories. If you need to do debugging,
you can look in the working directory.
Sometimes, though, these extra files are useful and may even be the whole point of running a script. We may be generating a PNG file containing a graph, or a JSON or CSV file containing data.
def add_new_files(self):
"""
Walk working directory and add a new dexy document for every newly
created file found.
"""
wd = self.workspace()
self.log_debug("adding new files found in %s for %s" % (wd, self.key))
add_new_files = self.setting('add-new-files')
if isinstance(add_new_files, basestring):
add_new_files = [add_new_files]
exclude = self.setting('exclude-add-new-files')
skip_dirs = self.setting('exclude-new-files-from-dir')
if isinstance(exclude, basestring):
raise dexy.exceptions.UserFeedback("exclude-add-new-files should be a list, not a string")
new_files_added = 0
for dirpath, subdirs, filenames in os.walk(wd):
# Prune subdirs which match exclude.
subdirs[:] = [d for d in subdirs if d not in skip_dirs]
# Iterate over files in directory.
for filename in filenames:
filepath = os.path.normpath(os.path.join(dirpath, filename))
relpath = os.path.relpath(filepath, wd)
self.log_debug("Processing %s" % filepath)
if relpath in self._files_workspace_populated_with:
# already have this file
continue
if isinstance(add_new_files, list):
is_valid_file_extension = False
for pattern in add_new_files:
if "*" in pattern:
if fnmatch.fnmatch(relpath, pattern):
is_valid_file_extension = True
continue
else:
if filename.endswith(pattern):
is_valid_file_extension = True
continue
if not is_valid_file_extension:
msg = "Not adding filename %s, does not match patterns: %s"
args = (filepath, ", ".join(add_new_files))
self.log_debug(msg % args)
continue
elif isinstance(add_new_files, bool):
if not add_new_files:
msg = "add_new_files method should not be called if setting is False"
raise dexy.exceptions.InternalDexyProblem(msg)
is_valid_file_extension = True
else:
msg = "add-new-files setting should be list or boolean. Type is %s value is %s"
args = (add_new_files.__class__, add_new_files,)
raise dexy.exceptions.InternalDexyProblem(msg % args)
# Check if should be excluded.
skip_because_excluded = False
for skip_pattern in exclude:
if skip_pattern in filepath:
msg = "skipping adding new file %s because it matches exclude %s"
args = (filepath, skip_pattern,)
self.log_debug(msg % args)
skip_because_excluded = True
continue
if skip_because_excluded:
continue
if not is_valid_file_extension:
raise Exception("Should not get here unless is_valid_file_extension")
with open(filepath, 'rb') as f:
contents = f.read()
self.add_doc(relpath, contents)
new_files_added += 1
if new_files_added > 10:
self.log_warn("%s additional files added" % (new_files_added))
By default, add-new-files is False so Dexy ignores any new files which appear
in the working directory.
def process_filters_have_add_new_files_false_by_default__test():
f = Filter.create_instance('process')
assert not f.setting('add-new-files')
Some filters like casperjs which are almost always invoked for side effects
will have add-new-files be True by default, so check the individual filter
documentation.
def casperjs_has_add_new_files_true_by_default__test():
f = Filter.create_instance('casperjs')
assert f.setting('add-new-files')
def if_add_new_files_false_new_files_not_added__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write("touch foo.txt")
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh
"""))
wrapper.run_from_new()
assert not "doc:foo.txt" in wrapper.nodes.keys()
def if_add_new_files_true_new_files_are_added__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write("touch foo.txt")
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: { add-new-files: True }
"""))
wrapper.run_from_new()
assert "doc:foo.txt" in wrapper.nodes.keys()
The add-new-files setting can also be a list of expressions to match.
def add_new_files_list__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write("touch foo.txt\ntouch bar.log\nls -l\n")
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: { add-new-files: [.txt] }
"""))
wrapper.run_from_new()
ls_wd = str(wrapper.nodes['doc:test.sh|sh'].output_data())
assert "bar.log" in ls_wd
assert "foo.txt" in ls_wd
assert "doc:foo.txt" in wrapper.nodes.keys()
assert not "doc:bar.log" in wrapper.nodes.keys()
def add_new_files_pattern__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write("touch foo.txt\ntouch bar.log\ntouch other.log\nls -l\n")
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: { add-new-files: ["b*.log"] }
"""))
wrapper.run_from_new()
ls_wd = str(wrapper.nodes['doc:test.sh|sh'].output_data())
assert "foo.txt" in ls_wd
assert "bar.log" in ls_wd
assert "other.log" in ls_wd
assert not "doc:foo.txt" in wrapper.nodes.keys()
assert "doc:bar.log" in wrapper.nodes.keys()
assert not "doc:other.log" in wrapper.nodes.keys()
There is also an exclude-add-new-files setting which lets you list
exceptions so you can skip directories, file names or patterns which otherwise
would be included.
def exclude_add_new_files__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write(trim("""\
touch foo.txt
touch bar.log
touch foo/hello.txt
ls -l"""))
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: {
add-new-files: True,
exclude-add-new-files: ["foo"]
}
"""))
wrapper.run_from_new()
ls_wd = str(wrapper.nodes['doc:test.sh|sh'].output_data())
assert "foo.txt" in ls_wd
assert "bar.log" in ls_wd
assert not "doc:foo.txt" in wrapper.nodes.keys()
assert "doc:bar.log" in wrapper.nodes.keys()
Additional Documents
Sometimes running a filter will cause extra documents to be added to the Dexy
run. The split filter, for example, takes a HTML file and splits it into
multiple files, each of which becomes an extra independent document. Extra
documents may also be added as a result of the add-new-files setting (see the
Adding New Files section).
When new documents are added, you may wish to customize some of their settings
or specify additional filters which should be applied to the new documents.
You can do this via additional-doc-filters and additional-doc-settings.
The additional-doc-filters setting can be a string listing a single filter or
single filter chain (a sequence of filters separated with pipes just as you
would write in a dexy file) in which case every new document has these
additional filters applied.
def additional_doc_filters__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write(trim("""\
echo "1 + 1 = {{ 1+1 }}" > hello.txt
ls -l"""))
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: {
add-new-files: True,
additional-doc-filters: "jinja|ch"
}
"""))
wrapper.run_from_new()
assert not "doc:hello.txt" in wrapper.nodes
hello_txt = wrapper.nodes["doc:hello.txt|jinja|ch"]
assert str(hello_txt.output_data()) == "1 + 1 = 2"
assert hello_txt.output_data().ext == ".html"
If additional-doc-filters is a list, then separate new documents are created
for each filter combination in the list.
def additional_doc_filters_list__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write(trim("""\
echo "1 + 1 = {{ 1+1 }}" > hello.txt
ls -l"""))
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: {
add-new-files: True,
additional-doc-filters: ["jinja|markdown", "jinja"]
}
"""))
wrapper.run_from_new()
assert not "doc:hello.txt" in wrapper.nodes
hello_txt = wrapper.nodes["doc:hello.txt|jinja"]
assert str(hello_txt.output_data()) == "1 + 1 = 2"
hello_html = wrapper.nodes["doc:hello.txt|jinja|markdown"]
assert str(hello_html.output_data()) == "<p>1 + 1 = 2</p>"
additional-doc-filters can also be a dictionary which maps file extensions to
the filters which should be applied to those file extensions. If a file is
found whose extension is not in the dictionary, then that file is added without
any extra filters being applied.
def additional_doc_filters_dict__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write(trim("""\
echo "1 + 1 = {{ 1+1 }}" > hello.txt
echo "print 'hello'\n" > example.py
touch "foo.rb"
ls -l"""))
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: {
add-new-files: True,
additional-doc-filters: {
'.txt' : 'jinja',
'.py' : ['py', 'pycon|pyg', 'pyg']
}
}
"""))
wrapper.run_from_new()
assert not "doc:hello.txt" in wrapper.nodes
assert "doc:foo.rb" in wrapper.nodes
assert len(wrapper.nodes) == 6
hello_txt = wrapper.nodes["doc:hello.txt|jinja"]
assert str(hello_txt.output_data()) == "1 + 1 = 2"
example_py_out = wrapper.nodes["doc:example.py|py"]
assert str(example_py_out.output_data()) == "hello\n"
example_pycon = wrapper.nodes["doc:example.py|pycon|pyg"]
assert ">>>" in str(example_pycon.output_data())
assert "'hello'" in str(example_pycon.output_data())
example_pyg = wrapper.nodes["doc:example.py|pyg"]
assert "'hello'" in str(example_pyg.output_data())
assert not ">>>" in str(example_pyg.output_data())
The keep-originals boolean setting can be combined with
additional-doc-filters and it instructs Dexy to also add the original files
without any extra filters applied.
def additional_doc_filters_keep_originals__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write(trim("""\
echo "1 + 1 = {{ 1+1 }}" > hello.txt
ls -l"""))
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: {
add-new-files: True,
keep-originals: True,
additional-doc-filters: "jinja"
}
"""))
wrapper.run_from_new()
assert "doc:hello.txt" in wrapper.nodes
hello_txt = wrapper.nodes["doc:hello.txt|jinja"]
assert str(hello_txt.output_data()) == "1 + 1 = 2"
The additional-doc-settings will apply extra settings to new documents. If
this is a dictionary, then the entries in the dictionary are assumed to be
setting names and values, and these will be applied to all new documents.
def additional_doc_settings__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write(trim("""\
touch hello.txt
ls -l"""))
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: {
add-new-files: True,
additional-doc-settings: {
'output' : True,
'ws-template' : "custom_template.html"
}
}
"""))
wrapper.run_from_new()
doc = wrapper.nodes["doc:hello.txt"]
assert doc.setting('output') == True
assert doc.setting('ws-template') == "custom_template.html"
additional-doc-settings can also be a list of lists where each element is a
file extension and a dictionary of settings which will be applied to all files
matching the extension. The ".*" extension can be used to provide default
settings.
def additional_doc_settings_list__test():
with wrap() as wrapper:
with open("test.sh", 'w') as f:
f.write(trim("""\
touch hello.html
touch hello.py
touch hello.rb
ls -l"""))
with open("dexy.yaml", 'w') as f:
f.write(trim("""
- test.sh|sh:
- sh: {
add-new-files: True,
additional-doc-filters: {
".py" : "pyg",
".rb" : "pyg"
},
additional-doc-settings: [
[".html", { "data-class" : "bs4" }],
[".*", { "ws-template" : "code-template.html" }],
]
}
"""))
wrapper.run_from_new()
assert wrapper.nodes["doc:hello.html"].output_data().alias == 'bs4'
assert wrapper.nodes["doc:hello.rb|pyg"].setting('ws-template') == "code-template.html"
assert wrapper.nodes["doc:hello.py|pyg"].setting('ws-template') == "code-template.html"
Templating Filters
One of the most common things you will probably want do in dexy is to insert
snippets of code into other documents using tags like
{{ d['foo.py|pyg'] }} using the jinja filter. The jinja filter is
an example of a templating filter, and this chapter describes how these filters
work and what elements are available for you to use in your documents.
A templating tool lets you insert content into a document template. Templating
tools typically evaluate template tags like {{ foo }} against an
environment. An environment can be thought of as a hashmap like { "foo" : 123
}. The values in the hashmap can be simple values like 123, or they can be
any type of object which is supported by the templating tool. Jinja2, for
example, supports almost any kind of Python object including functions.
The TemplateFilter base class in Dexy prepares a giant hashmap containing
various elements you might want to be able to refer to in your documents. It
does so by running several Template Plugins, each of which returns a hashmap.
For example, the DexyVersion template plugin returns a hashmap with one entry,
to let you refer to DEXY_VERSION (currently 1.0.2d) in your
documents.
class DexyVersion(TemplatePlugin):
"""
Exposes the current dexy version
"""
aliases = ['dexyversion']
def run(self):
return { "DEXY_VERSION" : ("The active dexy version. Currently %s." % DEXY_VERSION, DEXY_VERSION) }
All these individual hashmaps are combined together to generate the full environment.
Subclasses of TemplateFilter take this full environment and pass it to the templating system so it can be used to evalute template tags.
Choosing Template Plugins
By default, Dexy’s TemplateFilter includes all registered template plugins when it generates the template environment. (See the Cashew docs for details about how plugin registration works.)
def by_default_dexy_runs_all_template_plugins__test():
f = Filter.create_instance('template')
n_plugins = len(list(f.template_plugins()))
expected_plugins = len(list(instance for instance in TemplatePlugin))
assert n_plugins == expected_plugins
If, instead, you want to specify which plugins to run, then you can use the
plugins setting to specify a list of template plugin aliases to use.
def use_plugins_attribute_to_specify_whitelist__test():
f = Filter.create_instance('template')
f.update_settings({'plugins' : ['dexyversion']})
n_plugins = len(list(f.template_plugins()))
assert n_plugins == 1
If you just want to exclude a few plugins, then you can use the skip-plugins
filter setting to list template aliases you don’t want to be used.
def use_skip_plugins_attribute_to_specify_blacklist__test():
f = Filter.create_instance('template')
f.update_settings({'plugins' : ['dexyversion']})
n_plugins = len(list(f.template_plugins()))
assert n_plugins == 1
The dexy env command prints all the environment elements which are available
from running all the template plugins.
The Jinja Filter
The jinja filter is the recommended templating filter to use. It is the most
widely tested and used. It uses the jinja2 templating system.
TODO verfiy this URL
Jinja has a lot of nice features, and you should familiarize yourself with the jinja template documentation.
One nice feature is jinja template filters. These are functions which you call by placing them after a pipe symbol like {{ foo | indent }}.
Yeah, we have a lot of "filters" and pipe symbols happening here, but that’s
because both dexy and jinja are following similar conventions. Pipes are a
classic unix concept and indicate that you take the output from one process and
"pipe" it to the next process in a chain. It’s the same idea as when you write
a pipe in a dexy.yaml file. I’ll refer to jinja’s implementation of filters
as "jinja template filters" and dexy’s "jinja filter" as "the jinja dexy
filter".
You can use all of the standard jinja template filters in dexy documents which you process through the jinja dexy filter. In addition to the built-in jinja template filters, dexy also implements some custom jinja template filters.
You can see the additional jinja template filters provided by dexy when you run
dexy env. They are indicated by an asterisk.
Assertions in Documents
The jinja dexy filter allows you to make assertions about content you are including via jinja. For example, you may wish to assert that an error message does not appear in output, or you may wish to assert that a given HTML element is present, or that it contains certain text.
Here’s an example of a text file which includes output from a Python script:
Here is some python code:
{{ d['broken.py|pycon'] | indent(4) }}
There’s no indication that anything is wrong when we run dexy, but the results show that there’s a problem:
Here is some python code:
Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> printx "foo"
File "", line 1
printx "foo"
^
SyntaxError: invalid syntax While Dexy does monitor for nonzero exit codes and notifies you (depending on the filter setting), not all errors will result in a nonzero exit code (for a variety of reasons). Making assertions is an alternative approach to ensuring that you are getting the content you expect from your inputs.
Now here’s the document with an assertion filter:
Here is some python code:
{{ d['broken.py|pycon'] | assert_does_not_contain("SyntaxError") | indent(4) }}
When we run dexy, this is the result:
$ dexy
ERROR while running docs.md|jinja: input text contained 'SyntaxError'
dexy run finished in 0.488 WITH ERRORS
dexy reports finished in 0.284
After fixing the file, we can run this again:
Here is some python code:
Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print 'foo'
fooThe assertion filters return the original passed content if the assertion passes.
All available assertions can be seen by running the dexy env command:
$ dexy env | grep assert
*assert_contains: Assert that input equals expected value.
*assert_does_not_contain: Assert that input equals expected value.
*assert_equals: Assert that input equals expected value.
*assert_matches: Assert that input matches the specified regular expressino.
*assert_selector_text: Asserts that the contents of CSS selector matches the expected text.
*assert_startswith: Assert that the input starts with the specified value.
Alternative Templating Filters
There are other subclasses of TemplateFilter available, although many of
these are proofs of concept. If you have a reason not to use jinja then please
get in touch to discuss alternatives.
Stubbing Out Dynamic Content
#cookbook #dummyfilter
Occasionally you may want to work on the prose of a document without worrying about the automation. For example, a technical writer may wish to concentrate on writing explanations which a developer will later help pair with examples. Or you may be working on a machine which doesn’t have everything configured for generating screenshots, which aren’t important to your work anyway. You want to be able to run subsequent filters like a markdown to HTML filter without having jinja crud get in the way.
This can be accomplished by adding an alternative configuration target which
calls the dummyjinja filter instead of the jinja filter. The dummyjinja
filter evaluates jinja tags but instead of using a real dexy environment, it
just inserts an insert stub which allows subsequent filters to run without
choking on curly braces.
Reporters
Reporters are what present the output from your dexy run.
To see the available reporters, run:
$ dexy reporters
alias default info
d3tree true Generates a tree graph using d3.js
graph true Emits a plain text graph of the network structure.
graphviz true Emits a graphviz graph of the network structure.
long false Creates complete dexy output with files given long, unique filenames.
output true Creates canonical dexy output with files given short filenames.
run true Returns info about a dexy run.
ws false Applies a template to create a website from your dexy output.
The default column indicates whether the reporter is enabled by default. You
can control which reporters run via the reports argument to the main dexy
command. If this is blank then each report’s default setting is used. If
reports is not blank, it is interpreted as a string containing
space-separated report aliases to be run in order.
To see full documentation and settings for an individual reporter, use the
alias argument:
$ dexy reporters -alias output
Output Reporter
settings:
default
Whether to run this report by default. Should be False for reports
with side effects.
(default: True)
dir
Directory where report output will be written.
(default: 'output')
filename
Name of report file to generate (used when the report consists of
just a single file).
(default: None)
in-cache-dir
Whether to create report dir in the cache directory (instead of
project root).
(default: False)
no-delete
List of file names not to delete when resetting report dir (only
effective if report dir is cleaned element-wise).
(default: ['.git', '.nojekyll'])
plugins
List of template plugin aliases which should be included in jinja
environment.
(default: ['debug', 'inflection', 'builtins', 'operator', 'datetime', 'pprint', 'pygments', 'markdown'])
readme-contents
Contents to be written to README file.
(default: 'This directory was generated by the %(alias)s dexy reporter and may be deleted without notice.\n')
readme-filename
Name of README file, or None to omit.
(default: 'README.md')
run-for-wrapper-states
Wrapper states for which report is valid.
(default: ['ran'])
safety-filename
Name of a file which will be created in generated dir to indicate
it was created by dexy and is safe to auto-remove.
(default: '.dexy-generated')
You can customize settings for a reporter using a dexyplugin.yaml file. An
entry should start with reporter:alias: where alias is the reporter alias,
and then have setting keys and new values listed in an indented block (YAML
format, like dexy.yaml):
reporter:output:
help: Store output in separate gh-pages branch.
readme-filename: None
dir: ../dexy-user-guide-generated
Output Reporter
The output reporter populates the output/ directory which contains short,
canonical but potentially non-unique names for files. You can control whether a
document appears in the output/ directory via the output setting. Setting
this to True will ensure the document appears, setting this to False will
ensure the document does not appear. The default behavior is determined by
filters. If any filter’s default output setting is True then a document
passed through that filter will default to a True setting. It’s easiest to
start with defaults and then fine-tune to remove or add files from output/.
To change the name of the output/ directory, or to change any other report
settings, requires writing a custom plugin, which can be in YAML or Python
format.
The dir setting controls where the output is written. You can provide a
different name for dir or even a different path, for example you can set the
output to be in a gh-pages github branch to generate github pages content:
reporter:output:
help: Store output in separate gh-pages branch.
readme-filename: None
dir: ../dexy-user-guide-generated
If you set the dir to be outside of the dexy project root then you will need
to run dexy with the --writeanywhere setting, which you can pass on the
command line or in a dexy.conf file:
writeanywhere: true
debug: True
loglevel: DEBUG
Website Reporter
The Website reporter is similar to the Output reporter, and it uses the same
criteria to determine whether it should include a file in output-site/ or
not. The difference is that the Website reporter adds some features intended to
help create a website-like output, such as applying HTML templates and help
with creating navigation and links to other pages.
HTML Templates
By default the website reporter looks for templates named _template.html. For
a given file foo/bar/baz.html the website reporter will look first in
foo/bar/, then in foo/, then in the project root for a _template.html
file, so you can override templates in subdirectories without needing to do any
configuration.
You can change the default template name from _template.html to something
else via the default-template setting.
You can also use the ws-filter setting on an individual file to specify an
alternative template file. The ws-filter setting can also take boolean values
to override the default behavior of whether a template should be applied or
not. By default, templates are applied unless HTML header tags are already
found in the HTML.
When writing a template, a {{ content }} tag should indicate where the main
content should go.
Here’s a simple HTML template:
<html>
<head>
</head>
<body>
{{ content }}
</body>
</html>
And here’s an index.html file containing some very simple content:
<p>This is page content.</p>
Here’s the index.html file in the output-site/ directory:
<html>
<head>
</head>
<body>
<p>This is page content.</p>
</body>
</html>
The project was configured with --reports setting including the ws reporter:
reports: run ws
Sectioned Documents
If HTML content has been split into sections then you can access section
content via the content object, using either dictionary style access or
jinja’s dot syntax (if the section name forms a valid python attribute name).
Here is an example of a HTML file with sections delimited using idio filter
syntax:
<!-- @export "Introductory Section" -->
<p>This is the introductory section.</p>
<!-- @export "thanks" -->
<p>I'd like to thank Starfleet Academy.</p>
<!-- @export "footer" -->
<p>Some fine print to bury at the bottom.</p>
When this is put through the htmlsections filter the content will be split
into sections:
index.html|htmlsections
The htmlsections filter is actually one of the aliases for the idio filter.
It behaves a little differently than |idio: it splits content into sections
following the idio rules, it doesn’t apply syntax highlighting, and then it
sets output to True. It’s intended for use in just the scenario being
described here.
The template can access these sections using either content.foo or
content['foo']. The former is shorter, the latter works even when section
names have spaces in them:
<html>
<head>
</head>
<body>
<div class="introduction">
{{ content['Introductory Section'] }}
</div>
<div class="acknowledgements">
{{ content.thanks }}
</div>
<footer>
{{ content['footer'] }}
</footer>
</body>
</html>
Here’s the result:
None
You can also just call {{ content }} on a sectioned document and dexy
will insert all sections combined.
Navigation
The website template provides several elements to assist with creating website navigation.
page_title
The page_title element contains the title of the current page:
<title>{{ page_title }}</title>
Here’s the result for index.html:
<title>Welcome</title>
Current Page Location
The source element contains the canonical output filename of the current
page. The current_dir and parent_dir elements refer to the directory
containing the current page and its parent (if any), respectively:
<p>This document name is: {{ source }}</p>
<p>This document is in: '{{ current_dir }}'</p>
<p>This document's parent dir is: '{{ parent_dir }}'</p>
Here’s the result for index.html:
<p>This document name is: index.html</p>
<p>This document is in: ''</p>
<p>This document's parent dir is: ''</p>
Here’s the result for foo/index.html:
<p>This document name is: foo/index.html</p>
<p>This document is in: 'foo'</p>
<p>This document's parent dir is: ''</p>
Here’s the result for foo/bar/index.html:
<p>This document name is: foo/bar/index.html</p>
<p>This document is in: 'foo/bar'</p>
<p>This document's parent dir is: 'foo'</p>
Current Page’s Data Object
The s element contains the current page’s data object. (The same
information is contained in the d element, you can use either one.)
<p>This document's key is: {{ s.key }}</p>
<p>This document's file size is: {{ s.filesize() }}</p>
<p>'s' is an object of type: {{ s.__class__.__name__ }}</p>
Here’s the result for index.html:
<p>This document's key is: index.html</p>
<p>This document's file size is: 29</p>
<p>'s' is an object of type: Generic</p>
Here’s the result for foo/index.html:
<p>This document's key is: foo/index.html</p>
<p>This document's file size is: 0</p>
<p>'s' is an object of type: Generic</p>
Here’s the result for foo/bar/index.html:
<p>This document's key is: foo/bar/index.html</p>
<p>This document's file size is: 0</p>
<p>'s' is an object of type: Generic</p>
Navigation & Node Objects
The Website reporter creates a Navigation object which represents the tree of
directories in the project. Each directory is represented by a Node object.
Two node objects are made directly available within a website template, the
root object representing the root of the project, and the nav object
representing the node corresponding to the file being processed. The whole
navigation tree is availalbe via the navtree objects.
Here are some basic methods and attributes of nodes, using the nav object
representing the node for the document being processed:
<p>'nav' is an object of type: {{ nav.__class__.__name__ }}</p>
<p>ancestors are {{ nav.ancestors }}</p>
<p>breadcrumbs are {{ nav.breadcrumbs() }}</p>
<p>children are {{ nav.children }}</p>
<p>children with index pages? {{ nav.has_children_with_index_pages() }}</p>
<p>docs are {{ nav.docs }}</p>
<p>index page is {{ "%r" % nav.index_page }}</p>
<p>level is {{ nav.level }}</p>
<p>location is '{{ nav.location }}'</p>
<p>parent is {{ nav.parent }}</p>
The children attribute represents subdirectories. The docs attribute
represents all documents found in the node’s directory. The index_page
attribute corresponds to the index.html page, if there is one, in the node’s
directory. The level attribute represents the number of directories above the
node’s directory in the project.
Here’s the result for index.html:
<p>'nav' is an object of type: Node</p>
<p>ancestors are [Node(/)]</p>
<p>breadcrumbs are <a href="/">Welcome</a></p>
<p>children are [Node(/foo)]</p>
<p>children with index pages? True</p>
<p>docs are [Generic('index.html')]</p>
<p>index page is Generic('index.html')</p>
<p>level is 0</p>
<p>location is '/'</p>
<p>parent is None</p>
Here’s the result for foo/index.html:
<p>'nav' is an object of type: Node</p>
<p>ancestors are [Node(/), Node(/foo)]</p>
<p>breadcrumbs are <a href="/">Welcome</a> > <a href="/foo">Welcome</a></p>
<p>children are [Node(/foo/bar)]</p>
<p>children with index pages? True</p>
<p>docs are [Generic('foo/index.html')]</p>
<p>index page is Generic('foo/index.html')</p>
<p>level is 1</p>
<p>location is '/foo'</p>
<p>parent is Node(/)</p>
Here’s the result for foo/bar/index.html:
<p>'nav' is an object of type: Node</p>
<p>ancestors are [Node(/), Node(/foo), Node(/foo/bar)]</p>
<p>breadcrumbs are <a href="/">Welcome</a> > <a href="/foo">Welcome</a> > <a href="/foo/bar">Welcome</a></p>
<p>children are []</p>
<p>children with index pages? False</p>
<p>docs are [Generic('foo/bar/index.html')]</p>
<p>index page is Generic('foo/bar/index.html')</p>
<p>level is 2</p>
<p>location is '/foo/bar'</p>
<p>parent is Node(/foo)</p>
Here’s these attributes for the root node:
<p>'root' is an object of type: {{ root.__class__.__name__}}</p>
<p>ancestors are {{ root.ancestors }}</p>
<p>breadcrumbs are {{ root.breadcrumbs() }}</p>
<p>children are {{ root.children }}</p>
<p>children with index pages? {{ root.has_children_with_index_pages() }}</p>
<p>docs are {{ root.docs }}</p>
<p>index page is {{ "%r" % root.index_page }}</p>
<p>level is {{ root.level }}</p>
<p>location is '{{ root.location }}'</p>
<p>parent is {{ root.parent }}</p>
Here’s the result for index.html:
<p>'root' is an object of type: Node</p>
<p>ancestors are [Node(/)]</p>
<p>breadcrumbs are <a href="/">Welcome</a></p>
<p>children are [Node(/foo)]</p>
<p>children with index pages? True</p>
<p>docs are [Generic('index.html')]</p>
<p>index page is Generic('index.html')</p>
<p>level is 0</p>
<p>location is '/'</p>
<p>parent is None</p>
It’s the same for foo/bar/index.html:
<p>'root' is an object of type: Node</p>
<p>ancestors are [Node(/)]</p>
<p>breadcrumbs are <a href="/">Welcome</a></p>
<p>children are [Node(/foo)]</p>
<p>children with index pages? True</p>
<p>docs are [Generic('index.html')]</p>
<p>index page is Generic('index.html')</p>
<p>level is 0</p>
<p>location is '/'</p>
<p>parent is None</p>
A common usage is to iterate over children to produce a list of subdirectories.
You can create relative navigation on each page by using the nav object, or
navgation for the entire site using the root object and iterating recursively over
children.
There are some macros bundled with dexy which illustrate ways to use these elements to construct site navigation. You can use them as-is or copy them and modify them for your own needs.
The Navigation object is primarily used to generate the tree, but it can also
be accessed via the navtree element. Its nodes attribute is a dictionary of
all nodes accessed by path. The root attribute is the root node, which you
can already access via root. There is a debug method which prints out all
nodes and their attributes which you can include in a document.
<p>'navtree' is an object of type: {{ navtree.__class__.__name__ }}</p>
Nodes: {{ pprint(navtree.nodes) }}
Root: {{ pprint(navtree.root) }}
Debug Method:
{{ navtree.debug() }}
Here’s the result for index.html:
<p>'navtree' is an object of type: Navigation</p>
Nodes: {'/': Node(/), '/foo': Node(/foo), '/foo/bar': Node(/foo/bar)}
Root: Node(/)
Debug Method:
node: /
index-page:
index.html
docs:
index.html
children:
/foo
node: /foo
index-page:
foo/index.html
docs:
foo/index.html
children:
/foo/bar
node: /foo/bar
index-page:
foo/bar/index.html
docs:
foo/bar/index.html
Links to Pages and Sections
The link() and section() methods allow you to create a HTML <a> link to
a dexy page based on the page’s title or key and a section’s name.
The link() method links to a specified page, and optionally to a section on
that page. The section() method lets you link to a specified section without
needing to know which page it’s on.
Here’s an example of using these methods:
{{ link("Home") }}
{{ link("All About Foo") }}
{{ link("All About Foo", section_name = "Welcome") }}
{{ section("Welcome") }}
{{ section("Foo") }}
And here are the results:
<a href="index.html">Home</a>
<a href="foo/index.html">All About Foo</a>
<a href="foo/index.html#welcome">Welcome</a>
<a href="foo/index.html#welcome">Welcome</a>
<a href="foo/index.html#foo">Foo</a>
The page entitled Home has its title specified in dexy.yaml:
- index.html:
- title: Home
- ws-template: _home.html
The page entitled All About Foo, on which we link to the Foo and Welcome
sections uses two filters wihch are useful to know about.
Yamlargs
The yamlargs filter lets you add some YAML metadata to the top of a page. The
YAML is stripped off and the metadata is added to the page’s settings.
title: All About Foo
---
<h1>Welcome</h1>
<p>This page is all about foo.</p>
<h2>Foo</h2>
<p>This is the foo section.</p>
Soups
The soups filter uses BeautifulSoup 4’s HTML parser to look for any <h1> or
<hn> tags and creates a section for each one it finds, and also creates an id
attribute for each header element so they can be linked to.
<html>
<head>
<title>All About Foo</title>
</head>
<body>
<h1 id="welcome">Welcome</h1>
<p>This page is all about foo.</p>
<h2 id="foo">Foo</h2>
<p>This is the foo section.</p>
</body>
</html>
Listing Pages and Sections
You can use the dexy links command to print out a list of all valid keys
which are possible to use as the target of the link() and section()
commands:
$ dexy links
Nodes:
'All About Foo'
Sectioned('foo/index.html|yamlargs|soups')
'Home'
Generic('index.html')
'foo/index.html'
Sectioned('foo/index.html|yamlargs|soups')
'foo/index.html|yamlargs|soups'
Sectioned('foo/index.html|yamlargs|soups')
'index.html'
Generic('index.html')
Sections:
'Actual Document Contents'
Sectioned('foo/index.html|yamlargs|soups')
'Foo'
Sectioned('foo/index.html|yamlargs|soups')
'Welcome'
Sectioned('foo/index.html|yamlargs|soups')
Dexy will complain if you use a non-unique title or section name.
Other Plugins
In addiiton to the specific template elements covered above, the Website
reporter makes use of many of dexy’s Template Plugins. You can configure this
via the plugins setting.
Template Element Reference
The dexy info command provides information about indvidual documents in a
dexy run, and this command takes a ws argument which adds customized
documentation about the website template elements available, and their values,
for this individual document.
Here’s an example. It includes website related tags, elements defined via template plugins, the contents of the Navigation tree, and methods available on nodes in the tree:
$ dexy info -expr hello.txt -ws
search expr: hello.txt
Info for Document 'hello.txt|jinja'
document output data type: generic
settings:
canonical-name: hello.txt
canonical-output: True
output-name: None
shortcut: None
storage-type: generic
title: None
attributes:
ext: .txt
key: hello.txt|jinja
name: hello.txt
methods:
basename(): hello.txt
baserootname(): hello
filesize(): 7
long_name(): hello.txt-jinja.txt
parent_dir():
title(): Hello
web_safe_document_key(): hello.txt-jinja.txt
website reporter methods:
Website Template Environment Variables:
Navigation and Content Related:
current_dir: The directory containing the document being processed.
d: 'generic' type data object representing the current doc.
link: Function to create link to other page.
nav: The node for the current document's directory.
navtree: The complete navigation tree for the website.
page_title: Title of the current page.
parent_dir: The directory one level up from the document being processed.
root: The root node of the navigation tree.
s: 'generic' type data object representing the current doc.
section: Function to create link to section on any page.
source: Output name of current doc.
title: Title of the current page.
wrapper: The current wrapper object.
navobj Node attributes (using root node):
ancestors: [Node(/)]
children: []
docs: [Generic('hello.txt|jinja')]
index_page: None
level: 0
location: '/'
parent: None
navobj Node methods (using root node):
breadcrumbs()
Navigation breadcrumbs showing each parent directory.
has_children_with_index_pages()
Boolean value.
False
Variables From Plugins:
abs: The python builtin function abs
all: The python builtin function all
any: The python builtin function any
attrgetter: The attrgetter method from Python's operator module.
basestring: The python builtin function basestring
bin: The python builtin function bin
bool: The python builtin function bool
bytearray: The python builtin function bytearray
cal: Shortcut for `calendar`.
caldates: List of calendar dates in current month.
calendar: A Calendar instance from Python calendar module.
callable: The python builtin function callable
camelize: The camelize method from Python inflection module.
chr: The python builtin function chr
cmp: The python builtin function cmp
complex: The python builtin function complex
dasherize: The dasherize method from Python inflection module.
datetime: The Python datetime module.
debug: A debugging method - prints content to command line stdout.
dict: The python builtin function dict
dir: The python builtin function dir
divmod: The python builtin function divmod
enumerate: The python builtin function enumerate
filter: The python builtin function filter
float: The python builtin function float
format: The python builtin function format
hasattr: The python builtin function hasattr
hex: The python builtin function hex
humanize: The humanize method from Python inflection module.
id: The python builtin function id
int: The python builtin function int
isinstance: The python builtin function isinstance
issubclass: The python builtin function issubclass
itemgetter: The itemgetter method from Python's operator module.
iter: The python builtin function iter
len: The python builtin function len
list: The python builtin function list
locals: The python builtin function locals
long: The python builtin function long
map: The python builtin function map
markdown: Function which converts markdown to HTML.
max: The python builtin function max
md: Function which converts markdown to HTML.
min: The python builtin function min
month: Current month.
oct: The python builtin function oct
ord: The python builtin function ord
ordinal: The ordinal method from Python inflection module.
ordinalize: The ordinalize method from Python inflection module.
parameterize: The parameterize method from Python inflection module.
pformat: Pretty prints Python objects.
pluralize: The pluralize method from Python inflection module.
pow: The python builtin function pow
pprint: Pretty prints Python objects.
pygments: Dictionary of pygments stylesheets.
raise: Alias for `throw`.
range: The python builtin function range
reduce: The python builtin function reduce
repr: The python builtin function repr
reversed: The python builtin function reversed
round: The python builtin function round
set: The python builtin function set
singularize: The singularize method from Python inflection module.
slice: The python builtin function slice
sorted: The python builtin function sorted
str: The python builtin function str
sum: The python builtin function sum
throw: A debugging utility which raises exception with argument.
titleize: The titleize method from Python inflection module.
today: Result of datetime.today().
transliterate: The transliterate method from Python inflection module.
tuple: The python builtin function tuple
type: The python builtin function type
underscore: The underscore method from Python inflection module.
unicode: The python builtin function unicode
xrange: The python builtin function xrange
year: Current year.
zip: The python builtin function zip
navobj Nodes:
node: /
no index page.
docs:
hello.txt|jinja
active template plugins are:
debug, inflection, builtins, operator, datetime, pprint, pygments, markdown
For more information about methods available on this data type run
`dexy datas -alias generic`
Run Reporter
The Run reporter generates an HTML file with information about the dexy run. It
is enabled by default, you can find its output in .dexy/reports/run/index.html.
Colophon
This documentation was generated by Dexy and the Asciidoctor implementation of Asciidoc.
The source code lives on Github. If you have feedback or suggestions about this document please fell free to email info@dexy.it or open a github issue.
Here is the dexy.yaml file for this document:
user-guide.adoc|jinja|asciidoctor|customize:
# document settings
- output-name: index.html
# filter settings
- customize: { scripts: ['jquery.js', 'view-result.js'] }
- jinja: { assertion-passed-indicator: "<div style=\"background-color: green; width: 20px; height: 20px;\">OK</div>" }
# inputs
- assets
- dexy-source
- tests
- examples
- github-issues
assets:
- .js
github-issues:
- github.py|py:
- py: {
add-new-files: True,
additional-doc-settings: { output: False }
}
dexy-source:
- modules.txt|pydoc:
- contents: "dexy"
tests:
- tests/*.py|pydoc:
- tests/utils.py:
- output: False
examples:
- examples/*.py|idio|pycon|asciisyn
- examples/*.sh|idio|shint|asciisyn:
- except: "run"
- shint: {
add-new-files: True,
exclude-new-files-from-dir: [".dexy"],
keep-originals: True,
additional-doc-filters: idio|t,
additional-doc-settings: [['.html', { 'data-type' : 'bs4' }]]
}
- examples/run.sh|idio|shint|asciisyn:
- examples/run-example.yaml
- .conf|asciisyn:
- asciisyn: { lexer: yaml }
- .yaml|asciisyn
- .yaml|idio|asciisyn:
- idio: { ext : '.txt' }
- .yaml|idio:
- idio: { ext : '.txt' }