Remember, this is the simple module that we want to analyze:
defmodule Example do
def hello(name) do
"Hello, #{name}!"
end
endIt contains a simple function which returns a string created with the string interpolation syntax and the variable provided.
So how do we go about analyzing this? Let's think about what patterns we want to find in the ideal solution, and what patterns we don't want to find in the ideal solution.
Remember, analysis is not a replacement for the test unit, and testing the presence and output of public functions is better and more clearly tested using appropriate testing.
Some discrete things that we want to find:
- a parameter named
name - the use of string interpolation
Some discrete things that we don't want to find:
- binary concatenation
So now let's revisit the analyzer extension from step 2, and write an analyzer test to check if the name of the parameter is name. Our analyzer extension module so far:
defmodule ElixirAnalyzer.TestSuite.Example do
use ElixirAnalyzer.ExerciseTest
endWe are going to add a call to the the macro feature/1 which we gained use of by including use ElixirAnalyzer.ExerciseTest on line 2.
defmodule ElixirAnalyzer.TestSuite.Example do
use ElixirAnalyzer.ExerciseTest
+ feature "has a parameter called 'name'" do
+ end
endNow we have the to add in some details that describe the feature and what it should do if it fails to match our description.
The form block is how we describe what we want elixir to pattern match on. Pattern matching is an important concept which we are going to expand on. Similar to pattern matching on a nested list/tuple structure, we are going to pattern match on the AST (abstract syntax tree) generated by the elixir compiler. Let's look at the AST of the function in our module:
iex> ast = quote do
...> def hello(name) do
...> "Hello, #{name}!"
...> end
...> end
{:def, [context: Elixir, import: Kernel],
[
{:hello, [context: Elixir], [{:name, [], Elixir}]},
[
do: {:<<>>, [],
[
"Hello, ",
{:"::", [],
[
{{:., [], [Kernel, :to_string]}, [], [{:name, [], Elixir}]},
{:binary, [], Elixir}
]},
"!"
]}
]
]}This is the AST of our function that we want to pattern match on. We can experiment with this and see that we can get a match on subsets of this by using _ as a placeholder:
# Matches
{:def, _, _} = ast
{:def, _,[_|_]} = ast
{:def, _, [{:hello, _, _}, _]} = astAnd we can match on a subset of this tree, if we traverse this structure using an algorithm like Macro.prewalk/3 to find matches. But pattern matching manually would be a chore to do by hand, and likely prone to error, so let's make use of the functions in ElixirAnalyzer.ExerciseTest and let our DSL (domain specific language) define this for us. So what we need to worry about is finding a piece of syntax that will generate an AST snippet that we can match on:
# `form` is a special word in this DSL to define a pattern to look for
form do
def hello(name) do
...
end
endSo we know we want a function named hello and with a parameter named name to be matched, but we haven't looked at how to ignore all the other stuff that make it complicated. In this DSL, we use the keyword _ignore inside of the form block which is transformed into _ when our pattern is compiled:
form do
def hello(name) do
+ _ignore
end
endNow we have the pattern to match, let's insert it into our extension:
defmodule ElixirAnalyzer.TestSuite.Example do
use ElixirAnalyzer.ExerciseTest
feature "has a parameter called 'name'" do
+ form do
+ def hello(name) do
+ _ignore
+ end
+ end
end
end_ignore will match any node in the AST. In some special cases, that might be too permissive. You can use _shallow_ignore to match any node (name and metadata), but continue matching its children.
The _block_includes feature can be used inside of form to match a number of lines inside of a block of code. The block of code may contain other lines before, after or between the lines you are trying to match. The order of the lines needs to be the same.
For example, the following feature
feature "opens and closes the resource" do
form do
_block_includes do
_ignore = open(resource)
close(resource)
end
end
endwill match the following function
def write(resource, content) do
pipe = open(resource)
dump_content(pipe, content)
|> flush()
close(resource)
:ok
endNote that _block_includes consumes all the lines of a block of code, so it cannot be used right before or after another match.
# do not use _block_includes in this way
form do
_block_includes do
_ignore = open(resource)
close(resource)
end
# the following line can never match because _block_includes has consumed the full block
:ok
endThe _block_ends_with feature is very similar to _block_includes, the only difference is that the last line of _block_ends_with must match the last line of the code being matched.
For example, the following feature
feature "opens and closes the resource" do
form do
_block_ends_with do
_ignore = open(resource)
close(resource)
end
end
endwill match the following function
def write(resource, content) do
pipe = open(resource)
dump_content(pipe, content)
|> flush()
close(resource)
endbut would not match the _block_includes example above ending with :ok.
_block_ends_with also consumes all the lines of a block of code, so it cannot be used right before or after another match, just like _block_includes.
Additionally, when trying to match a single line, if _block_includes is not placed within a context (module, function...) the line will match even if it's not the last in the code.
For example
# do not use _block_ends_with in this way
form do
# No context given, _block_ends_with will attempt to match line by line
_block_ends_with do
close(resource)
end
endwill match
def write(resource, content) do
pipe = open(resource)
dump_content(pipe, content)
|> flush()
close(resource)
:ok
endInstead, use
form do
def write(_ignore, _ignore) do
_block_ends_with do
close(resource)
end
end
endSlightly different syntax can produce exactly the same AST. That means that certain details cannot be distinguished by this analyzer. If they cannot be distinguished, they cannot be verified by the analyzer, but they also don't need to be both listed when specifying all of the forms for a feature.
For example:
-
Single line vs multiline string
a = quote do """ 1 2 """ end b = quote do "1\n2\n" end a == b # => true
-
Single line vs multiline
doblocksa = quote do def foo do "hello" end end b = quote do def foo, do: "hello" end a == b # => true
-
Usage of parenthesis in certain contexts
a = quote do def foo("hello") end b = quote do def foo "hello" end a == b # => true
So far we have defined a feature to test, and a pattern to match, but we need to further specify how we want it to match and what it should do when it doesn't. We need to add the find, on_fail, and comment attributes:
find tells our analyzer how we want it to match the patterns we define for the test (we will soon look at defining multiple patterns for a single test). It has a few different accepted arguments:
:all- the analyzer must match all of the patterns defined for the test to pass:any- the analyzer must match any of the patterns defined for the test to pass:one- the analyzer must only match one of the patterns defined for the test to pass:none- the analyzer must not match any of the patterns defined for the test to pass
Because we only have one pattern, :any, :all, or :one would all produce the same output, so let's just choose :all
defmodule ElixirAnalyzer.TestSuite.Example do
use ElixirAnalyzer.ExerciseTest
feature "has a parameter called 'name'" do
+ find :all
form do
def hello(name) do
_ignore
end
end
end
endtype tells our analyzer the purpose of the test. There are 4 types:
:celebratory- if the test passes, append a comment to celebrate the student's achievement:actionable- if the test fails, append a comment with an actionable improvement to the solution:informative- if the test fails, append a comment with a point of learning:essential- if the test fails, append a comment which soft-blocks the student on the website
For our test, let's soft-block the solution if the pattern doesn't match the submission:
defmodule ElixirAnalyzer.TestSuite.Example do
use ElixirAnalyzer.ExerciseTest
feature "has a parameter called 'name'" do
find :all
+ type :essential
form do
def hello(name) do
_ignore
end
end
end
endAnalysis of the solution is coordinated by a central service in exercism's infrastructure. Delivery of the report to the student is through the website. The website copy would be cumbersome to add into this test framework, so a string to locate the copy in another repo is used. It takes the form: elixir.__exercise_slug__.__comment_to_display__ in snake case format.
At this moment, the comments are found in the exercism/website-copy repo.
so lets add that to our analyzer as well:
defmodule ElixirAnalyzer.TestSuite.Example do
use ElixirAnalyzer.ExerciseTest
feature "has a parameter called 'name'" do
find :all
type :essential
+ comment "elixir.example.use_name_parameter"
form do
def hello(name) do
_ignore
end
end
end
endSo this next part will likely take a bit of testing to isolate the exact pattern you wish to isolate, but the workflow is generally:
- make changes to the analyzer function
- rebuild the escript
- run the analyzer on the file via CLI
- determine if the analysis.json has to appropriate output
- (repeat)
If you have made it this far, great! You are doing awesome. Once you have a good idea about what kind of features you want the analyzer extension to look for, it's time for step 4 - write unit tests for your new extension.