How about support for verbs/commands?

[alexanderfast]

Hi.

I like this library a lot! But one thing I'm missing is support for "verbs". Like when the first argument on command line is a verb which then has options specific to that verb. Similar to git commit -m "message".

I put the code below together as a proof of concept. Though, looking at it now I think I would have preferred to have the Option attribute assigned to parameters of the method instead of a dedicated class, but that would require some more intrusive changes. Is this something you would be interested in exploring or is totally out of scope for this library?

using System;
using System.Linq;
using System.Reflection;

using CommandLine;

namespace SampleApp
{
    public class VerbProgram
    {
        private static void Main(string[] args)
        {
            //MyVerbs.FooOptions options = new MyVerbs.FooOptions();
            //if (!CommandLineParser.Default.ParseArguments(
            //    new[] { "--flag" }, options))
            //{
            //    Environment.Exit(1);
            //}

            MyVerbs verbs = new MyVerbs();
            if (!VerbParser.Default.ParseArguments(
                new[] { "foo", "--flag" }, verbs))
            {
                Environment.Exit(1);
            }

            Environment.Exit(0);
        }
    }

    internal class MyVerbs
    {
        internal class FooOptions : CommandLineOptionsBase
        {
            [Option("f", "flag", Required = true)]
            public bool Flag { get; set; }
        }

        [Verb]
        public void Foo(FooOptions options)
        {
            Console.WriteLine(options.Flag);
        }
    }

    public class VerbAttribute : Attribute
    {
        /// <summary>
        /// By default the name of the verb is a lowercase version of the
        /// method name, set this property to override.
        /// </summary>
        public string Name { get; set; }

        public string HelpText { get; set; }
    }

    internal class VerbParser
    {
        public static VerbParser Default = new VerbParser();
        internal ICommandLineParser ArgumentParser = CommandLineParser.Default;

        public bool ParseArguments(string[] args, object verbs)
        {
            // store name of verb and remove it from args
            var verbName = args[0];
            args = args.Skip(1).ToArray();

            // find verb method
            Pair<MethodInfo, VerbAttribute> verb = null;
            foreach (Pair<MethodInfo, VerbAttribute> pair in
                ReflectionUtil.RetrieveMethods<VerbAttribute>(verbs))
            {
                string name = pair.Right.Name;
                if (string.IsNullOrEmpty(name))
                {
                    name = pair.Left.Name.ToLower();
                }
                if (verbName.Equals(name))
                {
                    verb = pair;
                    break;
                }
            }
            if (verb == null)
            {
                return false; // verb not found
            }

            object options = Activator.CreateInstance(
                verb.Left.GetParameters()[0].ParameterType);
            if (!ArgumentParser.ParseArguments(args, options))
            {
                return false; // couldnt parse verb arguments
            }

            verb.Left.Invoke(verbs, new[] {options});
            return true;
        }
    }
}

[alexanderfast]

Oh, just realized that the new ReflectionUtil method might not be obvious, but it simply returns all methods with the attribute rather than just the first one:

public static IEnumerable<Pair<MethodInfo, TAttribute>> RetrieveMethods<TAttribute>(object target)
        where TAttribute : Attribute
{
    var info = target.GetType().GetMethods();

    foreach (MethodInfo method in info)
    {
        if (!method.IsStatic)
        {
            Attribute attribute =
                Attribute.GetCustomAttribute(method, typeof(TAttribute), false);
            if (attribute != null)
                yield return new Pair<MethodInfo, TAttribute>(method, (TAttribute)attribute);
        }
    }
}

[alexanderfast]

Pushed this to a verb branch in my fork, if you prefer to look at the code that way.

[gsscoder]

Hi all,
thanks everyone for using and liking this library!!! Thanks for all valuable suggestions and great pull request. Really!
I'll try to examine every post and code asap...

The inspiration that made ​​me write this library is to see more and more utility (better open source) that runs in the shell: so I encourage all library users to open their preferred IDE or text editor and program with fun! (Another thing: if someone likes it, please tell me your open source software that uses this library; I'll list it in github home README.md and CodePlex project home).

Alexander (mizipzor), your suggestion for extend the API is very similar to what I had in mind. This is very good because two persons that feels the same way are better then one. If anyone like to suggest a different way please write here his point of view.

So when the library will support verb commands? Very soon... I'm about to promote 1.9.3.* branch to stable. The first 1.9.4.* branch will contain this feature.

The interesting thing is that verb commands will lead the creation of a REPL (http://goo.gl/aojts) module that will allow an easier use of this library outside console applications domain. E.g. a WinForms (or other layout engine) app could make available to the user a command window (likes Visual Studio Package Manager Console) for input textual commands during execution. Isn't exciting? For me it is, but let me know your opinions...

Since we're on the subject of news... I'd like to let a subset of the library (I really don't know now nothing more than the basic idea) to be able to compile for .NET Micro Framework and .NET Compact Framework. I'll see...

Thanks to all again!!!
Regards,
Giacomo

[alexanderfast]

Hi,

I'm happy to hear that you like the idea of verbs. 👍

REPL was a new word for me (though I knew the concept) but I didn't know it was so commonly recognized. Truly exciting!

I actually use your library for some projects here at work. Sadly, I'm not allow to share the code so there are no links you can add. The latest project was the one requiring the verb support. Since I created this issue I've added quite a bit to the concept and have been switching back an forth between using the NuGet package and a custom compiled version. Currently, the verb support is in that specific project bundled with the application logic. It's not pretty since I couldn't access any of the library internals so it's more built around the library rather than into. But I'll see if I can update my branch with some of the new code.

But one of my hobby projects is using the NuGet packace, you can link to that if you want. Though its very basic right now since I'm waiting for a pull request to be integrated into another library I use.

Another thing, I know you like the idea of being able to use the library by just copying one file to the project. But the code is growing and navigating that file is becoming cumbersome. Is splitting that file into a one-class-per-file structure completely out of the question? If you ask me, NuGet is a by par better dependency manager. And, really, copying and including ten files isn't all that harder than one. A restructure is officially on my wishlist.

Regards,
Alex

[gsscoder]

About talking of projects that use the library, I mean only open source one! I'll list your teamcity-artifact-downloader as soon.

These day I was reasoning over the question of splitting source in more files. For development is better use a splitted source while for deployment I prefer one or two files.

For these reason I'm about to design a command line utility that split / join these kind of source...

In this way if some one one to study the source of this (or similar) projects, he can split the source in more files. Modify a single source and join back the group into a single file.

I think it's not a bad idea... (but this is my personal opinion).

Regards, Giacomo

[gsscoder]

Another solution for managing the source without splitting: you can use the class view of your IDE (MonoDevelop supports it).
Visual Studio 2012 has a wonderful class view mixed within solution explorer...

Regards, Giacomo

[nemec]

[Off Topic]
I'm going to have to put in a vote for @mizipzor's request to split up the project into multiple source files. Now that the project has a NuGet package, no one should be adding the .cs file manually to a project. If anything, the .nupkg file is a single file and just as easy to include in a project as a binary while leaving the source easy to navigate in multiple files.
[/Off Topic]

I've usually seen this feature referred to as a subparser. Since each subparser often has wildly different argument choices, why not code it so that each subparser is a separate subclass of CommandLineOptionsBase?

No one has really defined the commandline syntax for a parser with suboptions yet, so here's an initial draft:

MyExample.exe --mainparam1 value -m otherValue suboptionsName --arg=something --other=val

• Arguments specified before the suboptions name are applied toward the main parser only.
• Arguments specified after the suboptions name are applied toward the suboptions only.
• SubOption properties must either be already initialized or have a public, parameterless constructor (so that the parser can create a new instance if the SubOption is specified). I'd like to explore adding a where T : new() generic constraint somewhere in here to avoid reflection, but it doesn't look like CommandLine uses generics.
• SubOptions are identified by an additional attribute along with the subcommand name used to split the argument list.
• No nested SubOptions... let's just not get into that right now.

This simplifies parsing a lot, since you can split the argument list by each subparser name to get the list of arguments for that subparser (eg. [ MyExample.exe --mainparam1 value -m otherValue] [ suboptionsName --arg=something --other=val])

I'm working on a sample implementation that works something like this:

class SimpleSubOptions : OptionsBase
{
    [Option("i", "int")]
    public int IntegerValue { get; set; }
}

class SimpleOptionsWithSubOptions : OptionsBase
{
    [Option("s", "string")]
    public string StringValue { get; set; }

    [SubOption("suboption")]
    public SimpleSubOptions SubOptions { get; set; }
}

>>> MyExample.exe -s something suboption --int 4

Thoughts?

[nemec]

Check out the new tests in my pull request for examples of the final implementation. Feel free to do what you like with the pull request.

[alexanderfast]

Since each subparser often has wildly different argument choices, why not code it so that each subparser is a separate subclass of CommandLineOptionsBase?

I agree completely, in fact that's what I intended with my initial attempt pasted in my first comment. Check the Foo function in the MyVerbs class above. Though I went with the approach of tying a subparser to a single method taking a single argument, a CommandLineOptionsBase implementation.

I did that because, as you said, "each subparser has wildly different argument choices" and I'd argue they have wildly different implementations as well. With your approach the program main function would have many if's checking which suboption got arguments and call a method with those options (which name is probably similar to the suboption). Any thoughts on how to help users of this library structure their code in regards to this?

Many thanks for your efforts to realize this! :)

[alexanderfast]

(I'm not sure if this should be under your pull request, but to have everything in the same place I put it here.)

Another thing, in your approach its valid to specify every suboption. While this is useful in some cases I would like see a way to generate an error message if more than one is specified. This is currently valid:

namespace SampleApp
{
    sealed class Program
    {
        private class MySubOption : CommandLineOptionsBase
        {
            [Option("v", "value")]
            public string Value { get; set; }
        }

        private class MyOptions : CommandLineOptionsBase
        {
            [SubOption("commit")]
            public MySubOption Commit { get; set; }

            [SubOption("checkout")]
            public MySubOption CheckOut { get; set; }
        }

        private static void Main(string[] args)
        {
            args = "commit --value foo checkout --value bar".Split();

            var myOptions = new MyOptions();
            var myParser = new CommandLineParser();
            if (!myParser.ParseArguments(args, myOptions))
                Environment.Exit(1);
        }
    }
}

[nemec]

@mizipzor Good point. Developers will probably want multiple "main" methods, one for each subparser since they'll only be using one at a time, though if we allow parameters in between the main parser and subparser, the subparser's "main" method probably won't know how to handle it. I'd like to keep the "suboptions are objects" idea since commit -m and checkout -m could have completely different meanings with different Types. I'll give some thought to this...

The point about specifying every suboption is by design... mostly. There's even a unit test for it haha. Later last night I was thinking that instead of being able to specify multiple subparsers, we keep it recursive. For example, say there's a git subcommand plugin that lets you commit git repos to svn: git svn checkin would call the svn subparser which, in turn, would call svn's checkin subparser.

[alexanderfast]

If the user specifies more than one subparser the order of the subparsers execution probably matters, and the user probably expects them to be executed in the order specified. The program main function becomes non-trivial. Either way I would like to see some framework for this in the library.

You don't happen to know of a program that allows the use of multiple subparsers at once to use as a reference? The git svn checkin seems to be more about svn wrapping or mimicking checkin rather than multiple subparsers where order matters.

[nemec]

After doing some research, what actually happens is that git checks the GIT_BIN folder for a file named git-ARG1 and executes it, if found, with the rest of the arguments -- not exactly what I was envisioning.

I think I wasn't quite clear before, but I'm talking about allowing nested parsers. For example, the git executable contains an svn subparser that itself contains a checkin subparser. All of the arguments after the checkin directive are passed to the checkin subparser and may be accessed by something like Options.Svn.Checkin.Verbose and determining whether or not the checkin argument was provided is as simple as if(Options.Svn.Checkin != null).

If you want a specific example, git-remote has subcommands like git remote add and git remote rm. My thought is that the add and rm subparsers end up as children of the remote subparser, not the git parser. Theoretically, this could allow you to have a separate remote.exe binary with the same functionality that can reuse the existing subparser classes with no extra work.

[gsscoder]

Hi all,
I'm coming late to discussion...

In my opinion we can also suppose that a developer wants choose by itself its app design of plugin architecture employed. In favor of this there the unknown arguments feature introduced with Kevin Moore pull request (see #4 (comment) for info).

Another point could be the adoption of existing projects to employ the library. Enforcing architecture by specific design will enforce standardization too, but can also raise learning curve and changes required to the code (also if could be more elegant and clean / and probably it is).

Just one final thing about splitting code in one-class-per-file. As you will see in the next commit, I've put some effort in re-organizing CommandLine.cs and CommandLineText.cs. Collaborative work always lead to great things, but there's a reason also in this. When all types have converged in two files, I had the idea to offer developers a way to ship a self-contained exe alone without external dependencies. Also the grouping make sense: in fact you can include CommandLine.cs without CommandLineText.cs (this makes the last as an extension for formatting help, an important part of the library but not something that you MUST have in order to compile the first file - the inverse is obviously true).

Verb commands are great thing and will be in the next official beta. At 99% new types will be placed in a third file and also this will be treated a "module" or if you prefer as a C include that depends on another (main module CommandLine.cs). CommandLineParser class will be made partial and if you include CommandLineVerb.cs verbs feature will come to life.
This is just another point for left CommandLine.cs can intact: it will not grow a lot over actual size and now it's still not-so-long that made maintenance difficult.

Anyway I was always open to debate and more over forks do exist also for experimenting new ideas and new views after all!

Good job anyone!!!

Regards, Giacomo

[gsscoder]

Hi again,
I'd just want also point out about naming. I think that everyone here agree that like coding style, formatting etc also naming cares, right?

OK, in my opinion [SubOption] could lead to some doubt. The "Sub" in the noun suggests that there's something nested with an option. A concept with a child-relationship to an option... In contrast it describes a complete set of ordinary options and hence here there's the concept of grouping.

Am I wrong? I rather call it [VerbOption] or something like this.

For be completely correct I have to say that the thing can be viewed that normal options be come a "group of option" of "macro group" and hence the "SubOption" term became more correct. I would name the class in this way (but this will not be a constraint, I think), e.g.:

class AddSubOptions
{
// ... ordinary options
}

class Options
{
[VerbOption("add")]
public AddSubOptions AddVerb { get; set; }

[VerbOption("commit")]
public CommitSubOptions CommitVerb { get; set; }
}

Don't get me wrong, Nemec, I really appreciate all your suggestions and I'm not arguing; I just want to find the best name for the new attribute just for get the API as more clear as possible!

Opinions?

Regards, Giacomo

[nemec]

Giacomo,

I think you're trying to do a little bit of premature optimization here. If an application is going to go as far as including a third party command line parser to parse a few command line args, the ~100kb saved by giving users the option to "opt out" of certain unused functionality by choosing not to include a .cs file will not make any difference in performance. Unless CommandLineText.cs includes non-system DLLs, I think giving users the option to pick and choose which files to include will just make things more complicated.

One of the primary benefits of .Net is that the compiled binary (or dll) is platform-independent, so what benefit does a user get by manually including the .cs files (if they're not going to modify them)?

I'll put forth two final arguments toward splitting up into multiple class files, but obviously it's up to you:

1. You cannot open the same file multiple times in Visual Studio. I like to keep multiple classes open side-by-side when developing, but when everything is in the same file I lose all context. Doing a "go to definition" means I lose where I came from when I navigate to a new definition. This makes it really, really hard to debug issues and step through code.
2. Merge conflicts. Say I add a new class to the project, if it's in its own file there is absolutely no chance of it introducing a merge conflict. When all the code is in the same file, there's a much greater chance that two simultaneous changes to unrelated classes will create a merge conflict.

As for naming conventions, I have absolutely no issue. You're absolutely right that it suggests a "nested" set of options and the way I see it that's exactly what it is. I designed my implementation explicitly so that one set of options can be the "parent" of one (or more) sets of other options. This is, of course, just my first draft on how suboptions should work (and a sample implementation to prove that "it works"). There's a lot more work to be done based on the responses from other devs here in this thread. If this isn't the direction you were looking to go with the "verb" feature, I don't mind just throwing away my pull request.

Just my two cents :)

[gsscoder]

Hi,
thanks for comments.

I'm referring to all yours opinion in providing verbs implementation. Discussion about naming is purely philosophical.

I'm trying to follow all your (and other contributors) suggestion and I'm aware of problems of left a partial class open.

I've also evaluating the extension-method way.

For now I've choose and "old old way" to proceed (that is used also in a very popular piece of C# code thought for inclusion like TinyIoC - among others) -> one clear simple preprocessor directive (COMMANDLINE_EXTENSIONS) that opens/close with 'partial' keyword ICommandLineParser/CommandLineParser (the verbs partial will be placed in another file).

I know that it could lead to disagreements... But after all, as you said, there's the NuGet deployment where's everything will be deployed and also a more "advanced" developer that opt for source inclusion will care of this old (and maybe ugly, I admit) detail!

Moreover I ever can opt with a trade-off that I'm experimenting in a (for now) toy library that I recently published here (https://github.com/gsscoder/exprengine). If you look at the code you can see 3 projects. The main is a simple lib for math expression evaluation. There's obviously a test project and the third is a REPL command line app that call includes all file of the main lib (it not references it).
The point is also if the whole project back to be splitted in X-files there's always room for inclusion.

I don't want to appear fanatic for this possibility (and I'm probably getting this), but at certain point of evolution this was one of the main feature.

It's not about optimization of resulting exe (this is not the point!). The point is to allow the developer to ship the resulting app without distributing the library dll file. The exe will be obviously be dependent upon .NET infrastructure but this part of its choice to use the library...

If you're designing a such complex system like git this is not obviously a problem: the system will be will structured in directories with config files etc etc; but for simple utilities that don't need external reference I want to let developers make this option (source inclusion) simple and friendly.

Anyway thanks for all suggestions/comments. I will off course refer to your implementation for complete mine.

Regards, Giacomo

[nemec]

Ah, I understand now. Don't worry, you're not appearing fanatic at all :)

[gsscoder]

Hi Nemec,
thank you! And thank you for all the (extremely good) effort made to discussion / project.

I've just few moments ago cloned your subparser branch. It's very cool piece of code! I love that you mixed verbs with ordinary options (putting computation in original DoParseArguments).

I was about to provide a particular ParseArguments but your implementation is really more clean.

Except for naming a will move for VerbOption -> your implementation will be the reference of that feature and the credit will be put in evidence.

Thank you! I'll going to integrate it in the work in progress...

Regards,
Giacomo

[gsscoder]

... another thing ...

Please be patient but ever if there's unit tests I ever think twice about modifying something related to core code.

Please give me your valuable opinion, Nemec (others invited too).

In your branch your treating suboptions/verb like ordinary option: DoParseArguments discriminates it with OptionMap and also because it can't create a specialized parser for an option without dashes (one or two doesn't matter, it can't).

I was reasoning about name conflicts.

If the master class that contains verb holds an option defined "exactly" in the same way of the suboption type? Who win? Should I place it before the verb? Suppose I have program to manipulate file with a syntax like this:

fileman rm filename

fileman mv filesrc filedst

Suppose that suboptions "rm" and suboption "mv" accepts a -v switch (verbose -> for print more messages).

I can define it in a super class -> no problem.

I can define it in both class -> no problem.

I can the define it only in the master class -> no problem, again.

But what if developer define it in both suboption (or only one) and also in the master option class? Maybe it really don't know what it wants neither from the parser neither from its users... Anyway this situation could lead to some confusion. Am I wrong? What do you think? Are you treating this case?

Another thing that could create problem is nesting verbs, but I thing that it simply not possible (and not make sense).

To be clear I'm also don't know if when I define verbs I need normal options in the master class or to be clear, maybe I that verbs should be mutually exclusive by default.

After all they are completely new sub-sections of your program. If you think to git like in your example: they are brand new exe part of a subsystem.

Options appreciated.

Regards,
Giacomo

[gsscoder]

Compare ideas is fantastic. This last thoughts + your implementation as referral -> made my path to verbs clear!!!
Thanks, ...
Giacomo

PS: please add more comments anyway if you have time.

[nemec]

The idea is that any "master" specific arguments are specified before the verb. It's technically possible to specify them after all of the verb's arguments (fileman rm filename -v), but only if the argument isn't also defined by the verb (otherwise, it's parsed in the context of the verb).

Documenting this would be a good idea so that users know what to expect. The automated help will probably need some extra work, too. If both the master and verb define "-v", to apply the verbose argument to the master the command should read fileman -v rm filename.

As it's designed at the moment you can specify multiple verbs, one after the other (each with its own arguments -- see test ParseOptionsWithMultipleSuboptions). The better idea, I think, is to allow nested verbs. Say your rm verb defines its own verb/suboption remote (for remote deleting files or something...), then you'd run it as fileman rm -v remote filename. Instead of the while loop I added you can just recursively call DoParseArguments, passing in the argument iterator and the "verb object". Since verbs have the exact same parsing structure as the master, it's parsed pretty much automatically -- this was my first implementation, but I scrapped it for what you see now.

As others have mentioned, it would be nice to, in addition to the SubOption/VerbOption object, allowing VerbOption to be applied to a method that's run (which would allow you to execute subsystem exes, among other things). So what I have implemented now is only a starting point!

[alexanderfast]

I can add a specific usage example of "master" specific arguments. This is what I do in the project where I hacked together verb support (parts of what you see in the first post).

I've created a command line client wrapping a rest api, so for simplicity lets call my two verbs get and post. Both have a bunch of identical arguments like url, username and password. My current solution was to create a subclass of CommandLineOptionsBase called ServerDependantOptions which both verbs use as the base class for their specific options. The main drawback is that the help text for both verbs feel bloated.

I think a simple REST wrapper is a great example of where both verbs and "master arguments" would be oh so nice.

Edit: Think of a specialized and verb based variant of curl.

[alexanderfast]

It was a truly great discussion! But time has come to close this, it's been implemented.

[gsscoder]

agree...

[DriverEntry]

Hi,
could you say why did you stop on verbs mapped to the option properties instead of verbs mapped to methods and thier parameters (just like in mizipzor's first post). Don't you think that the second way would be much more handfull since parser could find and invoke our methods for us. So we need not to do odd work to check the verb value in onVerbCommand delegate and to convert subOptions type after calling ParseArguments() as described in wiki's "Verb Commands" page.

[gsscoder]

@DriverEntry may be you're quoting at stable-1.9.*? Some time passed... Now I don't remember the whole story and why that route was favored. The interesting thing of having source code is forking for PR or forking for private use or whatever you want, isn't it? Also my absence to the project was not determinant since someone else can fork and maintain it...

Backing to the problem. If you look at 2.0 (which as I said is an attempt to simplify all inside/out) you define X normal types for parsing and the parser will give you the instance associated with the correct verb.
Why this architecture in 2.0 pre-rel? Because apps that uses verbs are similar to the one which not use one, making things easy.

Hope all this, helped! And thank you for your interest in this project and for comments that ALWAYS help.
R. Giacomo