Skip to content

Stopwords remover xml documentation #3413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Apr 20, 2019
Merged

Stopwords remover xml documentation #3413

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ea847a9
Update documentation for stopwords
Apr 18, 2019
eb72063
Merge branch 'master' into Ivanidze/StopWordsRemoverXmlDocumentation
Apr 18, 2019
7e1016c
string to text
Apr 18, 2019
818ec4a
address comments
Apr 19, 2019
a8049d9
address comments
Apr 19, 2019
f3f9003
ReadOnly to TextDataViewType
Apr 19, 2019
5c5fc26
address comments
Apr 19, 2019
abff8d4
small update for types
Apr 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 38 additions & 13 deletions src/Microsoft.ML.Transforms/Text/StopWordsRemovingTransformer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,10 +61,7 @@ public ITransformer CreateComponent(IHostEnvironment env, IDataView input, OneTo
}

/// <summary>
/// A Stopword remover transform based on language-specific lists of stop words (most common words)
/// from Office Named Entity Recognition project.
/// The transform is usually applied after tokenizing text, so it compares individual tokens
/// (case-insensitive comparison) to the stopwords.
/// <see cref="ITransformer"/> resulting from fitting a <see cref="StopWordsRemovingEstimator"/>.
/// </summary>
public sealed class StopWordsRemovingTransformer : OneToOneTransformerBase
{
Expand Down Expand Up @@ -483,10 +480,25 @@ private protected override Func<int, bool> GetDependenciesCore(Func<int, bool> a
}

/// <summary>
/// Stopword remover removes language-specific list of stop words (most common words)
/// This is usually applied after tokenizing text, so it compares individual tokens
/// (case-insensitive comparison) to the stopwords.
/// <see cref="IEstimator{TTransformer}"/> for the <see cref="CustomStopWordsRemovingTransformer"/>.
/// </summary>
/// <remarks>
/// <format type="text/markdown"><![CDATA[
/// ### Estimator Characteristics
/// | | |
/// | -- | -- |
/// | Does this estimator need to look at the data to train its parameters? | No |
/// | Input column data type | Vector of [Text]<xref:System.ReadOnlyMemory{System.Char}> |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Text]xref:System.ReadOnlyMemory{System.Char} [](start = 45, length = 47)

nice. let's check how this renders when the preview site is refreshed tomorrow

Copy link

@shmoradims shmoradims Apr 19, 2019 •
edited by Ivanidzo4ka
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vector [](start = 35, length = 6)

should this be known-sized? #ByDesign

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it can be either known-sized or unknown size.
output would always be unknown size

In reply to: 276871092 [](ancestors = 276871092)

/// | Output column data type | Vector of unknown size of [Text]<xref:System.ReadOnlyMemory{System.Char}> |
///
/// The resulting <xref:Microsoft.ML.Transforms.Text.StopWordsRemovingTransformer/> creates a new column, named as specified in the output column name parameters, and
/// fills it with vector of strings similar to vector of strings in input column but removing all, predefined for certain language, strings from it.
Copy link
Contributor Author

@Ivanidzo4ka Ivanidzo4ka Apr 18, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strings [](start = 32, length = 7)

replace strings with texts? #Resolved

/// All text comparison made by casting predefined text and text from input column to lower case using casing rules of invariant culture.
/// See the See Also section for links to examples of the usage.
Copy link
Contributor

@natke natke Apr 19, 2019 •
edited by Ivanidzo4ka
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest: The resulting xref:Microsoft.ML.Transforms.Text.StopWordsRemovingTransformer/ creates a new column, named as specified in the output column name parameter, and fills it with a vector of words containing all of the words in the input column **except the predefined list of stopwords for the specified language. #Resolved

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Small type: extra ** in there

/// ]]>
/// </format>
/// </remarks>
/// <seealso cref="TextCatalog.RemoveDefaultStopWords(TransformsCatalog.TextTransforms, string, string, Language)" />
public sealed class StopWordsRemovingEstimator : TrivialEstimator<StopWordsRemovingTransformer>
{
/// <summary>
Expand Down Expand Up @@ -627,9 +639,7 @@ public override SchemaShape GetOutputSchema(SchemaShape inputSchema)
}

/// <summary>
/// Custom stopword remover removes specified list of stop words.
/// This is usually applied after tokenizing text, so it compares individual tokens
/// (case-insensitive comparison) to the stopwords.
/// <see cref="ITransformer"/> resulting from fitting a <see cref="CustomStopWordsRemovingEstimator"/>.
/// </summary>
public sealed class CustomStopWordsRemovingTransformer : OneToOneTransformerBase
{
Expand Down Expand Up @@ -1076,10 +1086,25 @@ protected override Delegate MakeGetter(DataViewRow input, int iinfo, Func<int, b
}

/// <summary>
/// Custom stopword remover removes specified list of stop words.
/// This is usually applied after tokenizing text, so it compares individual tokens
/// (case-insensitive comparison) to the stopwords.
/// <see cref="IEstimator{TTransformer}"/> for the <see cref="CustomStopWordsRemovingTransformer"/>.
/// </summary>
/// <remarks>
/// <format type="text/markdown"><![CDATA[
/// ### Estimator Characteristics
/// | | |
/// | -- | -- |
/// | Does this estimator need to look at the data to train its parameters? | No |
/// | Input column data type | Vector of [Text]<xref:System.ReadOnlyMemory{System.Char}> |
/// | Output column data type | Vector of unknown size of [Text]<xref:System.ReadOnlyMemory{System.Char}> |
Copy link

@shmoradims shmoradims Apr 19, 2019 •
edited by Ivanidzo4ka
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vector of unknown size of [](start = 36, length = 25)

how about "Unknown-sized vector of"? #Resolved

Copy link
Contributor

@natke natke Apr 19, 2019 •
edited by Ivanidzo4ka
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unspecified? #ByDesign

///
/// The resulting <xref:Microsoft.ML.Transforms.Text.CustomStopWordsRemovingTransformer/> creates a new column, named as specified in the output column name parameters, and
/// fills it with vector of strings similar to vector of strings in input column but removing all specified strings from it.
/// All text comparison made by casting specified text and text from input column to lower case using casing rules of invariant culture.
/// See the See Also section for links to examples of the usage.
/// ]]>
/// </format>
/// </remarks>
/// <seealso cref="TextCatalog.RemoveStopWords(TransformsCatalog.TextTransforms, string, string, string[])" />
public sealed class CustomStopWordsRemovingEstimator : TrivialEstimator<CustomStopWordsRemovingTransformer>
{
/// <summary>
Expand Down
24 changes: 14 additions & 10 deletions src/Microsoft.ML.Transforms/Text/TextCatalog.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,12 +251,14 @@ internal static NgramExtractingEstimator ProduceNgrams(this TransformsCatalog.Te
=> new NgramExtractingEstimator(Contracts.CheckRef(catalog, nameof(catalog)).GetEnvironment(), columns);

/// <summary>
/// Removes stop words from incoming token streams in <paramref name="inputColumnName"/>
/// and outputs the token streams without stopwords as <paramref name="outputColumnName"/>.
/// Create a <see cref="CustomStopWordsRemovingEstimator"/>, which copies the data from the column specified in <paramref name="inputColumnName"/>
/// to a new column: <paramref name="outputColumnName"/> and removes predifined set of text specific for <paramref name="language"/> from it.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

specific for [](start = 100, length = 40)

for later: we should that these language-specific lists to docs.

/// </summary>
/// <param name="catalog">The text-related transform's catalog.</param>
/// <param name="outputColumnName">The column containing output text. Null means <paramref name="inputColumnName"/> is replaced.</param>
/// <param name="inputColumnName">The column containing text to remove stop words on.</param>
/// <param name="catalog">The transform's catalog.</param>
/// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
/// This column's data type will be unknown size vector of text.</param>
/// <param name="inputColumnName">Name of the column to copy the data from.
/// This estimator operates over vector of text.</param>
/// <param name="language">Langauge of the input text column <paramref name="inputColumnName"/>.</param>
/// <example>
/// <format type="text/markdown">
Expand All @@ -272,12 +274,14 @@ public static StopWordsRemovingEstimator RemoveDefaultStopWords(this TransformsC
=> new StopWordsRemovingEstimator(Contracts.CheckRef(catalog, nameof(catalog)).GetEnvironment(), outputColumnName, inputColumnName, language);

/// <summary>
/// Removes stop words from incoming token streams in <paramref name="inputColumnName"/>
/// and outputs the token streams without stopwords as <paramref name="outputColumnName"/>.
/// Create a <see cref="CustomStopWordsRemovingEstimator"/>, which copies the data from the column specified in <paramref name="inputColumnName"/>
/// to a new column: <paramref name="outputColumnName"/> and removes text specified in <paramref name="stopwords"/> from it.
/// </summary>
/// <param name="catalog">The text-related transform's catalog.</param>
/// <param name="outputColumnName">The column containing output text. Null means <paramref name="inputColumnName"/> is replaced.</param>
/// <param name="inputColumnName">The column containing text to remove stop words on.</param>
/// <param name="catalog">The transform's catalog.</param>
/// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
/// This column's data type will be unknown size vector of text.</param>
/// <param name="inputColumnName">Name of the column to copy the data from.
/// This estimator operates over vector of text.</param>
/// <param name="stopwords">Array of words to remove.</param>
/// <example>
/// <format type="text/markdown">
Expand Down