Skip to content

Added RankingEvaluatorOptions and removed the truncation limit. #4081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 58 commits into from
Oct 10, 2019
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
8749a10
Merge pull request #1 from dotnet/master
harishsk Jun 20, 2019
fe25bf6
Fixed build errors resulting from upgrade to VS2019 compilers
harishsk Jun 21, 2019
cb446be
Added additional message describing the previous fix
harishsk Jun 21, 2019
b5ee220
Merge pull request #2 from dotnet/master
harishsk Jun 21, 2019
b9a7471
Merge pull request #3 from dotnet/master
harishsk Jun 24, 2019
80e238d
Merge pull request #4 from dotnet/master
harishsk Jun 27, 2019
2ef424d
Merge pull request #5 from dotnet/master
harishsk Jul 9, 2019
3958f01
Merge pull request #6 from dotnet/master
harishsk Aug 7, 2019
56d4595
Fixed #3993
harishsk Aug 7, 2019
00bc7ef
Merge pull request #7 from dotnet/master
harishsk Aug 14, 2019
d0462f1
Merge pull request #8 from dotnet/master
harishsk Aug 16, 2019
87cefbc
Merge branch 'master' into bugfix_3993
harishsk Aug 16, 2019
c3a908b
Reverted previous change and added a separate class to control evalua…
harishsk Aug 21, 2019
c0a430a
Merge pull request #9 from dotnet/master
harishsk Aug 21, 2019
0b55903
Syncing upstream fork (#10)
harishsk Aug 30, 2019
56983d5
Syncing upstream fork (#11)
harishsk Aug 30, 2019
3382d1d
Merge pull request #13 from dotnet/master
harishsk Sep 6, 2019
8ca5d01
Merge remote-tracking branch 'upstream/master'
harishsk Sep 10, 2019
4ac459e
Added unit test for ranking evaluation with options
harishsk Sep 11, 2019
8f20ea4
Merge remote-tracking branch 'upstream/master'
harishsk Sep 11, 2019
f9f9e1d
Changed visibility of OutputGroupSummary to internal until we expose …
harishsk Sep 11, 2019
21cb8f3
Added a unit test for maml ranking evaluation
harishsk Sep 12, 2019
138f201
Merge remote-tracking branch 'upstream/master'
harishsk Sep 13, 2019
55e3460
Merge remote-tracking branch 'upstream/master'
harishsk Sep 13, 2019
e43bba3
Merge remote-tracking branch 'upstream/master'
harishsk Sep 16, 2019
421d713
Merge branch 'master' of ssh://github.com/harishsk/machinelearning
harishsk Sep 16, 2019
4f4f81c
Merge remote-tracking branch 'upstream/master'
harishsk Sep 17, 2019
89082a5
Merge branch 'master' into bugfix_3993
harishsk Sep 17, 2019
f167af8
Merge branch 'master' of ssh://github.com/harishsk/machinelearning
harishsk Sep 17, 2019
0d4d34f
getting rid of maxTruncationLevel
Lynx1820 Sep 18, 2019
6cd2f15
removed unnecessary imports
Lynx1820 Sep 18, 2019
1424ab3
removed old comment
Lynx1820 Sep 18, 2019
3ee03ca
removed old comment
Lynx1820 Sep 18, 2019
34b7a91
Merge remote-tracking branch 'upstream/master'
harishsk Sep 19, 2019
5539127
Merge remote-tracking branch 'upstream/master'
harishsk Sep 19, 2019
02053a6
adding a maml baseline test
Lynx1820 Sep 23, 2019
35ad3c0
reformatted maml test and moved to where all the other maml tests are
Lynx1820 Sep 25, 2019
0eb3e2b
Reverted back some spacing. Accidentally reverted some changes in Eva…
Lynx1820 Sep 25, 2019
a3291b1
added more relaxed precision
Lynx1820 Sep 25, 2019
37af437
Merge remote-tracking branch 'upstream/master'
harishsk Sep 26, 2019
68f1f35
baseline precision set 1
Lynx1820 Sep 26, 2019
5b90a34
added more helpful debugging comment
Lynx1820 Sep 26, 2019
0efe238
temp
Lynx1820 Sep 27, 2019
b6584aa
Merge remote-tracking branch 'upstream/master'
harishsk Sep 27, 2019
7d47832
Merge branch 'master' of ssh://github.com/harishsk/machinelearning
harishsk Sep 27, 2019
0e99776
Merge branch 'master' into bugfix_3993
harishsk Sep 27, 2019
20a4490
Revert "temp"
Lynx1820 Sep 30, 2019
0d111f4
Revert "added more relaxed precision"
Lynx1820 Sep 30, 2019
72d1a4d
using fastRankRanking to test RankingEvaluator instead of lightgbm
Lynx1820 Sep 30, 2019
ea9ebed
Merge branch 'bugfix_3993' of git://github.com/harishsk/machinelearni…
Lynx1820 Sep 30, 2019
013be4f
reverted some indenting and LightGBM imports
Lynx1820 Sep 30, 2019
d2ae365
removed the stratification flag to get real numbers
Lynx1820 Oct 7, 2019
a9e6db8
testcase change added
Lynx1820 Oct 7, 2019
5855f99
Added the Bestfriend attribute back
Lynx1820 Oct 7, 2019
724bb12
Changed implementation of discount map computation to minimize mutabl…
harishsk Oct 9, 2019
d009f55
Added doc strings
harishsk Oct 9, 2019
8f7b6cd
Fixed bug in creation of fixed discount map
harishsk Oct 9, 2019
30d56a0
Merge remote-tracking branch 'upstream/master' into bugfix_3993
harishsk Oct 10, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions src/Microsoft.ML.Data/Commands/EvaluateCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,7 @@ namespace Microsoft.ML.Data
/// This class contains information about an overall metric, namely its name and whether it is a vector
/// metric or not.
/// </summary>
[BestFriend]
internal sealed class MetricColumn
public sealed class MetricColumn
{
/// <summary>
/// An enum specifying whether the metric should be maximized or minimized while sweeping. 'Info' should be
Expand Down
2 changes: 1 addition & 1 deletion src/Microsoft.ML.Data/Evaluators/AucAggregator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

namespace Microsoft.ML.Data
{
internal abstract partial class EvaluatorBase<TAgg>
public abstract partial class EvaluatorBase<TAgg>
{
internal abstract class AucAggregatorBase
{
Expand Down
3 changes: 1 addition & 2 deletions src/Microsoft.ML.Data/Evaluators/EvaluatorBase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ namespace Microsoft.ML.Data
/// <see cref="GetPerInstanceMetricsCore"/>. Note that the input <see cref="RoleMappedData"/> is assumed to contain all the column
/// roles needed for evaluation, including the score column.
/// </summary>
[BestFriend]
internal abstract partial class EvaluatorBase<TAgg> : IEvaluator
public abstract partial class EvaluatorBase<TAgg> : IEvaluator
where TAgg : EvaluatorBase<TAgg>.AggregatorBase
{
protected readonly IHost Host;
Expand Down
5 changes: 2 additions & 3 deletions src/Microsoft.ML.Data/Evaluators/RankingEvaluator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,7 @@

namespace Microsoft.ML.Data
{
[BestFriend]
internal sealed class RankingEvaluator : EvaluatorBase<RankingEvaluator.Aggregator>
public sealed class RankingEvaluator : EvaluatorBase<RankingEvaluator.Aggregator>
Copy link
Contributor

@justinormont justinormont Aug 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is exposing the four classes the right method?

/cc @eerhardt #Resolved

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No this is not the right approach. @harishsk - can you follow the same pattern we are following for the other evaluators? None of the other evaluator classes are public. There should be no reason why this is.


In reply to: 314017216 [](ancestors = 314017216)

{
public sealed class Arguments
{
Expand Down Expand Up @@ -271,7 +270,7 @@ public sealed class Aggregator : AggregatorBase
{
public sealed class Counters
{
public const int MaxTruncationLevel = 10;
public const int MaxTruncationLevel = 100;
Copy link
Contributor

@justinormont justinormont Aug 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a technical need for any fixed limit? From the data science side, I don't know of one.

Is there a ramification in our code for simply removing the limit? #Resolved

Copy link
Contributor Author

@harishsk harishsk Aug 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original author of this code has assumed 100 to be a reasonable limit and that assumption is enforced in many places in the code. The MaxTruncationLevel of 10 set above made it a bit inconsistent and my fix just makes the max limit consistent.

In theory it should be possible to remove the limit. But the way the code is structured makes it a bit more invasive than necessary.

Right now, in RankingEvaluator.cs the static class RankingUtils class has a static method DiscountMap which carries this assumption with a hard coded constant of 100 and also has a comment to reflect that.

//RankingEvaluator.cs on line 954
var discountMap = new Double[100]; //Hard to believe anyone would set truncation Level higher than 100

The way the code is structured, this requires some well defined maximum. It should be possible to remove this but that would require a bigger code change. I would like input on whether or the original author's assumption that 100 is a reasonable limit is a good assumption. If it is really necessary to not have a limit at all, it is a bigger code change.

Can we take this fix as is and open a new bug to remove the limit and we can triage that separately?

#Resolved

Copy link
Contributor

@justinormont justinormont Aug 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a datapoint, TensorFlow has no limit, and if the user doesn't specify topn, it defaults to infinity:
https://github.com/tensorflow/ranking/blob/7ae75da8a9cbe4c41a64dc7df394301cdfabbb02/tensorflow_ranking/python/metrics.py#L202-L203

It looks to be a pretty small fix to have no hard limit.

Having no limit lets the user set their truncation level to say 500 should they want to.

Currently the value of 10 assumes users won't care about the quality of their search results beyond the 1st page of results. Setting to 100 similarly assumes users won't care about the quality of results after page 10. #Resolved


public readonly int TruncationLevel;
private readonly List<Double[]> _groupNdcg;
Expand Down