Skip to content

Commit b916d37

Browse files
Prashanth GovindarajaneerhardtzHaytamjwood803MgSam
authored
Move DataFrame to machinelearning (#5641)
* Change namespace to Microsoft.Data.Analysis (#2773) * Update namespace to Microsoft.Data.Analysis * Remove "DataFrame" from the test project name * APIs for reversed binary operators (#2769) * Support reverse binary operators * Fix file left behind in a rebase * Fix whitespace * Throw for incompatible inPlace (#2778) * Throw if inPlace is set and types mismatch * Unit test * Better error message * Remove empty lines * Version, Tags and Description for Nuget (#2779) * Version, Tags and Description for Nuget * sq * Flags for release (#2781) * Publish packages to artifacts * Flags for release * Fix the Description method to not throw (#2786) * Fix the Description method to not crash Adds an Info method * sq * Address feddback * Last round of feedback * Use dataTypes if it passed in to LoadCsv (#2791) * Fix LoadCsv to use dataType if it passed in * sq * Don't read the full file after guessRows lines have been read * Address feedback * Last round of feedback * Creating a `Rows` property, similar to `Columns` (#2794) * Rows collection, similar to Columns * Doc * Some minor clean up * Make DataFrameRow a view into the DataFrame * sq * Address feedback * Remove DataFrame.RowCount * More row count changes * sq * Address feedback * Merge upstream * DataFrame.LoadCsv throws an exception on projects targeting < netcore3.0 (#2797) Fixing by passing in an encoding and a default buffer size. Also, get our tests running on .NET Framework. Fix #2783 * Params constructor on DataFrame (#2800) * Params constructor on DataFrame * Delete redundant constructors * Remove `T : unmanaged` constraint from DataFrameColumn.BinaryOperations (#2801) * Remove T : unmanaged constraint from DataFrameColumn.BinaryOperations * Address feedback * Rename the value version of the APIs * sq * Fix build * Address feedback * Remove Value from the APIs * sq * Address feedback * Bump version to 0.2.0 (#2803) * Add Apply<TResult>method to PrimitiveDataFrameColumn (#2807) * Add Apply method to PrimitiveDataFrameColumn and its container * Add TestApply test * Remove unused df variable in DataFrameTests * Add xml doc comments to Apply method * Add additional tests for ReadCsv (#2811) * Add additional tests for ReadCsv * Update asserts * Add empty row and skip test pending another fix * Remove test for another issue * Added static factory methods to DataFrameColumn (#2808) * Added static factory methods to DataFrameColumn where they make sense (for the overloads where its possible to infer the column's type). * Remove regions * Update some parts of the unit tests to use static factory methods to create DataFrameColumns. * Remove errant {T} on StringDataFrameColumn. * PR feedback Co-authored-by: Eric Erhardt <[email protected]> * Append rows to a DataFrame (#2823) * Append rows to a DataFrame * Unit test * Update unit tests and doc * Need to perfrom a type check every time * sq * Update unit test * Address comments * Move corefxlab to arcade (#2795) * Add eng folder * First cut of moving corefxlab to arcade * Move arcade symbol validation inside official buil * Move base yml file to root * Arcade will build, publish packages and symbols * UpdateXlf. Review this * Arcade Update to version 5.0.0-beta.19575.4 to include Experimental Channel * Remove property that was causing the build to fail * Moving global properties to the main Yaml instead of step in order to unblock publishing * Committing xlfs and changing the build script to not update Xlf on build * clean up corefxlab-base.yml * sq * Delete unused files and scripts * Get rid of all the xlf stuff * Remove UpdateXlfOnBuild for non-NT builds * Minor cleanup * More cleanup * update eng\build.sh permission * Rename to Nuget.config * sq * Remove the runtime spec from global.json * Don't publish test projs * Typo * Move version prefix to versions.props Change prereleaselabel to alpha * Increment version number to list as the latest package Increment version number of Microsoft.Experimental.Collections to list as the latest package Turn off graph generation * Update the Readme * Test removing the scripts folder * Touch readme to force a change * Address Jose's comments * Typo * Move versions to eng/versions.props * Benchmark.proj needs to refer to xunit * Clean up dependencies.props * Remove dependencies.props Co-authored-by: Jose Perez Rodriguez <[email protected]> * Rename Sort to OrderBy (#2814) * Rename sort to orderby and add orderbydescending method * Add doc strings * Update bench mark test * Update tests * Update DataFrameColumn to use orderby * Update doc comment * Additions to sortby * Revert "Additions to sortby" This reverts commit 3931d4e2a72ce44a539be7c27b2592395f3efd35. * Revert "Update doc comment" This reverts commit 192f7797fe2b77625486637badf77046162fedbf. * Revert "Update DataFrameColumn to use orderby" This reverts commit 8f94664c5fd18570cd2b601535e816ca5dd5e3c4. * Explode column types and generate converters (#2857) * Explode column types and generate converters * Clean this * sq * sq * Cherry pick for next commit * sq * Undo unnecessary change * Address remaining concerns from the 2nd DataFrame API Review (#2861) * Move string indexer to Columns * API changes from the 2nd API review * Unit tests * Address comments * Add binary operations and operators on the exploded columns (#2867) * Generate combinations of binary operations and Add * Numeric Converters and CloneAsNumericColumns * Binary, Comparison and Shift operations * Clean up and bug fix * Fix the binary op apis to not be overridden * Internal constructors for exploded types * Proper return types for exploded types * Update unit tests * Update csproj * Revert "Fix the binary op apis to not be overridden" This reverts commit 2dc2240c9449930139c1492d1388d5e1f8ba5fa1. * Bug fix and unit test * Constructor that takes in a container * Unit tests * Call the implementation where possible * Review sq * sq * Cherry pick for next commit * sq * Undo unnecessary change * Rename to the system namespace column types * Address comments * Push to pull locally * Mimic C#'s arithmetic grammar in DataFrame * Address feedback * Reduce the number of partial column definitions * Address feedback * Add APIs to get the strongly typed columns from a DataFrame (#2878) * CP * sq * sq * Improve docs * Enable xml docs for Data.Analysis (#2882) * Enable xml docs for Data.Analysis * Fix /// summary around inheritdoc * Minor doc changes * sq * sq * Address feedback * Add Apply to ArrowStringDataFrameColumn (#2889) * Support for Exploded columns types in Arrow and IO scenarios (#2885) * Support for Exploded columns types in Arrow and IO scenarios * Unit tests * Address feedback * Bump version (#2890) * Fix versioning to allow for individual stable packages (#2891) * Fix versioning to allow for individual stable packages * sq * Bump Microsoft.Data.Analysis version to 0.4.0 (#2892) * Bump Microsoft.Data.Analysis version to 0.4.0 * Fix dotnet/corefxlab#2906 (#2907) * Fix dotnet/corefxlab#2906 * Improvements and unit tests * sq * Better fix * sq * Improve LoadCsv to handle null values when deducing the column types (#2916) * Unit test to repro * Fix dotnet/corefxlab#2915 Append a null value to a column when encountering it instead of changing the column type to a StringDataFrameColumn * Update src/Microsoft.Data.Analysis/DataFrame.IO.cs Co-authored-by: Günther Foidl <[email protected]> * Update src/Microsoft.Data.Analysis/DataFrame.cs Co-authored-by: Günther Foidl <[email protected]> * Feedback Co-authored-by: Günther Foidl <[email protected]> * Create a 0.4.0 package (#2918) * Revert "Create a 0.4.0 package (#2918)" (#2919) This reverts commit 0bef531. * Produce a 0.4.0 build (#2920) * Default Length for StringDataFrameColumn (#2921) (#2923) * Increment version and stop producing stable packages (#2922) * Increment version and stop producing stable packages * Add DataFrame object formatter. (#2931) * Add DataFrame object formatter. * Update nuget dependencies. * Apply CR fixes. * Fix a bug in InsertColumn * Add Microsoft.Data.Analysis.nuget project (#2933) * Add DataFrame object formatter. * Update nuget dependencies. * Apply CR fixes. * Remove ReferenceOutputAssembly added to from Microsoft.Data.Analysys.csproj. * Add Microsoft.Data.Analysis.nuget project. * Move project to src. Fix nuget project settings. * Remove NoBuild property from project. * Remove IncludeBuildOutput and IncludeSymbols from project. * Add VersionPrefix to project. * Add IncludeBuildOutput property. * Add unit tests. * Downgrade from netcoreapp3.1 to netcoreapp3.0 * Upgrade from netcoreapp3.0 to netcoreapp3.1 (dotnet interactive is not compatible with 3.0) * Add netcoreapp3.1 to global settings * Add dotnet 3.1.5 runtime to global settings * Build fixes * Moving MDAI into interactive-extensions folder of the package * Minor refactoring * Respond to PR feedback Co-authored-by: Prashanth Govindarajan <[email protected]> Co-authored-by: Jose Perez Rodriguez <[email protected]> Co-authored-by: Eric Erhardt <[email protected]> * ColumnName indexer on DataFrame (#2959) * ColumnName indexer on DataFrame Fixes dotnet/corefxlab#2934 * Unit tests * Null column name * Implement FillNulls() for ArrowStringDataFrameColumn with inPlace: false (#2956) * implement FillNulls method for ArrowStringDataFrameColumn * additional asserts for testcase * Prevent DataFrame.Sample() method from returning duplicated rows (#2939) * resolves #2806 * replace forloop with ArraySegment<T> * reduce shuffle loop operations from O(Rows.Count) to O(numberOfRows) * Add WriteCsv plus unit tests. (#2947) * Add WriteCsv plus unit tests. * Add CultureInfo to WriteCsv. Remove index column param. Update unit tests. * Add CR changes. CultureInfo. Separator. * Format decimal types individually. Fix culture info. Fix unit tests. * Format decimal types individually. Fix culture info. Fix unit tests. * Missing values default to a `StringDataFrameColumn` (#2982) * Make LoadCsv more robust * Test empty string column * Retain prev guess where possible * Update FromArrowRecordBatches for dotnet-spark (#2978) * Support for RecordBatches with StructArrays * Sq * Address comments * Nits * Nits * Implement DataFrame.LoadCsvFromString (#2988) * Implement DataFrame.LoadCsvFromString * Address comments * Part 1 of porting the csv reader (#2997) * Move to the test folder * Suppress warnings * Move extensions reference out of props Make MDA.test use the props defined TFM Comment out 2 unit tests * Address feedback * Address feedback * Default to preview version * Update nuget.config Co-authored-by: Eric Erhardt <[email protected]> Co-authored-by: Haytam Zanid <[email protected]> Co-authored-by: Jon Wood <[email protected]> Co-authored-by: Sam <[email protected]> Co-authored-by: Jose Perez Rodriguez <[email protected]> Co-authored-by: Günther Foidl <[email protected]> Co-authored-by: Rhys Parry <[email protected]> Co-authored-by: daniel costea <[email protected]> Co-authored-by: Ramon <[email protected]>
1 parent 8dd47c1 commit b916d37

File tree

87 files changed

+64318
-10
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

87 files changed

+64318
-10
lines changed

Microsoft.ML.sln

+108
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,14 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.ML", "src\Microso
141141
EndProject
142142
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.ML.Mkl.Redist", "src\Microsoft.ML.Mkl.Redist\Microsoft.ML.Mkl.Redist.csproj", "{4584326B-C5B3-4CAE-B98A-34C5F5AA16F3}"
143143
EndProject
144+
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Data.Analysis", "src\Microsoft.Data.Analysis\Microsoft.Data.Analysis.csproj", "{84150C22-0627-4A11-81C9-F214762855EA}"
145+
EndProject
146+
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Data.Analysis.Interactive", "src\Microsoft.Data.Analysis.Interactive\Microsoft.Data.Analysis.Interactive.csproj", "{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}"
147+
EndProject
148+
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Data.Analysis.Tests", "test\Microsoft.Data.Analysis.Tests\Microsoft.Data.Analysis.Tests.csproj", "{0B765344-11A4-4738-9759-5060599DC134}"
149+
EndProject
150+
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Data.Analysis.Interactive.Tests", "test\Microsoft.Data.Analysis.Interactive.Tests\Microsoft.Data.Analysis.Interactive.Tests.csproj", "{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}"
151+
EndProject
144152
Global
145153
GlobalSection(SolutionConfigurationPlatforms) = preSolution
146154
Debug|Any CPU = Debug|Any CPU
@@ -1680,6 +1688,102 @@ Global
16801688
{4584326B-C5B3-4CAE-B98A-34C5F5AA16F3}.Release-netfx|Any CPU.Build.0 = Release-netfx|Any CPU
16811689
{4584326B-C5B3-4CAE-B98A-34C5F5AA16F3}.Release-netfx|x64.ActiveCfg = Release-netfx|Any CPU
16821690
{4584326B-C5B3-4CAE-B98A-34C5F5AA16F3}.Release-netfx|x64.Build.0 = Release-netfx|Any CPU
1691+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
1692+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug|Any CPU.Build.0 = Debug|Any CPU
1693+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug|x64.ActiveCfg = Debug|Any CPU
1694+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug|x64.Build.0 = Debug|Any CPU
1695+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug-netcoreapp3_1|Any CPU.ActiveCfg = Debug-netcoreapp3_1|Any CPU
1696+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug-netcoreapp3_1|Any CPU.Build.0 = Debug-netcoreapp3_1|Any CPU
1697+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug-netcoreapp3_1|x64.ActiveCfg = Debug-netcoreapp3_1|Any CPU
1698+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug-netcoreapp3_1|x64.Build.0 = Debug-netcoreapp3_1|Any CPU
1699+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug-netfx|Any CPU.ActiveCfg = Debug-netfx|Any CPU
1700+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug-netfx|Any CPU.Build.0 = Debug-netfx|Any CPU
1701+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug-netfx|x64.ActiveCfg = Debug-netfx|Any CPU
1702+
{84150C22-0627-4A11-81C9-F214762855EA}.Debug-netfx|x64.Build.0 = Debug-netfx|Any CPU
1703+
{84150C22-0627-4A11-81C9-F214762855EA}.Release|Any CPU.ActiveCfg = Release|Any CPU
1704+
{84150C22-0627-4A11-81C9-F214762855EA}.Release|Any CPU.Build.0 = Release|Any CPU
1705+
{84150C22-0627-4A11-81C9-F214762855EA}.Release|x64.ActiveCfg = Release|Any CPU
1706+
{84150C22-0627-4A11-81C9-F214762855EA}.Release|x64.Build.0 = Release|Any CPU
1707+
{84150C22-0627-4A11-81C9-F214762855EA}.Release-netcoreapp3_1|Any CPU.ActiveCfg = Release-netcoreapp3_1|Any CPU
1708+
{84150C22-0627-4A11-81C9-F214762855EA}.Release-netcoreapp3_1|Any CPU.Build.0 = Release-netcoreapp3_1|Any CPU
1709+
{84150C22-0627-4A11-81C9-F214762855EA}.Release-netcoreapp3_1|x64.ActiveCfg = Release-netcoreapp3_1|Any CPU
1710+
{84150C22-0627-4A11-81C9-F214762855EA}.Release-netcoreapp3_1|x64.Build.0 = Release-netcoreapp3_1|Any CPU
1711+
{84150C22-0627-4A11-81C9-F214762855EA}.Release-netfx|Any CPU.ActiveCfg = Release-netfx|Any CPU
1712+
{84150C22-0627-4A11-81C9-F214762855EA}.Release-netfx|Any CPU.Build.0 = Release-netfx|Any CPU
1713+
{84150C22-0627-4A11-81C9-F214762855EA}.Release-netfx|x64.ActiveCfg = Release-netfx|Any CPU
1714+
{84150C22-0627-4A11-81C9-F214762855EA}.Release-netfx|x64.Build.0 = Release-netfx|Any CPU
1715+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
1716+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug|Any CPU.Build.0 = Debug|Any CPU
1717+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug|x64.ActiveCfg = Debug|Any CPU
1718+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug|x64.Build.0 = Debug|Any CPU
1719+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug-netcoreapp3_1|Any CPU.ActiveCfg = Debug-netcoreapp3_1|Any CPU
1720+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug-netcoreapp3_1|Any CPU.Build.0 = Debug-netcoreapp3_1|Any CPU
1721+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug-netcoreapp3_1|x64.ActiveCfg = Debug-netcoreapp3_1|Any CPU
1722+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug-netcoreapp3_1|x64.Build.0 = Debug-netcoreapp3_1|Any CPU
1723+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug-netfx|Any CPU.ActiveCfg = Debug-netfx|Any CPU
1724+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug-netfx|Any CPU.Build.0 = Debug-netfx|Any CPU
1725+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug-netfx|x64.ActiveCfg = Debug-netfx|Any CPU
1726+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Debug-netfx|x64.Build.0 = Debug-netfx|Any CPU
1727+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release|Any CPU.ActiveCfg = Release|Any CPU
1728+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release|Any CPU.Build.0 = Release|Any CPU
1729+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release|x64.ActiveCfg = Release|Any CPU
1730+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release|x64.Build.0 = Release|Any CPU
1731+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release-netcoreapp3_1|Any CPU.ActiveCfg = Release-netcoreapp3_1|Any CPU
1732+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release-netcoreapp3_1|Any CPU.Build.0 = Release-netcoreapp3_1|Any CPU
1733+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release-netcoreapp3_1|x64.ActiveCfg = Release-netcoreapp3_1|Any CPU
1734+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release-netcoreapp3_1|x64.Build.0 = Release-netcoreapp3_1|Any CPU
1735+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release-netfx|Any CPU.ActiveCfg = Release-netfx|Any CPU
1736+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release-netfx|Any CPU.Build.0 = Release-netfx|Any CPU
1737+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release-netfx|x64.ActiveCfg = Release-netfx|Any CPU
1738+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923}.Release-netfx|x64.Build.0 = Release-netfx|Any CPU
1739+
{0B765344-11A4-4738-9759-5060599DC134}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
1740+
{0B765344-11A4-4738-9759-5060599DC134}.Debug|Any CPU.Build.0 = Debug|Any CPU
1741+
{0B765344-11A4-4738-9759-5060599DC134}.Debug|x64.ActiveCfg = Debug|Any CPU
1742+
{0B765344-11A4-4738-9759-5060599DC134}.Debug|x64.Build.0 = Debug|Any CPU
1743+
{0B765344-11A4-4738-9759-5060599DC134}.Debug-netcoreapp3_1|Any CPU.ActiveCfg = Debug-netcoreapp3_1|Any CPU
1744+
{0B765344-11A4-4738-9759-5060599DC134}.Debug-netcoreapp3_1|Any CPU.Build.0 = Debug-netcoreapp3_1|Any CPU
1745+
{0B765344-11A4-4738-9759-5060599DC134}.Debug-netcoreapp3_1|x64.ActiveCfg = Debug-netcoreapp3_1|Any CPU
1746+
{0B765344-11A4-4738-9759-5060599DC134}.Debug-netcoreapp3_1|x64.Build.0 = Debug-netcoreapp3_1|Any CPU
1747+
{0B765344-11A4-4738-9759-5060599DC134}.Debug-netfx|Any CPU.ActiveCfg = Debug-netfx|Any CPU
1748+
{0B765344-11A4-4738-9759-5060599DC134}.Debug-netfx|Any CPU.Build.0 = Debug-netfx|Any CPU
1749+
{0B765344-11A4-4738-9759-5060599DC134}.Debug-netfx|x64.ActiveCfg = Debug-netfx|Any CPU
1750+
{0B765344-11A4-4738-9759-5060599DC134}.Debug-netfx|x64.Build.0 = Debug-netfx|Any CPU
1751+
{0B765344-11A4-4738-9759-5060599DC134}.Release|Any CPU.ActiveCfg = Release|Any CPU
1752+
{0B765344-11A4-4738-9759-5060599DC134}.Release|Any CPU.Build.0 = Release|Any CPU
1753+
{0B765344-11A4-4738-9759-5060599DC134}.Release|x64.ActiveCfg = Release|Any CPU
1754+
{0B765344-11A4-4738-9759-5060599DC134}.Release|x64.Build.0 = Release|Any CPU
1755+
{0B765344-11A4-4738-9759-5060599DC134}.Release-netcoreapp3_1|Any CPU.ActiveCfg = Release-netcoreapp3_1|Any CPU
1756+
{0B765344-11A4-4738-9759-5060599DC134}.Release-netcoreapp3_1|Any CPU.Build.0 = Release-netcoreapp3_1|Any CPU
1757+
{0B765344-11A4-4738-9759-5060599DC134}.Release-netcoreapp3_1|x64.ActiveCfg = Release-netcoreapp3_1|Any CPU
1758+
{0B765344-11A4-4738-9759-5060599DC134}.Release-netcoreapp3_1|x64.Build.0 = Release-netcoreapp3_1|Any CPU
1759+
{0B765344-11A4-4738-9759-5060599DC134}.Release-netfx|Any CPU.ActiveCfg = Release-netfx|Any CPU
1760+
{0B765344-11A4-4738-9759-5060599DC134}.Release-netfx|Any CPU.Build.0 = Release-netfx|Any CPU
1761+
{0B765344-11A4-4738-9759-5060599DC134}.Release-netfx|x64.ActiveCfg = Release-netfx|Any CPU
1762+
{0B765344-11A4-4738-9759-5060599DC134}.Release-netfx|x64.Build.0 = Release-netfx|Any CPU
1763+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
1764+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug|Any CPU.Build.0 = Debug|Any CPU
1765+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug|x64.ActiveCfg = Debug|Any CPU
1766+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug|x64.Build.0 = Debug|Any CPU
1767+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug-netcoreapp3_1|Any CPU.ActiveCfg = Debug-netcoreapp3_1|Any CPU
1768+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug-netcoreapp3_1|Any CPU.Build.0 = Debug-netcoreapp3_1|Any CPU
1769+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug-netcoreapp3_1|x64.ActiveCfg = Debug-netcoreapp3_1|Any CPU
1770+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug-netcoreapp3_1|x64.Build.0 = Debug-netcoreapp3_1|Any CPU
1771+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug-netfx|Any CPU.ActiveCfg = Debug-netfx|Any CPU
1772+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug-netfx|Any CPU.Build.0 = Debug-netfx|Any CPU
1773+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug-netfx|x64.ActiveCfg = Debug-netfx|Any CPU
1774+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Debug-netfx|x64.Build.0 = Debug-netfx|Any CPU
1775+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release|Any CPU.ActiveCfg = Release|Any CPU
1776+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release|Any CPU.Build.0 = Release|Any CPU
1777+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release|x64.ActiveCfg = Release|Any CPU
1778+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release|x64.Build.0 = Release|Any CPU
1779+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release-netcoreapp3_1|Any CPU.ActiveCfg = Release-netcoreapp3_1|Any CPU
1780+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release-netcoreapp3_1|Any CPU.Build.0 = Release-netcoreapp3_1|Any CPU
1781+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release-netcoreapp3_1|x64.ActiveCfg = Release-netcoreapp3_1|Any CPU
1782+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release-netcoreapp3_1|x64.Build.0 = Release-netcoreapp3_1|Any CPU
1783+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release-netfx|Any CPU.ActiveCfg = Release-netfx|Any CPU
1784+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release-netfx|Any CPU.Build.0 = Release-netfx|Any CPU
1785+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release-netfx|x64.ActiveCfg = Release-netfx|Any CPU
1786+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25}.Release-netfx|x64.Build.0 = Release-netfx|Any CPU
16831787
EndGlobalSection
16841788
GlobalSection(SolutionProperties) = preSolution
16851789
HideSolutionNode = FALSE
@@ -1749,6 +1853,10 @@ Global
17491853
{C8DB58DC-6434-4431-A81F-263D86E2A5F3} = {AED9C836-31E3-4F3F-8ABC-929555D3F3C4}
17501854
{6CF88209-69DB-4B36-9604-3ECD9F163E96} = {09EADF06-BE25-4228-AB53-95AE3E15B530}
17511855
{4584326B-C5B3-4CAE-B98A-34C5F5AA16F3} = {09EADF06-BE25-4228-AB53-95AE3E15B530}
1856+
{84150C22-0627-4A11-81C9-F214762855EA} = {09EADF06-BE25-4228-AB53-95AE3E15B530}
1857+
{D9FDD2D5-BFFC-4A4D-8589-7F63AA3EA923} = {09EADF06-BE25-4228-AB53-95AE3E15B530}
1858+
{0B765344-11A4-4738-9759-5060599DC134} = {AED9C836-31E3-4F3F-8ABC-929555D3F3C4}
1859+
{8AFB8CC3-DA0B-4364-BFB3-296A7C54CC25} = {AED9C836-31E3-4F3F-8ABC-929555D3F3C4}
17521860
EndGlobalSection
17531861
GlobalSection(ExtensibilityGlobals) = postSolution
17541862
SolutionGuid = {41165AF1-35BB-4832-A189-73060F82B01D}

NuGet.config

+1-1
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
<add key="vs-buildservices" value="https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-buildservices/nuget/v3/index.json" />
1212
<add key="dotnet5-roslyn" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json" />
1313
<add key="mlnet-daily" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json" />
14-
<add key="mlnet-testdata" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/machinelearning-testdata/nuget/v3/index.json" />
14+
<add key="mlnet-assets" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/machinelearning-assets/nuget/v3/index.json" />
1515
</packageSources>
1616
<disabledPackageSources>
1717
<clear />

eng/Versions.props

+4-1
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,11 @@
3232
<TensorFlowVersion>2.3.1</TensorFlowVersion>
3333
<TensorFlowMajorVersion>2</TensorFlowMajorVersion>
3434
<TensorflowDotNETVersion>0.20.1</TensorflowDotNETVersion>
35-
3635
<MicrosoftCodeAnalysisCSharpInternalAnalyzerVersion>3.3.1</MicrosoftCodeAnalysisCSharpInternalAnalyzerVersion>
36+
<MicrosoftDotNetInteractiveVersion>1.0.0-beta.20410.1</MicrosoftDotNetInteractiveVersion>
37+
<MicrosoftDotNetInteractiveFormattingVersion>1.0.0-beta.20410.1</MicrosoftDotNetInteractiveFormattingVersion>
38+
<ApacheArrowVersion>2.0.0</ApacheArrowVersion>
39+
<SystemTextEncodingVersion>4.3.0</SystemTextEncodingVersion>
3740
<MicrosoftCSharpVersion>4.5.0</MicrosoftCSharpVersion>
3841
<SystemCompositionVersion>1.2.0</SystemCompositionVersion>
3942

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
// Licensed to the .NET Foundation under one or more agreements.
2+
// The .NET Foundation licenses this file to you under the MIT license.
3+
// See the LICENSE file in the project root for more information.
4+
5+
using System;
6+
using System.Collections.Generic;
7+
using System.Linq;
8+
using System.Threading.Tasks;
9+
using Microsoft.AspNetCore.Html;
10+
using Microsoft.DotNet.Interactive;
11+
using Microsoft.DotNet.Interactive.Formatting;
12+
using static Microsoft.DotNet.Interactive.Formatting.PocketViewTags;
13+
14+
namespace Microsoft.Data.Analysis.Interactive
15+
{
16+
public class DataFrameKernelExtension : IKernelExtension
17+
{
18+
public Task OnLoadAsync(Kernel kernel)
19+
{
20+
RegisterDataFrame();
21+
22+
return Task.CompletedTask;
23+
}
24+
25+
public static void RegisterDataFrame()
26+
{
27+
Formatter<DataFrame>.Register((df, writer) =>
28+
{
29+
const int MAX = 10000;
30+
const int SIZE = 10;
31+
32+
var uniqueId = DateTime.Now.Ticks;
33+
34+
var header = new List<IHtmlContent>
35+
{
36+
th(i("index"))
37+
};
38+
header.AddRange(df.Columns.Select(c => (IHtmlContent)th(c.Name)));
39+
40+
if (df.Rows.Count > SIZE)
41+
{
42+
var maxMessage = df.Rows.Count > MAX ? $" (showing a max of {MAX} rows)" : string.Empty;
43+
var title = h3[style: "text-align: center;"]($"DataFrame - {df.Rows.Count} rows {maxMessage}");
44+
45+
// table body
46+
var maxRows = Math.Min(MAX, df.Rows.Count);
47+
var rows = new List<List<IHtmlContent>>();
48+
for (var index = 0; index < maxRows; index++)
49+
{
50+
var cells = new List<IHtmlContent>
51+
{
52+
td(i((index)))
53+
};
54+
foreach (var obj in df.Rows[index])
55+
{
56+
cells.Add(td(obj));
57+
}
58+
rows.Add(cells);
59+
}
60+
61+
//navigator
62+
var footer = new List<IHtmlContent>();
63+
BuildHideRowsScript(uniqueId);
64+
65+
var paginateScriptFirst = BuildHideRowsScript(uniqueId) + GotoPageIndex(uniqueId, 0) + BuildPageScript(uniqueId, SIZE);
66+
footer.Add(button[style: "margin: 2px;", onclick: paginateScriptFirst]("⏮"));
67+
68+
var paginateScriptPrevTen = BuildHideRowsScript(uniqueId) + UpdatePageIndex(uniqueId, -10, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE);
69+
footer.Add(button[style: "margin: 2px;", onclick: paginateScriptPrevTen]("⏪"));
70+
71+
var paginateScriptPrev = BuildHideRowsScript(uniqueId) + UpdatePageIndex(uniqueId, -1, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE);
72+
footer.Add(button[style: "margin: 2px;", onclick: paginateScriptPrev]("◀️"));
73+
74+
footer.Add(b[style: "margin: 2px;"]("Page"));
75+
footer.Add(b[id: $"page_{uniqueId}", style: "margin: 2px;"]("1"));
76+
77+
var paginateScriptNext = BuildHideRowsScript(uniqueId) + UpdatePageIndex(uniqueId, 1, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE);
78+
footer.Add(button[style: "margin: 2px;", onclick: paginateScriptNext]("▶️"));
79+
80+
var paginateScriptNextTen = BuildHideRowsScript(uniqueId) + UpdatePageIndex(uniqueId, 10, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE);
81+
footer.Add(button[style: "margin: 2px;", onclick: paginateScriptNextTen]("⏩"));
82+
83+
var paginateScriptLast = BuildHideRowsScript(uniqueId) + GotoPageIndex(uniqueId, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE);
84+
footer.Add(button[style: "margin: 2px;", onclick: paginateScriptLast]("⏭️"));
85+
86+
//table
87+
var t = table[id: $"table_{uniqueId}"](
88+
caption(title),
89+
thead(tr(header)),
90+
tbody(rows.Select(r => tr[style: "display: none"](r))),
91+
tfoot(tr(td[colspan: df.Columns.Count + 1, style: "text-align: center;"](footer)))
92+
);
93+
writer.Write(t);
94+
95+
//show first page
96+
writer.Write($"<script>{BuildPageScript(uniqueId, SIZE)}</script>");
97+
}
98+
else
99+
{
100+
var rows = new List<List<IHtmlContent>>();
101+
for (var index = 0; index < df.Rows.Count; index++)
102+
{
103+
var cells = new List<IHtmlContent>
104+
{
105+
td(i((index)))
106+
};
107+
foreach (var obj in df.Rows[index])
108+
{
109+
cells.Add(td(obj));
110+
}
111+
rows.Add(cells);
112+
}
113+
114+
//table
115+
var t = table[id: $"table_{uniqueId}"](
116+
thead(tr(header)),
117+
tbody(rows.Select(r => tr(r)))
118+
);
119+
writer.Write(t);
120+
}
121+
}, "text/html");
122+
}
123+
124+
private static string BuildHideRowsScript(long uniqueId)
125+
{
126+
var script = $"var allRows = document.querySelectorAll('#table_{uniqueId} tbody tr:nth-child(n)'); ";
127+
script += "for (let i = 0; i < allRows.length; i++) { allRows[i].style.display='none'; } ";
128+
return script;
129+
}
130+
131+
private static string BuildPageScript(long uniqueId, int size)
132+
{
133+
var script = $"var page = parseInt(document.querySelector('#page_{uniqueId}').innerHTML) - 1; ";
134+
script += $"var pageRows = document.querySelectorAll(`#table_{uniqueId} tbody tr:nth-child(n + ${{page * {size} + 1 }})`); ";
135+
script += $"for (let j = 0; j < {size}; j++) {{ pageRows[j].style.display='table-row'; }} ";
136+
return script;
137+
}
138+
139+
private static string GotoPageIndex(long uniqueId, long page)
140+
{
141+
var script = $"document.querySelector('#page_{uniqueId}').innerHTML = {page + 1}; ";
142+
return script;
143+
}
144+
145+
private static string UpdatePageIndex(long uniqueId, int step, long maxPage)
146+
{
147+
var script = $"var page = parseInt(document.querySelector('#page_{uniqueId}').innerHTML) - 1; ";
148+
script += $"page = parseInt(page) + parseInt({step}); ";
149+
script += $"page = page < 0 ? 0 : page; ";
150+
script += $"page = page > {maxPage} ? {maxPage} : page; ";
151+
script += $"document.querySelector('#page_{uniqueId}').innerHTML = page + 1; ";
152+
return script;
153+
}
154+
}
155+
}

0 commit comments

Comments
 (0)