Merge pull request #3461 from input-output-hk/Squad1/CO-325/api-v1-improvements

[CO-325] API V1 Improvements - Part 2

Author: KtorZ

README.md

@@ -73,25 +73,53 @@ $ stack exec cardano-node -- --topology=wallet-new/topology-examples/testnet.yam
 
 From there, you can browse the API documentation for V0 and V1 through the following URLs:
 
-- http://localhost:8090/docs/v0/index/
-- http://localhost:8090/docs/v1/index/
+- https://localhost:8091/docs/v0/index/
+- https://localhost:8091/docs/v1/index/
 
-### HTTPS
 
-By default, wallet backend only accepts HTTPS connections:
+You may also run a simple cURL command to check whether the node is up-and-running:
 
 ```
-$ curl localhost:8090/docs/v1/index/
-This server only accepts secure HTTPS connections.
+$ curl https://localhost:8090/api/v1/node-info \
+   --cacert scripts/tls-files/ca.crt \
+   --cert scripts/tls-files/client.pem
 ```
 
-We should provide our `ca.crt`:
+> *NOTE* 
+>
+> Every node running a wallet API needs x509 certificates for enabling TLS support. By default,
+> those certificates are located in `./scripts/tls-files`. Use them if you need a CA or a
+> client certificate. 
+
+
+## Local Cluster
+
+Running a node against `mainnet_staging` may not be ideal for testing. The node will also need
+time to synchronize and won't run the full API capabilities until having done so. To cope with
+this, one may run a local cluster of nodes, acting upon a fresh database, speeding up most of
+the operations. To run a local cluster, _nix_ is your friend:
+
+```
+$ nix-build -A demoCluster -o run-demo --arg useStackBinaries true && ./run-demo
+```
+
+This will run a local cluster after having set up a fresh environment for it in `./state-demo`.
+There are some files of interest in this folder you may need like the tls certificates or the
+logging configurations. 
+
+
+### HTTPS
+
+By default, wallet backend only accepts HTTPS connections:
 
 ```
-$ curl --cacert scripts/tls-files/ca.crt https://localhost:8090/docs/v1/index/
+$ curl localhost:8090/api/v1/node-info
+This server only accepts secure HTTPS connections.
 ```
 
-But if we launch a node with `--wallet-debug` option, we can send simple `http`-requests.
+Read the documentation about TLS authentication in [docs/tls-authentication.md](../docs/tls-authentication.md)
+for details about how to contact a wallet node with TLS. 
+
 
 ### Swagger Specification
 
@@ -126,9 +154,33 @@ $ stack test cardano-sl-wallet-new
 Wallet integration tests can be run using this command (from the project *root* directory):
 
 ```
-$ nix-build release.nix -A walletIntegrationTests
+$ nix-build -A walletIntegrationTests --arg useStackBinaries true 
 ```
 
+> **NOTE**:
+> `nix-build -A walletIntegrationTests` (with or without `useStackBinaries`) runs a
+> local demo cluster, either via stack or nix by default on your local machine
+> that is fully usable by daedalus/curl etc...  and requires port 8090 and
+> ports 3001-3004 and 3101 to be available. This cluster has four core nodes, 1
+> relay, and a single wallet and has full x509 CA cert enabled. It then
+> pre-loads some genesis poor keys for testing and runs the wal-integr-test
+> haskell program, which connects to the running cluster. When it completes, it
+> terminates the demo cluster and wallet. This will fail if ports aren't
+> available to bind (although cardano-node will happily run without crashing,
+> it just will be broken), you try running two of these at once, etc...
+>
+> This is differentiated from `nix-build -A tests.walletIntegration` which **DOES
+> NOT** support `useStackBinaries` and builds/runs the entire cluster in a sandbox
+> isolated from the rest of the system (assuming nix sandboxing is enabled).
+> This is how hydra runs the tests and why hydra is capable or running more
+> than one cluster at the same time. This will use any binaries cached by hydra
+> if you have the IOHK binary cache enabled, or will build everything cleanly
+> in nix if the binaries aren't available in the local nix store. One other
+> thing to note is that tests.walletIntegration will only run once and will
+> cache the results (unless of a failure). If you have a need to rerun the
+> test, you can pass the `--check` flag to force the test to run again. `--check`
+> is used to confirm that results from one test match the results again.
+
 ## Developing
 
 We have a [`Makefile`](./Makefile) with some helpful commands for development.
@@ -152,7 +204,8 @@ Now use following command (from the `cardano-sl` *root* directory):
 $ curl -X POST                                  \
        -H "Content-Type: application/json"      \
        -d '"PATH_TO_SECRET_KEY"'                \
-       --cacert scripts/tls-files/ca.crt
+       --cacert scripts/tls-files/ca.crt        \
+       --cert scripts/tls-files/client.pem      \
        https://localhost:8090/api/wallets/keys
 ```
 
@@ -217,6 +270,7 @@ using environment variables as follows:
 LANG=en_GB.UTF-8 LC_ALL=en_GB.UTF-8 stack exec -- ...
 ```
 
+
 ##### API returns `415  Unsupported Media Type`
 
 The wallet's API can be quite picky about media-types and expect both a given type and an
@@ -232,3 +286,10 @@ value:
 ```
 application/json;charset=utf-8
 ```
+
+
+##### API returns `error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown`
+
+You're missing a valid client certificate to contact the node. For development, you may run the
+node with `--no-client-auth` or provide a valid corresponding client x509 certificates. More
+information in [docs/tls-authentication.md](../docs/tls-authentication.md).

cardano-sl-wallet-new.cabal

@@ -129,6 +129,7 @@ library
                        Cardano.Wallet.Server.CLI
                        Cardano.Wallet.Server.Plugins
                        Cardano.Wallet.TypeLits
+                       Cardano.Wallet.Types.UtxoStatistics
                        Cardano.Wallet.Client
                        Cardano.Wallet.Client.Http
 
@@ -175,7 +176,6 @@ library
                      , cardano-sl-node-ipc
                      , cardano-sl-util
                      , cardano-sl-wallet
-                     , cardano-sl-wallet-test
                      , cereal
                      , conduit
                      , connection
@@ -194,6 +194,7 @@ library
                      , http-client-tls
                      , http-types
                      , ixset-typed
+                     , foldl
                      , lens
                      , log-warper
                      , memory
@@ -207,6 +208,7 @@ library
                      , reflection
                      , resourcet
                      , retry
+                     , stm
                      , safecopy
                      , safe-exceptions
                      , serokell-util
@@ -354,6 +356,7 @@ executable wal-integr-test
                      , aeson-diff
                      , aeson-pretty
                      , bytestring
+                     , cardano-sl
                      , cardano-sl-core
                      , cardano-sl-wallet
                      , cardano-sl-wallet-new
@@ -502,7 +505,6 @@ test-suite wallet-unit-tests
                     , data-default
                     , formatting
                     , hspec
-                    , ixset-typed
                     , lens
                     , log-warper
                     , mtl
@@ -640,7 +642,6 @@ benchmark cardano-sl-wallet-new-bench
                      , bytestring
                      , cardano-sl-client
                      , cardano-sl-core
-                     , cardano-sl-db
                      , cardano-sl-wallet
                      , cassava
                      , connection

integration/QuickCheckSpecs.hs

@@ -6,6 +6,7 @@ module QuickCheckSpecs (mkSpec) where
 
 import           Universum
 
+import           GHC.TypeLits (KnownSymbol)
 import           Network.HTTP.Client hiding (Proxy)
 import           Network.HTTP.Types
 import           Servant
@@ -21,6 +22,7 @@ import qualified Cardano.Wallet.API.V1 as V0
 import qualified Cardano.Wallet.API.V1 as V1
 import           Cardano.Wallet.API.V1.Parameters (WalletRequestParams,
                      WithWalletRequestParams)
+import           Pos.Util.Servant (CustomQueryFlag)
 
 -- Our API apparently is returning JSON Arrays which is considered bad practice as very old
 -- browsers can be hacked: https://haacked.com/archive/2009/06/25/json-hijacking.aspx/
@@ -72,6 +74,12 @@ instance HasGenRequest sub => HasGenRequest (Tags tags :> sub) where
 instance HasGenRequest (sub :: *) => HasGenRequest (WalletRequestParams :> sub) where
     genRequest _ = genRequest (Proxy @(WithWalletRequestParams sub))
 
+instance ( KnownSymbol sym
+         , HasGenRequest sub
+         ) =>
+         HasGenRequest (CustomQueryFlag sym flag :> sub) where
+    genRequest _ = genRequest (Proxy @(QueryFlag sym :> sub))
+
 --
 -- RESTful-abiding predicates
 --

integration/TransactionSpecs.hs

@@ -11,9 +11,12 @@ import           Control.Lens
 import           Test.Hspec
 import           Text.Show.Pretty (ppShow)
 
+import           Util
+
+import qualified Data.Map.Strict as Map
 import qualified Pos.Core as Core
+import qualified Pos.Core.Txp as Txp
 
-import           Util
 
 {-# ANN module ("HLint: ignore Reduce duplication" :: Text) #-}
 
@@ -24,7 +27,7 @@ ppShowT :: Show a => a -> Text
 ppShowT = fromString . ppShow
 
 transactionSpecs :: WalletRef -> WalletClient IO -> Spec
-transactionSpecs wRef wc = do
+transactionSpecs wRef wc =
     describe "Transactions" $ do
         it "posted transactions appear in the index" $ do
             genesis <- genesisWallet wc
@@ -187,3 +190,43 @@ transactionSpecs wRef wc = do
             etxn <- postTransaction wc payment
 
             void $ etxn `shouldPrism` _Left
+
+        xit "posted transactions gives rise to nonempty Utxo histogram" $ do
+            genesis <- genesisWallet wc
+            (fromAcct, _) <- firstAccountAndId wc genesis
+
+            wallet <- sampleWallet wRef wc
+            (_, toAddr) <- firstAccountAndId wc wallet
+
+            let payment val = Payment
+                    { pmtSource =  PaymentSource
+                        { psWalletId = walId genesis
+                        , psAccountIndex = accIndex fromAcct
+                        }
+                    , pmtDestinations = pure PaymentDistribution
+                        { pdAddress = addrId toAddr
+                        , pdAmount = V1 (Core.mkCoin val)
+                        }
+                    , pmtGroupingPolicy = Nothing
+                    , pmtSpendingPassword = Nothing
+                    }
+
+            eresp0 <- getUtxoStatistics wc (walId wallet)
+            utxoStatistics0 <- fmap wrData eresp0 `shouldPrism` _Right
+            let utxoStatistics0Expected = computeUtxoStatistics log10 []
+            utxoStatistics0 `shouldBe` utxoStatistics0Expected
+
+            void $ postTransaction wc (payment 1)
+            threadDelay 120000000
+
+            let txIn  = Txp.TxInUnknown 0 "test"
+            let txOut = Txp.TxOutAux Txp.TxOut
+                    { Txp.txOutAddress = unV1 (addrId toAddr)
+                    , Txp.txOutValue = Core.mkCoin 1
+                    }
+            let utxos = [Map.fromList [(txIn, txOut)]]
+
+            eresp <- getUtxoStatistics wc (walId wallet)
+            utxoStatistics <- fmap wrData eresp `shouldPrism` _Right
+            let utxoStatisticsExpected = computeUtxoStatistics log10 utxos
+            utxoStatistics `shouldBe` utxoStatisticsExpected

integration/WalletSpecs.hs

@@ -13,7 +13,7 @@ import           Util
 
 
 walletSpecs :: WalletRef -> WalletClient IO -> Spec
-walletSpecs _ wc = do
+walletSpecs _ wc =
     describe "Wallets" $ do
         it "Creating a wallet makes it available." $ do
             newWallet <- randomWallet CreateWallet
@@ -51,6 +51,15 @@ walletSpecs _ wc = do
                 }
 
             eresp `shouldPrism_` _Right
+
+        it "creating wallet gives rise to an empty Utxo histogram" $ do
+            newWallet <- randomWallet CreateWallet
+            wallet <- createWalletCheck wc newWallet
+
+            eresp <- getUtxoStatistics wc (walId wallet)
+            utxoStatistics <- fmap wrData eresp `shouldPrism` _Right
+            let utxoStatisticsExpected = computeUtxoStatistics log10 []
+            utxoStatistics `shouldBe` utxoStatisticsExpected
   where
     testWalletAlreadyExists action = do
             newWallet1 <- randomWallet action

src/Cardano/Wallet/API/V1/Handlers/Info.hs

@@ -6,12 +6,16 @@ import           Servant
 
 import           Cardano.Wallet.API.Response (WalletResponse, single)
 import qualified Cardano.Wallet.API.V1.Info as Info
-import           Cardano.Wallet.API.V1.Types (NodeInfo)
+import           Cardano.Wallet.API.V1.Types (ForceNtpCheck, NodeInfo)
 import           Cardano.Wallet.WalletLayer (ActiveWalletLayer)
 import qualified Cardano.Wallet.WalletLayer as WalletLayer
 
 handlers :: ActiveWalletLayer IO -> ServerT Info.API Handler
 handlers = getNodeInfo
 
-getNodeInfo :: ActiveWalletLayer IO -> Handler (WalletResponse NodeInfo)
-getNodeInfo w = liftIO $ single <$> WalletLayer.getNodeInfo w
+getNodeInfo
+    :: ActiveWalletLayer IO
+    -> ForceNtpCheck
+    -> Handler (WalletResponse NodeInfo)
+getNodeInfo w forceNtp =
+    liftIO $ single <$> WalletLayer.getNodeInfo w forceNtp

src/Cardano/Wallet/API/V1/Handlers/Wallets.hs

@@ -2,17 +2,17 @@ module Cardano.Wallet.API.V1.Handlers.Wallets where
 
 import           Universum
 
-import           Pos.Core (Coin)
+import           Servant
 
 import           Cardano.Wallet.API.Request
 import           Cardano.Wallet.API.Response
 import           Cardano.Wallet.API.V1.Types as V1
 import qualified Cardano.Wallet.API.V1.Wallets as Wallets
-
 import           Cardano.Wallet.WalletLayer (PassiveWalletLayer)
 import qualified Cardano.Wallet.WalletLayer as WalletLayer
 
-import           Servant
+import           Pos.Core.Common (Coin (..))
+
 
 -- | All the @Servant@ handlers for wallet-specific operations.
 handlers :: PassiveWalletLayer IO -> ServerT Wallets.API Handler
@@ -22,6 +22,7 @@ handlers pwl =  newWallet pwl
            :<|> deleteWallet pwl
            :<|> getWallet pwl
            :<|> updateWallet pwl
+           :<|> getUtxoStatistics pwl
            :<|> checkExternalWallet pwl
            :<|> newExternalWallet pwl
            :<|> deleteExternalWallet pwl
@@ -96,6 +97,17 @@ updateWallet pwl wid walletUpdateRequest = do
          Left e  -> throwM e
          Right w -> return $ single w
 
+getUtxoStatistics
+    :: PassiveWalletLayer IO
+    -> WalletId
+    -> Handler (WalletResponse UtxoStatistics)
+getUtxoStatistics pwl wid = do
+    res <- liftIO $ WalletLayer.getUtxos pwl wid
+    case res of
+         Left e  -> throwM e
+         Right w ->
+            return $ single $ V1.computeUtxoStatistics V1.log10 (map snd w)
+
 checkExternalWallet :: PassiveWalletLayer IO
                     -> PublicKeyAsBase58
                     -> Handler (WalletResponse WalletAndTxHistory)

src/Cardano/Wallet/API/V1/Info.hs

@@ -4,9 +4,11 @@ import           Cardano.Wallet.API.Response (ValidJSON, WalletResponse)
 import           Cardano.Wallet.API.Types
 import           Cardano.Wallet.API.V1.Types
 
+import           Pos.Util.Servant (CustomQueryFlag)
 import           Servant
 
 type API = Tags '["Info"] :>
-         ( "node-info" :> Summary "Retrieves the dynamic information for this node."
-                       :> Get '[ValidJSON] (WalletResponse NodeInfo)
+         (    "node-info" :> Summary "Retrieves the dynamic information for this node."
+                          :> CustomQueryFlag "force_ntp_check" ForceNtpCheck
+                          :> Get '[ValidJSON] (WalletResponse NodeInfo)
          )