Skip to content

Commit 46e5908

Browse files
szeigerszeiger
committed
Initial version of parallel collections from the Scala standard library
Moved over from scala/scala @ 36d6e48da258648413e50e5a9d46c94cfc062634 with the necessary adaptations to make it work as a separate module on top of scala/scala#5603. - Since the sequential collections cannot implement `Parallelizable` anymore, all conversions have to be performed via extension methods provided by `scala.collection.parallel.CollectionConverters._` - `ParIterableLike.sequentially` was changed to use a `newCombiner` for building the parallel version from the temporary sequential collection. - `Combiner` has a new convenience method `fromSequential`. `par` in `Parallelizable` is now implemented on top of this. The build-related parts were copied from https://github.com/scala/scala-swing.
0 parents  commit 46e5908

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+8781
-0
lines changed

‎.gitignore

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#
2+
# Are you tempted to edit this file?
3+
#
4+
# First consider if the changes make sense for all,
5+
# or if they are specific to your workflow/system.
6+
# If it is the latter, you can augment this list with
7+
# entries in .git/info/excludes
8+
#
9+
10+
*.jar
11+
*~
12+
13+
# eclipse, intellij
14+
/.classpath
15+
/.project
16+
/.cache
17+
/.idea
18+
/.settings
19+
20+
# bak files produced by ./cleanup-commit
21+
*.bak
22+
23+
# Mac specific, but that is common enough a dev platform to warrant inclusion.
24+
.DS_Store
25+
26+
target/

‎.travis.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
language: scala
2+
3+
env:
4+
global:
5+
- PUBLISH_JDK=oraclejdk8
6+
7+
script: admin/build.sh
8+
9+
jdk: oraclejdk8

‎LICENSE

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
Copyright (c) 2002-2016 EPFL
2+
Copyright (c) 2011-2016 Lightbend, Inc.
3+
4+
All rights reserved.
5+
6+
Redistribution and use in source and binary forms, with or without modification,
7+
are permitted provided that the following conditions are met:
8+
9+
* Redistributions of source code must retain the above copyright notice,
10+
this list of conditions and the following disclaimer.
11+
* Redistributions in binary form must reproduce the above copyright notice,
12+
this list of conditions and the following disclaimer in the documentation
13+
and/or other materials provided with the distribution.
14+
* Neither the name of the EPFL nor the names of its contributors
15+
may be used to endorse or promote products derived from this software
16+
without specific prior written permission.
17+
18+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
19+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
20+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
21+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
22+
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
23+
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
24+
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
25+
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
26+
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
27+
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
28+
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Scala Parallel Collections
2+
3+
This Scala standard library module contains the package `scala.collection.parallel` with all parallel collections that
4+
used to be part of `scala-library` before Scala 2.13.

‎admin/README.md

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
## Tag Driven Releasing
2+
3+
Copied from https://github.com/scala/scala-java8-compat/commit/4a6cfc97cd95227b86650410e1b632e5ff79335b.
4+
5+
### Background Reading
6+
7+
- http://docs.travis-ci.com/user/environment-variables/
8+
- http://docs.travis-ci.com/user/encryption-keys/
9+
- http://docs.travis-ci.com/user/encrypting-files/
10+
11+
### Initial setup for the repository
12+
13+
To configure tag driven releases from Travis CI.
14+
15+
1. Generate a key pair for this repository with `./admin/genKeyPair.sh`.
16+
Edit `.travis.yml` and `admin/build.sh` as prompted.
17+
2. Publish the public key to https://pgp.mit.edu
18+
3. Store other secrets as encrypted environment variables with `admin/encryptEnvVars.sh`.
19+
Edit `.travis.yml` as prompted.
20+
4. Edit `.travis.yml` to use `./admin/build.sh` as the build script,
21+
and edit that script to use the tasks required for this project.
22+
5. Edit `.travis.yml` to select which JDK will be used for publishing.
23+
24+
It is important to add comments in .travis.yml to identify the name
25+
of each environment variable encoded in a `:secure` section.
26+
27+
After all of these steps, your .travis.yml should contain config of the
28+
form:
29+
30+
language: scala
31+
env:
32+
global:
33+
- PUBLISH_JDK=openjdk6
34+
# PGP_PASSPHRASE
35+
- secure: "XXXXXX"
36+
# SONA_USER
37+
- secure: "XXXXXX"
38+
# SONA_PASS
39+
- secure: "XXXXXX"
40+
script: admin/build.sh
41+
42+
If Sonatype credentials change in the future, step 3 can be repeated
43+
without generating a new key.
44+
45+
Be sure to use SBT 0.13.7 or higher to avoid [#1430](https://github.com/sbt/sbt/issues/1430)!
46+
47+
### Testing
48+
49+
1. Follow the release process below to create a dummy release (e.g. 0.1.0-TEST1).
50+
Confirm that the release was staged to Sonatype but do not release it to Maven
51+
central. Instead, drop the staging repository.
52+
53+
### Performing a release
54+
55+
1. Create a GitHub "Release" (with a corresponding tag) via the GitHub
56+
web interface.
57+
2. Travis CI will schedule a build for this release. Review the build logs.
58+
3. Log into https://oss.sonatype.org/ and identify the staging repository.
59+
4. Sanity check its contents
60+
5. Release staging repository to Maven and send out release announcement.
61+

‎admin/build.sh

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/bin/bash
2+
3+
set -e
4+
5+
# prep environment for publish to sonatype staging if the HEAD commit is tagged
6+
7+
# git on travis does not fetch tags, but we have TRAVIS_TAG
8+
# headTag=$(git describe --exact-match ||:)
9+
10+
if [ "$TRAVIS_JDK_VERSION" == "$PUBLISH_JDK" ] && [[ "$TRAVIS_TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[A-Za-z0-9-]+)? ]]; then
11+
echo "Going to release from tag $TRAVIS_TAG!"
12+
myVer=$(echo $TRAVIS_TAG | sed -e s/^v//)
13+
publishVersion='set every version := "'$myVer'"'
14+
extraTarget="+publish-signed"
15+
cat admin/gpg.sbt >> project/plugins.sbt
16+
cp admin/publish-settings.sbt .
17+
18+
# Copied from the output of genKeyPair.sh
19+
K=$encrypted_e923b9d88d53_key
20+
IV=$encrypted_e923b9d88d53_iv
21+
22+
openssl aes-256-cbc -K $K -iv $IV -in admin/secring.asc.enc -out admin/secring.asc -d
23+
fi
24+
25+
sbt "$publishVersion" clean update +test +publishLocal $extraTarget

‎admin/encryptEnvVars.sh

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
#
3+
# Encrypt sonatype credentials so that they can be
4+
# decrypted in trusted builds on Travis CI.
5+
#
6+
set -e
7+
8+
read -s -p 'SONA_USER: ' SONA_USER
9+
travis encrypt SONA_USER="$SONA_USER"
10+
read -s -p 'SONA_PASS: ' SONA_PASS
11+
travis encrypt SONA_PASS="$SONA_PASS"

‎admin/genKeyPair.sh

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/bin/bash
2+
#
3+
# Generates a key pair for this repository to sign artifacts.
4+
# Encrypt the private key and its passphrase in trusted builds
5+
# on Travis CI.
6+
#
7+
set -e
8+
9+
# Based on https://gist.github.com/kzap/5819745:
10+
function promptDelete() {
11+
if [[ -f "$1" ]]; then
12+
echo About to delete $1, Enter for okay / CTRL-C to cancel
13+
read
14+
rm "$1"
15+
fi
16+
}
17+
for f in admin/secring.asc.enc admin/secring.asc admin/pubring.asc; do promptDelete "$f"; done
18+
19+
echo Generating key pair. Please enter 1. repo name 2. [email protected], 3. a new passphrase
20+
echo Be careful when using special characters in the passphrase, see http://docs.travis-ci.com/user/encryption-keys/#Note-on-escaping-certain-symbols
21+
cp admin/gpg.sbt project
22+
sbt 'set pgpReadOnly := false' \
23+
'set pgpPublicRing := file("admin/pubring.asc")' \
24+
'set pgpSecretRing := file("admin/secring.asc")' \
25+
'pgp-cmd gen-key'
26+
rm project/gpg.sbt
27+
28+
echo ============================================================================================
29+
echo Encrypting admin/secring.asc. Update K and IV variables in admin/build.sh accordingly.
30+
echo ============================================================================================
31+
travis encrypt-file admin/secring.asc
32+
rm admin/secring.asc
33+
mv secring.asc.enc admin
34+
35+
echo ============================================================================================
36+
echo Encrypting environment variables. Add each to a line in .travis.yml. Include a comment
37+
echo with the name of the corresponding variable
38+
echo ============================================================================================
39+
read -s -p 'PGP_PASSPHRASE: ' PGP_PASSPHRASE
40+
travis encrypt PGP_PASSPHRASE="$PGP_PASSPHRASE"
41+

‎admin/gpg.sbt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
2+
addSbtPlugin("com.typesafe.sbt" % "sbt-pgp" % "0.8.3") // only added when publishing, see build.sh

‎admin/publish-settings.sbt

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
def env(key: String) = Option(System.getenv(key)).getOrElse("")
2+
3+
pgpPassphrase := Some(env("PGP_PASSPHRASE").toArray)
4+
5+
pgpPublicRing := file("admin/pubring.asc")
6+
7+
pgpSecretRing := file("admin/secring.asc")
8+
9+
credentials += Credentials("Sonatype Nexus Repository Manager", "oss.sonatype.org", env("SONA_USER"), env("SONA_PASS"))

‎build.sbt

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
import com.typesafe.tools.mima.plugin.{MimaPlugin, MimaKeys}
2+
3+
scalaModuleSettings
4+
5+
name := "scala-parallel-collections"
6+
7+
version := "1.0.0-SNAPSHOT"
8+
9+
scalaVersion := crossScalaVersions.value.head
10+
11+
resolvers += "scala-pr" at "https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/"
12+
13+
crossScalaVersions := Seq("2.13.0-SNAPSHOT")
14+
15+
scalacOptions ++= Seq("-deprecation", "-feature")
16+
17+
// important!! must come here (why?)
18+
scalaModuleOsgiSettings
19+
20+
OsgiKeys.exportPackage := Seq(s"scala.collection.parallel.*;version=${version.value}")
21+
22+
mimaPreviousVersion := None
23+
24+
lazy val root = project.in( file(".") )
25+
26+
fork in Test := true
27+
28+
libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test"
29+
30+
testOptions += Tests.Argument(TestFrameworks.JUnit, "-a", "-v")

‎project/build.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
sbt.version=0.13.8

‎project/plugins.sbt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
addSbtPlugin("org.scala-lang.modules" % "scala-module-plugin" % "1.0.3")

‎src/main/scala/scala/collection/CustomParallelizable.scala

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
/* __ *\
2+
** ________ ___ / / ___ Scala API **
3+
** / __/ __// _ | / / / _ | (c) 2003-2013, LAMP/EPFL **
4+
** __\ \/ /__/ __ |/ /__/ __ | http://scala-lang.org/ **
5+
** /____/\___/_/ |_/____/_/ | | **
6+
** |/ **
7+
\* */
8+
9+
package scala
10+
package collection
11+
12+
import parallel.Combiner
13+
14+
trait CustomParallelizable[+A, +ParRepr <: Parallel] extends Any with Parallelizable[A, ParRepr] {
15+
override def par: ParRepr
16+
override protected[this] def parCombiner: Combiner[A, ParRepr] = throw new UnsupportedOperationException("")
17+
}
18+

‎src/main/scala/scala/collection/Parallelizable.scala

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
/* __ *\
2+
** ________ ___ / / ___ Scala API **
3+
** / __/ __// _ | / / / _ | (c) 2003-2013, LAMP/EPFL **
4+
** __\ \/ /__/ __ |/ /__/ __ | http://scala-lang.org/ **
5+
** /____/\___/_/ |_/____/_/ | | **
6+
** |/ **
7+
\* */
8+
9+
package scala
10+
package collection
11+
12+
import parallel.Combiner
13+
14+
/** This trait describes collections which can be turned into parallel collections
15+
* by invoking the method `par`. Parallelizable collections may be parameterized with
16+
* a target type different than their own.
17+
*
18+
* @tparam A the type of the elements in the collection
19+
* @tparam ParRepr the actual type of the collection, which has to be parallel
20+
*/
21+
trait Parallelizable[+A, +ParRepr <: Parallel] extends Any {
22+
23+
def seq: TraversableOnce[A]
24+
25+
/** Returns a parallel implementation of this collection.
26+
*
27+
* For most collection types, this method creates a new parallel collection by copying
28+
* all the elements. For these collection, `par` takes linear time. Mutable collections
29+
* in this category do not produce a mutable parallel collection that has the same
30+
* underlying dataset, so changes in one collection will not be reflected in the other one.
31+
*
32+
* Specific collections (e.g. `ParArray` or `mutable.ParHashMap`) override this default
33+
* behaviour by creating a parallel collection which shares the same underlying dataset.
34+
* For these collections, `par` takes constant or sublinear time.
35+
*
36+
* All parallel collections return a reference to themselves.
37+
*
38+
* @return a parallel implementation of this collection
39+
*/
40+
def par: ParRepr = parCombiner.fromSequential(seq)
41+
42+
/** The default `par` implementation uses the combiner provided by this method
43+
* to create a new parallel collection.
44+
*
45+
* @return a combiner for the parallel collection of type `ParRepr`
46+
*/
47+
protected[this] def parCombiner: Combiner[A, ParRepr]
48+
}
49+

‎src/main/scala/scala/collection/generic/CanCombineFrom.scala

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/* __ *\
2+
** ________ ___ / / ___ Scala API **
3+
** / __/ __// _ | / / / _ | (c) 2010-2013, LAMP/EPFL **
4+
** __\ \/ /__/ __ |/ /__/ __ | http://scala-lang.org/ **
5+
** /____/\___/_/ |_/____/_/ | | **
6+
** |/ **
7+
\* */
8+
9+
package scala
10+
package collection
11+
package generic
12+
13+
import scala.collection.parallel._
14+
15+
/** A base trait for parallel builder factories.
16+
*
17+
* @tparam From the type of the underlying collection that requests a
18+
* builder to be created.
19+
* @tparam Elem the element type of the collection to be created.
20+
* @tparam To the type of the collection to be created.
21+
* @since 2.8
22+
*/
23+
trait CanCombineFrom[-From, -Elem, +To] extends CanBuildFrom[From, Elem, To] with Parallel {
24+
def apply(from: From): Combiner[Elem, To]
25+
def apply(): Combiner[Elem, To]
26+
}
27+

‎src/main/scala/scala/collection/generic/GenericParCompanion.scala

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
/* __ *\
2+
** ________ ___ / / ___ Scala API **
3+
** / __/ __// _ | / / / _ | (c) 2010-2013, LAMP/EPFL **
4+
** __\ \/ /__/ __ |/ /__/ __ | http://scala-lang.org/ **
5+
** /____/\___/_/ |_/____/_/ | | **
6+
** |/ **
7+
\* */
8+
9+
package scala
10+
package collection
11+
package generic
12+
13+
import scala.collection.parallel.Combiner
14+
import scala.collection.parallel.ParIterable
15+
import scala.collection.parallel.ParMap
16+
import scala.language.higherKinds
17+
18+
/** A template class for companion objects of parallel collection classes.
19+
* They should be mixed in together with `GenericCompanion` type.
20+
*
21+
* @define Coll `ParIterable`
22+
* @tparam CC the type constructor representing the collection class
23+
* @since 2.8
24+
*/
25+
trait GenericParCompanion[+CC[X] <: ParIterable[X]] {
26+
/** The default builder for $Coll objects.
27+
*/
28+
def newBuilder[A]: Combiner[A, CC[A]]
29+
30+
/** The parallel builder for $Coll objects.
31+
*/
32+
def newCombiner[A]: Combiner[A, CC[A]]
33+
}
34+
35+
trait GenericParMapCompanion[+CC[P, Q] <: ParMap[P, Q]] {
36+
def newCombiner[P, Q]: Combiner[(P, Q), CC[P, Q]]
37+
}
38+

0 commit comments

Comments
 (0)