break apart illinois-common-resources #186

cogcomp-dev · 2016-09-15T15:34:56Z

...into several smaller pieces. Gazetteers can go in one; brown clusters (and maybe other clusters) in another. This may help reduce the problems with large dependencies in CI.

cowchipkid · 2016-09-15T16:41:30Z

I would suggest anything approaching 100M within this jar file warrants it's own jar file.

mssammon · 2016-09-19T16:59:59Z

There's already a version just for SRL; please check and see if this overlaps with the NER when deciding how to break down the common-resources jar. The SRL resources jar itself may merit further segmentation. Before this ticket is closed (after testing NER -- including running the training/evaluation) please open a ticket on the illinois-srl project to migrate to the new jars, if the corresponding resources have been affected.

shatu · 2016-09-28T23:34:42Z

NER uses an older version of illinois-common-resources (1.3). So, which one should I break apart -- the one that NER uses, or the latest one?

Also, what should be the right approach :-

Should we keep the current illinois-common-resources as it is and create new jars containing portions of it? Or,
Should we modify the current illinois-common-resources to contain only a portion of its current members and move the rest to separate jars. -- This will probably work only if nothing else depends on common-resources. Or, we'll have to open issue in all such projects to update their dependencies correspondingly.

shatu · 2016-09-29T00:30:17Z

BTW, I can also see a illinois-common-resources-1.5-ner jar in the repo .. which is around 200M .. any idea what is that being used for?

mssammon · 2016-09-29T01:29:13Z

I think that was me trying to generate an ner-only jar.

shatu · 2016-09-29T07:38:17Z

Okay, so can someone familiar with the following resources, suggest me a logical partitioning, along with the suggested names of the new jars :-

Size -- Name (File/Dir)
64K -- levin-verbClass.txt (File)
504K -- verb-lemDict.txt (File)
12M -- cbcData (Dir)

145M -- CORLEX (Dir)

91M -- WordEmbedding (Dir)

2.6M -- rogetThesaurus (Dir)
97M -- gazetteers (Dir)

144M -- lin-clusters (Dir)
57M -- brown-clusters (Dir)

217M -- lin-similarity (Dir)

Also, kindly address the queries posed in the previous comment.

Thanks,
Shashank

cowchipkid · 2016-09-29T13:47:06Z

Definately used the 1.5 version, it includes an updated gazetteer.

I can only speak to a few of resources, but these are the ones I know are used by NER, although there may be others. The brown-clusters and gazetteers are well named and these are used in NER. I would suggest for these two datums, you split them each into their own jar files, I suspect there are cases where each are used independently. I would name them gazetteers.jar and brown-clusters.jar.

I would suggest you retain the structure within the jar to minimize code changes. In that way, all we should have to change is the pom.xml files…

danyaljj · 2016-09-29T18:22:04Z

Briefly repeating Shashank, here is what the file contains:

My suggestion is one jar file per resource. For example I would put brownCluster folder inside a jar artifact with the same name (and retain the current version 1.3).

danyaljj · 2016-09-29T19:39:30Z

Something to notice though: the latest version number on m2repo is 1.5, not 1.3 (which is currently being used inside the Edison code).
Also there three files:

illinois-common-resources-1.5-illinoisSRL.jar
illinois-common-resources-1.5-ner.jar
illinois-common-resources-1.5.jar

which seem to have some commonality. Here is what I see inside each:

danyaljj · 2016-09-29T19:43:45Z

I just realized that there used to resource-based separation of resources. Some of them existing resources are already packaged here.. Could potentially save some time.

cowchipkid · 2016-10-04T15:15:07Z

@shatu where do we stand on this? I am again in a holding pattern waiting for something, and I badly want to get the NER release wrapped up. Can I suggest we do as @danyaljj suggests and put everything in it's own separate jar and release them all. As long as we don't trash the existing illinois-common-resources, everything should still work, but we should create tickets for all the projects needing them to convert to these new resources. Is there a way to deprecate a mvn artifact?

shatu · 2016-10-04T16:30:03Z

Sure, I'll finish it off by the end of the day; I was waiting for some other proposals, but I think we should now go ahead with what you suggested -- Sorry for the delay.

I'm not sure about deprecating a mvn artifact; in fact am not even sure about its relevance to the current thread.

cowchipkid · 2016-10-04T16:48:00Z

@shatu I may not be using the right words here, I am not even close to mvn expert. The illinois-common-resources is what I am referring to, I am just wondering if there is a way to deprecate a resource that can be referenced as a dependency in a pom file. That would be very useful say if you encounter a major bug in a software library or something.

shatu · 2016-10-05T13:48:55Z

Okay, one last question ... what do we want the artifactIds to be .... "gazetteers" or as Daniel suggested above "resources.gazetteers"?

danyaljj · 2016-10-05T13:55:02Z

I like "gazetteers" more as the artifactId. "resources" can/should be part of the group id.

cowchipkid · 2016-10-05T14:57:06Z

what @danyaljj said

shatu · 2016-10-05T15:23:04Z

I'm done breaking apart the common-resources. For the time-being, I've kept the names of the jars as they were inside the common-resources, and have deployed them on the repo. Will it make sense to do something similar for cogcomp-resources jar as well (or is that already deprecated?)?

I'm not sure what all things from the common-resources, the NER depends upon, so I'm not sure what all dependencies to include in place of common-resources. Can someone familiar with NER help me with that? Also, what all tests to run in order to close this ticket?

danyaljj · 2016-10-05T15:30:18Z

Will it make sense to do something similar for cogcomp-resources jar as well?

Nah I think it's deprecated. We probably should remove these resources from m2repo.

I'm not sure what all things from the common-resources, the NER depends upon, so I'm not sure what all dependencies to include in place of common-resources. Can someone familiar with NER help me with that?

why not doing the greedy approach? I'd first start by Edison, drop the common-resources and add the necessary resources. As far as I know, we always load the resource with IOUtils.lsResources. So if you search for this keyword, you will see what resource is used where....

Also, what all tests to run in order to close this ticket?

I think we have to replace all definitions of common-resources in the pom files, and make sure all tests pass.

cowchipkid · 2016-10-05T15:40:44Z

Awesome! I will integrate this with NER as part of the NER release (ere-reader) fork. It's really just a matter of making sure I get everything I need.

shatu · 2016-10-05T23:12:07Z

@danyaljj .. I added all the dependencies separately to Edison's pom file, and am getting the following error :-

testGazetteerFeatures(edu.illinois.cs.cogcomp.edison.features.factory.TestWordFeatureFactory)
java.lang.RuntimeException: java.io.EOFException at edu.illinois.cs.cogcomp.edison.annotators.GazetteerViewGenerator.addView(GazetteerViewGenerator.java:402)
at edu.illinois.cs.cogcomp.annotation.Annotator.lazyAddView(Annotator.java:176)
at edu.illinois.cs.cogcomp.annotation.Annotator.getView(Annotator.java:161)
....

Does this has to do something with #146 ?

Is it possible to confirm whether it's a edison only issue or not? Is there any other project that uses gazetteers and is easy to test?

shatu · 2016-10-05T23:20:57Z

BTW, if anyone was using the 1.3 version of common-resources, I've deployed the corresponding (broken-apart) individual jars for that as well.

danyaljj · 2016-10-05T23:45:27Z

Your stack trace is cut. I don't completely know what is causing it, but I don't think it related to anything else. We are just replacing the containers of the resources. Everything should work the same, if the foldering structure is the same.

I would look more carefully into the stack trace...

shatu · 2016-10-06T00:22:39Z

Okay, here's what you need to replace common-resources with ..

    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>brown-clusters</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>cbc-clusters</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>corlex</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>gazetteers</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>levin-verbClass</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>lin-clusters</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>lin-similarity</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>rogetThesaurus</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>verb-lemDict</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>edu.illinois.cs.cogcomp.resources</groupId>
        <artifactId>wordEmbedding</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>

shatu · 2016-10-06T00:24:20Z

Also, here's the full stacktrace :-

testGazetteerFeatures(edu.illinois.cs.cogcomp.edison.features.factory.TestWordFeatureFactory) Time elapsed: 0.013 sec <<< ERROR!
java.lang.RuntimeException: java.io.EOFException
at edu.illinois.cs.cogcomp.edison.annotators.GazetteerViewGenerator.addView(GazetteerViewGenerator.java:402)
at edu.illinois.cs.cogcomp.annotation.Annotator.lazyAddView(Annotator.java:176)
at edu.illinois.cs.cogcomp.annotation.Annotator.getView(Annotator.java:161)
at edu.illinois.cs.cogcomp.core.datastructures.textannotation.TextAnnotation.addView(TextAnnotation.java:111)
at edu.illinois.cs.cogcomp.edison.features.factory.WordFeatureExtractorFactory$12.getWordFeatures(WordFeatureExtractorFactory.java:423)
at edu.illinois.cs.cogcomp.edison.features.factory.TestWordFeatureFactory.runTest(TestWordFeatureFactory.java:209)
at edu.illinois.cs.cogcomp.edison.features.factory.TestWordFeatureFactory.testGazetteerFeatures(TestWordFeatureFactory.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:81)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164)
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79)
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91)
at edu.illinois.cs.cogcomp.edison.annotators.GazetteerViewGenerator.lazyLoadGazetteers(GazetteerViewGenerator.java:319)
at edu.illinois.cs.cogcomp.edison.annotators.GazetteerViewGenerator.addView(GazetteerViewGenerator.java:400)
... 31 more

shatu · 2016-10-06T00:26:09Z

I manually downloaded the jars, decompressed them and did a diff with the corresponding resources present in common-resources .. they are exactly same -- So, not sure where the problem might be.

danyaljj · 2016-10-06T00:30:59Z

So all the other tests pass except this?

Seems like GazetteerViewGenerator is not able to extract the resource. Have you tried creating Gazetteer instance directly?

 new GazetteerViewGenerator("resources/gazetteers/gazetteers", ViewNames.GAZETTEER + "Gazetteers");

shatu · 2016-10-06T00:39:48Z

Yup, other tests work just fine :-

Tests run: 103, Failures: 0, Errors: 1, Skipped: 0

And, yes, with your above suggested replacement, it works! I'm not sure I understand why is that the case.

danyaljj · 2016-10-06T00:45:08Z

Actually that probably wasn't a good suggestion, as the annotators are lazy. So creating an instance doesn't cause it to load the resource. How about you run instance.doInitialize() afterwards?

shatu · 2016-10-06T00:51:11Z

doInitialize works for your suggestion above, but not for the original code i.e.

new GazetteerViewGenerator("gazetteers", ViewNames.GAZETTEER)

danyaljj · 2016-10-06T00:54:13Z

ok, so there must a problematic/missing resource. Why don't you add enough logging into the constructor to see for what input it's failing?

shatu · 2016-10-06T00:56:25Z

Sure, I'll debug it more thoroughly then -- just wanted to see if you guys have encountered similar problems elsewhere, and if it's an edison-only issue or not.

danyaljj · 2016-10-17T14:03:30Z

@shatu no progress on this?

shatu · 2016-10-19T05:52:16Z

@danyaljj .. apparently, this was over long ago. The same code just worked now; most likely, our maven repo had some issues back then in fetching the resources jar.

Edison tests pass now; should we now create tickets in the projects that use common-resources?

mssammon · 2016-10-19T12:22:45Z

@shatu great -- and yes, it would be helpful to open tickets in affected projects...

mssammon assigned mayhewsw and shatu and unassigned mayhewsw Sep 17, 2016

mssammon added this to the CCG Collective Bug Bash Round Two milestone Sep 19, 2016

danyaljj mentioned this issue Sep 23, 2016

Removed brownBllipClusters, since it already exists elsewhere #220

Merged

danyaljj mentioned this issue Sep 29, 2016

WIP: Adding SRL #215

Open

3 tasks

shatu mentioned this issue Oct 19, 2016

Edison Pom update due to disintegration of common-resources #246

Merged

mssammon closed this as completed Oct 19, 2016

mssammon mentioned this issue Nov 4, 2016

update common resources #43

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

break apart illinois-common-resources #186

break apart illinois-common-resources #186

cogcomp-dev commented Sep 15, 2016

cowchipkid commented Sep 15, 2016

mssammon commented Sep 19, 2016

shatu commented Sep 28, 2016 •

edited

Loading

shatu commented Sep 29, 2016

mssammon commented Sep 29, 2016

shatu commented Sep 29, 2016 •

edited

Loading

cowchipkid commented Sep 29, 2016

danyaljj commented Sep 29, 2016

danyaljj commented Sep 29, 2016

danyaljj commented Sep 29, 2016

cowchipkid commented Oct 4, 2016

shatu commented Oct 4, 2016 •

edited

Loading

cowchipkid commented Oct 4, 2016

shatu commented Oct 5, 2016 •

edited

Loading

danyaljj commented Oct 5, 2016

cowchipkid commented Oct 5, 2016

shatu commented Oct 5, 2016 •

edited

Loading

danyaljj commented Oct 5, 2016

cowchipkid commented Oct 5, 2016

shatu commented Oct 5, 2016 •

edited

Loading

shatu commented Oct 5, 2016

danyaljj commented Oct 5, 2016

shatu commented Oct 6, 2016

shatu commented Oct 6, 2016

shatu commented Oct 6, 2016

danyaljj commented Oct 6, 2016 •

edited

Loading

shatu commented Oct 6, 2016

danyaljj commented Oct 6, 2016

shatu commented Oct 6, 2016 •

edited

Loading

danyaljj commented Oct 6, 2016

shatu commented Oct 6, 2016

danyaljj commented Oct 17, 2016

shatu commented Oct 19, 2016 •

edited

Loading

mssammon commented Oct 19, 2016

break apart illinois-common-resources #186

break apart illinois-common-resources #186

Comments

cogcomp-dev commented Sep 15, 2016

cowchipkid commented Sep 15, 2016

mssammon commented Sep 19, 2016

shatu commented Sep 28, 2016 • edited Loading

shatu commented Sep 29, 2016

mssammon commented Sep 29, 2016

shatu commented Sep 29, 2016 • edited Loading

cowchipkid commented Sep 29, 2016

danyaljj commented Sep 29, 2016

danyaljj commented Sep 29, 2016

danyaljj commented Sep 29, 2016

cowchipkid commented Oct 4, 2016

shatu commented Oct 4, 2016 • edited Loading

cowchipkid commented Oct 4, 2016

shatu commented Oct 5, 2016 • edited Loading

danyaljj commented Oct 5, 2016

cowchipkid commented Oct 5, 2016

shatu commented Oct 5, 2016 • edited Loading

danyaljj commented Oct 5, 2016

cowchipkid commented Oct 5, 2016

shatu commented Oct 5, 2016 • edited Loading

shatu commented Oct 5, 2016

danyaljj commented Oct 5, 2016

shatu commented Oct 6, 2016

shatu commented Oct 6, 2016

shatu commented Oct 6, 2016

danyaljj commented Oct 6, 2016 • edited Loading

shatu commented Oct 6, 2016

danyaljj commented Oct 6, 2016

shatu commented Oct 6, 2016 • edited Loading

danyaljj commented Oct 6, 2016

shatu commented Oct 6, 2016

danyaljj commented Oct 17, 2016

shatu commented Oct 19, 2016 • edited Loading

mssammon commented Oct 19, 2016

shatu commented Sep 28, 2016 •

edited

Loading

shatu commented Sep 29, 2016 •

edited

Loading

shatu commented Oct 4, 2016 •

edited

Loading

shatu commented Oct 5, 2016 •

edited

Loading

shatu commented Oct 5, 2016 •

edited

Loading

shatu commented Oct 5, 2016 •

edited

Loading

danyaljj commented Oct 6, 2016 •

edited

Loading

shatu commented Oct 6, 2016 •

edited

Loading

shatu commented Oct 19, 2016 •

edited

Loading