Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/source/contributor-guide/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,12 @@ still assert while recording, so a recorded file cannot contradict what hadoop-a
file is written only when every case ran. Review the diff of the regenerated files before
committing them: each line that changes is a behaviour change on that hadoop-azure release.

The FileSystem cross-check in `AbfsAuthParitySuite` is tagged `org.apache.comet.tags.AzureTest`.
It initializes a real `AzureBlobFileSystem`, so it passes only when the test-scoped hadoop-azure
matches the Hadoop the build runs on. A build against another Hadoop, where no matching
hadoop-azure is pinned, can skip it with `-DtagsToExclude=org.apache.comet.tags.AzureTest` and
still run the resolver cases.

## Benchmark

There's a `make` command to run micro benchmarks in the repo. For
Expand Down
5 changes: 4 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,9 @@ under the License.
reflection only). Each Spark profile pins it to its Hadoop line; override with
-Dhadoop-azure.version to test another release. -->
<hadoop-azure.version>3.4.2</hadoop-azure.version>
<!-- Extra scalatest tags to exclude, comma separated, for example
-DtagsToExclude=org.apache.comet.tags.AzureTest. -->
<tagsToExclude></tagsToExclude>
<jni.dir>${project.basedir}/../native/target/debug</jni.dir>
<platform>darwin</platform>
<arch>x86_64</arch>
Expand Down Expand Up @@ -1124,7 +1127,7 @@ under the License.
<filereports>SparkTestSuite.txt</filereports>
<stdout>D</stdout>
<stderr/>
<tagsToExclude>org.apache.comet.IntegrationTestSuite</tagsToExclude>
<tagsToExclude>org.apache.comet.IntegrationTestSuite,${tagsToExclude}</tagsToExclude>
<systemProperties>
<!-- emit test logs to target/unit-tests.log -->
<log4j.configurationFile>file:src/test/resources/log4j2.properties</log4j.configurationFile>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ import com.fasterxml.jackson.databind.{JsonNode, ObjectMapper}
import org.apache.comet.objectstore.AbfsAuthParityCases._
import org.apache.comet.objectstore.AbfsAuthResolver._
import org.apache.comet.objectstore.AbfsReflection.Handles
import org.apache.comet.tags.AzureTest

/**
* Pins the resolver's behaviour per hadoop-azure version and cross-checks it against the real
Expand Down Expand Up @@ -184,7 +185,9 @@ class AbfsAuthParitySuite extends AnyFunSuite with Matchers with BeforeAndAfterA
}
}

test("FileSystem.newInstance agrees with the resolver on every case") {
// Tagged AzureTest: AzureBlobFileSystem links against Hadoop classes that only a matching
// Hadoop has, while the resolver cases above need nothing newer than AbfsConfiguration.
test("FileSystem.newInstance agrees with the resolver on every case", AzureTest) {
val timings = mutable.ArrayBuffer[(String, Long)]()
val disagreements = cases.flatMap { c =>
val outcome = resolve(hadoopConf(c), c.uri)
Expand Down
30 changes: 30 additions & 0 deletions spark/src/test/scala/org/apache/comet/tags/AzureTest.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.comet.tags

import org.scalatest.Tag

/**
* Marks tests that initialize a real `AzureBlobFileSystem` from the hadoop-azure on the test
* classpath. They pass only when that hadoop-azure matches the Hadoop the build runs on, so a
* build against another Hadoop can skip them with
* `-DtagsToExclude=org.apache.comet.tags.AzureTest`.
*/
object AzureTest extends Tag("org.apache.comet.tags.AzureTest")
Loading