Skip to content

Commit d8afbb8

Browse files
YUNQIUGUOYUNQIUGUO
authored
[Mobile] Add super resolution sample android app with ort-extensions support for ppp (#182)
* add initial super resolution android app with updated model * refine the project * update and save * add super res model * address pr comments * move to superRes/android dir * update * minor fix * update * add readme file * minor update * minor update * address pr comments * check in a workable aar package from packaging pipeline * update aar version * revert version and see * update * update pr comments
1 parent 5214e20 commit d8afbb8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+1035
-0
lines changed

‎mobile/examples/super_resolution/android/.gitignore

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
*.iml
2+
.gradle
3+
/local.properties
4+
/.idea/caches
5+
/.idea/libraries
6+
/.idea/modules.xml
7+
/.idea/workspace.xml
8+
/.idea/navEditor.xml
9+
/.idea/assetWizardSettings.xml
10+
.DS_Store
11+
/build
12+
/captures
13+
.externalNativeBuild
14+
.cxx
15+
local.properties

‎mobile/examples/super_resolution/android/README.md

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# ONNX Runtime Mobile Super Resolution Android sample application with Ort-Extensions support for pre/post processing
2+
3+
## Overview
4+
5+
This is a basic Super Resolution example application for [ONNX Runtime](https://github.com/microsoft/onnxruntime) on Android with [Ort-Extensions](https://github.com/microsoft/onnxruntime-extensions) support for pre/post processing. The demo app accomplishes the task of recovering a high resolution (HR) image from its low resolution counterpart.
6+
7+
The model used here is from source: [Pytorch Super Resolution](https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html) and accomodated into [ONNX](https://github.com/onnx/onnx) version with pre/post processing support.
8+
9+
10+
### Requirements
11+
- Android Studio Dolphin | 2021.3.1 Patch + (installed on Mac/Windows/Linux)
12+
- Android SDK 29+
13+
- Android NDK r22+
14+
- An Android device or an Android Emulator
15+
16+
### Steps to build and run
17+
18+
19+
### Step 1: Clone the ONNX runtime mobile examples source code
20+
21+
Clone this repository to get the sample application. Then open the project under folder `mobile\examples\super_resolution\android`.
22+
23+
24+
### Step 2: Prepare the model and required packages
25+
26+
- The required aar package is under lib's folder `mobile\examples\super_resolution\android\app\libs`.
27+
- The model used is under `mobile\examples\super_resolution\android\app\src\main\res\raw`.
28+
29+
Note: When update to pre-release/release version ort-extensions android package, it will no longer consume a local AAR package that's included in this repo.
30+
31+
32+
### Step 3: Connect Android Device and Run the app
33+
Connect your Android Device to your computer or select the Android Emulator in Android Studio Device manager.
34+
35+
Then select `Run -> Run app` and this will prompt the app to be built and installed on your device or emulator.
36+
37+
Now you can try and test the super resolution android app by clicking the perform action button.
38+
39+
#
40+
Here's an example screenshot of the app.
41+
42+
<img width=20% src="images/sample_screenshot_1.png" alt="App Screenshot 1" />
43+
<img width=20% src="images/sample_screenshot_2.png" alt="App Screenshot 2" />

‎mobile/examples/super_resolution/android/app/.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/build

‎mobile/examples/super_resolution/android/app/build.gradle

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
plugins {
2+
id 'com.android.application'
3+
id 'org.jetbrains.kotlin.android'
4+
id 'kotlin-android-extensions'
5+
}
6+
7+
android {
8+
namespace 'ai.onnxruntime.example.superresolution'
9+
compileSdk 32
10+
11+
defaultConfig {
12+
applicationId "ai.onnxruntime.example.superresolution"
13+
minSdk 24
14+
targetSdk 32
15+
versionCode 1
16+
versionName "1.0"
17+
18+
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
19+
}
20+
21+
buildTypes {
22+
release {
23+
minifyEnabled false
24+
proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
25+
}
26+
}
27+
compileOptions {
28+
sourceCompatibility JavaVersion.VERSION_1_8
29+
targetCompatibility JavaVersion.VERSION_1_8
30+
}
31+
kotlinOptions {
32+
jvmTarget = '1.8'
33+
}
34+
buildFeatures {
35+
viewBinding true
36+
}
37+
}
38+
39+
dependencies {
40+
implementation 'androidx.core:core-ktx:1.7.0'
41+
implementation 'androidx.appcompat:appcompat:1.5.1'
42+
implementation 'com.google.android.material:material:1.7.0'
43+
implementation 'androidx.constraintlayout:constraintlayout:2.1.4'
44+
testImplementation 'junit:junit:4.13.2'
45+
androidTestImplementation 'androidx.test.ext:junit:1.1.4'
46+
androidTestImplementation 'androidx.test.espresso:espresso-core:3.5.0'
47+
48+
implementation 'com.microsoft.onnxruntime:onnxruntime-android:latest.release'
49+
// TODO: update with released version aar package when available
50+
implementation files('libs/onnxruntime-extensions-android-0.6.0.aar')
51+
}

‎mobile/examples/super_resolution/android/app/libs/onnxruntime-extensions-android-0.6.0.aar

3.03 MB
Binary file not shown.

‎mobile/examples/super_resolution/android/app/proguard-rules.pro

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Add project specific ProGuard rules here.
2+
# You can control the set of applied configuration files using the
3+
# proguardFiles setting in build.gradle.
4+
#
5+
# For more details, see
6+
# http://developer.android.com/guide/developing/tools/proguard.html
7+
8+
# If your project uses WebView with JS, uncomment the following
9+
# and specify the fully qualified class name to the JavaScript interface
10+
# class:
11+
#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
12+
# public *;
13+
#}
14+
15+
# Uncomment this to preserve the line number information for
16+
# debugging stack traces.
17+
#-keepattributes SourceFile,LineNumberTable
18+
19+
# If you keep the line number information, uncomment this to
20+
# hide the original source file name.
21+
#-renamesourcefileattribute SourceFile

‎mobile/examples/super_resolution/android/app/src/androidTest/java/ai/onnxruntime/example/superresolution/ExampleInstrumentedTest.kt

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
package ai.onnxruntime.example.superresolution
2+
3+
import ai.onnxruntime.extensions.OrtxPackage;
4+
import ai.onnxruntime.OrtEnvironment
5+
import ai.onnxruntime.OrtSession
6+
7+
import androidx.test.platform.app.InstrumentationRegistry
8+
import androidx.test.ext.junit.runners.AndroidJUnit4
9+
10+
import org.junit.Test
11+
import org.junit.runner.RunWith
12+
13+
import org.junit.Assert.*
14+
15+
/**
16+
* Instrumented test, which will execute on an Android device.
17+
*
18+
* See [testing documentation](http://d.android.com/tools/testing).
19+
*/
20+
@RunWith(AndroidJUnit4::class)
21+
class ExampleInstrumentedTest {
22+
@Test
23+
fun useAppContext() {
24+
// Context of the app under test.
25+
val appContext = InstrumentationRegistry.getInstrumentation().targetContext
26+
assertEquals("ai.onnxruntime.example.superresolution", appContext.packageName)
27+
}
28+
29+
@Test
30+
fun loadModelAndCreateOrtSession() {
31+
// Context of the app under test.
32+
val appContext = InstrumentationRegistry.getInstrumentation().targetContext
33+
val resources = appContext.resources
34+
val modelBytes = resources.openRawResource(R.raw.pt_super_resolution_op16).readBytes()
35+
val env = OrtEnvironment.getEnvironment()
36+
env.use {
37+
assertNotNull(env)
38+
val sessionOptions: OrtSession.SessionOptions = OrtSession.SessionOptions()
39+
sessionOptions.registerCustomOpLibrary(OrtxPackage.getLibraryPath())
40+
val session = env.createSession(modelBytes, sessionOptions)
41+
session.use {
42+
assertNotNull(session)
43+
}
44+
}
45+
}
46+
}

‎mobile/examples/super_resolution/android/app/src/main/AndroidManifest.xml

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
3+
xmlns:tools="http://schemas.android.com/tools">
4+
5+
<uses-feature android:name="android.hardware.camera.any" />
6+
<uses-permission android:name="android.permission.CAMERA" />
7+
8+
<application
9+
android:allowBackup="true"
10+
android:dataExtractionRules="@xml/data_extraction_rules"
11+
android:fullBackupContent="@xml/backup_rules"
12+
android:icon="@mipmap/ic_launcher"
13+
android:label="@string/app_name"
14+
android:roundIcon="@mipmap/ic_launcher_round"
15+
android:supportsRtl="true"
16+
android:theme="@style/Theme.Super_resolution"
17+
tools:targetApi="31">
18+
<activity
19+
android:name=".MainActivity"
20+
android:exported="true">
21+
<intent-filter>
22+
<action android:name="android.intent.action.MAIN" />
23+
24+
<category android:name="android.intent.category.LAUNCHER" />
25+
</intent-filter>
26+
27+
<meta-data
28+
android:name="android.app.lib_name"
29+
android:value="" />
30+
</activity>
31+
</application>
32+
33+
</manifest>

‎mobile/examples/super_resolution/android/app/src/main/assets/test_superresolution.png

58.6 KB
Loading

‎mobile/examples/super_resolution/android/app/src/main/java/ai/onnxruntime/example/superresolution/MainActivity.kt

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
package ai.onnxruntime.example.superresolution
2+
3+
import ai.onnxruntime.*
4+
import ai.onnxruntime.extensions.OrtxPackage
5+
import android.annotation.SuppressLint
6+
import android.os.Bundle
7+
import android.widget.Button
8+
import android.widget.ImageView
9+
import android.widget.Toast
10+
import androidx.activity.*
11+
import androidx.appcompat.app.AppCompatActivity
12+
import kotlinx.android.synthetic.main.activity_main.*
13+
import kotlinx.coroutines.*
14+
import java.io.InputStream
15+
import java.util.*
16+
import java.util.concurrent.ExecutorService
17+
import java.util.concurrent.Executors
18+
19+
20+
class MainActivity : AppCompatActivity() {
21+
private var ortEnv: OrtEnvironment = OrtEnvironment.getEnvironment()
22+
private var outputImage: ImageView? = null
23+
private var superResolutionButton: Button? = null
24+
25+
@SuppressLint("UseCompatLoadingForDrawables")
26+
override fun onCreate(savedInstanceState: Bundle?) {
27+
super.onCreate(savedInstanceState)
28+
setContentView(R.layout.activity_main)
29+
30+
outputImage = findViewById(R.id.imageView2);
31+
superResolutionButton = findViewById(R.id.super_resolution_button)
32+
33+
superResolutionButton?.setOnClickListener {
34+
performSuperResolution()
35+
Toast.makeText(baseContext, "Super resolution performed!", Toast.LENGTH_SHORT).show()
36+
}
37+
}
38+
39+
override fun onDestroy() {
40+
super.onDestroy()
41+
ortEnv.close()
42+
}
43+
44+
private fun updateUI(result: Result) {
45+
outputImage?.setImageBitmap(result.outputBitmap)
46+
}
47+
48+
private fun readModel(): ByteArray {
49+
val modelID = R.raw.pt_super_resolution_op16
50+
return resources.openRawResource(modelID).readBytes()
51+
}
52+
53+
private fun readInputImage(): InputStream {
54+
return assets.open("test_superresolution.png")
55+
}
56+
57+
private fun createOrtSession(): OrtSession {
58+
val sessionOptions: OrtSession.SessionOptions = OrtSession.SessionOptions()
59+
sessionOptions.registerCustomOpLibrary(OrtxPackage.getLibraryPath())
60+
return ortEnv.createSession(readModel(), sessionOptions)
61+
}
62+
63+
private fun performSuperResolution() {
64+
var superResPerformer = SuperResPerformer(createOrtSession())
65+
var result = superResPerformer.upscale(readInputImage(), ortEnv)
66+
updateUI(result);
67+
}
68+
69+
companion object {
70+
const val TAG = "ORTSuperResolution"
71+
}
72+
}

‎mobile/examples/super_resolution/android/app/src/main/java/ai/onnxruntime/example/superresolution/SuperResPerformer.kt

+62
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
package ai.onnxruntime.example.superresolution
2+
3+
import ai.onnxruntime.OnnxJavaType
4+
import ai.onnxruntime.OrtSession
5+
import ai.onnxruntime.OnnxTensor
6+
import ai.onnxruntime.OrtEnvironment
7+
import android.graphics.Bitmap
8+
import android.graphics.BitmapFactory
9+
import java.io.InputStream
10+
import java.nio.ByteBuffer
11+
import java.util.*
12+
13+
internal data class Result(
14+
var outputBitmap: Bitmap? = null
15+
) {}
16+
17+
internal class SuperResPerformer(
18+
private val ortSession: OrtSession
19+
) {
20+
21+
fun upscale(inputStream: InputStream, ortEnv: OrtEnvironment): Result {
22+
var result = Result()
23+
24+
// Step 1: convert image into byte array (raw image bytes)
25+
val rawImageBytes = inputStream.readBytes()
26+
27+
// Step 2: get the shape of the byte array and make ort tensor
28+
val shape = longArrayOf(rawImageBytes.size.toLong())
29+
30+
ortEnv.use {
31+
val inputTensor = OnnxTensor.createTensor(
32+
ortEnv,
33+
ByteBuffer.wrap(rawImageBytes),
34+
shape,
35+
OnnxJavaType.UINT8
36+
)
37+
inputTensor.use {
38+
// Step 3: call ort inferenceSession run
39+
val output = ortSession.run(Collections.singletonMap("image", inputTensor))
40+
41+
// Step 4: output analysis
42+
output.use {
43+
val rawOutput = (output?.get(0)?.value) as ByteArray
44+
val outputImageBitmap =
45+
byteArrayToBitmap(rawOutput)
46+
47+
// Step 5: set output result
48+
result.outputBitmap = outputImageBitmap
49+
}
50+
}
51+
}
52+
return result
53+
}
54+
55+
private fun byteArrayToBitmap(data: ByteArray): Bitmap {
56+
return BitmapFactory.decodeByteArray(data, 0, data.size)
57+
}
58+
59+
protected fun finalize() {
60+
ortSession.close()
61+
}
62+
}

‎mobile/examples/super_resolution/android/app/src/main/res/drawable-v24/ic_launcher_foreground.xml

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
<vector xmlns:android="http://schemas.android.com/apk/res/android"
2+
xmlns:aapt="http://schemas.android.com/aapt"
3+
android:width="108dp"
4+
android:height="108dp"
5+
android:viewportWidth="108"
6+
android:viewportHeight="108">
7+
<path android:pathData="M31,63.928c0,0 6.4,-11 12.1,-13.1c7.2,-2.6 26,-1.4 26,-1.4l38.1,38.1L107,108.928l-32,-1L31,63.928z">
8+
<aapt:attr name="android:fillColor">
9+
<gradient
10+
android:endX="85.84757"
11+
android:endY="92.4963"
12+
android:startX="42.9492"
13+
android:startY="49.59793"
14+
android:type="linear">
15+
<item
16+
android:color="#44000000"
17+
android:offset="0.0" />
18+
<item
19+
android:color="#00000000"
20+
android:offset="1.0" />
21+
</gradient>
22+
</aapt:attr>
23+
</path>
24+
<path
25+
android:fillColor="#FFFFFF"
26+
android:fillType="nonZero"
27+
android:pathData="M65.3,45.828l3.8,-6.6c0.2,-0.4 0.1,-0.9 -0.3,-1.1c-0.4,-0.2 -0.9,-0.1 -1.1,0.3l-3.9,6.7c-6.3,-2.8 -13.4,-2.8 -19.7,0l-3.9,-6.7c-0.2,-0.4 -0.7,-0.5 -1.1,-0.3C38.8,38.328 38.7,38.828 38.9,39.228l3.8,6.6C36.2,49.428 31.7,56.028 31,63.928h46C76.3,56.028 71.8,49.428 65.3,45.828zM43.4,57.328c-0.8,0 -1.5,-0.5 -1.8,-1.2c-0.3,-0.7 -0.1,-1.5 0.4,-2.1c0.5,-0.5 1.4,-0.7 2.1,-0.4c0.7,0.3 1.2,1 1.2,1.8C45.3,56.528 44.5,57.328 43.4,57.328L43.4,57.328zM64.6,57.328c-0.8,0 -1.5,-0.5 -1.8,-1.2s-0.1,-1.5 0.4,-2.1c0.5,-0.5 1.4,-0.7 2.1,-0.4c0.7,0.3 1.2,1 1.2,1.8C66.5,56.528 65.6,57.328 64.6,57.328L64.6,57.328z"
28+
android:strokeWidth="1"
29+
android:strokeColor="#00000000" />
30+
</vector>

0 commit comments

Comments
 (0)