iOS library used to implement comic vignettes segmentation using a machine learning method named deep learning.
DeepPanel let's you extract all the panels' location from a comic page using a machine learning model trained here. This iOS library does not use Open CV at all, this means the size of the final app you generates will be as small as possible. We've optimized our model in terms of size and performance and implemented the computer vision algorithms we needed without any third party library to optimize the final library size and performance. Powered by TensorFlow lite, our already trained panels' segmentation model and some native code, DeepPanel is able to find all the panels' location in less than a second.
Keep in mind, this artificial intelligence solution is designed to simulate the human reading behavior and it will group panels that are related as a human would do while reading a comic book page. When analyzing a page we do not use image processing algorithms in order to find the panels. Our solution is based on deep learning. We do not use contour detection algorithms at all even when the samples analysis can look similar. If you are looking for a panels contouring solution you can use Open CV or any other image analysis tools like this project does.
Original | Analyzed |
---|---|
![]() |
![]() |
![]() |
![]() |
In case you are looking for the Android version of this project, you can find it here.
All the pages used in our sample project are created by Peper & Carrot which is a free comic book you should read right now ❤️
Include the library in your Podfile
pod 'DeepPanel'
To start using the library you just need to call DeepPanel.initialize
. You can initialize this library from any ViewController
or from your AppDelegate
class:
class AppDelegate: UIResponder, UIApplicationDelegate {
func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
DeepPanel.initialize()
return true
}
}
Once you've initialized the library get any comic book page, transform it into a UIImage
instance and extract the panels' information using extractPanelsInfo
method as follows:
let deepPanel = DeepPanel()
let result = deepPanel.extract(bitmapSamplePage)
result.panels.panelsInfo.forEach { panel in
print("\(panel.left) - \(panel.top) - \(panel.right) - \(panel.bottom)")
}
PredictionResult
contains a list of panels with the position of every panel on the page and also a 2d matrix of integers with the following labels inside:
- 0 - Label associated to the page of the content.
- N - Content related to the same panel on the page.
Do not invoke DeepPanel extractPanelsInfo
from the main app thread. Even when we can analyze a page in less than 500ms using regular device, you should not block your app UI at all. Our recommendation is to extract the analysis computation out of the UI thread using any queue.
Keep in mind for the first version of this library we consider every panel as a rectangle even if the content panel inside has a different shape. We will improve this in the future, for our very first release we've decided to use this approach because we will always represent the information of the panel on a rectangular screen.
Feel free to contribute, we will be glad to improve this project with your help. Keep in mind that your PRs must be validated by GitHub actions before being reviewed by any core contributor.
Special thanks to Asun, who never doubt we could publish this project ❤️
- Pedro Vicente Gómez Sánchez - [email protected]
Copyright 2020 Pedro Vicente Gómez Sánchez
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.