OpenCV and Tesseract can be associated with Artificial Intelligence due to their involvement in tasks that often fall under the AI umbrella, such as computer vision and text recognition. To automate solving image CAPTCHAs using Java, you will typically need several dependencies for tasks such as image processing, machine learning, and possibly computer vision.

OpenCV: A powerful library for computer vision and image processing. You can use the Java bindings for OpenCV.

Tesseract OCR: Tesseract OCR is an optical character recognition library that extracts text from images.

OpenCV (Open-Source Computer Vision Library)

• Category: AI Toolkit for Computer Vision and Image Processing.
• Purpose: Provides comprehensive tools for image and video processing, essential for many AI applications.
• Capabilities: Includes image transformation, filtering, feature detection, object detection, and support for machine learning and deep learning.
• Usage: Commonly used in AI projects for tasks like object detection, face recognition, and image classification.

Tesseract

• Category: AI Toolkit for Optical Character Recognition (OCR)
• Purpose: Converts images of text into machine-readable text using machine learning techniques.
• Capabilities: Recognizes and extracts text from images, supporting multiple languages and fonts.
• Usage: Utilized in AI projects for tasks such as document digitization, data extraction from scanned documents, and integrating text recognition into applications.

Step 1: Set up Dependencies

First, add the necessary dependencies to your pom.xml file:

Step 2: Write the Java Code

Create a Java class to preprocess the CAPTCHA image and extract the text using Tesseract.

Step 3: Set up Tesseract Data

You must create your Tesseract data files and place them in a project directory (e.g., tessdata). but you do need at least the data file for the language used in the CAPTCHA. Tesseract uses trained data files (often referred to as “language data” or “Tessdata”) to recognize text in different languages.

For example, you only need the English-trained data file if your CAPTCHA contains English text. If contains text in another language, you’ll need the corresponding trained data file.

Step 4: Run the code

Ensure you have an image CAPTCHA in your project directory(e.g.,captcha//captcha.png) and adjust the paths in the code accordingly.

Then, run the CaptchaSolver class.

Final Console Output of Extracted Image Text

Explanation

1. Image Preprocessing:
   • Load the image in grayscale mode.
   • Apply a binary threshold to convert the image to black and white.
   • Save the preprocessed image to disk.
2. Text Extraction with Tesseract:
   • Initialize Tesseract and point it to the tessdata directory.
   • Process the preprocessed image with Tesseract to extract the text.

By running this code, you should be able to automate the solving of simple image CAPTCHAs. Adjust the preprocessing steps as necessary for more complex CAPTCHAs.

Summary

Referring to OpenCV and Tesseract as components of an “AI toolkit” accurately reflects their roles in enabling and enhancing AI applications, particularly in the domains of computer vision and text recognition. They are essential tools for implementing AI-driven solutions, making them integral parts of the AI development ecosystem.

As we learned in the last blog, how to set up Appium 2.0 from scratch and install Appium Inspector too. This blog will teach about Appium Inspector’s desired capabilities to start our Android apps. Appium Inspector is part of the Appium 2.0 framework used to inspect elements within mobile applications for automation testing. Appium Inspector enables users to identify elements such as text boxes, buttons, drop-downs, etc., by getting their attributes like ID, class, and paths.

Capabilities of Appium

• appium:platformName – This capability is used to call Appium to automate testing on the specified platform. It specifies the platform name. For example, if you are testing an Android application, then set platformName to “Android”; if you are testing an iOS application, you set platformName to “IOS”.
• appium:platformVersion – It specifies the version of the mobile operating system running on the device. For example, if you are testing an Android application on a device running Android 10 then, set platformVersion to “10”. Similarly, if you are testing an iOS application on a device running iOS 14, you set platformVersion to “12.0”.
• appium:automationName – Which driver is used to automate the application.For Android, “UiAutomator2” and “Espresso” for native apps, and “XCUITest” for iOS.
• appium:deviceName – It is used to specify the name of the mobile devices or emulator. This capability allows Appium to identify the target device.
• appium:appPackage – It launches a particular application from your mobile device.
• appium:appActivity – It launches a particular activity within the Android application. It is used for navigating to the initial screen where your test case is started.

How to Set Capabilities in Appium Inspector

• Open the Android Studio and Appium Inspector installed on your system in our previous blog Appium 2.0.
• Create and open Emulator under Android Studio.
• Start the Appium server by entering the command ‘appium’ on Git bash Terminal.

  appium 
  
  [Appium] Welcome to Appium v2.0.1 [Appium] Attempting to load driver uiautomator2... [debug] 
  [Appium] Requiring driver at C:\Users\ASUS\.appium\node_modules\appium-uiautomator2-driver 
  [Appium] Appium REST http interface listener started on http://0.0.0.0:4723 [Appium] You can provide the following URLS in your client code to connect to this server: [Appium] http://192.168.1.3:4723/ [Appium] http://127.0.0.1:4723/ (only accessible from the same host)

• To check your Android device or emulator is up and running, use the command ‘adb devices’.

  $ adb devices
  List of devices attached
  emulator-5554   device
  

• Download a sample native APK app for testing from apk native app
• Then drag and drop the downloaded apk app file to your emulator.
  
• Click and open your downloaded apk app under your emulator.

Set Capabilities under Appium Inspector

• By default, our appium runs on ‘http://127.0.0.1:4723/’; we need to set ‘Remote Host: 127.0.0.1’ and “Remote Port: 4723”.
  
• Set desired capabilities appium:platformName, if you are testing an Android application then, set platformName to “Android”; if you are testing an iOS application, you set platformName to “IOS”. Then click on ‘+’.
  
• Set desired capabilities appium:platformVersion, to check your platform version by using the command ‘adb shell getprop ro.build.version.release’.

  $ adb shell getprop ro.build.version.release
  14

  

• Set desired capabilities appium:automationName, Which driver is used to automate the application. For Android, “UIAutomator2” and “XCUITest” are for iOS.
  
• Set Capabilities appium:deviceName, To check your Android device name, use the command ‘adb devices’.

  $ adb devices
  List of devices attached
  emulator-5554   device

  

• Set capabilities appium:appPackage and appium:appActivity, to check your app package and app activity details, we need to open apk app on your device or emulator and then, enter the command in git bash terminal ‘adb shell dumpsys window displays | grep mFocusedApp’.
  
  
• Let’s close apk app on your device or emulator, then click on the ‘Start Session’ button under Appium Inspector.
  

         See, our APK app is successfully up and running.

• You can see locators such as XPath, class name, and accessibility of apk app home screen with the source code. We can use that xpath class name to automate our Android scripts.

Conclusion:

This blog should help users set up desired capabilities under Appium Inspector to launch Android apps. We can find locators (xpaths, ID, class name) and it is easier to locate app components such as textbox, button, dropdown, etc. You can select individual elements in the app by clicking on them by inspector. It is part of the Appium 2.0 framework to inspect elements within mobile applications for automation testing. In the next blog, we will create one project and automate apk app.

In this blog, we will set up Appium plugins, Appium drivers, Appium Doctor, NodeJS, UI Automator, Appium inspector, and Android Studio to create our automation scripts for Android devices. Appium 2.0 is an Android platform tool where drivers and plugins are easily created. It will allow us to cover mobile applications, desktop Apps, and Flutter Apps.

New Changes in Appium 2.0

Independent Drivers: Ability to install and work with WebDrivers based on your application platform (IOS, Android, Windows OS, Flutter).

Appium 2.0 Plugins: A few Appium plugins are available such as Images, Gestures, Wait, Reporter, and Execute-Driver.

It will no longer support the old JSONWP and MJSONWP Protocols. Only W3C WebDriver protocols are supported. By default, the Appium server runs at http://127.0.0.1:4723/.

Let’s Setup Appium 2.0

• First, we need to download open-JDK Version 17 or the latest Version from OpenJDK, which is responsible for executing your Java code.
  • Step 1: Extract the Downloaded JDK file.
  • Step 2: Copy and paste the Extracted JDK file under ‘C:\Program Files\Java folder’.
• After installation of JDK, we need to download Maven from Maven, which is used to build and manage your project dependencies.
  • Step 1: Extract the Downloaded Maven file.
  • Step 2: Copy and paste the Extracted Maven file under ‘C:\Program Files\Java folder’.
• Download and Install Git from Git to track code changes in script files.
• If we want to install and run the Appium server, then NodeJS will play an important role. Let’s Download and install the NodeJS MSI file Version 20.11 or the latest version from Nodejs. While installing NodeJS, check the checkbox ‘Install All necessary Tools’.
  • By default, Nodejs is installed under ‘C:\Program Files\nodejs’
  • By default chocolatey is installed under ‘C:\ProgramData\chocolatey’.

Install Android Studio

• Once all the above steps are successfully done, download and install Android Studio IDE from Android Studio, which is used to test Android applications on various physical or emulated devices.
  
  • Note: By default, SDK is installed under the user’s directory.
    Ex: ‘C:\Users\ASUS\AppData\Local\Android\Sdk

Set Environment Variables into your system (Windows 10/11)

Environment variables are used to configure your app data to decrease the chances of unexpected errors and bugs.

1. Search ‘Edit the system environment variables’ in your system.
2. Click on ‘Environment variables’, then click ‘New’ under system variables and add the below path as per your file location.
   Variable Name: ANDROID_HOME
   Variable Value: C:\Users\ASUS\AppData\Local\Android\Sdk
   Variable Name: JAVA_HOME
   Variable Value: C:\Program Files\Java\jdk-21.0.1
   Variable Name: MVN_HOME
   Variable Value: C:\Program Files\Java\apache-maven-3.9.6
   
3. Click on ‘path’ and set the path below as per your system directory location.
   

How to Install Appium 2.0

• Open Git bash Terminal and enter the command ‘npm i -g appium@next’ to install Appium 2.0.

  npm i -g appium@next

• To Check the Appium Version, enter the command ‘appium -v’ on Git bash Terminal.

  appium -v 
  2.0.1

• Start the Appium server by entering the command ‘appium’ on Git bash Terminal.

  appium
  
  [Appium] Welcome to Appium v2.0.1
  [Appium] Attempting to load driver uiautomator2...
  [debug] [Appium] Requiring driver at C:\Users\ASUS\.appium\node_modules\appium-uiautomator2-driver
  [Appium] Appium REST http interface listener started on http://0.0.0.0:4723
  [Appium] You can provide the following URLS in your client code to connect to this server:
  [Appium] http://192.168.1.3:4723/
  [Appium] http://127.0.0.1:4723/ (only accessible from the same host)
  

• Use the ‘appium driver list’ command to check appium driver lists.

  appium driver list
  
  - Listing available drivers
  √ Listing available drivers
  - uiautomator2@2.43.3 [not installed]
  - xcuitest [not installed]
  - mac2 [not installed]
  - espresso [not installed]
  - safari [not installed]
  - gecko [not installed]
  - chromium [not installed]
  

• To Install the Appium uiautomator2 driver, enter the command ‘appium driver install uiautomator2’ on Git bash Terminal.

  appium driver install uiautomator2

• To Update Installed drivers, enter the command ‘appium driver list –updates’ on Git bash Terminal.

  $ appium driver list --updates
  - Listing available drivers
  √ Listing available drivers
  - uiautomator2@2.43.3 [installed (npm)] [Up to date]
  - xcuitest [not installed]
  - mac2 [not installed]
  - espresso [not installed]
  - safari [not installed]
  - gecko [not installed]
  - chromium [not installed]

• Install Appium Doctor using the command ‘npm install -g appium-doctor’ to check Android setup-related issues.

  npm install -g appium-doctor

• To check the Appium Doctor version, use the command ‘appium-doctor –version’.

  appium-doctor --version
  1.16.2

• To check android setup related issues type ‘appium-doctor –android’.

  appium-doctor --android
  WARN AppiumDoctor [Deprecated] Please use appium-doctor installed with "npm install @appium/doctor --location=global"
  info AppiumDoctor Appium Doctor v.1.16.2
  info AppiumDoctor ### Diagnostic for necessary dependencies starting ###
  info AppiumDoctor   The Node.js binary was found at: C:\Program Files\nodejs\node.EXE
  info AppiumDoctor   Node version is 20.11.0
  info AppiumDoctor   ANDROID_HOME is set to: C:\Users\ASUS\AppData\Local\Android\Sdk
  info AppiumDoctor   JAVA_HOME is set to: C:\Program Files\Java\jdk-21.0.1
  info AppiumDoctor    Checking adb, android, emulator, apkanalyzer.bat
  info AppiumDoctor      'adb' is in C:\Users\ASUS\AppData\Local\Android\Sdk\platform-tools\adb.exe
  info AppiumDoctor      'android' is in C:\Users\ASUS\AppData\Local\Android\Sdk\tools\android.bat
  info AppiumDoctor      'emulator' is in C:\Users\ASUS\AppData\Local\Android\Sdk\emulator\emulator.exe
  info AppiumDoctor      'apkanalyzer.bat' is in C:\Users\ASUS\AppData\Local\Android\Sdk\cmdline-tools\latest\bin\apkanalyzer.bat
  info AppiumDoctor   adb, android, emulator, apkanalyzer.bat exist: C:\Users\ASUS\AppData\Local\Android\Sdk
  info AppiumDoctor   'bin' subfolder exists under 'C:\Program Files\Java\jdk-21.0.1'

• To Check Appium Plugins, Type the command ‘appium plugin list’ on Git bash Terminal.

  $ appium plugin list
  - Listing available plugins
  √ Listing available plugins
  - images [not installed]
  - execute-driver [not installed]
  - relaxed-caps [not installed]
  - universal-xml [not installed]

• To Install Appium Plugin (Ex: images) enter command ‘appium plugin install images’ on Git bash Terminal.

  $ appium plugin install images
  - Installing 'images' using NPM install spec '@appium/images-plugin'
  √ Installing 'images' using NPM install spec '@appium/images-plugin'
  i Plugin images@3.0.1 successfully installed
  

• Download the Appium Inspector zip file from Appium Inspector. Extract the zip file and open Appium Inspector from the folder. Appium Inspector is used to find elements across different mobile platforms.
  
• Open Android Studio that is already installed on your system.  Go to the Device Manager in the top right corner and select any Device.
  
•  Check how many devices are connected with your system by using Command ‘adb devices’.

  $ adb devices
  List of devices attached
  emulator-5554 device
  

Appium 2.0 is now successfully installed on your system!

Conclusion:

This blog should help users install and set up Appium 2.0 smoothly on their system. The main purpose of Appium 2.0 is to make the test automation process more modular and flexible. This includes transitioning to a new architecture for extensions and drivers. It will no longer support the old JSONWP and MJSONWP Protocols. Only W3C Webdriver protocols are supported. By default, the Appium server runs at http://127.0.0.1:4723/. I hope this blog post was useful in helping your transition to Appium 2.0.