Jan 10, 2019 - You can now run Appium tests written in Ruby or Node.js against your native, hybrid and browser-based apps on AWS Device Farm.
Compare the features of private and public device clouds to see which one is best for your mobile app testing needs.
Join the DZone community and get the full member experience.
Join For FreeInformation Technology is all about optimizing things and getting work done effectively. Instead of hiring a whole new department and equipping it, it’s sometimes better to leave it off to the team of professionals providing full service and ensuring everything will be done in a cost-efficient and productive way. We’ve handpicked 10 reliable companies owning Device clouds. They offer an arrange of flexible plans, full support along the way and free trials to know what you’re paying for.
How to Choose Between Public and Private Mobile Cloud
In some cases, you may want to consider using emulators to test the apps. Though Public clouds are definitely cheaper, you risk waiting in queues.
Choose public cloud if:
- You don’t need any customized device sets but seek for the nice budget first.
- Test flows are simple and can be run on a single device.
- You need different devices for each test iteration.
- You are OK with limited interaction time since you don’t have continuous test execution.
- Standard security measures of public clouds fully satisfy your company’s IPSEC policies.
Choose a private cloud if:
- You need more than simple testing logic such as advanced scenarios with armbands, fitness gear, and smart clothes; complex flows when the single test case requires interaction with several mobile platforms, WEB, and APIs.
- You have a large number of test cases to run on a daily basis (over 1k).
- You want to debug your tests on the real hardware when you face device-specific issues.
- You need a fully secured environment with custom requirements implemented.
AWS Device Farm (by Amazon)
Official Service Description: 'AWS Device Farm allows developers to increase application quality, time to market, and customer satisfaction by testing and interacting with real Android and iOS devices in the AWS Cloud. Developers can upload their app and test scripts and run automated tests in parallel across 100s of real devices, getting results, screenshots, video, and performance data in minutes. They can also debug and reproduce customer issues by swiping, gesturing, and interacting with a device through their web browser.'
Types of testing: automated, manual, debugging
Types of apps: native, hybrid, web
Browsers: latest versions of Chrome, Firefox, Safari, IE 9
Devices: AWS Device Farm tests are run on real, non-rooted devices. The devices are a mixture of OEM and carrier-branded devices (367 total devices - Android, iOS phones, tablets, phablets).
Features:
- Supports files up to 4 GB/ the number of simultaneous test runs is 5/ 60-minute limit length of a remote access session/ 60 minute limit on the automated session
- Plugins and APIs to automatically initiate tests and get results from IDEs and continuous integration environments like Android Studio and Jenkins
- Testing cross-platform apps including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks
- Remote access via a web browser (gesturing, swiping and interacting with devices for Android apps)
- Parallel tests with collected high-level results, low-level logs, videos, pixel-to-pixel screenshots, and performance data
- For automated tests, issues are identified and grouped
- The test environment can be configured by location, language, network connection, application data, pre-installed apps to simulate real-world customer conditions
- Company offers its own built-in, script-free test suite, but tests are also customizable by open source frameworks (for every type of app - Appium Java Unit, Appium Java TestNG, Appium Python; for Android apps - Calabash, Instrumentation (like Unit, Espresso, Robotium, or any instrumentation-based tests), UI Automator, Explorer; for iOS - UI Automation, XCTest (UI))
Pricing: $0.17 per device minute (pay-as-you-go model); unlimited testing is available for a flat fee of $250/month per device; private devices start from $200/month
Firebase Test Lab (by Google)
Official Service Description: 'Firebase Test Lab for Android provides cloud-based infrastructure for testing Android apps. With one operation, you can initiate testing of your app across a wide variety of devices and device configurations. Test results—including logs, videos, and screenshots—are made available in your project in the Firebase console. Even if you haven't written any test code for your app, Test Lab can exercise your app automatically, looking for crashes.'
Types of testing: automated
Types of Apps: Android (web, C++, Unity, game)
Browsers: desktop - Chrome, Firefox, Safari, and Edge; mobile - browsers are currently not fully supported
Devices: Android, iOS (devices and emulators)
Features:
- Devices are installed and run in a Google data center. Test Lab helps you to find issues that only occur on specific device configurations (e.g., a Nexus 5 running a specific Android API level and specific local settings)
- In case you don’t have written app tests, Robo test tool is there to help you. It analyzes the structure of your app’s user interface and automatically simulates user activities. You can run your pre-written tests as well
- Test Lab is integrated with Android studio, the Firebase console, and the gcloud command line. Continuous integration system can also be used
- Allows to runs tests via Espresso, UI Automator 2.0, Orchestrator instrumentation, Robo test, Game loop test (Beta)
Pricing (by request): Spark - free plan with limitations, Flame - low price for early-stage startups and small-production apps, Blaze - a pay-as-you-go plan for large-scale production apps (extendable with paid Google Cloud Platform features).
MCloud (by Solvd)
Official Service Description: MCloud provides 'a cross-platform solution - a private mobile cloud for testing apps. The private cloud provides much higher security level. We can set up the entire infrastructure right in your office or alternatively within a remote data center, ensuring it’s compliant with all IPSEC and data protection standards. Submit your device suite required for testing and we’ll set up solution custom fit just for you. We can also help you assess the relative importance of particular devices if requested. Since it’s a private cloud, you won’t have to wait in queues - it’s dedicated only to your project and your specific needs.'
Types of testing: manual, automated
Types of apps: native, hybrid, web
Devices: Android, iOS devices (phones, tablets)
Features:
- Exclusive usage of devices 24/7
- Set of devices is handpicked individually for every customer and is not used by anyone else
- High level of provided security allows testing private applications
- Multithreaded execution - you can split test suite in multiple threads if you need rapid results for your test run. Single threads are used for deep regression coverage for each device
- Full control over the device - it’s crucial for testing smart gear
- Cover all platform using a single test suite. Run Selenium/Appium-based tests on the real hardware
- Perform manual tests within the user-friendly admin tool. Install builds, capture screenshots, videos or console logs
- Results are dynamically retrieved in the tools you prefer: Jenkins, TeamCity, TestRail, JIRA, Slack and many more
- Each device has its own Dockerized container, so whenever you need to restart Appium or ADB service you don’t need to restart the entire cloud, you’ll just need to restart the container for a specific device
- Integrated with Carina, Selenium, Appium
- plug-n-play - easily remove the mobile device when you need to use it in manual testing. When you’re done, connect it back - it’s up and running automatically
- Airplane and offline modes available
Pricing: From $ 3000/month for 10 devices to $11000 /month for 50 devices
Xamarin (by Microsoft)
Official Service Description: With Xamarin,'Ship apps your users will love. Find bugs before you ship with automated tests running on the devices your customers use. You can test everything your users can do, from pinching and swiping to double-tapping and scrolling. Find performance problems on any OS automatically with step-by-step memory and performance tracking. Beautiful reports make it easy to know when you’re ready for launch.'
Types of testing: manual, automated
Types of apps: native, hybrid
Devices: 3400+ real devices
Features:
- powerful testing framework Calabash is suggested to the customer for app automation purpose (Appium also supported)
- Interaction with app mimics user behavior ((double)tapping, swiping, pinching, etc.). You can build custom gestures programmatically
- Device features are available (Bluetooth, GPS, Wi-fi, physical buttons, camera, portrait/landscape models, simulating different battery conditions)
- Continuous tests are available via Visual Studio Team Services and integrate with TFS, Jenkins, TeamCity, etc.
- Detailed reports with results, full-time screenshots for every step and performance metrics (memory, CPU, duration, etc.) to find regressions and bottlenecks
- You can write tests in C# (Visual Studio), Ruby (Cucumber), Java (Appium), integration with NUnit test runner
- Cross-platform code reuse
- 3400+ real devices with approximately 100 devices added every month. The company is taking requests for devices. You can select devices by various criteria such as manufacturer, OS or even popularity in your target market
- The company offers expired automation engineers’ help to get started. You can also ask for building and maintaining a test suite specifically for your app
Pricing: Xamarin is now available through Visual Studio App Center, which is free for individuals, open source projects and small teams; pricing for enterprises - by request.
Kobiton
Official Service Description: 'Kobiton is the complete mobile device cloud platform you need to manage the devices you own and to access the real devices you want. Simple to use, easy to access from anywhere, and flexible enough to scale capacity across internal or external devices, Kobiton minimizes costs while increasing productivity. Use Kobiton to get your app to market faster.'
Types of testing: manual, automated
Types of apps: web, native, hybrid
Devices: Android, iOS, Windows (phones, tablets, phablets)
Features:
- Smart App directory tool to upload your app directly or through your build process to start testing in minutes
- detailed activity logs, Appium log, network activities, screenshots and videos to help identify and resolve issues faster
- Historial test execution sessions are stored for reports and bug submission
- Flexible test cloud - scalable testing capacity and customizable test cloud (a combination of an internal device with Kobiton cloud device)
- Device Lab Management feature allows for connecting internally
- Unlimited concurrent sessions are available
- Maximum size of the app is 500 Mb / no phone calls or SMS available
Pricing: Indie - $10/month for 100 device minutes (single user, access to all devices in the public cloud, minutes never expire); Business - $ 50/month (unlimited users, access to all devices in the public cloud, unlimited concurrent sessions, customizable plan); overage charges - $0.14/minute. Kobiton charges $0.10 per device minute, but you cannot purchase less than 100 minutes ($10) at a time.
Perfecto
Official Service Description: 'Release better digital apps faster. Our Continuous Quality Lab helps you easily mobilize your brand by perfecting the digital experiences that define it. Our 14 highly-secure data centers are home to thousands of real devices that are accessed from any browser - simply leave the device management to us. Perfecto has saved our customers over $156M each year through the elimination of manual testing.'
Types of testing: manual, automated
Types of apps: web, mobile, IoT
Devices: iOS, Android, Windows (mobile, tablet, phablet, desktop)
Features:
- All devices have calling, email, and web browsing capabilities as well as voice injection for voice-related applications
- Testing for real-world conditions - network, battery level, device orientation, app vitals and timers, incoming alerts, background running apps, etc.
- A company offers its own persona-based testing framework WindTunnel allowing to test against real-user conditions
- Integration with IDEs (Android Studio, Eclipse, IntelliJ, Visual Studio); CI (Bamboo, Circle CI, Jenkins, Microsoft Team Foundation (TFS); Perfecto REST API, TeamCity); Automated testing framework (Appium, Calabash, Espresso, Nightwatch, Protractor, Quantum, Selenium, WebdriverIO, XCUITest KIFF); Commercial automation tool integrations (CA Agile Requirements designer (ARD), CA Application Test, HP UFT, IBM Rational Test Workbench, SAP For Mobile, Tosca, Worksoft); APIs (Applitools, Jira, Perfecto REST API, Performance tools (LoadRunner, Neotys, HP BSM), Slack)
Pricing: free trial (12 devices for 2 hours of manual testing with no available integrations), basic - $1.188/year or $129/month (top 150 devices for manual testing with no available integrations), advanced - $3.588/year or $399/month (top 150 devices for automated testing, integration with HP ALM/UFT/LoadRunner/Sprinter and Microsoft Visual Studios/TFS), enterprise - by request (automated testing on private cloud, integration with HP ALM/UFT/LoadRunner/Sprinter, Microsoft Visual Studios/TFS, IBM Rational, Selenium WebDriver, Jenkins CI, HP BSM, Apache JMeter, Open REST API)
Sauce Labs
Official Service Description: 'Sauce Labs offers public and private Real Device Cloud for customers looking to expedite testing for their mobile apps without compromising quality. Sauce Labs Real Device Cloud (RDC) has thousands of Android and iOS devices for an intuitive manual and automated testing experience that is very easy to use.'
Types of testing: manual, automated
Types of apps: native, hybrid, mobile
Devices: Android, iOS (smartphones, tablets)
Features:
- supports Appium, Espresso, and Robotium
- Private device cloud offers a dedicated pool of devices with provided configuration, implementation and ongoing maintenance 24/7
- The selection of devices can be optimized for specific requirements (swiping devices from public cloud is available)
- Secure proxy connection and VPN tunneling
Pricing: testing on virtual platforms - Live from $19/month (for 1 concurrent session), automated - from $149/month, unlimited automated - from $ 298/month, enterprise - by request; real device testing - Live from $ 89/month (for 1 device); automated - from $ 349/month, unlimited automated - from $600/month, enterprise - by request.
free trial for 14 days with 100 minutes of automated and 60 minutes of manual testing
Browser Stack
Official Service Description: Browser Stack offers 'Live, web-based browser testing. Instant access to all real mobile and desktop browsers. Say goodbye to your lab of devices and virtual machines.'
Types of testing: automated
Types of apps: native, hybrid
Browsers: latest versions of IE, Edge, Safari, Chrome, Firefox, Opera, Yandex, etc. (1100+ mobile and desktop browsers)
Devices: Android, iOS (phones, tablets)
Features:
- 1000+ real devices (iOS and Android phones and tablets)
- Tests are run concurrently across devices for quicker results. Test execution time can be reduced by more than 10x by using parallel tests
- BrowserStack is integrated with different frameworks and languages supported by Appium
- An encrypted, secure tunnel is available for testing apps on internal development and staging environments with no setup required
- Instant debugging by using text logs, video recordings, and screenshots of the test run
- REST API can be used to query the most recent builds or to upload a new build
Pricing: Browser: Live - from $29/month; Automate Pro - from $99/month; Automate Mobile - $149/month; Enterprise - by request.
Mobile: App Live - from $49/month; App Automate - from $199/month; Freelancer - from $12.50/month; Enterprise - by request
CrossBrowserTesting (by SmartBear)
Official Service Description: 'Explore the only web testing platform built for the entire testing process. Developers and testers have run millions of tests on our real desktop & mobile browsers. See what CrossBrowserTesting can do for your team. Make every web experience perfect, on any browser or mobile device. Ditch your VMs and Device Lab. Easily run manual, visual, and Selenium tests in the cloud on 1500+ real desktop and mobile browsers.'
Types of testing: automated (Selenium (Java, JavaScript, Python, C#, Ruby, PHP available), Appium, Watir, Cucumber, etc.), manual
Types of apps: web (browser)
Browsers: Chrome, Firefox, Opera, Safari, Camino, IE, Edge, Netscape, Mozilla, AOL Explorer, Mobile: Dolphin, Maxthon, UC Browser, Sleipnir, Opera Mini, Samsung Browser, Android Browser, Mobile Safari, Chrome Mobile
Devices: Android, iOS (smartphones and tablets)
Features:
- live testing on real devices with native debugging tools (FireBug, Chrome Dev Tools, etc.)
- local testing via encrypted tunnel is available for live testing or screenshots against any of the access internal sites
- Integrated with: Frameworks - Appium, Behat, Codeception, Cucumber, InternJS, Karma, Lettuce, Mocha, GitHub, Nightwatch.JS, Protractor, SpecFlow, Watir, Webdriver.IO; CI/CD - Jenkins CI, TeamCity, VS Team Services; Communication - Slack, HipChat, JIRA
Pricing: Live - from $29/month (unlimited testing and screenshots on real devices for 1 user), Automated - from $60/month (1 parallel live test/ 1 parallel automated test with 1000 automation minuses for 1 user), Unlimited - from $100/month (2 parallel live tests, 1 parallel automated test with unlimited device minutes for 2 users)
SIGOS
Official Service Description: 'We help businesses deliver better apps their customers love. SIGOS App Experience is a comprehensive, fully customizable suite of testing and monitoring solutions using real devices available worldwide, anytime and anywhere. Test and monitor any type of mobile app - native, hybrid, or web - with flexible deployment options in the cloud or on-premise. Enterprise-class monitoring enables app performance optimization across devices and detects issues early on 24/7. Measure, report, and identify trends on key transactions in your application. Together, reach your business’ fullest potential by delivering the best-in-class apps for your users.'
Types of testing: manual, automated
Types of apps: native, hybrid, web
Devices: iOS, Android smartphones, and tablets (the company is taking requests; feature phones are also available)
Features:
- Manual testing allows to mimic user interaction with the touchscreen, swipe, battery life, and video, switching between real networks and wi-fi, send/receive text messages, manually interact with an app, capture test results, videos, and screenshots
- Test automation allows to run multiple tests simultaneously on multiple devices, hundreds of test cases are available, right away or scheduled execution, the company also offers their own Java API for device interaction
- Performance monitoring is available for checking the app’s reliability, competitiveness, and to ensure SLA compliance with the option of getting immediate alerts and insights and monitoring apps’ transactions
- Comprehensive cloud library of real mobile devices with day-one support of newly released devices
- Global market coverage with real mobile carrier networks and wi-fi
- Flexible deployment: public cloud, private cloud, and on-premise
- Public cloud is a cost-effective solution to test apps remotely, with reserved devices for specific timeslots and full device hardware control
- Private cloud is a dedicated device pool reserved especially for the customer. The platform is accessible anytime and anywhere through secure and centralized connections with devices
- On-premise is an option for securely installing device cloud behind the corporate firewall. This particular option is convenient for companies that meet strict security compliance requirements and need all the testing conducted locally.
- Using a camera is available, as well as capturing audio and visual
Pricing: free trial for 3 hours over 7 days; Manual testing from $15/hr (min. 10 hrs/month); Test automation from $14/hr; performance monitoring - by request.
It’s hard to keep up with all the platform updates and new devices constantly flowing to the market and addressing the widest-ever diversity of needs is pretty challenging
Skilled teams of testers and engineers are ready to meet those needs for reasonable prices and help you deliver better apps faster.
Public, Private, and On-Premise Clouds
AWS Device Farm | MCloud (Solvd) | Sauce Labs | SIGOS | |
Type(s) of cloud | Public, Private | Private, on-premise | Public, Private | Public, Private, on-premise |
Type(s) of testing | Manual, Automated | Manual, Automated | manual, automated | Manual, Automated |
Devices | 376 non-rooted devices: Android and iOS (phones, tablets, phablets) | Up to the client’s needs; 50+ real devices with WEB access (Android, iOS) | Approx. 400 Android and iOS devices (phones, tablets) | iOS, Android (smartphones and tablets ) |
Free Trial | 1000 device minutes | Limited (fitted for specific project needs) | 14 days with 100 min (automated)/ 60 minutes (manual) | 3 hrs/day for 7 days |
Pricing | $0.17 per device minute (pay-as-you-go). unlimited testing for a flat fee of $250/mo per device; private devices start from $200/mo | $300/mo per dedicated device (multiple threads available) | Live - from $ 89/mo, Automated - from $349/mo, Unlimited automated - from $600/mo | 15 $/hr - manual testing; 14 $/hr - test automation, performance monitoring - by request |
Public Clouds
Xamarin (Microsoft) | Firebase Test Lab (Google) | Kobiton | Perfecto | BrowserStack | CrossBrowser Testing (SmartBear) | |
Type(s) of cloud | Public | Public | Public | Pubic | Public | Public |
Type(s) of testing | manual, automated | manual, automated | manual, automated | manual, automated | automated | manual, automated |
Devices | 3400+ devices - phones, tablets (Android, iOS) | iOS, Android (real devices and emulators) | Android, iOS, Windows (phones, tablets, phablets) | 3000+ real Android and iOS devices | Android, iOS phones and tablets | Android, iOS phones and tablets |
Free trial | 90 min for 30 days | Free limited plan | 120 device min on 30 devices for 15 days | 2 hours | By request | 7 days |
Pricing | Free for individuals and small startups through Microsoft Visual Studio | Spark - Free plan, Flame - $25/mo, Blaze - pay-as-you-go | Indie - $10/mo for 100 device minutes (single user); Business - $ 50/mo (unlimited users/concurrent sessions); overage charges - $0.14/min | Basic - from $ 99/mo/user, Advanced - from $299/mo/user, Enterprise - by request | Browser: Live - from $29/mo; Automate Pro - from $99/mo; Automate Mobile - $149/mo; Enterprise - by request. Mobile: App Live - from $49/mo; App Automate - from $199/mo; Freelancer - from $12.50/mo; Enterprise - by request | Live - from $29/month (1 user), Automated - from $60/month (1 parallel live test/ 1 parallel automated test with 1000 automation minuses for 1 user), Unlimited - from $100/month (2 parallel live tests, 1 parallel automated test with unlimited device minutes for 2 users) |
Like This Article? Read More From DZone
Published at DZone with permission of Alexandra Gorobets . See the original article here.
Opinions expressed by DZone contributors are their own.