Thursday, 8 September 2011

Testing Models


Scrum Development Process

Scrum is an agile method for project management. Scrum is a method for managing work and achieving very high productivity. A popular agile method for project management, Scrum is noted for its simplicity, its high level of transparency, and a team based approach to work.
Characteristics of Scrum
a. A product backlog of prioritized work to be done;
b. Completion of a fixed set of backlog items in a series of short iterations or sprints;
c. A brief daily meeting or scrum, at which progress is explained, upcoming work is described and impediments are raised.
d. A brief sprint planning session in which the backlog items for the sprint will be defined.
e. A brief sprint retrospective, at which all team members reflect about the past sprint.
Following are some practices for the Scrum:
a. Customers become a part of the development team. (i.e., Customer must be genuinely interested in the output.)
b. Frequent intermediate deliveries with working functionality.
c. Frequent risk and mitigation plans developed by the development team itself. Make it live, and continuous activity.
d. Daily status discussion with the team. – Standup meetings.
e. A daily discussion asking each team member:
1. What have you done since yesterday?
2. What are you planning to do by tomorrow?
3. Do you have any problems preventing you from accomplishing your goal?
f. Transparency in planning and module development – Let everyone know, who is accountable for what and by when.
g. Workplaces and working hours must be energized. – "Working more hours" does not necessarily mean "producing more output."
Waterfall Model
This is the most common and classic of life cycle models, also referred to as a linear-sequential life cycle model. It is very simple to understand and use. In a waterfall model, each phase must be completed in its entirety before the next phase can begin. At the end of each phase, a review takes place to determine if the project is on the right path and whether or not to continue or discard the project.
Requirement (Diagram)
a. Design
b. Implementation & Unit Testing
c. Integration & System Testing
d. Operation
Advantages
a. Simple and easy to use.
b. Easy to manage due to the rigidity of the model – each phase has specific deliverables and a review process.
c. Phases are processed and completed one at a time.
d. Works well for smaller projects where requirements are very well understood.
Disadvantages
a. Adjusting scope during the life cycle can kill a project
b. Poor model for complex and object-oriented projects.
c. Poor model for long and ongoing projects.
d. Poor model where requirements are at a moderate to high risk of changing.
V-Shaped Model
Just like the waterfall model, the V-Shaped life cycle is a sequential path of execution of processes. Each phase must be completed before the next phase begins. Testing is emphasized in this model more so than the waterfall model though. The testing procedures are developed early in the life cycle before any coding is done, during each of the phases preceding implementation.
Requirements begin the life cycle model just like the waterfall model. Before development is started, a system test plan is created. The test plan focuses on meeting the functionality specified in the requirements gathering.
The high-level design phase focuses on system architecture and design. An integration test plan is created in this phase as well in order to test the pieces of the software systems ability to work together.
The low-level design phase is where the actual software components are designed, and unit tests are created in this phase as well.
The implementation phase is, again, where all coding takes place. Once coding is complete, the path of execution continues up the right side of the V where the test plans developed earlier are now put to use.
Advantages
a. Simple and easy to use.
b. Each phase has specific deliverables.
c. Higher chance of success over the waterfall model due to the development of test plans early on during the life cycle.
d. Works well for small projects where requirements are easily understood.
Disadvantages
a. Very rigid, like the waterfall model.
b. Software is developed during the implementation phase, so no early prototypes of the software are produced.
c. Model doesn’t provide a clear path for problems found during testing phases.
Spiral Model
The spiral model gives more emphases placed on risk analysis. The spiral model has four phases: Planning, Risk Analysis, Engineering and Evaluation. A software project repeatedly passes through these phases in iterations (called Spirals in this model). The baseline spiral, starting in the planning phase, requirements are gathered and risk is assessed. Each subsequent spirals builds on the baseline spiral.
Requirements are gathered during the planning phase. In the risk analysis phase, a process is undertaken to identify risk and alternate solutions. A prototype is produced at the end of the risk analysis phase.
Software is produced in the engineering phase, along with testing at the end of the phase. The evaluation phase allows the customer to evaluate the output of the project to date before the project continues to the next spiral.
In the spiral model, the angular component represents progress, and the radius of the spiral represents cost.
Advantages
a. High amount of risk analysis.
b. Good for large and mission-critical projects.
c. Software is produced early in the software life cycle.
Disadvantages
a. Can be a costly model to use.
b. Risk analysis requires highly specific expertise.
c. Project’s success is highly dependent on the risk analysis phase.
d. Doesn’t work well for smaller projects.

Software Testing Techniques


Black Box Testing Techniques.
a. Equivalence Partitioning.
b. Boundary Value Analysis.
c. Cause-Effect Graphing.
d. Error-Guessing.

Equivalence Partitioning

Equivalence partitioning is a software testing related technique with the goal: 
1. To reduce the number of test cases to a necessary minimum. 
2. To select the right test cases to cover all possible scenarios. 
Following example of a function has the pass parameter "month" of a date. The valid range for the month is 1 to 12, standing for January to December. This valid range is called a partition. In this example there are two further partitions of invalid ranges. The first invalid partition would be <= 0 and the second invalid partition would be >= 13. 

........ -2 -1 0 1 ....................... 12 13 14 15 ............... 
------------------------------------------------------ 
invalid partition 1 valid partition invalid partition 2 

It is sufficient to select one test case out of each partition to check the behavior of the program. The values within one partition are considered to be "equivalent". Thus the number of test cases can be reduced considerably. Equivalence partitioning is no stand alone method to determine test cases. It has to be supplemented by boundary value analysis. 

Boundary Value AnalysisBoundary value analysis is a software testing related technique to determine test cases covering known areas of frequent problems at the boundaries of software component input ranges. 

To set up boundary value analysis test cases you first have to determine which boundaries you have at the interface of a software component. This has to be done by applying the equivalence partitioning technique. Boundary value analysis and equivalence partitioning are inevitably linked together. For the example of the month in a date you would have the following partitions: 

......... -2 -1 0 1 ...................... 12 13 14 15 ..... 
-------------- --------------------------------------- 
invalid partition 1 valid partition invalid partition 2 

Applying boundary value analysis you have to select now a test case at each side of the boundary between two partitions. The boundary value analysis can have 6 text cases: n,n-1,n+1 for the upper limit and n,n-1,n+1 for the lower limit. 

 
White-Box testing techniques
a. Statement coverage
b. Decision coverage
c. Condition coverage
d. Decision-condition coverage
e. Multiple condition coverage
f. Basis Path Testing
g. Loop testing
h. Data flow testing

GUI Testing

GUI testing is a process to test application's user interface and to make sure that it confirms the design requirements.


1. Text Box
a. Move the Mouse Cursor over all Enterable Text Boxes. Cursor should change from arrow to Insert Bar.
b. If it doesn't then the text in the box should be grey or non-updateable.
c. Try to overflow the text by typing to many characters.
d. Enter invalid characters - Letters in amount fields, try strange characters like + , - * etc. in All fields.
e. SHIFT and Arrow should Select Characters. Selection should also be possible with mouse. Double Click should select all text in box.

 
2. Radio Button: 
Left and Right arrows should move ON Selection. So should Up and Down. Select with mouse by clicking. 
Check Boxes: Clicking with the mouse on the box, or on the text should SET/UNSET the box. SPACE should do the same.

 
3. Command Buttons
a. If Command Button leads to another Screen, and if the user can enter or change details on the other screen then the Text on the button should be followed by three dots.
b. All Buttons except for OK and Cancel should have a letter Access to them. This is indicated by a letter underlined in the button text. The button should be activated by pressing ALT+Letter. Make sure there is no duplication.
c. Click each button once with the mouse - This should activate
Tab to each button - Press SPACE - This should activate
Tab to each button - Press RETURN - This should activate
d. If there is a Cancel Button on the screen , then pressing should activate it.

 
4. Aesthetic Conditions:
a. Is the general screen background the correct colour?
b. Are the field prompts the correct colour?
c. Are the field backgrounds the correct colour?
d. In read-only mode, are the field prompts the correct colour?
e. In read-only mode, are the field backgrounds the correct colour?
f. Are all the screen prompts specified in the correct screen font?
g. Is the text in all fields specified in the correct screen font?
h. Are all the field prompts aligned perfectly on the screen?
i. Are all the field edit boxes aligned perfectly on the screen?
j. Should the screen be resizable?
k. Should the screen be minimisable?
l. Is all user input captured in UPPER case or lower case consistently?

 
5. Validation Conditions:
a. Does a failure of validation on every field cause a sensible user error message?
b. Is the user required to fix entries which have failed validation tests?
c. Have any fields got multiple validation rules and if so are all rules being applied?
d. If the user enters an invalid value and clicks on the OK button (i.e. does not TAB off the field) is the invalid entry identified and highlighted correctly with an error message.?
e. Is validation consistently applied at screen level unless specifically required at field level?
f. For all numeric fields check whether negative numbers can and should be able to be entered.
g. For all numeric fields check the minimum and maximum values and also some mid-range values allowable?
h. For all character/alphanumeric fields check the field to ensure that there is a character limit specified and that this limit is exactly correct for the specified database size?
i. Do all mandatory fields require user input?
j. If any of the database columns don't allow null values then the corresponding screen fields must be mandatory. (If any field which initially was mandatory has become optional then check whether null values are allowed in this field.)

6. Usability Conditions:
a. Is all date entry required in the correct format?
b. When an error message occurs does the focus return to the field in error when the user cancels it?
c. Do all the fields edit boxes indicate the number of characters they will hold by there length? e.g. a 30 character field should be a lot longer

7. Data Integrity Conditions:
a. Check the maximum field lengths to ensure that there are no truncated characters?
b. Check maximum and minimum field values for numeric fields?
c. If numeric fields accept negative values can these be stored correctly on the database and does it make sense for the field to accept negative numbers?

8. Date Field Checks
a. Assure that leap years are validated correctly & do not cause errors/miscalculations
b. Assure that month code 00 and 13 are validated correctly & do not cause errors/miscalculations
c. Assure that 00 and 13 are reported as errors
d. Assure that day values 00 and 32 are validated correctly & do not cause errors/miscalculations
e. Assure that Feb. 28, 29, 30 are validated correctly & do not cause errors/ miscalculations
f. Assure that Feb. 30 is reported as an error
g. Assure that century change is validated correctly & does not cause errors/ miscalculations

9. Alpha Field Checks
a. Use blank and non-blank data.
b. Include lowest and highest values.
c. Include invalid characters & symbols.
d. Include valid characters.
e. Include data items with first position blank.
f. Include data items with last position blank.

Testing Concepts


1. SDLC
a. Feasibility study
b. Requirements analysis
c. Systems design: Describes desired features and business rules
d. Implementation: The real code is written here.
e. Integration and testing: Brings all the pieces together
f. Acceptance, installation, deployment
g. Maintenance

2. STLC
a. Requirements Analysis
b. Test Planning
c. Test Development
d. Test Execution
e. Bug Reporting and Result Analysis
f. Defect Retesting and Regression testing

a. Table of Contents
b. Objective of testing
c. Personnel responsible for each task
d. Personnel contact-info
e. Relevant naming conventions
f. Features to be Tested, Features not to be Tested
g. Assumptions and dependencies
h. Test environment - hardware, operating systems, Browsers etc
i. Project risk analysis
j. Initial smoke testing period and criteria
k. Process used to manage the bugs
l. Test suspension and restart criteria
m. Duration of test


a. Component Name
b. Prerequsitie
c. Test Cases Steps
d. Expected Result
e. Status
f. Comments

5. Bug Life Cycle
a. New
b. Assigned
c. Resolved
d. Verified
e. Closed
f. Reopen

6. Testing Techniques (Testing approach)
The most popular Black box testing techniques are:
1. Equivalence Partitioning
2. Boundary Value Analysis
3. Error-Guessing

1. Bug Tracking Tool: Bugzilla
2. Recording the bug in video format: Screencast-o-matic
3. Tool for taking Screen-shots for reporting the bug: Fireshot
4. Compatibility testing tools: Browser Sandbox and Adobe BrowserLab
5. Tool to Monitor CSS and HTML: Firebug
6. Tool to ensure valid HTML: HTML Validator
7. Performance Testing tool: Page Speed by Google
8. To find Broken Links: Pinger Ad-on and brokenlinkcheck
9. For checking the spelling of content: Spell Check

8. Severity: Determines the defect's effect
Priority: Determines the defect urgency

9. Hypertext Transfer Protocol Secure (HTTPS) is a combination of the Hypertext Transfer Protocol with the SSL/TLS protocol to provide encrypted communication and secure identification of a network web server.

10. Testing Process

a. Content should be correct.
b. Wrap-around properly.
c. Switch images off for ALT texts
d. Switch JavaScript off and See if appropriate messages are displayed to user.
e. Check sensible page titles.

a. Participation of QA and Designers
b. Participation of sample end user
c. Observation by test morderator
d. Development of research questions
e. Recomendation of improvements to the design of the product

c. Functional Testing
a. Check for broken links.
b. Validate the HTML (Validator W3).

Functional testing types:
a. Smoke testing / Sanity testing
b. Usability Testing
c. Regression Testing
d. Pre User Acceptance Testing which includes Alpha and Beta
e. User Acceptance Testing
f. Localization Testing

Non-functional testing types:
a. Load, Stress, Performance, Volume Testing
b. Compatibility
c. Security Testing
d. Installation Testing

d. Interface
a. Data display on browser should match with data available on server.

e. Compatibility
a. Windows
b. Browsers

f. Security Testing
a. Limit the number of tries.
b. Verify rules for password selection.
c. Is there a timeout limit?
d. Paste internal url directly without log-in
e. Test if SSL is used for security measures.
f. Type a single quote
g. SQL Injection: Try submitting following: (') single quote
h. Session hijacking: If your app has a session identifier number in the URL decrease that number

a. Can your site handle a large amount of users requesting a certain page.
b. Long period of continuous use: Is site able to run for long period, without downtime.
c. Apply Web page performance (speed) Tool
  • Performance Testing: The goal of performance testing is not to find bugs, but to eliminate bottlenecks and establish a baseline for future regression testing.
  • Load test: To verify application behavior under normal and peak load conditions
  • Stress Testing: To determine application’s behavior when it is pushed beyond normal or peak load conditions.
    Here are some ways in which stress can be applied to the system:
    a. double the baseline number for concurrent users/HTTP connections
  • Volume testing:
    a. testing a word processor by editing a very large document
    b. testing a printer by sending it a very large job
  • Capacity test: To determine how many users and/or transactions a given system will support and still meet performance goals.
If chair is designed for 100 kg weight, and my weight is 70 kg then that testing is called as normal testing. If my weight is 100 kg then that testing is called as load testing. If my wt is 120 kg then that testing called as stress testing.
11. Smoke testing A smoke test is a cursory examination of all of the basic components of a software system to ensure that they work. Typically, smoke testing is conducted immediately after a software build is made. It ensures that the future testing is not blocked. 

12. System testing 
Testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing is actually done to the entire system against the Functional Requirement Specification(s) (FRS) and/or the System Requirement Specification (SRS).
Types Of System Tests:
a. Functional testing 
b. User interface testing 
c. Usability testing 
d. Compatibility testing 
e. Security testing 
f. Performance testing 
g. Sanity testing 
h. Regression testing 
i. Installation testing


13. Defect or Bug density: Defect density is equal to the ratio of number of defects to the number of lines of code.

DD=Total Defect/KLOC( Kilo lines of Code)
Ex: Suppose 10 bugs are found in 1 KLOC
Therefore DD is 10/KLOC (Kilo lines of code)

14. 
Traceability Matrix - Requirement Traceability matrix (RTM)
Traceability matrix is a matrix which is used to keep track of the requirements. It is a mapping between the requrements and test cases, we will do this for identify missing test cases.

Exact requirements from the requirement doc given by the client are copied in this matrix. These requirements are assigned a unique number and the remark as testable or not. Against each testable requirement test objective and test case is identified. It is highly possible that for one req there could be multiple test objectives and test cases. For each of the test objective and test case unique number is assigned.

Advantages:
a. We can trace the missing test cases.
b. Whenever requirements changes then we can easily refer to matrix document, change the usecase and go to corresponding testcases and change them.
c. Easy to test any functionality. Only we need to refer matrix document and we can reach to related test cases.
d. We can trace the impact of functionalities on one another. Because different functionalities can have same test cases. 


15. Bug: A software bug is the common term used to describe an error, mistake, failure, or fault in a computer program.

16. Monkey Testing
Monkey Testing is random testing.
a. Dumb Monkeys: A dumb monkey doesn't know anything about the software being tested; it just clicks or types randomly.

b. Semi-Smart Monkeys: Add logging to your monkey so that everything it does is recorded to a file. When the monkey finds a bug, you need only to look at the log file to see what it was doing before the failure.
Another solution to track what your monkey does is to set up a video camera to record what happens on the screen. When you notice that the software has failed, just rewind and replay the tape.

c. Smart Monkeys: A true smart monkey knows Where he is, What he can do there, Where he can go, Where he's been, If what he's seeing is correct.
A smart monkey isn't limited to just looking for crashing bugs, either. It can examine data as it goes, checking the results of its actions and looking for differences from what it expects.

17. Web Testing: Web testing nothing but Browser applications testing. They are usually a most visible, widely-used - and potentially most vulnerable - applications.

18. Server Side Interface: In web testing the server side interface should be tested. This is done by verify that communication is done properly. Compatibility of server with software, hardware, network and database should be tested.

19. Client Side Compatibility: The client side compatibility is also tested in various platforms, using various browsers etc.
 

20. Desktop – Client server – Web Applications

Desktop application runs on personal computers and work stations.
Client server application you have two different components to test.  Application is loaded on server machine while the application (exe) on every client machine. 
Web application: Application is loaded on the server whose location may or may not be known and no exe is installed on the client machine, you have to test it on different web browsers and different Operating Systems.

21. Why are there so many software bugs?
a. Unclear software requirements because there is miscommunication as to what the software should or shouldn’t do.
b. Software complexity.
c. Programming errors occur because programmers and software engineers, like everyone else, can make mistakes.
d. Changing requirements.

Testing on Smartphones


1. About smartphones:
Mobile Phone: A mobile phone is more frequently called a cellular phone or cellphone. Most mobile phones provide voice communications, Short Message Service (SMS), Multimedia Message Service (MMS).










PDA: Personal Digital Assistant (PDA) is a handheld computer for managing contacts, appointments and tasks. Wireless PDAs may also offer e-mail and Web browsing, and data are synchronized between the PDA and desktop computer via USB or wireless.










Smartphone: A smartphone is considered to be the combination of the traditional PDA and cellular phone, with a bigger focus on the cellular phone part. Smartphones allow users to store information, e-mail, install programs, along with using a mobile phone in one device.










Some Popular Examples:
a. The popular Apple iPhone is a Smartphone and is designed and marketed by Apple Inc. The first iPhone was introduced on January 9, 2007. An iPhone functions as a camera phone, including text messaging and visual voice-mail, a portable media player, and an Internet client, with e-mail, web browsing, and Wi-Fi connectivity.











iPhone Features: FaceTime, Camera, Mail, Photos, Messages, App Store, Retina display, Folders, safari, AirPrint, Maps + Compass, iTunes Store, iMovie (from app store), Multitasking, Game center, iPod, AirPlay, Keyboard, Accessibility, iBooks (from app store), HD Video Recording, Phone, Home Screen, Voice Control, Search, Find My iPhone.

b. The iPad (PDA) is a tablet computer designed, developed and marketed by Apple primarily as a platform for audio-visual media including books, periodicals, movies, music, games, and web content. Apple released the iPad in April 2010. The iPad runs the same operating system as the iPod Touch and iPhone. Without modification, it will only run programs approved by Apple and distributed via its online store.






iPad Features: Safari, Mail, Photos, Videos, You Tube, iPod, iTunes, App Store, Maps, Notes, Calendar, Contacts, Home Screen, Multitasking, Folders, Airprint, Airplay, game Center, Spotlight search, Accessibility, Find My iPad, iBooks (from app store).

c. iPod (such as the newer iTouch are PDA's) is a portable media player designed and marketed by Apple and launched on October 23, 2001.








iPod Features: FaceTime, Music, mail, Voice Control, AirPrint, Find My iPod touch, Retina display, Movies and TV shows, Safari, Maps, AirPlay, HD Video Recording, App Store, Photos, You Tube, Voice Memos, Game center, iTunes, Home Screen, Nike _ iPod, Accessibility, iBooks (from app store).

d. The HP iPAQ Mobile Messenger is a Smartphone.











e. The LG Prada is a cellular phone with a touch screen — but its not a smartphone


f. The RIM BlackBerry 8800 is considered a smartphone











g. The Palm Treo 700p is a PDA phone











h. The Motorola Q is also considered to be a PDA phone



2. Operating systems for Smartphones



















a. Android operating system: Android is a mobile operating system initially developed by Android Inc. Android was purchased by Google in 2005. Android operating system is currently being run by mobile phones built by a variety of different companies, including Samsung, Acer, Motorola, and HTC.

b. iPhone OS: iOS (known as iPhone OS prior to June 2010) is Apple's mobile operating system. Iphone OS is the mobile phone version of Mac OSX.

c. Palm: The palm OS (also known as Garnet OS) is a mobile operating system that was designed by Palm, Inc., in 1996. In 2009, the main licensee of Palm OS, Palm, Inc., switched from Palm OS to webOS for their forthcoming devices. HP webOS is a proprietary mobile operating system running on the Linux kernel, initially developed by Palm and purchased by Hewlett Packard (HP) in 2010.

d. Blackberry OS: Blackberry OS is developed and designed by Canadian company Research In Motion (RIM) since 1999. One bad side is that it limits one to viewing MS Office documents only.

e. Windows Phone 7 series (Windows Mobile): Windows Mobile is a mobile operating system developed by Microsoft that was for use in Smartphones and mobile devices. Windows Phone 7 mobile operating system is the successor to its Windows Mobile platform. It launched in Europe, Singapore and Australia on October 21, 2010 with Asia to follow in 2011.

f. Symbian: Symbian is an operating system for smartphones and mobile devices designed by Symbian Ltd. Nokia acquired Symbian Software Limited in 2008. Symbian OS was made available in 2010 as open source software.

3. TESTING on Smartphones

1. Installation testing: Tester’s need to focus on installation & uninstallation of the app (remember, user will install the app through Market place).

2. Requirement Functionality Testing: Test for the application functionality according to designed test cases.

3. Widget testing: Test the widget by adding widget, using widget as per functionality i.e. music widget should play music, deleting widget.

4. Phone Interrupt testing: Test the application behavior on phone call, SMS & other notifications.

5. Stress testing (monkey test): There is a command in android where it generates series of events at different frequencies. Make sure our app passes this monkey test. The command used for generating event is:
$ adb shell monkey [options] <event-count>
(http://developer.android.com/guide/developing/tools/monkey.html)

6. UI testing: All mobile platforms have certain submission guidelines to follow before the application can be available commercially. Apple AppStore is very strict on guidelines. Most of the applications submitted are rejected due to small errors on AppStore. We need to verify (UI test) that application is meeting guidelines:


a. Apple Guidelines
b. Android Guidelines

7. Compatibility Test: Developers do the unit testing on the emulators. Testers need to test on actual environment i.e. devices. There are multiple devices of different resolutions & having various OS. We need to check for compatibility on maximum devices.

8. Following are some basic scenarios which should be considered while testing mobile application:

a. UI of the app is as per the screen size of the device, no text/control should be cutting off.
b. Text should be readable.
c. Suspend/Resume (Call/SMS/Alarm) when app is running.
d. If app has timer/sound then Suspend/Resume when timer/sound is active.
e. Behavior of app on Flip/Slider close.
f. Behavior of app when memory of the device is almost full (very critical).
g. Behavior of when network is not available.
h. Backup and restore
i. Behavior of app if app is running for longer period of time
j. Behavior of app when keys are pressed randomly.
k. If app support app/port directed SMS then behavior on receiving such SMS.

9. Performance tests
a. System tests (booting the phone, or resuming from standby)
b. User interface tests (rotating the screen, displaying a menu)
c. Network (downloading a Web page or application)
d. Media (starting a video)
e. Phone (calling a contact)
f. Application (displaying the day’s calendar, viewing the photo folder)
g. File system (reading an mp3 file)

10. Testing Checklist for Mobile Applications
http://mobileappstesting.com/2009/10/23/testing-checklist-for-mobile-applications/

11. iPhone Testing: Applications for the iPhone are written in Objective-C. Objective-C is an object-oriented programming language which is very similar the C programming language.

To write iPhone applications the iPhone SDK (software development kit) is needed which uses Xcode (Mac OS X development software) as the development environment for the programming and debugging, and an iPhone simulator to test the program.

Testing: As with all software, thorough testing is essential before release. However, all faults will not be found as they may be in rarely used sections of the application, but there is a limit to how much time is spent on finding faults because removing a high percentage of faults may only improve the application by a very small percentage.

The interfaces and the main functionality must be tested in various ways to make sure that the chance of faults that would cause a failure in the system is low. This testing will be carried out in three different ways, first, the system will be tested on a simulator, then by experienced iPhone users and finally by inexperienced iPhone users. The use of inexperienced and experienced users will test the interface for ease of use.

Resources Required:
iPhone SDK – needed for iPhone API incorporation
iPhone Simulator – needed to test application
iPhone Developer Membership – needed to test application on an iPhone
Apple Mac – iPhone Simulator and SDK only designed to work on Mac OS

4. Why testing should be done on actual device?

a. Resource availability: Simulator may have access to more or less resources, and the speed at which those resources become available is liable to be very different.

b. Program performance: The simulator may run at very different speeds to the device.

c. Communications availability: If app communicates over 3g/GPRS/WAN/Bluetooth, simulators tend to simulate in such a way that they always work, whereas in the real world you get outages, performance drops, etc.

d. Hardware exceptions: how does your code handle low battery, power-offs, insufficent memory, etc.

e. Given you're only dealing with a service, you are possibly not worried about the user experience so much, but you really need to carry out usability testing on the device in its target environment, rather than at your desk.

f. Some of the UI elements looks different in simulator and in emulator.

Issues with different Simulators:
One of the guy has posted following points on a testing forum:
1. Issues with Blackberry Simulator
a. Blackberry simulator is extremely slow
b. mouse does not work while working on Blackberry simulator
c. it crashes a number of times
d. impossible to close the application (Have to kill it via task manager)

2. Issues with Android Simulator
a. Android is definitely faster than Blackberry simulator - but the smaller screen of the phone makes things a little impossible to test the entire application
b. We cannot zoom in once we have the modal window
c. In the original zoomed out size, we can hardly see anything for testing a specific feature. I was trying to search the net to extend the simulator for better testing- but do not really think that’s possible

3. Simulator for Ipad and Iphone
a. They are extremely slow.

5. Problems occur while doing Testing on Mobile Applications:

a. Mobile software is written on an emulator and eventually runs on the actual device.
b. There are less or more subtle differences between emulators and devices (and in particular across devices from different vendors) in terms of supported APIs, hardware and Java Virtual Machine implementation details.

c. Uploading and testing the application on each and every device is a tedious, time consuming process.

6. Tools used

1. Simulator [Blackberry]

2. Simulator [iPhone]

3. Simulator [iPhone]

4. Demo [Tool]

5. Simulator [Windows Mobile]