读书笔记 - How Google Test Software

<How Google Test Software>(《谷歌如何测试软件》)的确为神秘谷歌公司揭开一层面纱，讲到了谷歌的代码文化和测试文化，讲到了角色划分，职责划分，测试种类划分，讲到优秀的不同角色的人应该具有什么样子的，讲到测试的创新和工具，还有大量的人物访谈。

这里的笔记主要包含：个人感兴趣的，值得备忘的，需要后续关注的东西记录。测试和质量都不仅仅是是测试人员的事情，还需要团队，公司的支持。

谷歌内部开发和测试角色和职责范围

The software engineer (SWE) is the traditional developer role. SWEs write functional code that ships to users. They create design documentation, choose data structures and overall architecture, and they spend the vast majority of their time writing and reviewing code. SWEs write a lot of test code, including test-driven design (TDD), unit tests, and, as we explain later in this chapter, participate in the construction of small, medium, and large tests. SWEs own quality for everything they touch whether they wrote it, fixed it, or modified it. That's right, if a SWE has to modify a function and that modification breaks an existing test or requires a new one, they must author that test. SWEs spend close to 100 percent of their time writing code.

The software engineer in test (SET) is also a developer role, except his focus is on testability and general test infrastructure. SETs review designs and look closely at code quality and risk. They refactor code to make it more testable and write unit testing frameworks and automation. They are a partner in the SWE codebase, but are more concerned with increasing quality and test coverage than adding new features or increasing performance. SETs also spend close to 100 percent of their time writing code, but they do so in service of quality rather than coding features a customer might use. 

The test engineer(TE) is related to the SET role, but it has a different focus. It is a role that puts testing on behalf of the user first and developers second. Some Google TEs spend a good deal of their time writing code in the form of automation scripts and code that drives usage scenarios and even mimics the user. They also organize the testing work of SWEs and SETs, interpret test results, and drive test execution, particularly in the late stages of a project as the push toward release intensifies. TEs are product experts, quality advisers, and analyzers of risk. Many of them write a lot of code; many of them write only a little.

测试组织结构和流程

测试尽早参与，各个环节参与，多Review文档，代码，架构。Code Review 是专门有一套Submit的流程。

高度自动化，强调持续集成。

测试分大中小测试，大中小范围、执行人、时间和要求不一样。

测试度（测试成熟度）的等级

Summary of Test Certified Levels
Level 1

Set up test coverage bundles.
Set up a continuous build.
Classify your tests as Small, Medium, and Large.
Identify nondeterministic tests.
Create a smoke test suite.

Level 2

No releases with red tests.
Require a smoke test suite to pass before a submit.
Incremental coverage by all tests >= 50%.
Incremental coverage by small tests >= 10%.
At least one feature tested by an integration test.

Level 3

Require tests for all nontrivial changes.
Incremental coverage by small tests >= 50%.
New significant features are tested by integration tests.

Level 4

Automate running of smoke tests before submitting new code.
Smoke tests should take less than 30 minutes to run.
No nondeterministic tests.
Total test coverage should be at least 40%.
Test coverage from small tests alone should be at least 25%.
All significant features are tested by integration tests.

Level 5

Add a test for each nontrivial bug fix.
Actively use available analysis tools.
Total test coverage should be at least 60%.

面试

面试测试的题目，看似简单，但需要优秀候选人更广泛更完备测试思维（挖掘更深更多方面的条件、情况），敢于反思、质疑问题，敢于多一些建议。

ACC (Attribute, Component, Capability)

可用于风险分析，用例设计，用例指派，探索式测试。个人十分行赏。

Example: Determining Attributes, Components, and apabilities for Google+

ACC can be performed quickly in a document, spreadsheet, or even on a napkin! Here is an abbreviated example of ACC for Google+. 

• Google+ Attributes (derived exclusively by watching an executive discuss Google+)

Social: Empowers users to share information and what they're up to.
Expressive: Users can express themselves through the features.
Easy: Intuitive. Easy to figure out how to do what you want to do. 
Relevant: Shows only information the user cares about.
Extensible: Capable of integrating with Google properties and third-party sites and applications.
Private: Users' data won't be shared. 

• Google+ Components (derived from architecture documents)
Profile: Information and preferences for the logged in user.
People: Profiles that the user has connected with.
Stream: Aranked stream of posts, comments, notifications, photos, and so on. 
Circles: Groups to put contacts into "friends," "co-workers," and so on.
Notifications: Indicators for when you are mentioned in a post.
Interests or +1: Indication for user likes.
Posts: Buzz posts from the users and their contacts.
Comments: Comments on posts, photos, videos, and so on.
Photos: Photos uploaded by the users and their contacts.

• Google+ Capabilities:
Profile:

Social: Share profiles and preferences with friends and contacts.
Expressive: Users can create an online version of themselves.
Expressive: Personalize your experience with Google+.
Easy: Easy to enter and update information and have it propagate.
Extensible: Serves profile information to applications with appropriate access.
Private: Enables a user to keep private data private. 
Private: Share data only with approved and appropriate parties.

People:

Social: Users can connect with users' friends, coworkers, and families.
Expressive: Profiles of other users are personalized and easily distinguishable.
Easy: Provides tools to easily manage a user's contacts.
Relevant: Users can filter their contacts based on criteria for relevance.
Extensible: Serves contact data to authorized services and applications.
Private: Keeps data about a user's contacts private to approved parties.

Stream:

Social: Informs the users of updates from their social networks.
Relevant: Filters for updates the user would be interested in.
Extensible: Serves stream updates to services and applications.

Circles:

Social: Groups contacts into circles based on social context.
Expressive: New circles can be created based on the context of the user.
Easy: Facilitates adding, updating, and removing contacts to and from circles.
Easy: Facilitates creating and modifying circles.
Extensible: Serves data about circles for use to services and applications.

Notifications:

Easy: Presents notifications concisely.
Extensible: Posts notifications for use by other services and applications.

Hangouts:

Social: Users can invite their circles to hang out.
Social: Users can open hangouts to the public.
Social: Others are notified of hangouts they access in their streams.
Easy: Hangouts can be created and participated in within a few clicks.
Easy: Video and audio inputs can be disabled with a single click.
Easy: Additional users can be added to an existing hangout.
Expressive: Before joining a hangout, users can preview how they will appear to others.
Extensible: Users can chat through text while in a hangout.
Extensible: Videos from YouTube can be added to a hangout.
Extensible: Devices can be configured and adjusted in Settings.
Extensible: Users without a webcam can participate in hangouts through audio.
Private: Only invited guests can access a hangout.
Private: Only invited guests are notified of a hangout.

Posts:

Expressive: Expresses the thoughts of the user through Buzz.
Private: Posts are restricted to the intended audience.

Comments:

Expressive: Expresses the thoughts of the user through comments.
Extensible: Posts data on comments for use by other services and applications.
Private: Posts are restricted to the intended audience.

Photos:

Social: Users can share their photos with their contacts and friends.
Easy: Users can easily upload new photos.
Easy: Users can easily import photos from other sources.
Extensible: Integration with other photo services.
Private: Photos are restricted so that they're only visible to the intended audience.

谷歌测试工具

Google Test Case Manager (GTCM) - 测试用例管理

Buganizer - 缺陷管理

Google Feedback - 方便报缺陷的系统

Quality Bots Experiment - 对比系统

对比页面变化，在Chrome的浏览器的测试，每个版本浏览器都需要对比N个网页确认没有破坏掉什么功能。
http://code.google.com/p/qualitybots/

BITE

BITE stands for Browser Integrated Test Environment. The Browser Integrated Testing Environment, or BITE, is an open source Chrome extension (http://code.google.com/chrome/extensions/index.html)

• Report a bug
• Viewing bugs
• Record / Playback
• Executing Manual and Exploratory Tests with BITE

Google Test Analytics - Risk Analysis

Support ACC - Link test case / bugs
we're open sourcing Test Analytics (http://code.google.com/p/test-analytics/), a tool built at Google to make generating an ACC simple.

Record/Playback Framework

Record/Playback Framework (RPF) works in the Browser Integrated Test Environment (BITE) (http://googletesting.blogspot.com/2011/10/takebite-out-of-bugs-and-redundant.html).

SETs 和 TEs的区别

SETs' roles and TEs' roles are related but different in fundamental ways. I've been both and managed both. Look at the lists that follow and find which description most fits you—maybe you should switch roles.

You might be an SET if

• You can take a specification, a clean whiteboard, and code up a solid and efficient solution. 
• When you code, you guiltily think of all the unit tests you should be writing. Then, you end up thinking of all the ways to generate the test code and validation instead of hand crafting each unit test. 
• You think an end user is someone making an API call.
• You get cranky if you look at a poorly written API documentation, but sometimes forget why the API is interesting in the first place. 
• You find yourself geeking out with people about optimizations in code or about looking for race conditions.
• You prefer to communicate with other human beings via IRC or comments in check-ins.
• You prefer a command line to a GUI and rarely touch the mouse.
• You dream of machines executing your code across thousands of machines, beating up algorithms, and testing algorithms—showing their correctness through sheer numbers of CPU cycles and network packets.
• You have never noticed or customized your desktop background.
• Seeing compiler warnings makes you anxious.
• When asked to test a product, you open up the source code and start thinking about what needs to be mocked out.
• Your idea of leadership is to build out a great low-level unit test framework that everyone leverages or is highly exercised millions of times a day by a test server.
• When asked if the product is ready to ship, you might just say, "All tests are passing."

You might be a TE if

• You can take existing code, look for errors, and immediately understand the likely failure modes of that software, but don't much care about coding it from scratch or making the change. 
• You prefer reading Slashdot or News.com to reading other people's code all day.
• You read a spec for a product that is half-baked, you take it upon yourself to fill in all the gaps, and just merge this into the document.
• You dream of working on a product that makes a huge impact on people's lives, and people recognize the product you work on.
• You find yourself appalled by some websites' UI and wonder how they could ever have users.
• You get excited about visualizing data.
• You find yourself wanting to talk to humans in meat space.
• You don't understand why you have to type "i" to start typing in a certain text editor.
• Your idea of leadership is nurturing other engineers' ideas and
• challenging their ideas with an order of magnitude more scale.
• When asked if the product is ready to ship, you might say, "I think it's ready."

It is important for testers to figure out who they are. Often, TEs are simply seen as SETs who don't code as much or as well. The reality is that they see things that people with their head in the code all day will never see. SETs should also realize they aren't TEs and let go of any guilt or pressure to find UI issues or think about the system overall or competitive products; focus instead on high quality, testable, and reusable modules and amazing automation. 
It takes a diverse family of testers to raise an amazing product.