Prototype Evaluation 7/27/16

Week 5: Prototype Evaluation 

This week, we first refined our 3D model in Unity by replacing the floor plan with a measurement grid and adding extra walls in the virtual space for participants to manipulate.

Final model for this week's prototype evaluation

Our research questions remain unchanged from the previous week:

• What aspects of our design help collaborators establish common ground efficiently?
• How does our design differ from traditional mediums in terms of interactions?
• What elements of traditional mediums “map” well to a virtual environment?
• What challenges do clients face when using our solution?

We decided that for this round of evaluation, we might not be able to answer the first question since it requires synchronous communication during the design process to evaluate effectively, and our prototype currently does not support networking.  So, we temporarily removed the question from this first round and focused on how our solution helps clients to better understand and communicate spatial concepts to designers. 

For the evaluation, we focused on three major research goals:

• Understand how clients will use our solution to illustrate their ideas
• Identify possible challenges and the learning curve of our solution for further refinement
• Evaluate the core values that our solution brings to clients as well as its advantages and disadvantages

Evaluation Overview

Based on our research goals, we determined the prototype evaluation focus:

• Understand how changing viewpoints between small-scale and full-scale might facilitate clients’ spatial understanding.
• Understand how clients utilize and value object manipulation and 3D sketching functions for changing the spatial layout. 
• Compare the client experience between 2D floor plans and 3D virtual space to identify advantages and disadvantages.

Evaluation Process

We designed a test protocol based on two conditions. Each session evaluated a single participant and took about one hour. Participants performed the same four tasks with both our solution and pen/paper. The order of conditions was counterbalanced. At the end, participants were interviewed about their experience using both mediums.

Evaluation Approach

Recruitment

After the model refinement, we started our recruitment of participants. To ensure the participant group could represent our target users which were project owners/investors, our criteria were that the participants should have a non-architecture background and have experience working with architects/interior designers or experience designing their home/workplace.

Scenario-based

The participants were described a scenario where they are planning a 1B1B one-floor home and just received a design concept from the architect. 

Participants were asked to perform four tasks based on the scenario: 

1. Removing an extra room
2. Separating the living room space
3. Increasing the amount of natural lighting
4. Further decoration of the whole space

Two conditions

Participants completed a series of tasks in both conditions (traditional floorplan & our solution) which were counterbalanced across participants.  This allowed us to compare how their experiences and understanding differed between traditional markup methods and our solution.

User Testing

We recruited four participants based on a set of criteria that helped us approximate the target profile of non-expert clients that might be working with designers to plan a space.  We also accounted for other confounding variables such as gender differences and VR experience.

• 4 participants (2 male, 2 female)
• Non-architecture background
• Has experience planning home or workplace
• Little to no previous VR experience

The testing took place at Startup Hall at the Univesity of Washington. We set up a room-scale VR system and ran one session on Friday afternoon and three sessions on Tuesday.

We used video cameras as our primary method of capture. In the VR task, we recorded the projected screen together with the participant due to the importance of body language in understanding their intentions. The audio was also recorded throughout each session since participants were asked to think aloud as they completed the tasks.

Findings and Implications

Key finding #1

Participants had more satisfying and reactive experiences with the manipulation function in virtual space because it provides instant feedback on the spatial change.

Implication

The real-time feedback and visualization help users feel more satisfied with their design decisions, which could be a core value of our solution.  We should consider the effect of gratification on client involvement.

Key finding #2

Participants were more willing to experiment with different changes and specify the design when they are in VR compared to using pen and paper. Participants found that they easily got lost when sketching on paper due to the permanent nature of the medium.

Implication

Our solution is valuable for encouraging user involvement due to the ease of experiencing changes without the need to commit to permanent modifications.  We should continue to facilitate this kind of interaction and redesign the sketching function to reflect this value.

Key finding #3

The flexibility of jumping between small-scale full-scale views increased participants’ spatial awareness.  In particular, participants used the small-scale view for an overview of the spatial layout much like a traditional floor plan.

Implication

The intuitiveness of the small-scale view transfers from traditional floor plans since it allows users to see the overview of the designed space.  Perhaps we should emphasize this view over the full-scale and refine it to carry over more valuable aspects from floor plans.

Key finding #4

The sense of measurement was more subjective to our participant group. In our prototype, they measured the size and distance (when manipulating objects) relative to other elements in the space instead of referring to the measurement grid.
However, most participants expressed a desire for precise positioning and explicit measurement visualizations.

Implication

Clients considered our solution more as a tool to experience layout and design changes, but it is possible that designers have a different concern when looking at the space. We should investigate more on the need for precise measurement.

Refinements

Based on user's behavior and their comments, we found several user experience and usability issues that we should take into consideration and go through a refinement.

User experience

• Participants found it challenging to sketch 3D concepts precisely to present their ideas.
• Participants found that it was hard for them to measure distance when moving walls and furniture.
• Participants could not tell whether the furniture was the right size
• Participants often made changes while in small-scale view. However, many reported dizziness or neck strain from looking down to perform tasks.

Based on the evaluation, we proposed following solutions to refine the user experience.

• We plan to provide an indicator or automated snap-to-surface function to help users with sketching.
• Dynamic measurements that will pop-out showing the exact distance while moving an object. 
• When pointing to furniture, users can see the dimension information.
• We are lifting the small-scale model up based on participant height and making it adjustable so that they no longer need to keep looking down.

Usability

• Participants sometimes lost their sense of direction and location in VR, especially after teleportation.
• The controller button mapping was often difficult to remember, causing participants to trigger unintended actions.
• It was sometimes difficult for participants to make small manipulations with precision (e.g. to the walls and furniture).

Changes we plan to make to improve our prototype:

• Add reference objects or marks into the space to help participants locate themselves.
• Add labels to the virtual controller model for reference and streamline button interaction.
• Add a snapping function to help align objects to other objects automatically or tweak manipulation interaction.

Next Steps

Evaluate the common ground 

We will present the virtual markup to designers via POV screen recordings of participant sessions and let them interpret the participants’ design intentions. We will also revisit video recordings and listen to participants’ thought processes so we can evaluate the accuracy of interpretation. This will help us determine if mutual understanding is reached and what parts of our solution were effective.

Refine the prototype

We will select the most significant usability problems relative to our topic and refine our prototype for the second round of testing.

Final Takeaways 

Despite our initial assumptions, the 3D sketching function was not as intuitive as we thought.  While most participants thought that the sketching tool was valuable for labeling and marking up the space, it was often challenging to use.

Another assumption was that our solution would provide an increased sense of scale. However, it was compromised by technical limitations in our prototype. It would be ideal to maximize the real-size feeling of the full-scale view by increasing the freedom for participants to walk around in the space so they would not have to rely on the teleport function quite as often.

Reflection

Our evaluation helped validate our design solution as participants expressed enthusiasm for the value of real-time 3D markup. Participants’ overall positive feedback on the manipulation function made us realize how instant visual feedback can help participants experiment with and understand spatial concepts to inform their decisions.  While rough, our initial prototype was functional enough to be able to help us evaluate our research questions and reveal specific improvements for refinement.  We are confident moving forward with our solution in the final weeks.

Prototyping 07/20/16

Week 3 & 4: Prototyping

During the last two weeks, we were focused on detailing the interaction flows of our final concept and planning our prototype evaluation. We made our decision to focus on the asynchronous collaboration experience between clients and designers based on two reasons. First, asynchronous communication is the most common channel in remote collaboration context. Second, it is easier to implement into prototype due to time and technical constrains.

The record of the first prototyping meeting.

Interaction Flow

To understand how our target user will use our solution, we created an interaction flowchart to show all possible interactions. 

The whole concept contains three main interaction flows, which are synchronous collaboration in the same virtual space, synchronous collaboration in separate spaces, and asynchronous collaboration between clients and designers. The asynchronous collaboration experience will be the main focus for our prototyping evaluation. 

Although we are still in the process of finalizing our research protocol, we have identified the big picture and direction for our prototype testing tasks and evaluation.

Prototyping Overview

After consulted with instructors, we began to discuss about which technology should be used to build our solution. In the meanwhile, we also brainstormed some research questions that we would like to answer.

Meeting notes from the second meeting

During our third meeting, we further put down all the requirements and functions that should be included in our prototype and split the work so we could start working. 

The second meeting mainly focused on two parts, which technology should we use, and who should be our participants.Since we are going to focus on client's needs, we determined that our target participants should be:

• Non-architecture background
• Has experience decorating home/workspace
• On the market for new houses or apartments (either currently or in the past year)
• Has experience working with architects or interior designers to design a space

And we decided that our prototype should have three basic functions.

• 3D sketching
• Object manipulation (translation and rotation)
• Full-scale and small-scale viewpoints

Further details of the prototype was discussed during our third meeting. We discussed and listed our all the details including where to get the resources, how will each function being used, how should we test our prototype, and how our prototype will answer our research questions.

Research Question Formation

On Wednesday and Thursday, we had a meeting with the instructors and sponsors. We presented our ideas and collected feedback, some feedback we got included

• Focus on big picture rather than interaction details
• Emphasize synchronous collaboration
• We should consider Wizard of Oz method

The meeting helped us revisiting our design concept and forced us to think about what is our design hypothesis, and how are we going to proof that our solution is better than the traditional method.

Design Concept

Based on the concept we discussed during the concept refinement phase, our solution is a model-based interface shared between remote collaborators that facilitates dialog and spatial understanding through the establishment of common ground.

Furthermore, stakeholders share the same virtual environment and experience different viewpoints to gain a full sense of scale while allowing real-time manipulation of the model.

Design hypothesis

Our design provides clients with 3D markup tools in a full-scale virtual space to help them better understand and communicate spatial concepts to designers.

Research Questions:

• What aspects of our design help collaborators establish common ground efficiently?
• How does our design differ from traditional mediums in terms of interactions?
• What elements of traditional mediums “map” well to a virtual environment?
• What challenges do clients face when using our solution?

Research Goals:

• Understand how clients will use our solution to illustrate their idea
• Understand possible challenges and learning curve of our solution
• Understand how successfully the solution helps clients communicate their design intentions to designers

Evaluation Plan

After we got our design hypothesis and research questions ready, we started to put down more details of how are we going to test the prototype.

Research Protocol:

1.Design with pen and paper on 2D medium

An empty floor plan and a list of requirements will be provided to the participant, and they will be asked to leave markup on the spatial arrangement on paper.

2. Design in VR

The client will first go through a tutorial to learn the basic controls in VR, then be placed in an empty virtual room with a list of requirements. They can sketch in 3D, manipulate objects, and jump between viewpoints to communicate their ideas.

3. Debrief and evaluation

The client will be asked to compare their experiences between using traditional pen & paper on 2D floor plan and using our solution. We want to understand what tools they prefer and why as well as any challenges they encountered during the session.

4. Expert evaluation

The client’s markup will be shown to the designer. The designer will be asked to evaluate the two different mediums to help us understand what aspects of our solution are the most effective and whether our solution helps the formation of common ground.

Documentation Methods: 

We plan to record video of the participants as well as screen recordings to capture the complete point of view in virtual space. We will also take notes during the sessions to record any particularly interesting insights.

Materials: 

For the prototype, we have made a floor plan of a 1B1B one-floor interior, and also built the virtual environment of the same interior in Unity. Both of them will be used for the testing in order to compare the 2D and 3D methods.

Prototype Demo

Finally, we had our pre-alpha version prototype ready. Although it is still pretty rough, we got all the functions we need for testing. The next step is to run some pilot study and start recruiting participants to evaluate our solution.

Reflection

Our prototyping phase began with extensive discussion to figure out what part of our solution should be built and evaluated.  While this was difficult to determine initially, our time constraints and interaction model made it clear that we should focus on asynchronous collaboration for our initial prototype.  Building our prototype in Unity was relatively straightforward and we were able to build the virtual environment and implement the basic functionality over a weekend.  Getting this first iteration completed also made it clear what features still needed to be added in preparation for our user tests.  Instructor and peer feedback on our presentation was positive and we are confident in moving forward with our prototype.

Concept Refinement 7/6/16

Week 2: Concept Refinement

In week 2, we further narrowed down our ideation to one concept. We revisited the problem space and decided that the core value of our solution is to "help form common ground between designers and project owners." Based on this value and our previous investigation, we created a concept image, storyboard, interaction model, and system diagram to help us convey the idea.

Refined Project Focus

Facilitate client contribution in early stages of the spatial design process to help the formation of common ground between designers and project owners.

Background

Sharing common ground (i.e. mutual knowledge, beliefs, and assumptions) is essential for communication between collaborators (Clark, H.H. & Brennan, S. E., 1991). However, architects and project owners working in a remote context face two major challenges.

First, it can be difficult for collaborators without spatial design backgrounds to fully understand design intentions. Differences of expertise in project owners make it easy to misunderstand 2D artifacts such as floor plans and images, or how the final design might look after being built. This kind of miscommunication can be extremely costly in the long run.

The second challenge is ensuring that people are referring to the same objects or locations during discussions. With gesture and gaze, referential communication diminishes the effort needed for mutual understanding during face-to-face communication. However, in a remote context, people might not be able to rely on these non-verbal cues to establish common ground. Remote collaborators need to spend more effort to get on the same page, resulting in inefficiency and possible misunderstanding.

Concept Overview

A model-based interface shared between remote collaborators that facilitates dialog and spatial understanding through the establishment of common ground. Stakeholders share the same virtual environment and experience asymmetrical viewpoints to gain a full sense of scale while allowing real-time manipulation of the model. 

Target User

Project owners and investors 

Architects and interior designers

Problem Space

Most project owners or investors have a difficult time understanding space and sense of scale from 2D images of design concepts such as floor plans. On the other hand, it is also difficult for designers to fully capture client needs and expectations by referring solely on text-based markup without more in-depth social cues. The communication of design feedback through traditional channels such as email also causes considerable delays to the project schedule.

Value

Remote collaborators can establish common ground by breaking technical barriers and utilizing referential communication. The solution facilitates the design process by presenting a modular experience with tools to help visualize non-verbal cues.

Interaction Model

The model/space itself stands as the common interface for both designers and project owners. Designers manipulate the model in the small scale while project owners experience the updated changes in the full scale. Designers can understand project owners’ attention and needs by sharing their gaze, reading their gesture and listening to their voice.

System Diagram

The system diagram maps out the stakeholders involved in the design process and their relationship with other components in the system. Steps 1 to 7 indicate the general design process and show how the solution is integrated into the system.

Reflection

Overall, we feel confident about our design direction and will push forward with prototyping the concept.  After spending a long time debating as a group on whether to present several concept variations or just a single concept, we found that the single concept stood better on its own than several incomplete variations.  Instructor and peer feedback brought up the issue of asynchronous vs. synchronous collaboration and how our solution might continue to function in the project workflow.  As we begin prototyping, we plan to pick a smaller function of the interaction to build in order to evaluate it.

Concept Development 6/27/16

Week 1: Ideation

The goal of this week is to ideate potential design direction based on our secondary and primary research. The technique we used for ideation includes design prompts generation, interaction, individual ideation, wishing, and group ideation.

Design Prompts

Design prompts were the core of our ideation, by generating design prompts, we had a better understanding of what kinds of problem are we try to solve, who are the potential user, and how are we going to solve it.

The design prompts we used were:

1. 
   
2. How might we empower clients to feel like they are part of the design team?
3. How do we elicit emotional connections between stakeholders and the project?
4. How can we make remote interactions feel more dynamic and personal?
5. How can we make complicated design tools intuitive enough for non-experts to pick up and use?
6. How might we help architects connect with clients on an emotional level in the absence of explicit social cues?
7. How can we minimize the technical language barrier for clients? 
8. How can we help stakeholders transition between 2D and 3D perspectives for better understanding of spatial flow?
9. How can we help clients view the design concept like an interactive story?

Ideation Process Overview

We started our ideation with design prompts and interaction map among stakeholders. After the analysis, we moved to individual ideation by coming up with 20 concepts from each person. The second ideation method was "wishing" which enabled us to generate bold and unrealistic ideas, group them under different themes and develop them into more realistic concepts. At the end, we group all the concepts into nine themes, combined overlapping ideas, and picked five design concepts which were most interesting and promising to us.

Interaction Map

Individual Ideation

Generating ideas individually is an efficient way to start the process since we might have different interpretations of the problem and expectations of how the problem might be solved. This also allowed us ideate without being interfered by others’ opinions. 

Process

1. Each team member generated 20 ideas with visual and verbal descriptions overnight
2. We had group discussion where each of us walked through each idea
3. The ideas were grouped and labeled based on emergent themes

Individual ideas

Wishing

Empathy is essential to user-centered design. We used a “wishing” approach to come out with unrealistic but valuable “wishes” based on stakeholder perspectives.  This gave us a diverse range of themes that were unrestricted by realism.  These wishes were then turned into more feasible solutions through group ideation.

Process

1. We tried to put ourselves into the architect’s and client’s shoes, and generated ideas based on “I wish we had... “ or “I wish I could”
2. Wishes were grouped into several categories based on similar themes.
3. “Realistic” ideas were then generated to try to fulfill the wishes.

"Wishing"

Group Ideation

We conducted a group ideation session to organize all of our individual ideas and to form our potential design directions. We took our 60 ideas and grouped some of them while combining others to arrive at five design directions.

Process

1. Each team member generated 20 ideas with visual and verbal descriptions overnight.
2. All ideas were laid out on a table for group discussion.
3. The ideas were grouped and merged to form themes and potential design directions.

Grouping and combining ideas

Based on grouping and combing out ideas, we summarized five design concepts, which are:

1. Increasing client involvement in the design process by providing intuitive toolkits.
2. Improving spatial understanding for non-experts by translating 2D concepts to 3D.
3. Improving understanding of static concepts through simulations of real-life scenarios.
4. Increasing the viability of implicit social cues for a more emotional connection between stakeholders.
5. Providing or creating 3D visualization of pre-existing spaces to facilitate communication of design intentions.

And for each design direction, we creates a storyboard to show how we might solve the problem.

Design Concepts

Concept One: Design Toolkit

Intuitive design kit in AR or VR format for clients to manipulate virtual 3D models.

Problem space

Clients are not trained in spatial design so it is often a challenge for them to explain what a building should look like to the architect. It might take weeks to wait for architects to rebuild a model when it does not meet the client’s expectation.

How it works

After architects import models into the design toolkit, the model will be automatically broken down into smaller components. Clients can move components around just like playing with Legos to express their design intention.

Target user

Clients

Value

The design kit provides and easy way for clients to play around with the model without learning how to use the traditional software, and it also provides instant feedback on how the space will look like after changes.

Concept Two: 3D Model Reviewer

Virtual space where clients can make 3D sketches and comments on the design concepts, and experience real-time update of the space design based on their request. 

Problem Space

For clients, most current solutions either present the model in 2D or in low-fidelity 3D that could not truly represent the design. Furthermore, clients are required to learn how to use the review software to leave comments. For designers, it can also be challenging for them to understand the comments and 2D sketches left by the client.

How it works

Clients can leave sketches and comments on the space, and see real-time updates in the space based on their request.

Target user

Clients, designers

Value

The solution provides an easy way for clients to review and comment on the 3D model. Designers can update the model immediately without needing a back and forth communication just to clarify what clients want.

Concept Three: Real-life Scenario Simulator

Animation-based simulation system that helps clients understand the spatial flow through real-life scenarios.

Problem space

Lots of money is spent to change or revising designs when clients find out that a room actually doesn’t work for what they intended.

How it works

By experiencing spatial flow in real-life scenarios, clients can better understand how the space might function. The solution can also provide a preview of how the design might look in different scenarios (raining days, at night, or 30 years later).

Target user

Clients

Value

The solution allows client to have a complete overview of how the room or building will look after being built. Since clients rely heavily on narrative storytelling as a means of understanding space, these solutions help communication during the design process.

Concept Four: Non-verbal Behavior Translator

Visualization of social cues that helps connect collaborators through understanding of emotion and common ground in the remote context.

Problem space

Designers rely heavily on non-verbal expressions so they prefer to meet in-person to tell whether the client likes the design or not. In the remote context, these social cues are often unavailable. For example, collaborators could not tell whether they are pointing to the same object during the discussion.

How it works

With gestural tracking together with attention and gaze information, the solution can record and translate how the client feels or which part of the design they are referencing in to the architect in real-time. With this, the designer could understand the non-verbal expression under remote context just like they are meeting them in-person.

Target user

Client-facing professionals

Value

The solution provides non-verbal social cues to designers so they are able to tell whether the client likes the design or not with the provided information, particularly in remote contexts. It could also ensure that designers and clients are establishing common ground during the discussion.

Concept Five: 3D Reference Database

 Providing or creating a 3D library that consists of visualizations of pre-existing spaces to facilitate communication of design intentions.

Problem space

Finding reference images is challenging, especially for non-experts. However, it is also challenging for them to express how they imagine the space, which architects are often struggling with to figure out exactly what the client wants.

How it works

The solution provides a database with a variety of indexed pre-existing designs. All designs are presented in 3D with environmental context so that clients and designers can use them as references.

Target user

Clients and architects

Value

The solution provides an easy way for both clients and architects to present their ideas. Architects can simply set up the 3D space to show to the client, and clients no longer need to try to find reference images to convey how they want the building to look.

Reflection

Although we tried to generate the idea as broad as possible, we were still somehow limited to some of the specific domains. In the ideation process, we encourage ourselves to go wild and go crazy, which we are going to keep ideate some more compelling and creative potential solutions. Also, we still had not clearly define the user and the value of each concept that it will be the main focus next week.